Repository: NVIDIA/NeMo
Pull request: Audio Norm
- added jenkins tests for the normalization_with_audio script (text only, text + audio, manifest)
- all default normalization unit tests will be executed for normalization with audio as well
- added serial class to the default normalization (deterministic): using single digits by default
- addressed [this](https://github.com/NVIDIA/NeMo/issues/2266) and partially [this](https://github.com/NVIDIA/NeMo/issues/2267) (`k` -> `kay` is not supported yet)
- more verbalization options added for cardinals and whitelist
- now all possible verbalization options are provided not only the first shortest 100

Problem statement:
Normalizer and NormalizerWithAudio fail with dashed tokens eg "550-pound bomb"
**Describe the bug**

Example sentence fragments for text normalizer:
"It had 12-inch guns and a 21-knot cruising speed."
"Each 550-pound bomb exploded."

**Steps/Code to reproduce bug**

```python
from nemo_text_processing.text_normalization.normalize import Normalizer
from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio

normalizer = Normalizer(input_case='cased')
audio_normalizer = NormalizerWithAudio(input_case='cased')

texts = [
    "It had 12-inch guns and a 21-knot cruising speed.",
    "Each 550-pound bomb exploded.",
]

for text in texts:
    normalized_text = normalizer.normalize(text, verbose=False)
    print(normalized_text)

    normalized_texts = audio_normalizer.normalize_with_audio(text, verbose=False)
    print(normalized_texts)
```
The results from normalized_texts will be something like (notice the spaces between each letter in the word for the units):
"five hundred fifty p o u n d bomb"
and
"twenty one k n o t cruising speed"

It seems like the sparrow/pynini rules don't allow for dashed text input followed by a unit and interpret the unit as an abbreviation. The CER calculation in NormalizerWithAudio doesn't repair this because the results from normalize_with_audio() don't ever contain the expected outputs.

**Expected behavior**

"five hundred fifty pound bomb"
"twenty one knot cruising speed"

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip install from source, main branch

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version Ubuntu 20.10
- PyTorch version 1.7.1
- Python version 3.8.5

I'm trying to track down where in pynini or the nemo rules this is happening and if I find it I will follow-up.

NormalizerWithAudio  - Allow plural Serial verbalizers
**Is your feature request related to a problem? Please describe.**

Pluralized verbalizers of Serial objects are not available in the results from NormalizerWithAudio.normalize_with_audio(), but a large number of plural serials exist in English eg "B-52s" should allow for a verbalizer of "B fifty twos" in addition to "B fifty two S".  The correctly pluralized serial should be discernible with the CER and audio transcript. 

These appear often in the financial, military, and other realms (think "401ks" "1099s" "W2s" etc).

**Describe the solution you'd like**

Add a parameter to the Serial FST. If it ends in "s" (or a plural form), offer a plural form of serial[-2] to the verbalizers as an option (and drop serial[-1]).

This may be non-trivial when serials end in a letter eg "401k" --> "four oh one kays" may be more useful than "four oh one ks"

**Describe alternatives you've considered**

The usual regex preprocessing which should be more efficiently implemented in FSTs?

**Additional context**

```python
from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio

audio_normalizer = NormalizerWithAudio(input_case='cased')

texts = [
    "Twelve B-17s were totally destroyed.",
    "8 401ks were emptied.",
]

for text in texts:
    normalized_texts = audio_normalizer.normalize_with_audio(text, verbose=False)
    print('\n'.join([x for x in normalized_texts]))

    assert "eight four oh one kays were emptied." in normalized_texts or "Twelve B seventeens were totally destroyed" in normalized_texts
```

Hints from issue discussion:
The taggers pick these examples up as Serials and not as Measures. The options with the spaces between the letters make sense if it's a Serial.

I think a modification to the tagger for Measure that allows the dash between Cardinal and Unit may do the trick. If I can figure out the syntax...
Yes, this is a known issue, the fix is a part of the upcoming update. 
I'll update this bug when the PR is ready.
I'll be following the progress closely and am happy to help if I can get more familiar with pynini -- is this repo the best place to file / discuss similar issues?

Commit message: add serial to the default normalizer, add tests

Signed-off-by: ekmb <ebakhturina@nvidia.com>

Patch to review:
diff --git a/Jenkinsfile b/Jenkinsfile
index ba2e3606e5a5..3daec034db46 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -167,6 +167,24 @@ pipeline {
             sh 'rm -rf /home/TestData/nlp/text_denorm/output/*'
           }
         }
+        stage('L2: TN with Audio (audio and raw text)') {
+          steps {
+            sh 'cd nemo_text_processing/text_normalization && \
+            python python normalize_with_audio.py --text "The total amounts to \$4.76." \
+            --audio_data /home/TestData/nlp/text_norm/audio_based/audio.wav | tail -n 1 > /home/TestData/nlp/text_norm/audio_based/output/out_raw.txt 2>&1 && \
+            cmp --silent /home/TestData/nlp/text_norm/audio_based/output/out_raw.txt /home/TestData/nlp/text_norm/audio_based/result.txt || exit 1'
+            sh 'rm -rf /home/TestData/nlp/text_norm/audio_based/output/out_raw.txt'
+          }
+        }
+        stage('L2: TN with Audio (audio and text file)') {
+          steps {
+            sh 'cd nemo_text_processing/text_normalization && \
+            python python normalize_with_audio.py --text /home/TestData/nlp/text_norm/audio_based/text.txt \
+            --audio_data /home/TestData/nlp/text_norm/audio_based/audio.wav | tail -n 1 > /home/TestData/nlp/text_norm/audio_based/output/out_file.txt 2>&1 && \
+            cmp --silent /home/TestData/nlp/text_norm/audio_based/output/out_file.txt /home/TestData/nlp/text_norm/audio_based/result.txt || exit 1'
+            sh 'rm -rf /home/TestData/nlp/text_norm/audio_based/output/out_file.txt'
+          }
+        }
       }
     }
 
diff --git a/nemo_text_processing/text_normalization/data/suppletive.tsv b/nemo_text_processing/text_normalization/data/suppletive.tsv
index dea620f37ab3..be59872c28d9 100644
--- a/nemo_text_processing/text_normalization/data/suppletive.tsv
+++ b/nemo_text_processing/text_normalization/data/suppletive.tsv
@@ -34,4 +34,5 @@ revolution per minute	revolutions per minute
 mile per hour	miles per hour
 megabit per second	megabits per second
 square foot	square feet
-kilobit per second	kilobits per second
\ No newline at end of file
+kilobit per second	kilobits per second
+lb
\ No newline at end of file
diff --git a/nemo_text_processing/text_normalization/data/whitelist_alternatives.tsv b/nemo_text_processing/text_normalization/data/whitelist_alternatives.tsv
index ffb60fceb1c4..32f896aa8d95 100644
--- a/nemo_text_processing/text_normalization/data/whitelist_alternatives.tsv
+++ b/nemo_text_processing/text_normalization/data/whitelist_alternatives.tsv
@@ -1,17 +1,11 @@
-II.	the Second
-II	Second
-III.	the Third
-III	Third
-IV.	the Fourth
-IV	Fourth
-VIII.	the Eighth
-Hon.	honorable
-Hon.	honourable
-St.	street
-St	street
-St.	saint
-St	saint
-Dr.	drive
-Dr.	doctor
-Mr	mister
-Mrs	misses
\ No newline at end of file
+Hon.	Honorable
+Hon.	Honourable
+St.	Street
+St	Street
+St.	Saint
+St	Saint
+Dr.	Drive
+Dr.	Doctor
+Mr.	Mister
+Mrs.	Misses
+Ms.	Miss
\ No newline at end of file
diff --git a/nemo_text_processing/text_normalization/normalize_with_audio.py b/nemo_text_processing/text_normalization/normalize_with_audio.py
index 880606718633..f3ae0e48f04e 100644
--- a/nemo_text_processing/text_normalization/normalize_with_audio.py
+++ b/nemo_text_processing/text_normalization/normalize_with_audio.py
@@ -15,6 +15,7 @@
 import json
 import os
 import re
+import time
 from argparse import ArgumentParser
 from typing import List, Tuple
 
@@ -28,6 +29,7 @@
 
 try:
     import pynini
+    from pynini.lib import rewrite
 
     PYNINI_AVAILABLE = True
 except (ModuleNotFoundError, ImportError):
@@ -76,14 +78,13 @@ def __init__(self, input_case: str):
         self.tagger = ClassifyFst(input_case=input_case, deterministic=False)
         self.verbalizer = VerbalizeFinalFst(deterministic=False)
 
-    def normalize_with_audio(self, text: str, verbose: bool = False) -> str:
+    def normalize(self, text: str, verbose: bool = False) -> str:
         """
         Main function. Normalizes tokens from written to spoken form
             e.g. 12 kg -> twelve kilograms
 
         Args:
             text: string that may include semiotic classes
-            transcript: transcription of the audio
             verbose: whether to print intermediate meta information
 
         Returns:
@@ -94,39 +95,35 @@ def normalize_with_audio(self, text: str, verbose: bool = False) -> str:
             if verbose:
                 print(text)
             return text
-        text = pynini.escape(text)
-
-        def get_tagged_texts(text):
-            tagged_lattice = self.find_tags(text)
-            tagged_texts = self.select_all_semiotic_tags(tagged_lattice)
-            return tagged_texts
 
-        tagged_texts = set(get_tagged_texts(text))
+        text = pynini.escape(text)
+        # TODO add preprocess?
+        tagged_texts = rewrite.top_rewrites(text, self.tagger.fst, nshortest=100)
         normalized_texts = []
 
         for tagged_text in tagged_texts:
+            print(tagged_text)
             self.parser(tagged_text)
             tokens = self.parser.parse()
             tags_reordered = self.generate_permutations(tokens)
             for tagged_text_reordered in tags_reordered:
+                print(tagged_text_reordered)
                 tagged_text_reordered = pynini.escape(tagged_text_reordered)
-
-                verbalizer_lattice = self.find_verbalizer(tagged_text_reordered)
-                if verbalizer_lattice.num_states() == 0:
+                try:
+                    verbalized = rewrite.rewrites(tagged_text_reordered, self.verbalizer.fst)
+                    normalized_texts.extend(verbalized)
+                except pynini.lib.rewrite.Error:
                     continue
 
-                verbalized = self.get_all_verbalizers(verbalizer_lattice)
-                for verbalized_option in verbalized:
-                    normalized_texts.append(verbalized_option)
-
         if len(normalized_texts) == 0:
             raise ValueError()
-
-        normalized_texts = [post_process(t) for t in normalized_texts]
+        normalized_texts = [post_process(t) for t in normalized_texts] + normalized_texts
         normalized_texts = set(normalized_texts)
         return normalized_texts
 
-    def select_best_match(self, normalized_texts: List[str], transcript: str, verbose: bool = False):
+    def select_best_match(
+        self, normalized_texts: List[str], transcript: str, verbose: bool = False, remove_punct: bool = False
+    ):
         """
         Selects the best normalization option based on the lowest CER
 
@@ -134,11 +131,12 @@ def select_best_match(self, normalized_texts: List[str], transcript: str, verbos
             normalized_texts: normalized text options
             transcript: ASR model transcript of the audio file corresponding to the normalized text
             verbose: whether to print intermediate meta information
+            remove_punct: whether to remove punctuation before calculating CER
 
         Returns:
             normalized text with the lowest CER and CER value
         """
-        normalized_texts = calculate_cer(normalized_texts, transcript)
+        normalized_texts = calculate_cer(normalized_texts, transcript, remove_punct)
         normalized_texts = sorted(normalized_texts, key=lambda x: x[1])
         normalized_text, cer = normalized_texts[0]
 
@@ -149,16 +147,6 @@ def select_best_match(self, normalized_texts: List[str], transcript: str, verbos
             print('-' * 30)
         return normalized_text, cer
 
-    def select_all_semiotic_tags(self, lattice: 'pynini.FstLike', n=100) -> List[str]:
-        tagged_text_options = pynini.shortestpath(lattice, nshortest=n)
-        tagged_text_options = [t[1] for t in tagged_text_options.paths("utf8").items()]
-        return tagged_text_options
-
-    def get_all_verbalizers(self, lattice: 'pynini.FstLike', n=100) -> List[str]:
-        verbalized_options = pynini.shortestpath(lattice, nshortest=n)
-        verbalized_options = [t[1] for t in verbalized_options.paths("utf8").items()]
-        return verbalized_options
-
 
 def calculate_cer(normalized_texts: List[str], transcript: str, remove_punct=False) -> List[Tuple[str, float]]:
     """
@@ -172,7 +160,7 @@ def calculate_cer(normalized_texts: List[str], transcript: str, remove_punct=Fal
     """
     normalized_options = []
     for text in normalized_texts:
-        text_clean = text.replace('-', ' ').lower().strip()
+        text_clean = text.replace('-', ' ').lower()
         if remove_punct:
             for punct in "!?:;,.-()*+-/<=>@^_":
                 text_clean = text_clean.replace(punct, " ")
@@ -191,10 +179,12 @@ def pre_process(text: str) -> str:
 
     Returns: text with spaces around punctuation marks
     """
+    print(text)
     text = text.replace('--', '-')
     space_right = '!?:;,.-()*+-/<=>@^_'
     space_both = '-()*+-/<=>@^_'
 
+    # TODO
     for punct in space_right:
         text = text.replace(punct, punct + ' ')
     for punct in space_both:
@@ -215,8 +205,7 @@ def post_process(text: str, punctuation='!,.:;?') -> str:
     Returns: text with normalized spaces and quotes
     """
     text = (
-        text.replace('--', '-')
-        .replace('( ', '(')
+        text.replace('( ', '(')
         .replace(' )', ')')
         .replace('  ', ' ')
         .replace('”', '"')
@@ -266,16 +255,17 @@ def parse_args():
         '--model', type=str, default='QuartzNet15x5Base-En', help='Pre-trained model name or path to model checkpoint'
     )
     parser.add_argument("--verbose", help="print info for debugging", action='store_true')
+    parser.add_argument("--remove_punct", help="remove punctuation before calculating cer", action='store_true')
     return parser.parse_args()
 
 
 def normalize_manifest(args):
     """
     Args:
-        manifest: path to .json manifest file.
+        args.audio_data: path to .json manifest file.
     """
     normalizer = NormalizerWithAudio(input_case=args.input_case)
-    manifest_out = args.audio_data.replace('.json', '_nemo_wfst.json')
+    manifest_out = args.audio_data.replace('.json', '_normalized.json')
     asr_model = None
     with open(args.audio_data, 'r') as f:
         with open(manifest_out, 'w') as f_out:
@@ -288,11 +278,12 @@ def normalize_manifest(args):
                     if asr_model is None:
                         asr_model = get_asr_model(args.model)
                     transcript = asr_model.transcribe([audio])[0]
-                normalized_texts = normalizer.normalize_with_audio(line['text'], args.verbose)
-                normalized_text, cer = normalizer.select_best_match(normalized_texts, transcript, args.verbose)
-
-                line['nemo_wfst'] = normalized_text
-                line['CER_nemo_wfst'] = cer
+                normalized_texts = normalizer.normalize(line['text'], args.verbose)
+                normalized_text, cer = normalizer.select_best_match(
+                    normalized_texts, transcript, args.verbose, args.remove_punct
+                )
+                line['nemo_normalized'] = normalized_text
+                line['CER_nemo_normalized'] = cer
                 f_out.write(json.dumps(line, ensure_ascii=False) + '\n')
     print(f'Normalized version saved at {manifest_out}')
 
@@ -300,18 +291,25 @@ def normalize_manifest(args):
 if __name__ == "__main__":
     args = parse_args()
 
+    start = time.time()
     if args.text:
         normalizer = NormalizerWithAudio(input_case=args.input_case)
         if os.path.exists(args.text):
             with open(args.text, 'r') as f:
-                args.text = f.read()
-        normalized_texts = normalizer.normalize_with_audio(args.text, args.verbose)
-        for norm_text in normalized_texts:
-            print(norm_text)
+                args.text = f.read().strip()
+        normalized_texts = normalizer.normalize(args.text, args.verbose)
         if args.audio_data:
             asr_model = get_asr_model(args.model)
             transcript = asr_model.transcribe([args.audio_data])[0]
-            normalized_text, cer = normalizer.select_best_match(normalized_texts, transcript, args.verbose)
+            normalized_text, cer = normalizer.select_best_match(
+                normalized_texts, transcript, args.verbose, args.remove_punct
+            )
+            print(f'Transcript: {transcript}')
+            print(f'Normalized: {normalized_text}')
+        else:
+            print('Normalization options:')
+            for norm_text in normalized_texts:
+                print(norm_text)
     elif not os.path.exists(args.audio_data):
         raise ValueError(f'{args.audio_data} not found.')
     elif args.audio_data.endswith('.json'):
@@ -322,3 +320,4 @@ def normalize_manifest(args):
             + "'--audio_data' path to audio file and '--text' path to a text file OR"
             "'--text' string text (for debugging without audio)"
         )
+    print(f'Execution time: {round((time.time() - start)/60, 2)} min.')
diff --git a/nemo_text_processing/text_normalization/taggers/cardinal.py b/nemo_text_processing/text_normalization/taggers/cardinal.py
index 7f9f821129ea..24f8bf1ac95a 100644
--- a/nemo_text_processing/text_normalization/taggers/cardinal.py
+++ b/nemo_text_processing/text_normalization/taggers/cardinal.py
@@ -69,8 +69,23 @@ def __init__(self, deterministic: bool = True):
                     1,
                 )
             )
-
-            self.graph = self.graph | self.single_digits_graph | get_hundreds_graph() | single_digits_graph_with_commas
+            range_graph = (
+                self.graph
+                + (
+                    pynini.cross("-", " to ")
+                    | pynini.cross("-", " ")
+                    | pynini.cross("x", " by ")
+                    | pynini.cross(" x ", " by ")
+                )
+                + self.graph
+            )
+            self.graph = (
+                self.graph
+                | self.single_digits_graph
+                | get_hundreds_graph()
+                | single_digits_graph_with_commas
+                | range_graph
+            )
 
         optional_minus_graph = pynini.closure(pynutil.insert("negative: ") + pynini.cross("-", "\"true\" "), 0, 1)
         final_graph = optional_minus_graph + pynutil.insert("integer: \"") + self.graph + pynutil.insert("\"")
diff --git a/nemo_text_processing/text_normalization/taggers/measure.py b/nemo_text_processing/text_normalization/taggers/measure.py
index 1c4f87566c8f..d9fbf26552e4 100644
--- a/nemo_text_processing/text_normalization/taggers/measure.py
+++ b/nemo_text_processing/text_normalization/taggers/measure.py
@@ -16,6 +16,7 @@
 from nemo_text_processing.text_normalization.data_loader_utils import get_abs_path
 from nemo_text_processing.text_normalization.graph_utils import (
     NEMO_NON_BREAKING_SPACE,
+    NEMO_NOT_SPACE,
     NEMO_SIGMA,
     SINGULAR_TO_PLURAL,
     GraphFst,
@@ -101,6 +102,25 @@ def __init__(self, cardinal: GraphFst, decimal: GraphFst, deterministic: bool =
             + pynutil.insert(" } ")
             + unit_singular
         )
-        final_graph = subgraph_decimal | subgraph_cardinal
+
+        subgraph_cardinal_dash = (
+            pynutil.insert("cardinal { integer: \"")
+            + cardinal.single_digits_graph
+            + pynini.cross('-', '')
+            + pynutil.insert("\" } units: \"")
+            + pynini.closure(NEMO_NOT_SPACE, 1)
+            + pynutil.insert("\"")
+        )
+
+        subgraph_decimal_dash = (
+            pynutil.insert("decimal { ")
+            + decimal.final_graph_wo_negative
+            + pynini.cross('-', '')
+            + pynutil.insert(" } units: \"")
+            + pynini.closure(NEMO_NOT_SPACE, 1)
+            + pynutil.insert("\"")
+        )
+
+        final_graph = subgraph_decimal | subgraph_cardinal | subgraph_cardinal_dash | subgraph_decimal_dash
         final_graph = self.add_tokens(final_graph)
         self.fst = final_graph.optimize()
diff --git a/nemo_text_processing/text_normalization/taggers/serial.py b/nemo_text_processing/text_normalization/taggers/serial.py
index 1447a88ccf28..39e3a1117504 100644
--- a/nemo_text_processing/text_normalization/taggers/serial.py
+++ b/nemo_text_processing/text_normalization/taggers/serial.py
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_text_processing.text_normalization.graph_utils import NEMO_ALPHA, GraphFst, delete_space
+from nemo_text_processing.text_normalization.graph_utils import NEMO_ALPHA, NEMO_NOT_SPACE, GraphFst, delete_space
 
 try:
     import pynini
@@ -40,18 +40,22 @@ class SerialFst(GraphFst):
             for False multiple transduction are generated (used for audio-based normalization)
     """
 
-    def __init__(self, cardinal: GraphFst, deterministic: bool = False):
+    def __init__(self, cardinal: GraphFst, deterministic: bool = True):
         super().__init__(name="serial", kind="classify", deterministic=deterministic)
 
-        num_graph = cardinal.graph
+        if deterministic:
+            num_graph = cardinal.single_digits_graph
+        else:
+            num_graph = cardinal.graph
         serial_graph_cardinal_start = (
             pynini.closure((NEMO_ALPHA + pynutil.insert(" ")) | (NEMO_ALPHA + pynini.cross('-', ' ')), 1) + num_graph
         )
         serial_end = pynini.closure(pynutil.insert(" ") + NEMO_ALPHA + pynini.closure(pynutil.insert(" ") + num_graph))
 
         serial_graph_cardinal_end = num_graph + (
-            (pynutil.insert(" ") + NEMO_ALPHA) | (pynini.cross('-', ' ') + NEMO_ALPHA)
+            pynini.closure(pynutil.insert(" ") + NEMO_ALPHA) | (pynini.cross('-', ' ') + NEMO_ALPHA)
         )
+
         serial_end2 = pynini.closure(
             pynutil.insert(" ")
             + num_graph
@@ -62,12 +66,12 @@ def __init__(self, cardinal: GraphFst, deterministic: bool = False):
             serial_end | serial_end2
         )
 
-        graph = (
-            pynutil.insert("cardinal { integer: \"")
-            + serial_graph
-            + delete_space
-            + pynutil.insert("\" } units: \"serial\"")
-        )
+        graph = pynutil.insert("cardinal { integer: \"") + serial_graph
+
+        if not deterministic:
+            graph += pynini.closure(pynini.accep("s"))
+
+        graph += pynutil.insert("\" } units: \"serial\"")
 
         graph = self.add_tokens(graph)
         self.fst = graph.optimize()
diff --git a/nemo_text_processing/text_normalization/taggers/tokenize_and_classify.py b/nemo_text_processing/text_normalization/taggers/tokenize_and_classify.py
index 1ee63e55770a..9b3a2b9a5434 100644
--- a/nemo_text_processing/text_normalization/taggers/tokenize_and_classify.py
+++ b/nemo_text_processing/text_normalization/taggers/tokenize_and_classify.py
@@ -71,6 +71,7 @@ def __init__(self, input_case: str, deterministic: bool = True):
         money_graph = MoneyFst(cardinal=cardinal, decimal=decimal, deterministic=deterministic).fst
         whitelist_graph = WhiteListFst(input_case=input_case, deterministic=deterministic).fst
         punct_graph = PunctuationFst(deterministic=deterministic).fst
+        serial_graph = SerialFst(cardinal=cardinal, deterministic=deterministic).fst
 
         classify = (
             pynutil.add_weight(whitelist_graph, 1.01)
@@ -83,13 +84,14 @@ def __init__(self, input_case: str, deterministic: bool = True):
             | pynutil.add_weight(money_graph, 1.1)
             | pynutil.add_weight(telephone_graph, 1.1)
             | pynutil.add_weight(electonic_graph, 1.1)
+            | pynutil.add_weight(serial_graph, 1.2)
             | pynutil.add_weight(word_graph, 100)
         )
 
-        if not deterministic:
-            serial_graph = SerialFst(cardinal, deterministic=deterministic).fst
-            classify |= pynutil.add_weight(serial_graph, 1.1)
-            classify = classify.optimize()
+        # if not deterministic:
+        #     serial_graph = SerialFst(cardinal=cardinal, deterministic=deterministic).fst
+        #     classify |= pynutil.add_weight(serial_graph, 1.1)
+        #     classify = classify.optimize()
 
         punct = pynutil.insert("tokens { ") + pynutil.add_weight(punct_graph, weight=1.1) + pynutil.insert(" }")
         token = pynutil.insert("tokens { ") + classify + pynutil.insert(" }")
diff --git a/nemo_text_processing/text_normalization/taggers/whitelist.py b/nemo_text_processing/text_normalization/taggers/whitelist.py
index 59c0725f32a5..50c883f80358 100644
--- a/nemo_text_processing/text_normalization/taggers/whitelist.py
+++ b/nemo_text_processing/text_normalization/taggers/whitelist.py
@@ -44,7 +44,7 @@ class WhiteListFst(GraphFst):
     def __init__(self, input_case: str, deterministic: bool = True):
         super().__init__(name="whitelist", kind="classify")
 
-        def _get_whitelist_graph(file="data/whitelist.tsv"):
+        def _get_whitelist_graph(input_case, file="data/whitelist.tsv"):
             whitelist = load_labels(get_abs_path(file))
             if input_case == "lower_cased":
                 whitelist = [(x.lower(), y) for x, y in whitelist]
@@ -53,9 +53,16 @@ def _get_whitelist_graph(file="data/whitelist.tsv"):
             graph = pynini.string_map(whitelist)
             return graph
 
-        graph = _get_whitelist_graph()
+        def _get_whitelist_non_deterministic_graph(file="data/whitelist_alternatives.tsv"):
+            whitelist = load_labels(get_abs_path(file))
+            whitelist_lower = [(x.lower(), y.lower()) for x, y in whitelist]
+            whitelist_cased = [(x, y) for x, y in whitelist]
+            graph = pynini.string_map(whitelist_lower + whitelist_cased)
+            return graph
+
+        graph = _get_whitelist_graph(input_case)
         if not deterministic:
-            graph |= _get_whitelist_graph("data/whitelist_alternatives.tsv")
+            graph |= _get_whitelist_graph("lower_cased") | _get_whitelist_non_deterministic_graph()
 
         graph = pynutil.insert("name: \"") + convert_space(graph) + pynutil.insert("\"")
         self.fst = graph.optimize()
diff --git a/nemo_text_processing/text_normalization/verbalizers/measure.py b/nemo_text_processing/text_normalization/verbalizers/measure.py
index 78cf0eb9ddde..dbbc73c79445 100644
--- a/nemo_text_processing/text_normalization/verbalizers/measure.py
+++ b/nemo_text_processing/text_normalization/verbalizers/measure.py
@@ -43,7 +43,7 @@ def __init__(self, decimal: GraphFst, cardinal: GraphFst, deterministic: bool =
         unit = pynutil.insert(" ") + pynini.closure(NEMO_CHAR - " ", 1)
 
         unit = pynutil.delete("units: \"") + unit + pynutil.delete("\"") + delete_space
-        graph_decimal = (
+        self.graph_decimal = (
             pynutil.delete("decimal {")
             + delete_space
             + optional_sign
@@ -61,6 +61,6 @@ def __init__(self, decimal: GraphFst, cardinal: GraphFst, deterministic: bool =
             + delete_space
             + pynutil.delete("}")
         )
-        graph = (self.graph_cardinal | graph_decimal) + delete_space + unit
+        graph = (self.graph_cardinal | self.graph_decimal) + delete_space + unit
         delete_tokens = self.delete_tokens(graph)
         self.fst = delete_tokens.optimize()
diff --git a/nemo_text_processing/text_normalization/verbalizers/serial.py b/nemo_text_processing/text_normalization/verbalizers/serial.py
index 66361a8b2342..cf328035bba9 100644
--- a/nemo_text_processing/text_normalization/verbalizers/serial.py
+++ b/nemo_text_processing/text_normalization/verbalizers/serial.py
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_text_processing.text_normalization.graph_utils import GraphFst, delete_space
+from nemo_text_processing.text_normalization.graph_utils import NEMO_NOT_SPACE, GraphFst, delete_space
 
 try:
     import pynini
@@ -38,7 +38,9 @@ class SerialFst(GraphFst):
     def __init__(self, measure: GraphFst, deterministic: bool = False):
         super().__init__(name="serial", kind="verbalize", deterministic=deterministic)
 
-        serial = pynutil.delete("units: \"") + pynini.cross("serial", "") + pynutil.delete("\"") + delete_space
-        graph = measure.graph_cardinal + delete_space + serial
+        serial = (
+            pynini.cross("units: \"serial", "") + pynini.closure(NEMO_NOT_SPACE) + pynutil.delete("\"") + delete_space
+        )
+        graph = measure.graph_cardinal + pynini.closure(delete_space) + serial
         delete_tokens = self.delete_tokens(graph)
         self.fst = delete_tokens.optimize()
diff --git a/nemo_text_processing/text_normalization/verbalizers/verbalize.py b/nemo_text_processing/text_normalization/verbalizers/verbalize.py
index 92ff74cabcd1..651e0b9f7cb1 100644
--- a/nemo_text_processing/text_normalization/verbalizers/verbalize.py
+++ b/nemo_text_processing/text_normalization/verbalizers/verbalize.py
@@ -54,6 +54,7 @@ def __init__(self, deterministic: bool = True):
         date_graph = DateFst(ordinal=ordinal, deterministic=deterministic).fst
         money_graph = MoneyFst(decimal=decimal, deterministic=deterministic).fst
         whitelist_graph = WhiteListFst(deterministic=deterministic).fst
+        serial_graph = SerialFst(measure=measure, deterministic=deterministic).fst
         graph = (
             time_graph
             | date_graph
@@ -64,10 +65,11 @@ def __init__(self, deterministic: bool = True):
             | cardinal_graph
             | telephone_graph
             | electronic_graph
+            | serial_graph
             | whitelist_graph
         )
 
-        if not deterministic:
-            serial_graph = SerialFst(measure, deterministic=deterministic).fst
-            graph |= serial_graph
+        # if not deterministic:
+        #     serial_graph = SerialFst(measure=measure, deterministic=deterministic).fst
+        #     graph |= serial_graph
         self.fst = graph
diff --git a/tests/nemo_text_processing/data_text_normalization/test_cases_cardinal.txt b/tests/nemo_text_processing/data_text_normalization/test_cases_cardinal.txt
index 5b9100f6e949..1495a7c046b3 100644
--- a/tests/nemo_text_processing/data_text_normalization/test_cases_cardinal.txt
+++ b/tests/nemo_text_processing/data_text_normalization/test_cases_cardinal.txt
@@ -7,3 +7,4 @@
 13,000~thirteen thousand
 123,123,000~one hundred twenty three million one hundred twenty three thousand
 123,000,012~one hundred twenty three million twelve
+a 4-kilogram bag~a four kilogram bag
diff --git a/tests/nemo_text_processing/data_text_normalization/test_cases_decimal.txt b/tests/nemo_text_processing/data_text_normalization/test_cases_decimal.txt
index 1b839abc2052..d9634ffc815c 100644
--- a/tests/nemo_text_processing/data_text_normalization/test_cases_decimal.txt
+++ b/tests/nemo_text_processing/data_text_normalization/test_cases_decimal.txt
@@ -6,3 +6,4 @@
 0.1 billion~zero point one billion
 .1 trillion~point one trillion
 -0.1~minus zero point one
+7.2-millimeter bullet~seven point two millimeter bullet
diff --git a/tests/nemo_text_processing/data_text_normalization/test_cases_normalize_with_audio.txt b/tests/nemo_text_processing/data_text_normalization/test_cases_normalize_with_audio.txt
index d6d12defbb4f..7dadcf864fb8 100644
--- a/tests/nemo_text_processing/data_text_normalization/test_cases_normalize_with_audio.txt
+++ b/tests/nemo_text_processing/data_text_normalization/test_cases_normalize_with_audio.txt
@@ -1,11 +1,57 @@
-123~123|one twenty three|one two three|one hundred twenty three|one hundred and twenty three
-$123~$123|one twenty three dollars|one two three dollars|one hundred twenty three dollars|one hundred and twenty three dollars
-$123.2~$123.2|one twenty three dollars and two cents|one two three dollars two cents|one hundred twenty three dollars and two cents|one hundred and twenty three point two dollars
-1.24~1.24|one point two four|one two four|one point twenty four|one twenty four
-$1.21~one dollar and twenty one cents|one dollar twenty one cents|one point two one dollars|one twenty one dollars
-£1.00~one pound|one point o o pounds
-t-0t25d12-f~t zero t twenty five d twelve f
-133-A~one hundred thirty three A
-B2A23C~B two A twenty three C
-25d08A~twenty five d zero eight A
-C24~C twenty four
\ No newline at end of file
+~123
+123
+one twenty three
+one two three
+one hundred twenty three
+one hundred and twenty three
+~$123
+$123
+one twenty three dollars
+one two three dollars
+one hundred twenty three dollars
+one hundred and twenty three dollars
+~$123.2
+$123.2
+one twenty three dollars and two cents
+one two three dollars two cents
+one hundred twenty three dollars and two cents
+one hundred and twenty three point two dollars
+~1.24
+1.24
+one point two four
+one two four
+one point twenty four
+one twenty four
+~$1.21
+one dollar and twenty one cents
+one dollar twenty one cents
+one point two one dollars
+one twenty one dollars
+~£1.00
+one pound
+one point o o pounds
+~t-0t25d12-f
+t zero t twenty five d twelve f
+t zero t two five d one two f
+~133-A
+one hundred thirty three A
+~B2A23C
+B two A twenty three C
+~25d08A
+twenty five d zero eight A
+~C24
+C twenty four
+~It seemed to her that the jacket Oswald wore was darker than Commission Exhibit No. 162.
+It seemed to her that the jacket Oswald wore was darker than Commission Exhibit number one sixty two.
+It seemed to her that the jacket Oswald wore was darker than Commission Exhibit number one hundred sixty two.
+It seemed to her that the jacket Oswald wore was darker than Commission Exhibit number one hundred and sixty two.
+It seemed to her that the jacket Oswald wore was darker than Commission Exhibit number one six two.
+~"Father, let this cup pass." He prayed--was heard. What cup was it that passed away from him? Sure not the death-cup, now filled to the brim! There was no quailing in the awful word; He still was king of kings, of lords the lord:-- He feared lest, in the suffering waste and grim, His faith might grow too faint and sickly dim."
+"Father, let this cup pass." He prayed--was heard. What cup was it that passed away from him? Sure not the death-cup, now filled to the brim! There was no quailing in the awful word; He still was king of kings, of lords the lord:-- He feared lest, in the suffering waste and grim, His faith might grow too faint and sickly dim."
+~The box was 5 x 7 m.
+The box was five by seven meters.
+~25-30
+twenty five to thirty
+twenty five thirty
+~W2s
+W twos
\ No newline at end of file
diff --git a/tests/nemo_text_processing/data_text_normalization/test_cases_serial.txt b/tests/nemo_text_processing/data_text_normalization/test_cases_serial.txt
new file mode 100644
index 000000000000..6df8c17d5af2
--- /dev/null
+++ b/tests/nemo_text_processing/data_text_normalization/test_cases_serial.txt
@@ -0,0 +1,6 @@
+t-0t25d12-f~t oh t two five d one two f
+133-A~one three three A
+B2A23C~B two A two three C
+25d08A~two five d oh eight A
+C24~C two four
+W2s~W two s
diff --git a/tests/nemo_text_processing/test_boundary.py b/tests/nemo_text_processing/test_boundary.py
index cad8b2715854..547a7589bf6d 100644
--- a/tests/nemo_text_processing/test_boundary.py
+++ b/tests/nemo_text_processing/test_boundary.py
@@ -14,6 +14,7 @@
 
 import pytest
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -21,6 +22,7 @@
 class TestBoundary:
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_boundary.txt'))
     @pytest.mark.skipif(
@@ -31,3 +33,5 @@ class TestBoundary:
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_cardinal.py b/tests/nemo_text_processing/test_cardinal.py
index a8baa5859dd7..3e4c3f6fdf33 100644
--- a/tests/nemo_text_processing/test_cardinal.py
+++ b/tests/nemo_text_processing/test_cardinal.py
@@ -15,6 +15,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -33,6 +34,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_cardinal.txt'))
     @pytest.mark.skipif(
@@ -43,3 +45,5 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_date.py b/tests/nemo_text_processing/test_date.py
index 4d796152447b..54b9296b0334 100644
--- a/tests/nemo_text_processing/test_date.py
+++ b/tests/nemo_text_processing/test_date.py
@@ -15,6 +15,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -33,6 +34,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_date.txt'))
     @pytest.mark.skipif(
@@ -43,6 +45,8 @@ def test_denorm(self, test_input, expected):
     def test_norm_uncased(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input, verbose=False)
+        assert expected in pred_non_deterministic
 
     normalizer_uppercased = Normalizer(input_case='cased') if PYNINI_AVAILABLE else None
     cases_uppercased = {"Aug. 8": "august eighth", "8 Aug.": "the eighth of august", "aug. 8": "august eighth"}
@@ -56,3 +60,5 @@ def test_norm_uncased(self, test_input, expected):
     def test_norm_cased(self, test_input, expected):
         pred = self.normalizer_uppercased.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_decimal.py b/tests/nemo_text_processing/test_decimal.py
index 778476dfd790..59fece29438c 100644
--- a/tests/nemo_text_processing/test_decimal.py
+++ b/tests/nemo_text_processing/test_decimal.py
@@ -15,6 +15,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -33,6 +34,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case="lower_cased") if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_decimal.txt'))
     @pytest.mark.skipif(
@@ -43,3 +45,20 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input, verbose=False)
+        assert expected in pred_non_deterministic
+
+
+if __name__ == '__main__':
+    test_cases = parse_test_case_file('data_text_normalization/test_cases_decimal.txt')
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
+
+    for test_input, expected in test_cases:
+        import pdb
+
+        pdb.set_trace()
+        pred_non_deterministic = normalizer_with_audio.normalize(test_input)
+        print('input:', test_input)
+        for p in pred_non_deterministic:
+            print(p)
+        print('=' * 30)
diff --git a/tests/nemo_text_processing/test_electronic.py b/tests/nemo_text_processing/test_electronic.py
index 190dd379993d..a4238be40dec 100644
--- a/tests/nemo_text_processing/test_electronic.py
+++ b/tests/nemo_text_processing/test_electronic.py
@@ -15,6 +15,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -33,6 +34,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case="lower_cased") if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_electronic.txt'))
     @pytest.mark.skipif(
@@ -42,4 +44,6 @@ def test_denorm(self, test_input, expected):
     @pytest.mark.unit
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
-        assert pred == expected, f'|{pred}| {len(pred)}- |{expected}| {len(expected)}'
+        assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input, verbose=False)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_measure.py b/tests/nemo_text_processing/test_measure.py
index 5bd191bf5d60..ec61ec4112a6 100644
--- a/tests/nemo_text_processing/test_measure.py
+++ b/tests/nemo_text_processing/test_measure.py
@@ -16,6 +16,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -34,6 +35,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case="lower_cased") if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_measure.txt'))
     @pytest.mark.skipif(
@@ -44,3 +46,5 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_money.py b/tests/nemo_text_processing/test_money.py
index 86ed5659be0f..04ce35ec0b8a 100644
--- a/tests/nemo_text_processing/test_money.py
+++ b/tests/nemo_text_processing/test_money.py
@@ -16,6 +16,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -34,6 +35,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_money.txt'))
     @pytest.mark.skipif(
@@ -44,3 +46,5 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_normalization_with_audio.py b/tests/nemo_text_processing/test_normalization_with_audio.py
index d30d4837f53d..290267471a21 100644
--- a/tests/nemo_text_processing/test_normalization_with_audio.py
+++ b/tests/nemo_text_processing/test_normalization_with_audio.py
@@ -29,5 +29,5 @@ class TestNormalizeWithAudio:
     @pytest.mark.run_only_on('CPU')
     @pytest.mark.unit
     def test_norm(self, test_input, expected):
-        pred = self.normalizer.normalize_with_audio(test_input)
+        pred = self.normalizer.normalize(test_input)
         assert len(set(pred).intersection(set(expected))) == len(expected)
diff --git a/tests/nemo_text_processing/test_ordinal.py b/tests/nemo_text_processing/test_ordinal.py
index 5cb766d3b185..4cb47b2a40cf 100644
--- a/tests/nemo_text_processing/test_ordinal.py
+++ b/tests/nemo_text_processing/test_ordinal.py
@@ -16,6 +16,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -34,6 +35,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_ordinal.txt'))
     @pytest.mark.skipif(
@@ -44,3 +46,5 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_serial.py b/tests/nemo_text_processing/test_serial.py
new file mode 100644
index 000000000000..0b858fd663eb
--- /dev/null
+++ b/tests/nemo_text_processing/test_serial.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import pytest
+from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
+from parameterized import parameterized
+from utils import PYNINI_AVAILABLE, parse_test_case_file
+
+
+class TestSerial:
+    normalizer = Normalizer(input_case='cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
+
+    @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_serial.txt'))
+    @pytest.mark.skipif(
+        not PYNINI_AVAILABLE, reason="`pynini` not installed, please install via nemo_text_processing/setup.sh"
+    )
+    @pytest.mark.run_only_on('CPU')
+    @pytest.mark.unit
+    def test_norm(self, test_input, expected):
+        pred = self.normalizer.normalize(test_input, verbose=False)
+        assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_telephone.py b/tests/nemo_text_processing/test_telephone.py
index a59414c181bb..0c13f72605c5 100644
--- a/tests/nemo_text_processing/test_telephone.py
+++ b/tests/nemo_text_processing/test_telephone.py
@@ -16,6 +16,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -34,6 +35,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_telephone.txt'))
     @pytest.mark.skipif(
@@ -44,3 +46,5 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_time.py b/tests/nemo_text_processing/test_time.py
index 6a5b9e342f93..6add727d936c 100644
--- a/tests/nemo_text_processing/test_time.py
+++ b/tests/nemo_text_processing/test_time.py
@@ -15,6 +15,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -33,6 +34,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_time.txt'))
     @pytest.mark.skipif(
@@ -43,3 +45,5 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_whitelist.py b/tests/nemo_text_processing/test_whitelist.py
index 4146d91dcc31..0dfc67266f66 100644
--- a/tests/nemo_text_processing/test_whitelist.py
+++ b/tests/nemo_text_processing/test_whitelist.py
@@ -16,6 +16,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -34,6 +35,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_whitelist.txt'))
     @pytest.mark.skipif(
@@ -44,6 +46,8 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input, verbose=False)
+        assert expected in pred_non_deterministic
 
     normalizer_uppercased = Normalizer(input_case='cased') if PYNINI_AVAILABLE else None
     cases_uppercased = {"Dr. Evil": "doctor Evil", "No. 4": "number four", "dr. Evil": "dr. Evil", "no. 4": "no. four"}
@@ -57,3 +61,5 @@ def test_norm(self, test_input, expected):
     def test_norm_cased(self, test_input, expected):
         pred = self.normalizer_uppercased.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/test_word.py b/tests/nemo_text_processing/test_word.py
index 9d1eb9849afc..eea3c7802177 100644
--- a/tests/nemo_text_processing/test_word.py
+++ b/tests/nemo_text_processing/test_word.py
@@ -16,6 +16,7 @@
 import pytest
 from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
 from nemo_text_processing.text_normalization.normalize import Normalizer
+from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio
 from parameterized import parameterized
 from utils import PYNINI_AVAILABLE, parse_test_case_file
 
@@ -34,6 +35,7 @@ def test_denorm(self, test_input, expected):
         assert pred == expected
 
     normalizer = Normalizer(input_case='lower_cased') if PYNINI_AVAILABLE else None
+    normalizer_with_audio = NormalizerWithAudio(input_case='cased') if PYNINI_AVAILABLE else None
 
     @parameterized.expand(parse_test_case_file('data_text_normalization/test_cases_word.txt'))
     @pytest.mark.skipif(
@@ -44,3 +46,5 @@ def test_denorm(self, test_input, expected):
     def test_norm(self, test_input, expected):
         pred = self.normalizer.normalize(test_input, verbose=False)
         assert pred == expected
+        pred_non_deterministic = self.normalizer_with_audio.normalize(test_input)
+        assert expected in pred_non_deterministic
diff --git a/tests/nemo_text_processing/utils.py b/tests/nemo_text_processing/utils.py
index ca9750fc54f9..d2bb3554ef65 100644
--- a/tests/nemo_text_processing/utils.py
+++ b/tests/nemo_text_processing/utils.py
@@ -31,7 +31,6 @@ def parse_test_case_file(file_name: str):
         for line in f:
             spoken, written = line.split('~')
             test_pairs.append((spoken, written.strip("\n")))
-    print(test_pairs)
     return test_pairs
 
 
@@ -41,8 +40,15 @@ def get_test_cases_multiple(file_name: str = 'data_text_normalization/test_cases
     """
     test_pairs = []
     with open(os.path.dirname(os.path.abspath(__file__)) + os.path.sep + file_name, 'r') as f:
+        written = None
+        normalized_options = []
         for line in f:
-            written, normalized_options = line.split('~')
-            normalized_options = normalized_options.strip().split('|')
-            test_pairs.append((written, normalized_options))
+            if line.startswith('~'):
+                if written:
+                    test_pairs.append((written, normalized_options))
+                    normalized_options = []
+                written = line.strip().replace('~', '')
+            else:
+                normalized_options.append(line.strip())
+    test_pairs.append((written, normalized_options))
     return test_pairs


Code review comments:
diff_hunk: @@ -101,6 +102,25 @@ def __init__(self, cardinal: GraphFst, decimal: GraphFst, deterministic: bool =
             + pynutil.insert(" } ")
             + unit_singular
         )
-        final_graph = subgraph_decimal | subgraph_cardinal
+
+        subgraph_cardinal_dash = (
+            pynutil.insert("cardinal { integer: \"")
+            + cardinal.single_digits_graph
path: nemo_text_processing/text_normalization/taggers/measure.py
text: @user1:
subgraph_decimal_dash will allow for the test case of "7.2-millimeter," and I believe another test case like "7-millimeter" would work here with cardinal.single_digits_graph, but I don't think the following work after this MR:
"70-millimeter bullet"
"12-inch guns"
"21-knot cruising speed"
"550-pound bomb"

I may have missed something when I applied these manually to my local version, but I believe those will all still fail (though decimal versions of them would work eg "70.1-millimeter bullet").

@user1:
```suggestion
            + cardinal.graph
```