Repository: alvenirai/punctfix (language: python)

Issue:
0.10.0 fails when input text contains a word which is only non-alphanumerical (" % ")
Since the last update punctfix no longer handles % in the input string. I get the following exception:

```
<class 'Exception'> Something went wrong while matching word list ... Tried matching the word: der with 
Traceback (most recent call last):
  File "/home/frederik/audio/speech_recognition/src/speech_recognition/text_formater.py", line 88, in punctuate
    tokens = self.punctuation.punctuate(' '.join(stripped_tokens)).split(' ')
  File "/home/frederik/.local/lib/python3.10/site-packages/punctfix/inference.py", line 195, in punctuate
    word_prediction_list = self.populate_word_prediction_with_labels(chunks, word_prediction_list)
  File "/home/frederik/.local/lib/python3.10/site-packages/punctfix/inference.py", line 139, in populate_word_prediction_with_labels
    assert word_prediction_list[current_index].word == word, \
AssertionError: Something went wrong while matching word list ... Tried matching the word: der with 
```

For this input string:
```
overordnet set så har vi 22 % der rapporterer en eller anden form for enten brunstigst opmærksomhed eller kønskrænkende adfærd eller tvang
```

This worked completely fine in 0.9.0


Discussion / hints:
Maybe this is desired behavior as per https://github.com/alvenirai/punctfix/commit/e009f7a53b26e8b14a734303b9d4ee016b09ce49?
Temporarily fixed by calling:
```
punctuation = PunctFixer()
punctuation.punctuate(string.replace('%', 'procent')).replace('procent', '%')
```
Hi @FredHaa, thank you for filing an issue and giving a nice example; I'll look at it ASAP.

 I was not aware that it previously even worked with input string containing %, as the model has been trained with completely cleaned input text and the previous version had inconsistent behaviour depending on the non-alphanumerical characters. 

The desired default behaviour would be for the puncfixer to remove the `%` but this apparently fails; I'll fix soon. The alternative behaviour, to keep the `%` like in the previous version can be enabled by instantiating the model with 
```py
>>> fixer = PunctFixer(skip_normalization=True)
```
which makes your example work on version 0.10:
```py
>>> fixer.punctuate(example_text)
"Overordnet set, så har vi 22 %, der rapporterer en eller anden form for enten brunstigst, opmærksomhed eller kønskrænkende adfærd eller tvang."
```

Fix patch:
diff --git a/README.md b/README.md
index dbd106f..b891e8b 100644
--- a/README.md
+++ b/README.md
@@ -30,6 +30,7 @@ Its quite simple to use!
 'En dag bliver vi sku glade for, at vi nu kan sætte punktummer og kommaer i en sætning. Det fungerer da meget godt, ikke?' 
 ```
 
+Note that, per default, the input text will be normalied. See next section for more details.
 
 ## Parameters for PunctFixer
 * Pass `device="cuda"` or `device="cpu"` to indicate where to run inference. Default is `device="cpu"`
@@ -38,7 +39,8 @@ lower acuracy use a chunk size of 150-200 and very little overlap i.e. 5-10. The
 default values `word_chunk_size=100`, `word_overlap=70` which makes it run a bit slow. The default parameters
 will be updated when we have some results on variations. 
 * Supported languages are "en" for English, "da" for Danish and "de" for German. Default is `language="da"`.
-
+* Note that the fixer has been trained on normalized text (lowercase letters and numbers) and will per default normalize input text. You can instantiate the model with `skip_normalization=True` to disable this but this might yield errors on some input text.
+* To raise warnings every time the input is normalied, set `warn_on_normalization=True`.
 
 ## Contribute
 If you encounter issues, feel free to open issues in the repo and then we will fix. Even better, create issue and 
diff --git a/punctfix/inference.py b/punctfix/inference.py
index 437a813..9c6b38b 100644
--- a/punctfix/inference.py
+++ b/punctfix/inference.py
@@ -210,6 +210,9 @@ def _split_input_text(self, text: str) -> List[str]:
                 continue
             if len(norm_word) < len(word):
                 to_warn.append(r"Non-word (r'\W') characters were removed.")
+            # We might have removed the entire word
+            if not norm_word:
+                continue
             if not norm_word.islower():
                 norm_word = norm_word.lower()
                 to_warn.append("Text was lowercased.")
diff --git a/setup.py b/setup.py
index 6ad7952..2117628 100644
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@
 
 setuptools.setup(
     name="punctfix",
-    version="0.10.0",
+    version="0.10.1",
     author="Martin Carsten Nielsen",
     author_email="martin@alvenir.ai",
     description="Punctuation restoration library",


Test patch:
diff --git a/tests/test_punctuation.py b/tests/test_punctuation.py
index 797065c..da412c1 100644
--- a/tests/test_punctuation.py
+++ b/tests/test_punctuation.py
@@ -203,7 +203,8 @@ def test_do_normalize(self):
         self.model.warn_on_normalization = False
         expected_output = ["hejsa", "mand"]
         for model_input in ("hejsa, mand", " hejsa mand", "hejsa mand",
-                "Hejsa mand", "hejsa  mand", "  hejsa mand", "  hejsa, Mand"):
+                "Hejsa mand", "hejsa  mand", "  hejsa mand", "  hejsa, Mand",
+                "hejsa % mand ! % "):
             actual_output = self.model._split_input_text(model_input)
             self.assertEqual(actual_output, expected_output)