Repository: barrust/pyspellchecker

Issue:
English spellchecking
Hello Team!
I am new to the Project and I have a question.

I use python 3.7 and run into problem with this test program:
``` python
from spellchecker import SpellChecker
spell = SpellChecker()                         
split_words = spell.split_words
spell_unknown = spell.unknown

words = split_words("That's how t and s don't fit.")
print(words)
misspelled = spell_unknown(words)
print(misspelled)
```
With pyspellchecker ver 0.5.4 the printout is:
``` python
['that', 's', 'how', 't', 'and', 's', 'don', 't', 'fit']
set()
```
So free standing 't' and  's' are not marked as errors neither are contractions.

If I change the phrase to:
```python
words = split_words("That is how that's and don't do not fit.")
```
and use pyspellchecker ver 0.5.6 the printout is:
``` python
['that', 'is', 'how', 'that', 's', 'and', 'don', 't', 'do', 'not', 'fit']
{'t', 's'}
```
So contractions are marked as mistakes again.

(I read barrust comment on Oct 22, 2019}

Please, assist.


Discussion:

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e4a891d..7aa5b30 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,10 +1,9 @@
 # pyspellchecker
 
-## Future Release
-* Updated automated `scripts/build_dictionary.py` script to support adding missing words
-
-## Version 0.6.0
+## Version 0.6.0 (future version)
 * Remove **python 2.7** support
+* Updated automated `scripts/build_dictionary.py` script to support adding missing words
+* Updated `split_words()` to attempt to better handle punctuation; [#84](https://github.com/barrust/pyspellchecker/issues/84)
 
 ## Version 0.5.6
 * ***NOTE:*** Last planned support for **Python 2.7**
diff --git a/spellchecker/utils.py b/spellchecker/utils.py
index deafce4..e00484d 100644
--- a/spellchecker/utils.py
+++ b/spellchecker/utils.py
@@ -64,9 +64,11 @@ def write_file(filepath, encoding, gzipped, data):
 
 
 def _parse_into_words(text):
-    """ Parse the text into words; currently removes punctuation
+    """ Parse the text into words; currently removes punctuation except for
+        apostrophies.
 
         Args:
             text (str): The text to split into words
     """
-    return re.findall(r"\w+", text.lower())
+    # see: https://stackoverflow.com/a/12705513
+    return re.findall(r"(\w[\w']*\w|\w)", text.lower())


Test patch:
diff --git a/tests/spellchecker_test.py b/tests/spellchecker_test.py
index 165371a..b403117 100644
--- a/tests/spellchecker_test.py
+++ b/tests/spellchecker_test.py
@@ -191,7 +191,6 @@ class TestSpellChecker(unittest.TestCase):
                 cnt += 1
         self.assertEqual(cnt, 0)
 
-
     def test_remove_by_threshold_using_items(self):
         ''' test removing everything below a certain threshold; using items to test '''
         spell = SpellChecker()
@@ -398,3 +397,9 @@ class TestSpellChecker(unittest.TestCase):
 
         self.assertTrue(var in spell)
         self.assertEqual(spell[var], 60)
+
+    def test_split_words(self):
+        ''' test using split_words '''
+        spell = SpellChecker()
+        res = spell.split_words("This isn't a good test, but it is a test!!!!")
+        self.assertEqual(set(res), set(["this", "isn't", "a", "good", "test", "but", "it", "is", "a", "test"]))