Repository: nltk/nltk

Issue:
POS tagger crashes when text starts with a white space.
Dear NLTK team,

the POS parsing crashes when the text starts with one (or several) white space(s). The problem occurs in the normalize() call.

/usr/local/lib/python3.6/dist-packages/nltk/tag/perceptron.py in normalize(self, word)
    238         elif word.isdigit() and len(word) == 4:
    239             return '!YEAR'
--> 240         elif word[0].isdigit():
    241             return '!DIGITS'
    242         else:

IndexError: string index out of range

I guess that testing the length() of the word could solve the issue.
lstriping the input text is not a correct one, as for many reasons, the text has to be preserved untouched by ntlk.

Best regards

Jerome


Discussion:
This can be reproduced, for example, with the following code:
```python
from nltk.tag.perceptron import PerceptronTagger

tagger = PerceptronTagger(load=False)
[tagger.normalize(w) for w in ['', 'is', 'a', 'beautiful', 'day']]
```

The code above produces the following output:
```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-29-a7ae37b09fb1> in <module>
----> 1 [tagger.normalize(w) for w in ['', 'is', 'a', 'beautiful', 'day']]

<ipython-input-29-a7ae37b09fb1> in <listcomp>(.0)
----> 1 [tagger.normalize(w) for w in ['', 'is', 'a', 'beautiful', 'day']]

nltk/nltk/tag/perceptron.py in normalize(self, word)
    278         elif word.isdigit() and len(word) == 4:
    279             return "!YEAR"
--> 280         elif word[0].isdigit():
    281             return "!DIGITS"
    282         else:

IndexError: string index out of range
```

Hi @jeromemassot . What kind of result would you expect from the `normalize` method on an empty string? Could you please provide some examples?

Resolving patch:
diff --git a/nltk/tag/__init__.py b/nltk/tag/__init__.py
--- a/nltk/tag/__init__.py
+++ b/nltk/tag/__init__.py
@@ -108,7 +108,7 @@ def _get_tagger(lang=None):
 
 
 def _pos_tag(tokens, tagset=None, tagger=None, lang=None):
-    # Currently only supoorts English and Russian.
+    # Currently only supports English and Russian.
     if lang not in ["eng", "rus"]:
         raise NotImplementedError(
             "Currently, NLTK pos_tag only supports English and Russian "
diff --git a/nltk/tag/perceptron.py b/nltk/tag/perceptron.py
--- a/nltk/tag/perceptron.py
+++ b/nltk/tag/perceptron.py
@@ -275,12 +275,11 @@ def normalize(self, word):
         """
         if "-" in word and word[0] != "-":
             return "!HYPHEN"
-        elif word.isdigit() and len(word) == 4:
+        if word.isdigit() and len(word) == 4:
             return "!YEAR"
-        elif word[0].isdigit():
+        if word and word[0].isdigit():
             return "!DIGITS"
-        else:
-            return word.lower()
+        return word.lower()
 
     def _get_features(self, i, word, context, prev, prev2):
         """Map tokens into a feature representation, implemented as a
@@ -296,7 +295,7 @@ def add(name, *args):
         # It's useful to have a constant feature, which acts sort of like a prior
         add("bias")
         add("i suffix", word[-3:])
-        add("i pref1", word[0])
+        add("i pref1", word[0] if word else "")
         add("i-1 tag", prev)
         add("i-2 tag", prev2)
         add("i tag+i-2 tag", prev, prev2)


Test patch:
diff --git a/nltk/test/tag.doctest b/nltk/test/tag.doctest
--- a/nltk/test/tag.doctest
+++ b/nltk/test/tag.doctest
@@ -31,3 +31,14 @@ and does not fail with:
     >>> from nltk.tag import RegexpTagger
     >>> patterns = [(str(i), 'NNP',) for i in range(200)]
     >>> tagger = RegexpTagger(patterns)
+
+Regression Testing for issue #2483
+==================================
+
+Ensure that tagging with pos_tag (PerceptronTagger) does not throw an IndexError
+when attempting tagging an empty string. What it must return instead is not 
+strictly defined.
+
+    >>> from nltk.tag import pos_tag
+    >>> pos_tag(['', 'is', 'a', 'beautiful', 'day'])
+    [...]
\ No newline at end of file