Repository: cltk/cltk

Issue:
Latinlibrary corpus word/sentence loading raises error
The NLTK PlaintextCorpusReader methods words() and sents() raise errors when using commands such as:

```
from cltk.corpus.latin import latinlibrary
latinlibrary.sents()

```
- Words: The reader streams the input, sometimes returning empty lists, which cause index out of range errors with the current Latin lemmatizer. 
- Sents: Currently relies on an external NLTK corpus; will raise errors if not installed.

Questions about corpus reader
Following up from PR #296.

I like this, have a few questions as I have not used the NLTK's PlainCorpusReader.

Using your new feature, a few observations and questions.

``` python
In [1]: from cltk.corpus.latin import latinlibrary

In [2]: type(latinlibrary)
Out[2]: nltk.corpus.reader.plaintext.PlaintextCorpusReader

In [3]: dir(latinlibrary)
Out[3]: 
['CorpusView',
 '__class__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__unicode__',
 '__weakref__',
 '_encoding',
 '_fileids',
 '_get_root',
 '_para_block_reader',
 '_read_para_block',
 '_read_sent_block',
 '_read_word_block',
 '_root',
 '_sent_tokenizer',
 '_tagset',
 '_word_tokenizer',
 'abspath',
 'abspaths',
 'citation',
 'encoding',
 'ensure_loaded',
 'fileids',
 'license',
 'open',
 'paras',
 'raw',
 'readme',
 'root',
 'sents',
 'unicode_repr',
 'words']

In [5]: latinlibrary.words()
Out[5]: ['DUODECIM', 'TABULARUM', 'LEGES', 'DUODECIM', ...]
```

How do I view all words in this?

``` python
In [9]: latinlibrary.sents()
Out[9]: ---------------------------------------------------------------------------
LookupError                               Traceback (most recent call last)
/Users/kyle/cltk/venv/lib/python3.5/site-packages/IPython/core/formatters.py in __call__(self, obj)
    697                 type_pprinters=self.type_printers,
    698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
    700             printer.flush()
    701             return stream.getvalue()

…

/Users/kyle/cltk/venv/lib/python3.5/site-packages/nltk/data.py in find(resource_name, paths)
    639     sep = '*' * 70
    640     resource_not_found = '\n%s\n%s\n%s' % (sep, msg, sep)
--> 641     raise LookupError(resource_not_found)
    642 
    643 

LookupError: 
**********************************************************************
  Resource 'tokenizers/punkt/PY3/english.pickle' not found.
  Please use the NLTK Downloader to obtain the resource:  >>>
  nltk.download()
  Searched in:
    - '/Users/kyle/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
    - ''
**********************************************************************

```

What kind of tokenizer is it looking for, sentence or word? I made a sentence tokenizer for Latin (in the models repo), so I assume it's usable here, though I don't know the code.

``` python
In [17]: latinlibrary.raw()
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-17-8861dd5120c2> in <module>()
----> 1 latinlibrary.raw()

…

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/encodings/utf_8.py in decode(input, errors)
     14 
     15 def decode(input, errors='strict'):
---> 16     return codecs.utf_8_decode(input, errors, True)
     17 
     18 class IncrementalEncoder(codecs.IncrementalEncoder):

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 32177: invalid start byte
```

Having full Unicode support is a must for us, obviously.

Finally, could this allow for sub-selection of texts? Or can this just choose the entirety of a corpus.

Thanks for this! I'm not trying to be too critical. Just want to understand the implementation's current limitations.



Discussion:

Have a look at this gist for some examples (and output) that I generated this morning. It shows access to words, sentences, fileids, and sub-selection of texts.  "raw" and "sents" work on my machine—I'll try to figure out why.

As far as unicode support—do you have the most recent copy of the latin_text_latin_library corpus installed? I had converted all of the txt files to utf-8 and pushed the changes earlier in the month.

I am definitely going to do some more research on using custom sent/word tokenizers with this reader.

And nothing too critical—really want this to work! https://gist.github.com/diyclassics/5f4e7ff7963e255dd44278577ffcbf6e#file-gistfile1-txt

You're right, I don't have the latest vers of the corpus. That certainly is
why.

I can answer any question about the sentence tokenizer. Seems that we'll
need to override where it looks for it, from the nltk_data to our dir.

On Sunday, May 29, 2016, Patrick J. Burns notifications@github.com wrote:

> Have a look at this gist for some examples (and output) that I generated
> this morning. It shows access to words, sentences, fileids, and
> sub-selection of texts. "raw" and "sends" work on my machine—I'll try to
> figure out why.
> 
> As far as unicode support—do you have the most recent copy of the
> latin_text_latin_library corpus installed? I had converted all of the txt
> files to utf-8 and pushed the changes earlier in the month.
> 
> I am definitely going to do some more research on using custom sent/word
> tokenizers with this reader.
> 
> And nothing too critical—really want this to work!
> https://gist.github.com/diyclassics/5f4e7ff7963e255dd44278577ffcbf6e#file-gistfile1-txt
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/cltk/cltk/issues/297#issuecomment-222370091, or mute
> the thread
> https://github.com/notifications/unsubscribe/ABxN-FIludiJCVlJsgMa5QSH20iIx8OLks5qGcHegaJpZM4IpUFX
> .

## 

Kyle P. Johnson, Ph.D.

Natural language processing, data science, architecture
https://kyle-p-johnson.comkyle@kyle-p-johnson.com |
https://kyle-p-johnson.com | https://github.com/kylepjohnson

Classical Language Toolkit, Founder
http://cltk.org | https://github.com/cltk/cltk

Looks like PlaintextCorpusReader can take the following arguments "sent_tokenizer" & "word_tokenizer"—just ran this successfully with the CLTK Latin sentence and word tokenizers. Makes sense for these to be the defaults for importing the latin corpora, no?

Btw—did updating the corpus take care of just the unicode problem or did that also affect the other functions [like .sents()]?

So I updated and that fixed the Unicode error. The sents() issue remains, as I do not have (nor do I want) the NLTK's english tokenizer. Error here: https://gist.github.com/kylepjohnson/6bbfef58f2eec14d222fca9b32f72fe6.


Resolving patch:
diff --git a/cltk/corpus/latin/__init__.py b/cltk/corpus/latin/__init__.py
--- a/cltk/corpus/latin/__init__.py
+++ b/cltk/corpus/latin/__init__.py
@@ -6,17 +6,36 @@
 """
 CLTK Latin corpus readers
 """
+
 import os.path
 from nltk.corpus.reader.plaintext import PlaintextCorpusReader
+from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
+
+from cltk.tokenize.sentence import TokenizeSentence
+from cltk.tokenize.word import WordTokenizer
 
 # Would like to have this search through a CLTK_DATA environment variable
 # Better to use something like make_cltk_path in cltk.utils.file_operations?
 home = os.path.expanduser('~')
 cltk_path = os.path.join(home, 'CLTK_DATA')
 
+word_tokenizer = WordTokenizer('latin')
+
+if os.path.exists(cltk_path + 'latin/model/latin_models_cltk/tokenizers/sentence'):
+    sent_tokenizer = TokenizeSentence('latin')
+else:
+    punkt_param = PunktParameters()
+    abbreviations = ['c', 'l', 'm', 'p', 'q', 't', 'ti', 'sex', 'a', 'd', 'cn', 'sp', "m'", 'ser', 'ap', 'n', 'v', 'k', 'mam', 'post', 'f', 'oct', 'opet', 'paul', 'pro', 'sert', 'st', 'sta', 'v', 'vol', 'vop']
+    punkt_param.abbrev_types = set(abbreviations)
+    sent_tokenizer = PunktSentenceTokenizer(punkt_param)
+
 # Latin Library
 try:
-    latinlibrary = PlaintextCorpusReader(cltk_path + '/latin/text/latin_text_latin_library', '.*\.txt', encoding='utf-8')
+    latinlibrary = PlaintextCorpusReader(cltk_path + '/latin/text/latin_text_latin_library', 
+    '.*\.txt',
+    word_tokenizer=word_tokenizer, 
+    sent_tokenizer=sent_tokenizer, 
+    encoding='utf-8')    
     pass
 except IOError as e:
     print("Corpus not found. Please check that the Latin Library is installed in CLTK_DATA.")
diff --git a/cltk/tokenize/sentence.py b/cltk/tokenize/sentence.py
--- a/cltk/tokenize/sentence.py
+++ b/cltk/tokenize/sentence.py
@@ -88,3 +88,9 @@ def tokenize_sentences(self: object, untokenized_string: str):
         for sentence in tokenizer.sentences_from_text(untokenized_string, realign_boundaries=True):  # pylint: disable=C0301
             tokenized_sentences.append(sentence)
         return tokenized_sentences
+        
+    def tokenize(self: object, untokenized_string: str):
+        # NLTK's PlaintextCorpusReader needs a function called tokenize
+        # in functions used as a parameter for sentence tokenization.
+        # So this is an alias for tokenize_sentences().
+        return self.tokenize_sentences(untokenized_string)
diff --git a/cltk/tokenize/word.py b/cltk/tokenize/word.py
--- a/cltk/tokenize/word.py
+++ b/cltk/tokenize/word.py
@@ -125,18 +125,21 @@ def replace(m):
 
     for sent in sents:
         temp_tokens = word_tokenizer.word_tokenize(sent)
-        if temp_tokens[0].endswith('ne'):
-            if temp_tokens[0].lower() not in exceptions:
-                temp = [temp_tokens[0][:-2], '-ne']
-                temp_tokens = temp + temp_tokens[1:]
-
-        if temp_tokens[-1].endswith('.'):
-            final_word = temp_tokens[-1][:-1]
-            del temp_tokens[-1]
-            temp_tokens += [final_word, '.']
-
-        for token in temp_tokens:
-            tokens.append(token)
+        # Need to check that tokens exist before handling them; needed to make stream.readlines work in PlaintextCorpusReader
+        
+        if temp_tokens:
+            if temp_tokens[0].endswith('ne'):
+                if temp_tokens[0].lower() not in exceptions:
+                    temp = [temp_tokens[0][:-2], '-ne']
+                    temp_tokens = temp + temp_tokens[1:]
+
+            if temp_tokens[-1].endswith('.'):
+                final_word = temp_tokens[-1][:-1]
+                del temp_tokens[-1]
+                temp_tokens += [final_word, '.']
+
+            for token in temp_tokens:
+                tokens.append(token)
 
     # Break enclitic handling into own function?
     specific_tokens = []


Test patch: