Repository: nltk/nltk

Issue:
AttributeError: 'itertools.chain' object has no attribute 'lookup'
nltk version: 3.7.
python version: 3.9.15
From the following code:

```
padded_line = [list(pad_both_ends(word_tokenize(input_text), n=N))]
train_data, vocab = padded_everygram_pipeline(order=N, text=padded_line)
lm = Lidstone(gamma=0.2, vocabulary=vocab, order=N)
lm.fit(text=train_data)
```

where input_text is a long string and N as in the Ngram model. From the last line of code above, I get the following error:

>lm.fit(text=train_data)
>  File "env\lib\site-packages\nltk\lm\api.py", line 110, in fit
>  self.counts.update(self.vocab.lookup(sent) for sent in text)
>  File "env\lib\site-packages\nltk\lm\counter.py", line 115, in update
>    for sent in ngram_text:
>  File "env\lib\site-packages\nltk\lm\api.py", line 110, in <genexpr>
>    self.counts.update(self.vocab.lookup(sent) for sent in text)
> AttributeError: 'itertools.chain' object has no attribute 'lookup'

Can anyone fix this error for me?


Discussion:
Hello!

This feels to me to be a bit of a design oversight that occurs whenever you provide a vocabulary that is not a `nltk.lm.Vocabulary` straight to the language model initialisation. To be concrete:
* The `vocab` as returned by `padded_everygram_pipeline` is an iterator over text as vocabulary data (a `itertools.chain` instance)
* The `Lidstone` initialization expects a `Vocabulary` instance.
* Alternatively, you can not provide a vocabulary to `Lidstone`, but pass an iterator over text as vocabulary data to `lm.fit(...)`.

So, there are two working solutions:

### Solution 1: Pass a `Vocabulary` instance to `Lidstone`:
```python
from nltk.lm import Lidstone, Vocabulary
from nltk.lm.preprocessing import pad_both_ends, padded_everygram_pipeline
from nltk.tokenize import word_tokenize

input_text = """
Hello! This is a pretty long text, or so I would hope. I intend
for it to have multiple sentences. I should be able to do that, I reckon.
However, that does mean that I have to keep coming up with some text.
Luckily, that is not all too difficult. This text should be approaching
a good length by now.
"""
N = 2
padded_line = [list(pad_both_ends(word_tokenize(input_text), n=N))]
train_data, vocab = padded_everygram_pipeline(order=N, text=padded_line)
# Initialize vocabulary instance, and update it using the vocab iterator
lm = Lidstone(gamma=0.2, vocabulary=Vocabulary(vocab), order=N)
lm.fit(train_data)

print(lm.generate(10, text_seed=["This"]))
# e.g. ['text', 'should', 'be', 'able', 'to', 'keep', 'coming', 'up', 'with', 'some']
```

### Solution 2: Pass `vocab` to `lm.fit` instead
```python
from nltk.lm import Lidstone, Vocabulary
from nltk.lm.preprocessing import pad_both_ends, padded_everygram_pipeline
from nltk.tokenize import word_tokenize

input_text = """
Hello! This is a pretty long text, or so I would hope. I intend
for it to have multiple sentences. I should be able to do that, I reckon.
However, that does mean that I have to keep coming up with some text.
Luckily, that is not all too difficult. This text should be approaching
a good length by now.
"""
N = 2
padded_line = [list(pad_both_ends(word_tokenize(input_text), n=N))]
train_data, vocab = padded_everygram_pipeline(order=N, text=padded_line)
lm = Lidstone(gamma=0.2, order=N)
lm.fit(train_data, vocab)

print(lm.generate(10, text_seed=["This"]))
# e.g. ['text', 'should', 'be', 'approaching', 'a', 'good', 'length', 'by', 'now', '.']
```

Normally I believe that warnings for incorrect use of functions/methods are a bit unnecessary, but in this case it is a very, very reasonable mistake to make. With other words, I'll be preparing a quick PR that causes a warning to be thrown when a language model is initialized with a vocabulary that isn't a `Vocabulary` instance.

Thank you for raising this issue, and I hope that this comment helps you solve your issues!

- Tom Aarsen

Resolving patch:
diff --git a/nltk/lm/api.py b/nltk/lm/api.py
--- a/nltk/lm/api.py
+++ b/nltk/lm/api.py
@@ -7,6 +7,7 @@
 """Language Model Interface."""
 
 import random
+import warnings
 from abc import ABCMeta, abstractmethod
 from bisect import bisect
 from itertools import accumulate
@@ -83,7 +84,7 @@ def __init__(self, order, vocabulary=None, counter=None):
             of creating a new one when training.
         :type vocabulary: `nltk.lm.Vocabulary` or None
         :param counter: If provided, use this object to count ngrams.
-        :type vocabulary: `nltk.lm.NgramCounter` or None
+        :type counter: `nltk.lm.NgramCounter` or None
         :param ngrams_fn: If given, defines how sentences in training text are turned to ngram
             sequences.
         :type ngrams_fn: function or None
@@ -91,6 +92,12 @@ def __init__(self, order, vocabulary=None, counter=None):
         :type pad_fn: function or None
         """
         self.order = order
+        if vocabulary and not isinstance(vocabulary, Vocabulary):
+            warnings.warn(
+                f"The `vocabulary` argument passed to {self.__class__.__name__!r} "
+                "must be an instance of `nltk.lm.Vocabulary`.",
+                stacklevel=3,
+            )
         self.vocab = Vocabulary() if vocabulary is None else vocabulary
         self.counts = NgramCounter() if counter is None else counter
 


Test patch: