Repository: piskvorky/gensim

Issue:
Word2Vec docs indicate sents can be a generator, but it can't
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

Tried to pass a generator as the `sentences` argument when training Word2Vec, as suggested [in the docs](https://radimrehurek.com/gensim/models/word2vec.html#usage-examples:~:text=sentences%20can%20be%20a%20generator). But then I get this error:

    TypeError: You can't pass a generator as the sentences argument. Try a sequence.

#### Steps/code/corpus to reproduce

```

from gensim.models import Word2Vec

def get_sents():
    for ii in range(100):
        yield ['hi']

sents = get_sents()
model = Word2Vec(sents, size=100, window=5, min_count=1, workers=4)
```

#### Versions

Please provide the output of:

```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import struct; print("Bits", 8 * struct.calcsize("P"))
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
```
```
Linux-5.8.5-arch1-1-x86_64-with-glibc2.2.5
Python 3.8.5 (default, Jul 27 2020, 08:42:51) 
[GCC 10.1.0]
Bits 64
NumPy 1.19.1
SciPy 1.4.1
gensim 3.8.3
FAST_VERSION 1
```

Word2Vec docs indicate sents can be a generator, but it can't
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

Tried to pass a generator as the `sentences` argument when training Word2Vec, as suggested [in the docs](https://radimrehurek.com/gensim/models/word2vec.html#usage-examples:~:text=sentences%20can%20be%20a%20generator). But then I get this error:

    TypeError: You can't pass a generator as the sentences argument. Try a sequence.

#### Steps/code/corpus to reproduce

```

from gensim.models import Word2Vec

def get_sents():
    for ii in range(100):
        yield ['hi']

sents = get_sents()
model = Word2Vec(sents, size=100, window=5, min_count=1, workers=4)
```

#### Versions

Please provide the output of:

```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import struct; print("Bits", 8 * struct.calcsize("P"))
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
```
```
Linux-5.8.5-arch1-1-x86_64-with-glibc2.2.5
Python 3.8.5 (default, Jul 27 2020, 08:42:51) 
[GCC 10.1.0]
Bits 64
NumPy 1.19.1
SciPy 1.4.1
gensim 3.8.3
FAST_VERSION 1
```



Discussion:
Yeah, that doc section is weird. It also refers to an undefined (at that point) variable `sentences`.

@polm Could you open a PR to fix the docs please?
Yeah, that doc section is weird. It also refers to an undefined (at that point) variable `sentences`.

@polm Could you open a PR to fix the docs please?

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -39,18 +39,22 @@
 
 .. sourcecode:: pycon
 
-    >>> from gensim.test.utils import common_texts, get_tmpfile
+    >>> from gensim.test.utils import common_texts
     >>> from gensim.models import Word2Vec
     >>>
-    >>> path = get_tmpfile("word2vec.model")
-    >>>
     >>> model = Word2Vec(common_texts, size=100, window=5, min_count=1, workers=4)
     >>> model.save("word2vec.model")
 
-The training is streamed, meaning `sentences` can be a generator, reading input data
-from disk on-the-fly, without loading the entire corpus into RAM.
 
-It also means you can continue training the model later:
+The training is streamed, so ``sentences`` can be an iterable, reading input data
+from disk on-the-fly. This lets you avoid loading the entire corpus into RAM.
+However, note that because the iterable must be re-startable, `sentences` must
+not be a generator. For an example of an appropriate iterator see
+:class:`~gensim.models.word2vec.BrownCorpus`,
+:class:`~gensim.models.word2vec.Text8Corpus` or
+:class:`~gensim.models.word2vec.LineSentence`.
+
+If you save the model you can continue training it later:
 
 .. sourcecode:: pycon
 


Test patch: