Repository: piskvorky/gensim

Issue:
Passing nonsense (int) for `corpus_file` generates confusing OSError in get_offsets_and_start_doctags_for_corpusfile
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

I am trying to train a doc2vec model over 1.35 million docs. Each line of the opened file corresponds to a doc and I am creating a TaggedDocument object from that. I am getting these errors which I cannot resolve. Note that the 1351747 below corresponds to the number of docs.
```
OSError: [Errno 9] Bad file descriptor: 1351747
```

This is my code:
```python
taggeddoc = []
f = open('EnglishT.txt')
for line in f:
    td = TaggedDocument(gensim.utils.to_unicode(str.encode(line)).split(),[str(index)])
    taggeddoc.append(td)
    index += 1
f.close()
print(len(taggeddoc))
model = gensim.models.doc2vec.Doc2Vec(vector_size=50, dm=0, dbow_words=0, workers=30, alpha=0.025, min_alpha=0.025)

model.build_vocab(taggeddoc)
# start training
for epoch in range(500):
    model.train(taggeddoc, model.corpus_count, epochs=model.iter)
    model.alpha -= 0.002  
    model.min_alpha = model.alpha 
```

Using Python3:
```
Traceback (most recent call last):
  File "en_d2v.py", line 25, in <module>
    model.train(taggeddoc, model.corpus_count, epochs=model.iter)
  File "/home/junlp/.local/lib/python3.6/site-packages/gensim/models/doc2vec.py", line 799, in train
    offsets, start_doctags = self._get_offsets_and_start_doctags_for_corpusfile(corpus_file, self.workers)
  File "/home/junlp/.local/lib/python3.6/site-packages/gensim/models/doc2vec.py", line 828, in _get_offsets_and_start_doctags_for_corpusfile
    corpus_file_size = os.path.getsize(corpus_file)
  File "/usr/lib/python3.6/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
OSError: [Errno 9] Bad file descriptor: 1351747
```

Using Python2
```
Traceback (most recent call last):
  File "en_d2v.py", line 32, in <module>
    model.train(taggeddoc, model.corpus_count, epochs=model.iter)
  File "/home/junlp/.local/lib/python2.7/site-packages/gensim/models/doc2vec.py", line 799, in train
    offsets, start_doctags = self._get_offsets_and_start_doctags_for_corpusfile(corpus_file, self.workers)
  File "/home/junlp/.local/lib/python2.7/site-packages/gensim/models/doc2vec.py", line 828, in _get_offsets_and_start_doctags_for_corpusfile
    corpus_file_size = os.path.getsize(corpus_file)
  File "/usr/lib/python2.7/genericpath.py", line 57, in getsize
    return os.stat(filename).st_size
TypeError: coercing to Unicode: need string or buffer, int found
```

#### Versions

Linux-4.15.0-46-generic-x86_64-with-Ubuntu-18.04-bionic
('Python', '2.7.15rc1 (default, Nov 12 2018, 14:31:15) \n[GCC 7.3.0]') 
('NumPy', '1.16.2')
('SciPy', '1.2.1')
No handlers could be found for logger "smart_open.ssh"
('gensim', '3.7.2')
('FAST_VERSION', 1)

For Python3,
Python 3.6.7 (default, Oct 22 2018, 11:32:17)


Discussion:
This sort of troubleshooting should be handled via the discussion list, https://groups.google.com/forum/#!forum/gensim , rather than a Github issue, unless it's clear that a bug or required new feature is involved. 

Your code seems based on an outdated/bad example, in a couple ways. In particular, the 2nd positional argument you've provided to `train()`, `model.corpus_count`, is actually binding to a `corpus_file` parameter (added in recent gensim versions). Explicitly naming the intended parameter, `train(taggeddoc, total_examples=model.corpus_count, epochs=model.iter)`, should resolve this problem. 

But separately: it's rare for any user to need to call `train()` multiple times in a loop. (It's unnecessary and prone to other errors, such as decrementing `alpha` in destructive way, as has also happened here.)  Leave the default `alpha`/`min_alpha` in place, call `train()` once with your preferred `epochs` value, and it will do the right thing. 
@gojomo Thanks for your help. I made the necessary changes, and cleaned up the code. Closing this. 
Reopening with new name to address issue that passing an obviously-bad `corpus_file` value generated such an unhelpful error. 
Hi @gojomo If no one is working on this, can I take this up ? 
@saraswatmks of course!
@gojomo Thanks, I am on it. Incase, I need some clarifications I can ask here, right?

Resolving patch:
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -70,7 +70,7 @@
 except ImportError:
     from Queue import Queue  # noqa:F401
 
-from collections import namedtuple, defaultdict
+from collections import namedtuple, defaultdict, Iterable
 from timeit import default_timer
 
 from numpy import zeros, float32 as REAL, empty, ones, \
@@ -794,6 +794,19 @@ def train(self, documents=None, corpus_file=None, total_examples=None, total_wor
 
         """
         kwargs = {}
+
+        if corpus_file is None and documents is None:
+            raise TypeError("Either one of corpus_file or documents value must be provided")
+
+        if corpus_file is not None and documents is not None:
+            raise TypeError("Both corpus_file and documents must not be provided at the same time")
+
+        if documents is None and not os.path.isfile(corpus_file):
+            raise TypeError("Parameter corpus_file must be a valid path to a file, got %r instead" % corpus_file)
+
+        if documents is not None and not isinstance(documents, Iterable):
+            raise TypeError("documents must be an iterable of list, got %r instead" % documents)
+
         if corpus_file is not None:
             # Calculate offsets for each worker along with initial doctags (doctag ~ document/line number in a file)
             offsets, start_doctags = self._get_offsets_and_start_doctags_for_corpusfile(corpus_file, self.workers)
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -280,10 +280,12 @@
 """
 
 import logging
+import os
 
 import numpy as np
 from numpy import ones, vstack, float32 as REAL, sum as np_sum
 import six
+from collections import Iterable
 
 import gensim.models._fasttext_bin
 
@@ -901,6 +903,19 @@ def train(self, sentences=None, corpus_file=None, total_examples=None, total_wor
             >>> model.train(sentences, total_examples=model.corpus_count, epochs=model.epochs)
 
         """
+
+        if corpus_file is None and sentences is None:
+            raise TypeError("Either one of corpus_file or sentences value must be provided")
+
+        if corpus_file is not None and sentences is not None:
+            raise TypeError("Both corpus_file and sentences must not be provided at the same time")
+
+        if sentences is None and not os.path.isfile(corpus_file):
+            raise TypeError("Parameter corpus_file must be a valid path to a file, got %r instead" % corpus_file)
+
+        if sentences is not None and not isinstance(sentences, Iterable):
+            raise TypeError("sentences must be an iterable of list, got %r instead" % sentences)
+
         super(FastText, self).train(
             sentences=sentences, corpus_file=corpus_file, total_examples=total_examples, total_words=total_words,
             epochs=epochs, start_alpha=start_alpha, end_alpha=end_alpha, word_count=word_count,


Test patch:
diff --git a/gensim/test/test_doc2vec.py b/gensim/test/test_doc2vec.py
--- a/gensim/test/test_doc2vec.py
+++ b/gensim/test/test_doc2vec.py
@@ -173,6 +173,17 @@ def testLoadOldModel(self):
             sims_to_infer = loaded_model.docvecs.most_similar([doc0_inferred], topn=len(loaded_model.docvecs))
             self.assertTrue(sims_to_infer)
 
+    def testDoc2vecTrainParameters(self):
+
+        model = doc2vec.Doc2Vec(vector_size=50)
+        model.build_vocab(documents=list_corpus)
+
+        self.assertRaises(TypeError, model.train, corpus_file=11111)
+        self.assertRaises(TypeError, model.train, documents=11111)
+        self.assertRaises(TypeError, model.train, documents=sentences, corpus_file='test')
+        self.assertRaises(TypeError, model.train, documents=None, corpus_file=None)
+        self.assertRaises(TypeError, model.train, corpus_file=sentences)
+
     @unittest.skipIf(os.name == 'nt', "See another test for Windows below")
     def test_get_offsets_and_start_doctags(self):
         # Each line takes 6 bytes (including '\n' character)
@@ -387,7 +398,7 @@ def model_sanity(self, model, keep_training=True):
             tmpf = get_tmpfile('gensim_doc2vec.tst')
             model.save(tmpf)
             loaded = doc2vec.Doc2Vec.load(tmpf)
-            loaded.train(sentences, total_examples=loaded.corpus_count, epochs=loaded.epochs)
+            loaded.train(documents=sentences, total_examples=loaded.corpus_count, epochs=loaded.epochs)
 
     def test_training(self):
         """Test doc2vec training."""
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -95,6 +95,17 @@ def test_training(self):
         oov_vec = model.wv['minor']  # oov word
         self.assertEqual(len(oov_vec), 10)
 
+    def testFastTextTrainParameters(self):
+
+        model = FT_gensim(size=10, min_count=1, hs=1, negative=0, seed=42, workers=1)
+        model.build_vocab(sentences=sentences)
+
+        self.assertRaises(TypeError, model.train, corpus_file=11111)
+        self.assertRaises(TypeError, model.train, sentences=11111)
+        self.assertRaises(TypeError, model.train, sentences=sentences, corpus_file='test')
+        self.assertRaises(TypeError, model.train, sentences=None, corpus_file=None)
+        self.assertRaises(TypeError, model.train, corpus_file=sentences)
+
     @unittest.skipIf(os.name == 'nt' and six.PY2, "corpus_file training is not supported on Windows + Py27")
     def test_training_fromfile(self):
         with temporary_file(get_tmpfile('gensim_fasttext.tst')) as corpus_file: