Repository: piskvorky/gensim

Issue:
evaluate_word_pairs gives ValueError: x and y must have length at least 2.
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

I am trying to run an evaluation on a pair of trained vectors on a datafile, namely [this one](https://github.com/tca19/near-lossless-binarization/blob/master/datasets/MEN.txt). However, when using the function `wordembeddingmodel.evaluate_word_pairs(datapath("MEN.txt"))` (it is stored in the datapath folder) I get the error `ValueError: x and y must have length at least 2.`. I suspect this has to do with the formatting, but I cannot figure out what the issue is. 

#### Steps/code/corpus to reproduce



Traceback: 

```bash
File "/.../src/evaluate.py", line 25, in evaluate_wordsimilarity
    ] = wordembeddingmodel.evaluate_word_pairs(datapath(worddataset))[0][0]
  File "/home/users/filip/.conda/envs/sskgnn/lib/python3.9/site-packages/gensim/models/keyedvectors.py", line 1321, in evaluate_word_pairs
    pearson = stats.pearsonr(similarity_gold, similarity_model)
  File "/home/users/filip/.conda/envs/sskgnn/lib/python3.9/site-packages/scipy/stats/stats.py", line 4016, in pearsonr
    raise ValueError('x and y must have length at least 2.')
ValueError: x and y must have length at least 2.
```
Dataset: Downloaded from [here](https://raw.githubusercontent.com/tca19/near-lossless-binarization/master/datasets/MEN.txt). I download it and create it using: 

```python3
import requests
from gensim.test.utils import datapath
import os.path as osp

fpath = datapath("")
file = "https://raw.githubusercontent.com/tca19/near-lossless-binarization/master/datasets/MEN.txt"
r = requests.get(file)
filename = file.split("/")[-1]
with open(osp.join(fpath,filename), "wb") as f:
    f.write(r.text.encode("utf-8").strip())
```


```python
>>> print(wordembeddingmodel.lifecycle_events)
*** AttributeError: 'Word2VecKeyedVectors' object has no attribute 'lifecycle_events'
```

#### Versions

Please provide the output of:

```python
>>> import platform; print(platform.platform())
ec.FAST_VERSION)Linux-5.4.0-90-generic-x86_64-with-glibc2.31
>>> import sys; print("Python", sys.version)
Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0]
>>> import struct; print("Bits", 8 * struct.calcsize("P"))
Bits 64
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.21.2
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.7.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.8.3
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 1
```

Any help is greatly appreciated! Thank you :)

Cheers
Filip
evaluate_word_pairs gives ValueError: x and y must have length at least 2.
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

I am trying to run an evaluation on a pair of trained vectors on a datafile, namely [this one](https://github.com/tca19/near-lossless-binarization/blob/master/datasets/MEN.txt). However, when using the function `wordembeddingmodel.evaluate_word_pairs(datapath("MEN.txt"))` (it is stored in the datapath folder) I get the error `ValueError: x and y must have length at least 2.`. I suspect this has to do with the formatting, but I cannot figure out what the issue is. 

#### Steps/code/corpus to reproduce



Traceback: 

```bash
File "/.../src/evaluate.py", line 25, in evaluate_wordsimilarity
    ] = wordembeddingmodel.evaluate_word_pairs(datapath(worddataset))[0][0]
  File "/home/users/filip/.conda/envs/sskgnn/lib/python3.9/site-packages/gensim/models/keyedvectors.py", line 1321, in evaluate_word_pairs
    pearson = stats.pearsonr(similarity_gold, similarity_model)
  File "/home/users/filip/.conda/envs/sskgnn/lib/python3.9/site-packages/scipy/stats/stats.py", line 4016, in pearsonr
    raise ValueError('x and y must have length at least 2.')
ValueError: x and y must have length at least 2.
```
Dataset: Downloaded from [here](https://raw.githubusercontent.com/tca19/near-lossless-binarization/master/datasets/MEN.txt). I download it and create it using: 

```python3
import requests
from gensim.test.utils import datapath
import os.path as osp

fpath = datapath("")
file = "https://raw.githubusercontent.com/tca19/near-lossless-binarization/master/datasets/MEN.txt"
r = requests.get(file)
filename = file.split("/")[-1]
with open(osp.join(fpath,filename), "wb") as f:
    f.write(r.text.encode("utf-8").strip())
```


```python
>>> print(wordembeddingmodel.lifecycle_events)
*** AttributeError: 'Word2VecKeyedVectors' object has no attribute 'lifecycle_events'
```

#### Versions

Please provide the output of:

```python
>>> import platform; print(platform.platform())
ec.FAST_VERSION)Linux-5.4.0-90-generic-x86_64-with-glibc2.31
>>> import sys; print("Python", sys.version)
Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0]
>>> import struct; print("Bits", 8 * struct.calcsize("P"))
Bits 64
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.21.2
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.7.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.8.3
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 1
```

Any help is greatly appreciated! Thank you :)

Cheers
Filip


Discussion:
Please upgrade your Gensim first, we no longer support the 3.X versions.
Alright, I upgraded it, such that: 

```python3
>>> import gensim; print("gensim", gensim.__version__)
gensim 4.1.2
```

The error persist however :/
Please upgrade your Gensim first, we no longer support the 3.X versions.
Alright, I upgraded it, such that: 

```python3
>>> import gensim; print("gensim", gensim.__version__)
gensim 4.1.2
```

The error persist however :/

Resolving patch:
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -256,6 +256,9 @@ def __init__(self, vector_size, count=0, dtype=np.float32, mapfile_path=None):
 
         self.mapfile_path = mapfile_path
 
+    def __str__(self):
+        return f"{self.__class__.__name__}<vector_size={self.vector_size}, {len(self)} keys>"
+
     def _load_specials(self, *args, **kwargs):
         """Handle special requirements of `.load()` protocol, usually up-converting older versions."""
         super(KeyedVectors, self)._load_specials(*args, **kwargs)
@@ -1420,7 +1423,8 @@ def log_evaluate_word_pairs(pearson, spearman, oov, pairs):
         logger.info('Pairs with unknown words ratio: %.1f%%', oov)
 
     def evaluate_word_pairs(
-            self, pairs, delimiter='\t', restrict_vocab=300000, case_insensitive=True, dummy4unknown=False,
+            self, pairs, delimiter='\t', encoding='utf8',
+            restrict_vocab=300000, case_insensitive=True, dummy4unknown=False,
         ):
         """Compute correlation of the model with human similarity judgments.
 
@@ -1471,16 +1475,12 @@ def evaluate_word_pairs(
         similarity_model = []
         oov = 0
 
-        original_key_to_index = self.key_to_index
-        self.key_to_index = ok_vocab
-
-        with utils.open(pairs, 'rb') as fin:
-            for line_no, line in enumerate(fin):
-                line = utils.to_unicode(line)
-                if line.startswith('#'):
-                    # May be a comment
-                    continue
-                else:
+        original_key_to_index, self.key_to_index = self.key_to_index, ok_vocab
+        try:
+            with utils.open(pairs, encoding=encoding) as fin:
+                for line_no, line in enumerate(fin):
+                    if not line or line.startswith('#'):  # Ignore lines with comments.
+                        continue
                     try:
                         if case_insensitive:
                             a, b, sim = [word.upper() for word in line.split(delimiter)]
@@ -1490,19 +1490,27 @@ def evaluate_word_pairs(
                     except (ValueError, TypeError):
                         logger.info('Skipping invalid line #%d in %s', line_no, pairs)
                         continue
+
                     if a not in ok_vocab or b not in ok_vocab:
                         oov += 1
                         if dummy4unknown:
                             logger.debug('Zero similarity for line #%d with OOV words: %s', line_no, line.strip())
                             similarity_model.append(0.0)
                             similarity_gold.append(sim)
-                            continue
                         else:
-                            logger.debug('Skipping line #%d with OOV words: %s', line_no, line.strip())
-                            continue
+                            logger.info('Skipping line #%d with OOV words: %s', line_no, line.strip())
+                        continue
                     similarity_gold.append(sim)  # Similarity from the dataset
                     similarity_model.append(self.similarity(a, b))  # Similarity from the model
-        self.key_to_index = original_key_to_index
+        finally:
+            self.key_to_index = original_key_to_index
+
+        assert len(similarity_gold) == len(similarity_model)
+        if not similarity_gold:
+            raise ValueError(
+                f"No valid similarity judgements found in {pairs}: either invalid format or "
+                f"all are out-of-vocabulary in {self}"
+            )
         spearman = stats.spearmanr(similarity_gold, similarity_model)
         pearson = stats.pearsonr(similarity_gold, similarity_model)
         if dummy4unknown:


Test patch: