Repository: piskvorky/gensim

Issue:
gensim.matutils.hellinger d(x,y) != d(y,x) if len(x) = len(y)
<!--
If your issue is a usage or a general question, please submit it here instead:
- Mailing List: https://groups.google.com/forum/#!forum/gensim
For more information, see Recipes&FAQ: https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ
-->

<!-- Instructions For Filing a Bug: https://github.com/RaRe-Technologies/gensim/blob/develop/CONTRIBUTING.md -->

#### Description
<!-- Example: Vocabulary size is not what I expected when training Word2Vec. -->
Compute the distance between 2 distribution with gensim.matutils.hellinger. The d(x,y) do not equal to d(y,x) if len(x) = len(y)
#### Steps/Code/Corpus to Reproduce
<!--
Example:
```
from gensim.models import word2vec

sentences = ['human', 'machine']
model = word2vec.Word2Vec(sentences)
print(model.syn0.shape) 
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->
from gensim.matutils import hellinger
vec_1 = [(2, 0.1), (3, 0.4), (4, 0.1), (5, 0.1), (1, 0.1), (7, 0.2)]
vec_2 = [(1, 0.1), (3, 0.8), (4, 0.1), (8, 0.1), (10, 0.8), (9, 0.1)]
hellinger(vec_1,vec_2) == hellinger(vec_2,vec_1)

#### Expected Results
<!-- Example: Expected shape of (100,2).-->
True
#### Actual Results
<!-- Example: Actual shape of (100,5). 
Please paste or specifically describe the actual output or traceback. -->
False
#### Versions
<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
-->
Linux-4.13.0-26-generic-x86_64-with-debian-stretch-sid
('Python', '2.7.11 |Anaconda custom (64-bit)| (default, Dec  6 2015, 18:08:32) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]')
('NumPy', '1.13.3')
('SciPy', '1.0.0')
('gensim', '3.1.0')
('FAST_VERSION', 1)

<!-- Thanks for contributing! -->




Discussion:
Thanks for report @elituan, reproduced on `3.2.0`. 

Problem here - https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/matutils.py#L900 (and lower)

@parulsethi can you fix this?
hi, I am willing to work on this.  It seems that d(x,y) do not equal to d(y,x) becasue that x and y has different index sets, and d(x,y) is computed based on the indexs in x while d(y,x) is computed based on indexs in y. 

So I am wording if it should have same  result in this situation, and if it should,  maybe one approach  is that we collect all the indexs in x and y into a set firstly, then we can iter the index set and get values in x and y.   



@caiyulun sounds like a good idea, feel free to contribute! For all indexes that defined for one vector and undefined for other - I think we should use `0.0` as a default value (because of BoW representation for this case -> all undefined stuff is zero).


Resolving patch:
diff --git a/gensim/matutils.py b/gensim/matutils.py
--- a/gensim/matutils.py
+++ b/gensim/matutils.py
@@ -897,10 +897,9 @@ def hellinger(vec1, vec2):
     if isbow(vec1) and isbow(vec2):
         # if it is a BoW format, instead of converting to dense we use dictionaries to calculate appropriate distance
         vec1, vec2 = dict(vec1), dict(vec2)
-        if len(vec2) < len(vec1):
-            vec1, vec2 = vec2, vec1  # swap references so that we iterate over the shorter vector
+        indices = set(list(vec1.keys()) + list(vec2.keys()))
         sim = np.sqrt(
-            0.5 * sum((np.sqrt(value) - np.sqrt(vec2.get(index, 0.0)))**2 for index, value in iteritems(vec1))
+            0.5 * sum((np.sqrt(vec1.get(index, 0.0)) - np.sqrt(vec2.get(index, 0.0)))**2 for index in indices)
         )
         return sim
     else:


Test patch:
diff --git a/gensim/test/test_similarity_metrics.py b/gensim/test/test_similarity_metrics.py
--- a/gensim/test/test_similarity_metrics.py
+++ b/gensim/test/test_similarity_metrics.py
@@ -105,12 +105,21 @@ def test_inputs(self):
 
     def test_distributions(self):
 
-        # checking bag of words as inputs
+        # checking different length bag of words as inputs
         vec_1 = [(2, 0.1), (3, 0.4), (4, 0.1), (5, 0.1), (1, 0.1), (7, 0.2)]
         vec_2 = [(1, 0.1), (3, 0.8), (4, 0.1)]
         result = matutils.hellinger(vec_1, vec_2)
-        expected = 0.185241936534
+        expected = 0.484060507634
+        self.assertAlmostEqual(expected, result)
+
+        # checking symmetrical bag of words inputs return same distance
+        vec_1 = [(2, 0.1), (3, 0.4), (4, 0.1), (5, 0.1), (1, 0.1), (7, 0.2)]
+        vec_2 = [(1, 0.1), (3, 0.8), (4, 0.1), (8, 0.1), (10, 0.8), (9, 0.1)]
+        result = matutils.hellinger(vec_1, vec_2)
+        result_symmetric = matutils.hellinger(vec_2, vec_1)
+        expected = 0.856921568786
         self.assertAlmostEqual(expected, result)
+        self.assertAlmostEqual(expected, result_symmetric)
 
         # checking ndarray, csr_matrix as inputs
         vec_1 = np.array([[1, 0.3], [0, 0.4], [2, 0.3]])