Repository: piskvorky/gensim

Issue:
annoy.py conversion of cosine distance to cosine similarity is incorrect
in [this function](https://github.com/RaRe-Technologies/gensim/blob/f35faae7a7b0c3c8586fb61208560522e37e0e7e/gensim/similarities/annoy.py#L169) the code to calculate cosine similarity is incorrect

    def most_similar(self, vector, num_neighbors):
        """Find `num_neighbors` most similar items.

        Parameters
        ----------
        vector : numpy.array
            Vector for word/document.
        num_neighbors : int
            Number of most similar items

        Returns
        -------
        list of (str, float)
            List of most similar items in format [(`item`, `cosine_distance`), ... ]

        """
        ids, distances = self.index.get_nns_by_vector(
            vector, num_neighbors, include_distances=True)

        return [(self.labels[ids[i]], 1 - distances[i] / 2) for i in range(len(ids))]

according to annoy documentation `get_nns_by_vector` with `include_distances=True` will return the distances and not the square power of the distance (this was changed since aug 2016):

_`a.get_distance(i, j)` returns the distance between items i and j. NOTE: this used to return the squared distance, but has been changed as of Aug 2016._

[link](https://github.com/spotify/annoy#:~:text=a.get_distance(i%2C%20j)%20returns%20the%20distance%20between%20items%20i%20and%20j.%20NOTE%3A%20this%20used%20to%20return%20the%20squared%20distance%2C%20but%20has%20been%20changed%20as%20of%20Aug%202016.)

also:
Annoy uses Euclidean distance of normalized vectors for its angular distance, which for two vectors u,v is equal to sqrt(2(1-cos(u,v)))
[link](https://github.com/spotify/annoy#:~:text=Annoy%20uses%20Euclidean%20distance%20of%20normalized%20vectors%20for%20its%20angular%20distance%2C%20which%20for%20two%20vectors%20u%2Cv%20is%20equal%20to%20sqrt(2(1%2Dcos(u%2Cv))))

so this means that in order to calculate the cosine similarity correctly we should do this:
`
return [(self.labels[ids[i]], 1 - distances[i]^2 / 2) for i in range(len(ids))]
`


Discussion:
As this Gensim code was last touched July 2016, & the Annoy library says it changed its behavior August 2016, I can see how this is out-of-sync.

An effect would be that the max distance returned by `get_nns_by_vector()` of 2.0 will be turned into a cosine-similarity of 0.0 instead of the correct -1.0 – squashed range of returned values. (However, in no case will the relative ranking of nearest-neighbors be incorrect – which may have contributed to this going unnnoticed for so long.)

The proposed fix looks correct. 
Thanks @monash8 for the report and investigation. And @gojomo for confirming.

@monash8 are you able to open a PR with the fix?
> Thanks @monash8 for the report and investigation. And @gojomo for confirming.
> 
> @monash8 are you able to open a PR with the fix?

sure! @piskvorky can you give me access to create a new branch?
@monash8 please create a branch in your fork & open a PR again gensim's `develop` branch from there.

Resolving patch:
diff --git a/gensim/similarities/annoy.py b/gensim/similarities/annoy.py
--- a/gensim/similarities/annoy.py
+++ b/gensim/similarities/annoy.py
@@ -185,4 +185,4 @@ def most_similar(self, vector, num_neighbors):
         ids, distances = self.index.get_nns_by_vector(
             vector, num_neighbors, include_distances=True)
 
-        return [(self.labels[ids[i]], 1 - distances[i] / 2) for i in range(len(ids))]
+        return [(self.labels[ids[i]], 1 - distances[i] ** 2 / 2) for i in range(len(ids))]


Test patch: