Repository: piskvorky/gensim

Issue:
SoftCosineSimilarity: Unexpected Behavior for Query Input
<!--
If your issue is a usage or a general question, please submit it here instead:
- Mailing List: https://groups.google.com/forum/#!forum/gensim
For more information, see Recipes&FAQ: https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ
-->

<!-- Instructions For Filing a Bug: https://github.com/RaRe-Technologies/gensim/blob/develop/CONTRIBUTING.md -->

#### Description
Expected SoftCosineSimilarity self[query] syntax to accept any iterable of list of (int, number) as indicated by: https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/similarities/docsim.py#L925

#### Steps/Code/Corpus to Reproduce
```
import scipy as sp
import gensim

# import the common_corpus as a TextCorpus object
# We assume common_corpus is a text file containing the lines:
#
# Human machine interface for lab abc computer applications
# A survey of user opinion of computer system response time
# The EPS user interface management system
# System and human system engineering testing of EPS
# Relation of user perceived response time to error measurement
# The generation of random binary unordered trees
# The intersection graph of paths in trees
# Graph minors IV Widths of trees and well quasi ordering
# Graph minors A survey

corpus = gensim.corpora.TextCorpus(input="path/common_corpus")
tfidf = gensim.models.TfidfModel(
    corpus, id2word=corpus.dictionary, normalize=True)

# Behavior of MatrixSimilarity
cos_index = gensim.similarities.MatrixSimilarity(tfidf[corpus])
cos_index[tfidf[corpus]]  # :)

# Behavior of SoftCosineSimilarity
dim = len(corpus.dictionary)
M = sp.sparse.identity(dim).tocsc() # generate the identity matrix
softcos_index = gensim.similarities.SoftCosineSimilarity(tfidf[corpus], M)
softcos_index[tfidf[corpus]] # :(
softcos_index[tfidf[[bow for bow in corpus]]] # :|
softcos_index[[vec for vec in tfidf[corpus]]] # :|
```


#### Expected Results
I believe that the TransformedCorpus tfidf[corpus] is an iterable of list of (int, number):
```
In [248]: next(iter(tfidf[corpus]))
Out[248]:
[(0, 0.4500498962785324),
 (1, 0.4500498962785324),
 (2, 0.3080749612015952),
 (3, 0.3080749612015952),
 (4, 0.4500498962785324),
 (5, 0.4500498962785324)]
```
I would expect that to be taken as query input without incident. It seems that SoftCosineSimilarity was adapted from a copy of MatrixSimilarity and was meant to function in a similar spirit, however, they do not accept the same types of input. 

#### Actual Results
```
In [249]: softcos_index[tfidf[corpus]]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-249-2d8c4d759cc9> in <module>()
----> 1 softcos_index[tfidf[corpus]]

/Users/natalied/Projects/canonical_answers/.venv/lib/python2.7/site-packages/gensim/interfaces.pyc in __getitem__(self, query)
    365                 else:
    366                     query = matutils.unitvec(query)
--> 367         result = self.get_similarities(query)
    368
    369         if self.num_best is None:

/Users/natalied/Projects/canonical_answers/.venv/lib/python2.7/site-packages/gensim/similarities/docsim.pyc in get_similarities(self, query)
    936             query = [self.corpus[i] for i in query]
    937
--> 938         if not query or not isinstance(query[0], list):
    939             query = [query]
    940

/Users/natalied/Projects/canonical_answers/.venv/lib/python2.7/site-packages/gensim/interfaces.pyc in __getitem__(self, docno)
    215             return self.obj[self.corpus[docno]]
    216         else:
--> 217             raise RuntimeError('Type {} does not support slicing.'.format(type(self.corpus)))
    218
    219

RuntimeError: Type <class 'gensim.corpora.textcorpus.TextCorpus'> does not support slicing.
```

#### Possible Code Involved
Compare the MatrixSimilarity get_similarities method: 
https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/similarities/docsim.py#L825
with the SoftCosineSimilarity get_similarities method:
https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/similarities/docsim.py#L938

Also we may need to expand this test: 
https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_similarities.py#L377

#### Versions
```
Darwin-15.6.0-x86_64-i386-64bit
('Python', '2.7.13 (default, Apr  4 2017, 08:46:44) \n[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]')
('NumPy', '1.14.1')
('SciPy', '1.0.0')
('gensim', '3.4.0')
('FAST_VERSION', 0)
```


<!-- Thanks for contributing! -->


SoftCosineSimilarity: Unexpected Behavior for Query Input
<!--
If your issue is a usage or a general question, please submit it here instead:
- Mailing List: https://groups.google.com/forum/#!forum/gensim
For more information, see Recipes&FAQ: https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ
-->

<!-- Instructions For Filing a Bug: https://github.com/RaRe-Technologies/gensim/blob/develop/CONTRIBUTING.md -->

#### Description
Expected SoftCosineSimilarity self[query] syntax to accept any iterable of list of (int, number) as indicated by: https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/similarities/docsim.py#L925

#### Steps/Code/Corpus to Reproduce
```
import scipy as sp
import gensim

# import the common_corpus as a TextCorpus object
# We assume common_corpus is a text file containing the lines:
#
# Human machine interface for lab abc computer applications
# A survey of user opinion of computer system response time
# The EPS user interface management system
# System and human system engineering testing of EPS
# Relation of user perceived response time to error measurement
# The generation of random binary unordered trees
# The intersection graph of paths in trees
# Graph minors IV Widths of trees and well quasi ordering
# Graph minors A survey

corpus = gensim.corpora.TextCorpus(input="path/common_corpus")
tfidf = gensim.models.TfidfModel(
    corpus, id2word=corpus.dictionary, normalize=True)

# Behavior of MatrixSimilarity
cos_index = gensim.similarities.MatrixSimilarity(tfidf[corpus])
cos_index[tfidf[corpus]]  # :)

# Behavior of SoftCosineSimilarity
dim = len(corpus.dictionary)
M = sp.sparse.identity(dim).tocsc() # generate the identity matrix
softcos_index = gensim.similarities.SoftCosineSimilarity(tfidf[corpus], M)
softcos_index[tfidf[corpus]] # :(
softcos_index[tfidf[[bow for bow in corpus]]] # :|
softcos_index[[vec for vec in tfidf[corpus]]] # :|
```


#### Expected Results
I believe that the TransformedCorpus tfidf[corpus] is an iterable of list of (int, number):
```
In [248]: next(iter(tfidf[corpus]))
Out[248]:
[(0, 0.4500498962785324),
 (1, 0.4500498962785324),
 (2, 0.3080749612015952),
 (3, 0.3080749612015952),
 (4, 0.4500498962785324),
 (5, 0.4500498962785324)]
```
I would expect that to be taken as query input without incident. It seems that SoftCosineSimilarity was adapted from a copy of MatrixSimilarity and was meant to function in a similar spirit, however, they do not accept the same types of input. 

#### Actual Results
```
In [249]: softcos_index[tfidf[corpus]]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-249-2d8c4d759cc9> in <module>()
----> 1 softcos_index[tfidf[corpus]]

/Users/natalied/Projects/canonical_answers/.venv/lib/python2.7/site-packages/gensim/interfaces.pyc in __getitem__(self, query)
    365                 else:
    366                     query = matutils.unitvec(query)
--> 367         result = self.get_similarities(query)
    368
    369         if self.num_best is None:

/Users/natalied/Projects/canonical_answers/.venv/lib/python2.7/site-packages/gensim/similarities/docsim.pyc in get_similarities(self, query)
    936             query = [self.corpus[i] for i in query]
    937
--> 938         if not query or not isinstance(query[0], list):
    939             query = [query]
    940

/Users/natalied/Projects/canonical_answers/.venv/lib/python2.7/site-packages/gensim/interfaces.pyc in __getitem__(self, docno)
    215             return self.obj[self.corpus[docno]]
    216         else:
--> 217             raise RuntimeError('Type {} does not support slicing.'.format(type(self.corpus)))
    218
    219

RuntimeError: Type <class 'gensim.corpora.textcorpus.TextCorpus'> does not support slicing.
```

#### Possible Code Involved
Compare the MatrixSimilarity get_similarities method: 
https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/similarities/docsim.py#L825
with the SoftCosineSimilarity get_similarities method:
https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/similarities/docsim.py#L938

Also we may need to expand this test: 
https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_similarities.py#L377

#### Versions
```
Darwin-15.6.0-x86_64-i386-64bit
('Python', '2.7.13 (default, Apr  4 2017, 08:46:44) \n[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]')
('NumPy', '1.14.1')
('SciPy', '1.0.0')
('gensim', '3.4.0')
('FAST_VERSION', 0)
```


<!-- Thanks for contributing! -->




Discussion:
Thanks for reporting. CC @Witiko 
Just to expand the original issue report, the `get_similarities` method of `SoftCosineSimilarity` was taken verbatim from `WmdSimilarity`. Any digression from the expected behavior is likely to impact that class as well.
Not sure if this is the correct place to put this info, but I also noticed that these related unit tests do not seem to be run:  

https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_keyedvectors.py#L30

And if they did, I am not sure this is correctly checking for ones on the diagonal of the term similarity matrix: 

https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_keyedvectors.py#L40


You are correct, @nataliedurgin. Additionally, if the test were run, they would have failed at https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_keyedvectors.py#L38
since the up-to-date signature of `similarity_matrix` is `similarity_matrix(dictionary, tfidf, …)`. Do you have any idea why the tests are not executed by the CI services, @menshikh-iv?
I think unittest might require methods to be prefixed by 'test_' in order to find them. The tests might be formatted like this: 
https://github.com/nataliedurgin/gensim/commit/5bb6aea3a92b8e7ff7cecfca054a6057bb9b8fa8

All seem to pass in that case except for checking that there are ones on the diagonals. (Not to say they shouldn't be expanded.)

(Sorry I wasn't working on this issue. I was working on making a Levenshtein distance term similarity matrix function to feed into softcosine and was cannibalizing some of the existing term similarity matrix functionality.)  
Nice catch! This seems to be a test case that I sketched and then forgot about, since the CI service did not complain. It is somewhat unrelated to the original issue, but should be fixed nevertheless.
@Witiko @nataliedurgin can you fix it?
@piskvorky Not earlier than this weekend. @nataliedurgin has already fixed the unrelated tests, but the original issue still needs fixing and making into a new test case.
> (Sorry I wasn't working on this issue. I was working on making a Levenshtein distance term similarity matrix function to feed into softcosine and was cannibalizing some of the existing term similarity matrix functionality.)

Most of the method should be directly reusable if you are going for the same greedy algorithm that achieves a constant number of non-zero elements per a row / column. If you are going just for a constant threshold, like Sidorov et al. (2014) did, you will end up with something much simpler.

Building the similarity matrix off the Levenshtein distance achieved competitive results to word embeddings for SimBow at SemEval 2017 and in my experiments as well; the inclusion of a corresponding method / function in Gensim could be beneficial.
Those actually were the relevant tests, but because the diagonals test case still needed to be corrected, I didn't necessarily want to take responsibility for it. I guess I've written that test for my purposes already too, so it won't be a bother. I'll open another ticket for just the tests and close it in my eventual pull request.

@Witiko Exactly my reasoning. I am implementing aspects of the Simbow paper, including Levenshtein, and saw that there was precedent to add it to gensim because softcosine had already been done. I am almost finished. (Though I am sure performance improvements could be made.)

https://github.com/nataliedurgin/gensim/commits/levenshtein-term-similarity-matrix
This issues surrounding unit tests for similarity_matrix can now be found #1961 and #1960.
@nataliedurgin sorry, I'm not familiar with the application here, but what edit distance functionality is actually needed? What operations specifically?

Is it a string-string query, or string-set_of_strings query? Is there an upper threshold on the "maximum relevant distance"? Is it really Levenshtein, or would a better edit-distance model help (such as discounting errors due to keyboard typos, omitted syllables, pronunciation similarity etc).

I'm asking because we've done a tremendous amount of work in this area in the past. We have some pretty well optimized algorithms and implementations for the various use-cases that fall under or expand "Levenshtein distance".
@piskvorky That would be fantastic if it exists. I couldn't find it in the repo. The application in SimBow at SemEval 2017 http://www.aclweb.org/anthology/S17-2051 (the paper mentioned by @Witiko) is constructing a word-word edit distance similarity matrix. I am not actually implementing levenshtein, but planned to use either pylev.levenshtein or distance.levenshtein (which would perhaps create unwanted external dependencies). I will happily substitute any method that takes in two word strings and spits back some notion of edit-distance. I am only implementing the functionality to take a Dictionary and output the term similarity matrix that can be fed into SoftCosineSimilarity. I need it for work regardless, but since there seemed to be existing interest in other portions of that paper, thought that I would offer it to the community. No worries either way!   
Aside: every time I read "softcos" my brain turns it into "soft tacos" and I get hungry. 
Well that's my question: what are the operations actually needed in this app? Are we comparing two static sets of words (which can be done much faster than a menial pairwise computation)?

Or do we have one static set of words and comparing that against a dynamic unknown string? If that's the case, some structures can be precomputed that make the dynamic distance computations much faster. Do we need only top-N most similar words? Everything under distance <= 3 (ignoring very distant words)?

It is rather rare that a full slavish N^2 Levenshtein is actually needed. And the runtime implications can be huge.
Gotcha. You are correct that what I really want is to input a pair of documents and output a score related to their edit distance. Since I have no idea how sensitive the results in the paper are to this particular method of creating a levenshtein term similarity matrix as input into soft cosine, my first instinct was to create a literal implementation. I'll hold off on the pull request and experiment internally with both a literal implementation and an existing gensim edit distance based similarity. 

At the very least this endevour has uncovered some bugs in existing functionality. 

Would you be willing to point me in the direction of existing edit-distance functionality? 

Thanks so much for your help! 
> Are we comparing two static sets of words (which can be done much faster than a menial pairwise computation)?

@piskvorky If we are actually building a term similarity matrix, then we will be computing the edit distance between all two-sets of words in the dictionary (unless we pre-filter some) regardless of the end application. This can likely be optimized already.
Aha, so we're comparing two static sets of words?

And do we care about all the distances = filling this word x word distance matrix fully, or only a subset, such as only top-N smallest values in each row, or only cell values <= 3 etc?
@nataliedurgin none of that work is open sourced (yet). I'm just trying to figure out if it's worth it, what the actual use case is here :)
Definitely only top-N closest in each row.

Cool, that should optimize very nicely. Nothing urgent -- IMO let's get the algo correct first and check the performance. See if it's actually worth optimizing = how much difference that would make in reality.
Thanks for reporting. CC @Witiko 
Just to expand the original issue report, the `get_similarities` method of `SoftCosineSimilarity` was taken verbatim from `WmdSimilarity`. Any digression from the expected behavior is likely to impact that class as well.
Not sure if this is the correct place to put this info, but I also noticed that these related unit tests do not seem to be run:  

https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_keyedvectors.py#L30

And if they did, I am not sure this is correctly checking for ones on the diagonal of the term similarity matrix: 

https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_keyedvectors.py#L40


You are correct, @nataliedurgin. Additionally, if the test were run, they would have failed at https://github.com/RaRe-Technologies/gensim/blob/f9669bb8a0b5b4b45fa8ff58d951a11d3178116d/gensim/test/test_keyedvectors.py#L38
since the up-to-date signature of `similarity_matrix` is `similarity_matrix(dictionary, tfidf, …)`. Do you have any idea why the tests are not executed by the CI services, @menshikh-iv?
I think unittest might require methods to be prefixed by 'test_' in order to find them. The tests might be formatted like this: 
https://github.com/nataliedurgin/gensim/commit/5bb6aea3a92b8e7ff7cecfca054a6057bb9b8fa8

All seem to pass in that case except for checking that there are ones on the diagonals. (Not to say they shouldn't be expanded.)

(Sorry I wasn't working on this issue. I was working on making a Levenshtein distance term similarity matrix function to feed into softcosine and was cannibalizing some of the existing term similarity matrix functionality.)  
Nice catch! This seems to be a test case that I sketched and then forgot about, since the CI service did not complain. It is somewhat unrelated to the original issue, but should be fixed nevertheless.
@Witiko @nataliedurgin can you fix it?
@piskvorky Not earlier than this weekend. @nataliedurgin has already fixed the unrelated tests, but the original issue still needs fixing and making into a new test case.
> (Sorry I wasn't working on this issue. I was working on making a Levenshtein distance term similarity matrix function to feed into softcosine and was cannibalizing some of the existing term similarity matrix functionality.)

Most of the method should be directly reusable if you are going for the same greedy algorithm that achieves a constant number of non-zero elements per a row / column. If you are going just for a constant threshold, like Sidorov et al. (2014) did, you will end up with something much simpler.

Building the similarity matrix off the Levenshtein distance achieved competitive results to word embeddings for SimBow at SemEval 2017 and in my experiments as well; the inclusion of a corresponding method / function in Gensim could be beneficial.
Those actually were the relevant tests, but because the diagonals test case still needed to be corrected, I didn't necessarily want to take responsibility for it. I guess I've written that test for my purposes already too, so it won't be a bother. I'll open another ticket for just the tests and close it in my eventual pull request.

@Witiko Exactly my reasoning. I am implementing aspects of the Simbow paper, including Levenshtein, and saw that there was precedent to add it to gensim because softcosine had already been done. I am almost finished. (Though I am sure performance improvements could be made.)

https://github.com/nataliedurgin/gensim/commits/levenshtein-term-similarity-matrix
This issues surrounding unit tests for similarity_matrix can now be found #1961 and #1960.
@nataliedurgin sorry, I'm not familiar with the application here, but what edit distance functionality is actually needed? What operations specifically?

Is it a string-string query, or string-set_of_strings query? Is there an upper threshold on the "maximum relevant distance"? Is it really Levenshtein, or would a better edit-distance model help (such as discounting errors due to keyboard typos, omitted syllables, pronunciation similarity etc).

I'm asking because we've done a tremendous amount of work in this area in the past. We have some pretty well optimized algorithms and implementations for the various use-cases that fall under or expand "Levenshtein distance".
@piskvorky That would be fantastic if it exists. I couldn't find it in the repo. The application in SimBow at SemEval 2017 http://www.aclweb.org/anthology/S17-2051 (the paper mentioned by @Witiko) is constructing a word-word edit distance similarity matrix. I am not actually implementing levenshtein, but planned to use either pylev.levenshtein or distance.levenshtein (which would perhaps create unwanted external dependencies). I will happily substitute any method that takes in two word strings and spits back some notion of edit-distance. I am only implementing the functionality to take a Dictionary and output the term similarity matrix that can be fed into SoftCosineSimilarity. I need it for work regardless, but since there seemed to be existing interest in other portions of that paper, thought that I would offer it to the community. No worries either way!   
Aside: every time I read "softcos" my brain turns it into "soft tacos" and I get hungry. 
Well that's my question: what are the operations actually needed in this app? Are we comparing two static sets of words (which can be done much faster than a menial pairwise computation)?

Or do we have one static set of words and comparing that against a dynamic unknown string? If that's the case, some structures can be precomputed that make the dynamic distance computations much faster. Do we need only top-N most similar words? Everything under distance <= 3 (ignoring very distant words)?

It is rather rare that a full slavish N^2 Levenshtein is actually needed. And the runtime implications can be huge.
Gotcha. You are correct that what I really want is to input a pair of documents and output a score related to their edit distance. Since I have no idea how sensitive the results in the paper are to this particular method of creating a levenshtein term similarity matrix as input into soft cosine, my first instinct was to create a literal implementation. I'll hold off on the pull request and experiment internally with both a literal implementation and an existing gensim edit distance based similarity. 

At the very least this endevour has uncovered some bugs in existing functionality. 

Would you be willing to point me in the direction of existing edit-distance functionality? 

Thanks so much for your help! 
> Are we comparing two static sets of words (which can be done much faster than a menial pairwise computation)?

@piskvorky If we are actually building a term similarity matrix, then we will be computing the edit distance between all two-sets of words in the dictionary (unless we pre-filter some) regardless of the end application. This can likely be optimized already.
Aha, so we're comparing two static sets of words?

And do we care about all the distances = filling this word x word distance matrix fully, or only a subset, such as only top-N smallest values in each row, or only cell values <= 3 etc?
@nataliedurgin none of that work is open sourced (yet). I'm just trying to figure out if it's worth it, what the actual use case is here :)
Definitely only top-N closest in each row.

Cool, that should optimize very nicely. Nothing urgent -- IMO let's get the algo correct first and check the performance. See if it's actually worth optimizing = how much difference that would make in reality.

Resolving patch:
diff --git a/gensim/similarities/docsim.py b/gensim/similarities/docsim.py
--- a/gensim/similarities/docsim.py
+++ b/gensim/similarities/docsim.py
@@ -922,7 +922,7 @@ def get_similarities(self, query):
 
         Parameters
         ----------
-        query : {list of (int, number), iterable of list of (int, number), :class:`scipy.sparse.csr_matrix`
+        query : {list of (int, number), iterable of list of (int, number)
             Document or collection of documents.
 
         Return
@@ -931,29 +931,29 @@ def get_similarities(self, query):
             Similarity matrix.
 
         """
-        if isinstance(query, numpy.ndarray):
-            # Convert document indexes to actual documents.
-            query = [self.corpus[i] for i in query]
 
-        if not query or not isinstance(query[0], list):
-            query = [query]
+        is_corpus, query = utils.is_corpus(query)
+        if not is_corpus:
+            if isinstance(query, numpy.ndarray):
+                # Convert document indexes to actual documents.
+                query = [self.corpus[i] for i in query]
+            else:
+                query = [query]
 
-        n_queries = len(query)
         result = []
-        for qidx in range(n_queries):
+        for query_document in query:
             # Compute similarity for each query.
-            qresult = [matutils.softcossim(document, query[qidx], self.similarity_matrix)
-                       for document in self.corpus]
+            qresult = [matutils.softcossim(query_document, corpus_document, self.similarity_matrix)
+                       for corpus_document in self.corpus]
             qresult = numpy.array(qresult)
 
             # Append single query result to list of all results.
             result.append(qresult)
 
-        if len(result) == 1:
-            # Only one query.
-            result = result[0]
-        else:
+        if is_corpus:
             result = numpy.array(result)
+        else:
+            result = result[0]
 
         return result
 
@@ -1038,7 +1038,7 @@ def get_similarities(self, query):
 
         Parameters
         ----------
-        query : {list of (int, number), iterable of list of (int, number), :class:`scipy.sparse.csr_matrix`
+        query : {list of (int, number), iterable of list of (int, number)
             Document or collection of documents.
 
         Return


Test patch:
diff --git a/gensim/test/test_similarities.py b/gensim/test/test_similarities.py
--- a/gensim/test/test_similarities.py
+++ b/gensim/test/test_similarities.py
@@ -17,10 +17,10 @@
 import scipy
 
 from smart_open import smart_open
-from gensim.corpora import Dictionary
 from gensim.models import word2vec
 from gensim.models import doc2vec
 from gensim.models import KeyedVectors
+from gensim.models import TfidfModel
 from gensim import matutils, similarities
 from gensim.models import Word2Vec, FastText
 from gensim.test.utils import (datapath, get_tmpfile,
@@ -373,8 +373,7 @@ def testIter(self):
 class TestSoftCosineSimilarity(unittest.TestCase, _TestSimilarityABC):
     def setUp(self):
         self.cls = similarities.SoftCosineSimilarity
-        self.dictionary = Dictionary(texts)
-        self.corpus = [dictionary.doc2bow(document) for document in texts]
+        self.tfidf = TfidfModel(dictionary=dictionary)
         similarity_matrix = scipy.sparse.identity(12, format="lil")
         similarity_matrix[dictionary.token2id["user"], dictionary.token2id["human"]] = 0.5
         similarity_matrix[dictionary.token2id["human"], dictionary.token2id["user"]] = 0.5
@@ -382,33 +381,53 @@ def setUp(self):
 
     def factoryMethod(self):
         # Override factoryMethod.
-        return self.cls(self.corpus, self.similarity_matrix)
+        return self.cls(corpus, self.similarity_matrix)
 
     def testFull(self, num_best=None):
         # Override testFull.
 
-        index = self.cls(self.corpus, self.similarity_matrix, num_best=num_best)
-        query = self.dictionary.doc2bow(texts[0])
+        # Single query
+        index = self.cls(corpus, self.similarity_matrix, num_best=num_best)
+        query = dictionary.doc2bow(texts[0])
         sims = index[query]
-
         if num_best is not None:
             # Sparse array.
             for i, sim in sims:
                 self.assertTrue(numpy.alltrue(sim <= 1.0))
                 self.assertTrue(numpy.alltrue(sim >= 0.0))
         else:
-            self.assertTrue(sims[0] == 1.0)  # Similarity of a document with itself is 1.0.
+            self.assertAlmostEqual(1.0, sims[0])  # Similarity of a document with itself is 1.0.
             self.assertTrue(numpy.alltrue(sims[1:] >= 0.0))
             self.assertTrue(numpy.alltrue(sims[1:] < 1.0))
             expected = 2.1889350195476758
             self.assertAlmostEqual(expected, numpy.sum(sims))
 
+        # Corpora
+        for query in (
+                corpus,  # Basic text corpus.
+                self.tfidf[corpus]):  # Transformed corpus without slicing support.
+            index = self.cls(query, self.similarity_matrix, num_best=num_best)
+            sims = index[query]
+            if num_best is not None:
+                # Sparse array.
+                for result in sims:
+                    for i, sim in result:
+                        self.assertTrue(numpy.alltrue(sim <= 1.0))
+                        self.assertTrue(numpy.alltrue(sim >= 0.0))
+            else:
+                for i, result in enumerate(sims):
+                    self.assertAlmostEqual(1.0, result[i])  # Similarity of a document with itself is 1.0.
+                    self.assertTrue(numpy.alltrue(result[:i] >= 0.0))
+                    self.assertTrue(numpy.alltrue(result[:i] < 1.0))
+                    self.assertTrue(numpy.alltrue(result[i + 1:] >= 0.0))
+                    self.assertTrue(numpy.alltrue(result[i + 1:] < 1.0))
+
     def testNonIncreasing(self):
         """ Check that similarities are non-increasing when `num_best` is not `None`."""
         # NOTE: this could be implemented for other similarities as well (i.e. in _TestSimilarityABC).
 
-        index = self.cls(self.corpus, self.similarity_matrix, num_best=5)
-        query = self.dictionary.doc2bow(texts[0])
+        index = self.cls(corpus, self.similarity_matrix, num_best=5)
+        query = dictionary.doc2bow(texts[0])
         sims = index[query]
         sims2 = numpy.asarray(sims)[:, 1]  # Just the similarities themselves.
 
@@ -419,8 +438,8 @@ def testNonIncreasing(self):
     def testChunking(self):
         # Override testChunking.
 
-        index = self.cls(self.corpus, self.similarity_matrix)
-        query = [self.dictionary.doc2bow(document) for document in texts[:3]]
+        index = self.cls(corpus, self.similarity_matrix)
+        query = [dictionary.doc2bow(document) for document in texts[:3]]
         sims = index[query]
 
         for i in range(3):
@@ -438,7 +457,7 @@ def testChunking(self):
     def testIter(self):
         # Override testIter.
 
-        index = self.cls(self.corpus, self.similarity_matrix)
+        index = self.cls(corpus, self.similarity_matrix)
         for sims in index:
             self.assertTrue(numpy.alltrue(sims >= 0.0))
             self.assertTrue(numpy.alltrue(sims <= 1.0))