Repository: deepset-ai/haystack
Pull request: Cosine similarity for the rest of DocStores.
**Proposed changes**:
- as per https://github.com/deepset-ai/haystack/issues/1539#issuecomment-936979286

**Status (please check what you already did)**:
- [V ] First draft (up for discussions & feedback)
- [ ] Final code
- [ ] Added tests
- [ ] Updated documentation


Problem statement:
Implementing cosine similarity in Milvus and Weaviate doc stores
**Is your feature request related to a problem? Please describe.**
In Milvus docstore's documentation, it's mentioned that cosine similarity is not supported. In Weaviate's there is no mention of that fact, but from the source codes it can be seen that the similarity parameter is not used anywhere within the module.

**Describe the solution you'd like**
Can't we implement cosine on our side for both Milvus and Weaviate, since it's pretty easy?
I would suggest **normalize_embedding()** abstract method in the Base docstore, which would resolve to:

```python
def normalize_embedding(emb: np.ndarray)->np.ndarray:
  return faiss.normalize_L2(emb)
```
in the FAISS doc store, to a stub

```python
def normalize_embedding(emb: np.ndarray)->np.ndarray:
  return emb
```
in Elastic Search and InMemory doc stores, since they implement cosine similarity search directly, and to

```python
@njit(fastmath=True)
def normalize_embedding(emb: np.ndarray)->np.ndarray:
  norm=np.sqrt(emb.dot(emb))
  if norm==0.0:
    return emb
  else:
    return emb/norm

```
in Milvus and Weaviate.

_(in my testing np.sqrt(emb.dot(emb)) is twice faster than linalg.norm) And utilizing of Numba makes it another ~7 times faster._

normalize_embedding would be called at **write_documents**, **update_embeddings**, and **query_by_embedding** steps if sel.similarity is "cosine".

**write_documents()** method should acquire a new parameter, already_normalized: bool=False, which will only be used if documents come with precomputed embedding. With already_normalized=True, they will be treated as already normalized.

Plus maybe we could support some checking for normalization like **are_embeddings_normalized(fast:int=0)**, that, with fast=100, would sample 100 random embeddings and return if all they are normalized.

Smth like that. What do you think?

**Describe alternatives you've considered**
Are there some other useful transformations that can be applied to embeddings?

Hints from issue discussion:
Hi @fingoldo! I think it would definitely be nice to support cosine similarity across all document stores, given that it is recommended to use cosine similarity with sentence-transformers models. Nice that you already provided a way how this could be implemented! One minor thing: I think faiss' normalize_l2 method normalized the vector in place, so the method would probably need to look something like this:

```python
def normalize_embedding(emb: np.ndarray)->np.ndarray:
  faiss.normalize_L2(emb)
  return emb
```

@tholor What do you think about this?
> One minor thing: I think faiss' normalize_l2 method normalized the vector in place, so the method would probably need to look something like this:
> 
> ```python
> def normalize_embedding(emb: np.ndarray)->np.ndarray:
>   faiss.normalize_L2(emb)
>   return emb
> ```
> 
> @tholor What do you think about this?

Indeed, you are right. All needs to be done inplace then, actually it's no need to waste time allocating new memory. The signature becomes ->None then:

```python
@njit(fastmath=True)
def normalize_embedding(emb: np.ndarray)->None:
  norm=np.sqrt(emb.dot(emb))
  if norm!=0.0: emb/=norm
```
Yeah, it's a nice idea. We shouldn't rely on faiss though, as there's no need for this extra depenendency when running haystack with milvus / elastic etc
As far as I understood @fingoldo's suggestion, it relies on faiss only for the FAISSDocumentStore. For Milvus and Weaviate, he proposes to use numpy (possibly with numba) and for Elasticsearch and InMemory to keep on using the exisiting implementation.
Ok then we are aligned here. @fingoldo Are you interested in creating a PR for it? 
> > Ok then we are aligned here. @fingoldo Are you interested in creating a PR for it?
> 
> yes ) I don't mind, it should be fun and I will be honored. Would be my first contribution to open source )


Perfect! We really appreciate it :)
Working on it. Subtle question: in FAISS doc store, write_documents() method, when similarity is specified as cosine, the module performs normalization (by means of faiss). Is it not slightly misleading? Perhaps the user means that their vectors are normalized already and simply wants to use cosine later at the querying step? With L2 normalization it will be no harm to normalize twice, except the performance penalty, but if we'll support more options in the future it might be problematic. I think we have to either be explicit in the docs about unavoidable conversion, or, better, have a flag to skip or force conversion. What do you guys think?
Created a PR, please forgive any nonsense as I'm doing it the first time. Did not do any local tests, hoping for your automated testing. Ready to fix issues caused etc, waiting for your approval or suggestions.

Commit message: Fixed scores representation for cosine. Assuming Weavieate's rep needs no change.

Patch to review:
diff --git a/haystack/document_store/base.py b/haystack/document_store/base.py
index 0a68fc529a..75fa5f9eac 100644
--- a/haystack/document_store/base.py
+++ b/haystack/document_store/base.py
@@ -199,7 +199,14 @@ def get_document_by_id(self, id: str, index: Optional[str] = None) -> Optional[D
     @abstractmethod
     def get_document_count(self, filters: Optional[Dict[str, List[str]]] = None, index: Optional[str] = None) -> int:
         pass
-
+    
+    @abstractmethod
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings vector inplace.
+        """
+        pass        
+        
     @abstractmethod
     def query_by_embedding(self,
                            query_emb: np.ndarray,
diff --git a/haystack/document_store/elasticsearch.py b/haystack/document_store/elasticsearch.py
index 64e904313f..37136b2d45 100644
--- a/haystack/document_store/elasticsearch.py
+++ b/haystack/document_store/elasticsearch.py
@@ -745,7 +745,13 @@ def query(
 
         documents = [self._convert_es_hit_to_document(hit, return_embedding=self.return_embedding) for hit in result]
         return documents
-
+    
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings vector inplace.
+        """
+        pass
+        
     def query_by_embedding(self,
                            query_emb: np.ndarray,
                            filters: Optional[Dict[str, List[str]]] = None,
diff --git a/haystack/document_store/faiss.py b/haystack/document_store/faiss.py
index 30a5d409bd..fd286e8c09 100644
--- a/haystack/document_store/faiss.py
+++ b/haystack/document_store/faiss.py
@@ -15,9 +15,8 @@
 from haystack import Document
 from haystack.document_store.sql import SQLDocumentStore
 from haystack.retriever.base import BaseRetriever
-from haystack.utils import get_batches_from_generator
-from scipy.special import expit
 from haystack.document_store.base import DuplicateDocumentError
+from haystack.utils import get_batches_from_generator, normalize_vector_l2, finalize_raw_score
 
 logger = logging.getLogger(__name__)
 
@@ -197,8 +196,7 @@ def write_documents(self, documents: Union[List[dict], List[Document]], index: O
                 embeddings = [doc.embedding for doc in document_objects[i: i + batch_size]]
                 embeddings_to_index = np.array(embeddings, dtype="float32")
 
-                if self.similarity == 'cosine':
-                    faiss.normalize_L2(embeddings_to_index)
+                if self.similarity == 'cosine': self.normalize_embedding(embeddings_to_index)
 
                 self.faiss_indexes[index].add(embeddings_to_index)
 
@@ -278,8 +276,7 @@ def update_embeddings(
 
                 embeddings_to_index = np.array(embeddings, dtype="float32")
 
-                if self.similarity == 'cosine':
-                    faiss.normalize_L2(embeddings_to_index)
+                if self.similarity == 'cosine': self.normalize_embedding(embeddings_to_index)
 
                 self.faiss_indexes[index].add(embeddings_to_index)
 
@@ -415,7 +412,13 @@ def delete_documents(self, index: Optional[str] = None, filters: Optional[Dict[s
             else:
                 self.faiss_indexes[index].reset()
         super().delete_documents(index=index, filters=filters)
-
+    
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings vector inplace.
+        """
+        faiss.normalize_L2(emb)
+        
     def query_by_embedding(
         self,
         query_emb: np.ndarray,
@@ -447,8 +450,7 @@ def query_by_embedding(
 
         query_emb = query_emb.reshape(1, -1).astype(np.float32)
 
-        if self.similarity == 'cosine':
-            faiss.normalize_L2(query_emb)
+        if self.similarity == 'cosine': self.normalize_embedding(query_emb)
 
         score_matrix, vector_id_matrix = self.faiss_indexes[index].search(query_emb, top_k)
         vector_ids_for_query = [str(vector_id) for vector_id in vector_id_matrix[0] if vector_id != -1]
@@ -459,10 +461,8 @@ def query_by_embedding(
         scores_for_vector_ids: Dict[str, float] = {str(v_id): s for v_id, s in zip(vector_id_matrix[0], score_matrix[0])}
         for doc in documents:
             raw_score = scores_for_vector_ids[doc.meta["vector_id"]]
-            if self.similarity == 'cosine':
-                doc.score = (raw_score + 1) / 2
-            else:
-                doc.score = float(expit(np.asarray(raw_score / 100)))
+            doc.score = finalize_raw_score(raw_score,self.similarity)
+
             if return_embedding is True:
                 doc.embedding = self.faiss_indexes[index].reconstruct(int(doc.meta["vector_id"]))
 
diff --git a/haystack/document_store/memory.py b/haystack/document_store/memory.py
index 15d372a66d..e1c7872ef6 100644
--- a/haystack/document_store/memory.py
+++ b/haystack/document_store/memory.py
@@ -151,7 +151,13 @@ def get_documents_by_id(self, ids: List[str], index: Optional[str] = None) -> Li
         index = index or self.index
         documents = [self.indexes[index][id] for id in ids]
         return documents
-
+    
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings vector inplace.
+        """
+        pass
+        
     def query_by_embedding(self,
                            query_emb: np.ndarray,
                            filters: Optional[Dict[str, List[str]]] = None,
diff --git a/haystack/document_store/milvus.py b/haystack/document_store/milvus.py
index 7e317c16ce..22897fb81c 100644
--- a/haystack/document_store/milvus.py
+++ b/haystack/document_store/milvus.py
@@ -5,14 +5,13 @@
 import numpy as np
 
 from milvus import IndexType, MetricType, Milvus, Status
-from scipy.special import expit
 from tqdm import tqdm
 
 from haystack import Document
 from haystack.document_store.sql import SQLDocumentStore
 from haystack.retriever.base import BaseRetriever
-from haystack.utils import get_batches_from_generator
 from haystack.document_store.base import DuplicateDocumentError
+from haystack.utils import get_batches_from_generator, normalize_vector_l2, finalize_raw_score
 
 logger = logging.getLogger(__name__)
 
@@ -71,7 +70,7 @@ def __init__(
          (From https://milvus.io/docs/v1.0.0/performance_faq.md#How-can-I-get-the-best-performance-from-Milvus-through-setting-index_file_size)
         :param similarity: The similarity function used to compare document vectors. 'dot_product' is the default and recommended for DPR embeddings.
                            'cosine' is recommended for Sentence Transformers, but is not directly supported by Milvus.
-                           However, you can normalize your embeddings and use `dot_product` to get the same results.
+                           However, Haystack can normalize your embeddings and use `dot_product` to get the same results.
                            See https://milvus.io/docs/v1.0.0/metric.md?Inner-product-(IP)#floating.
         :param index_type: Type of approximate nearest neighbour (ANN) index used. The choice here determines your tradeoff between speed and accuracy.
                            Some popular options:
@@ -110,15 +109,15 @@ def __init__(
         self.vector_dim = vector_dim
         self.index_file_size = index_file_size
 
-        if similarity == "dot_product":
+        if similarity == "dot_product" or similarity == "cosine":
             self.metric_type = MetricType.IP
             self.similarity = similarity
         elif similarity == "l2":
             self.metric_type = MetricType.L2
             self.similarity = similarity
         else:
-            raise ValueError("The Milvus document store can currently only support dot_product and L2 similarity. "
-                             "Please set similarity=\"dot_product\" or \"l2\"")
+            raise ValueError("The Milvus document store can currently only support dot_product, cosine and L2 similarity. "
+                             "Please set similarity=\"dot_product\", \"cosine\", or \"l2\"")
 
         self.index_type = index_type
         self.index_param = index_param or {"nlist": 16384}
@@ -212,9 +211,16 @@ def write_documents(self, documents: Union[List[dict], List[Document]], index: O
                     for doc in document_batch:
                         doc_ids.append(doc.id)
                         if isinstance(doc.embedding, np.ndarray):
+                            if self.similarity == 'cosine': self.normalize_embedding(doc.embedding)
                             embeddings.append(doc.embedding.tolist())
                         elif isinstance(doc.embedding, list):
-                            embeddings.append(doc.embedding)
+                            if self.similarity == 'cosine':
+                                # temp conversion to ndarray
+                                np_embedding = np.array(doc.embedding, dtype="float32")
+                                self.normalize_embedding(np_embedding)
+                                embeddings.append(np_embedding.tolist())
+                            else:                            
+                                embeddings.append(doc.embedding)
                         else:
                             raise AttributeError(f'Format of supplied document embedding {type(doc.embedding)} is not '
                                                  f'supported. Please use list or numpy.ndarray')
@@ -296,6 +302,10 @@ def update_embeddings(
                 self._delete_vector_ids_from_milvus(documents=document_batch, index=index)
 
                 embeddings = retriever.embed_passages(document_batch)  # type: ignore
+                if self.similarity == 'cosine':
+                    for embedding in embeddings:
+                        self.normalize_embedding(embedding)
+                
                 embeddings_list = [embedding.tolist() for embedding in embeddings]
                 assert len(document_batch) == len(embeddings_list)
 
@@ -313,7 +323,10 @@ def update_embeddings(
 
         self.milvus_server.flush([index])
         self.milvus_server.compact(collection_name=index)
-
+    
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        normalize_vector_l2(emb)
+        
     def query_by_embedding(self,
                            query_emb: np.ndarray,
                            filters: Optional[dict] = None,
@@ -345,7 +358,10 @@ def query_by_embedding(self,
             return_embedding = self.return_embedding
         index = index or self.index
 
-        query_emb = query_emb.reshape(1, -1).astype(np.float32)
+        if self.similarity == 'cosine': self.normalize_embedding(query_emb)
+        
+        query_emb = query_emb.reshape(1, -1).astype(np.float32)               
+        
         status, search_result = self.milvus_server.search(
             collection_name=index,
             query_records=query_emb,
@@ -369,7 +385,7 @@ def query_by_embedding(self,
 
         for doc in documents:
             raw_score = scores_for_vector_ids[doc.meta["vector_id"]]
-            doc.score = float(expit(np.asarray(raw_score / 100)))
+            doc.score = finalize_raw_score(raw_score,self.similarity)
 
         return documents
 
diff --git a/haystack/document_store/sql.py b/haystack/document_store/sql.py
index 62243ffb34..c406346d35 100644
--- a/haystack/document_store/sql.py
+++ b/haystack/document_store/sql.py
@@ -447,7 +447,19 @@ def _convert_sql_row_to_label(self, row) -> Label:
             id=row.id
         )
         return label
-
+    
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings vector inplace.
+        """
+        pass
+    
+    def normalize_documents_embeddings(self, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings of already existing documents.
+        """
+        pass
+        
     def query_by_embedding(self,
                            query_emb: np.ndarray,
                            filters: Optional[dict] = None,
diff --git a/haystack/document_store/weaviate.py b/haystack/document_store/weaviate.py
index 9ed29bb0f7..1abfdea01f 100644
--- a/haystack/document_store/weaviate.py
+++ b/haystack/document_store/weaviate.py
@@ -5,7 +5,7 @@
 
 from haystack import Document
 from haystack.document_store.base import BaseDocumentStore
-from haystack.utils import get_batches_from_generator
+from haystack.utils import get_batches_from_generator, normalize_vector_l2, finalize_raw_score
 
 from weaviate import client, auth, AuthClientPassword
 from weaviate import ObjectsBatchRequest
@@ -227,7 +227,10 @@ def _convert_weaviate_result_to_document(
 
         if return_embedding and embedding:
             embedding = np.asarray(embedding, dtype=np.float32)
-
+        
+        #score = finalize_raw_score(score,self.similarity)
+        #if self.similarity == "cosine": score = ((logit(score) * 100) + 1) / 2
+        
         document = Document(
             id=id,
             text=text,
@@ -379,6 +382,8 @@ def write_documents(
 
                     doc_id = str(_doc.pop("id"))
                     vector = _doc.pop(self.embedding_field)
+                    if self.similarity == 'cosine': self.normalize_embedding(vector)
+                    
                     if _doc.get(self.faq_question_field) is None:
                         _doc.pop(self.faq_question_field)
 
@@ -570,7 +575,14 @@ def query(
             documents.append(doc)
 
         return documents
-
+    
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings vector inplace.
+        """
+        normalize_vector_l2(emb)
+        
+        
     def query_by_embedding(self,
                            query_emb: np.ndarray,
                            filters: Optional[dict] = None,
@@ -596,7 +608,10 @@ def query_by_embedding(self,
         properties = self._get_current_properties(index)
         properties.append("_additional {id, certainty, vector}")
 
-        query_emb = query_emb.reshape(1, -1).astype(np.float32)
+        if self.similarity == 'cosine': self.normalize_embedding(query_emb)
+        
+        query_emb = query_emb.reshape(1, -1).astype(np.float32)                
+        
         query_string = {
             "vector" : query_emb
         }
@@ -676,6 +691,7 @@ def update_embeddings(
                                    "Specify the arg `embedding_dim` when initializing WeaviateDocumentStore()")
             for doc, emb in zip(document_batch, embeddings):
                 # Using update method to only update the embeddings, other properties will be in tact
+                if self.similarity == 'cosine': self.normalize_embedding(emb)
                 self.weaviate_client.data_object.update({}, class_name=index, uuid=doc.id, vector=emb)
 
     def delete_all_documents(self, index: Optional[str] = None, filters: Optional[Dict[str, List[str]]] = None):
diff --git a/haystack/utils.py b/haystack/utils.py
index 60dca8b1c7..800768da16 100644
--- a/haystack/utils.py
+++ b/haystack/utils.py
@@ -1,15 +1,19 @@
 import json
-from collections import defaultdict
+import time
+import subprocess
 from itertools import islice
-import logging
+from collections import defaultdict
+
 import pprint
-import pandas as pd
+import logging
 from typing import Dict, Any, List, Optional
 from haystack.document_store.sql import DocumentORM
-import subprocess
-import time
-import torch
 
+import torch
+import numpy as np
+import pandas as pd
+from numba import njit
+#from scipy.special import expit, logit
 
 logger = logging.getLogger(__name__)
 
@@ -246,3 +250,23 @@ def get_batches_from_generator(iterable, n):
     while x:
         yield x
         x = tuple(islice(it, n))
+
+@njit(fastmath=True)
+def normalize_vector_l2(emb: np.ndarray)->None:
+    """
+        Performs L2 normalization of embeddings vector inplace.
+    """
+    norm = np.sqrt(emb.dot(emb))
+    if norm != 0.0:
+        emb /= norm
+
+@njit(fastmath=True)
+def expit(x: float) -> float:
+    return 1 / (1 + np.exp(-x))
+    
+@njit(fastmath=True)
+def finalize_raw_score(raw_score:float,similarity:str)->float:
+    if similarity == 'cosine':
+        return (raw_score + 1) / 2
+    else:
+        return float(expit(raw_score / 100))
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index 31823af3ed..99d9864720 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -61,3 +61,4 @@ SPARQLWrapper
 mmh3
 weaviate-client==2.5.0
 ray==1.5.0
+numba
\ No newline at end of file


Code review comments:
diff_hunk: @@ -246,3 +250,23 @@ def get_batches_from_generator(iterable, n):
     while x:
         yield x
         x = tuple(islice(it, n))
+
+@njit(fastmath=True)
+def normalize_vector_l2(emb: np.ndarray)->None:
path: haystack/utils.py
text: @user1:
The code structure is here now quite nested. We have an abstract `normalize_embedding` in the base class, then the implementations by the child classes and the child classes use this method.

IMO it would simplify things if we moved this code into the method of the base class and just overwrite it in FAISS with the custom normalization there. 

@author:
Yes makes sense. OK will change that.

diff_hunk: @@ -61,3 +61,4 @@ SPARQLWrapper
 mmh3
 weaviate-client==2.5.0
 ray==1.5.0
+numba
path: requirements.txt
text: @user1:
Can you please run a quick script to compare the speed of numba vs. plain numpy for the normalization? If the benefit is not significant, we would like to avoid adding any extra dependencies to Haystack.

@author:
Yes, I did that testing already and Numba gave ~7x improvement over plain NumPy. But if extra dependencies are a concern, we can, by switching from njit decorator to njit wrapper, use Numba only if is installed, and remove it from dependencies. What do you think?

@user1:
Yes, that sounds good! Let's make Numba optional then and remove it from dependencies.

diff_hunk: @@ -227,7 +227,10 @@ def _convert_weaviate_result_to_document(
 
         if return_embedding and embedding:
             embedding = np.asarray(embedding, dtype=np.float32)
-
+        
+        #score = finalize_raw_score(score,self.similarity)
+        #if self.similarity == "cosine": score = ((logit(score) * 100) + 1) / 2
+        
path: haystack/document_store/weaviate.py
text: @user1:
Some left overs from dev?

@author:
Was not sure ) Will remove )

@author:
We will actually need this if we'll want to have alternative normalization methods, i think. raw score will need to be computed as weaviate returns processed score already, not a raw one, if I'm correct.

diff_hunk: @@ -447,7 +447,19 @@ def _convert_sql_row_to_label(self, row) -> Label:
             id=row.id
         )
         return label
-
+    
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
+            Performs L2 normalization of embeddings vector inplace.
+        """
+        pass
+    
+    def normalize_documents_embeddings(self, kind:str="L2")->None:
+        """
path: haystack/document_store/sql.py
text: @user1:
Couldn't find the actual implementation for this one

@author:
Yes, forgot to delete it. I wanted to have a method to convert existing embedding to normalized one but realized it would be used not frequently at all and add way too much coding hassle. Will delete.

diff_hunk: @@ -199,7 +199,14 @@ def get_document_by_id(self, id: str, index: Optional[str] = None) -> Optional[D
     @abstractmethod
     def get_document_count(self, filters: Optional[Dict[str, List[str]]] = None, index: Optional[str] = None) -> int:
         pass
-
+    
+    @abstractmethod
+    def normalize_embedding(self, emb: np.ndarray, kind:str="L2")->None:
+        """
path: haystack/document_store/base.py
text: @user1:
Maybe I missed it but where is the `kind` parameter actually used? If it's not used, let's drop it. 

@author:
Yes, I could use it but there were no alternatives currently so I skipped trivial `if kind=="L2": proceed`. And what if we'll want to use other normalization methods in the future? Will add that parameter back then?

@user1:
Yes, we should remove it then and only introduce it when it's actually needed in future :)

@author:
Or, instead, I could implement ISC normalization that is L1/Helinger-based, too? from here https://journalofbigdata.springeropen.com/articles/10.1186/s40537-017-0083-6
It's reported to lead to higher classification accuracy on 5 real-world NLP datasets (when served as a feature), over a range of classifications. I would be keen to see results on my task, actually ).

@user1:
Before adding ISC normalization, we'd actually need some benchmarks to see how it really works in the semantic search case as we only add "curated options" to users. I'd prefer to rather get this PR finished in the next days and focus only on the L2 normalization here (we will have major refactorings coming up in the next week, which might complicate merging this PR later). If you are interested, you could of course explore other normalizations separately and if results are promising we can do a second PR. 

@author:
I am testing ISC now, needs some hours to finish. Will report the results back.

@author:
@user1  ISC seems to give meaningful results. I can't tell right away from just one task that it's better than cosine, but it's at least not worse. Here's the (TOP-10) results of querying an extensive collection of research papers (1.9M) for a specific topic, for cosine and isc metrics, respectively: [cosine](https://ibb.co/wzyxFwt) and [isc](https://ibb.co/gvd8Ztg). Results of ISC are definitely valuable and meaningful for my particular task. Since I have already added it, is there a chance that we can keep it? It would be so painful deleting it ) Rest of the discussed changes I have implemented.


@user1:
Thanks for testing it in a qualitative way :). 
From the screenshots it seems a bit suspicious to me that all ISC scores are 0.761. Evaluating the quality here from such a small sample and without gold annotations is super difficult, I think. 
We should at least benchmark it once on a retrieval dataset and collect quantiative metrics. You can use this benchmark script here to run it on the Natural Questions dev set and compare metrics to a regular `dot_product`: https://github.com/deepset-ai/haystack/blob/master/test/benchmarks/retriever_simplified.py  

My biggest doubt is that models like DPR were trained on dot product and therefore probably won't work better with ISC. 
Also: not sure about the implementation details yet as I didn't have time to read the paper in detail yet - can it be achieved with a pure normalization trick?  

My recommendation would still be to finish this PR first and then conduct the benchmarking / addition of ISC. Otherwise you will risk quite some merge conflicts once #1624 is merged. 

@author:
Yes good questions, there are restrictions as vectors can contain negative elements and I had to add 0.15 first to all elements to guarantee non-negativity (for sqrt to work) for my particular model. Yes, most likely they will work worse for DPR-class models, but for STS-class they can work same good or better. Ok let's go without ISC for now. Should I keep a second parameter to the normalize_embedding function? This time using it in the comparison
```python

    def normalize_embedding(self, emb: np.ndarray, similarity:str="cosine")->None:
        """
            Performs normalization of embeddings vector inplace.
        """
        if similarity=="cosine":
            normalize_vector_l2(emb)
```
with plans to extend it in future with

```python

        elif similarity=="isc":
            normalize_vector_l1(emb)
```
Or revert all back to just a single parameter for now?

@user1:
Let's remove the second parameter for a better code quality - won't be much work to add it later if needed :)