Issue: dataset.search_batch() function outputs all -1 indices sometime.

I am working with RAG and playing around with different faiss indexes. At the moment I use **index = faiss.index_factory(768, "IVF65536_HNSW32,Flat")**.

During the retrieval phase exactly in [this line of retrieval_rag.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/retrieval_rag.py#L231) an error issue when all retrieved indices are -1.  Please refer to the screenshot of a PID worker. 

![image](https://user-images.githubusercontent.com/16892570/113782387-37a67600-9786-11eb-9c29-acad661a9648.png)


Here, my retrieve batch size is 2 and n_docs is 5. I can solve this by working around np. stack, but I want to ask, why we get an output index of -1. Do you have any idea :) ?

Is this a problem of the index, where the faiss can't find any similar vector?
Is there documentation on the output index being -1?

@lhoestq 
 

Comments:
Actually, I found the answer [here](https://github.com/facebookresearch/faiss/wiki/FAQ#what-does-it-mean-when-a-search-returns--1-ids). 

So we have to do some modifications to the code for instances where the index doesn't retrieve any IDs.

@lhoestq  @patrickvonplaten 

I also found another short bug in the retrieval part.   Especially, when retrieving documents. If Faiss returns the -1 as the index, the retriever will always use the last element in the dataset.

please check [def get_doc_dicts function](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/retrieval_rag.py#L222)


Does the use of the HNSW guarantee to retrieve valid indexes always? 



Hi !
No it happens sometimes to return -1, especially if your dataset is small.
If your dataset is big enough it shouldn't happen in my experience.

Ideally we should ignore all the -1 that are returned. It should be possible to change that in RAG's code 

I also checked with some indexes it returns more -1s. Specially with IVF
when nprobr is very low. It doesn't happen when using HNSW though. But at
the moment if it happens, dataset will always return the last element.
Maybe we should change it to repeat the most last valid retrieved doc id.
What do you think?

On Wed, Apr 7, 2021, 21:09 Quentin Lhoest ***@***.***> wrote:

> Hi !
> No it happens sometimes to return -1, especially if your dataset is small.
> If your dataset is big enough it shouldn't happen.
>
> Ideally we should ignore all the -1 that are returned. It should be
> possible to change that in RAG's code
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2175#issuecomment-814746509>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGTENOTLBEZTXEO2RS3THQOMPANCNFSM42PRVYDA>
> .
>


That would be an easy way to workaround this issue. Feel free to open a PR on `transformers` and ping me ! :)

Sure.  Will push everything together with RAG end to end. :) thanks a lot.

On Wed, Apr 7, 2021, 21:16 Quentin Lhoest ***@***.***> wrote:

> That would be an easy way to workaround this issue. Feel free to open a PR
> on transformers and ping me ! :)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2175#issuecomment-814752589>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGWLROCGARKN7WOJYSTTHQPH5ANCNFSM42PRVYDA>
> .
>