[end2end rag] Slow speed when extending the external KB
Author: shunyuzh
Pull request: no

Hi folks,

@shamanez  Sorry to disturb you for some problems. It mainly about process about re-encode and re-index, related to kb_encode_utils.py and finetune_rag.py. 

First, When I extended your provided SQUAD-KB.csv to another five times larger file, the re-index process became too slow, sometimes even needing half or one hour to finish re-indexing. I am trying to use faiss-gpu to speed the re-index process, but it doesn't work well. 

If you are interested at the above BUG, you can try using the small split of dpr_wiki to test:  https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/psgs_w100.tsv.pkl

Second, I found that when I set index_gpus to more than 2, the prepare time for re-encoding became longer even to 5 or 10 minutes . I guess this cost mainly due to the I/O of load_dataset or spilt in def embed_update(ctx_encoder, total_processes, device, process_num, shard_dir, csv_path). 

Overall, these problems only occur when extending the external knowledge corpus. It works well when using your provided small squad-kb.csv.

Feel free to give any suggestion on what you are intereted at, in the way as you like.


Comments:
Hi, Thanks a lot for trying out our model and pointing out these valuable facts. I checked your problem and didn't see any problem with GPU selection, but yeah the time can increase, dramatically if we use a large dataset with the current index. So please go through my answer below. 

Actually, it is not a bug. If you only check the time taken to re-encode, you can always reduce it by using a lot more GPUs, because encoding process is embarrassingly parallel. In the encoding process, the model creates dataset splits using the HF dataset library and saves them into a disk. Then, those splits will only be merged when we need to start the re-indexing process.  So this increase of time is not because of having more GPUs, but because of the re-indexing process.  In re-indexing, we use FAISS and HNSW index. I had some long chats with FAISS people and they say usually HNSW index time is slow, and changes according to the number of vectors and status of the vectors (since we are changing the embeddings). This bug can be solved by using another index like IVF, which is very fast.   







Yes, it's just related to larger dataset. Once the process of re-encoding started, it goes fast. But your mentioned process of creating dataset splits, saving them into a disk and merging them from the disk indeed cost much time.

As for the re-index process, I agree that your said IVF and other hyperparameters of Faiss may helps. If you find what setting really works better, please let me know.

@Dopaminezsy

sure.  I think you might need to do hyperparameters tuning. Anyways  I trained a model where that the external KB consisted of 7.5 million passages. Although  KB update time has increased it worked fine. Another thing is if you have access to enough computational power you can easily make the entire process much more efficient.   When it comes to the indexing process, you can try completely neglecting it and using a greedy search during the training.  I have noticed this method in REALM paper. 

On Mon, Aug 2, 2021 at 8:52 PM Dopaminezsy ***@***.***> wrote:

> Yes, it's just related to larger dataset. Once the process of re-encoding
> started, it goes fast. But your mentioned process of creating dataset
> splits, saving them into a disk and merging them from the disk indeed cost
> much time.
>
> As for the re-index process, I agree that your said IVF and other
> hyperparameters of Faiss may helps. If you find what setting really works
> better, please let me know.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12949#issuecomment-890849702>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGRYJLGPI2VLXEYDL5TT2ZMFLANCNFSM5BIEELFA>
> .
>


-- 
[image: Augmented Human Lab] <http://www.ahlab.org/> [image: uni]
<https://www.auckland.ac.nz/en/abi.html>

Gayal Shamane
Ph.D. Candidate
Augmented Human Lab
Auckland Bioengineering Institute | The University of Auckland


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.