Fix rag finetuning + add finetuning test
Author: lhoestq
Pull request: yes

Following #7715 we need more test coverage of the RAG example scripts.
In this PR I'm adding a test for the finetuning script.
The test includes a single gpu test and a multi gpu test. Both are passing.

As mentioned in #7816 and #8345 there were some errors in the script that I had to fix.

Moreover since @amogkam has been working on the finetuning script as well to integrate Ray, I made sure to reduce the possible conflicts with his PR #8583 . More precisely I'm reusing the CustomAccel class that will allow to init either the pytorch distributed retrieval or the ray distributed retrieval.

Also fixed a bug in RAG forward pass (see #8665 )

Fix #7816
Fix #8345

Comments:
@lhoestq 

Hi, I tried to execute finetune.py on two GPUs.  It mainly fails with the following error. But when I run with the single GPU it works. I have also attached a screen shot.



**RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [130.216.209.142]:55728**

![Selection_006](https://user-images.githubusercontent.com/16892570/99660519-57831900-2ac7-11eb-933b-a99845ee3516.png)
![Selection_007](https://user-images.githubusercontent.com/16892570/99660537-5c47cd00-2ac7-11eb-9e91-272f2dd7f007.png)



What command did you run exactly ?

> What command did you run exactly ?

`python examples/rag/finetune.py     --data_dir ./examples/rag/test_data/dummy_seq2seq     --output_dir ./examples/rag/outputs   --model_name_or_path facebook/rag-token-base      --model_type rag_sequence       --do_train     --do_predict     --n_val -1     --val_check_interval 0.25     --train_batch_size 1     --eval_batch_size 1     --max_source_length 128     --max_target_length 25     --val_max_target_length 25     --test_max_target_length 25     --label_smoothing 0.1     --dropout 0.1     --attention_dropout 0.1     --weight_decay 0.001     --adam_epsilon 1e-08     --max_grad_norm 0.1     --lr_scheduler polynomial     --learning_rate 3e-05     --num_train_epochs 100     --warmup_steps 500     --gradient_accumulation_steps 1    --index_name custom    --passages_path ./examples/rag/data/my_knowledge_dataset    --index_path  ./examples/rag/data/my_knowledge_dataset_hnsw_index.faiss --gpus 2
`

Does changing the port with `--distributed_port 8888` help in your case ?

It says, 
`finetune.py: error: unrecognized arguments: --distributed-port 8888`


I tried with `--distributed-port 8888` still gives the same error. 
btw my torch version is **Version: 1.7.0+cu110**




What's your pytorch lightning version ?
(also sorry I misspelled distributed-port)

> pytorch lightning

**Version: 1.0.4**


@lhoestq 

Hi just wanted to know .. did you managed to run the finetune.sh script
without any errors.



Yes I have no issue on my side. The finetuning test works fine too.
Could you try to update pytorch lightning and see if it fixes your issue ?
Let me know if you manage to fix it

Did you try with custom index ?

On Fri, Nov 20, 2020, 22:43 Quentin Lhoest <notifications@github.com> wrote:

> Yes I have no issue on my side. The finetuning test works fine too.
> Could you try to update pytorch lightning and see if it fixes your issue ?
> Let me know if you manage to fix it
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/8585#issuecomment-731061351>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGRRHXMLZB2KC5WTSJ3SQY25LANCNFSM4TYMOBIQ>
> .
>


Let me test right now

Can you also send me your pytorch and tranformers versions.




On Fri, Nov 20, 2020, 22:49 Shamane Siriwardhana <shamane@ahlab.org> wrote:

> Did you try with custom index ?
>
> On Fri, Nov 20, 2020, 22:43 Quentin Lhoest <notifications@github.com>
> wrote:
>
>> Yes I have no issue on my side. The finetuning test works fine too.
>> Could you try to update pytorch lightning and see if it fixes your issue ?
>> Let me know if you manage to fix it
>>
>> —
>> You are receiving this because you commented.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/transformers/pull/8585#issuecomment-731061351>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AEA4FGRRHXMLZB2KC5WTSJ3SQY25LANCNFSM4TYMOBIQ>
>> .
>>
>


Awesome I managed to reproduce your issue using the custom index :)
I will investigate
And I'm using pytorch 1.7.0 cu11 and transformers with the latest changes from master and this PR

Perfect. I feel it is some tensor issue that happens in the validation
sanity check.

On Fri, Nov 20, 2020, 23:02 Quentin Lhoest <notifications@github.com> wrote:

> Awesome I managed to reproduce your issue using the custom index :)
> I will investigate
> And I'm using pytorch 1.7.0 cu11 and transformers with the latest changes
> from master and this PR
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/8585#issuecomment-731071657>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGXTIZMVU76CGOR3FZTSQY5EHANCNFSM4TYMOBIQ>
> .
>


Indeed it was an issue with the precision of the tensor. I'm fixing it

Amazing. If you can .. once you fixed this can you please add finetuning
commands for a custom dataset in the read me. The current one  is not with
all commands. I really think this RAG framework will be a game changer if
we can apply cor other tasks:)

On Fri, Nov 20, 2020, 23:18 Quentin Lhoest <notifications@github.com> wrote:

> Indeed it was an issue with the precision of the tensor. I'm fixing it
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/8585#issuecomment-731080487>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGWGLNSO3ITQGNLLUYTSQY7BFANCNFSM4TYMOBIQ>
> .
>


Ok I fixed the tensor issue and updated the readme

I also had to rename some the examples files of RAG to avoid collisions with the files of the seq2seq examples. The name collision broke the CI tests with failed imports. 

I did:
```
examples/rag/utils.py -> exmaples/rag/utils_rag.py
examples/rag/callbacks.py -> exmaples/rag/callbacks_rag.py
examples/rag/finetune.py -> exmaples/rag/finetune_rag.py
examples/rag/finetune.sh -> exmaples/rag/finetune_rag.sh
```

All tests are green now :)

Thanks a lot for your quick response.

On Sat, Nov 21, 2020, 00:15 Quentin Lhoest <notifications@github.com> wrote:

> Ok I fixed the tensor issue and updated the readme
>
> I also had to rename some the examples files of RAG to avoid collisions
> with the files of the seq2seq examples. The name collision broke the CI
> tests with failed imports.
>
> I did:
>
> examples/rag/utils.py -> exmaples/rag/utils_rag.py
> examples/rag/callbacks.py -> exmaples/rag/callbacks_rag.py
> examples/rag/finetune.py -> exmaples/rag/finetune_rag.py
> examples/rag/finetune.sh -> exmaples/rag/finetune_rag.sh
>
> All tests are green now :)
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/8585#issuecomment-731108211>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGTA626DKBJ5S7JXLFDSQZFVNANCNFSM4TYMOBIQ>
> .
>


I took your comment into account @patrickvonplaten 
The only thing I didn't change is the return_dict=True - I kept them to avoid playing with tuples indices.

@lhoestq hello, thank you for this amazing feature.

when I try to create my custom dataset I receveing this error:

`2020-12-16 00:48:44.645715: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
INFO:__main__:Step 1 - Create the dataset
Using custom data configuration default
Reusing dataset csv (/root/.cache/huggingface/datasets/csv/default-d44cf86c96b535d8/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2)
Loading cached processed dataset at /root/.cache/huggingface/datasets/csv/default-d44cf86c96b535d8/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2/cache-ad363af188e673b0.arrow
100% 1/1 [00:00<00:00, 10.92ba/s]
INFO:__main__:Step 2 - Index the dataset
Traceback (most recent call last):
  File "examples/rag/use_own_knowledge_dataset.py", line 200, in <module>
    main(rag_example_args, processing_args, index_hnsw_args)
  File "examples/rag/use_own_knowledge_dataset.py", line 102, in main
    index = faiss.IndexHNSWFlat(index_hnsw_args.d, index_hnsw_args.m, faiss.METRIC_INNER_PRODUCT)
  File "/usr/local/lib/python3.6/dist-packages/faiss/swigfaiss.py", line 3746, in __init__
    this = _swigfaiss.new_IndexHNSWFlat(*args)
NotImplementedError: Wrong number or type of arguments for overloaded function 'new_IndexHNSWFlat'.
  Possible C/C++ prototypes are:
    faiss::IndexHNSWFlat::IndexHNSWFlat()
    faiss::IndexHNSWFlat::IndexHNSWFlat(int,int)`


I'm using Google Colab to test this - https://colab.research.google.com/drive/1Cjj18rYmeS0Bueis_KPB5Wbybl-JNDLL?usp=sharing



Well, i didn't install the specific dependencies you defined. excuse me.

Solved running - !pip install -r /transformers/examples/rag/requirements.txt

At least it is registered if someone has the same problem. haha