Unable to finetune RAG end2end due to error in finetune_rag.py file
Author: AntonioLopardo
Pull request: no

## Environment info
- `transformers` version: 4.8.2
- Platform: Ubuntu 20.04
- Python version: 3.8.10
- PyTorch version (GPU?): 1.10.0.dev20210717+cu111
- Tensorflow version (GPU?): none
- Using GPU in script?: yes, 8 A6000s
- Using distributed or parallel set-up in script?: yes, ray 2.0.0.dev0

### Who can help
Quentin Lhoest (@lhoestq), Patrick von Platen (@patrickvonplaten)

## Information

Model I am using (Bert, XLNet ...): RAG 

The problem arises when using:
* [ ] the official example scripts: I used the finetune_rag_ray_end2end.sh script, but it doesn't run in my env with either ray or pytorch as distributed retriever mode. 
* [ ] my own modified scripts: I only modified the args to finetune_rag.py
``` python finetune_rag.py \
    --data_dir  squad-training \
    --output_dir model_checkpoints \
    --model_name_or_path facebook/rag-token-base \
    --model_type rag_token \
    --fp16 \
    --gpus 8  \
    --profile \
    --do_train \
    --end2end \
    --do_predict \
    --n_val -1  \
    --train_batch_size 8 \
    --eval_batch_size 1 \
    --max_source_length 128 \
    --max_target_length 25 \
    --val_max_target_length 25 \
    --test_max_target_length 25 \
    --label_smoothing 0.1 \
    --dropout 0.1 \
    --attention_dropout 0.1 \
    --weight_decay 0.001 \
    --adam_epsilon 1e-08 \
    --max_grad_norm 0.1 \
    --lr_scheduler polynomial \
    --learning_rate 3e-05 \
    --num_train_epochs 10 \
    --warmup_steps 500 \
    --gradient_accumulation_steps 8 \
    --distributed_retriever pytorch \
    --num_retrieval_workers 4  \
    --passages_path SQUAD-KB/my_knowledge_dataset \
    --index_path  SQUAD-KB/my_knowledge_dataset_hnsw_index.faiss \
    --index_name custom \
    --context_encoder_name facebook/dpr-ctx_encoder-multiset-base \
    --csv_path SQUAD-KB/squad-kb.csv \
    --index_gpus 1 \
    --gpu_order [5,6,7,8,9,0,1,2,3,4] \
    --shard_dir test_dir/kb-shards \
    --indexing_freq 500 
```

The tasks I am working on is:
* [ ] my own task or dataset: I'm working with different datasets, but for now I've only tested on the data made available by @shamanez [here](https://drive.google.com/drive/folders/1qyzV-PaEARWvaU_jjpnU_NUS3U_dSjtG?usp=sharing) 

## To reproduce

Steps to reproduce the behavior:

0. pip install torch, transformers, pytorch_lightning, ray[default] 
1. Download the data from [here](https://drive.google.com/drive/folders/1qyzV-PaEARWvaU_jjpnU_NUS3U_dSjtG?usp=sharing) and put the SQUAD-KB and squad-training directories in the same directory of the script
2. Change the args to finetune_rag.py
3. testing with ‘distributed_retriever pytorch‘ I get this error 
```
INFO:__main__:please use RAY as the distributed retrieval method
Traceback (most recent call last):
  File "finetune_rag.py", line 793, in <module>
    main(args)
  File "finetune_rag.py", line 730, in main
    model: GenerativeQAModule = GenerativeQAModule(args)
  File "finetune_rag.py", line 138, in __init__
    model = self.model_class.from_pretrained(hparams.model_name_or_path, config=config, retriever=retriever)
UnboundLocalError: local variable 'retriever' referenced before assignment
Stopped all 12 Ray processes.
```
4. testing with ‘distributed_retriever ray‘ I get this error
```
Global seed set to 42
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/8
test_dir/kb-shards
2021-07-18 10:38:55,472	INFO worker.py:805 -- Connecting to existing Ray cluster at address: 135.181.63.142:6379
INFO:__main__:Getting named actors for NODE_RANK 0, LOCAL_RANK 7
Traceback (most recent call last):
  File "/root/Retr_Exp/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 793, in <module>
    main(args)
  File "/root/Retr_Exp/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 725, in main
    named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
  File "/root/Retr_Exp/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 725, in <listcomp>
    named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
  File "/root/CS-Env/lib/python3.8/site-packages/ray-2.0.0.dev0-py3.8-linux-x86_64.egg/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/root/CS-Env/lib/python3.8/site-packages/ray-2.0.0.dev0-py3.8-linux-x86_64.egg/ray/worker.py", line 1746, in get_actor
    return worker.core_worker.get_named_actor_handle(name)
  File "python/ray/_raylet.pyx", line 1565, in ray._raylet.CoreWorker.get_named_actor_handle
  File "python/ray/_raylet.pyx", line 158, in ray._raylet.check_status
ValueError: Failed to look up actor with name 'retrieval_worker_0'. You are either trying to look up a named actor you didn't create, the named actor died, or the actor hasn't been created because named actor creation is asynchronous.
```

## Expected behavior
The expected behaviour consist in being able to finetune RAG without errors 


Comments:
Hi, 

1. first, the  RAG-end2end cannot get a train with pytorch_retriever, since it's not enabled and you have to use ray_retriever (Since it is very slow and hard to update the indexed KB).


2. The second issue is related you your distributed system and we have discussed it in [this issue](https://github.com/huggingface/transformers/issues/12050#issuecomment-856646135).

For a quick fix, change the [line](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag-end2end-retriever/finetune_rag.py#L707) as follows (change the variables in to int) :

        **if ("LOCAL_RANK" not in os.environ or int(os.environ["LOCAL_RANK"]) == 0) and (
            "NODE_RANK" not in os.environ or int(os.environ["NODE_RANK"]) == 0
        ):**



The error persists for me even after the quick fix. I've tried to run the script using a single GPU, a single A6000, to verify that everything else works and it does but the projections for how long a single epoch should take are a bit worrying.
```
Epoch 0:   0%|                      | 157/257383819 [01:12<33135:48:12,  2.16it/s, loss=nan, v_num=7
```
On previous attempts I have let it run for longer but the projected time doesn't lower significantly, is this reasonable or am I getting something else wrong?
Btw for reference I'm using the instances from https://datacrunch.io/ they might have gpus running on different nodes so the ray issues could be caused by the setup. Any suggestion on how I could proceed? Is it a problem with Pytorch Lightning, should I use an older version of ray? And is a single A6000 not enough to conclude the Squad pretraining?
Thank you very much for any help!

Edit: The scripts in the test_run folder are able to do training and validation on a single GPU in a reasonable amount of time

@AntonioLopardo  ,

Oh yeah sorry :) , I thought I fixed this issue during the PR. It is basically a problem with PL's parameter naming.  Comment the following [line](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag-end2end-retriever/lightning_base.py#L405) in the lightning-base.


_"Btw for reference I'm using the instances from https://datacrunch.io/ they might have gpus running on different nodes so the ray issues could be caused by the setup. Any suggestion on how I could proceed? Is it a problem with Pytorch Lightning, should I use an older version of ray? "_

No there's no problem with PL or RAY. It is just how you set up your cluster. Basically, when using RAY, we only create processes in the master process, and then during the other processes, we use these actors.  So what is happening is the "if" condition gets messed up with different naming of nodes and master processes. To solve this,  play around with these environmental variables. 

**LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"**




On your second question, @Dopaminezsy solved it as follows.


![image](https://user-images.githubusercontent.com/16892570/126244104-49ee25f3-27a6-41e8-afff-4cafbeb0f24d.png)


I'm still having issues running the code on more than one GPU, but it runs in about 3 hours for each epoch on one now, thank you very much for your help @shamanez! I'll continue to try to run on more GPUs, but I've already tried many variations of the "LOCAL_RANK" and "NODE_RANK" lines without much success. 
Btw do you think it would be possible to run the retriever with a different encoder, say the "facebook/dpr-ctx_encoder-single-nq-base" or another one finetuned on a specific domain? 

Happy to help you. Well the GPU problem is very hard to solve without having access to a server. But I hope you get the logic! where you need to make sure the "if" condition is true in the main process.  Try to add the node parameter to the trainer from [here](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag-end2end-retriever/lightning_base.py#L401). Anyways you can also check how to execute codes in node architecture. 



Yeah, it is possible. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@AntonioLopardo @shamanez Hi! 
I think the problem is the version of the ray library. I had the same problem with version 2.0. In version 1.3, the error doesn't appear.

I believe part of the reason that the code runs slowly is, because the validation step is run after every training step. See [this](https://github.com/huggingface/transformers/blob/main/examples/research_projects/rag-end2end-retriever/lightning_base.py#L404).  @shamanez is my understanding correct, and is it necessary for the validation step to be run after every training step?

P.S. Thank you for updating the implementation with the newer Ray and PL versions, it's very helpful 😄 

@aidansan  Thanks a pointing this out. Seems like PL lightning document has changed. 

Please read this -  https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#val-check-interval

seems like now we need to change it to a float. 



> I'm still having issues running the code on more than one GPU, but it runs in about 3 hours for each epoch on one now, thank you very much for your help @shamanez! I'll continue to try to run on more GPUs, but I've already tried many variations of the "LOCAL_RANK" and "NODE_RANK" lines without much success. Btw do you think it would be possible to run the retriever with a different encoder, say the "facebook/dpr-ctx_encoder-single-nq-base" or another one finetuned on a specific domain?

yes it is possible.

https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#check-val-every-n-epoch check-val-every-n-epoch might work as well?

yeah I think so