The phenomenon of being suddenly killed during RAG evaluation processing
Author: Juno82
Pull request: no
Labels: name: bug

### System Info

Platform : Ubuntu 18.04.5
Python version : 3.8.8
pytorch-lightning version : 1.3.1
torch version : 1.12.1

### Who can help?

@patrickvonplaten Hello, an error occurs during the RAG Evaluation, and please help me.
 It suddenly becomes killed without any error message.

![image](https://user-images.githubusercontent.com/110461567/185853225-0f0b0cd3-de89-48d2-9351-a8b069e8c418.png)

The script was executed as follows.

python examples/research_projects/rag/eval_rag.py \
    --model_name_or_path facebook/rag-sequence-base \
    --model_type rag_sequence \
    --evaluation_set output/biencoder-nq-dev.questions \
    --gold_data_path output/biencoder-nq-dev.pages \
    --predictions_path output/retrieval_preds.tsv  \
    --eval_mode retrieval \
    --k 1

If look at the code, it occurs when get the model.

![image](https://user-images.githubusercontent.com/110461567/185853809-07f857d8-4ebd-45b5-8474-a32007cb72af.png)


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run evaluation script after fine tuning

python examples/research_projects/rag/eval_rag.py \
    --model_name_or_path facebook/rag-sequence-base \
    --model_type rag_sequence \
    --evaluation_set output/biencoder-nq-dev.questions \
    --gold_data_path output/biencoder-nq-dev.pages \
    --predictions_path output/retrieval_preds.tsv  \
    --eval_mode retrieval \
    --k 1

### Expected behavior

Evaluation should be executed normally and the result should be saved as a tsv file.

Comments:
This is very likely an out of memory error, unfortunately

Dear @LysandreJik ,
Thank you for your comment.
I am trying again from fine tuning.

When I run it with ray, I get the following error:

![image](https://user-images.githubusercontent.com/110461567/186671785-2dfdb66b-f5d8-4eff-a635-c9728cb30753.png)

The script was run as below.

python examples/research_projects/rag/finetune_rag.py \
    --data_dir '/home/junhokim/workspace/transformers/examples/research_projects/rag/finetune_data' \
    --output_dir './finetune_rag_token' \
    --model_name_or_path facebook/rag-token-base \
    --model_type rag_token \
    --do_train \
    --do_predict \
    --train_batch_size 1 \
    --accumulate_grad_batches 1 \
    --num_retrieval_workers 4 \
    --max_epochs 3 \
    --distributed_retriever ray\
    --fp16 \
    --gpus 2

cuda version : 11.6
torch version : 1.12.1+cu116
pytorch-lightning version : 1.5.10
torchmetrics version : 0.9.3
transformers version : 4.21.1

How can I fix the error?

Best Regards,
Junho Kim



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any update on this @Juno82?

Getting similar behaviour. 

Same Error as Killed after downloading the model successfully. 
sagemaker-user@studio$ python main.py \
>     --model hf-causal \
>     --model_args pretrained=EleutherAI/gpt-j-6B \
>     --tasks lambada_*,hellaswag \
>     --device cuda:0
Selected Tasks: ['hellaswag', 'lambada_openai', 'lambada_openai_cloze', 'lambada_openai_mt_de', 'lambada_openai_mt_en', 'lambada_openai_mt_es', 'lambada_openai_mt_fr', 'lambada_openai_mt_it', 'lambada_standard', 'lambada_standard_cloze']
Device not specified
Cuda Available? False
Downloading (…)lve/main/config.json: 100%|█████████████████████████████████████████████████████████| 930/930 [00:00<00:00, 61.1kB/s]
Downloading pytorch_model.bin: 100%|███████████████████████████████████████████████████████████| 24.2G/24.2G [05:56<00:00, 68.0MB/s]
**Killed**

Any suggestions to use the LM-Evaluation-harness framework ?

Same issue. How to resolve?