[rag] missing a working End-to-end evaluation example
Author: stas00
Pull request: no
Labels: name: Feature request

I'm going to try to write tests for `examples/rag` (https://github.com/huggingface/transformers/issues/7715), but first I'm trying to figure out how it works.

Would it be possible to add a full `End-to-end evaluation` invocation example in https://github.com/huggingface/transformers/blob/master/examples/rag/README.md#end-to-end-evaluation? i.e. with the correct data.

I tested https://github.com/huggingface/transformers/blob/master/examples/rag/README.md#retrieval-evaluation and it worked, but if I try to adapt the same params for e2e it crashes with:
```
$ python eval_rag.py --model_name_or_path facebook/rag-sequence-nq --model_type rag_sequence \
--evaluation_set output/biencoder-nq-dev.questions --gold_data_path output/biencoder-nq-dev.pages \
--predictions_path output/retrieval_preds.tsv --eval_mode e2e --gold_data_mode qa --n_docs 5 \
--print_predictions
2020-11-03 22:07:33.124277: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
INFO:__main__:Evaluate the following checkpoints: ['facebook/rag-sequence-nq']
INFO:__main__:Calculating metrics based on an existing predictions file: output/retrieval_preds.tsv
Traceback (most recent call last):
  File "eval_rag.py", line 314, in <module>
    main(args)
  File "eval_rag.py", line 280, in main
    score_fn(args, args.predictions_path, args.gold_data_path)
  File "eval_rag.py", line 46, in get_scores
    data = pd.read_csv(gold_data_path, sep="\t", header=None)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 458, in _read
    data = parser.read(nrows)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 1196, in read
    ret = self._engine.read(nrows)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 2155, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 847, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 862, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 918, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 905, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2042, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 2, saw 6
```

I think it needs a different input data.

And we need 2 functional examples: for `qa` and `ans` each.

I can handle adding this to the doc if you tell me what to add.

Thanks.

@patrickvonplaten, @lhoestq



Comments:
@stas00 

Can you please write a test code for **finetune.sh**. 

As you can see I'm waiting for this ticket to be addressed before I'm able to write the tests. 

Perhaps you can address that, and then I will have all the info needed to write the tests.

Until then please file a normal issue about it. I haven't done any rag work yet, so that's why I'm asking for support.

@lhoestq is working on this at the moment :-) 

Actually I'm working on the finetuning script example, not eval ;)
But maybe this can help with adding a test for the eval script example.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


stale