RuntimeError: Empty or `None` reference sentence found.
Author: SadGare
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.18.0.dev0
- Platform: Windows-10-10.0.22000-SP0
- Python version: 3.7.11 
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help
@patrickvonplaten , @patil-suraj
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- Longformer, BigBird: @ydshieh
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Marian

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

And here is a example of my dataset

> {"translation": {"Cls": "谁司票拟？", "Mdn": "谁起草的这个命令？"}}
{"translation": {"Cls": "百司章奏，置急足驰白乃下。", "Mdn": "百官的奏章，要用快马才能赶上。"}}

I organize the data as I'm told in readme. Although in readme it says that it should be jsonl file, I find that jsonl file won't work and just rename it as json works. It works well for several thousands steps.

## To reproduce

Steps to reproduce the behavior:

1. run this command `python examples/pytorch/translation/run_translation.py --model_name_or_path "Helsinki-NLP/opus-mt-zh-en" --do_train --do_eval --source_lang Cls --target_lang Mdn --source_prefix "translate Classical to Modern: " --train_file examples\pytorch\translation\train.json --validation_file examples\pytorch\translation\dev.json --test_file examples\pytorch\translation\test.json --output_dir D:/Gare-translation/Helsinki-NLP --per_device_train_batch_size=32 --per_device_eval_batch_size=16 --overwrite_output_dir --predict_with_generate --num_train_epochs=200 --save_total_limit=200 --save_steps=10000 --load_best_model_at_end True --evaluation_strategy "steps"`
2. finish first training epoch and run first evaluation
3. pops up error 

> INFO|trainer.py:2412] 2022-03-29 15:36:18,174 >> ***** Running Evaluation *****
[INFO|trainer.py:2414] 2022-03-29 15:36:18,174 >>   Num examples = 8438
[INFO|trainer.py:2417] 2022-03-29 15:36:18,174 >>   Batch size = 4
                                                                                                                       Traceback (most recent call last):██████████████████████████████████████████████████| 2110/2110 [08:51<00:00,  3.68it/s]
  File "examples/pytorch/translation/run_translation.py", line 624, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 541, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\transformers\trainer.py", line 1493, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\transformers\trainer.py", line 1620, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\transformers\trainer_seq2seq.py", line 70, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\transformers\trainer.py", line 2287, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\transformers\trainer.py", line 2528, in evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
  File "examples/pytorch/translation/run_translation.py", line 515, in compute_metrics
    result = metric.compute(predictions=decoded_preds, references=decoded_labels)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\datasets\metric.py", line 430, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "C:\Users\Gare\.cache\huggingface\modules\datasets_modules\metrics\sacrebleu\daba8f731596c6a1a68d61f20220697f68c420a55e2096b4eea8e3ffdc406d96\sacrebleu.py", line 130, in _compute
    **(dict(tokenize=tokenize) if tokenize else {}),
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\sacrebleu\compat.py", line 35, in corpus_bleu
    return metric.corpus_score(hypotheses, references)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\sacrebleu\metrics\base.py", line 421, in corpus_score
    stats = self._extract_corpus_statistics(hypotheses, references)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\sacrebleu\metrics\base.py", line 366, in _extract_corpus_statistics
    ref_cache = self._cache_references(references)
  File "C:\ProgramData\Anaconda3\envs\gare\lib\site-packages\sacrebleu\metrics\base.py", line 333, in _cache_references
    raise RuntimeError("Empty or `None` reference sentence found.")
RuntimeError: Empty or `None` reference sentence found.
  0%|                                                                       | 500/3418600 [10:49<1234:06:48,  1.30s/it]

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Any advice that helps with it would be appreciated.
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hi @Gare-Ng ,

> I organize the data as I'm told in readme. Although in readme it says that it should be jsonl file, I find that jsonl file won't work and just rename it as json works

The file format is expected to be `jsonl` but the scripts expect the file extension to be `.json`.

I think the reason for this error is that, there seems to be an example which has an empty sting as the translation/target. The example scripts are kept simple and easy to adapt so they don't do any extra pre-processing to detect and remove empty examples. You should inspect the dataset to remove such problematic examples and run the scripts. 

Thank you @patil-suraj, and I do write a small script to check whether there is such empty things that I guess it is string there. 

> which has an empty sting as the translation/target.

But it turns out that there isn't such problematic examples in my test dev and test file. I train this model without modifying json file today morning and it works every well. Vaguely I remenber it pops this errors when I decide to change a model and train. At first  I think it is related to the new model, but errors remains when change model back. Till now I still get this error. I'm certain I didn't change json file. My little script is here just in case.

```
import jsonlines
jsonl_name=r'C:\Users\Gare\PycharmProjects\Gare\transformers\examples\pytorch\translation\test.json'
f=jsonlines.open(jsonl_name, "r")
lines=1
errors=[]
for i in f:
    if len((i['translation']['Mdn']))==0 or len((i['translation']['Cls']))==0:
        errors.append(lines)
    lines += 1
if len(errors)==0:
    print('No Error')
else:
    print(errors)
```


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.