BLEURT score calculation raises UnrecognizedFlagError
Author: nadavo
State: closed

Calling the `compute` method for **bleurt** metric fails with an `UnrecognizedFlagError` for `FLAGS.bleurt_batch_size`. 

My environment:
```
python==3.8.5
datasets==1.2.0
tensorflow==2.3.1
cudatoolkit==11.0.221
```

Test code for reproducing the error:
```
from datasets import load_metric
bleurt = load_metric('bleurt')
gen_text = "I am walking on the promenade today"
ref_text = "I am walking along the promenade on this sunny day"
bleurt.compute(predictions=[test_text], references=[test_text])
```

Error Output:
```
Using default BLEURT-Base checkpoint for sequence maximum length 128. You can use a bigger model for better results with e.g.: datasets.load_metric('bleurt', 'bleurt-large-512').
INFO:tensorflow:Reading checkpoint /home/ubuntu/.cache/huggingface/metrics/bleurt/default/downloads/extracted/9aee35580225730ac5422599f35c4986e4c49cafd08082123342b1019720dac4/bleurt-base-128.
INFO:tensorflow:Config file found, reading.
INFO:tensorflow:Will load checkpoint bert_custom
INFO:tensorflow:Performs basic checks...
INFO:tensorflow:... name:bert_custom
INFO:tensorflow:... vocab_file:vocab.txt
INFO:tensorflow:... bert_config_file:bert_config.json
INFO:tensorflow:... do_lower_case:True
INFO:tensorflow:... max_seq_length:128
INFO:tensorflow:Creating BLEURT scorer.
INFO:tensorflow:Loading model...
INFO:tensorflow:BLEURT initialized.
---------------------------------------------------------------------------
UnrecognizedFlagError                     Traceback (most recent call last)
<ipython-input-12-8b3f4322318a> in <module>
      2 gen_text = "I am walking on the promenade today"
      3 ref_text = "I am walking along the promenade on this sunny day"
----> 4 bleurt.compute(predictions=[gen_text], references=[ref_text])

~/anaconda3/envs/noved/lib/python3.8/site-packages/datasets/metric.py in compute(self, *args, **kwargs)
    396             references = self.data["references"]
    397             with temp_seed(self.seed):
--> 398                 output = self._compute(predictions=predictions, references=references, **kwargs)
    399 
    400             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/bleurt/b1de33e1cbbcb1dbe276c887efa1fad68c6aff913885108078fa1ad408908778/bleurt.py in _compute(self, predictions, references)
    103 
    104     def _compute(self, predictions, references):
--> 105         scores = self.scorer.score(references=references, candidates=predictions)
    106         return {"scores": scores}

~/anaconda3/envs/noved/lib/python3.8/site-packages/bleurt/score.py in score(self, references, candidates, batch_size)
    164     """
    165     if not batch_size:
--> 166       batch_size = FLAGS.bleurt_batch_size
    167 
    168     candidates, references = list(candidates), list(references)

~/anaconda3/envs/noved/lib/python3.8/site-packages/tensorflow/python/platform/flags.py in __getattr__(self, name)
     83     # a flag.
     84     if not wrapped.is_parsed():
---> 85       wrapped(_sys.argv)
     86     return wrapped.__getattr__(name)
     87 

~/anaconda3/envs/noved/lib/python3.8/site-packages/absl/flags/_flagvalues.py in __call__(self, argv, known_only)
    643     for name, value in unknown_flags:
    644       suggestions = _helpers.get_flag_suggestions(name, list(self))
--> 645       raise _exceptions.UnrecognizedFlagError(
    646           name, value, suggestions=suggestions)
    647 

UnrecognizedFlagError: Unknown command line flag 'f'
```

Possible Fix:
Modify `_compute` method https://github.com/huggingface/datasets/blob/7e64851a12263dc74d41c668167918484c8000ab/metrics/bleurt/bleurt.py#L104
to receive a `batch_size` argument, for example:
```
def _compute(self, predictions, references, batch_size=1):
    scores = self.scorer.score(references=references, candidates=predictions, batch_size=batch_size)
    return {"scores": scores}
```

Comments: Upgrading tensorflow to version 2.4.0 solved the issue.

I still have the same error even with TF 2.4.0.

And I have the same error with TF 2.4.1. I believe this issue should be reopened. Any ideas?!

I'm seeing the same issue with TF 2.4.1 when running the following in https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb:
```
!pip install git+https://github.com/google-research/bleurt.git
references = ["foo bar baz", "one two three"]
bleurt_metric = load_metric('bleurt')
predictions =  ["foo bar", "four five six"]
bleurt_metric.compute(predictions=predictions, references=references)
```

@aleSuglia @oscartackstrom - Are you getting the error when running your code in a Jupyter notebook ?

I tried reproducing this error again, and was unable to do so from the python command line console in a virtual environment similar to the one I originally used (and unfortunately no longer have access to) when I first got the error. 
However, I've managed to reproduce the error by running the same code in a Jupyter notebook running a kernel from the same virtual environment.
This made me suspect that the problem is somehow related to the Jupyter notebook.

More environment details:
```
OS: Ubuntu Linux 18.04
conda==4.8.3
python==3.8.5
datasets==1.3.0
tensorflow==2.4.0
BLEURT==0.0.1
notebook==6.2.0
```

This happens when running the notebook on colab. The issue seems to be that colab populates sys.argv with arguments not handled by bleurt.

Running this before calling bleurt fixes it:
```
import sys
sys.argv = sys.argv[:1]
```

Not the most elegant solution. Perhaps it needs to be fixed in the bleurt code itself rather than huggingface?

This is the output of `print(sys.argv)` when running on colab:
```
['/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py', '-f', '/root/.local/share/jupyter/runtime/kernel-a857a78c-44d6-4b9d-b18a-030b858ee327.json']
```

I got the error when running it from the command line. It looks more like an error that should be fixed in the BLEURT codebase.

Seems to be a known issue in the bleurt codebase: https://github.com/google-research/bleurt/issues/24.

Hi, the problem should be solved now.

Hi @tsellam! I can verify that the issue is indeed fixed now. Thanks!