Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI
Author: lmvasque
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input. 

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."], 
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```
  
**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch). 

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).


Comments: Hi, thanks for reporting! We can add a `sources: datasets.Value("string")` feature to the `Features` dict in the `SARI` script to fix this. Would you be interested in submitting a PR?

Hi Mario,

Thanks for your message. I did try to add `sources` into the `Features` dict using a script for the metric:
```
            features=datasets.Features(
                {
                    "sources": datasets.Value("string", id="sequence"),
                    "predictions": datasets.Value("string", id="sequence"),
                    "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
                }
            ),
```

But that only avoids a failure in `encode_batch` in the `add_batch` method:
```
        batch = {"predictions": predictions, "references": references}
        batch = self.info.features.encode_batch(batch)
```

The real problem is that `add_batch()`, `add()` and `compute()` does not receive a `sources` param:
```
def add_batch(self, *, predictions=None, references=None):
def add(self, *, prediction=None, reference=None):
def compute(self, *, predictions=None, references=None, **kwargs)
```

And then, it fails:
`TypeError: add_batch() got an unexpected keyword argument sources`

I need this for adding any metric based on SARI or alike, not only for sari.py :)

Let me know if I understood correctly the proposed solution.


The `Metric` class has been modified recently to support this use-case, but the `add_batch` + `compute` pattern still doesn't work correctly. I'll open a PR.