SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Multi-task dataset mixing

Comment:
I agree that we should leave the flattening of the dataset to the user for now. Especially because although the T5 framing seems obvious, there are slight variations on how the T5 authors do it in comparison to other approaches such as gpt-3 and decaNLP.

In terms of sampling, Examples-proportional mixing does seem the simplest to implement so would probably be a good starting point.

Temperature-scaled mixing would probably most useful, offering flexibility as it can simulate the other 2 methods by setting the temperature parameter. There is a [relevant part of the T5 repo](https://github.com/google-research/text-to-text-transfer-transformer/blob/03c94165a7d52e4f7230e5944a0541d8c5710788/t5/data/utils.py#L889-L1118) which should help with implementation.

According to the T5 authors, equal-mixing performs worst. Among the other two methods, tuning the K value (the artificial dataset size limit) has a large impact.
Issue: Multi-task dataset mixing

Comment:
I agree with going with temperature-scaled mixing for its flexibility!

For the function that combines the datasets, I also find `dataset.add()` okay while also considering that users may want it to be easy to combine a list of say 10 data sources in one go.

`dataset.sample()` should also be good. By the looks of it, we're planning to have as main parameters: `temperature`, and `K`.

On converting the datasets to the same format, I agree that we can leave these to the users for now. But, I do imagine it'd be an awesome feature for the future to have this automatically handled, based on a chosen *approach* to formatting :smile: 

E.g. T5, GPT-3, decaNLP, original raw formatting, or a contributed way of formatting in text-to-text.
Issue: Multi-task dataset mixing

Comment:
This is an interesting discussion indeed and it would be nice to make multi-task easier.

Probably the best would be to have a new type of dataset especially designed for that in order to easily combine and sample from the multiple datasets.

This way we could probably handle the combination of datasets with differing schemas as well (unlike T5).
Issue: Multi-task dataset mixing

Comment:
@thomwolf Are you suggesting making a wrapper class which can take existing datasets as arguments and do all the required sampling/combining, to present the same interface as a normal dataset?

That doesn't seem too complicated to implement.
Issue: Multi-task dataset mixing

Comment:
I guess we're looking at the end user writing something like:
``` python
ds = nlp.load_dataset('multitask-t5',datasets=["squad","cnn_dm",...], k=1000, t=2.0)
```
Using the t5 method of combining here (or this could be a function passed in as an arg) 

Passing kwargs to each 'sub-dataset' might become tricky.
Issue: Multi-task dataset mixing

Comment:
From thinking upon @thomwolf 's suggestion, I've started experimenting:
```python
class MultitaskDataset(DatasetBuilder):
    def __init__(self, *args, **kwargs):
        super(MultitaskDataset, self).__init__(*args, **kwargs)
        self._datasets = kwargs.get("datasets")

    def _info(self):
        return nlp.DatasetInfo(
            description=_DESCRIPTION,
            features=nlp.Features({
                    "source": nlp.Value("string"),
                    "target": nlp.Sequence(nlp.Value("string"))
                })
        )

    def _get_common_splits(self):
        '''Finds the common splits present in all self._datasets'''
        min_set = None
        for dataset in self._datasets:
            if min_set != None:
                min_set.intersection(set(dataset.keys()))
            else:
                min_set = set(dataset.keys())
        return min_set

....

# Maybe this?:
squad = nlp.load_dataset("squad")
cnn_dm = nlp.load_dataset("cnn_dailymail","3.0.0")
multitask_dataset = nlp.load_dataset(
    'multitask_dataset',
    datasets=[squad,cnn_dailymail], 
    k=1000, 
    t=2.0
)

```

Does anyone know what methods of `MultitaskDataset` I would need to implement? Maybe `as_dataset` and `download_and_prepare`? Most of these should be just calling the methods of the sub-datasets. 

I'm assuming DatasetBuilder is better than the more specific `GeneratorBasedBuilder`, `BeamBasedBuilder`, etc....

One of the other problems is that the dataset size is unknown till you construct it (as you can pick the sub-datasets). Am hoping not to need to make changes to `nlp.load_dataset` just for this class.

I'd appreciate it if anyone more familiar with nlp's internal workings could tell me if I'm on the right track!
Issue: Multi-task dataset mixing

Comment:
I think I would probably go for a `MultiDataset` wrapper around a list of `Dataset`.

I'm not sure we need to give it `k` and `t` parameters at creation, it can maybe be something along the lines of:
```python
squad = nlp.load_dataset("squad")
cnn_dm = nlp.load_dataset("cnn_dailymail","3.0.0")

multitask_dataset = nlp.MultiDataset(squad, cnn_dm)

batch = multitask_dataset.sample(10, temperature=2.0, k=1000)
```

The first proof-of-concept for multi-task datasets could definitely require that the provided datasets have the same name/type for columns (if needed you easily rename/cast a column prior to instantiating the `MultiDataset`).

It's good to think about it for some time though and don't overfit too much on the T5 examples (in particular for the ways/kwargs for sampling among datasets).
Issue: Multi-task dataset mixing

Comment:
The problem with changing `k` and `t` per sampling is that you'd have to somehow remember which examples you'd already returned while re-weighting the remaining examples based on the new `k` and `t`values. It seems possible but complicated (I can't really see a reason why you'd want to change the weighting of datasets after you constructed the multidataset).

Wouldn't it be convenient if it implemented the dataset interface? Then if someone has code using a single nlp dataset, they can replace it with a multitask combination of more datasets without having to change other code. We would at least need to be able to pass it into a `DataLoader`.
Issue: Multi-task dataset mixing

Comment:
A very janky (but working) implementation of  `multitask_dataset.sample()` could be something like this:
```python
import nlp
import torch

class MultiDataset():
    def __init__(self, *args, temperature=2.0, k=1000, maximum=None, scale=1):
        self.datasets = args
        self._dataloaders = {}
        for split in self._get_common_splits():
            split_datasets = [ds[split] for ds in self.datasets]
            mixing_rates = self._calc_mixing_rates(split_datasets,temperature, k, maximum, scale)
            weights = []
            for i in range(len(self.datasets)):
                weights += [mixing_rates[i]]*len(self.datasets[i][split])
            self._dataloaders[split] = torch.utils.data.DataLoader(torch.utils.data.ConcatDataset(split_datasets),
                                                        sampler=torch.utils.data.sampler.WeightedRandomSampler(
                                                            num_samples=len(weights),
                                                            weights = weights,
                                                            replacement=True),
                                                        shuffle=False)

    def _get_common_splits(self):
        '''Finds the common splits present in all self.datasets'''
        min_set = None
        for dataset in self.datasets:
            if min_set != None:
                min_set.intersection(set(dataset.keys()))
            else:
                min_set = set(dataset.keys())
        return min_set


    def _calc_mixing_rates(self,datasets, temperature=2.0, k=1000, maximum=None, scale=1):
       '''Work out the weighting of each dataset based on t and k'''
        mixing_rates = []
        for dataset in datasets:
            rate = len(dataset)
            rate *= scale
            if maximum:
                rate = min(rate, maximum)
            if temperature != 1.0:
                rate = rate ** (1.0/temperature)
            mixing_rates.append(rate)
        return mixing_rates

    def sample(self,n,split):
        batch = []
        for example in self._dataloaders[split]:
            batch.append(example)
            n -= 1
            if n == 0:
                return batch


def flatten(dataset,flatten_fn):
    for k in dataset.keys():
        if isinstance(dataset[k],nlp.Dataset):
            dataset[k] = dataset[k].map(flatten_fn,remove_columns=dataset[k].column_names)

# Squad
def flatten_squad(example):
    return {"source": "squad context: " + example['context'] + " question: " + example['question'],"target":example["answers"]["text"]}
squad = nlp.load_dataset("squad")
flatten(squad,flatten_squad)

# CNN_DM
def flatten_cnn_dm(example):
    return {"source": "cnn_dm: " + example['article'],"target":[example["highlights"]]}
cnn_dm = nlp.load_dataset("cnn_dailymail", "3.0.0")
flatten(cnn_dm,flatten_cnn_dm)

multitask_dataset = MultiDataset(squad, cnn_dm)
batch = multitask_dataset.sample(100,"train")
```

There's definitely a more sensible way than embedding `DataLoader`s inside.
Issue: Multi-task dataset mixing

Comment:
Good spot! Here are my thoughts:

- Aside: Adding `MultitaskModel` to transformers might be a thing to raise - even though having task-specific heads has become unfashionable in recent times in favour of text-to-text type models.
- Adding the task name as an extra field also seems useful for these kind of models which have task-specific heads
- There is some validation of our approach that the user should be expected to `map` datasets into a common form.
- The size-proportional sampling (also called "Examples-proportional mixing") used here doesn't perform too badly in the T5 paper (it's comparable to temperature-scaled mixing in many cases but less flexible. This is only reasonable with a `K` maximum size parameter to prevent very large datasets dominating). This might be good for a first prototype using:
    ```python
    def __iter__(self):
        """
        For each batch, sample a task, and yield a batch from the respective
        task Dataloader.

        We use size-proportional sampling, but you could easily modify this
        to sample from some-other distribution.
        """
        task_choice_list = []
        for i, task_name in enumerate(self.task_name_list):
            task_choice_list += [i] * self.num_batches_dict[task_name]
        task_choice_list = np.array(task_choice_list)
        np.random.shuffle(task_choice_list)

        dataloader_iter_dict = {
            task_name: iter(dataloader) 
            for task_name, dataloader in self.dataloader_dict.items()
        }
        for task_choice in task_choice_list:
            task_name = self.task_name_list[task_choice]
            yield next(dataloader_iter_dict[task_name])    
    ```
    We'd just need to pull samples from the raw datasets and not from `DataLoader`s for each task. We can assume the user has done `dataset.shuffle()` if they want to.

    Other sampling methods can later be implemented by changing how the `task_choice_list` is generated. This should allow more flexibility and not tie us to specific methods for sampling among datasets.
Issue: Multi-task dataset mixing

Comment:
Another thought: Multitasking over benchmarks (represented as Meta-datasets in nlp) is probably a common use case. Would be nice to pass an entire benchmark to our `MultiDataset` wrapper rather than having to pass individual components.
Issue: Multi-task dataset mixing

Comment:
Hey! Happy to jump into the discussion here. I'm still getting familiar with bits of this code, but the reasons I sampled over data loaders rather than datasets is 1) ensuring that each sampled batch corresponds to only 1 task (in case of different inputs formats/downstream models) and 2) potentially having different batch sizes per task (e.g. some tasks have very long/short inputs). How are you currently dealing with these in your PR?
Issue: Multi-task dataset mixing

Comment:
The short answer is - I'm not! Everything is currently on a per-example basis. It would be fairly simple to add a `batch_size` argument which would ensure that every `batch_size` examples come from the same task. That should suit most use-cases (unless you wanted to ensure batches all came from the same task and apply something like `SortishSampler` on each task first)

Your notebook was really inspiring by the way - thanks!
Issue: Multi-task dataset mixing

Comment:
@zphang is having different batch sizes per task actually helpful? Would be interesting to know as it's not something I've come across as a technique used by any MTL papers.
Issue: Multi-task dataset mixing

Comment:
> @zphang is having different batch sizes per task actually helpful? Would be interesting to know as it's not something I've come across as a technique used by any MTL papers.

I think having different batch sizes per task is particularly helpful in some scenarios where each task has different amount of data. For example, the problem I'm currently facing is one task has tens of thousands of samples while one task has a couple hundreds. I think in this case different batch size could help. But if using the same batch size is a lot simpler to implement, I guess it makes sense to go with that.
Issue: Multi-task dataset mixing

Comment:
I think that instead of proportional to size sampling you should specify weights or probabilities for drawing a batch from each dataset. We should also ensure that the smaller datasets are repeated so that the encoder layer doesn't overtrain on the largest dataset.
Issue: Multi-task dataset mixing

Comment:
Are there any references for people doing different batch sizes per task in the literature? I've only seen constant batch sizes with differing numbers of batches for each task which seems sufficient to prevent the impact of large datasets (Read 3.5.3 of the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf) for example).
Issue: Multi-task dataset mixing

Comment:
Hi,
regarding building T5 dataset , I think we can use datasets https://github.com/huggingface/datasets and then need something similar to tf.data.experimental.sample_from_datasets, do you know if similar functionality exist in pytorch? Which can sample multiple datasets with the given rates. thanks.
Issue: Multi-task dataset mixing

Comment:
You can use `interleave_datasets` to mix several datasets together. By default it alternates between all the datasets, but you can also provide sampling probabilities if you want to oversample from one of the datasets

```python
from datasets import load_dataset, interleave_datasets

squad = load_dataset("squad", split="train")
cnn_dm = load_dataset("cnn_dailymail", "3.0.0", split="train")
ds = interleave_datasets([squad, cnn_dm])

print(ds[0])
# {'id': '5733be284776f41900661182',
#  'title': 'University_of_Notre_Dame',
#  'context': 'Architecturally, the school has a Catholic character...',
#  'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
#  'answers': {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]},
#  'article': None,
#  'highlights': None}
print(ds[1])
# {'id': '42c027e4ff9730fbb3de84c1af0d2c506e41c3e4',
#  'title': None,
#  'context': None,
#  'question': None,
#  'answers': None,
#  'article': 'LONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe...',
#  'highlights': "Harry Potter star Daniel Radcliffe..."}
```

see docs at https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.interleave_datasets
Issue: Multi-task dataset mixing

Comment:
I also have this implementation of multi-task sampler here which I used it to tune T5: https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py
Issue: ❓ How to get ROUGE-2 with the ROUGE metric ?

Comment:
For the rouge2 metric you can do

```python
rouge = nlp.load_metric('rouge')
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        rouge.add(lp, lg)
score = rouge.compute(rouge_types=["rouge2"])
```

Note that I just did a PR to have both `.add` and `.add_batch` for metrics, that's why now this is `rouge.add(lp, lg)` and not `rouge.add([lp], [lg])`
Issue: ❓ How to get ROUGE-2 with the ROUGE metric ?

Comment:
Well I just tested with the official script and both rouge1 and rougeL return exactly the same thing for the input you gave, so this is actually fine ^^

I hope it helped :)
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
I just ran it on colab and got this
```
[{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812,
dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train',
num_bytes=611607465, num_examples=533285, dataset_name='blog_authorship_corpus')},
{'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277,
dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation',
num_bytes=35652716, num_examples=30804, dataset_name='blog_authorship_corpus')}]
```
which is different from the `dataset_infos.json` and also different from yours.

It looks like the script for generating examples is not consistent
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
The files provided by the authors are corrupted and the script seems to ignore the xml files that can't be decoded (it does `try:... except UnicodeDecodeError`). Maybe depending of the environment some files can be opened and some others don't but not sure why
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
Feel free to do `ignore_verifications=True` for now... The verifications only include a check on the checksums of the downloaded files, and a check on the number of examples in each splits.
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
I'm getting this same issue when loading the `imdb` corpus via `dataset = load_dataset("imdb")`. When I try `ignore_verifications=True`, no examples are read into the `train` portion of the dataset.
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
> I'm getting this same issue when loading the `imdb` corpus via `dataset = load_dataset("imdb")`. When I try `ignore_verifications=True`, no examples are read into the `train` portion of the dataset.

When the checksums don't match, it may mean that the file you downloaded is corrupted. In this case you can try to load the dataset again `load_dataset("imdb", download_mode="force_redownload")`

Also I just checked on my side and it worked fine:

```python
from datasets import load_dataset

dataset = load_dataset("imdb")
print(len(dataset["train"]))
# 25000
```

Let me know if redownloading fixes your issue @EmilyAlsentzer .
If not, feel free to open a separate issue.
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
I wasn't aware of the "force_redownload" option and manually removed the '/home/me/.cache/huggingface/datasets/' dir, this worked for me (dataset 'cnn_dailymail')
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
Yes I think this might not be documented well enough. Let’s add it to the doc @lhoestq @SBrandeis.
And everything on how to control the cache behavior better (removing, overriding, changing the path, etc)
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
Already fixed:
```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("blog_authorship_corpus")

In [3]: ds
Out[3]: 
DatasetDict({
    train: Dataset({
        features: ['text', 'date', 'gender', 'age', 'horoscope', 'job'],
        num_rows: 689793
    })
    validation: Dataset({
        features: ['text', 'date', 'gender', 'age', 'horoscope', 'job'],
        num_rows: 37919
    })
})
Issue: NonMatchingSplitsSizesError when loading blog_authorship_corpus

Comment:
In my case, I had to remove the cache datasets directory completely as @putssander suggested, the download_mode='forced_redownload' was insufficient.

I had a private repository with data files that I loaded with a loading script. It was working fine until I pushed a new version of the data files and then the NonMatchingSplitsSizesError was raised.
Issue: [Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type

Comment:
Here the full error trace:

```
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-1-7aaf3f011358> in <module>
      1 import nlp
      2 ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
----> 3 ds.map(lambda x: x, load_from_cache_file=False)

~/python_bin/nlp/arrow_dataset.py in map(self, function, with_indices, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, arrow_schema, disable_nullable)
    549
    550         if update_data:
--> 551             writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file
    552
    553             # Create new Dataset from buffer or file

~/python_bin/nlp/arrow_writer.py in finalize(self, close_stream)
    182     def finalize(self, close_stream=True):
    183         if self.pa_writer is not None:
--> 184             self.write_on_file()
    185             self.pa_writer.close()
    186         if close_stream:

~/python_bin/nlp/arrow_writer.py in write_on_file(self)
    104         """
    105         if self.current_rows:
--> 106             pa_array = pa.array(self.current_rows, type=self._type)
    107             first_example = pa.array(self.current_rows[0:1], type=self._type)[0]
    108             # Sanity check

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Could not convert TagMe with type str: converting to null type
```
Issue: [Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type

Comment:
Actually thinking a bit more about it, it's probably a data sample that is not correct in `trivia_qa`. But I'm a bit surprised though that we managed to write it in .arrow format and now cannot write it anymore after an "identity" mapping.
Issue: [Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type

Comment:
Actually, I know what the problem is...I'm quite sure it's a bug. Here we take some test inputs: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L472

It might be that in the test inputs, a `Sequence` type value is an emtpy list. So in my case I have `ds[0]["entity_pages']["wiki_context"] = []`. => this leads to an `arrow_schema` equal to `null`  for `["entity_pages']["wiki_context"]` => see line: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L501  instead of list of string which it should for other examples. 

Guess it's an edge case, but it can happen.
Issue: [Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type

Comment:
Good point, I think the schema should be infered at the writing stage where we have a `writer_batch_size` number of examples (typically 10k) so it's even less likely to run into this scenario.
Issue: Remove test set from NLP viewer

Comment:
Appears that [two thirds of those polled on Twitter](https://twitter.com/srush_nlp/status/1265734497632477185) are in favor of _some_ mechanism for averting eyeballs from the test data.