Add CCAligned Multilingual Dataset
State: closed | Author: gchhablani (CONTRIBUTOR)

Hello,

I'm trying to add [CCAligned Multilingual Dataset](http://www.statmt.org/cc-aligned/). This has the potential to close #1756.

This dataset has two types - Document-Pairs, and Sentence-Pairs.

The datasets are huge, so I won't be able to test all of them. At the same time, a user might only want to download one particular language and not all. To provide this feature, `load_dataset`'s `**config_kwargs` should allow some random keyword args, in this case -`language_code`. This will be needed before the dataset is downloaded and extracted.

I'm expecting the usage to be something like - 
`load_dataset('ccaligned_multilingual','documents',language_code='en_XX-af_ZA')`. Ofcourse, at a later stage we can provide just two character language codes. This also has an issue where one language has multiple files (`my_MM` and `my_MM_zaw` on the link), but before that the required functionality must be added to `load_dataset`.

It would be great if someone could either tell me an alternative way to do this, or point me to where changes need to be made, if any, apart from the `BuilderConfig` definition. 

Additionally, I believe the tests will also have to be modified if this change is made, since it would not be possible to test for any random keyword arguments. 

A decent way to go about this would be to provide all the options in a list/dictionary for `language_code` and use that to test the arguments. In essence, this is similar to the pre-trained checkpoint dictionary as `transformers`. That means writing dataset specific tests, or adding something new to dataset generation script to make it easier for everyone to add keyword arguments without having to worry about the tests.

Thanks,
Gunjan

Requesting @lhoestq / @yjernite to review.

Comments: Hi !

We already have some datasets that can have many many configurations possible.
To be able to support that, we allow to subclass BuilderConfig to add as many additional parameters as you may need.
This way users can load any language they want. For example the [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py) dataset is a dataset for translation and therefore users should be able to provide any language pair. You can check how the subclass of BuilderConfig is defined [here](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L49).

For testing, only the configurations defined in the `BUILDER_CONFIGS` class attribute are used.
All the other configs combinations are not tested, but they can be used by users. If a config doesn't already exist in `BUILDER_CONFIGS`, then it is created on the fly.
For example in [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L61), only 6 configs are defined in `BUILDER_CONFIGS`.

So what I would do in your case is have something like
```python

class CCAlignedConfig(datasets.BuilderConfig):
    def __init__(self, *args, documents_or_sentences=None, language_code=None, **kwargs):
        super().__init__(
            *args,
            name=f"{documents_or_sentences}-{language_code}",
            **kwargs,
        )
        self.documents_or_sentences = documents_or_sentences
        self.language_code = language_code
```
And of course, feel free to change/rename things if you want to. In particular I think we can improve the name of the parameter `documents_or_sentences`

Hi @lhoestq,

Thanks a lot! I don't know why I didn't think about that. :P 
I'll make these changes and update.

Hi @lhoestq,

I have tested and added dummy files. Request you to review.

Also, does this mean BUILDER_CONFIGS is only needed while testing?

Hi @lhoestq,

Any changes required on this one?

Thanks,
Gunjan

Hi @lhoestq,

Sorry for the delay, I have added the changes from the review. For the ISO format language codes, I just selected the first two characters from the names, hoping those are correct. Let me know if you want me to verify :P

Thanks for taking the time to add such a detailed review. I'll keep all these changes in mind the next time I'm adding a dataset.

Thanks,
Gunjan

Hi @lhoestq,

I have changed the README, and added a single example per config. Even one example is long enough to make the files heavy. Hope that isn't an issue.

Thanks,
Gunjan

Hi @lhoestq,

Thanks for approving.