Problems with WMT dataset
State: closed
Labels: description: Something isn't working
name: bug

description: A bug in a dataset script provided in the library
name: dataset bug

## Describe the bug
I am trying to load WMT15 dataset and to define which data-sources to use for train/validation/test splits, but unfortunately it seems that the official documentation at [https://huggingface.co/datasets/wmt15#:~:text=Versions%20exists%20for,wmt_translate%22%2C%20config%3Dconfig)](https://huggingface.co/datasets/wmt15#:~:text=Versions%20exists%20for,wmt_translate%22%2C%20config%3Dconfig)) doesn't work anymore.

## Steps to reproduce the bug
```shell
>>> import datasets
>>> a = datasets.translate.wmt.WmtConfig()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'datasets' has no attribute 'translate'
>>> a = datasets.wmt.WmtConfig()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'datasets' has no attribute 'wmt'
```

## Expected results
To load WMT15 with given data-sources.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.10.0-10-amd64-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Comments:
Hi! Yes, the docs are outdated. Expect this to be fixed soon. 

In the meantime, you can try to fix the issue yourself.

These are the configs/language pairs supported by `wmt15` from which you can choose:
* `cs-en` (Czech - English)
* `de-en` (German - English)
* `fi-en` (Finnish- English)
* `fr-en` (French - English)
* `ru-en` (Russian - English)

And the current implementation always uses all the subsets available for a language, so to define custom subsets, you'll have to clone the repo from the Hub and replace the line https://huggingface.co/datasets/wmt15/blob/main/wmt_utils.py#L688 with:
`for split, ss_names in (self._subsets if self.config.subsets is None else self.config.subsets).items()`

Then, you can load the dataset as follows:
```python
from datasets import load_dataset
dset = load_dataset("path/to/local/wmt15_folder", "<one of 5 available configs>", subsets=...)

@mariosasko thanks a lot for the suggested fix! 

Hi @mariosasko 

Are the docs updated? If not, I would like to get on it. I am new around here, would we helpful, if you can guide.

Thanks

Hi @khushmeeet! The docs haven't been updated, so feel free to work on this issue. This is a tricky issue, so I'll give the steps you can follow to fix this:

First, this code:
https://github.com/huggingface/datasets/blob/7cff5b9726a223509dbd6224de3f5f452c8d924f/src/datasets/load.py#L113-L118

needs to be replaced with (makes the dataset builder search more robust and allows us to remove the ABC stuff from `wmt_utils.py`):
```python
    for name, obj in module.__dict__.items():
        if inspect.isclass(obj) and issubclass(obj, main_cls_type):
            if inspect.isabstract(obj):
                continue
            module_main_cls = obj
            obj_module = inspect.getmodule(obj)
            if obj_module is not None and module == obj_module:
               break
```

Then, all the `wmt_utils.py` scripts need to be updated as follows (these are the diffs with the requiered changes):
````diff
 import os
 import re
 import xml.etree.cElementTree as ElementTree
-from abc import ABC, abstractmethod

 import datasets
````

````diff
logger = datasets.logging.get_logger(__name__)


 _DESCRIPTION = """\
-Translate dataset based on the data from statmt.org.
+Translation dataset based on the data from statmt.org.

-Versions exists for the different years using a combination of multiple data
-sources. The base `wmt_translate` allows you to create your own config to choose
-your own data/language pair by creating a custom `datasets.translate.wmt.WmtConfig`.
+Versions exist for different years using a combination of data
+sources. The base `wmt` allows you to create a custom dataset by choosing
+your own data/language pair. This can be done as follows:

 ```
-config = datasets.wmt.WmtConfig(
-    version="0.0.1",
+from datasets import inspect_dataset, load_dataset_builder
+
+inspect_dataset("<insert the dataset name", "path/to/scripts")
+builder = load_dataset_builder(
+    "path/to/scripts/wmt_utils.py",
     language_pair=("fr", "de"),
     subsets={
         datasets.Split.TRAIN: ["commoncrawl_frde"],
         datasets.Split.VALIDATION: ["euelections_dev2019"],
     },
 )
-builder = datasets.builder("wmt_translate", config=config)
-```

+# Standard version
+builder.download_and_prepare()
+ds = builder.as_dataset()
+
+# Streamable version
+ds = builder.as_streaming_dataset()
+```
 """
````

````diff
+class Wmt(datasets.GeneratorBasedBuilder):
     """WMT translation dataset."""
+
+    BUILDER_CONFIG_CLASS = WmtConfig

     def __init__(self, *args, **kwargs):
-        if type(self) == Wmt and "config" not in kwargs:  # pylint: disable=unidiomatic-typecheck
-            raise ValueError(
-                "The raw `wmt_translate` can only be instantiated with the config "
-                "kwargs. You may want to use one of the `wmtYY_translate` "
-                "implementation instead to get the WMT dataset for a specific year."
-            )
         super(Wmt, self).__init__(*args, **kwargs)

     @property
-    @abstractmethod
     def _subsets(self):
         """Subsets that make up each split of the dataset."""
````
```diff
         """Subsets that make up each split of the dataset for the language pair."""
         source, target = self.config.language_pair
         filtered_subsets = {}
-        for split, ss_names in self._subsets.items():
+        subsets = self._subsets if self.config.subsets is None else self.config.subsets
+        for split, ss_names in subsets.items():
             filtered_subsets[split] = []
             for ss_name in ss_names:
                 dataset = DATASET_MAP[ss_name]
```

`wmt14`, `wmt15`, `wmt16`, `wmt17`, `wmt18`, `wmt19` and `wmt_t2t` have this script, so all of them need to be updated. Also, the dataset summaries from the READMEs of these datasets need to be updated to match the new `_DESCRIPTION` string. And that's it! Let me know if you need additional help.

Hi @mariosasko ,

I have made the changes as suggested by you and have opened a PR #4537.

Thanks

Resolved via #4554