How to reuse functionality of a (generic) dataset?
State: closed | Author: ArneBinder (NONE)

I have written a generic dataset for corpora created with the Brat annotation tool ([specification](https://brat.nlplab.org/standoff.html), [dataset code](https://github.com/ArneBinder/nlp/blob/brat/datasets/brat/brat.py)). Now I wonder how to use that to create specific dataset instances. What's the recommended way to reuse formats and loading functionality for datasets with a common format?

In my case, it took a bit of time to create the Brat dataset and I think others would appreciate to not have to think about that again. Also, I assume there are other formats (e.g. conll) that are widely used, so having this would really ease dataset onboarding and adoption of the library.

Comments: Hi @ArneBinder, we have a few "generic" datasets which are intended to load data files with a predefined format:
- csv: https://github.com/huggingface/nlp/tree/master/datasets/csv
- json: https://github.com/huggingface/nlp/tree/master/datasets/json
- text: https://github.com/huggingface/nlp/tree/master/datasets/text

You can find more details about this way to load datasets here in the documentation: https://huggingface.co/nlp/loading_datasets.html#from-local-files

Maybe your brat loading script could be shared in a similar fashion?

> Maybe your brat loading script could be shared in a similar fashion?

@thomwolf that was also my first idea and I think I will tackle that in the next days. I separated the code and created a real abstract class `AbstractBrat` to allow to inherit from that (I've just seen that the dataset_loader loads the first non abstract class), now `Brat` is very similar in its functionality to https://github.com/huggingface/nlp/tree/master/datasets/text but inherits from `AbstractBrat`.

However, it is still not clear to me how to add a specific dataset (as explained in https://huggingface.co/nlp/add_dataset.html) to your repo that uses this format/abstract class, i.e. re-using the `features` entry of the  `DatasetInfo` object and `_generate_examples()`. Again, by doing so, the only remaining entries/functions to define would be `_DESCRIPTION`, `_CITATION`, `homepage` and `_URL` (which is all copy-paste stuff) and `_split_generators()`.
 
In a lack of better ideas, I tried sth like below, but of course it does not work outside `nlp` (`AbstractBrat` is currently defined in [datasets/brat.py](https://github.com/ArneBinder/nlp/blob/5e81fb8710546ee7be3353a7f02a3045e9a8351e/datasets/brat/brat.py)):
```python
from __future__ import absolute_import, division, print_function

import os

import nlp

from datasets.brat.brat import AbstractBrat

_CITATION = """
@inproceedings{lauscher2018b,
  title = {An argument-annotated corpus of scientific publications},
  booktitle = {Proceedings of the 5th Workshop on Mining Argumentation},
  publisher = {Association for Computational Linguistics},
  author = {Lauscher, Anne and Glava\v{s}, Goran and Ponzetto, Simone Paolo},
  address = {Brussels, Belgium},
  year = {2018},
  pages = {40–46}
}
"""

_DESCRIPTION = """\
This dataset is an extension of the Dr. Inventor corpus (Fisas et al., 2015, 2016) with an annotation layer containing 
fine-grained argumentative components and relations. It is the first argument-annotated corpus of scientific 
publications (in English), which allows for joint analyses of argumentation and other rhetorical dimensions of 
scientific writing.
"""

_URL = "http://data.dws.informatik.uni-mannheim.de/sci-arg/compiled_corpus.zip"


class Sciarg(AbstractBrat):

    VERSION = nlp.Version("1.0.0")

    def _info(self):

        brat_features = super()._info().features
        return nlp.DatasetInfo(
            # This is the description that will appear on the datasets page.
            description=_DESCRIPTION,
            # nlp.features.FeatureConnectors
            features=brat_features,
            # If there's a common (input, target) tuple from the features,
            # specify them here. They'll be used if as_supervised=True in
            # builder.as_dataset.
            #supervised_keys=None,
            # Homepage of the dataset for documentation
            homepage="https://github.com/anlausch/ArguminSci",
            citation=_CITATION,
        )

    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        # TODO: Downloads the data and defines the splits
        # dl_manager is a nlp.download.DownloadManager that can be used to
        # download and extract URLs
        dl_dir = dl_manager.download_and_extract(_URL)
        data_dir = os.path.join(dl_dir, "compiled_corpus")
        print(f'data_dir: {data_dir}')
        return [
            nlp.SplitGenerator(
                name=nlp.Split.TRAIN,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "directory": data_dir,
                },
            ),
        ]
```  

Nevertheless, many thanks for tackling the dataset accessibility problem with this great library!

As temporary fix I've created [ArneBinder/nlp-formats](https://github.com/ArneBinder/nlp-formats) (contributions welcome).

Hi! You can either copy&paste the builder script and import the builder from there or use `datasets.load_dataset_builder` inside the script and call the methods of the returned builder object.