Question:
I am trying to load a custom dataset locally.  This is a test dataset, will be revised soon, and will probably never be public so we would not want to put it on the HF Hub,
The dataset is in the same format as Conll2003.  The idea is to train Bert on conll2003+the custom dataset.
The setup I am testing (I am open to changes) is to use a folder under the project folder called “ADPConll” with all the data files (just like the Conll2003 folder in git datasets) in it like so:
MainProjectFolder
ADPConll
ADPConll.py                    ← copy of conll2003.oy with minor changes (see below)
ADPConll.py.lock             ← HF internal file
ADPConll.json                 ← train dataset in the format you see in the HF data browser.
ADPConll_test.json         ← test dataset
ADPConll_valid.json       ← validation dataset
Now the question comes as to how to set up the details.  I think the key is in the ADPConll.py script where I have the following:
_URL = “./ADPConll/”
_TRAINING_FILE = “ADPConll.json”
_DEV_FILE = “ADPConll_valid.txt”
_TEST_FILE = “ADPConll_test.txt”
I have tried several different settings but I am trying to cut short the trail/error approach and see if we can get this documented.
Note that the format of the files matches what you see in the HF Streamlit data explorer:
Streamlit 3
Ideally we would train on a combination of the 2 datasets but we would add new labels to our custom dataset so it can recognize additional entities.

Context:
Hi ! The conll2003 script doesn’t parse json files, but rather text files in the conll format. This format has one line per token, with space-separated labels for the POS tag, the chunk tag and the NER tag.
So if you want to use a dataset script like conll you’d have to use the right format, or reimplement the _generate_examples method to parse the json data (this should be straightforward since you just have to iterate over the json data and yield each example).
Otherwise if you don’t want to implement a dataset script, it’s probably simpler to directly use the json loader:
from datasets import load_dataset

data_files = {
    "train": “./ADPConll/ADPConll.json”,
    "validation": “./ADPConll/ADPConll_valid.json”,
    "test": “./ADPConll/ADPConll_test.json”
}

dataset = load_dataset("json", data_files=data_files)

Also if you want you can define your own features (with your own labels):
import datasets

features = datasets.Features(
    {
        "id": datasets.Value("string"),
        "tokens": datasets.Sequence(datasets.Value("string")),
        "pos_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=[
                    '"',
                    "''",
                    "#",
                    "$",
                    "(",
                    ")",
                    ",",
                    ".",
                    ":",
                    "``",
                    "CC",
                    "CD",
                    "DT",
                    "EX",
                    "FW",
                    "IN",
                    "JJ",
                    "JJR",
                    "JJS",
                    "LS",
                    "MD",
                    "NN",
                    "NNP",
                    "NNPS",
                    "NNS",
                    "NN|SYM",
                    "PDT",
                    "POS",
                    "PRP",
                    "PRP$",
                    "RB",
                    "RBR",
                    "RBS",
                    "RP",
                    "SYM",
                    "TO",
                    "UH",
                    "VB",
                    "VBD",
                    "VBG",
                    "VBN",
                    "VBP",
                    "VBZ",
                    "WDT",
                    "WP",
                    "WP$",
                    "WRB",
                ]
            )
        ),
        "chunk_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=[
                    "O",
                    "B-ADJP",
                    "I-ADJP",
                    "B-ADVP",
                    "I-ADVP",
                    "B-CONJP",
                    "I-CONJP",
                    "B-INTJ",
                    "I-INTJ",
                    "B-LST",
                    "I-LST",
                    "B-NP",
                    "I-NP",
                    "B-PP",
                    "I-PP",
                    "B-PRT",
                    "I-PRT",
                    "B-SBAR",
                    "I-SBAR",
                    "B-UCP",
                    "I-UCP",
                    "B-VP",
                    "I-VP",
                ]
            )
        ),
        "ner_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=[
                    "O",
                    "B-PER",
                    "I-PER",
                    "B-ORG",
                    "I-ORG",
                    "B-LOC",
                    "I-LOC",
                    "B-MISC",
                    "I-MISC",
                ]
            )
        ),
    }
)

and then load the dataset with your own features:
dataset = load_dataset("json", data_files=data_files, features=features)

Let me know if that helps !


Building a dataset file for machine translation and add it to Huggingface Datasets

Question:
Hi all,
I am trying to add a dataset for machine translation for Dravidian languages (South India). However, I am facing an error that I cannot resolve at the moment. If anyone could help, it would be great. The dataset is stored in a csv format if anyone would like to have a look at it. [en_ml_test.csv - Google Drive 7]
(en_ml_test.csv - Google Drive 7)

Screenshot 2021-05-22 1307431920×1080 151 KB

The code for the dataset is as follows:
      # coding=utf-8
# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Machine Translation in Dravidian languages"""

import datasets
import csv
import pandas as pd
_HOMEPAGE = "https://competitions.codalab.org/competitions/27650"

_CITATION = """ \
@inproceedings{chakravarthi-etal-2021-findings-shared,
    title = "Findings of the Shared Task on Machine Translation in {D}ravidian languages",
    author = "Chakravarthi, Bharathi Raja  and
      Priyadharshini, Ruba  and
      Banerjee, Shubhanker  and
      Saldanha, Richard  and
      McCrae, John P.  and
      M, Anand Kumar  and
      Krishnamurthy, Parameswari  and
      Johnson, Melvin",
    booktitle = "Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages",
    month = apr,
    year = "2021",
    address = "Kyiv",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.dravidianlangtech-1.15",
    pages = "119--125",
    abstract = "This paper presents an overview of the shared task on machine translation of Dravidian languages. We presented the shared task results at the EACL 2021 workshop on Speech and Language Technologies for Dravidian Languages. This paper describes the datasets used, the methodology used for the evaluation of participants, and the experiments{'} overall results. As a part of this shared task, we organized four sub-tasks corresponding to machine translation of the following language pairs: English to Tamil, English to Malayalam, English to Telugu and Tamil to Telugu which are available at https://competitions.codalab.org/competitions/27650. We provided the participants with training and development datasets to perform experiments, and the results were evaluated on unseen test data. In total, 46 research groups participated in the shared task and 7 experimental runs were submitted for evaluation. We used BLEU scores for assessment of the translations.",
}
"""

_DESCRIPTION = """\
The goal of this task is to improve access and the production of information for monolingual speakers of Dravidian languages, it is necessary to have machine translation. The shared task aims to promote research towards this goal.
"""

_LICENSE = "Creative Commons Attribution 4.0 International Licence"

_URLS = {
    "malayalam-english": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/file/d/11A0kPNhrv9xd1ZwqCCw9HltfnsVvm_o9/view?usp=sharing",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/file/d/1MJjjE5ieQ1xygMhqs_VYD0veZlBq794y/view?usp=sharing",
    },
    "tamil-english": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/file/d/1-2E-bdG_Ad_npW_QFKSUON2LeH1gENsX/view?usp=sharing",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/file/d/1-24hW9E_GiyvAaeN-utR_97ghrFwnn9i/view?usp=sharing",
    },
    "telugu-english": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/file/d/1-7v7HUbT1bGMIs7THc2XnUb0rFK0wRHO/view?usp=sharing",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/file/d/1-2tjohkFrqFj0qnMho22yZiVbTrgZ34_/view?usp=sharing",
    },
    "tamil-telugu": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/file/d/1-AvUG7RAeNPuFOQDIGa-xEL8FbtxNBHZ/view?usp=sharing",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/file/d/1-AIIHxdbSTSru01pvNPiMe3UPldnz033/view?usp=sharing",
    },
}

class DravidianMT(datasets.GeneratorBasedBuilder):
    """ Machine Translation in Dravidian languages dataset"""

    VERSION = datasets.Version("1.0.0")

    BUILDER_CONFIGS = [
        datasets.BuilderConfig(
            name="malayalam-english",
            version=VERSION,
            description="This part of the dataset covers the translation between Malayalam and English",
        ),
        datasets.BuilderConfig(
            name="tamil-english",
            version=VERSION,
            description="This part of the dataset covers the translation between Tamil and English",
        ),
        datasets.BuilderConfig(
            name="telugu-english",
            version=VERSION,
            description="This part of the dataset covers the translation between Telugu and English",
        ),
        datasets.BuilderConfig(
            name="tamil-telugu",
            version=VERSION,
            description="This part of the dataset covers the translation between Tamil and Telugu",
        ),
    ]

    def _info(self):

        if (
                self.config.name == "malayalam-english"
        ):  # This is the name of the configuration selected in BUILDER_CONFIGS above
            features = datasets.Features(
                {
                    "translation": datasets.features.Translation(
                        language_pair=("en", "ml")
                    ),
                }
            )

        elif self.config.name == "tamil-english":
            features = datasets.Features(
                {
                    "translation": datasets.features.Translation(
                        language_pair=("en", "ta")
                    ),
                }
            )
        elif self.config.name == "telugu-english":
            features = datasets.Features(
                {
                    "translation": datasets.features.Translation(
                        language_pair=("en", "te")
                    ),
                }
            )
        # else self.config.name == "tamil-telugu"
        else:
            features = datasets.Features(
                {
                    "translation": datasets.features.Translation(
                        language_pair=("ta", "te")
                    ),
                }
            )

        return datasets.DatasetInfo(
            # This is the description that will appear on the datasets page.
            description=_DESCRIPTION,
            # This defines the different columns of the dataset and their types
            features=features,  # Here we define them above because they are different between the two configurations
            # If there's a common (input, target) tuple from the features,
            # specify them here. They'll be used if as_supervised=True in
            # builder.as_dataset.
            supervised_keys=None,
            # Homepage of the dataset for documentation
            homepage=_HOMEPAGE,
            # License for the dataset if available
            license=_LICENSE,
            # Citation for the dataset
            citation=_CITATION,
        )

    def _split_generators(self, dl_manager):
        """ Returns SplitGenerators"""

        my_urls = _URLS[self.config.name]

        train_path = dl_manager.download_and_extract(my_urls["TRAIN_DOWNLOAD_URL"])
        test_path = dl_manager.download_and_extract(my_urls["TEST_DOWNLOAD_URL"])

        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                gen_kwargs={
                    "filepath": train_path,
                    "split": "train",
                },
            ),
            datasets.SplitGenerator(
                name=datasets.Split.TEST,
                gen_kwargs={"filepath": test_path, "split": "test"},
            ),
        ]

    def _generate_examples(self, filepath, split):
        """ Generate Dravidian MT examples"""

        with open(filepath, encoding="utf-8") as f:
            reader = csv.DictReader(f, quoting=csv.QUOTE_NONE)
            for idx, row in enumerate(reader):

                if self.config.name == "tamil-english":
                    result = {"translation": {"en": row["en"], "ta": row["ta"]}}

                elif self.config.name == "malayalam-english":
                    result = {"translation": {"en": row["en"], "ml": row["ml"]}}

                elif self.config.name == "telugu-english":
                    result = {"translation": {"en": row["en"], "te": row["te"]}}

                else:
                    result = {"translation": {"ta": row["ta"], "te": row["te"]}}

                yield idx, result

Context:
Hi ! This is because you provide URLs to see the file on google drive, not download them.
You can fix this by changing the urls to download urls:
_URLS = {
    "malayalam-english": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=11A0kPNhrv9xd1ZwqCCw9HltfnsVvm_o9&export=download",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=1MJjjE5ieQ1xygMhqs_VYD0veZlBq794y&export=download",
    },
    "tamil-english": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=1-2E-bdG_Ad_npW_QFKSUON2LeH1gENsX&export=download",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=1-24hW9E_GiyvAaeN-utR_97ghrFwnn9i&export=download",
    },
    "telugu-english": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=1-7v7HUbT1bGMIs7THc2XnUb0rFK0wRHO&export=download",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=1-2tjohkFrqFj0qnMho22yZiVbTrgZ34_&export=download",
    },
    "tamil-telugu": {
        "TRAIN_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=1-AvUG7RAeNPuFOQDIGa-xEL8FbtxNBHZ&export=download",
        "TEST_DOWNLOAD_URL": "https://drive.google.com/u/0/uc?id=1-AIIHxdbSTSru01pvNPiMe3UPldnz033&export=download",
    },
}


Question:
We have uploaded a dataset hatexplain datset 1 from a different account. How can we link this dataset to our organisation hatealert 1?
Similarly, we want to link the models trained using this dataset to the dataset page. How can we do that ?

Context:
Looking at your shared dataset, I think it was uploaded as a canonical dataset 3 and not a community provided dataset (instructions to upload dataset). Please correct me if I’m wrong.
Answering your second question, if you want to link a dataset and a model, you can simply change the model card in the model repo (README.md). For example:
---
datasets:
  - bookcorpus
  - wikipedia
---

Here is an example 1


Use load dataset to load a sample of the dataset

Question:
Hello  ,
I would really love to load a sample of the dataset rather than the whole data at first. Can I do this with hugging face library. I don’t want to download the full dataset as it is 23GB large rather just download a sample and work on it asap before working on the whole dataset.
Any ideas ??

Context:
As far as I know, this is something that is actively being worked on.

      github.com/huggingface/datasets

      [WIP] Dataset Streaming 32

      huggingface:master ← huggingface:dataset-streaming

        opened

        May 19, 2021

          lhoestq

          +862
          -68

    # Dataset Streaming

## API

Current API is

```python
from datasets.stre…aming import load_dataset

# Load an IterableDataset without downloading data
snli = load_dataset("snli")

# Access examples by streaming data
print(next(iter(snli["train"])))
# {'premise': 'A person on a horse jumps over a broken down airplane.',
#  'hypothesis': 'A person is training his horse for a competition.',
#  'label': 1}
```

I already implemented a few methods:
- IterableDataset.map: apply transforms on-the-fly to the examples
- IterableDataset.shuffle: shuffle the data _a la_ TFDS, i.e. with a shuffling buffer
- IterableDataset.with_format: set the format to `"torch"` to get a `torch.utils.data.IterableDataset`
- merge_datasets: merge two iterable datasets by alternating one or the other (you can specify the probabilities)

I would love to have your opinion on the API design :)

## Implementation details

### Streaming

Data streaming is done using `fsspec` which has nice caching features.

To make dataset streaming work I extend the `open` function of dataset scripts to support opening remote files without downloading them entirely. It also works with remote compressed archives (currently only zip is supported):

```python
# Get a file-like object by streaming data from a remote file
open("https://github.com/davidsbatista/NER-datasets/raw/master/CONLL2003/train.txt")

# Get a file-like object by streaming data from a remote compressed archive by using the hop separator "::"
open("zip://snli_1.0_train.txt::https://nlp.stanford.edu/projects/snli/snli_1.0.zip")
```

I also extend the `os.path.join` function to support navigation in remote compressed archives, since it has to deal with the `"::"` separator. This separator is used by `fsspec`.

Finally I also added a retry mechanism in case the connection fails during data streaming.

### Transforms

An IterableDataset wraps an ExamplesIterable instance. There are different subclasses depending on the transforms we want to apply:
- ExamplesIterable: the basic one
- MappedExamplesIterable: an iterable with a `map` function applied on the fly
- BufferShuffledExamplesIterable: an iterable with a shuffling buffer
- CyclingMultiSourcesExamplesIterable: alternates between several ExamplesIterable
- RandomlyCyclingMultiSourcesExamplesIterable: randomly alternates between several ExamplesIterable

### DatasetBuilder

I use the same builders as usual. I just added a new method `_get_examples_iterable_for_split` to get an ExamplesIterable for a given split. Currently only the GeneratorBasedBuilder and the ArrowBasedBuilder implement it.

The BeamBasedBuilder doesn't implement it yet.
It means that datasets like wikipedia and natural_questions can't be loaded as IterableDataset for now.

## Other details

I did some changes in many dataset script to use `download` instead of `download_and_extract` when extraction is not needed. This will avoid errors for streaming

## TODO

usual stuff:

- [ ] make streaming dependency "aiohttp" optional: `pip install datasets[streaming]`
- [ ] tests
- [ ] docs

@lhoestq might be able to provide more info


Querying column is slow for datasets with indices mapping

Question:
Hi,
I have a large dataset and wanna query values for a specific column for batch packing. It turns out that querying is super slow if I use datasets with indices mapping (e.g., after train_test_split or select operations).
The code below reproduces the issue:
import time
import numpy as np
from datasets import Dataset

input_ids = np.random.randint(0, 60_000, (500_000, 128)).tolist()
length = np.random.randint(3, 128, (500_000)).tolist()

dataset = Dataset.from_dict({"input_ids": input_ids, "length": length})
dataset_dict = dataset.train_test_split(test_size=0.1)

# ---------------------------------------------------------------
# Original dataset
start = time.time()
_ = dataset["length"]
print(f"Operation took {time.time() - start:.2f} seconds")
# Operation took 0.15 seconds
# ---------------------------------------------------------------

# ---------------------------------------------------------------
# Dataset with indices mapping
start = time.time()
_ = dataset_dict["train"]["length"]
print(f"Operation took {time.time() - start:.2f} seconds")
# Operation took 5.74 seconds
# ---------------------------------------------------------------

It takes forever to load values for my 500+ Gb dataset, and I have to use flatten_indices on each dataset split to get deep copies with super fast querying performance. Anyway, flatten_indices is no faster and also takes too much time. So preprocessing becomes super painful to achieve acceptable performance.
Is there a way to achieve acceptable performance without flattening indices?

Context: Which version are you using? The latest versions  give dramatic improvements on such querying operations.


Question:
Hi,
I’ve been using /datasets for a while and couldn’t find an elegant solution to transform list-like elements into separate rows.
I have a large source code corpus (>500 Gb) that needs to be preprocessed for further language modeling. The corpus consists of source code files that are too long to be fed into non-sparse self-attention models. So I tokenize files into chunks, resulting in each row containing a list of chunks.
Tokenization looks like this:
tokenizer = MBartTokenizerFast.from_pretrained("./code_mbart_spm_60K")

def map_fn(example: Dict[str, Any]) -> Dict[str, Any]:
    encoding = tokenizer(example["text"], return_overflowing_tokens=True)
    del encoding["overflow_to_sample_mapping"]
    return {**encoding}

dataset = datasets.load_from_disk("./code_dataset")
dataset = dataset.map(map_fn, batched=False, remove_columns=dataset.column_names, num_proc=24)
example = dataset[0] # type: {"input_ids": List[List[int]], "attention_mask": List[List[int]]}

After that, I need my dataset to be flat – i.e. each chunk to be a separate row in the dataset. So far I iterate over each example and each chunk and write tokenized chunks into jsonlines files. Then I load json files with /datasets and save results.
I can’t convert my tokenized dataset into a pandas dataframe and apply explode method due to RAM limitations.
Does anybody know if there is an elegant workaround/solution to perform such transformation?

Context: What if you try with batched=True ?


Setting format of columns for nested dictionary datasets with set_format

Question:
Hi ::
is there a way to set the format of nested columns, as the main columns are python dictionaries?
A batch would have the following format:
examples = {
         id": [],
        "source": {
            "something": [],
            "something_more": [],
        },
        "target": {
            "something": [],
            "something_more": [],
        },
    }

I want to set the format of “something” and “something_other”.
Thanks!

Context:
Hi ! Currently this is not possible unfortunately.
What I would suggest is either:

unnest your columns using dataset.flatten()

OR use your own formatting transform with dataset.set_transform. This way you can use your own transform that sets the format of your nested columns to whatever your want. The transform should take as input a python dictionary and should return the formatted data.


Saving dataset in the current state without cache

Question:
Hi!
I am working with a wikipedia dataset and want to distribute it across my machines. It is preprocessed roughly like this
TEST_SIZE = 10_000
VAL_SIZE = 3_000

wiki_processed = datasets.load_dataset("wikipedia", "20200501.en")["train"]
wiki_processed = wiki_processed.map(map_fn, num_proc=20)
wiki_processed = wiki_processed.filter(lambda x: bool(x["text"]), num_proc=20)

dataset_dict_val = wiki_processed.train_test_split(test_size=VAL_SIZE, seed=42)
dataset_dict = dataset_dict_val["train"].train_test_split(test_size=TEST_SIZE, seed=42)
dataset_dict["validation"] = dataset_dict_val["test"]

dataset_dict.save_to_disk("../data/wikipedia_rank")

Originally, wikipedia takes about 16Gb, but wikipedia_rank is more than 90Gb.
This happens, because every subset train, validation, test has the original dataset.arrow in its directory + cache files.
~/d/c/d/wikipedia_rank ❯❯❯ du -hs *
4.0K	dataset_dict.json
16G	test
61G	train
16G	validation
~/d/c/d/wikipedia_rank ❯❯❯ du -hs test/*
4.3M	test/cache-0924ed04d225940e.arrow
4.1M	test/cache-0ca033cdbd106a2d.arrow
4.3M	test/cache-1a97b84558ed7b1d.arrow
4.3M	test/cache-22a7c31dbee9c03c.arrow
80K	test/cache-2bb6f88c5c8f4329.arrow
4.4M	test/cache-2f128b0e055ed3cf.arrow
4.4M	test/cache-3c97985cec531e1a.arrow
4.4M	test/cache-482dc3ba3be09fbc.arrow
4.0M	test/cache-4c932850df42ad39.arrow
4.7M	test/cache-4ce30cf00c698ef6.arrow
4.7M	test/cache-4e36968451cc71ba.arrow
4.5M	test/cache-550a1d3d28c3487a.arrow
4.9M	test/cache-599e1bb413fe3195.arrow
4.2M	test/cache-637a2658f481ab57.arrow
4.7M	test/cache-7c27537563c55dd1.arrow
5.1M	test/cache-924c2f977e10633f.arrow
4.7M	test/cache-9510e0cd305aa3fa.arrow
4.5M	test/cache-9664e702137250f3.arrow
4.4M	test/cache-9c3a9b369f101759.arrow
4.1M	test/cache-ab0b7947030b6ce7.arrow
4.0M	test/cache-e124027841f7ed5b.arrow
16G	test/dataset.arrow  # I want test/dataset.arrow only to hold 10K examples, not the whole wikipedia
200K	test/dataset_info.json
4.0K	test/state.json

What is the recommended way to only save a sampled and preprocessed version of my dataset without cache files?

Context:
UPD: this workaround is not feasible in my case even with 1Tb of RAM =(
Here’s a workaround that I figured out, but it is slow and requires to put all of your data into RAM (and I don’t have enough).
dataset_dict = datasets.DatasetDict({
    "train": datasets.arrow_dataset.Dataset.from_dict(dataset_dict["train"].data.to_pydict()),
    "validation": datasets.arrow_dataset.Dataset.from_dict(dataset_dict["validation"].data.to_pydict()),
    "test": datasets.arrow_dataset.Dataset.from_dict(dataset_dict["test"].data.to_pydict()),
})

dataset_dict.save_to_disk("../data/wikipedia_rank_nocache")


Question: I have fine tuned a model using a dataset. I want the model name to be displayed alongside the dataset in the dataset hub. How to do this?

Context: If you tag the dataset on the model card it should work


HF Datasets not working with Language Modeling notebook

Question:
Posted this in the beginners part of the forum, but didnt get any responses so decided to repost here.
I am trying to use this notebook (https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb 5) to finetune and generate text with GPT-Neo using my own custom dataset. I uploaded my own text file to my dataset, but when trying to use it, it just gives me this error
FileNotFoundError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
354 try:
→ 355 local_path = cached_path(file_path, download_config=download_config)
356 except FileNotFoundError:
4 frames
FileNotFoundError: Couldn’t find file at htts://huggingface.co/datasets/Trainmaster9977/zbakuman/resolve/main/zbakuman.py
During handling of the above exception, another exception occurred:
FileNotFoundError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
357 raise FileNotFoundError(
358 “Couldn’t find file locally at {}, or remotely at {}. Please provide a valid {} name”.format(
→ 359 combined_path, file_path, “dataset” if dataset else “metric”
360 )
361 )
FileNotFoundError: Couldn’t find file locally at Trainmaster9977/zbakuman/zbakuman.py, or remotely at htts://huggingface.co/datasets/Trainmaster9977/zbakuman/resolve/main/zbakuman.py. Please provide a valid dataset name
(Slightly modified beginning of url due to new users only able to put 2 links in posts)
I tried to use this to load it
from datasets import load_dataset
dataset = load_dataset(“Trainmaster9977/zbakuman”)

Context:
hey @Trainmaster9977, as described in the docs 18 you need to provide an argument to load_dataset that indicates the file format (csv, json, etc).
ps. in future please don’t create duplicate posts (either edit the original one or delete it if necessary)


Cant save Dataset as Parquet-File since Updating Datasets?

Question:
Hi Guys,
i was using Datasets==1.5.0 while Preprocessing my Dataset & Saving it. I updated to the latest Datasets Version (1.6.1) and since then i cant export my Datasets as Parquet File.
With Version 1.5.0 i could just do:
import pyarrow.parquet as pq
...
...
pq.write_table(train_dataset.data, 'train.parquet')
pq.write_table(eval_dataset.data, 'eval.parquet')

When i run the same code with the latest datasets version i get:
  File "../preprocess_dataset.py", line 132, in <module>
    pq.write_table(train_dataset.data, f'{resampled_data_dir}/{data_args.dataset_config_name}.train.parquet')
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 1674, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 588, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
TypeError: Argument 'table' has incorrect type (expected pyarrow.lib.Table, got ConcatenationTable)

Should i just use 1.5.0 or is the a quick and easy work around?
Im not that familar with python. in java i could just use one version for one project and another version for another project. can / should i do the same here?

Context:
pq.write_table(train_dataset.data.table,
pq.write_table(eval_dataset.data.table, 'eval.parquet')

is working , so this can be closed


Question:
Hi guys,
I wanted to train a net based on HuggingFace. Unfortunately the validation process runs out of memory at the end. The values are still estimated by the model, but when the labels are to be converted to texts and the WER is to be determined, the program crashes (out of memory):
***** Running Evaluation *****
  Num examples = 15588
  Batch size = 4
100%|███████████████████████████████████████| 3897/3897 [21:27<00:00,  3.07it/s]Traceback (most recent call last):
  File "/tmp/pycharm_project_263/audioengine/model/finetuning/wav2vec2/finetune_parquet.py", line 151, in <module>
  File "/tmp/pycharm_project_263/audioengine/model/finetuning/wav2vec2/finetune_parquet.py", line 128, in main
    max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1757, in evaluate
    output = self.prediction_loop(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1930, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "/tmp/pycharm_project_263/audioengine/model/finetuning/wav2vec2/wav2vec2_trainer.py", line 191, in __call__
    wer = wer_metric.compute(predictions=pred_str, references=label_str)
  File "/usr/local/lib/python3.8/dist-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/warmachine/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/usr/local/lib/python3.8/dist-packages/jiwer/measures.py", line 80, in wer
    measures = compute_measures(
  File "/usr/local/lib/python3.8/dist-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/usr/local/lib/python3.8/dist-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError
100%|███████████████████████████████████████| 3897/3897 [21:40<00:00,  3.00it/s]

Process finished with exit code 1

wer_metric = load_metric("wer")

def compute_metrics(processor):
    def __call__(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)

        pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

        pred_str = processor.batch_decode(pred_ids)
        # we do not want to group tokens when computing the metrics
        label_str = processor.batch_decode(pred.label_ids, group_tokens=False)

        wer = wer_metric.compute(predictions=pred_str, references=label_str)

        return {"wer": wer}

    return __call__
...

   trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        tokenizer=processor.feature_extractor,
        train_seq_lengths=train_dataset.input_seq_lengths,
        compute_metrics=compute_metrics(processor),
    )

Context: I’m guessing you don’t have enough RAM to host all your decoded predictions in memory. You should split your dataset in smaller pieces to get the decoded predictions.


Question: Is there any native compatibility in datasets to construct it from NumPy arrays to be further used in transformers without writing it to a file and loading it that way?

Context: As far as I know there isn’t a native Dataset.from_numpy method, but you could map your array to a Python dictionary and use the from_dict method: Loading a Dataset — datasets 1.5.0 documentation 5


Question:
Hi,
May I ask when the next release will take place?
I would like to pip install the new features

Context:
hey @Hwijeen, i don’t know when the next datasets release will be, but in the meantime you can pip install the new features directly from master as follows:
pip install git+https://github.com/huggingface/datasets.git

note that this is probably not stable (in the sense that master is continuously updated, some features might change etc)


Using load_dataset.set_transform() function along with Trainer class

Question:
Hello,
I’m trying to use the load_dataset.set_transform(…) function along with DataCollatorForLanguageModeling  3 and the Trainer 1 class from the transformers library to pretrain a model. Since I have a large dataset, tokenization does not fit in RAM and using the .map() function uses way too much disk space (> 500Go) which is limited in my case. So I need to tokenize on the fly.
While the set_transform works as expected if I index the dataset, I don’t know why it fails when I plug it with a DataCollatorForLanguageModeling  and a Trainer.
tokenizer = ...

    def encode(batch):
        return tokenizer(batch["text"], padding="max_length", truncation=True, max_length=512, return_tensors="pt")

    train_dataset = load_dataset('text', data_files={'train': txt_train_dataset})
    train_dataset.set_transform(encode)

    validation_dataset = load_dataset('text', data_files={'validation': txt_validation_dataset})
    validation_dataset.set_transform(encode)

    print(train_dataset["train"][:3]) # works as expected: {'input_ids': ..., 'attention_mask': ...}

    data_collator = DataCollatorForLanguageModeling(
            tokenizer=tokenizer, mlm=True, mlm_probability=0.15
        )

    trainer = Trainer(
            model=...,
            args=...,
            data_collator=data_collator,
            train_dataset=train_dataset["train"],
            eval_dataset=validation_dataset["validation"],
            compute_metrics=...,
        )

Im probably missing something.

Context: hey @tkon3, when you say “it fails” what do you mean exactly? there’s a related thread on set_transform here that might be useful: Understanding set_transform 11


How to use dataset with run_language_modeling?

Question:
I have downloaded the s2orc dataset and saved it to disk.
Since it’s in the arrow format, I cannot figure out how to use the run_language_modeling command, since that seems to require a text file.
It seems like it would be simple. Can anyone help?

Context:
I modified run_language_modeling.py a little to make it work.
I would still be interested to hear if this is supported.


Question:
This is a problem that I have gotten: full info on this thread 4 I had a quick question -
I have constructed Dataset object using numpy arrays, however when I use this:-
def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=2000)
  return encodings

train_encoded_dataset = train_dataset.map(tok, batched=True)
val_encoded_dataset = val_dataset.map(tok, batched=True)

and I explore my train_encoded_dataset, I see that when trying to view the source sequence:
>>> train_encoded_dataset
>>>Dataset({
    features: ['attention_mask', 'input_ids', 'src', 'tgt'],
    num_rows: 4572
})
>>> train_encoded_dataset['src'][0]

The output of this last command produces text that is completely raw (basically untokenized) string- (like: ‘lorem ipsum…’) which is expected since I didn’t call tokenizer.tokenize
So does anyone have any idea how to get the model to tokenized as well? I tried a few obvious ways, but it didn’t yield anything

Context:
hey @Neel-Gupta, could you share a minimal example of the Dataset object you’re working with?

 Neel-Gupta:

The output of this last command produces text that is completely raw (basically untokenized) string- (like: ‘lorem ipsum…’) which is expected since I didn’t call tokenizer.tokenize
So does anyone have any idea how to get the model to tokenized as well? I tried a few obvious ways, but it didn’t yield anything

just so i understand, you’re saying that the tokenizer is not tokenizing the strings in the src field right?


How to use Dataset with Pytorch Lightning

Question:
Hi,
I’m trying to load the cnn-dailymail 3 dataset to train a model for summarization using pytorch lighntning. To load the dataset with DataLoader I tried to follow the documentation 13 but it doesnt work (the pytorch lightning code I am using does work when the Dataloader isnt using a dataset from huggingface so there shouldnt be a problem in the training procedure).
Here is the code:
def train_dataloader(self):
    train_dataset = load_dataset('cnn_dailymail','3.0.0', split='train')
    train_dataset = train_dataset.map(lambda e: tokenizer(e['article'],e['highlights'], truncation=True, padding='max_length'), batched=True)
    train_dataset.set_format(type='torch')
    dataloader = DataLoader(train_dataset, batch_size=self.hparams.train_batch_size)
    return dataloader

I put ‘article’ and ‘highlight’ in tokenizer as these are the 2 columns in the dataset that correspond to target and source.
Here is the stack trace:
Traceback (most recent call last):
  File "MBART.py", line 346, in <module>
    trainer.fit(model)
  File "venv/lib/python3.6/site-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "venv/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1073, in fit
    results = self.accelerator_backend.train(model)
  File "venv/lib/python3.6/site-packages/pytorch_lightning/accelerators/gpu_backend.py", line 51, in train
    results = self.trainer.run_pretrain_routine(model)
  File "venv/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1224, in run_pretrain_routine
    self._run_sanity_check(ref_model, model)
  File "venv/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1257, in _run_sanity_check
    eval_results = self._evaluate(model, self.val_dataloaders, max_batches, False)
  File "venv/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 305, in _evaluate
    for batch_idx, batch in enumerate(dataloader):
  File "venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "venv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "venv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1087, in __getitem__
    format_kwargs=self._format_kwargs,
  File "venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1074, in _getitem
    format_kwargs=format_kwargs,
  File "venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 890, in _convert_outputs
    v = map_nested(command, v, **map_nested_kwargs)
  File "venv/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 225, in map_nested
    return function(data_struct)
  File "venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 851, in command
    return torch.tensor(x, **format_kwargs)
TypeError: new(): invalid data type 'str'

Any ideas on how to make it work ?
Thanks !

Context:
I think you also need to specify which columns you’d like to keep when doing .set_format(type='torch'). If you don’t do this, then the text columns are still part of the dataset, and converting strings to PyTorch tensors causes an error.
So I think you just need to update that line to:
train_dataset.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])


Apply batched zero shot classification on HuggingFace datasets object

Question:
Hi,
UPDATE: notebook to reproduce: https://colab.research.google.com/drive/1t-ApjHqdSo90NoXSJ5baeh7h-gJx8bLt?usp=sharing 4
I have a large amount of unlabeled texts, stored as a Pandas dataframe. So just a single column called “text”.
I’d like to apply zero-shot classification on all these texts in a batched way using HuggingFace Datasets’ .map(function, batched=True) functionality. I defined the function that I want to apply on batches as follows:
def zero_shot_classify_sequences(examples, threshold=0.5):

    # first, send batch of texts through pipeline
    texts = examples['text']
    outputs = classifier(texts, candidate_labels, multi_label=True)
    # next, for each output:
    final_outputs = []
    for output in outputs:
        # create dictionary (predicted_labels, confidence)
        final_output = {}
        for label, score in zip(output['labels'], output['scores']):
            if score > threshold:
               final_output[label] = score
        final_outputs.append(final_output)

    assert len(final_outputs) == len(texts)
    # set final outputs
    examples['predicted_labels'] = final_outputs

    return examples

The candidate labels are defined outside of this function.
In other words, I’d like to add a new column “predicted_labels”, which, for a batch of texts, should be a list of dictionaries (each dictionary mapping labels to confidence values for a given text - only those for which the confidence value > 0.5). However, when I do updated_dataset = dataset.map(zero_shot_classify_sequences, batched=True, batch_size=10), the output does not look like I’d expect. For a given text, I get the following:
'predicted_labels': {'Delivery & fulfilment technology': None,
  'Novel processing techniques & Equipments': None,
  'Plant-based': None,
  'Retail tech': None}

This should not be the case. In case none of the confidence values is higher than the threshold of 0.5, then the dictionary of “predicted labels” should be empty for that given example.
It probably has to do with the fact that a list of dictionaries is not supported by Apache Arrow? Or is it?

Context: cc @lhoestq


How to combine local data files with an official  dataset

Question:
Hey,
I made a short notebook to show how local data files can be loaded into  Datasets and
consequently, be combined with local data files into one Dataset object.
Check out the google colab here 176.

Context:
So, if I have a very different dataset, but I have the sentence and the mp3 (doesnt matter the audio quality, silence or if major part of the file is silent/background noise).
I only need to create that json file and use it as a base dataset?


Question:
I’m getting this issue when I am trying to map-tokenize a large custom data set. Looks like a multiprocessing issue. Running it with one proc or with a smaller set it seems work. I’ve tried different batch_size and still get the same errors. I also tried sharding it into smaller data sets, but that didn’t help. Thoughts? Thanks!
dataset[‘test’].map(lambda e: tokenizer(e[‘texts’]), batched = True, batch_size = 1000, num_proc  = 8)

error                                     Traceback (most recent call last)
 in
----> 1 dataset[‘test’].map(lambda e: tokenizer(e[‘texts’]), batched = True, batch_size = 1000, num_proc  = 8)
/home/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
1316                 logger.info(“Spawning {} processes”.format(num_proc))
1317                 results = [pool.apply_async(self.class._map_single, kwds=kwds) for kwds in kwds_per_shard]
 → 1318                 transformed_shards = [r.get() for r in results]
1319                 logger.info(“Concatenating {} shards from multiprocessing”.format(num_proc))
1320                 result = concatenate_datasets(transformed_shards)
/home/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py in (.0)
1316                 logger.info(“Spawning {} processes”.format(num_proc))
1317                 results = [pool.apply_async(self.class._map_single, kwds=kwds) for kwds in kwds_per_shard]
 → 1318                 transformed_shards = [r.get() for r in results]
1319                 logger.info(“Concatenating {} shards from multiprocessing”.format(num_proc))
1320                 result = concatenate_datasets(transformed_shards)
/home/venv/lib/python3.6/site-packages/multiprocess/pool.py in get(self, timeout)
642             return self._value
643         else:
 → 644             raise self._value
645
646     def _set(self, i, obj):
/home/venv/lib/python3.6/site-packages/multiprocess/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
422                         break
423                     try:
 → 424                         put(task)
425                     except Exception as e:
426                         job, idx = task[:2]
/home/venv/lib/python3.6/site-packages/multiprocess/connection.py in send(self, obj)
207         self._check_closed()
208         self._check_writable()
 → 209         self._send_bytes(_ForkingPickler.dumps(obj))
210
211     def recv_bytes(self, maxlength=None):
/home/venv/lib/python3.6/site-packages/multiprocess/connection.py in _send_bytes(self, buf)
394         n = len(buf)
395         # For wire compatibility with 3.2 and lower
 → 396         header = struct.pack("!i", n)
397         if n > 16384:
398             # The payload is large so Nagle’s algorithm won’t be triggered
error: ‘i’ format requires -2147483648 <= number <= 2147483647

Context:
Hi there, I got a (maybe) similar issue caused by the multiprocessing in map. Instead of opening a new thread, I thought I would use this one. Note that the error occurs only if I specify num_proc > 1, i.e. use multi-processing:
Code:
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)

datasets = datasets.map(
    lambda sequence: tokenizer(sequence['text'], return_special_tokens_mask=True),
    batched=True,
    batch_size=1000,
    num_proc=2, #psutil.cpu_count()
    remove_columns=['text'],
)

datasets

Error:
Token indices sequence length is longer than the specified maximum sequence length for this model (8395 > 512). Running this sequence through the model will result in indexing errors
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
  File "c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\multiprocess\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\arrow_dataset.py", line 203, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\arrow_dataset.py", line 1695, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\arrow_dataset.py", line 1608, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "<ipython-input-18-25a1ecec1896>", line 9, in <lambda>
NameError: name 'tokenizer' is not defined
"""

The above exception was the direct cause of the following exception:

NameError                                 Traceback (most recent call last)
<ipython-input-18-25a1ecec1896> in <module>
      6 tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)
      7
----> 8 datasets = datasets.map(
      9     lambda sequence: tokenizer(sequence['text'], return_special_tokens_mask=True),
     10     batched=True,

c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc)
    430             cache_file_names = {k: None for k in self}
    431         return DatasetDict(
--> 432             {
    433                 k: dataset.map(
    434                     function=function,

c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\dataset_dict.py in <dictcomp>(.0)
    431         return DatasetDict(
    432             {
--> 433                 k: dataset.map(
    434                     function=function,
    435                     with_indices=with_indices,

c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1483                 logger.info("Spawning {} processes".format(num_proc))
   1484                 results = [pool.apply_async(self.__class__._map_single, kwds=kwds) for kwds in kwds_per_shard]
-> 1485                 transformed_shards = [r.get() for r in results]
   1486                 logger.info("Concatenating {} shards from multiprocessing".format(num_proc))
   1487                 result = concatenate_datasets(transformed_shards)

c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\datasets\arrow_dataset.py in <listcomp>(.0)
   1483                 logger.info("Spawning {} processes".format(num_proc))
   1484                 results = [pool.apply_async(self.__class__._map_single, kwds=kwds) for kwds in kwds_per_shard]
-> 1485                 transformed_shards = [r.get() for r in results]
   1486                 logger.info("Concatenating {} shards from multiprocessing".format(num_proc))
   1487                 result = concatenate_datasets(transformed_shards)

c:\Users\s_scho53\Desktop\L09_Desktop\_FiLMo\.venv\lib\site-packages\multiprocess\pool.py in get(self, timeout)
    769             return self._value
    770         else:
--> 771             raise self._value
    772
    773     def _set(self, i, obj):

NameError: name 'tokenizer' is not defined

I am grateful for any help!


Question:
Hi there,
I am wondering, what is currently the most elegant way to perform a three-way random split (into train, val and test set)? Let’s assume I load_dataset so that:
Dataset({
    features: ['text'],
    num_rows: 19122
})

Subsequently, I’d like to perform the split. Currently I am performing dataset.train_test_split() twice and then recombine the three datasets into one using DatasetDict. However, I assume that this is not the most elegant approach right? I also experimented with ReadInstructions, however, I could only split the data deterministically instead of randomly…
Any one got a better soultion?

Context: cc @lhoestq


Fetching rows of a large Dataset by index

Question:
I was referred here by @lhoestq from this github issue 1.
Background
I have a large dataset, ds_all_utts, of user utterances. I load it using load_from_disk because I saved it with save_to_disk:
ds_all_utts = load_from_disk(ds_all_utts_fname)

ds_all_utts has 2,732,013 rows and these features:
{'ANY': Value(dtype='int64', id=None),
 'COMPLAINTCLARIFICATION': Value(dtype='int64', id=None),
 'COMPLAINTMISHEARD': Value(dtype='int64', id=None),
 'COMPLAINTPRIVACY': Value(dtype='int64', id=None),
 'COMPLAINTREPETITION': Value(dtype='int64', id=None),
 'CRITICISM': Value(dtype='int64', id=None),
 'NEGATIVENAVIGATION': Value(dtype='int64', id=None),
 'OFFENSIVE': Value(dtype='int64', id=None),
 'STOP': Value(dtype='int64', id=None),
 'embedding': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None),
 'frequency': Value(dtype='int64', id=None),
 'user_utterance': Value(dtype='string', id=None)}

user_utterance is a short piece of text (usually just a few words), embedding is a 1280-length vector representing that utterance, frequency is an int, and the rest are binary labels (0 or 1) for the utterance. It’s sorted by descending frequency.
I have another Dataset called neuralgen_ds whose rows represent turns of dialogue along with their context. It has 385,580 rows and these features:
{'session_id': Value(dtype='string', id=None),
 'treelet': Value(dtype='string', id=None),
 'context': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None),
 'bot_utt': Value(dtype='string', id=None),
 'bot_utt_labels': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None),
 'user_utt': Value(dtype='string', id=None),
 'user_utt_labels': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None),
 'GPT2ED': Value(dtype='bool', id=None),
 '__index_level_0__': Value(dtype='int64', id=None)}

Of these, the important one is user_utt, which is the same type of data as ds_all_utts['user_utterance']. Some user utterances appear multiple times in neuralgen_ds; there are 190,602 unique utterances in neuralgen_ds['user_utt'].
What I want to do
For each row of neuralgen_ds, I want to look up the user utterance in ds_all_utts, and copy over certain columns into neuralgen_ds. In particular, I want to copy over embedding and all the capitalized binary labels (ANY, COMPLAINTCLARIFICATION, etc).
My code
First I create a dictionary mapping from a user utterance to its position in ds_all_utts:
ds_all_utts_lookup = {utt: idx for idx, utt in enumerate(ds_all_utts['user_utterance'])}

Then I use .map to add the columns to neuralgen_ds:
cols = ['embedding', 'ANY', 'COMPLAINTCLARIFICATION', 'COMPLAINTMISHEARD', 'COMPLAINTPRIVACY', 'COMPLAINTREPETITION', 'CRITICISM', 'NEGATIVENAVIGATION', 'OFFENSIVE', 'STOP']

def map_fn(examples):
    user_utts = examples['user_utt']  # list of str
    idxs = [ds_all_utts_lookup[user_utt] for user_utt in user_utts]  # list of int
    ds_slice = ds_all_utts[idxs]  # dict
    result = {col: ds_slice[col] for col in cols}
    return result

neuralgen_ds = neuralgen_ds.map(map_fn, batched=True, batch_size=100)

The tqdm estimate says this .map will take over 8 hours. Adjusting batch_size doesn’t seem to help. The slowest part of map_fn is this line:
ds_slice = ds_all_utts[idxs]  # dict

Other questions

Are you on a SSD or an HDD ?

I’m not sure, but I followed these instructions 2 and got
>>> lsblk -o name,rota
NAME   ROTA
sda       1
├─sda1    1
├─sda2    1
├─sda5    1
├─sda6    1
├─sda7    1
└─sda8    1
sdb       1
└─sdb1    1

Context:
Hi !
To speed up map operations, you can run it with multiprocessing with specifying num_proc= in map. Usually it’s better to set it to the number of cores your CPU has.
Let me know if that helps.
Also it looks like you’re using a HDD, which is slower than an SSD. Since your script does a lot of read/write operations (writing a dataset when reading data from another one), I’d expect that the HDD slows down the process a bit unfortunately.


Question:
I’ve been working on a side project that uses phonetic English language models for text generation. Since I’m not aware of any existing phonetic English datasets, I’ve been preprocessing existing English text datasets with my phonemization script to give myself enough training data. Mainly OSCAR for pretraining the model, and then my own small datasets for fine-tuning on specific tasks.
My workflow has been:

downloading the 2.5TB oscar_en shuffled text file
processing it (in chunks) to its phonetic representation and saving those text files to disk
batch tokenizing those files and saving them to a local HuggingFace dataset, because it takes hours (or days) to tokenize the whole thing at the beginning of a training

Even with only 3% of the original OSCAR corpus phonemized, my dataset is up to over a 1.2TB on disk. Which I was okay with – I’m running out of local storage, but I was never going to be able to use the whole OSCAR corpus anyway on my rinky-dink home setup.
But this month has brought two things to HuggingFace – OSCAR in the datasets library, and on-the-fly transforms.
Am I right in understanding that I could load the oscar_en corpus from the HF Dataset, and then pass to set_transform a function that would phonemize and tokenize the samples, and the only hit to my disk would be the arrow cache of the original OSCAR dataset? And that I would be able to quickly resume training from my most recent checkpoint, since it’d just be loading from that cache?
I imagine it’ll slow the overall training down and I might not be able to feed my GPUs as quickly as I’d like, but the simplified workflow might be worth the performance hit (especially if I find another bug in my phonemizer script that makes me want to redo everything)
Building off of that, if one wanted to do a BART-style pretraining, would it be possible start with a single-column dataset, and pass to set_transform a function that returns the tokenized original dataset as the targets, and a randomly-masked version of the original tokens as the inputs, all on the fly? [Forgive me if this last question is dumb or nonsensical, I have a very limited understanding of seq2seq training / how BART works]

Context: set_transform does not cache 23 the resulting data. Depending on the data/storage you have available you may want to opt for map 13. Both have a low memory footprint.


Question: Hi, I am a beginner. I was wondering if there was a predefined dataset that contains various service’s terms and conditions and or privacy policies to fine-tune a model. Thanks

Context:
Hi ! Afaik there’s no such dataset available on the datasets hub yet.
Do you know any dataset that we could add ?


Can dataset.map accept multiple arguments like python map

Question:
In Python, map works as follows:
map(func, arg_1, arg_2)

In datasets.map, we are required to pass in a callable (which expects objects of form dataset[idx], which means that certain things like tokenizer have to be defined and should be accessible within the scope of this function, along with that other parameters that we want to pass. Can we pass arguments like a normal func call as shown above ? I’m asking because I have two preprocess_func for train and validation split, and I have to write both functions twice which looks repetitive (since there are slight changes in both).

Context: Hi @prajjwal1, as described in the docs 202, you can pass a dict called fn_kwargs that can include the extra arguments for your map function


Question:
Hello all,
I would like to start an NLP project, analysing financial data in Hebrew. To do so, I would like to train a Bert model on the Financial PhraseBank. Currently, the sentences are in English and was wondering if anyone would like to/know anyone that could assist me in translating this.
Thank you

Context:
Hi The Udster,
You could take a look at https://towardsdatascience.com/going-global-how-to-multi-task-in-multiple-languages-with-the-mt5-transformer-892617cd890c 5
Somewhere in the article the author describes how to use the simpletransformers library (based on Huggingface Transformers) to translate datasets into several language using the MarianMT models. (opus_xyz)
The translation quality is not so high as the one from google cloud, but it still beats manual translation
regards,
Yeb


Is it possible to filter/select dataset class by a column’s values?

Question:
I am wondering if it possible to use the dataset indices to:

get the values for a column
use (#1) to select/filter the original dataset by the order of those values

The problem I have is this: I am using HF’s dataset class for SQuAD 2.0 data like so:
from datasets import load_dataset
dataset = load_dataset("squad_v2")

When I train, I collect the indices and can use those indices to filter/select the dataset in order of how it was trained with randomly shuffled batches like so:
dataset['train'].select(indices=[list of indices here])

The problem is when I also want to use HF’s scoring method for SQuAD 2.0. I need the original data set and its tokenized “features” to be in the same order. When the training data is tokenized, it does not share the same length; it gets larger, owing to text of varying length. For this reason, I cannot use the indices emitted during training to align my original training data properly; the sizes are not the same.
One way that I could align both data sets is to:

collect the indices used during training
use those indices to create a new training data set in the right order dataset['train'].select(indices=[list of indices here])

then from the output of step 2, get each a list of all the strings found in the id column
use the strings found in the id column to then re-order the dataset class by the each and every unique string value.

I do not think this is possible judging from the docs here, but I am curious if anyone has a recommendation:
https://huggingface.co/docs/datasets/package_reference/main_classes.html 1
A painful way of doing what I want is through elastic search (its very slow, but I am feeling that there has to be a better way to filter / query a dataset.
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'localhost'}]).ping()
dataset['train'].add_elasticsearch_index(column='id')
out1 = dataset['train'].search(index_name='id', query='56be85543aeaaa14008c9063')
out1[1]

Context:
From what I can understand, what you want is somehow filter the dataset and then use the same dataset to compute metrics, is this right?
You should be able to do this by

get your filtered dataset
create a dataloader
iterate over the batches and do prediction for each batch
compute the metrics.

 for batch in dataloader:
    model_input, targets = batch
    predictions = model(model_inputs)
    metric.add_batch(predictions, targets)

score = metric.compute()


Does load_dataset load the data in to the memory?

Question:
What if the data file on the disk of a loaded dataset (with load_dataset function) changed?
Please consider the data structure and names stays the same and only some values change. So without loading the dataset again can I access new data points?

Context:
Hi ! If the data files changes - for example if one value was changed, then load_dataset will return a new updated dataset that takes this change into account.
Indeed the datasets library looks for any change in your data file before returning the dataset to make sure it doesn’t reload an outdated version from the cache.


NLP dataset for ByteLevelTokenizer Training

Question:
Hi I would like to train my own ByteLevelBPETokenizer using an nlp dataset.
tokenizer = ByteLevelBPETokenizer()

tokenizer.train(files=???, vocab_size=52000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

The dataset is from:
from datasets import load_dataset
dataset = load_dataset('wikicorpus', 'raw_en')

How can I process this dataset to input it in the tokenizer.train() function?
Thanks

Context:
You can take a look at the example script here:

      github.com

    huggingface/tokenizers/blob/master/bindings/python/examples/train_with_datasets.py 16
import datasets
from tokenizers import normalizers, pre_tokenizers, Tokenizer, models, trainers

# Build a tokenizer
bpe_tokenizer = Tokenizer(models.BPE())
bpe_tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
bpe_tokenizer.normalizer = normalizers.Lowercase()

# Initialize a dataset
dataset = datasets.load_dataset("wikitext", "wikitext-103-raw-v1")

# Build an iterator over this dataset
def batch_iterator():
    batch_length = 1000
    for i in range(0, len(dataset["train"]), batch_length):
        yield dataset["train"][i : i + batch_length]["text"]

# And finally train
bpe_tokenizer.train_from_iterator(batch_iterator(), length=len(dataset["train"]))


Question:
Hi,
when I tried to load the weights of model on my device,
it is shown “no space left on device”. And when I restart the machine, this message might go.
How can I clean the cache without restarting my machine?

Context: If you’re using torch, you can try torch.cuda.empty_cache(), and if that’s not enough, send your model to the CPU first, then empty the cache. But, if we’re being honest, I can’t ever quite get it to work the way I expect, and usually I end up restarting my kernel, as it’s the only way I know to reliably clear up GPU memory.


AttributeError: ‘DatasetDict’ object has no attribute ‘train_test_split’

Question:
Shouldn’t this work?
dataset = load_dataset('json', data_files='path/to/file')
dataset.train_test_split(test_size=0.15)

I’m getting this following error:
Using custom data configuration default

Downloading and preparing dataset json/default-cf892ee5bc3fc36a (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/json/default-cf892ee5bc3fc36a/0.0.0/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514...
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/json/default-cf892ee5bc3fc36a/0.0.0/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514. Subsequent calls will reuse this data.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-59d55201b8c3> in <module>()
      1 dataset = load_dataset('json', data_files='/path/to/file')
----> 2 dataset.train_test_split(test_size=0.15)
      3 dataset.shard(10)
      4 dataset

AttributeError: 'DatasetDict' object has no attribute 'train_test_split'

Context:
Hi @thecity2, as far as I know train_test_split operates on Dataset objects, not DatasetDict objects.
For example, this works
squad = (load_dataset('squad', split='train')
        .train_test_split(train_size=800, test_size=200))

because I’ve picked the train split and so load_dataset returns a Dataset object. On the other hand, this does not work:
squad = load_dataset('squad').train_test_split(train_size=800, test_size=200)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-10-d3fb264651eb> in <module>
----> 1 squad = load_dataset('squad').train_test_split(train_size=800, test_size=200)

AttributeError: 'DatasetDict' object has no attribute 'train_test_split'

It seems that your load_dataset is returning the latter, so you could try applying train_test_split on one of the Dataset objects that lives in your dataset.


Getting PermissionError: [WinError 32] When Using Load_Dataset()

Question:
I am trying to use the load_dataset command to create a dataset of my CSV train and test files. However, when attempting to load in my csv files, I’m getting a windows error.
My code:
from datasets import load_dataset

# load data
train_dataset = load_dataset('csv', data_files='C:/Users/WTF/Desktop/cleaned_dataset/train.csv')

The error return:
Downloading and preparing dataset csv/default-6eb8a0ce457cdcea (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to C:\Users\WTF\.cache\huggingface\datasets\csv\default-6eb8a0ce457cdcea\0.0.0\2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2...

0 tables [00:00, ? tables/s]AAAA
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\datasets\builder.py in incomplete_dir(dirname)
    484                     try:
--> 485                         yield tmp_dir
    486                         if os.path.isdir(dirname):

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    526                         self._download_and_prepare(
--> 527                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    528                         )

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    603                 # Prepare split will record examples associated to the split
--> 604                 self._prepare_split(split_generator, **prepare_split_kwargs)
    605             except OSError as e:

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    958         not_verbose = bool(logger.getEffectiveLevel() > WARNING)
--> 959         for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
    960             writer.write_table(table)

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\tqdm\std.py in __iter__(self)
   1103
-> 1104         for obj in iterable:
   1105             yield obj

C:\Users\WTF\.cache\huggingface\modules\datasets_modules\datasets\csv\2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2\csv.py in _generate_tables(self, files)
    126                 float_precision=self.config.float_precision,
--> 127                 chunksize=self.config.chunksize,
    128             )

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)
    701
--> 702         return _read(filepath_or_buffer, kwds)
    703

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds)
    428     # Create the parser.
--> 429     parser = TextFileReader(filepath_or_buffer, **kwds)
    430

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\parsers.py in __init__(self, f, engine, **kwds)
    894
--> 895         self._make_engine(self.engine)
    896

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\parsers.py in _make_engine(self, engine)
   1121         if engine == 'c':
-> 1122             self._engine = CParserWrapper(self.f, **self.options)
   1123         else:

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\parsers.py in __init__(self, src, **kwds)
   1852
-> 1853         self._reader = parsers.TextReader(src, **kwds)
   1854         self.unnamed_cols = self._reader.unnamed_cols

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader.__cinit__()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._get_header()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 57: invalid start byte

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
<ipython-input-10-ab3af5dabfdc> in <module>()
      2
      3 # load data
----> 4 train_dataset = load_dataset('csv', data_files='C:/Users/WTF/Desktop/cleaned_dataset/train.csv')
      5 test_dataset = load_dataset('csv', data_files='C:/Users/WTF/Desktop/cleaned_dataset/test.csv')

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    610         download_config=download_config,
    611         download_mode=download_mode,
--> 612         ignore_verifications=ignore_verifications,
    613     )
    614

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    532                     self.info.size_in_bytes = self.info.dataset_size + self.info.download_size
    533                     # Save info
--> 534                     self._save_info()
    535
    536             # Download post processing resources

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\contextlib.py in __exit__(self, type, value, traceback)
    128                 value = type()
    129             try:
--> 130                 self.gen.throw(type, value, traceback)
    131             except StopIteration as exc:
    132                 # Suppress StopIteration *unless* it's the same exception that

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\site-packages\datasets\builder.py in incomplete_dir(dirname)
    489                     finally:
    490                         if os.path.exists(tmp_dir):
--> 491                             shutil.rmtree(tmp_dir)
    492
    493             # Print is intentional: we want this to always go to stdout so user has

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\shutil.py in rmtree(path, ignore_errors, onerror)
    514             # can't continue even if onerror hook returns
    515             return
--> 516         return _rmtree_unsafe(path, onerror)
    517
    518 # Allow introspection of whether or not the hardening against symlink

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\shutil.py in _rmtree_unsafe(path, onerror)
    398                 os.unlink(fullname)
    399             except OSError:
--> 400                 onerror(os.unlink, fullname, sys.exc_info())
    401     try:
    402         os.rmdir(path)

C:\Users\WTF\AppData\Local\Programs\Python\Python37\lib\shutil.py in _rmtree_unsafe(path, onerror)
    396         else:
    397             try:
--> 398                 os.unlink(fullname)
    399             except OSError:
    400                 onerror(os.unlink, fullname, sys.exc_info())

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\WTF\\.cache\\huggingface\\datasets\\csv\\default-6eb8a0ce457cdcea\\0.0.0\\2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2.incomplete\\csv-train.arrow```

Context: cc @lhoestq


Question: There is any way to create a dataset from a generator (without it being loaded into memory). Something similar to tf.data.Dataset.from_generator

Context: The datasets are not completely read into memory so you should not have to worry about memory usage. It’s mostly fast on-disk access thanks to memory mapping.


Bleurt metric throwing this error: UnrecognizedFlagError: Unknown command line flag ‘f’

Question:
Running this in Jupyter notebook and getting this error when attempting to use the bleurt metric:
UnrecognizedFlagError: Unknown command line flag 'f'

Any ideas?

Context: What’s the code you used ?


[Open-to-the-community] One week team-effort to reach v2.0 of HF datasets library

Question:
Hi all,
We are planning to do one of the biggest team effort we have ever done next week (Nov 30th to Dec 4th) to reach the v2.0 of the datasets library (Edit: final day extended to next Wednesday Dec 9th!).
The effort will involve more than half of HuggingFace (!) with about 15 people including members who’ve defined the library like @lhoestq @yjernite, @joeddav, @jplu @patrickvonplaten, members of the research team like @teven @VictorSanh and the OSS team like @Narsil, newcomers like @abhishek, awesome part-time members like @aymm and @canwenxu and many others including @madlag or yours truly. (Edit: And now over 200 external participants as well )
It will be targetted toward adding and tagging a large number of NLP datasets to the  datasets library with the goal being to reach +500 datasets and covers and organize as much of the NLP dataset eco-system as we find possible.
We are taking the occasion to develop some tools to more easily add and tag datasets in the library as well as create dataset cards for them.
After internal discussion, we have decided to open this time-limited project to external contributors if you want to have a little taste of what it is to participate in an internal HuggingFace team effort.
Basically, you can ping me or anyone of us and I will add you to the slack channel and give you access to the tools we use as well as detailed information on the workflow and a list of datasets that we think are worth adding.
There might be (Edit: “will definitely be”) a small reward as HuggingFace swag and of course sharing your contribution to this project but keep in mind that this is an open-source effort so join if you want to do an open-contribution and enjoy a bit of HuggingFace vibe, this is not an internship or work offer (for this you should check and apply on our profile on AngelList!). We expect most of the work to be done by the full-time members of HuggingFace but we are also always happy to share how we work and collaborate with external contributors which why we are opening this project.
what is it about:

we are adding a lot of new datasets to the library (in particular in many NLP tasks and we would like to have more datasets in low ressource languages as well) with the aim to cover as much ground as possible

how you can join:

post here to say that you want to participate and I will add you to our slack => That’s it

what you’ll get

enjoy a bit of HuggingFace vibe by joining the team sprint
receive a special event gift (actually 2 gifts, see this post further down the thread for details!) because it’s really amazing to see the community so involved here that we wanted to remember this event!

BIG UPDATE
We have just updated the deadline to next Wednesday (Dec 9th) So the late comers can still participate!
SECOND BIG UPDATE
A lot of people are still joining (on the way to be 300 participants ) so we are extending a bit the deadline again  — though it will a limited extension because we have to end the project at some point
 More precisely:
All the participants who will have open at least 1 PR before the end of Wednesday (Dec 9th) can continue adding additional datasets until the end of Sunday (Dec 13th) that will be counted in the sprint.
 In other word:
If you have open 1 PR before Wednesday (and thus are eligible for the special event tee-shirt goody ) you will have until the end of Sunday to add 2 others datasets if you want, and join the main-contributors channel of the slack (+ get the special event mug)
Open-sourcely yours,
Thom

Context: Love this! I’ll be too preoccupied the following weeks, but I’ll definitely join in in the future if such an event is done again!


Question:
I’m using a custom dataset from a CSV file where the labels are strings. I’m curious what the best way to encode these labels to integers would be.
Sample code:
datasets = load_dataset('csv', data_files={
    'train': 'train.csv',
    'test': 'test.csv'
    }
)

def tokenize(batch):
    return tokenizer(batch['text'], padding=True, truncation=True, max_length=128)

datasets = datasets.map(tokenize, batched=True)

datasets.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])

Context:
In your tokenize function, you can also add a line to convert your labels to ints:
def tokenize(batch):
    tokenized_batch = tokenizer(batch['text'], padding=True, truncation=True, max_length=128)
    tokenized_batch["labels"] = [str_to_int[label] for label in batch["labels"]]
    return tokenized_batch

with str_to_int your correspondence string label to int label.


Performance tips for shuffle and flatten_indices

Question:
Hey @lhoestq,
How can I speed up shuffle+flatten on a dataset with millions of instances?  It’s painfully slow for whatever setting I tried.
TIA

Context:
Hi !
By flatten you mean flattend_indices ?
Is your dataset made of strings ?
If so, then the speed bottleneck is the I/O: read from the current dataset arrow file, and write to a new file.
The new file is written when doing flatten_indices.
To speed up things you can use a SSD or distribute the writing (using shard on the shuffled dataset for example).


Question:
Hello, I am trying to download wikipedia dataset.
This is the code I try:
from datasets import load_dataset
dataset = load_dataset("wikipedia", "20200501.ang", beam_runner='DirectRunner')

Then it shows:
FileNotFoundError: Couldn’t find file at https://dumps.wikimedia.org/angwiki/20200501/dumpstatus.json 4
If I pick a recent one dump which is available from https://dumps.wikimedia.org/angwiki/ 3 :
from datasets import load_dataset
dataset = load_dataset("wikipedia", "20200620.ang", beam_runner='DirectRunner')

It shows:
ValueError: BuilderConfig 20200620.ang not found. Available: [‘20200501.aa’, ‘20200501.ab’, ‘20200501.ace’, …]
Any advice? Thank you.

Context:
Do you need English wikipedia? If so, all you need is:
dataset = load_dataset('wikipedia', "20200501.en", split='train')


Question:
Hello everyone,
Datasets provide this great feature of formatting datasets using set_format and then choosing the desired format (numpy, torch etc). The encoded dataset I prepared has columns/features of various data types (int32, int8 etc) but HF models require all features to be dtype torch.long/int64. Is there a simple trick to convert all features to torch.long tensors when selecting torch format?
I understand that I could have prepared the dataset with int64 type but that significantly increases the dataset file size footprint.
Thanks,
Vladimir

Context:
Nevermind, I found a way. RTFM.
format = {'type': 'torch', 'format_kwargs' :{'dtype': torch.long}}
dataset.set_format(**format)


Compressing, saving, and loading datasets

Question:
Hello everyone,
I am working with large datasets (Wikipedia), and use map transform to create new datasets. The workflow involves creating new datasets that are saved using save_to_disk, and subsequently, I use terminal compression utils to compress the dataset folder. Then I decompress these files and the use load_from_disk to load them on other machines. These manual steps are pita.

It would be great to use compression within datasets and have one compressed file as a result of save_to_disk if so desired.

If I could save these datasets to a remote location immediatelly bypassing save_to_disk, compression, copy manual steps that would be amazing.

Loading these created datasets via URL from s3, gs, etc. via a single load_dataset call would be a killer.

All the best,
Vladimir

Context: I agree that would be super cool to be able to archive and save/load archived dataset from/to a cloud storage. We’re thinking about this actively. Do you think that some dataset versioning logic could be interesting as well ?


Question:
Hey @lhoestq,
I am preparing datasets for BERT pre-training and often save_to_disk simply dies without saving the contents of the files to disk. The largest file I was able to save was 18 GB but above that, I am having no luck.
Any performance tips for dealing with large datasets? Should I simply shard before saving to disk? If I do that, then I get copies of 18 GB files in each shard’s directory. What are my options?
Forgot to mention. I am already using num_proc and larger batches to speed up dataset map invocations. Those work great. It’s the save_to_disk that I am not sure how to deal with. And sharding without additional copies of the underlying dataset being copies in all shard directories.
Thanks in advance.

Context:
Hi !
Is there an error message ?


How to deal with unpickable objects in map

Question:
During the creation of my dataset I would like to add sent2vec representations of input sentences to the dataset. The code would look like this:
import sent2vec
from datasets import load_dataset

sent2vec_model = sent2vec.Sent2vecModel()
sent2vec_model.load_model(sent2vec_path, inference_mode=True)

datasets = load_dataset("text", data_files={"train": train_f, "validation": valid_f})

def preprocess(sentences):
    embedded_sents = sent2vec_model.embed_sentences(sentences["text"])
    return {"text": sentences["text"], "embeddings": embedded_sents}

datasets.map(preprocess, batch_size=None, batched=True)

Unfortunately this won’t work as the sent2vec model can’t be pickled (it seems), and the fingerprint generation thus fails. At first I thought the issue was that map uses multiprocessing by default but using num_proc=1 does not help either. From the error trace it seems that the error arises during the fingerprint/hash update when the sent2vec model is trying to pickled.
File "/mnt/c/dev/python/neural-fuzzy-repair/nfr/finetuning.py", line 48, in create_datasets
    datasets.map(preprocess, batch_size=None, batched=True)
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/dataset_dict.py", line 283, in map
    {
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/dataset_dict.py", line 284, in <dictcomp>
    k: dataset.map(
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1240, in map
    return self._map_single(
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 156, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/fingerprint.py", line 157, in wrapper
    kwargs[fingerprint_name] = update_fingerprint(
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/fingerprint.py", line 105, in update_fingerprint
    hasher.update(transform_args[key])
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/fingerprint.py", line 57, in update
    self.m.update(self.hash(value).encode("utf-8"))
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/fingerprint.py", line 53, in hash
    return cls.hash_default(value)
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/fingerprint.py", line 46, in hash_default
    return cls.hash_bytes(dumps(value))
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 367, in dumps
    dump(obj, file)
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 339, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/dill/_dill.py", line 446, in dump
    StockPickler.dump(self, obj)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 485, in dump
    self.save(obj)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/dill/_dill.py", line 1435, in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 690, in save_reduce
    save(args)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 899, in save_tuple
    save(element)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 884, in save_tuple
    save(element)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/dill/_dill.py", line 1170, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 690, in save_reduce
    save(args)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 884, in save_tuple
    save(element)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 601, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 715, in save_reduce
    save(state)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/bram/.local/share/virtualenvs/neural-fuzzy-repair-b49KnSNp/lib/python3.8/site-packages/dill/_dill.py", line 933, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 969, in save_dict
    self._batch_setitems(obj.items())
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 995, in _batch_setitems
    save(v)
  File "/home/bram/.pyenv/versions/3.8.6/lib/python3.8/pickle.py", line 576, in save
    rv = reduce(self.proto)
  File "stringsource", line 2, in sent2vec.Sent2vecModel.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

Is there any way around this? For instance by completely disabling the fingerprinting?

Context:
Oh interesting, thanks Bram.
Yes I guess in this case we would have to disable the fingerprinting, right @lhoestq?
Which is a bit too bad because in the future we would have liked to leverage the fingerprint to allow the user to have a super robust reproducibility setup but  python (+cython in this case) will always be what it is (aka a huge open field).


Question:
Hello everyone,
I have a datasets.Dataset created with the following features and size.
Dataset(features: {'embeddings': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None), 'pid': Value(dtype='int32', id=None)}, num_rows: 8841823)
I added a FAISS index over ‘embeddings’ column in the GPU.
docs.load_faiss_index('embeddings', DATAPATH_INDEXED_ENCODED_PASSAGES, device=torch.cuda.current_device())
RuntimeError: Error in void faiss::gpu::allocMemorySpaceV(faiss::gpu::MemorySpace, void**, size_t) at gpu/utils/MemorySpace.cpp:26: Error: 'err == cudaSuccess' failed: failed to cudaMalloc 27162080256 bytes (error 2 out of memory)
I am working on a NVIDIA Tesla K80 with 1 GPU having 11.4GB memory. Well, it seems that with this hardward is impossible to index it on the GPU. Running the index on the CPU leads to infeasible execution time on the next step since I would like to get the top 1000 nearest examples for 6900 queries.
Any workarounds? I’m not sure if I’ll have the resources, but if I get access to several GPUs like the one that I’m working with, I just need to specify multiple GPU IDs when loading the faiss index, right?

Context: Well, it seems that you are simply running out if memory. I assume that you either have a very large index or you also have a model on the GPU at the same time?


Extend load_from_disk and save_to_disk to remote storage

Question:
Hello everyone,
As datasets are sliced, diced, and transformed it makes sense to load/save them from disk for future use. But why just local disk when this load/save can be extended to any remote storage (S3, GS etc)?
It would be great to have remote storage load/save added in the next release
Many thanks

Context:
You’re right that would be cool
Thanks for the suggestion
We think it could also be useful to have some sort of versioning for hosted datasets. Do you feel like it could be something you’d be interested to have ?


Question:
Hey guys,
After dataset map transformation, I have a new dataset with the following features:
{‘data’: {‘is_random_next’: Value(dtype=‘bool’, id=None),
‘tokens_a’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None),
‘tokens_b’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None)}}
The data column is useless so after invoking flatten, I get:
{‘data.is_random_next’: Value(dtype=‘bool’, id=None),
‘data.tokens_a’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None),
‘data.tokens_b’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None)}
which is exactly what I want. After renaming columns, I am done.
Since map forces us to return dict, I need to wrap a list of dict values (is_random_next, tokens_a, tokens_b) with dict to comply, so I did. How could I avoid that?

Context: Quentin @lhoestq I got the wikipedia/bookcorpus dataset processing to be super fast and everything works as advertised. I was wondering if I could somehow bypass the kludge that I have now  for dataset flattening - I am really curious if it could be done without it?


Question:
The current book corpus dataset is parsed into sentences directly, which is great, but then there is no way to determine document boundaries. Would it be useful to have another bookcorpus dataset that is chunked into books rather than sentences directly?
Shawn Presser went to great lenghts to preserve the structure of the books’ text and it is available at https://github.com/soskek/bookcorpus/issues/27 25 for download.

Context:
Indeed ! It was already suggested in https://github.com/huggingface/datasets/issues/486 28 to use this link. It would be very cool to add it to the library. You can make a script to use the new link if you want. You can take some inspiration from the docs 28 and from the current bookcorpus 11 script.
Let me know if you have questions, you can ping me on the forum or on github