Need permission to use dataset for blog post - who to reach out to?

Question: Hi all - I would like to use this dataset 1 to write a blog post about training an NLP model to identify Adverse Drug Events. The publication I want to publish at (https://towardsdatascience.com/) asked me to get permission to use this dataset. Who do I need to ask to use this dataset? Thanks, Heiko

Context: The citation information is right there. So if you click only a bit further you can find the authors’ email addresses in their paper.


Question:
Hi,
How does the map function’s cache behave if I pass in a function with partial into map? I’m passing in a function that takes in more arguments and changes behavior according to it which is wrapped with the partial function.

Context:
Hi ! partial-wrapped functions are supported
If any of the partial argument changes, the map function will re-compute the result.


Help understanding how to build a dataset for language as with the old TextDataset

Question:
Hello,
I am trying to load a custom dataset that I will then use for language modeling. The dataset consists of a text file that has a whole document in each line, meaning that each line overpasses the normal 512 tokens limit of most tokenizers.
I would like to understand what is the process to build a text dataset that tokenizes each line, having previously split the documents in the dataset into lines of a “tokenizable” size, as the old TextDataset 49 class would do, where you only had to do the following, and a tokenized dataset without text loss would be available to pass to a DataCollator:
model_checkpoint = 'distilbert-base-uncased'

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="path/to/text_file.txt",
    block_size=512,
)

For now, what I have is the following, which, of course, throws an error/warning because each line is longer than the maximum block size in the tokenizer:
import datasets
dataset = datasets.load_dataset('path/to/text_file.txt')

model_checkpoint = 'distilbert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

def tokenize_function(examples):
    return tokenizer(examples["text"])

tokenized_datasets = dataset.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])

tokenized_datasets

So what would be the “standard” way of creating a dataset in the way it was done before?
Thank you very much for the help :))

Context:
Hi !
If you want to tokenize line by line, you can use this:
max_seq_length = 512
num_proc = 4

def tokenize_function(examples):
    # Remove empty lines
    examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=max_seq_length,
    )

tokenized_dataset = dataset.map(
    tokenize_function,
    batched=True,
    num_proc=num_proc,
    remove_columns=["text"],
)

Though the TextDataset was doing a different processing by concatenating all the texts and building blocks of size 512. If you need this behavior, then you must apply an additional map function after the tokenization:
# Main data processing function that will concatenate all texts from
# our dataset and generate chunks of max_seq_length.
def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop,
    # you can customize this part to your needs.
    total_length = (total_length // max_seq_length) * max_seq_length
    # Split by chunks of max_len.
    result = {
        k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]
        for k, t in concatenated_examples.items()
    }
    return result

# Note that with `batched=True`, this map processes 1,000 texts together,
# so group_texts throws away a remainder for each of those groups of 1,000 texts.
# You can adjust that batch_size here but a higher value might be slower to preprocess.

tokenized_dataset = tokenized_dataset.map(
    group_texts,
    batched=True,
    num_proc=num_proc,
)

This code comes from the processing of the run_mlm.py 17 example script of transformers


Loading custom dataset without cache - using load script

Question:
I need to use IITCDIP dataset on a server where I get 6 hours time slot. The data_loader is taking more than 6 hours for caching. So, I can do my training. Is there any way:

load custom dataset with caching (Stream) using script similar to here 2.

Resume the caching process

Cache dataset on one system and use on other system.

Note that I have tried up to 64 num_proc  but did not get any speed up in caching processing.

Context:
You can first prepare your dataset on one system, then use my_dataset.save_to_disk() to save it in the directory of your choice. Then you can move this directory on another system and reload the dataset with datasets.load_from_disk()
Hope that helps !


Datasets behaving strange when calling filter twice

Question:
I think I’m seeing unexpected behavior when calling filter twice in succession. I wonder if it’s related to caching?
Here’s an easy reproduction:
from datasets import load_dataset

def debug_log(key, d):
    print(key, len(d), set(d['language']))

d = load_dataset('tydiqa', 'primary_task', split='validation')
debug_log('d', d)
d_filter_0 = d.filter(lambda x: x['language'] == 'english')
debug_log('d_filter_0', d_filter_0)
d_filter_1 = d_filter_0.filter(lambda x: x['language'] == 'english')
debug_log('d_filter_1', d_filter_1)
d_filter_2 = d.filter(lambda x: x['language'] == 'english')
debug_log('d_filter_2', d_filter_2)

Output (notice that languages for d_filter_0 != d_filter_1):
Reusing dataset tydiqa (/Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148)
d 18670 {'bengali', 'swahili', 'english', 'thai', 'telugu', 'indonesian', 'russian', 'arabic', 'finnish', 'japanese', 'korean'}
Loading cached processed dataset at /Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148/cache-1c4d9d00950a4a3d.arrow
d_filter_0 1031 {'english'}
Loading cached processed dataset at /Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148/cache-6c4a6a69baa0ff73.arrow
d_filter_1 1031 {'bengali', 'swahili', 'english', 'thai', 'telugu', 'indonesian', 'russian', 'arabic', 'finnish', 'japanese', 'korean'}
Loading cached processed dataset at /Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148/cache-1c4d9d00950a4a3d.arrow
d_filter_2 1031 {'english'}

It would be great to call filter twice! Basically, can filter the first time by language, and the second time by something else like length.

Context:
Hey @mrdrozdov instead of chaining multiple filter operations, I think you can use conditionals in the lambda function like:
dset.filter(lambda x: x['language'] == 'english' and x['language'] == 'german')


Loading community JSON based datasets without a script

Question:
Hi,
I can easily create and upload JSON community datasets to HF hub (without a script). However, when using the dataset I need to specify data_files parameter, i.e. dataset = load_dataset("vblagoje/dataset_abc", data_files={"train": "train.json", "validation": "validation.json", "test": "test.json"})  which is a bit of inconvenience.
Is there a way to avoid data_files parameters short of providing a loading script as well? If not, why don’t we rely on naming conventions to resolve the data splits anyway?
Cheers,
Vladimir

Context:
Hi ! I like the idea of using a name convention. By default it could use something like
data_files = {
    "train": ["*train*"],
    "test": ["*test*"],
    "validation": ["*dev*", "*valid*"]
}

Another alternative would be to let the user specify the data_files patterns in an additional JSON file (config.json or something like this). Let me know what you think  !


TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class ‘list’>

Question:
I am trying to transform my data to dataset format to use it with a bert tokenizer but I get this error :
raise TypeError(
TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class 'list'>. Make sure provided `function` returns a variable of type `dict` to update the dataset or `None` if you are only interested in side effects.

How can I solve it  ?
datasets library : 1.2.1
flaubert_tokenizer = FlaubertTokenizer.from_pretrained('flaubert/flaubert_small_cased')
model = FlaubertModel.from_pretrained('flaubert/flaubert_small_cased')

train_file = './corpusi1.xlsx'

Xtrain, ytrain, filename, len_labels = read_file_2(fic)
# Xtrain, lge_size = get_flaubert_layer(Xtrain, path_to_model_lge)

data_preprocessed = make_new_traindata(Xtrain)

my_dict = {"verbatim": data_preprocessed[1], "label": ytrain}
dataset = Dataset.from_dict(my_dict)

print(type(dataset))
#<class 'datasets.arrow_dataset.Dataset'>

#print(dataset)
Dataset({
    features: ['verbatim', 'label'],
    num_rows: 346
})
`

tokenized_dataset = dataset.map(lambda x: flaubert_tokenizer.encode(x['verbatim'], padding=max_length, truncation=True, max_length=512), batched=True) # this line raise the error

How can I convert dataset.arrow to dataset dict ?

Context:
Hi !
The output of the function passed to a batched map should be a dict with the structure {column_name → [list of values per element]}
Can you try this instead ?
tokenized_dataset = dataset.map(
    lambda x: {"input_ids": flaubert_tokenizer.encode(
        x['verbatim'], padding=max_length, truncation=True, max_length=512
    )},
    batched=True
)


Access denied when reading files in dataset

Question:
I have created a dataset by following the instructions. Basically I am doing this by choosing “New dataset”, cloning the repository, added the files to the local git (in my case *.json.gz) and pushing to the repo.
However, when trying to access the files in the browser I first get the message “This file is stored with [Git LFS]. It is too big to display, but you can still [download] it.” Clicking the “download” link it returns an “access denied” XML-object. This happens even if I set the status of the dataset as public.
Accessing the files with “load dataset”, gives me the exact same error.
EDIT: When my colleague in the same organisation creates a repo, it seem to work fine. However, when I clone and upload files, they are inaccessible. None of us are able to read the files that I have submitted. File permissions are identical on both the readable and the unreadable files.

Context:
Hi ! Is your dataset public ? So that I can take a look and see what’s going on.
Otherwise feel free to make a dummy public dataset that reproduces the error.
Or we can discuss this via DM if you don’t want to make the dataset public


Datasets not behaving as expected after random data augmentation with map

Question:
I’m using nlpaug 1 to augment a split of the sst2 dataset. As instructed in the documentation 1, I’m using map with batched=True for this purpose. The function I pass to map takes one instance (batch_size=1) and generates several instances. The important thing here is that this function is not a pure function, the sentence it generates and the number of instances it generates and returns are completely random. I get a warning each time that says there is a problem with caching and fingerprint, which I assumed is because of the random nature of my function.
After the data augmentation, the dataset acts weird; for example:
synonym_aug_datasets[0.2]["train"].filter(lambda x: x['idx'] > 200000)["idx"][:10]

synonym_aug_datasets is a python dictionary with several augmented datasets in it. I’m simply filtering all instances with idx larger than 200000 and then looking at the idx of the first 10 instances, all of them should have idx larger than 200000, right? Every time I run this code, I get a different result, sometimes even with idx smaller than 200000, and sometimes it won’t even run
output 1:
[55154, 55917, 200628, 409, 6218, 33825, 201639, 2063, 49115, 2959]

output 2:
ValueError: Keys mismatch: between {'indices': Value(dtype='uint64', id=None)} and {'idx': Value(dtype='int32', id=None), 'label': Value(dtype='int64', id=None), 'sentence': Value(dtype='string', id=None)}

output 3:
[200205,
 200168,
 200888,
 200157,
 201597,
 201784,
 200899,
 200466,
 201086,
 200435]

How is that even possible?

Context:
I found a solution for now. Just before tokenizing, I’m converting my datasets to pandas data frame and converting them back to datasets. By doing this, the datasets library doesn’t load the same cache each time and recognizes that my datasets are different.
new_datasets = DatasetDict({
    "train": Dataset.from_pandas(old_datasets["train"].to_pandas()),
    "validation": Dataset.from_pandas(old_datasets["validation"].to_pandas()),
    "test": Dataset.from_pandas(old_datasets["test"].to_pandas()),
})


How to split main dataset into train, dev, test as DatasetDict

Question:
It seems that a single dataset can be split up into different partitions but in such a way that the connection between them is still clear (by using a DatasetDict), which is neat. I am having difficulties trying to figure out how I can create them, and use them, though. I’ve been going through the documentation [1] 19,[2] 14 and the source code [1] 8,[2] 5 but it hasn’t become any clearer.
In some parts you speak of only a train, test split 50 other times you include validation 12. It is not clear how to split an existing Dataset into a DatasetDict containing train, dev (validation), and test keys, even though I think this is a common scenario.
The focus of the library seems to be on the use of existing datasets (which is a great feature!), but time permitting I would like to see more information and use cases for creating and managing your own datasets. I’m usually pretty good at figuring out a code base by its documentation and source code (e.g. transformers), but dataset seems to give me a harder time. That is not to say that I think it is bad, of course. But it does show, I think, that even though it seems simple from the examples, it is actually quite complex behind the scenes. For people who want do custom stuff it is then super important to have clear, verbose documentation about as many aspects of the library as possible.

Context:
Hi Bram,
Yes the documentation of train_test_split that you link to is the right one. The train_test_split method currently provided is just a copy of the famous sklearn train_test_split (that we kinda assume people to be familiar with), we just removed the stratified split options which are quite complex.
We could add an option to split in three with a validation split indeed, feel free to open a PR on this if you would like to have this feature fast.
Right now what you can do is splitting two times:
# 90% train, 10% test + validation
train_testvalid = dataset.train_test_split(test=0.1)
# Split the 10% test + valid in half test, half valid
test_valid = train_test_dataset['test'].train_test_split(test=0.5)
# gather everyone if you want to have a single DatasetDict
train_test_valid_dataset = DatasetDict({
    'train': train_testvalid['train'],
    'test': test_valid['test'],
    'valid': test_valid['train']})

The mention of a validation split that you point to is just an enum provided for dataset creators who would like to include a standard name for a validation split in there dataset.
We will try to add more doc on the code organization but bear in mind that (1) this library is still very young and (2) we are a lot less working on it (it’s really mostly one person, Quentin, that I try to help as much as I can), so it will definitely take some time before we have “have clear, verbose documentation about as many aspects of the library as possible”.
Basically, to give you an idea, the code is organized in two main parts:

the dataset building part which is defined in part by the people writing datasets and very open (hence the many options for splits in this part) => this is most of the complex code because it a wrapper around script provided externally. This includes files like builder.py, load.py, arrow_dataset.py.
the dataset processing part (after the dataset has been build) which is mostly contained in the arrow_dataset.py file and contains most of what the users will actually interact with => this is probably the part you need to read the most. The main complex part here is that we are deeply integrated with Apache Arrow which is very efficient but definitely not the easiest framework to understand.

You can also read this part in the doc where I tried to make a graph and give some information on how datasets are created (the first part in my list above): https://huggingface.co/docs/datasets/add_dataset.html 43


Question:
Hi,
It appears that the code in this example 1 returns NotImplementedError stating that Gleu is currently under construction. The last update was about a year ago.
I would like to know what is required for this metric to work from within datasets? Perhaps, I could make a pull request with necessary changes.

Context:
Hi ! I think someone started working on it but they stopped pretty early in the implementation.
The current code is here 2 and it appears that the _compute method is incomplete/not working. This method is supposed to return a dictionary of metrics (for example {"gleu_mean": mean, "gleu_std": std, "gleu_ci": ci}) from the list of references and predictions.
If you’re interested in contributing, feel free to try to make this method work and create a PR
Locally you can test your metric with
from datasets import load_metric

gleu = load_metric("path/to/gleu.py")
gleu.compute(refererences=references, prefictions=predictions)


Dataset map preprocess throws ArrowInvalid

Question:
I’m using wav2vec2 for emotion classification (following @m3hrdadfi’s notebook). In the dataset preprocessing step using .map(), it throws an error, and I’m not sure what is triggering it in the first place. So, any pointer resolving it would be much appreciated. Thanks!
(also, gently pinging @lhoestq and @patrickvonplaten)

Code Reference:
# Loading the created dataset using datasets
from datasets import load_dataset, load_metric

data_files = {
    "train": "/content/data/train.csv",
    "validation": "/content/data/test.csv",
}

dataset = load_dataset("csv", data_files=data_files, delimiter="\t", )
train_dataset = dataset["train"]
eval_dataset = dataset["validation"]

# We need to specify the input and output column
input_column = "path"
output_column = "emotion"

# we need to distinguish the unique labels in our SER dataset
label_list = train_dataset.unique(output_column)
label_list.sort()  # Let's sort it for determinism
num_labels = len(label_list)
# print(f"A classification problem with {num_labels} classes: {label_list}")

from transformers import AutoConfig, Wav2Vec2Processor

model_name_or_path = "arijitx/wav2vec2-large-xlsr-bengali"
pooling_mode = "mean"

# config
config = AutoConfig.from_pretrained(
    model_name_or_path,
    num_labels=num_labels,
    label2id={label: i for i, label in enumerate(label_list)},
    id2label={i: label for i, label in enumerate(label_list)},
    finetuning_task="wav2vec2_clf",
)
setattr(config, 'pooling_mode', pooling_mode)

processor = Wav2Vec2Processor.from_pretrained(model_name_or_path,)
target_sampling_rate = processor.feature_extractor.sampling_rate
# print(f"The target sampling rate: {target_sampling_rate}")

"""So far, we downloaded, loaded, and split the SER dataset into train and test sets. The instantiated our strategy configuration for using context representations in our classification problem SER. Now, we need to extract features from the audio path in context representation tensors and feed them into our classification model to determine the emotion in the speech.

Since the audio file is saved in the `.wav` format, it is easy to use **[Librosa](https://librosa.org/doc/latest/index.html)** or others, but we suppose that the format may be in the `.mp3` format in case of generality. We found that the **[Torchaudio](https://pytorch.org/audio/stable/index.html)** library works best for reading in `.mp3` data.

An audio file usually stores both its values and the sampling rate with which the speech signal was digitalized. We want to store both in the dataset and write a **map(...)** function accordingly. Also, we need to handle the string labels into integers for our specific classification task in this case, the **single-label classification** you may want to use for your **regression** or even **multi-label classification**.
"""

def speech_file_to_array_fn(path):
    speech_array, sampling_rate = torchaudio.load(path)
    resampler = torchaudio.transforms.Resample(sampling_rate, target_sampling_rate)
    speech = resampler(speech_array).squeeze().numpy()
    return speech

def label_to_id(label, label_list):

    if len(label_list) > 0:
        return label_list.index(label) if label in label_list else -1

    return label

def preprocess_function(examples):
    speech_list = [speech_file_to_array_fn(path) for path in examples[input_column]]
    target_list = [label_to_id(label, label_list) for label in examples[output_column]]

    result = processor(speech_list, sampling_rate=target_sampling_rate)
    result["labels"] = list(target_list)

    return result

train_dataset = train_dataset.map(
    preprocess_function,
    batch_size=8,
    batched=True,
    # num_proc=4
)
eval_dataset = eval_dataset.map(
    preprocess_function,
    batch_size=8,
    batched=True,
    # num_proc=4
)

Error Reference:
/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-47-fc0bb5a1a9e1> in <module>()
      2     preprocess_function,
      3     batch_size=8,
----> 4     batched=True,
      5     # num_proc=4
      6 )

13 frames
/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1684                 new_fingerprint=new_fingerprint,
   1685                 disable_tqdm=disable_tqdm,
-> 1686                 desc=desc,
   1687             )
   1688         else:

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    183         }
    184         # apply actual function
--> 185         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    186         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    187         # re-apply format to the output

/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc)
   2036                             else:
   2037                                 batch = cast_to_python_objects(batch)
-> 2038                                 writer.write_batch(batch)
   2039                 if update_data and writer is not None:
   2040                     writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    401             typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)
    402             typed_sequence_examples[col] = typed_sequence
--> 403         pa_table = pa.Table.from_pydict(typed_sequence_examples)
    404         self.write_table(pa_table, writer_batch_size)
    405

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_pydict()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._handle_arrow_array_protocol()

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in __arrow_array__(self, type)
    105                 out = numpy_to_pyarrow_listarray(self.data)
    106             else:
--> 107                 out = pa.array(self.data, type=type)
    108             if trying_type and out[0].as_py() != self.data[0]:
    109                 raise TypeError(

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Can only convert 1-dimensional array values

Context:
Hi ! What version of datasets are you using ?
Also do you know what kind of arrays the Wav2vec2Processor returns ? Ideally it should be numpy arrays


Help Finding Dataset with Increasing Vocab Size and/or Reading Difficulty

Question: Can anyone point me towards any corpus with a natural increase in the vocabulary size and/or reading difficulty? I’m thinking of something akin to how children and young adult books come with estimated grade levels (e.g. “5th grading reading level”).

Context: Have you considered data from CommonLit Readability Prize 1 Kaggle competition? I guess it doesn’t exactly fit your description but it’s kind of similar.


Loading div2k from super-image into Pytorch

Question:
I am following this page 6 to load div2k from super-image, and I want to convert it so that I can train my model built in PyTorch.
To do that I am following Using a Dataset with PyTorch/Tensorflow — datasets 1.11.0 documentation 3. However, I got an error
TypeError: new(): invalid data type 'numpy.str_'

My code is
from datasets import load_dataset
from super_image.data import EvalDataset, TrainDataset, augment_five_crop
import torch

augmented_dataset = load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='train')\
    .map(augment_five_crop, batched=True, desc="Augmenting Dataset")                                # download and augment the data with the five_crop method
train_dataset = TrainDataset(augmented_dataset)                                                     # prepare the train dataset for loading PyTorch DataLoader
eval_dataset = EvalDataset(load_dataset('eugenesiow/Div2k', 'bicubic_x4', split='validation'))      # prepare the eval dataset for the PyTorch DataLoader

augmented_dataset.set_format(type='torch', columns= ["lr", "hr"])
augmented_dataset.__getitem__(0)

and the full error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-5b6b28c36bc5> in <module>
     12
     13 augmented_dataset.set_format(type='torch', columns= ["lr", "hr"])
---> 14 augmented_dataset.__getitem__(0)

~\anaconda3\lib\site-packages\datasets\arrow_dataset.py in __getitem__(self, key)
   1515     def __getitem__(self, key: Union[int, slice, str]) -> Union[Dict, List]:
   1516         """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 1517         return self._getitem(
   1518             key,
   1519             format_type=self._format_type,

~\anaconda3\lib\site-packages\datasets\arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1508         formatter = get_formatter(format_type, **format_kwargs)
   1509         pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 1510         formatted_output = format_table(
   1511             pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   1512         )

~\anaconda3\lib\site-packages\datasets\formatting\formatting.py in format_table(table, key, formatter, format_columns, output_all_columns)
    420     else:
    421         pa_table_to_format = pa_table.drop(col for col in pa_table.column_names if col not in format_columns)
--> 422         formatted_output = formatter(pa_table_to_format, query_type=query_type)
    423         if output_all_columns:
    424             if isinstance(formatted_output, MutableMapping):

~\anaconda3\lib\site-packages\datasets\formatting\formatting.py in __call__(self, pa_table, query_type)
    192     def __call__(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
    193         if query_type == "row":
--> 194             return self.format_row(pa_table)
    195         elif query_type == "column":
    196             return self.format_column(pa_table)

~\anaconda3\lib\site-packages\datasets\formatting\torch_formatter.py in format_row(self, pa_table)
     57     def format_row(self, pa_table: pa.Table) -> dict:
     58         row = self.numpy_arrow_extractor().extract_row(pa_table)
---> 59         return self.recursive_tensorize(row)
     60
     61     def format_column(self, pa_table: pa.Table) -> "torch.Tensor":

~\anaconda3\lib\site-packages\datasets\formatting\torch_formatter.py in recursive_tensorize(self, data_struct)
     53
     54     def recursive_tensorize(self, data_struct: dict):
---> 55         return map_nested(self._recursive_tensorize, data_struct, map_list=False)
     56
     57     def format_row(self, pa_table: pa.Table) -> dict:

~\anaconda3\lib\site-packages\datasets\utils\py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)
    202         num_proc = 1
    203     if num_proc <= 1 or len(iterable) <= num_proc:
--> 204         mapped = [
    205             _single_map_nested((function, obj, types, None, True))
    206             for obj in utils.tqdm(iterable, disable=disable_tqdm)

~\anaconda3\lib\site-packages\datasets\utils\py_utils.py in <listcomp>(.0)
    203     if num_proc <= 1 or len(iterable) <= num_proc:
    204         mapped = [
--> 205             _single_map_nested((function, obj, types, None, True))
    206             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    207         ]

~\anaconda3\lib\site-packages\datasets\utils\py_utils.py in _single_map_nested(args)
    141     # Singleton first to spare some computation
    142     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 143         return function(data_struct)
    144
    145     # Reduce logging to keep things readable in multiprocessing with tqdm

~\anaconda3\lib\site-packages\datasets\formatting\torch_formatter.py in _recursive_tensorize(self, data_struct)
     50             if data_struct.dtype == np.object:  # pytorch tensors cannot be instantied from an array of objects
     51                 return [self.recursive_tensorize(substruct) for substruct in data_struct]
---> 52         return self._tensorize(data_struct)
     53
     54     def recursive_tensorize(self, data_struct: dict):

~\anaconda3\lib\site-packages\datasets\formatting\torch_formatter.py in _tensorize(self, value)
     42             default_dtype = {"dtype": torch.float32}
     43
---> 44         return torch.tensor(value, **{**default_dtype, **self.torch_tensor_kwargs})
     45
     46     def _recursive_tensorize(self, data_struct: dict):

TypeError: new(): invalid data type 'numpy.str_'

The same error is thrown if I use PyTorch DataLoader
my_ldr = torch.utils.data.DataLoader(augmented_dataset, batch_size=1, shuffle=False)
next(iter(my_ldr))

Context:
Hi ! What are the feature types of your dataset ?
print(augmented_dataset.features)

The ‘torch’ formats only work for columns of numeric data, since strings can’t be converted to pytorch tensors. Could it be that one of “lr” and “hr” has text data ?


Question:
Hey guys,
How do I properly encode/format json file dump (or use any other approach for creating JSON files) so that the created JSON file is easily digested by load_dataset JSON variant as described in the docs 2?
TIA,
Vladimir

Context:
Hi ! You can simply use .to_json() - see documentation here 3
Here is an example using SQuAD:
from datasets import load_dataset

squad = load_dataset("squad", split="train")
squad.to_json("squad.json")

data_files = {"train": "squad.json"}
re_squad = load_dataset("json", data_files=data_files, split="train")

This creates a JSON Lines file, then it reloads it using the JSON dataset loader


Question:
What would be the recommended usage of datasets given I have large dataset e.g. common crawl, and need distributed training? For example, is there a build-in functionality that I could preprocess the data once and save/load in disk in a binarized/efficient way? And is there anything worth noticing for efficient distributed training with large datasets?
Tried to go over the doc but didn’t find anything on this.
Thanks!

Context:
Hi ! Sure the datasets library is designed to support the processing of large scale datasets. Datasets are loaded using memory mapping from your disk so it doesn’t fill your RAM. You can parallelize your data processing using map since it supports multiprocessing. Then you can save your processed dataset using save_to_disk, and reload it later using load_from_disk
from datasets import load_dataset, load_from_disk

dataset = load_dataset(...)
dataset = dataset.map(..., num_proc=num_processes)
dataset.save_to_disk("path/to/save/directory")

# later
dataset = load_from_disk("path/to/save/directory")

Things worth noticing:

you can specify a cache_dir parameter in load_dataset so that you can store the raw + prepared data wherever you want and be able to delete them later to save space if needed.
If you are working on a cluster with a virtual filesystem, you may want to make sure that the memory mapping works efficiently. This is probably the case if you are doing distributed training. There is a discussion about this in here 21 if this is the case. We are still investigating why some virtual filesystems have such behaviors.


Question:
I’m currently making use of gcsfuse to load data from arrow tables stored in a Google Cloud Storage bucket mounted to a vm folder. The tables were saved automatically when the map() function cached the datasets to the mounted folder which is all well and good but gcsfuse often struggles with large datasets (io errors, sluggish, etc.).
Is there a way of using the equivalent of save_to_disk for the map() function’s caching? That way I could use the s3 filesystem to load the tables in a more stable fashion.

Context:
Hi ! When do you experience slow downs or error exactly ?
Have you tried making map write to a local directory first (you can pass cache_file_name to map to specify where to write the Arrow data), and then save it to your cloud storage using save_to_disk ?
This way accessing the data from your processed dataset will be much faster since your dataset will be on your disk.


Problems with Dataset.from_dict() and Feature types

Question:
I am building the training pipeline for a Distilbert and am trying to define the Feature types for a Dataset that is loaded from a dictionary.
This dictionary is actually the input_ids, labels and attention_mask fields that the tokenizer returns, and I can’t seem to achieve the correct data assignation, specially the label feature. Here is an example:
data = {
    'input_ids': [[101, 1, 2, 3, 102], [101, 11, 12, 13, 102]],
    'labels': [[-100, 'O', 'B-NOUN', 'L-NOUN', -100], [-100, 'U-NOUN', 'O', 'O', -100]],
    'attention_mask': [[0, 1, 1], [1, 0, 0]]
}

dt = Dataset.from_dict(data)

The problem comes because:

They are Sequences of values, and I am not sure how to build a Sequence[ClassLabel] feature and hence, the above code crashes on .from_dict(data) as it expect int not str.
If I map the label names previously to the .from_dict(data) (such as the code after those lines) it is able to generate the dataset without problem, but later on creates a strange behavior in the training and prediction time for the model. Here Distilbert just builds its own tag2id mapping such as {0: ‘LABEL_0’, 1: ‘LABEL_1’, 2:‘LABEL_2’} that I am not sure how to fix (and this generates problems whith the aggregation strategy on prediction time.

Example of mapped data such as explained in point 2:
data = {
    'input_ids': [[101, 1, 2, 3, 102], [101, 11, 12, 13, 102]],
    'labels': [[-100, 1, 2, 3, -100], [-100, 4, 1, 1, -100]],
    'attention_mask': [[0, 1, 1], [1, 0, 0]]
}

dt = Dataset.from_dict(data)

Any idea how to correctly generate the dataset and avoid that problem with Distilbert?
UPDATE: Decided to try another road and it is basically building first the dataset and  then tokenizing, but the problem I got right now is that using the function align_labels_with_mapping crashes when the feature is a Sequence, and casting the feature also was not able to cast the values inside the Sequence object.
What am I doing wrong?

Context:
Hi ! You can define a Sequence of ClassLabel with
Sequence(ClassLabel(names=...))

but in this case you need your data to be integers, not strings.
You can convert your data to integers with
data = features.encode_batch(data)

and then get your dataset:
dt = Dataset.from_dict(data, features=features)

Regarding the label2id of DistilBERT, you can define your own by specifying the id2label parameter of the model config when you load your model with .from_pretrained(..., id2label=...)

 zhcr:

the function align_labels_with_mapping crashes when the feature is a Sequence, and casting the feature also was not able to cast the values inside the Sequence object.

Could you open an issue on GitHub so that we can take a look into what went wrong ?
If you have a code snippet that reproduces the error that would be of great help as well.


Using load_datasets for newly created datasets

Question:
Hi all
I am created a new public datasets here (Sam2021/Arguement_Mining_CL2017 · Datasets at Hugging Face 2) and I have added loading script with link to actual data in github.
But I am not able to load the data remotely using
data = load_datasets('Sam2021/Arguement_Mining_CL2017)
Can anyone please let me know what I am doing wrong ?

Context:
Hi ! If your data is in the dataset repository, then you can just pass the name of the data files to be downloaded:
_TRAINING_FILE = "train.txt"
_TEST_FILE = "test.txt"

    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        urls_to_download = {"train": _TRAINING_FILE, "test": _TEST_FILE}
        downloaded_files = dl_manager.download_and_extract(urls_to_download)
        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": downloaded_files["train"]}),
            datasets.SplitGenerator(name=datasets.Split.TEST, gen_kwargs={"filepath": downloaded_files["test"]}),
        ]

Under the hood it will download it from an URL like this one:
https://huggingface.co/datasets/Sam2021/Arguement_Mining_CL2017/resolve/main/train.txt


[urgent]Can you reconstruct datasets using the cache file (.arrow file)?

Question:
Hi,
Basically I construct a dataset through the mapping function in jupyter notebook. I can find the cache file (.arrow). Is there a way I can reconstruct the dataset similar to “datasets.load_from_disk()”
Screen Shot 2021-01-02 at 2.59.22 PM746×120 16.3 KB

Context:
Sure ! If you only have the .arrow file you can do
from datasets import Dataset

dataset = Dataset.from_file(path_to_arrow_file)


Passing schema features to a load_dataset function

Question:
Hi.
I am trying to load a loca csv into a Hugging Face dataset, but I have this error. I can’t understand what is wrong. I am using dataset 1.6.2
This is the code:

And this is the error I have:

image1147×768 128 KB

Tha error says `Target schema’s field names are not matching the table’s field names: [‘text’, ‘labels’, ‘offerid’], [‘labels’, ‘offerid’, ‘text’]. It doesn’t make any sense for me, because the fields name matches.
Any help would be welcome.

Context: @Oigres Hi, Probably this link 4 might help you, let me know if you have already tried.


Question:
Hi everyone.
After creating a dataset consisting of all my data, I split it in train/validation/test sets. Following that, I am performing a number of preprocessing steps on all of them, and end up with three altered datasets, of type datasets.arrow_dataset.Dataset.
In order to save them and in the future load directly the preprocessed datasets, would I have to call
dataset.save_to_disk(FILE_PATH)
3 times, one for the training, one for the validation and one for the test set? Or is there any way to somehow save them all together? If yes, what is more efficient?
Thanks in advance.

Context:
Hi !
You can save them all as a dataset dictionary:
from datasets import DatasetDict, load_from_disk

dataset = DatasetDict({
    "train": train_dataset,
    "validation": validation_dataset,
    "test": test_dataset,
})

dataset.save_to_disk("path/to/dataset/dir")

# reload
dataset = load_from_disk("path/to/dataset/dir")

# access any split
train_dataset = dataset["train"]

This is especially useful to save several splits of a dataset together.


Allow streaming of large datasets with image/audio

Question:
Hi,
I’m trying to recreate the OpenAI subset of YFCC100M and since the dataset is very large, I’d like to push it to the hub and allow streaming.
I was seeing it as:

large metadata file (containing title, description, file, path): I had planned a JSON but I guess I should use a tsv or csv (because a JSON probably needs to be read entirely to be parsed?)
an archive with all images: it seems that it needs to be a zip file

Would streaming be feasible? Is there a loading script I can get inspiration from?

Context: pinging @lhoestq here


Question:
My office PC doesn’t have access to internet, and the load_metric function downloads the metric from internet. I tried pickling using the following code:
PC 1 (connected to internet)
import pickle
from datasets import load_metric
metric = load_metric("glue", "mrpc")
with open('metric.pickle', 'wb') as handle:
    pickle.dump(metric, handle, protocol=pickle.HIGHEST_PROTOCOL)

Now i copied the metric pickle file to the offline PC, and tried opening using the followng code:
import pickle
with open('metric.pickle', 'rb') as handle:
    metric = pickle.load(handle)

but it gived the following error:
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ModuleNotFoundError: No module named 'datasets_modules'

Is there way to use datasets load_metric in offline mode?
Thank you in advance.

Context:
Hi ! Yes you just need to have the metric python file from datasets/metrics at master · huggingface/datasets · GitHub 26 and load it with
load_metric("path/to/metric.py")


Question:
I have a dataset that is multi-label in nature. The explanatory feature is an image. And the labels are the target.
An example of the classes:
classes = [‘Smears’, ‘Loaded Language’, ‘Name calling/Labeling’, ‘Glittering generalities (Virtue)’,
‘Appeal to (Strong) Emotions’, ‘Appeal to fear/prejudice’, ‘Transfer’, ‘Doubt’,
‘Exaggeration/Minimisation’, ‘Whataboutism’, ‘Slogans’, ‘Flag-waving’,
“Misrepresentation of Someone’s Position (Straw Man)”, ‘Causal Oversimplification’,
‘Thought-terminating cliché’, ‘Black-and-white Fallacy/Dictatorship’, ‘Appeal to authority’,
‘Reductio ad hitlerum’, ‘Repetition’, ‘Obfuscation, Intentional vagueness, Confusion’,
‘Presenting Irrelevant Data (Red Herring)’, ‘Bandwagon’]
I wanted to setup using this pattern: https://theaisummer.com/hugging-face-vit/
The example uses datasets.Dataset from hugginface. I wanted to replicate this same example using my data found here: SEMEVAL-2021-task6-corpus/data at main · di-dimitrov/SEMEVAL-2021-task6-corpus · GitHub.
I loaded my data from the json file:
with open(’./data/training_set_task3/training_set_task3.json’, encoding=‘utf-8’) as f: ### Training data
training = json.load(f)
df_training = pd.DataFrame(training)
I use the multilabelbinerizer to convert my multi-labels to one hot encodings, load my images and an example of the dataset is below:

id
labels
text
image
img
label

0
128
[Black-and-white Fallacy/Dictatorship, Name ca…
THERE ARE ONLY TWO GENDERS\n\nFEMALE \n\nMALE\n
128_image.png
[[[255, 255, 255], [255, 255, 255], [255, 255,…
[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, …

1
189
[Reductio ad hitlerum, Smears, Transfer]
This is not an accident!
189_image.png
[[[11, 15, 64], [11, 15, 64], [11, 15, 64], [1…
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …

After subsetting my dataset here is the modeling dataset.

image748×287 7.91 KB

df_train = df_train.rename(columns={‘label’:‘labels’})
features = Features({
‘labels’: ClassLabel(
names= classes),
‘img’: Array3D(dtype=“int64”, shape=(3, 32, 32)),
‘pixel_values’: Array3D(dtype=“float32”, shape=(3, 224, 224)),
})
preprocessed_train_ds = dataset_training.map(preprocess_images, batched=True, batch_size = 8, features=features)
This last statement generates an error because I failed to account the example is using a single label. I was unable to find an equivalent usage for multi-labels. Is this feature available? Is it coming? Any recommendations for how to address using this pattern? Any ideas would be greatly appreciated. Worst case, I can use this pattern and go back to the other methods of processing but when I tried the example, I loved how everything was self contained and easy to organize.
My goal is to use this visual transformer model  from huggingface to process my images.  I have used hugging face for the text components so wanted to keep both techniques similar for an understandability perspective.

Error:
ArrowInvalid                              Traceback (most recent call last)
 in
----> 1 preprocessed_train_ds = dataset_training.map(preprocess_images, batched=True, batch_size = 8, features=features)
~\anaconda3\envs\pytorch\lib\site-packages\datasets\arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
1663
1664         if num_proc is None or num_proc == 1:
 → 1665             return self._map_single(
1666                 function=function,
1667                 with_indices=with_indices,
~\anaconda3\envs\pytorch\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
183         }
184         # apply actual function
 → 185         out: Union[“Dataset”, “DatasetDict”] = func(self, *args, **kwargs)
186         datasets: List[“Dataset”] = list(out.values()) if isinstance(out, dict) else [out]
187         # re-apply format to the output
~\anaconda3\envs\pytorch\lib\site-packages\datasets\fingerprint.py in wrapper(*args, **kwargs)
395             # Call actual function
396
 → 397             out = func(self, *args, **kwargs)
398
399             # Update fingerprint of in-place transforms + update in-place history of transforms
~\anaconda3\envs\pytorch\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc)
2032                             else:
2033                                 batch = cast_to_python_objects(batch)
 → 2034                                 writer.write_batch(batch)
2035                 if update_data and writer is not None:
2036                     writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file
~\anaconda3\envs\pytorch\lib\site-packages\datasets\arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
389             typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)
390             typed_sequence_examples[col] = typed_sequence
 → 391         pa_table = pa.Table.from_pydict(typed_sequence_examples)
392         self.write_table(pa_table, writer_batch_size)
393
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\table.pxi in pyarrow.lib.Table.from_pydict()
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\array.pxi in pyarrow.lib.asarray()
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\array.pxi in pyarrow.lib.array()
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\array.pxi in pyarrow.lib._handle_arrow_array_protocol()
~\anaconda3\envs\pytorch\lib\site-packages\datasets\arrow_writer.py in arrow_array(self, type)
96                 out = pa.ExtensionArray.from_storage(type, pa.array(self.data, type.storage_dtype))
97             else:
—> 98                 out = pa.array(self.data, type=type)
99             if trying_type and out[0].as_py() != self.data[0]:
100                 raise TypeError(
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\array.pxi in pyarrow.lib.array()
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\array.pxi in pyarrow.lib._sequence_to_array()
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~\anaconda3\envs\pytorch\lib\site-packages\pyarrow\error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Could not convert [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0] with type list: tried to convert to int
Thanks,
John R.

Context:
Hi !
The ClassLabel feature type is for single-label multi-class classification.
For multi-label classification you can use
from datasets.features import ClassLabel, Sequence
labels_type = Sequence(ClassLabel(names=classes))

and modify your the function you pass to map to convert your list of 21 booleans to the list of True indices. For example:
[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0] → [4, 12, 18]


Question:
Hi,
I need to make a train_test_split of the dataset (namely, of its train part). I want to split it in such a way so that the instances are not shuffled, but test part is consisted of first test_size elements (instead of last test_size elements, as it is done by default when doing dataset.train_test_split(test_size=test_size, shuffle=False)).
There is certainly a primitive way of doing it with
test_data = dataset.train_test_split(train_size=test_size, shuffle=False)['train']
but since I am writing a public code, it will cause misunderstanding.
To sum up, is there a way of doing it “more beautiful”?
Thanks!

Context:
Hi ! Indeed it’s not convenient to see that this is in reverse order compared to scikit-learn
Currently there isn’t a more satisfactory solution, but feel free to to open an issue on github to discuss this behavior and how we can improve it


I had collected data for a language text for translation How can I add it up into datsets

Question:
I just am trying to keep the repository of my efforts to translate a language. I have some 7k sentences in CSV format. I want to know, how can I upload into HF datasets. This may be a basic question however if there is a page already with detailed instructions do let me know.
I tried but no options exist.
Thanks.

Context:
Hi !
Currently a HF dataset on the Hub requires a python script to be loaded (see here 3 for example).
But we are adding a feature that will allow you to upload to the Hub with a method my_dataset.push_to_hub().
You can track the progress of the first PR we made here 4 : it allows to load a dataset without a python script. Then we’ll add the actual push_to_hub method


Question:
My office PC is  not connected to internet, and I want to use the datasets package to load the dataset. I first saved the already existing dataset using the following code:
from datasets import load_dataset
datasets = load_dataset("glue", "mrpc")
datasets.save_to_disk('glue-mrpc')

A folder is created with dataset_dict.json file and three folders for train, test, and validation respectively. I zipped the folder andcopied to my office PC, and now when i tried loading the dataset from folder using the following code:
datasets = load_dataset('/content/glue-mrpc')

But I am getting the following error:
FileNotFoundError: Couldn't find file locally at /content/glue-mrpc/glue-mrpc.py. Please provide a valid dataset name.

I also tried pickling, but didnt work.
Is there any way to save the dataset in one PC and load it in another PC?
Thank you.

Context: When loading from a directory created by save_to_disk function, try using load_from_disk method


How to perform unbatch operation with huggingface datasets

Question:
Hi
I asked this issue on mapping one entry to multiple entries with huggingface dataset library:

      github.com/huggingface/datasets

      equal operation to perform unbatch for huggingface datasets  2

        opened

        Aug 7, 2021

          dorooddorood606

          bug

    Hi
I need to use "unbatch" operation in tensorflow on a huggingface dataset, I …could not find this operation, could you kindly direct me how I can do it, here is the problem I am trying to solve:

I am considering "record" dataset in SuperGlue and I need to replicate each entery of the dataset for each answer, to make it similar to what T5 originally did:

https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L925

Here please find an example:

  For example, a typical example from ReCoRD might look like
  {
      'passsage': 'This is the passage.',
      'query': 'A @placeholder is a bird.',
      'entities': ['penguin', 'potato', 'pigeon'],
      'answers': ['penguin', 'pigeon'],
  }
  and I need a prosessor which would turn this example into the following two examples:
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'penguin',
  }
  and
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'pigeon',
  }

For doing this, one need unbatch, as each entry can map to multiple samples depending on the number of answers, I am not sure how to perform this operation with  huggingface datasets library and greatly appreciate your help

@lhoestq

Thank you very much.

Not sure if this is better to be asked here, please let me know to close the issue in the repo, or here.
thank you for your kind response and assistance on how one can model this with huggingface datasets.
really appreciate your response @lhoestq

Context:
Hi ! It looks like you managed to find a solution thanks to mario’s suggestion here 5
If you have other questions, feel free to post on github or on the forum as you did


Question:
Hi,
I’m using the datasets library to load in the popular medical dataset MIMIC 3 (only the notes) and creating a huggingface dataset to get it ready for language modelling using BERT. I have a script that loads creates a custom dataset and tokenizes it and writes it to the cache file. I set load_from_cache_file in the map function of the dataset to True. The tokenization process takes a while because of the moderately large dataset size.
When I run the script again, I would expect the dataset to be loaded from the cache file quickly and move on to the next step in the process. However, even though I get a warning saying Reusing dataset text, the tokenization process is running again. This is not good because I intend of training multiple models and measuring the training time as part of a project.
However, I did notice that the time it takes to run when running the second time is less. For example, when running the first time without any cache file, the process took 25 minutes. When running the second time with the cache file available and load_from_cache_file set to True, it took 13 minutes. Is this expected behavior?
I’m using version 1.6.1 of the datasets library. I would like some help in resolving this problem. Please let me know if more info is needed.

Context:
Hi !
Have you tried using the latest release 1.6.2 ? Have you changed the parameters passed to map between the two runs ?
If the parameters (both the tokenization function, the batch size, etc.) are the same, then it will reload the tokenized text from the cache.
Also if you share the code that you’re using it would be helpful


Question:
Hello everyone,
I am currently creating a dataset where the semantics of a split make no sense. It’s an Information Retrieval corpus that should not be split. I currently load the entire corpus into the “train” split because I’ve been copying what other dataset loading scripts do, and it works, but would rather drop splits altogether for the corpus config.
Can I use other strings to name the splits? Can I leave out splits altogether for a given self.config.name?

Context:
Hi ! By default you can use the train split, but feel free to name it whatever you want.
In practice, when you return the split generators in _split_generators(), you can specify the split name you want to datasets.SplitGenerator(name=...)


Question:
Can someone point me to resources on how to go about preparing custom datasets for fine-tuning NLP models?
Suppose I want to fine-tune a text completion model on a particular author’s works so that the outputs are in their style and I have access to the text in pdf format, how do I go from bunch of pdfs to a dataset that is acceptable for input to the model?

Context: I think you might need to convert the pdfs to a readable format like json or csv . Then you can clean and process it through pandas or the datasets library provided by hugging face …


Problems while filtering large datasets using `map`

Question:
As part of out flax-jax-community-week project, I encountered a problem when attempting to filter a large dataset to select only the items matching a certain condition. I’m processing the items with map using batched mode, but some of the batches contain no matching items. We need to return an empty result in that case, but I’m having trouble doing so. If I return an empty dictionary {}, then I encounter schema errors. I think they happen when empty batches are followed by non-empty ones. If I return a dictionary with keys containing empty lists {'column_name': []}, then I get an index out of bounds error.
I filed this issue 1 describing the problem in slightly more detail. If this is not a bug, or someone else has found a workaround, that’d be great to know.

Context: Thanks for reporting ! Hopefully we can have a fix soon


Question: Suppose I have a dataset d and I wanna have a copy of the whole dataset, something similar to torch.clone() for tensors and copy.deepcopy for regular python objects. Is there a way I could achieve this using datasets?

Context:
Why not using copy.deepcopy ?
As answered in this question: What is the diffrence between copy.deepcopy and flatten_indices? - #2 by lhoestq , copy.deepcopy will create a copy of dataset.
It is just a few days that I’m using transformers and datasets, but up until now, everything I did with a datasets object, was without mutation, for example sorting, shuffling, selecting, filtering and using map does not change your datasets object, when there is no mutation, most of the times you don’t need to clone anything. So make sure if you really need to make a copy of datasets object.


Question:
According to datasets document 1:
A few interesting features are provided out-of-the-box by the Apache Arrow backend:
    - **multi-threaded** or single-threaded reading
    - automatic decompression of input files (based on the filename extension, such as my_data.csv.gz)
    - fetching column names from the first row in the CSV file
    - column-wise type inference and conversion to one of null, int64, float64, timestamp[s], string or binary data
    - detecting various spellings of null values such as NaN or #N/A

How can I read, for example train.csv.gz in multi-threaded mode ?
I use datasets.load_dataset("csv", data_files="./train.csv.gz") but htop only show one cpu core is running

Context: pinging @lhoestq , sorry I can’t find the multi-threaded part in document


Keeping only current dataset state in cache

Question:
Hi,
How would I apply multiple processing functions on a dataset while only ever keeping one version of the dataset in cache. The code below, for example, seems to keep every state of the dataset in the /tmp directory until the session ends. This results in my dataset being duplicated on disk many times, and thus requiring a ton of disk space.
set_caching_enabled(False)

dataset = load_dataset(...)

# Creates first copy in /tmp
dataset = dataset.map(...)
# Creates second copy in /tmp
dataset = dataset.map(...)
# Creates third copy in /tmp
dataset = dataset.map(...)

dataset.save_to_disk(...)

I don’t care about the intermediate versions. I’m running this as a SageMaker processing job and my whole container is discarded afterwards. All I need is an input and an output version of the dataset. How would I get rid of the additional copies in /tmp so I can reduce my scripts’ space requirements?
Thanks,
Marcel

Context:
You can get the cache files to delete with dataset.cache_files. Feel free to delete them from your filesystem at each step:
import os

def delete_dataset(dataset):
    cached_files = [cache_file["filename"] for cache_file in dataset.cache_files]
    del dataset
    for cached_file in cached_files:
        os.remove(cached_file)

# This line creates a new dataset using map, and deletes the old dataset
dataset, _ = dataset.map(...), delete_dataset(dataset)


Question:
I am trying to load the timit_asr dataset however only the first record is shown (duplicated over all the raws).
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))
The above code result with the same sentence duplicated ten times.
It also happens when I use the dataset viewer at Streamlit 2 .
Can you please help?

Context: This issue has been fixed in the recent versions of  Datasets, and the viewer in updated


Desired behavior when calling `shuffle` or `select` on `interleave_datasets`

Question: Could someone explain these operations on interleave_datasets? I didn’t find any word in the doc on this. Thanks.

Context:
Hi ! You can find the documentation of interleave_datasets here:

      huggingface.co

Main classes 1

  DatasetInfo: Dataset: The base class datasets.Dataset implements a Dataset backed by an Apache Arrow table. DatasetDict: Dictionary with split names as keys ...

Interleaving datasets can be seen as a way to alternate between examples of several datasets. For example
from datasets import Dataset, interleave_datasets

d1, d2 = {"foo": ["a", "b", "c"]}, {"foo": ["x", "y", "z"]}
d1, d2 = Dataset.from_dict(d1), Dataset.from_dict(d2)
d3 = interleave_datasets([d1, d2])
print(d3["foo"])
# ['a', 'x', 'b', 'y', 'c', 'z']

You can call shuffle or select on the resulting dataset if you want.
Shuffle will shuffle the order of the dataset:
print(d3.shuffle(seed=42)["foo"])
['y', 'b', 'z', 'c', 'x', 'a']

Select will select the indices you want:
print(d3.select([0, 1, 2])["foo"])
# ['a', 'x', 'b']


What is the diffrence between copy.deepcopy and flatten_indices?

Question:
Hi, to be honest, I’m not sure what exactly flatten_indices do, I was doing something with a dataset and I got an error saying that I’m working on a shallow copy and I should use flatten_indices, now my question is what does exactly flatten_indices do and what is the difference between using flatten_indices and python copy.deepcopy?
import copy
dataset2 = copy.deepcopy(dataset1)

# or

dataset2 = dataset1.flatten_indices()

Context:
Hi !
Python deepcopy does what it says: it creates a copy of the dataset object.
However flatten_indices has another goal.
First you must know that dataset shuffling/sharding doesn’t actually shuffle or shard the Arrow data on your disk. Instead, it creates an indices mapping that maps the queries of the user (e.g. dataset[0]) to the actual position of the examples in the Arrow data on your disk. This allows to make shuffling and sharding really fast and doesn’t require to write new Arrow data to save disk space.
However a shuffled/sharded dataset is a little bit slower to query because of this mapping. In case you want to get the optimal speed back, you have to write your dataset in a new Arrow file with flatten_indices. The resulting dataset will be loaded from this new Arrow file, and won’t have an indices mapping anymore since all the examples in the Arrow file will already be in the right order.
Hope that clarifies things, let me know if you have other questions !


Question:
I am trying to write a custom dataset using GeneratorBasedBuilder and I faced some samples have missing columns sometimes.
I really don’t know how to deal with it. Maybe we need creating a “nullable” feature value? Is it possible?
something like:
datasets.features.Sequence({"text": datasets.Value("string",nullable=True),
                            "answer_start": datasets.Value("int32"),})

Context:
Hi ! We don’t support nullable since it creates issues with type inference.
Instead, feel free to use an empty string when a string in missing, or -1 for missing labels for example


How to use `map` or similar when one row is mapped to multiple rows?

Question:
Suppose I have a dataset with 100 rows and I have a func that could turn each row into 10 rows. What I want is a mapped dataset that has 1000 rows. Is there a way I could do it using the package? Currently I got a length mismatch issue when using map.
rrowInvalid: Column 1 named test_col expected length 100 but got length 1000

Context:
Hi ! You can use map in batched mode to return more rows than the input:
from datasets import Dataset

dataset = Dataset.from_dict({"foo": ["aaa", "bbb", "ccc"]})  # 3 rows

def augment(batch):
   return {"bar": [character for foo in batch["foo"] for character in foo]}

dataset = dataset.map(augment, batched=True, remove_columns=dataset.column_names)
print(dataset["bar"])  # now it has 9 rows
# ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c']

Note that I had to pass remove_columns=dataset.column_names in order to trash the old column “foo” that doesn’t have 9 rows


How to write my own metrics if it is not in datasets.metrics

Question: I would like to use a custom metric, probably mean squared error. It is in sklearn.metrics but not in datasets.metrics – what is the quickest way to do this if it is not available in datasets.metrics?

Context:
You could have a look at implementation of existing metrics available here on datasets repo 42. You can even use one of the simpler one like accuracy or f1 as base and then modify it for your case.
Instructions to load a custom metric are available on the documentation page 33.


Set dataset to pytorch tensors produce class list making the model unable to process the data

Question:
Hello,
I am folllowing this tutorial to use Fine-tuning a pretrained model — transformers 4.7.0 documentation in order to use the flauBert to produce embeddings to train my classifier. In one of the lines , I have to set my dataset to pytorch tensors but when applying that line I get a list format which I do not understand. When printing element of the dataset I get tensors but when trying to pass the “input_ids” to the model , it is actually a list so the model cannot treat the data. Could help me figure it out why I get list and not a pytoch tensors when using 'set_format to torch.
def get_flaubert_layer(texte, path_to_lge): # last version

    tokenized_dataset, lge_size = preprocessed_with_flaubert(texte, path_to_lge)
    print("Set data to torch format...")
    tokenized_dataset.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask'])
    print("Format of data after -set.format- :", type(tokenized_dataset))
    print("Format of input_ids after -set.format- :", type(tokenized_dataset['input_ids']))
    print("Format of one element in the dataset", type(tokenized_dataset['input_ids'][0]))
    print(type(tokenized_dataset))
    print('Loading model...')
    flaubert = FlaubertModel.from_pretrained(path_to_lge)
    hidden_state = flaubert(input_ids=tokenized_dataset['input_ids'],
                            attention_mask=tokenized_dataset['attention_mask'])
    print(hidden_state[0][:, 0])
    cls_embedding = hidden_state[0][:, 0]
    print(cls_embedding)

#  test with data
path = '/gpfswork/rech/kpf/umg16uw/expe_5/model/sm'
print("Load model...")
flaubert = FlaubertModel.from_pretrained(path)
emb, s = get_flaubert_layer(data1, path)

Stacktrace and results
  0%|          | 0/4 [00:00<?, ?ba/s]
 25%|██▌       | 1/4 [00:00<00:01,  1.63ba/s]
 50%|█████     | 2/4 [00:01<00:01,  1.72ba/s]
 75%|███████▌  | 3/4 [00:01<00:00,  1.81ba/s]
100%|██████████| 4/4 [00:01<00:00,  2.41ba/s]
Traceback (most recent call last):
  File "/gpfs7kw/linkhome/rech/genlig01/umg16uw/test/expe_5/traitements/remove_noise.py", line 130, in <module>
    emb, s = get_flaubert_layer(data1, path)
  File "/gpfs7kw/linkhome/rech/genlig01/umg16uw/test/expe_5/traitements/functions_for_processing.py", line 206, in get_flaubert_layer
    hidden_state = flaubert(input_ids=tokenized_dataset['input_ids'],
  File "/linkhome/rech/genlig01/umg16uw/.conda/envs/bert/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/linkhome/rech/genlig01/umg16uw/.conda/envs/bert/lib/python3.9/site-packages/transformers/models/flaubert/modeling_flaubert.py", line 174, in forward
    bs, slen = input_ids.size()
AttributeError: 'list' object has no attribute 'size'
srun: error: r11i0n6: task 0: Exited with exit code 1
srun: Terminating job step 381611.0

real	1m37.690s
user	0m0.013s

format of the data passed to the flaubert model :
Loading tokenizer...
Transform data to format Dataset...
Set data to torch format...
Format of data after -set.format- : <class 'datasets.arrow_dataset.Dataset'>
Format of input_ids after -set.format- : <class 'list'>
Format of one element in the dataset <class 'torch.Tensor'>

As you see the columns input_ids is in a formal list and not tensors

Context:
Try to change this:

 emmakelo:

hidden_state = flaubert(input_ids=tokenized_dataset['input_ids'],
                            attention_mask=tokenized_dataset['attention_mask

in this:
hidden_state = flaubert(**tokenized_dataset[0])


get batch indices when iterating DataLoader over a Dataset

Question:
The code below is taken from the tutorial 4
from datasets import load_metric

metric= load_metric("glue", "mrpc")
model.eval()
for batch in eval_dataloader:
    batch = {k: v.to(device) for k, v in batch.items()}
    with torch.no_grad():
        outputs = model(**batch)

    logits = outputs.logits
    predictions = torch.argmax(logits, dim=-1)
    metric.add_batch(predictions=predictions, references=batch["labels"])

metric.compute()

Inside the loop for batch in eval_dataloader:, how can I know which indices from the dataset this batch includes?
The DataLoader is created earlier using
eval_dataloader = DataLoader(
    tokenized_datasets["validation"], batch_size=8, collate_fn=data_collator
)

Note that it’s without the shuffling flag, so it’s possible to count manually using batch size, but how to do it with shuffling? Is it possible to to make it a field of the batch when creating the dataset and dataloader?

Context: Hi ! I think it’s possible to get the indices used by the sampler (see Indices of a dataset sampled by DataLoader - PyTorch Forums 84 for example).


Caching a dataset with map() when loaded with from_dict()

Question:
Using datasets 1.8.0.
Normal? Situations
If I use load_dataset() to load data, it generates cache files. If you then apply .map() on that dataset, corresponding cache files are generated as expected. Following is a simple code to reproduce the results.
from datasets import load_dataset, Dataset

def add_prefix(example):
    example['sentence1'] = 'My sentence: ' + example['sentence1']
    return example

def main():
    dataset = load_dataset('glue', 'mrpc', split='train')
    print(dataset)
    dataset = dataset.map(add_prefix)
    print(dataset)

if __name__ == "__main__":
    main()

Running the above will generate the following output. As we can see with the tqdm progress bar, the .map() is applied and the cache files are saved accordingly.
Downloading: 28.8kB [00:00, 25.6MB/s]
Downloading: 28.7kB [00:00, 28.8MB/s]
Downloading and preparing dataset glue/mrpc (download: 1.43 MiB, generated: 1.43 MiB, post-processed: Unknown size, total: 2.85 MiB) to /home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad...
Downloading: 6.22kB [00:00, 7.73MB/s]
Downloading: 1.05MB [00:00, 9.16MB/s]
Downloading: 441kB [00:00, 4.83MB/s]
0 examples [00:00, ? examples/s]2021-06-26 00:07:58.740056: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Dataset glue downloaded and prepared to /home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad. Subsequent calls will reuse this data.
Dataset({
    features: ['idx', 'label', 'sentence1', 'sentence2'],
    num_rows: 3668
})
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3668/3668 [00:00<00:00, 20903.25ex/s]
Dataset({
    features: ['idx', 'label', 'sentence1', 'sentence2'],
    num_rows: 3668
})

Running the exact code again will use the saved cache for both load_dataset() and .map().
Reusing dataset glue (/home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)
Dataset({
    features: ['idx', 'label', 'sentence1', 'sentence2'],
    num_rows: 3668
})
Loading cached processed dataset at /home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-53c01e7abeb2b20b.arrow
Dataset({
    features: ['idx', 'label', 'sentence1', 'sentence2'],
    num_rows: 3668
})

Not-so-normal Situations
However, using from_dict() to load dataset from the memory does not generate cache files. Because the dataset did not initially generate the cache files, applying .map() on this dataset also does not generate cache files. This can be observed in arrow_dataset.py lines 1803-1813.
# Check if we've already cached this computation (indexed by a hash)
if self.cache_files:  # !!!THIS WILL BE AN EMPTY LIST when loaded with .from_dict()!!!
    if cache_file_name is None:
        # we create a unique hash from the function,
        # current dataset file and the mapping args
        cache_file_name = self._get_cache_file_path(new_fingerprint)
    if os.path.exists(cache_file_name) and load_from_cache_file:
        logger.warning("Loading cached processed dataset at %s", cache_file_name)
        info = self.info.copy()
        info.features = features
        return Dataset.from_file(cache_file_name, info=info, split=self.split)

Why is this a problem?
In the thread below, it was suggested that one use torch.distributed.barrier() when using a distributed framework to process a large dataset.

    Slow processing with map when using deepspeed or fairscale 🤗Datasets

    I was also able to reproduce the result. Thanks for the prompt support @stas and @sgugger.

For my specific use-case, I create the dataset using the .from_dict() method. I then process the dataset using .map() using the main process (no cache files get saved automatically unless you specify the cache_file_name parameter of .map()). Once the non-main processes are resumed, they are not able to locate the cache files.

Possible Workarounds

I could save all data (maybe a csv) and load it using load_dataset(). This way, cache files are generated and all is well. However, my dataset is large (~100GB), and multiple processing using .map() further increases the data size making it very difficult to store and load data efficiently with limited HW resources. That’s why I’m using from_dict() to load smaller data on-the-fly.
Instead of using in-memory data to create the dataset, I could first save the data as csv or json. I could then use load_dataset('csv' or 'json', ...). I will try this method for now. I think it will work.

Context:
Hi !
When dataset.cache_files is empty (i.e. when your dataset comes from python objects, not from data from your disk), the map() method doesn’t know where to write the resulting dataset.
In this case, you have to pass cache_file_name=<path/to/resulting/cache/file.arrow> manually to map().
For the subsequent calls to .map(), you won’t need to specify this anymore, since it will store the cache files in the same directory as the path your provided in the first place.


Question:
Hi, I am fine-tuning a classification model and would like to log accuracy, precision, recall and F1 using Trainer API.
While I am using metric = load_metric("glue", "mrpc") it logs accuracy and F1, but when I am using metric = load_metric("precision", "recall", "f1") it only logs the first metric.
Is it by design? Do I need to write a custom script if I want to log all these metrics by epochs/steps using Trainer API?

Context: You need to load each of those metrics separately, I don’t think the loader accepts a list.


Question:
Hi guys,
I wanted to check the oscar data set and create a subsample of it for the experiment. I followed the instruction in the documentation:
from datasets import load_dataset

dataset = load_dataset("oscar", "unshuffled_deduplicated_fa")

and it return
    train: Dataset({
        features: ['id', 'text'],
        num_rows: 8203495
    })
})

my question was how can I access the text data it self.
The directory that pops up when I loading data set is
/root/.cache/huggingface/datasets/oscar/unshuffled_deduplicated_fa/1.0.0/e4f06cecc7ae02f7adf85640b4019bf476d44453f251a1d84aebae28b0f8d51d
Thanks

Context: dataset['train']['text'] for instance


Question:
Hey @lhoestq ,
Is there a roadmap/timeline for dataset streaming?
Cheers

Context: hey @vblagoje even better, there’s an open PR: Dataset Streaming by lhoestq · Pull Request #2375 · huggingface/datasets · GitHub 37


Question:
Does datasets support nested datasets? I have a nested dataset path dict, e.g.,
{
	a: {
		train:
		dev:
		test:
	},
	b: {
		train:
		dev:
		test:
	}
}

and load_dataset seems unhappy with this structure (TypeError: expected str, bytes or os.PathLike object, not dict). Does that imply I have to do load_dataset for each sub-dataset and then merge them together?
Further, is there any support for over/undersampling given multiple (sub-)datasets? Thanks a lot!

Context:
Hi !
Indeed load_dataset is used to load only one dataset. This dataset can have several splits though.
In your case you have to call load_dataset once per dataset.
Regarding over/undersampling, we are adding an interleave_datasets functions that creates a new dataset from several datasets. By default it alternates between each original dataset, but you can specify sampling probabilities to over/undersample from the original datasets
It will be available in the next release of the library.
What you can do in the meantime is use a mix of shuffle and conatenate_datasets.
If you have dataset1 and dataset2 and want to get dataset3 with 10 examples from datasets1 and 90 examples from dataset2 you can do:
from datasets import concatenate_datasets

seed = 42
dataset3 = concatenate_datasets([
    dataset1.shuffle(seed=seed).select(range(10)),
    dataset2.shuffle(seed=seed).select(range(90)),
])


Slow processing with map when using deepspeed or fairscale

Question:
I apply the tokenizer to my custom dataset using the  datasets.Dataset.map()  4 method as done in the run_mlm.py 3  example. FYI, I am using multiprocessing by setting num_proc parameter of map().
I have a multi-GPU system, and doing the above usually takes about ~10 minutes. But once I use DeepSpeed (deepspeed --include localhost:0,1,2), the process takes much longer (~20 minutes). I’m aware the preprocessing is done 100% on the CPU, so I don’t know why I am experiencing this slow down in speed in the first place. If I limit the number of visible devices to just one (deepspeed --include localhost:0), the preprocessing using map() seems to be fast again.

Context: Maybe @stas will have an idea?


Lazy-Loading binarized shard using Hf-dataset for Hf-Trainer

Question:
I have been in a dilemma for training language models. I wanted to train a model with large amount (>5TB) of data. What I did is,

Write my own data loading script using this 2.
Binarize the data
Load the data using load_dataset module
Use huggingface trainer to train the model. At first, I wrote my own training loop. But since hf-trainer comes with deepspeed I decided to move into it.
But what happens that my binarize data becomes large and it cannot load the full data into the RAM (not even in 428GB RAM).

I was wondering is there any way I can shard the data and perform lazy loading? What I mean by Lazy-loading is to load a binarized shard into memory only when I train the model with a specific shard.
The tradeoff becomes, either I learn deepspeed and integrate into my training loop which seems a very hectic job, or try solving this issue with hf-dataset.
@patrickvonplaten @valhalla

Context:
Hey there,
With datasets, all the pre-processed (binarized) data will be cached on the disk and will be loaded lazily by default. So you won’t run into any RAM issues at all.
So instead of first binarizing the data one could load it using datasets and then pre-process it using the .map method, this will cache the data which then you can load lazily.
You could refer to the training scripts in transformers examples 8.
Also, refer to the datasets dos 5
Hope this helps!


Question:
I trained a BPE tokenizer using the wiki-text and now I’m trying to use this tokenizer on a custom dataset from a csv file. What I want to achieve is additional of feature columns in my dataset. But the dataset.map is giving error.

Screenshot 2021-06-23 at 5.47.51 PM730×904 152 KB

Context: You should just use the tokenizer __call__: tokenizer(example["text"]).


Question:
Hi,
I’m trying to make sense out of the various versions of the bigpatent dataset.
In particular, the version in the huggingface’s dataset library is uncased and tokenized. Did you obtain that version out of the raw untokenized dataset here or it was provided by the authors?
My confusion comes from the fact that there are different versions of the dataset. For example, the Pegasus preserved casing for their experients. As far as I understood, their version of the dataset was obtained from the raw dataset by using this 1 preprocessing. However, I fail when trying to build the dataset*, even when considering a single code split only.
Also, the huggingface dataset seems to lack some (useless) sections (e.g. in relation to previous art) so I was wondering how you obtained the clean version of the dataset in the first place.
Can anybody help?
*I have seen from the TFDS issues that huggingface had the same problem when trying to replicate the results from Pegasus. Did you manage to solve that problem?

Context:
Hi ! We’re using the same source data as what TFDS used to use.
What error do you get when trying to build the dataset ?


Making multiple samples from single samples using HuggingFace Datasets

Question:
My use case involved building multiple samples from a single sample. Is there any way I can do that with Datasets.map().
Just a view of what I need to do:
# this is how my dataset looks like
dataset = [(1, 2, 3), (5, 7, 8)]

# this is how my output dataset should look like
output = [1, 2, 3, 5, 7, 8]

** these list & tuple are just for making my point more clear, in actual its a Huggingface dataset 😉.

Can I do something like that with Datasets.map() or any other way to do this??
cc @lhoestq

Context:
Hi ! Yes you can definitely do that using a batched map call.
Basically you can return more examples than the number of examples in per batch:
from datasets import Dataset

d = Dataset.from_dict({"foo": [(1, 2, 3), (5, 7, 8)]})
d = d.map(lambda x: {"foo": [i for row in x["foo"] for i in row]}, batched=True)

print(d["foo"])
# [1, 2, 3, 5, 7, 8]


Question: Hi, I am new here! But I tried out the STT colab and it looks like there is something wrong with the Timit dataset (Streamlit 9) because all the entries are the same. Thanks for any advice!

Context: Thanks for reporting ! This has been fixed now


Question:
Hi I’am trying to use nlp datasets to train a RoBERTa Model from scratch and I am not sure how to perpare the dataset to put it in the Trainer:
!pip install datasets
from datasets import load_dataset
dataset = load_dataset('wikicorpus', 'raw_en')

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer,
    mlm=True,
    mlm_probability=0.15)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=16,
    save_steps=10_000,
    save_total_limit=2)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset)

How do I have to dataset.set_format() such that it only takes the text of the dataset, line-by-line?
Or what’s the proper way to prepare the dataset for MLM?
In the past I have been doing it with:
from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="/dataset.txt"
)

which will be removed soon and does not support multiple txt files.
Thanks

Context: You should have a look at the preprocessing done in the run_mlm example 128. There is also the corresponding notebook 122 that can help.


Question:
To my understanding, set_transform should do transformations on the fly such that the gpu can immediately use if for training.
When I specify group_by_length=True on the trainer, set_transform no longer does lazy eval, it goes through the whole dataset – my hunch is that it needs to do all the transformations first to be able to group by length.
Is this behavior intended? I think the group_by_length should only be limited to the batch size (or a smaller subset of the dataset) and not to the whole dataset

Context: No, group_by_lengths need to read all the lengths of the dataset to be able to build batches of similar lengths.


Map method to tokenize raises index error

Question:
Hi I have a custom dataset of structure:
DatasetDict({
train: Dataset({
    features: ['sentence', 'label'],
    num_rows: 12638343
})
test: Dataset({
    features: ['sentence', 'label'],
    num_rows: 1560290
})
validation: Dataset({
    features: ['sentence', 'label'],
    num_rows: 1404261
})
})

When I encode training example using map method from dataset with tokenizer defined:
Note I can only use num_proc = 1 no matter how I code if name == “main”: after defining tokenizer for multiprocess it will say “tokenizer” not found… anyway lets focus on this error for the time being
train_encodings = dataset["train"].map(lambda examples: tokenizer(examples['sentence'], truncation = True, padding = True), batched=True, num_proc = 1, keep_in_memory = True)

I ran into Index Error 10% through, no matter I change batch_size it will still stuck at this:
C:\Users\xxxxx\AppData\Roaming\Python\Python38\site-packages\datasets\table.py:84:
RuntimeWarning: overflow encountered in int_scalars
k = i + ((j - i) * (x - arr[i]) // (arr[j] - arr[i]))
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-13-b3bd4b3dbd29> in <module>
  1 ## Pass our Tokenizer to our Train, test, validation set ##
  2
----> 3 train_encodings = dataset["train"].map(lambda examples: tokenizer(examples['sentence'],
truncation = True, padding = True), batched=True, num_proc = 1, keep_in_memory = True)
  4
  5 # val_encodings = dataset["validation"].map(lambda examples: tokenizer(examples['sentence'],
truncation = True, padding = True), batched=True, num_proc = 1, keep_in_memory = True)

~\AppData\Roaming\Python\Python38\site-packages\datasets\arrow_dataset.py in map(self,
function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns,
keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features,
disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
1481
1482         if num_proc is None or num_proc == 1:
-> 1483             return self._map_single(
1484                 function=function,
1485                 with_indices=with_indices,

~\AppData\Roaming\Python\Python38\site-packages\datasets\arrow_dataset.py in wrapper(*args,
**kwargs)
172         }
173         # apply actual function
--> 174         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
175         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
176         # re-apply format to the output

~\AppData\Roaming\Python\Python38\site-packages\datasets\fingerprint.py in wrapper(*args,
**kwargs)
338             # Call actual function
339
--> 340             out = func(self, *args, **kwargs)
341
342             # Update fingerprint of in-place transforms + update in-place history of transforms

~\AppData\Roaming\Python\Python38\site-packages\datasets\arrow_dataset.py in
_map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch,
remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size,
features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset)
1812                         if drop_last_batch and i + batch_size > input_dataset.num_rows:
1813                             continue
-> 1814                         batch = input_dataset[i : i + batch_size]
1815                         indices = list(
1816                             range(*(slice(i, i + batch_size).indices(input_dataset.num_rows)))

~\AppData\Roaming\Python\Python38\site-packages\datasets\arrow_dataset.py in __getitem__(self,
key)
1343     def __getitem__(self, key: Union[int, slice, str]) -> Union[Dict, List]:
1344         """Can be used to index columns (by string names) or rows (by integer index or iterable of
indices or bools)."""
-> 1345         return self._getitem(
1346             key,
1347             format_type=self._format_type,

~\AppData\Roaming\Python\Python38\site-packages\datasets\arrow_dataset.py in _getitem(self,
key, format_type, format_columns, output_all_columns, format_kwargs)
1335         format_kwargs = format_kwargs if format_kwargs is not None else {}
1336         formatter = get_formatter(format_type, **format_kwargs)
-> 1337         pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not
None else None)
1338         formatted_output = format_table(
1339             pa_subtable, key, formatter=formatter, format_columns=format_columns,
output_all_columns=output_all_columns

~\AppData\Roaming\Python\Python38\site-packages\datasets\formatting\formatting.py in
 query_table(table, key, indices)
 366     # Query the main table
 367     if indices is None:
--> 368         pa_subtable = _query_table(table, key)
 369     else:
 370         pa_subtable = _query_table_with_indices_mapping(table, key, indices=indices)

~\AppData\Roaming\Python\Python38\site-packages\datasets\formatting\formatting.py in
_query_table(table, key)
 82     if isinstance(key, range):
 83         if _is_range_contiguous(key) and key.start >= 0:
---> 84             return table.fast_slice(key.start, key.stop - key.start)
 85         else:
 86             pass  # treat as an iterable

~\AppData\Roaming\Python\Python38\site-packages\datasets\table.py in fast_slice(self, offset, length)
131             batches[0] = batches[0].slice(offset - self._offsets[i])
132         else:
--> 133             j = _interpolation_search(self._offsets, offset + length - 1)
134             batches = self._batches[i : j + 1]
135             batches[-1] = batches[-1].slice(0, offset + length - self._offsets[j])

~\AppData\Roaming\Python\Python38\site-packages\datasets\table.py in _interpolation_search(arr, x)
 89         else:
 90             i, j = i, k
---> 91     raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
 92
 93

IndexError: Invalid query '1698999' for size 12638343.

I could have load the csv using pandas then encode on it but I close to run out of memory on that so I try to use datasets_load class… Any help would be appreciated! Thanks

Context: I am facing similar issue. Have you found any solution?


Question:
Hi,
I have a very large dataset that fails in the preprocess stage due to its size.
I’ve looked into: sharding the dataset vs. using batched=True on the map(…) function.
What are the advantages/use cases of either cases?
Also can anyone point me to an example that uses each methodology?
Shards - The official docs lacks the use of shards (eg. how to do a full train using shards – do you run the training scheme such that you do 1 epoch for the shard and then switch to the next shard?)
Batched - Not sure what it means by:
Provided function which is applied to all elements of table returns a dict of types [<class 'list'>, <class 'list'>, <class 'str'>]. When using batched=True, make sure provided function returns a dict of types like (<class 'list'>, <class 'numpy.ndarray'>)
Any clear example showing batched=True will help!

Context:
Hi ! Do you need to shard your dataset because you don’t have enough disk space ?
Have you considered doing the preprocessing on-the-fly during training to avoid filling up your disk ?


Question:
How can I add a new column to my dataset?
I am working on Cosmos QA dataset and need to add a new column of the following format:
Value(dtype=‘string’, id=None)
The current dataset has the following features:
Dataset({
features: [‘id’, ‘context’, ‘question’, ‘answer0’, ‘answer1’, ‘answer2’, ‘answer3’, ‘label’],
num_rows: 25262
})
Thank you in advance!

Context:
Hi ! You can use the add_column method:
from datasets import load_dataset

ds = load_dataset("cosmos_qa", split="train")

new_column = ["foo"] * len(ds)
ds = ds.add_column("new_column", new_column)

and you get a dataset
Dataset({
    features: ['id', 'context', 'question', 'answer0', 'answer1', 'answer2', 'answer3', 'label', 'new_column'],
    num_rows: 25262
})


Question:
Hey guys. I’m trying to fine-tune a GPT2 model using the DataCollatorForLanguageModelling, but I’m getting a message like:

image1127×330 74 KB

I’m not asking to pad it anywhere, does anyone has a clue about what should be going wrong?

Context:
Hi ! I think some users have already experienced this issue and found a fix in this thread:

      github.com/huggingface/transformers

      ValueError: Unable to set proper padding strategy as the tokenizer does not have a padding token.  12

        opened

        Apr 19, 2020

          closed

        May 5, 2020

          banunitte

    # ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for b…ugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
Using pad_token, but it is not set yet.

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

tokens = tokenizer.batch_encode_plus(
    ["This is a sample", "This is another longer sample text"],
    pad_to_max_length=True ,max_length=10 ,return_attention_mask = True# First sentence will have some PADDED tokens to match second sequence length
)

for i in range(2):
    print("Tokens (int)      : {}".format(tokens['input_ids'][i]))
    print("Tokens (str)      : {}".format([tokenizer.convert_ids_to_tokens(s) for s in tokens['input_ids'][i]]))
    print("Tokens (attn_mask): {}".format(tokens['attention_mask'][i]))
    print()

I hope that helps