SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: strange datasets from OSCAR corpus

Comment:
Hi ! Thanks for reporting
cc @pjox is this an issue from the data ?

Anyway we should at least mention that OSCAR could contain such contents in the dataset card, you're totally right @jerryIsHere
Issue: strange datasets from OSCAR corpus

Comment:
Hi @jerryIsHere , sorry for the late response! Sadly this is normal, the problem comes form fasttext's classifier which we used to create the original corpus. In general the classifier is not really capable of properly recognizing Yue Chineese so the file ends un being just noise from Common Crawl. Some of these problems with OSCAR were already discussed [here](https://arxiv.org/pdf/2103.12028.pdf) but we are working on explicitly documenting the problems by language on our website. In fact, could please you open an issue on [our repo](https://github.com/oscar-corpus/oscar-website/issues) as well so that we can track it?
Issue: Missing original answers in kilt-TriviaQA

Comment:
That could be useful indeed! Feel free to open a PR on the dataset card if you already have some code that runs, otherwise we'll take care of it soon :)
Issue: Missing original answers in kilt-TriviaQA

Comment:
I can open a PR but there is 2 details to fix:
- the name for the corresponding key (e.g. `original_answer`)
- how to implement it: I’m not sure what happens when you map `lambda x: {'input': ...}` as it keeps the other keys (e.g. `output`) intact but here since we want to set a nested value (e.g. `x['output']['original_answer']`) I implemented it with a regular function (not lambda), see below

```py
def add_original_answer(x, trivia_qa, triviaqa_map):
    i = triviaqa_map[x['id']]
    x['output']['original_answer'] = trivia_qa['validation'][i]['answer']['value']
    return x
```
Issue: datasets 1.6 ignores cache

Comment:
Looks like there are multiple issues regarding this (#2386, #2322) and it's a WIP #2329. Currently these datasets are being loaded in-memory which is causing this issue. Quoting @mariosasko here for a quick fix:

> set `keep_in_memory` to `False` when loading a dataset (`sst = load_dataset("sst", keep_in_memory=False)`) to prevent it from loading in-memory. Currently, in-memory datasets fail to find cached files due to this check (always False for them)
Issue: datasets 1.6 ignores cache

Comment:
Hi ! Since `datasets` 1.6.0 we no longer keep small datasets (<250MB) on disk and load them in RAM instead by default. This makes data processing and iterating on data faster. However datasets in RAM currently have no way to reload previous results from the cache (since nothing is written on disk). We are working on making the caching work for datasets in RAM.

Until then, I'd recommend passing `keep_in_memory=False` to the calls to `load_dataset` like here:

https://github.com/huggingface/transformers/blob/223943872e8c9c3fc11db3c6e93da07f5177423f/examples/pytorch/language-modeling/run_clm.py#L233

This way you say explicitly that you want your dataset to stay on the disk, and it will be able to recover previously computed results from the cache.
Issue: datasets 1.6 ignores cache

Comment:
OK, It doesn't look like we can use the proposed workaround - see https://github.com/huggingface/transformers/issues/11801

Could you please add an env var for us to be able to turn off this unwanted in our situation behavior? It is really problematic for dev work, when one needs to restart the training very often and needs a quick startup time. Manual editing of standard scripts is not a practical option when one uses examples.

This could also be a problem for tests, which will be slower because of lack of cache, albeit usually we use tiny datasets there. I think we want caching for tests.

Thank you.
Issue: datasets 1.6 ignores cache

Comment:
Hi @stas00, 

You are right: an env variable is needed to turn off this behavior. I am adding it.

For the moment there is a config parameter to turn off this behavior: `datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES = None`

You can find this info in the docs:
- in the docstring of the parameter `keep_in_memory` of the function [`load_datasets`](https://huggingface.co/docs/datasets/package_reference/loading_methods.html#datasets.load_dataset):
- in a Note in the docs about [Loading a Dataset](https://huggingface.co/docs/datasets/loading_datasets.html#from-the-huggingface-hub)

> The default in 🤗Datasets is to memory-map the dataset on drive if its size is larger than datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES (default 250 MiB); otherwise, the dataset is copied in-memory. This behavior can be disabled by setting datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES = None, and in this case the dataset is not loaded in memory.
Issue: datasets 1.6 ignores cache

Comment:
Yes, but this still requires one to edit the standard example scripts, so if I'm doing that already I just as well can add `keep_in_memory=False`.

May be the low hanging fruit is to add `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES` env var to match the config, and if the user sets it to 0, then it'll be the same as `keep_in_memory=False` or `datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=0`?
Issue: datasets 1.6 ignores cache

Comment:
@stas00, however, for the moment, setting the value to `0` is equivalent to the opposite, i.e. `keep_in_memory=True`. This means the max size until which I load in memory is 0 bytes.

Tell me if this is logical/convenient, or I should change it.
Issue: datasets 1.6 ignores cache

Comment:
In my PR, to turn off current default bahavior, you should set env variable to one of: `{"", "OFF", "NO", "FALSE"}`.

For example:
```
MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=
```
Issue: datasets 1.6 ignores cache

Comment:
IMHO, this behaviour is not very intuitive, as 0 is a normal quantity of bytes. So `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=0` to me reads as don't cache ever.

Also "SIZE_IN_BYTES" that can take one of `{"", "OFF", "NO", "FALSE"}` is also quite odd.

I think supporting a very simple `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES` that can accept any numerical value to match the name of the variable, requires minimal logic and is very straightforward. 

So if you could adjust this logic - then `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=0` is all that's needed to not do in-memory datasets.

Does it make sense?
Issue: datasets 1.6 ignores cache

Comment:
I understand your point @stas00, as I am not very convinced with current implementation.

My concern is: which numerical value should then pass a user who wants `keep_in_memory=True` by default, independently of dataset size? Currently it is `0` for this case.
Issue: datasets 1.6 ignores cache

Comment:
That's a good question, and again the normal bytes can be used for that:
```
MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=1e12 # (~2**40)
```
Since it's unlikely that anybody will have more than 1TB RAM.

It's also silly that it uses BYTES and not MBYTES - that level of refinement doesn't seem to be of a practical use in this context.

Not sure when it was added and if there are back-compat issues here, but perhaps it could be renamed `MAX_IN_MEMORY_DATASET_SIZE` and support 1M, 1G, 1T, etc. 

But scientific notation is quite intuitive too, as each 000 zeros is the next M, G, T multiplier. Minus the discrepancy of 1024 vs 1000, which adds up. And it is easy to write down `1e12`, as compared to `1099511627776` (2**40).  (`1.1e12` is more exact).
Issue: datasets 1.6 ignores cache

Comment:
Great! Thanks, @stas00.

I am implementing your suggestion to turn off default value when set to `0`.

For the other suggestion (allowing different metric prefixes), I will discuss with @lhoestq to agree on its implementation.
Issue: ArrowDataset.save_to_disk produces files that cannot be read using pyarrow.feather

Comment:
Hi ! This is because we are actually using the arrow streaming format. We plan to switch to the arrow IPC format.
More info at #1933
Issue: ArrowDataset.save_to_disk produces files that cannot be read using pyarrow.feather

Comment:
Not sure if this was resolved, but I am getting a similar error when trying to load a dataset.arrow file directly: `ArrowInvalid: Not an Arrow file`
Issue: ArrowDataset.save_to_disk produces files that cannot be read using pyarrow.feather

Comment:
Since we're using the streaming format, you need to use `open_stream`:

```python
import pyarrow as pa

def in_memory_arrow_table_from_file(filename: str) -> pa.Table:
    in_memory_stream = pa.input_stream(filename)
    opened_stream = pa.ipc.open_stream(in_memory_stream)
    pa_table = opened_stream.read_all()
    return pa_table

def memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
    memory_mapped_stream = pa.memory_map(filename)
    opened_stream = pa.ipc.open_stream(memory_mapped_stream)
    pa_table = opened_stream.read_all()
    return pa_table
```
Issue: Loading dataset from local path

Comment:
Version below works, checked again in the docs, and data_files should be a path.
```
ds = datasets.load_dataset('my_script.py', 
                           data_files='/data/dir/corpus.txt', 
                           cache_dir='.')
```
Issue: `FaissIndex.save` throws error on GPU

Comment:
Just in case, this is a workaround that I use in my code and it seems to do the job.

```python
if use_gpu_index:
    data["train"]._indexes["text_emb"].faiss_index = faiss.index_gpu_to_cpu(data["train"]._indexes["text_emb"].faiss_index)
```
Issue: Add an API to access the language and pretty name of a dataset

Comment:
Hi ! With @bhavitvyamalik we discussed about having something like
```python
from datasets import load_dataset_card

dataset_card = load_dataset_card("squad")
print(dataset_card.metadata.pretty_name)
# Stanford Question Answering Dataset (SQuAD)
print(dataset_card.metadata.languages)
# ["en"]

```
What do you think ?

I don't know if you already have a way to load the model tags in `transformers` but we can agree on the API to have something consistent.

Also note that the pretty name would only be used to show users something prettier than a dataset id, but in the end the source of truth will stay the dataset id (here `squad`).
Issue: Add an API to access the language and pretty name of a dataset

Comment:
What dataset_info method are you talking about @julien-c ? In `huggingface_hub` I can only see `model_info`.
Issue: Add an API to access the language and pretty name of a dataset

Comment:
Indeed, this info can now be fetched with `huggingface_hub.dataset_info`, so I think we can close this issue.
Issue: [Question] How to move and reuse preprocessed dataset? 

Comment:

Hi :) Can you share with us the code you used ?


EDIT: from https://github.com/huggingface/transformers/issues/11665#issuecomment-838348291 I understand you're using the run_clm.py script. Can you share your logs ?
Issue: [Question] How to move and reuse preprocessed dataset? 

Comment:
Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same
Issue: [Question] How to move and reuse preprocessed dataset? 

Comment:
> Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

I only changed the `preprocessing_num_workers` maybe it is the problem~ I will try again~
Issue: Is there a way to join multiple datasets in one?

Comment:
Hi ! We don't have `join`/`merge` on a certain column as in pandas.
Maybe you can just use the [concatenate_datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets) function.
Issue: Is there a way to join multiple datasets in one?

Comment:
Hi! You can use `datasets_sql` for that now. As of recently, PyArrow also supports querying tables via Substrait, so I think we can start adding these methods to the API soon.
Issue: Columns are removed before or after map function applied?

Comment:
Hi! Columns are removed **after** applying the function and **before** updating the examples with the function's output (as per the docs [here](https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.map.remove_columns)). I agree the docs on this should be more clear.
Issue: NonMatchingChecksumError for web_of_science dataset

Comment:
I've raised a PR for this. Should work with `dataset = load_dataset("web_of_science", "WOS11967", ignore_verifications=True)`once it gets merged into the main branch. Thanks for reporting this!
Issue: Allow passing `desc` to `tqdm` in `Dataset.map()`

Comment:
I think the user could pass the `desc` parameter to `map` so that it can be displayed in the tqdm progress bar, as suggested by @cccntu.

When there's no multiprocessing, the `desc` of the progress bar could be the `desc` passed by the user.
In multiprocessing, we were already using a `desc` equal to `"#" + str(rank)`.
We can change it to be `(desc or "") + "#" + str(rank)` instead.

In the end, since both `desc` and `rank` could be None, we can have:
```python
pbar_desc = (desc or "") + "#" + str(rank) if rank is not None else desc
```

Finally let's remember that if we add `desc` as a new parameter to `map`, we should add it to the `ignore_kwargs` list of the `@fingerprint_transform` decorator of `Dataset._map_single` since we don't want this parameter to affect the fingerprint of the resulting dataset.
Issue: A syntax error in example

Comment:
cc @beurkinger but I think this has been fixed internally and will soon be updated right ?
Issue: load_dataset("timit_asr") gives back duplicates of just one sample text

Comment:
Thanks @ekeleshian for having reported.

I am closing this issue once that you updated `datasets`. Feel free to reopen it if the problem persists.
Issue: Calls to map are not cached.

Comment:
Hi,

set `keep_in_memory` to False when loading a dataset (`sst = load_dataset("sst", keep_in_memory=False)`) to prevent it from loading in-memory. Currently, in-memory datasets fail to find cached files due to this check (always False for them):

https://github.com/huggingface/datasets/blob/241a0b4a3a868778ee91e767ad406f9da7610df2/src/datasets/arrow_dataset.py#L1718

@albertvillanova It seems like this behavior was overlooked in #2182.
Issue: Calls to map are not cached.

Comment:
Hi @villmow, thanks for reporting. 

As @mariosasko has pointed out, we did not consider this case when introducing the feature of automatic in-memory for small datasets. This needs to be fixed.
Issue: Calls to map are not cached.

Comment:
Hi ! Currently a dataset that is in memory doesn't know doesn't know in which directory it has to read/write cache files.
On the other hand, a dataset that loaded from the disk (via memory mapping) uses the directory from which the dataset is located to read/write cache files.

Because of that, currently in-memory datasets simply don't use caching.

Maybe a Dataset object could have a `cache_dir` that is set to the directory where the arrow files are created during `load_dataset` ?
Issue: UnicodeDecodeError for OSCAR (Afrikaans)

Comment:
Thanks for reporting, @sgraaf.

I am going to have a look at it. 

I guess the expected codec is "UTF-8". Normally, when no explicitly codec is passed, Python uses one which is platform-dependent. For Linux machines, the default codec is `utf_8`, which is OK. However for Windows machine, the default codec is `cp1252`, which causes the problem.
Issue: UnicodeDecodeError for OSCAR (Afrikaans)

Comment:
@sgraaf, I have just merged the fix in the master branch.

You can either:
- install `datasets` from source code
- wait until we make the next release of `datasets`
- set the `utf-8` codec as your default instead of `cp1252`. This can be done by activating the Python [UTF-8 mode](https://www.python.org/dev/peps/pep-0540) either by passing the command-line option `-X utf8` or by setting the environment variable `PYTHONUTF8=1`.
Issue: [api request] API to obtain "dataset_module" dynamic path?

Comment:
Hi @richardliaw, 

First, thanks for the compliments.

In relation with your request, currently, the dynamic modules path is obtained this way:
```python
from datasets.load import init_dynamic_modules, MODULE_NAME_FOR_DYNAMIC_MODULES

dynamic_modules_path = init_dynamic_modules(MODULE_NAME_FOR_DYNAMIC_MODULES)
```

Let me know if it is OK for you this way. 

I could set `MODULE_NAME_FOR_DYNAMIC_MODULES` as default value, so that you could instead obtain the path with:
```
dynamic_modules_path = datasets.load.init_dynamic_modules()
```
Issue: [api request] API to obtain "dataset_module" dynamic path?

Comment:
Hi @richardliaw, the feature is on the master branch and will be included in the next release in a couple of weeks.
Issue: Unable to setup dev env on Windows

Comment:
Hi @gchhablani, 

There are some 3rd-party dependencies that require to build code in C. In this case, it is the library `python-Levenshtein`.

On Windows, in order to be able to build C code, you need to install at least `Microsoft C++ Build Tools` version 14. You can find more info here: https://visualstudio.microsoft.com/visual-cpp-build-tools/
Issue: Add VoxPopuli

Comment:
I'm happy to take this on:) One question: The original unlabelled data is stored unsegmented (see e.g. https://github.com/facebookresearch/voxpopuli/blob/main/voxpopuli/get_unlabelled_data.py#L30), but segmenting the audio in the dataset would require a dependency on something like soundfile or torchaudio. An alternative could be to provide the segments start and end times as a Sequence and then it's up to the user to perform the segmentation on-the-fly if they wish?
Issue: Add VoxPopuli

Comment:
Hey @jfainberg,

This sounds great! I think adding a dependency would not be a big problem, however automatically segmenting the data probably means that it would take a very long time to do:

```python
dataset = load_dataset("voxpopuli", "french")
```

=> so as a start I think your option 2 is the way to go!
Issue: Slow #0 when using map to tokenize.

Comment:
Hi ! Have you tried other values for `preprocessing_num_workers` ? Is it always process 0 that is slower ?
There are no difference between process 0 and the others except that it processes the first shard of the dataset.
Issue: Slow #0 when using map to tokenize.

Comment:
Hi, I have found the reason of it. Before using the map function to tokenize the data, I concatenate the wikipedia and bookcorpus first, like this:
```if args.dataset_name1 is not None:
        dataset1 = load_dataset(args.dataset_name1, args.dataset_config_name1, split="train")
        dataset1 = dataset1.remove_columns('title')
        if args.dataset_name2 is not None:
            dataset2 = load_dataset(args.dataset_name2, args.dataset_config_name2,split="train")
            assert dataset1.features.type == dataset2.features.type, str(dataset1.features.type)+';'+str(dataset2.features.type)
           datasets12 = concatenate_datasets([dataset1, dataset2], split='train')
```
When I just use one datasets, e.g. wikipedia, the problem seems no longer exist:
![image](https://user-images.githubusercontent.com/31714566/116967059-13d24380-ace4-11eb-8d14-b7b9c9a275cc.png)

Bookcorpus has more row numbers than Wikipedia, however, it takes much more time to process each batch of wiki than that of bookcorpus. When we first concatenate two datasets and then use _map_ to process the concatenated datasets,  e.g. `num_proc=5`, process 0 has to process all of the wikipedia data, causing the problem that #0 takes a longer time to finish the job. 

The problem is caused by the different characteristic of different datasets. One solution might be using _map_ first to process two datasets seperately, then concatenate the tokenized and processed datasets before input to the `Dataloader`.
Issue: Slow #0 when using map to tokenize.

Comment:
That makes sense ! You can indeed use `map` on both datasets separately and then concatenate.
Another option is to concatenate, then shuffle, and then `map`.
Issue: Load_dataset for local CSV files

Comment:
Hi,

this is not a standard CSV file (requires additional preprocessing) so I wouldn't label this as s bug. You could parse the examples with the regex module or the string API to extract the data, but the following approach is probably the easiest (once you load the data):
```python
import ast
# load the dataset and copy the features
def process(ex):
    return {"tokens": ast.literal_eval(ex["tokens"]), "labels": ast.literal_eval(ex["labels"])}
dataset = dataset.map(process, features=new_features)
```
Issue: Load_dataset for local CSV files

Comment:
Hi,

Thanks for the reply.
I  have already used  ```ast.literal_eval``` to evaluate the string into list, but I was getting another error:
```
ArrowInvalid: Could not convert X with type str: tried to convert to int
```
Why this happens ? Should labels be mapped to their ids and use int instead of str ?
Issue: Help understanding how to build a dataset for language modeling as with the old TextDataset

Comment:

I received an answer for this question on the HuggingFace Datasets forum by @lhoestq

Hi !

If you want to tokenize line by line, you can use this:

```
max_seq_length = 512
num_proc = 4

def tokenize_function(examples):
# Remove empty lines
examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
return tokenizer(
    examples["text"],
    truncation=True,
    max_length=max_seq_length,
)

tokenized_dataset = dataset.map(
tokenize_function,
batched=True,
num_proc=num_proc,
remove_columns=["text"],
)
```

Though the TextDataset was doing a different processing by concatenating all the texts and building blocks of size 512. If you need this behavior, then you must apply an additional map function after the tokenization:

```
# Main data processing function that will concatenate all texts from
# our dataset and generate chunks of max_seq_length.
def group_texts(examples):
# Concatenate all texts.
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
total_length = len(concatenated_examples[list(examples.keys())[0]])
# We drop the small remainder, we could add padding if the model supported it instead of this drop,
# you can customize this part to your needs.
total_length = (total_length // max_seq_length) * max_seq_length
# Split by chunks of max_len.
result = {
    k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]
    for k, t in concatenated_examples.items()
}
return result

# Note that with `batched=True`, this map processes 1,000 texts together,
# so group_texts throws away a remainder for each of those groups of 1,000 texts.
# You can adjust that batch_size here but a higher value might be slower to preprocess.

tokenized_dataset = tokenized_dataset.map(
group_texts,
batched=True,
num_proc=num_proc,
)
```

This code comes from the processing of the run_mlm.py example script of transformers
Issue: Compatibility with Ubuntu 18 and GLIBC 2.27?

Comment:
From the trace this seems like an error in the tokenizer library instead.

Do you mind opening an issue at https://github.com/huggingface/tokenizers instead?
Issue: Compatibility with Ubuntu 18 and GLIBC 2.27?

Comment:
Hi @tginart, thanks for reporting.

I think this issue is already open at `tokenizers` library: https://github.com/huggingface/tokenizers/issues/685
Issue: Loss result inGptNeoForCasual

Comment:
Hi ! I think you might have to ask on the `transformers` repo on or the forum at https://discuss.huggingface.co/

Closing since it's not related to this library
Issue: concatenate_datasets loads all the data into memory

Comment:
Hi ! this looks like an important issue. Let me try to reproduce this.
Cc @samsontmr this might be related to the memory issue you have in #2134