SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Datasets library not suitable for huge text datasets.

Comment:
Hi ! Looks related to #861 

You are right: tokenizing a dataset using map takes a lot of space since it can store `input_ids` but also `token_type_ids`, `attention_mask` and `special_tokens_mask`. Moreover if your tokenization function returns python integers then by default they'll be stored as int64 which can take a lot of space. Padding can also increase the size of the tokenized dataset.

To make things more convenient, we recently added a "lazy map" feature that allows to tokenize each batch at training time as you mentioned. For example you'll be able to do
```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

def encode(batch):
    return tokenizer(batch["text"], padding="longest", truncation=True, max_length=512, return_tensors="pt")

dataset.set_transform(encode)
print(dataset.format)
# {'type': 'custom', 'format_kwargs': {'transform': <function __main__.encode(batch)>}, 'columns': ['idx', 'label', 'sentence1', 'sentence2'], 'output_all_columns': False}
print(dataset[:2])
# {'input_ids': tensor([[  101,  2572,  3217, ... 102]]), 'token_type_ids': tensor([[0, 0, 0, ... 0]]), 'attention_mask': tensor([[1, 1, 1, ... 1]])}

```
In this example the `encode` transform is applied on-the-fly on the "text" column.

This feature will be available in the next release 2.0 which will happen in a few days.
You can already play with it by installing `datasets` from source if you want :)

Hope that helps !
Issue: Datasets library not suitable for huge text datasets.

Comment:
How recently was `set_transform` added? I am actually trying to implement it and getting an error:

`AttributeError: 'Dataset' object has no attribute 'set_transform'
`

I'm on v.1.2.1.

EDIT: Oh, wait I see now it's in the v.2.0. Whoops! This should be really useful.
Issue: Datasets library not suitable for huge text datasets.

Comment:
Yes indeed it was added a few days ago. The code is available on master
We'll do a release next week :)

Feel free to install `datasets` from source to try it out though, I would love to have some feedbacks
Issue: Datasets library not suitable for huge text datasets.

Comment:
For information: it's now available in `datasets` 1.3.0.
The 2.0 is reserved for even cooler features ;)
Issue: Datasets library not suitable for huge text datasets.

Comment:
Hi @alexvaca0 , we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.
Issue: Provide better exception message when one of many files results in an exception

Comment:
Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```
Issue: Loading local dataset raise requests.exceptions.ConnectTimeout

Comment:
Hi ! Thanks for reporting. This was indeed a bug introduced when we moved the `json` dataset loader inside the `datasets` package (before that, the `json` loader was fetched online, as all the other dataset scripts).

This should be fixed on master now. Feel free to install `datasets` from source to try it out.
The fix will be available in the next release of `datasets` in a few days
Issue: pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500

Comment:
Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```
Issue: Unable to add Multi-label Datasets

Comment:
Thanks for adding this dataset! As far as I know `supervised_keys` is mostly a holdover from TFDS, but isn't really used, so feel free to drop it (@lhoestq  or @thomwolf correct me if I'm wrong). It definitely shouldn't be blocking :)
Issue: Unable to add Multi-label Datasets

Comment:
Thanks @yjernite @lhoestq 

The template for new dataset makes it slightly confusing. I suppose the comment suggesting its update can be removed.
Issue: Add Hateful Memes Dataset

Comment:
Hi @gchhablani since Array2D doesn't support images of different sizes, I would suggest to store in the dataset the paths to the image file instead of the image data. This has the advantage of not decompressing the data (images are often compressed using jpeg, png etc.). Users can still apply `.map` to load the images if they want to. Though it would en up being Sequences features.

In the future we'll add support for ragged tensors for this case and update the relevant dataset with this feature.
Issue: writing Datasets in a human readable format

Comment:
AFAIK, there is currently no built-in method on the `Dataset` object to do this.
However, a workaround is to directly use the Arrow table backing the dataset, **but it implies loading the whole dataset in memory** (correct me if I'm mistaken @lhoestq).

You can convert the Arrow table to a pandas dataframe to save the data as csv as follows:
```python
arrow_table = dataset.data
dataframe = arrow_table.to_pandas()
dataframe.to_csv("/path/to/file.csv")
```

Similarly, you can convert the dataset to a Python dict and save it as JSON:
```python
import json
arrow_table = dataset.data
py_dict = arrow_table.to_pydict()
with open("/path/to/file.json", "w+") as f:
    json.dump(py_dict, f)
```
Issue: writing Datasets in a human readable format

Comment:
Indeed this works as long as you have enough memory.
It would be amazing to have export options like csv, json etc. !

It should be doable to implement something that iterates through the dataset batch by batch to write to csv for example.
There is already an `export` method but currently the only export type that is supported is `tfrecords`.
Issue: can't pickle SwigPyObject objects when calling dataset.get_nearest_examples from FAISS index

Comment:
Hi ! Indeed we used to require mapping functions to be picklable with `pickle` or `dill` in order to cache the resulting datasets. And FAISS indexes are not picklable unfortunately.

But since #1703 this is no longer required (the caching will simply be disabled). This change will be available in the next release of `datasets`, or you can also install `datasets` from source.
Issue: can't pickle SwigPyObject objects when calling dataset.get_nearest_examples from FAISS index

Comment:
I totally forgot to answer this issue, I'm so sorry. 

I was able to get it working by installing `datasets` from source. Huge thanks!
Issue: Querying examples from big datasets is slower than small datasets

Comment:
Hello, @lhoestq / @gaceladri  : We have been seeing similar behavior with bigger datasets, where querying time increases. Are you folks aware of any solution that fixes this problem yet?
Issue: Querying examples from big datasets is slower than small datasets

Comment:
Hi ! I'm pretty sure that it can be fixed by using the Arrow IPC file format instead of the raw streaming format but I haven't tested yet.
I'll take a look at it soon and let you know
Issue: Querying examples from big datasets is slower than small datasets

Comment:
My workaround is to shard the dataset into splits in my ssd disk and feed the data in different training sessions. But it is a bit of a pain when we need to reload the last training session with the rest of the split with the Trainer in transformers.

I mean, when I split the training and then reloads the model and optimizer, it not gets the correct global_status of the optimizer, so I need to hardcode some things. I'm planning to open an issue in transformers and think about it.
```
from datasets import load_dataset

book_corpus = load_dataset("bookcorpus", split="train[:25%]")
wikicorpus = load_dataset("wikicorpus", split="train[:25%]")
openwebtext = load_dataset("openwebtext", split="train[:25%]")

big_dataset = datasets.concatenate_datasets([wikicorpus, openwebtext, book_corpus])
big_dataset.shuffle(seed=42)
big_dataset = big_dataset.map(encode, batched=True, num_proc=20, load_from_cache_file=True, writer_batch_size=5000)
big_dataset.set_format(type='torch', columns=["text", "input_ids", "attention_mask", "token_type_ids"])


training_args = TrainingArguments(
    output_dir="./linear_bert",
    overwrite_output_dir=True,
    per_device_train_batch_size=71,
    save_steps=500,
    save_total_limit=10,
    logging_first_step=True,
    logging_steps=100,
    gradient_accumulation_steps=9,
    fp16=True,
    dataloader_num_workers=20,
    warmup_steps=24000,
    learning_rate=0.000545205002870214,
    adam_epsilon=1e-6,
    adam_beta2=0.98,
    weight_decay=0.01,
    max_steps=138974,  # the total number of steps after concatenating 100% datasets
    max_grad_norm=1.0,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=big_dataset,
    tokenizer=tokenizer))
```

I do one training pass with the total steps of this shard and I use len(bbig)/batchsize to stop the training (hardcoded in the trainer.py) when I pass over all the examples in this split.

Now Im working, I will edit the comment with a more elaborated answer when I left the work.
Issue: Querying examples from big datasets is slower than small datasets

Comment:
I just tested and using the Arrow File format doesn't improve the speed... This will need further investigation.

My guess is that it has to iterate over the record batches or chunks of a ChunkedArray in order to retrieve elements.

However if we know in advance in which chunk the element is, and at what index it is, then we can access it instantaneously. But this requires dealing with the chunked arrays instead of the pyarrow Table directly which is not practical.
Issue: Querying examples from big datasets is slower than small datasets

Comment:
I have a dataset with about 2.7 million rows (which I'm loading via `load_from_disk`), and I need to fetch around 300k (particular) rows of it, by index. Currently this is taking a really long time (~8 hours). I tried sharding the large dataset but overall it doesn't change how long it takes to fetch the desired rows.

I actually have enough RAM that I could fit the large dataset in memory. Would having the large dataset in memory speed up querying? To find out, I tried to load (a column of) the large dataset into memory like this:
```
column_data = large_ds['column_name']
```
but in itself this takes a really long time.

I'm pretty stuck - do you have any ideas what I should do?
Issue: Querying examples from big datasets is slower than small datasets

Comment:
Hi ! Feel free to post a message on the [forum](https://discuss.huggingface.co/c/datasets/10). I'd be happy to help you with this.

In your post on the forum, feel free to add more details about your setup:
What are column names and types of your dataset ?
How was the dataset constructed ?
Is the dataset shuffled ?
Is the dataset tokenized ?
Are you on a SSD or an HDD ?

I'm sure we can figure something out.
For example on my laptop I can access the 6 millions articles from wikipedia in less than a minute.
Issue: Connection error

Comment:
Hi ! For future references let me add a link to our discussion here : https://github.com/huggingface/datasets/issues/759#issuecomment-770684693

Let me know if you manage to fix your proxy issue or if we can do something on our end to help you :)
Issue: Filter on dataset too much slowww

Comment:
When I use the filter on the arrow table directly, it works like butter. But I can't find a way to update the table in `Dataset` object.

```
ds_table = dataset.data.filter(mask=dataset['flag'])
```
Issue: Filter on dataset too much slowww

Comment:
Hi ! Currently the filter method reads the dataset batch by batch to write a new, filtered, arrow file on disk. Therefore all the reading + writing can take some time.
Using a mask directly on the arrow table doesn't do any read or write operation therefore it's way quicker.

Replacing the old table by the new one should do the job:
```python
dataset._data = dataset._data.filter(...)
```

Note: this is a **workaround** and in general users shouldn't have to do that. In particular if you did some `shuffle` or `select` before that then it would not work correctly since the indices mapping (index from `__getitem__` -> index in the table) would not be valid anymore. But if you haven't done any `shuffle`, `select`, `shard`, `train_test_split` etc. then it should work.

Ideally it would be awesome to update the filter function to allow masking this way !
If you would like to give it a shot I will be happy to help :)
Issue: Filter on dataset too much slowww

Comment:
Hi @lhoestq @ayubSubhaniya,

If there's no progress on this one, can I try working on it?

Thanks,
Gunjan
Issue: Filter on dataset too much slowww

Comment:
Sure @gchhablani feel free to start working on it, this would be very appreciated :)
This feature is would be really awesome, especially since arrow allows to mask really quickly and without having to rewrite the dataset on disk
Issue: ModuleNotFoundError: No module named 'apache_beam', when specific languages.

Comment:
Hi !

Apache Beam is a framework used to define data transformation pipelines. These pipeline can then be run in many runtimes: DataFlow, Spark, Flink, etc. There also exist a local runner called the DirectRunner.
Wikipedia is a dataset that requires some parsing, so to allow the processing to be run on this kind of runtime we're using Apache Beam.

At Hugging Face we've already processed certain versions of wikipedia (the `20200501.en` one for example) so that users can directly download the processed version instead of using Apache Beam to process it.
However for the japanese language we haven't processed it so you'll have to run the processing on your side.
So you do need Apache Beam to process `20200501.ja`.

You can install Apache Beam with
```
pip install apache-beam
```

I think we can probably improve the error message to let users know of this subtlety.
What #498 implied is that Apache Beam is not needed when you process a dataset that doesn't use Apache Beam.
Issue: ModuleNotFoundError: No module named 'apache_beam', when specific languages.

Comment:
Hi @miyamonz,

I tried replicating this issue using the same snippet used by you. I am able to download the dataset without any issues, although I stopped it in the middle because the dataset is huge.

Based on a similar issue [here](https://github.com/google-research/fixmatch/issues/23), it could be related to your environment setup, although I am just guessing here. Can you share these details?
Issue: ModuleNotFoundError: No module named 'apache_beam', when specific languages.

Comment:
I don't know if this is related, but there is this issue on the wikipedia processing that you reported at #2031 (open PR is at #2037 ) .
Does the fix your proposed at #2037 helps in your case ?

And for information, the DirectRunner of Apache Beam is not optimized for memory intensive tasks, so you must be right when you say that it uses the memory for the entire file.
Issue: ModuleNotFoundError: No module named 'apache_beam', when specific languages.

Comment:
the #2037 doesn't solve my problem directly, but I found the point!

https://github.com/huggingface/datasets/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/datasets/wikipedia/wikipedia.py#L523
this `beam.transforms.Reshuffle()` cause the memory error.

it makes sense if I consider the shuffle means. Beam's reshuffle seems need put all data in memory.
Previously I doubt that this line causes error, but at that time another bug showed in #2037 made error, so I can't found it.

Anyway, I comment out this line, and run load_dataset, then it works!

```python
wiki = datasets.load_dataset(
    "./wikipedia.py",
    cache_dir="./datasets",
    beam_runner="DirectRunner",
    language="ja",
    date="20210120",
)["train"]
```
![image](https://user-images.githubusercontent.com/6331508/112283369-6a9f3300-8ccb-11eb-82e5-827bf7fddfb9.png)

Dataset has already shuffle function. https://github.com/huggingface/datasets/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/arrow_dataset.py#L2069
So, though I don't know it's difference correctly, but I think Beam's reshuffle isn't be needed. How do you think?
Issue: ModuleNotFoundError: No module named 'apache_beam', when specific languages.

Comment:
The reshuffle is needed when you use parallelism.
The objective is to redistribute the articles evenly on the workers, since the `_extract_content` step generated many articles per file. By using reshuffle, we can split the processing of the articles of one file into several workers. Without reshuffle, all the articles of one file would be processed on the same worker that read the file, making the whole process take a very long time.
Issue: ModuleNotFoundError: No module named 'apache_beam', when specific languages.

Comment:
Maybe the reshuffle step can be added only if the runner is not a DirectRunner ?
Issue: How to use split dataset 

Comment:
By default, all 3 splits will be loaded if you run the following:

```python
from datasets import load_dataset
dataset = load_dataset("lambada")
print(dataset["train"])
print(dataset["valid"])

```

If you wanted to do load this manually, you could do this:

```python
from datasets import load_dataset
data_files = {
   "train": "data/lambada/train.txt",
   "valid": "data/lambada/valid.txt",
   "test": "data/lambada/test.txt",
}
ds = load_dataset("text", data_files=data_files)
```
Issue: Not enough disk space (Needed: Unknown size) when caching on a cluster

Comment:
Hi ! 

What do you mean by "disk_usage(".").free` can't compute on the cluster's shared disk" exactly ?
Does it return 0 ?
Issue: Not enough disk space (Needed: Unknown size) when caching on a cluster

Comment:
Yes, that's right. It shows 0 free space even though there is. I suspect it might have to do with permissions on the shared disk.

```python
>>> disk_usage(".")
usage(total=999999, used=999999, free=0)
```
Issue: Not enough disk space (Needed: Unknown size) when caching on a cluster

Comment:
That's an interesting behavior...
Do you know any other way to get the free space that works in your case ?
Also if it's a permission issue could you try fix the permissions and let mus know if that helped ?
Issue: Not enough disk space (Needed: Unknown size) when caching on a cluster

Comment:
I think its an issue on the clusters end (unclear exactly why -- maybe something with docker containers?), will close the issue
Issue: Not enough disk space (Needed: Unknown size) when caching on a cluster

Comment:
@philippnoah I had fixed it with a small hack where I patched `has_sufficient_disk_space` to always return `True`. you can do that with an import without having to modify the `datasets` package
Issue: Not enough disk space (Needed: Unknown size) when caching on a cluster

Comment:
@olinguyen Thanks for the suggestion, it works but I had to to edit builder.py in the installed package. Can you please explain how were you able to do this using import?
Issue: Not enough disk space (Needed: Unknown size) when caching on a cluster

Comment:
I was able to patch the builder code in my notebook before the load data call and it works. 
```
import datasets
datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
```
Issue: JSONDecodeError on JSON with multiple lines

Comment:
Hi !

The `json` dataset script does support this format. For example loading a dataset with this format works on my side:
```json
{"key1":11, "key2":12, "key3":13}
{"key1":21, "key2":22, "key3":23}
```

Can you show the full stacktrace please ? Also which version of datasets and pyarrow are you using ?
Issue: JSONDecodeError on JSON with multiple lines

Comment:
Hi Quentin!

I apologize for bothering you. There was some issue with my pyarrow version as far as I understand. I don't remember the exact version I was using as I didn't check it.

I repeated it with `datasets 1.2.1` and `pyarrow  2.0.0` and it worked.

Closing this issue. Again, sorry for the bother.

Thanks,
Gunjan
Issue: Dataset Examples Explorer

Comment:
Hi @ChewKokWah,

We're working on it! In the meantime, you can still find the dataset explorer at the following URL: https://huggingface.co/datasets/viewer/
Issue: Dataset Examples Explorer

Comment:
Glad to see that it still exist, this existing one is more than good enough for me, it is feature rich, simple to use and concise. 
Hope similar feature can be retain in the future version.
Issue: AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 

Comment:
Hi ! I'm not able to reproduce the issue. Can you try restarting your runtime ?

The PyExtensionType is available in pyarrow starting 0.17.1 iirc. If restarting your runtime doesn't fix this, can you try updating pyarrow ?
```
pip install pyarrow --upgrade
```
Issue: AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 

Comment:
Yes indeed.

Also it looks like Pyarrow 3.0.0 got released on pypi 10 hours ago. This might be related to the bug, I'll investigate
EDIT: looks like the 3.0.0 release doesn't have unexpected breaking changes for us, so I don't think the issue comes from that
Issue: AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 

Comment:
Installing datasets installs pyarrow>=0.17.1 so in theory it doesn't matter which version of pyarrow colab has by default (which is currently pyarrow 0.14.1).

Also now the colab runtime refresh the pyarrow version automatically after the update from pip (previously you needed to restart your runtime).

I guess what happened is that Colab didn't refresh pyarrow for some reason, and the AttributeError was raised *before* the pyarrow version check from `datasets` at https://github.com/huggingface/datasets/blob/master/src/datasets/__init__.py#L60
Issue: AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 

Comment:
Yes colab doesn’t reload preloaded library unless you restart the instance. Maybe we should move the check on top of the init
Issue: [Question & Bug Report] Can we preprocess a dataset on the fly?

Comment:
We are very actively working on this. How does your dataset look like in practice (number/size/type of files)?
Issue: [Question & Bug Report] Can we preprocess a dataset on the fly?

Comment:
It's a text file with many lines (about 1B) of Chinese sentences. I use it to train language model using https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm_wwm.py