SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: train_test_split returns empty dataset item

Comment:
We'll do a release pretty soon to include the fix :)
In the meantime you can install the lib from source if you want to
Issue: load_dataset() won't download in Windows

Comment:
I have the same issue. Tried to download a few of them and not a single one is downloaded successfully.

This is the output:
```
>>> dataset = load_dataset('blended_skill_talk', split='train')
Using custom data configuration default               <-- This step never ends
```
Issue: load_dataset() won't download in Windows

Comment:
This was fixed in #644 
I'll do a new release soon :)

In the meantime you can run it by installing from source
Issue: load_dataset() won't download in Windows

Comment:
Closing since version 1.1.0 got released with Windows support :) 
Let me know if it works for you now
Issue: Questions about XSUM 

Comment:
We should try to regenerate the data using the official script.
But iirc that's what we used in the first place, so not sure why it didn't match in the first place.

I'll let you know when the dataset is updated
Issue: Questions about XSUM 

Comment:
Thanks, looking forward to hearing your update on this thread. 

This is a blocking issue for us; would appreciate any progress on this front. We can also help with the fix, if you deem it appropriately.
Issue: Questions about XSUM 

Comment:
I just started the generation on my side, I'll let you know how it goes :)
Issue: Questions about XSUM 

Comment:
Hmm after a first run I'm still missing 136668/226711 urls.
I'll relaunch it tomorrow to try to get the remaining ones.
Issue: Questions about XSUM 

Comment:
So I managed to download them all but when parsing only 226,181/226,711 worked.
Not sure if it's worth digging and debugging parsing at this point :/
Issue: Questions about XSUM 

Comment:
Thanks @lhoestq
It would be great to improve coverage, but IDs are the really crucial part for us. We'd really appreciate an update to the dataset with IDs either way!
Issue: Questions about XSUM 

Comment:
I gave up at an even earlier point. The dataset I use has 204,017 train examples.
Issue: Questions about XSUM 

Comment:
@lhoestq @sshleifer like @jbragg said earlier, the main issue for us is that the current XSUM dataset (in your package) does not have IDs suggested by the original dataset ([here is the file](https://raw.githubusercontent.com/EdinburghNLP/XSum/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json).) Would appreciate if you update the XSUM dataset to include the instance IDs. 

The missing instances is also a problem, but likely not worth pursuing given its relatively small scale.
Issue: Questions about XSUM 

Comment:
>So I managed to download them all but when parsing only 226,181/226,711 worked.

@lhoestq any chance we could update the HF-hosted dataset with the IDs in your new version? Happy to help if there's something I can do.
Issue: Questions about XSUM 

Comment:
Well I couldn't parse what I downloaded.
Unfortunately I think I won't be able to take a look at it this week.
I can try to send you what I got if you want to give it a shot @jbragg 
Otherwise feel free to re-run the xsum download script, maybe you'll be luckier than me
Issue: How to skip a example when running dataset.map

Comment:
Hi @xixiaoyao,
Depending on what you want to do you can:
- use a first step of `filter` to filter out the invalid examples: https://huggingface.co/docs/datasets/processing.html#filtering-rows-select-and-filter
- or directly detect the invalid examples inside the callable used with `map` and return them unchanged or even remove them at the same time if you are using `map` in batched mode. Here is an example where we use `map` in batched mode to add new rows on the fly but you can also use it to remove examples on the fly (that's what `filter` actually do under-the-hood): https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset
Issue: Loss not decrease with Datasets and Transformers

Comment:
Hi did you manage to fix your issue ?

If so feel free to share your fix and close this thread
Issue: Does both 'bookcorpus' and 'wikipedia' belong to the same datasets which Google used for pretraining BERT?

Comment:
No they are other similar copies but they are not provided by the official Bert models authors.
Issue: runing dataset.map, it raises TypeError: can't pickle Tokenizer objects

Comment:
Hi !
It works on my side with both the LongFormerTokenizer and the LongFormerTokenizerFast.

Which version of transformers/datasets are you using ?
Issue: runing dataset.map, it raises TypeError: can't pickle Tokenizer objects

Comment:
Then I guess you need to give us more informations on your setup (OS, python, GPU, etc) or a Google Colab reproducing the error for us to be able to debug this error.
Issue: runing dataset.map, it raises TypeError: can't pickle Tokenizer objects

Comment:
> I have the same issue with `transformers/BertJapaneseTokenizer`.

It looks like it this tokenizer is not supported unfortunately.
This is because `t.word_tokenizer.mecab` is a `fugashi.fugashi.GenericTagger` which is not compatible with pickle nor dill.

We need objects passes to `map` to be picklable for our caching system to work properly.
Here it crashes because the caching system is not able to pickle the GenericTagger.

\> Maybe you can create an issue on [fugashi](https://github.com/polm/fugashi/issues) 's repo and ask to make `fugashi.fugashi.GenericTagger` compatible with pickle ?

What you can do in the meantime is use a picklable wrapper of the tokenizer:


```python
from transformers import BertJapaneseTokenizer, MecabTokenizer

class PicklableTokenizer(BertJapaneseTokenizer):

    def __getstate__(self):
        state = dict(self.__dict__)
        state["do_lower_case"] = self.word_tokenizer.do_lower_case
        state["never_split"] = self.word_tokenizer.never_split 
        del state["word_tokenizer"]
        return state

    def __setstate__(self, state):
        do_lower_case = state.pop("do_lower_case")
        never_split = state.pop("never_split")
        self.__dict__ = state
        self.word_tokenizer = MecabTokenizer(
            do_lower_case=do_lower_case, never_split=never_split)
        )

t = PicklableTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")
encoded = train_ds.map(lambda examples: {'tokens': t.encode(examples['title'])}, batched=True)  # it works
```
Issue: runing dataset.map, it raises TypeError: can't pickle Tokenizer objects

Comment:
We can also update the `BertJapaneseTokenizer` in `transformers` as you just shown @lhoestq to make it compatible with pickle. It will be faster than asking on fugashi 's repo and good for the other users of `transformers` as well.

I'm currently working on `transformers` I'll include it in the https://github.com/huggingface/transformers/pull/7141 PR and the next release of `transformers`.
Issue: runing dataset.map, it raises TypeError: can't pickle Tokenizer objects

Comment:
Thank you for the rapid and polite response!

@lhoestq Thanks for the suggestion! I've passed the pickle phase, but another `ArrowInvalid` problem occored. I created another issue #687 .

@thomwolf Wow, really fast work. I'm looking forward to the next release 🤗
Issue: load_dataset from local squad.py, raise error: TypeError: 'NoneType' object is not callable 

Comment:
Hi !
Thanks for reporting.
It looks like no object inherits from `datasets.GeneratorBasedBuilder` (or more generally from `datasets.DatasetBuilder`) in your script.

Could you check that there exist at least one dataset builder class ?
Issue: load_dataset from local squad.py, raise error: TypeError: 'NoneType' object is not callable 

Comment:
It happened when try to change the old project which use 'nlp' to new project which use 'datasets'. You should check you old 'my_squad.py' file, change the inherit class from `nlp.xxx` to `datasets.xxx`. Otherwise datasets - load.py - import_main_class() `if inspect.isclass(obj) and issubclass(obj, main_cls_type):` can not find the main_cls.
Issue: Squad Metric Description & Feature Mismatch

Comment:
Thanks for reporting !
There indeed a mismatch between the features and the kwargs description

I believe `answer_start` was added to match the squad dataset format for consistency, even though it is not used in the metric computation. I think I'd rather keep it this way, so that you can just give `references=squad["answers"]` to `.compute()`.
Maybe we can just fix the description then.
Issue: Squad Metric Description & Feature Mismatch

Comment:
But then providing the `answer_start` becomes mandatory since the format of the features is checked against the one provided in the squad [file](https://github.com/huggingface/datasets/pull/658/files).
Issue: Problem with JSON dataset format

Comment:
Currently the `json` dataset doesn't support this format unfortunately.
However you could load it with
```python
from datasets import Dataset
import pandas as pd

df = pd.read_json("path_to_local.json", orient="index")
dataset = Dataset.from_pandas(df)
```
Issue: dummy data testing can't test datasets using `dl_manager.extract` in `_split_generators`

Comment:
Hi :) 
In your dummy data zip file you can just have `subset000.xz` as directories instead of compressed files.
Let me know if it helps
Issue: dummy data testing can't test datasets using `dl_manager.extract` in `_split_generators`

Comment:
Thanks for your comment @lhoestq ,
Just for confirmation, changing dummy data like this won't make dummy test test the functionality to extract `subsetxxx.xz` but actually kind of circumvent it. But since we will test the real data so it is ok ?
Issue: dummy data testing can't test datasets using `dl_manager.extract` in `_split_generators`

Comment:
Yes it's fine for now. We plan to add a job for slow tests.
And at one point we'll also do another pass on the dummy data handling and consider extracting files.
Issue: Inconsistent behavior in map

Comment:
Thanks for reporting !

This issue must have appeared when we refactored type inference in `nlp`
By default the library tries to keep the same feature types when applying `map` but apparently it has troubles with nested structures. I'll try to fix that next week
Issue: Cannot download dataset_info.json

Comment:
Thanks for reporting !
We should add support for servers without internet connection indeed
I'll do that early next week
Issue: Cannot download dataset_info.json

Comment:
Right now the recommended way is to create the dataset on a server with internet connection and then to save it and copy the serialized dataset to the server without internet connection.
Issue: Cannot download dataset_info.json

Comment:
#652 should allow you to load text/json/csv/pandas datasets without an internet connection **IF** you've the dataset script locally.

Example: 
If you have `datasets/text/text.py` locally, then you can do `load_dataset("./datasets/text", data_files=...)`
Issue: Caching processed dataset at wrong folder

Comment:
Thanks for reporting !
It uses a temporary file to write the data.
However it looks like the temporary file is not placed in the right directory during the processing
Issue: Caching processed dataset at wrong folder

Comment:
Well actually I just tested and the temporary file is placed in the same directory, so it should work as expected.
Which version of `datasets` are you using ?
Issue: Caching processed dataset at wrong folder

Comment:
It looks like a pyarrow issue with google colab.
For some reason this code increases the disk usage of google colab while it actually writes into google drive:

```python
import pyarrow as pa

stream = pa.OSFile("/content/drive/My Drive/path/to/file.arrow", "wb")
writer = pa.RecordBatchStreamWriter(stream, schema=pa.schema({"text": pa.string()}))
writer.write_table(pa.Table.from_pydict({"text": ["a"*511 + "\n"] * ((1 << 30) // 512)}))  # 1GiB
writer.close()
stream.close()
```

Moreover if I `rm` the file on google drive, it frees disk space on google colab.
Issue: Caching processed dataset at wrong folder

Comment:
Actually I did more tests it doesn't >.<
I'll let you know if I find a way to fix that
Issue: Caching processed dataset at wrong folder

Comment:
Actually I also have the issue when writing a regular text file

```python
f = open("/content/drive/My Drive/path/to/file", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```

Is that supposed to happen ?
Issue: Caching processed dataset at wrong folder

Comment:
The code you wrote should write a 1GB file in the Google Drive folder. Doesn't it?
Issue: Caching processed dataset at wrong folder

Comment:
I could check it and as you say as I write to te Drive disk the colab disk also increases...
Issue: Caching processed dataset at wrong folder

Comment:
To reproduce it: 
```bash
!df -h | grep sda1
```
```python
f = open("/content/drive/My Drive/test_to_remove.txt", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```
```bash
!ls -lh /content/drive/My\ Drive/test_to_remove.txt

!df -h | grep sda1

!rm -rf /content/drive/My\ Drive/test_to_remove.txt

```
[Colab](https://colab.research.google.com/drive/1D0UiweCYQwwWZ65EEhuqqbaDDbhJYXfm?usp=sharing)
Issue: Caching processed dataset at wrong folder

Comment:
Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Not sure what could cause that on the `datasets` side. Could this be a `Trainer` issue ? cc @julien-c @sgugger ?
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
There was a memory leak issue fixed recently in master. You should install from source and see if it fixes your problem.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
@lhoestq @sgugger Thanks for your comments. I have install from source code as you told, but the problem is still there.
To reproduce the issue, just replace [these lines](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L241-L258) with: 
(load_dataset and DataCollatorForDatasetsLanguageModeling as [above mentioned](https://github.com/huggingface/datasets/issues/633#issue-702440484))
```python
    dataset = load_dataset("bookcorpus")
    dataset = dataset.train_test_split(test_size=0.1)
    train_dataset = dataset['train']
    eval_dataset = dataset['test'] if training_args.do_eval else None

    data_collator = DataCollatorForDatasetsLanguageModeling(
        tokenizer=tokenizer,
        mlm=data_args.mlm,
        mlm_probability=data_args.mlm_probability,
        block_size=data_args.block_size
    )
```
and run by:
```bash
python run_language_modeling.py
--output_dir=output \
--model_type=bert \
--model_name_or_path=bert-base-uncased \
--do_train \
--do_eval \
--mlm 
```
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Same here. Pre-training on wikitext-103 to do some test. At the end of the training it takes 32GB of RAM + ~30GB of SWAP. I installed dataset==1.1.0, not built from source. I will try uninstalling and building from source when it finish.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
This seems to be on the `transformers` library side.

If you have more informations (pip env) or even better, a colab reproducing the error we can investigate.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
It seems like it's solved with freshed versions of transformers. I have tried to replicate the error doing a fresh pip install transformers & datasets on colab and the error doesn't continue. On colab it keeps stable on 5GB! (Y)

Edit: **Thanks for your great work**. Have a good day.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
@gaceladri witch version transformers and datasets  are you using now? I want to try again. Thanks.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
It's happening to me again. After 4 hours of pre-training, my ram memory gets full and the kernel dies. I am using the last transformers version as today. 4.4.0 and the last version of datasets 1.2.1, both installed from master. The memory consumption keeps increasing.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Thanks for the investigation @gaceladri 

Apparently this happens when `num_workers>0` and has to do with objects being copied-on-write.
Did you try setting num_workers to 0 @gaceladri ?
If the issue doesn't happen with `num_workers=0` then this would confirm that it's indeed related to this python/pytorch issue.

Since a `Dataset` object is a wrapper of a pyarrow Table, we should investigate if the data being copied comes from the Table itself or from metadata in the `Dataset` object. If it comes from the metadata in the `Dataset` object, we should be able to implement a workaround. But if it comes from the Table, we'll need to see with the pyarrow team what we can do...
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Hmmm so this might come from another issue...
Since it doesn't seem to be related to multiprocessing it should be easier to investigate though.
Do you have some ideas @gaceladri ?
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
@lhoestq But despite this, I got lost into the [class Dataset()](https://huggingface.co/docs/datasets/_modules/datasets/arrow_dataset.html#Dataset) reading the pyarrow files.

Edit: but you should be rigth, that it does not have to be related to multiprocessing since it keeps happening when `num_workers=0`
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Or maybe wandb uses multiprocessing ? One process for wandb logging and one for actual training ? If this is the case then even setting `num_workers=0` would cause the process to be forked for wandb and therefore cause the memory issue.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
@lhoestq could be, but if we set wandb to false this should not happen. I am going to try.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
@lhoestq It keeps happening. I have uninstalled wandb from my env, setted `%env WANDB_DISABLED=true` on my notebook, and commented this func:

```
def get_available_reporting_integrations():
    integrations = []
    if is_azureml_available():
        integrations.append("azure_ml")
    if is_comet_available():
        integrations.append("comet_ml")
    if is_mlflow_available():
        integrations.append("mlflow")
    if is_tensorboard_available():
        integrations.append("tensorboard")
    # if is_wandb_available():
    #     integrations.append("wandb")
    return integrations
```
As a fast test and it keeps increasing the ram memory. Wandb could not be the blameworthy here.
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Thanks for checking @gaceladri . Let's investigate the single process setting then.
If you have some sort of colab notebook with a minimal code example that shows this behavior feel free to share it @gaceladri so that we can play around with it to find what causes this. Otherwise I'll probably try to reproduce on my side at one point
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
@lhoestq sure. Here you have https://colab.research.google.com/drive/1ba09ZOpyHGAOQLcsxiQAHRXl10qnMU5o?usp=sharing let me know if the link works and it reproduces the issue. To me, it reproduces the issue, since if you start the training the ram memory keeps increasing.

Let me know. Thanks!
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Could the bug be comming from tokenizers?

I got this warning at the terminal from my jupyter notebook: 
```
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
```
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
I've never experienced memory issues with tokenizers so I don't know
Cc @n1t0 are you aware of any issue that would cause memory to keep increasing when the tokenizer is used in the Data Collator for language modeling ?
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Thanks for sharing your results.
So you still had the issue for fine-tuning ?
And the issue still appears with a bare-bone dataset from an arrow file...
Issue: Load large text file for LM pre-training resulting in OOM

Comment:
Yes, on both cases. Fine-tuning a pre-trained model and pre-training from scratch with a local arrow file already pre-processed.
Issue: Text dataset not working with large files

Comment:
Basically ~600MB txt files(UTF-8) * 59. 
contents like ```안녕하세요, 이것은 예제로 한번 말해보는 텍스트입니다. 그냥 이렇다고요.
\n```

Also, it gets stuck for a loooong time at ```Testing the mapped function outputs```, for more than 12 hours(currently ongoing)
Issue: Text dataset not working with large files

Comment:
It gets stuck while doing `.map()` ? Are you using multiprocessing ?
If you could provide a code snippet it could be very useful
Issue: Text dataset not working with large files

Comment:
From transformers/examples/language-modeling/run-language-modeling.py :
```
def get_dataset(
    args: DataTrainingArguments,
    tokenizer: PreTrainedTokenizer,
    evaluate: bool = False,
    cache_dir: Optional[str] = None,
):
    file_path = args.eval_data_file if evaluate else args.train_data_file
    if True:
        dataset = load_dataset("text", data_files=glob.glob(file_path), split='train', use_threads=True, 
        ignore_verifications=True, save_infos=True, block_size=104857600)
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                                truncation=True, max_length=args.block_size), batched=True)
        dataset.set_format(type='torch', columns=['input_ids'])
        return dataset
    if args.line_by_line:
        return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
    else:
        return TextDataset(
            tokenizer=tokenizer,
            file_path=file_path,
            block_size=args.block_size,
            overwrite_cache=args.overwrite_cache,
            cache_dir=cache_dir,
        )
```

No, I'm not using multiprocessing.
Issue: Text dataset not working with large files

Comment:
I am not able to reproduce on my side :/

Could you send the version of `datasets` and `pyarrow` you're using ?
Could you try to update the lib and try again ?
Or do you think you could try to reproduce it on google colab ?
Issue: Text dataset not working with large files

Comment:
Huh, weird. It's fixed on my side too.
But now ```Caching processed dataset``` is taking forever - how can I disable it? Any flags?
Issue: Text dataset not working with large files

Comment:
Right after `Caching processed dataset`, your function is applied to the dataset and there's a progress bar that shows how much time is left. How much time does it take for you ?

Also caching isn't supposed to slow down your processing. But if you still want to disable it you can do `.map(..., load_from_cache_file=False)`
Issue: Text dataset not working with large files

Comment:
Ah, it’s much faster now(Takes around 15~20min). 
BTW, any way to set default tensor output as plain tensors with distributed training? The ragged tensors are incompatible with tpustrategy :(