SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Compare different Rouge implementations  

Comment:
> This is a real issue, sorry for missing the mention @ibeltagy
> 
> We implemented a more involved [solution](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L481) that enforces that sentences are split with `\n` so that rougeLsum scores match papers even if models don't generate newlines.
> 
> Unfortunately, the best/laziest way I found to do this introduced an `nltk` dependency (For sentence splitting, all sentences don't end in `.`!!!), but this might be avoidable with some effort.

Thanks for the details, I didn't know about that. Maybe we should consider adding this processing step or at least mention it somewhere in the library or the documentation

> #### Sidebar: Wouldn't Deterministic Be Better?
> `rouge_scorer.scoring.BootstrapAggregator` is well named but is not deterministic which I would like to change for my mental health, unless there is some really good reason to sample 500 observations before computing f-scores.
> 
> I have a fix on a branch, but I wanted to get some context before introducting a 4th way to compute rouge. Scores are generally within .03 Rouge2 of boostrap after multiplying by 100, e.g 22.05 vs 22.08 Rouge2.

I think the default `n_samples` of the aggregator is 1000. We could increase it or at least allow users to change it if they want more precise results.
Issue: Compare different Rouge implementations  

Comment:
Hi, thanks for the solution. 

I am not sure if this is a bug, but on line [510](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L510), are pred, tgt supposed to be swapped?
Issue: Compare different Rouge implementations  

Comment:
Hi, so I took this example from the HF implementation. What I can see is that  the precision of `Hello there` being summarized to `general kenobi` is 1. I don't understand how this calculation is correct.
Is the comparison just counting the words?
and if Yes, then how does this translates to summarization evaluation?
```
    >>> rouge = datasets.load_metric('rouge')
    >>> predictions = ["hello there", "general kenobi"]
    >>> references = ["hello there", "general kenobi"]
    >>> results = rouge.compute(predictions=predictions, references=references)
    >>> print(list(results.keys()))
    ['rouge1', 'rouge2', 'rougeL', 'rougeLsum']
    >>> print(results["rouge1"])
    AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0))
    >>> print(results["rouge1"].mid.fmeasure)
    1.0
""", stored examples: 0)
```
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
I think the only way to avoid this warning would be to do a copy of the numpy array before providing it.

This would slow down a bit the iteration over the dataset but maybe it would be safer. We could disable the copy with a flag on the `set_format` command.

In most typical cases of training a NLP model, PyTorch shouldn't modify the input so it's ok to have a non-writable array but I can understand the warning is a bit scary so maybe we could choose the side of non-warning/slower by default and have an option to speedup.

What do you think @lhoestq ?
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
@thomwolf Would it be possible to have the array look writeable, but raise an error if it is actually written to?

I would like to keep my code free of warning, but I also wouldn't like to slow down the program because of unnecessary copy operations.
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
Well because I don't know the internal of numpy as well as you I guess hahahah, do you want to try to open a PR proposing a solution?
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
@thomwolf @AndreasMadsen I think this is a terrible idea, n/o, and I am very much against it. Modifying internals of an array in such a hacky way is bound to run into other (user) issues down the line. To users it would not be clear at all what is going on e.g. when they check for write access (which will return True) but then they get a warning that the array is not writeable. That's extremely confusing. 

If your only goal is to get rid of warnings in your code, then you can just use a [simplefilter](https://docs.python.org/3.8/library/warnings.html#temporarily-suppressing-warnings) for UserWarnings in your own code. Changing the code-base in such an intuitive way does not seem like a good way to go and sets a bad precedent, imo. 

(Feel free to disagree, of course.)

IMO a warning can stay (as they can be filtered by users anyway), but it can be clarified why the warning takes place.
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
> To users it would not be clear at all what is going on e.g. when they check for write access (which will return True) but then they get a warning that the array is not writeable. That's extremely confusing.

Confusion can be resolved with a helpful error message. In this case, that error message can be controlled by huggingface/datasets. The right argument here is that if code depends on `.flags.writable` being truthful (not just for warnings), then it will cause unavoidable errors. Although, I can't imagine such a use-case.

> If your only goal is to get rid of warnings in your code, then you can just use a simplefilter for UserWarnings in your own code. Changing the code-base in such an intuitive way does not seem like a good way to go and sets a bad precedent, imo.

I don't want to ignore all `UserWarnings`, nor all warnings regarding non-writable arrays. Ignoring warnings leads to hard to debug issues.

> IMO a warning can stay (as they can be filtered by users anyway), but it can be clarified why the warning takes place.

Plain use cases should really not generate warnings. It teaches developers to ignore warnings which is a terrible practice.

---

The best solution would be to allow non-writable arrays in `DataLoader`, but that is a PyTorch issue.
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
> The right argument here is that if code depends on `.flags.writable` being truthful (not just for warnings), then it will cause unavoidable errors. Although, I can't imagine such a use-case.

That's exactly the argument in my first sentence. Too often someone "cannot think of a use-case", but you can not foresee the use-cases of a whole research community.
 
> I don't want to ignore all `UserWarnings`, nor all warnings regarding non-writable arrays. Ignoring warnings leads to hard to debug issues.

That's fair.

> Plain use cases should really not generate warnings. It teaches developers to ignore warnings which is a terrible practice.

But this is not a plain use-case (because Pytorch does not support these read-only tensors). Manually setting the flag to writable will solve the issue on the surface but is basically just a hack to compensate for something that is not allowed in another library. 

What about an "ignore_warnings" flag in `set_format` that when True wraps the offending code in a block to ignore userwarnings at that specific step in [_convert_outputs](https://github.com/huggingface/datasets/blob/880c2c76a8223a00c303eab2909371e857113063/src/datasets/arrow_dataset.py#L821)? Something like:

```python
def _convert_outputs(..., ignore_warnings=True):
    ...
    with warnings.catch_warnings():
        if ignore_warnings:
            warnings.simplefilter("ignore", UserWarning)
        return torch.tensor(...)
# continues without warning filter after context manager...
```
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
> But this is not a plain use-case (because Pytorch does not support these read-only tensors).

By "plain", I mean the recommended way to use `datasets` with PyTorch according to the `datasets` documentation.
Issue: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

Comment:
This error is what I see when I run the first lines of the Pytorch Quickstart.  It should also say that it should be ignored and/or how to fix it.   BTW, this is a Pytorch error message -- not a Huggingface error message.   My code runs anyway.
Issue: Offset overflow when slicing a big dataset with an array of indices in Pyarrow >= 1.0.0

Comment:
Related: https://issues.apache.org/jira/browse/ARROW-9773

It's definitely a size thing. I took a smaller dataset with 87000 rows and did:
```
for i in range(10,1000,20):
    table = pa.concat_tables([dset._data]*i)
    table.take([0])
```
and it broke at around i=300.

Also when `_indices` is not None, this breaks indexing by slice. E.g. `dset.shuffle()[:1]` breaks.

Luckily so far I haven't seen `_indices.column(0).take` break, which means it doesn't break `select` or anything like that which is where the speed really matters, it's just `_getitem`. So I'm currently working around it by just doing the arrow v0 method in `_getitem`:
```
#if PYARROW_V0:
data_subset = pa.concat_tables(
    self._data.slice(indices_array[i].as_py(), 1) for i in range(len(indices_array))
)
#else:
    #data_subset = self._data.take(indices_array)
```
Issue: Offset overflow when slicing a big dataset with an array of indices in Pyarrow >= 1.0.0

Comment:
This specific issue has been fixed in https://github.com/huggingface/datasets/pull/645

If you still have this error, could you open a new issue and explain how to reproduce the error ?
Issue: Offset overflow when slicing a big dataset with an array of indices in Pyarrow >= 1.0.0

Comment:
Facing the same issue. 
Steps to reproduce: (dataset is a few GB big so try in colab maybe)
Datasets version - 2.11.0
```
import datasets
import re

ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')

def get_text_caption(example):
    regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
    example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
    return example

ds = ds.map(get_text_caption)
```

I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.
Issue: Offset overflow when slicing a big dataset with an array of indices in Pyarrow >= 1.0.0

Comment:
Got this error on a very large data set (900m rows, 35 cols) performing a similar batch map operation.
Issue: ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648

Comment:
```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 17136104 entries, 0 to 17136103
Data columns (total 6 columns):
 #   Column        Dtype  
---  ------        -----  
 0   item_id       int64  
 1   item_titl     object 
 2   start_price   float64
 3   shipping_fee  float64
 4   picture_url   object 
 5   embeddings    object 
dtypes: float64(2), int64(1), object(3)
memory usage: 915.2+ MB
```
Issue: ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648

Comment:
Thanks and some more on the `embeddings` and `picture_url` would be nice as well (type and max lengths of the elements)
Issue: ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648

Comment:
It looks like a Pyarrow limitation.
I was able to reproduce the error with 

```python
import pandas as pd
import numpy as np
import pyarrow as pa

 n = 1713614
df = pd.DataFrame.from_dict({"a": list(np.zeros((n, 128))), "b": range(n)})
pa.Table.from_pandas(df)
```

I also tried with 50% of the dataframe and it actually works.
I created an issue on Apache Arrow's JIRA [here](https://issues.apache.org/jira/browse/ARROW-9976)

One way to fix that would be to chunk the dataframe and concatenate arrow tables.
Issue: ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648

Comment:
It looks like it's going to be fixed in pyarrow 2.0.0 :)

In the meantime I suggest to chunk big dataframes to create several small datasets, and then concatenate them using [concatenate_datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets)
Issue: Load text file for RoBERTa pre-training. 

Comment:
Could you try
```python
load_dataset('text', data_files='test.txt',cache_dir="./", split="train")
```
?

`load_dataset` returns a dictionary by default, like {"train": your_dataset}
Issue: Load text file for RoBERTa pre-training. 

Comment:
You need to tokenize the string inputs to convert them in integers before you can feed them to a pytorch dataloader.

You can read the quicktour of the datasets or the transformers libraries to know more about that:
- transformers: https://huggingface.co/transformers/quicktour.html
- dataset: https://huggingface.co/docs/datasets/quicktour.html
Issue: Load text file for RoBERTa pre-training. 

Comment:
Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
But finally got it working. This is what I did after looking into the documentation.

1. split the whole dataset file into smaller files
```bash
mkdir ./shards
split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
````
2. Pass paths of small data files to `load_dataset`
```python
files = glob.glob('shards/*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files, split='train')
```
(On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)

3. Tokenization
```python
def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')
dataset = dataset.map(encode, batched=True)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
```
 Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
```python
dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
next(iter(dataloader))
```
Hope this helps
Issue: Load text file for RoBERTa pre-training. 

Comment:
Thanks, @thomwolf  and @sipah00 ,

I tried to implement your suggestions in my scripts. 
Now, I am facing some connection time-out error. I am using my local file, I have no idea why the module request s3 database.

The log is:
```
Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    raise err
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    timeout=timeout
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    chunked=chunked,
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/text/text.py (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection obj
ect at 0x7fff401e0e48>: Failed to establish a new connection: [Errno 110] Connection timed out',))

Traceback (most recent call last):
  File "/scratch/roberta_emohash/run_language_modeling.py", line 1019, in <module>
    main()
  File "/scratch/roberta_emohash/run_language_modeling.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 117, in HG_Datasets
    dataset = load_dataset('text', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/evn_py36/datasets/datasets/src/datasets/load.py", line 590, in load_dataset
    self._validate_conn(conn)
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 300, in connect
    conn = self._new_conn()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fff401e0da0>: Failed to establish a new connection: [Errno 110] Connection timed out

``` 

Do you have any experience on this issue?
Issue: Load text file for RoBERTa pre-training. 

Comment:
I noticed this is because I use a cloud server where does not provide for connections from our standard compute nodes to outside resources. 

For the `datasets` package, it seems that if the loading script is not already cached in the library it will attempt to connect to an AWS resource to download the dataset loading script. 

I am wondering why the package works in this way. Do you have any suggestions to solve this issue?
Issue: Load text file for RoBERTa pre-training. 

Comment:
I solved the above issue by downloading text.py manually and passing the path to the `load_dataset` function. 

Now, I have a new issue with the Read-only file system.

The error is: 
```
I0916 22:14:38.453380 140737353971520 filelock.py:274] Lock 140734268996072 acquired on /scratch/chiyuzh/roberta/text.py.lock
Found main folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text
Creating specific version folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014
I0916 22:14:38.530371 140737353971520 filelock.py:318] Lock 140734268996072 released on /scratch/chiyuzh/roberta/text.py.lock
Traceback (most recent call last):
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 1019, in <module>
    main()
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 117, in HG_Datasets
    dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 590, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 385, in prepare_module
    os.makedirs(hash_folder_path)
  File "/project/chiyuzh/evn_py36/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014'

```

I installed datasets at /project/chiyuzh/evn_py36/datasets/src where is a writable directory.
I also tried change the environment variables to the writable directory:
`export HF_MODULES_PATH=/project/chiyuzh/evn_py36/datasets/cache_dir/`
`export HF_DATASETS_CACHE=/project/chiyuzh/evn_py36/datasets/cache_dir/`
 
In my scripts, I also changed to:
`dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")`
`data_cache_dir = $TMPDIR/data/` that also a writable directory.
 
But it still try to make directory at /home/chiyuzh/.cache/huggingface/modules/.
Do you have any idea about this issue? @thomwolf
Issue: Load text file for RoBERTa pre-training. 

Comment:
> Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
> But finally got it working. This is what I did after looking into the documentation.
> 
> 1. split the whole dataset file into smaller files
> 
> ```shell
> mkdir ./shards
> split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
> ```
> 
> 1. Pass paths of small data files to `load_dataset`
> 
> ```python
> files = glob.glob('shards/*')
> from datasets import load_dataset
> dataset = load_dataset('text', data_files=files, split='train')
> ```
> 
> (On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)
> 
> 1. Tokenization
> 
> ```python
> def encode(examples):
>   return tokenizer(examples['text'], truncation=True, padding='max_length')
> dataset = dataset.map(encode, batched=True)
> dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
> ```
> 
> Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
> 
> ```python
> dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
> next(iter(dataloader))
> ```
> 
> Hope this helps

When I run 'dataset = dataset.map(encode, batched=True)',
I encountered a problem like this:

> Testing the mapped function outputs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in map
    for k, dataset in self.items()
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in <dictcomp>
    for k, dataset in self.items()
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "<stdin>", line 3, in encode
TypeError: __init__() takes 1 positional argument but 2 were given