SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Text dataset not working with large files

Comment:
> Ah, it’s much faster now(Takes around 15~20min).

Glad to see that it's faster now. What did you change exactly ?

> BTW, any way to set default tensor output as plain tensors with distributed training? The ragged tensors are incompatible with tpustrategy :(

Oh I didn't know about that. Feel free to open an issue to mention that.
I guess what you can do for now is set the dataset format to numpy instead of tensorflow, and use a wrapper of the dataset that converts the numpy arrays to tf tensors.
Issue: Text dataset not working with large files

Comment:
>>> Glad to see that it's faster now. What did you change exactly ?
I don't know, it just worked...? Sorry I couldn't be more helpful.

Setting with numpy array is a great idea! Thanks.
Issue: dtype of tensors should be preserved

Comment:
Indeed we convert tensors to list to be able to write in arrow format. Because of this conversion we lose the dtype information. We should add the dtype detection when we do type inference. However it would require a bit of refactoring since currently the conversion happens before the type inference..

And then for your information, when reading from arrow format we have to cast from arrow to numpy (which is fast since pyarrow has a numpy integration), and then to torch.

However there's one thing that can help you: we make sure that the dtypes correspond to what is defined in `features`.
Therefore what you can do is provide `features` in `.map(preprocess, feature=...)` to specify the output types.

For example in your case:
```python
from datasets import Features, Value, Sequence

features = Features({
    "input_ids": Sequence(Value("int32")),
    "sembedding": Sequence(Value("float32"))
})
preprocessed_dataset = dataset.map(preprocess, features=features)

preprocessed_dataset.set_format("torch", columns=["input_ids", "sembedding"])
print(preprocessed_dataset[0]["sembedding"].dtype)
# "torch.float32"
```

Let me know if it helps
Issue: dtype of tensors should be preserved

Comment:
If the arrow format is basically lists, why is the intermediate step to numpy necessary? I am a bit confused about that part.

Thanks for your suggestion. as I have currently implemented this, I cast to torch.Tensor in my collate_fn to save disk space (so I do not have to save padded tensors to max_len but can pad up to max batch len in collate_fn) at the cost of a bit slower processing. So for me this is not relevant anymore, but I am sure it is for others!
Issue: dtype of tensors should be preserved

Comment:
I'm glad you managed to figure something out :)

Casting from arrow to numpy can be 100x faster than casting from arrow to list.
This is because arrow has an integration with numpy that allows it to instantiate numpy arrays with zero-copy from arrow.
On the other hand to create python lists it is slow since it has to recreate the list object by iterating through each element in python.
Issue: dtype of tensors should be preserved

Comment:
I encountered a simliar issue: `datasets` converted my float numpy array to `torch.float64` tensors, while many pytorch operations require `torch.float32` inputs and it's very troublesome. 

I tried @lhoestq 's solution, but since it's mixed with the preprocess function, it's not very intuitive. 

I just want to share another possible simpler solution: directly cast the dtype of the processed dataset.

Now I want to change the type of `labels` in `train_dataset` from float64 to float32, I can do this.

```
from datasets import Value, Sequence, Features
feats = train_dataset.features.copy()
feats['labels'].feature = Value(dtype='float32')
feats = Features(feats)
train_dataset.cast_(feats)
```
Issue: dtype of tensors should be preserved

Comment:
Reopening since @bhavitvyamalik started looking into it !

Also I'm posting here a function that could be helpful to support preserving the dtype of tensors.

It's used to build a pyarrow array out of a numpy array and:
- it doesn't convert the numpy array to a python list
- it keeps the precision of the numpy array for the pyarrow array
- it works with multidimensional arrays (while `pa.array` can only take a 1D array as input)
- it builds the pyarrow ListArray from offsets created on-the-fly and values that come from the flattened numpy array

```python
from functools import reduce
from operator import mul

import numpy as np
import pyarrow as pa

def pa_ndarray(a):
    """Build a PyArrow ListArray from a multidimensional NumPy array"""
    values = pa.array(a.flatten()) 
    for i in range(a.ndim - 1): 
        n_offsets = reduce(mul, a.shape[:a.ndim - i - 1], 1) 
        step_offsets = a.shape[a.ndim - i - 1] 
        offsets = pa.array(np.arange(n_offsets + 1) * step_offsets, type=pa.int32()) 
        values = pa.ListArray.from_arrays(offsets, values) 
    return values 

narr = np.arange(42).reshape(7, 2, 3).astype(np.uint8)
parr = pa_ndarray(narr)
assert isinstance(parr, pa.Array)
assert parr.type == pa.list_(pa.list_(pa.uint8()))
assert narr.tolist() == parr.to_pylist()
```

The only costly operation is the offsets computations. Since it doesn't iterate on the numpy array values this function is pretty fast.
Issue: dtype of tensors should be preserved

Comment:
@lhoestq Have you thought about this further?

We have a use case where we're attempting to load data containing numpy arrays using the `datasets` library.

When using one of the "standard" methods (`[Value(...)]` or `Sequence()`) we see ~200 samples processed per second during the call to `_prepare_split`. This slowdown is caused by the vast number of calls to `encode_nested_example` (each sequence is converted to a list, and each element in the sequence...). 

Using the `Feature` `ArrayND` improves this somewhat to ~500/s as it now uses numpy's `tolist()` rather than iterating over each value in the array and converting them individually.

However, it's still pretty slow and in theory it should be possible to avoid the `numpy -> python -> arrow` dance altogether. To demonstrate this, if you keep the `Feature` set to an `ArrayND` but instead return a `pa_ndarray(...)` in `_generate_examples` it skips the conversion (`return obj, False`) and hits ~11_000/s. Two orders of magnitude speed up! The problem is this then fails later on when the `ArrowWriter` tries to write the examples to disk :-( 

It would be nice to have first-class support for user-defined PyArrow objects. Is this a possibility? We have _large_ datasets where even an order of magnitude difference is important so settling on the middle ~500/s is less than ideal! 

Is there a workaround for this or another method that should be used instead that gets  near-to or equal performance to returning PyArrow arrays?
Issue: dtype of tensors should be preserved

Comment:
Hi !

It would be awesome to achieve this speed for numpy arrays !
For now we have to use `encode_nested_example` to convert numpy arrays to python lists since pyarrow doesn't support multidimensional numpy arrays (only 1D).

Maybe let's start a new PR from your PR @bhavitvyamalik (idk why we didn't answer your PR at that time, sorry about that).
Basically the idea is to allow `TypedSequence` to support numpy arrays as you did, and remove the numpy->python casting in `_cast_to_python_objects`.

This is really important since we are starting to have a focus on other modalities than text as well (audio, images).

Though until then @samgd, there is another feature that may interest you and that may give you the speed you want:

In a dataset script you can subclass either a GeneratorBasedBuilder (with the `_generate_examples ` method) or an ArrowBasedBuilder if you want. the ArrowBasedBuilder allows to yield arrow data by implementing the `_generate_tables` method (it's the same as `_generate_examples` except you must yield arrow tables). Since the data are already in arrow format, it doesn't call `encode_nested_example`. Let me know if that helps.
Issue: Custom feature types in `load_dataset` from CSV

Comment:
Currently `csv` doesn't support the `features` attribute (unlike `json`).
What you can do for now is cast the features using the in-place transform `cast_`

```python
from datasets import load_dataset

dataset = load_dataset('csv', data_files=file_dict, delimiter=';', column_names=['text', 'label'])
dataset.cast_(emotion_features)
```
Issue: Custom feature types in `load_dataset` from CSV

Comment:
Hi @lhoestq we've tried out your suggestion but are now running into the following error:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-163-81ffd5ac18c9> in <module>
----> 1 dataset.cast_(emotion_features)

/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py in cast_(self, features)
    125         self._check_values_type()
    126         for dataset in self.values():
--> 127             dataset.cast_(features=features)
    128 
    129     def remove_columns_(self, column_names: Union[str, List[str]]):

/usr/local/lib/python3.6/dist-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    161             # Call actual function
    162 
--> 163             out = func(self, *args, **kwargs)
    164 
    165             # Update fingerprint of in-place transforms + update in-place history of transforms

/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in cast_(self, features)
    602         self._info.features = features
    603         schema = pa.schema(features.type)
--> 604         self._data = self._data.cast(schema)
    605 
    606     @fingerprint(inplace=True)

/usr/local/lib/python3.6/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.cast()

ValueError: Target schema's field names are not matching the table's field names: ['text', 'label'], ['label', 'text']
```

Looking at the types in `emotion_features` we see that `label` and `text` appear to be swapped in the Arrow table:

```
emotion_features.type
StructType(struct<label: int64, text: string>)
```

Did we define the `emotion_features` incorrectly? We just followed the instructions from the [docs](https://huggingface.co/docs/datasets/features.html?highlight=features#dataset-features), but perhaps we misunderstood something 😬
Issue: Custom feature types in `load_dataset` from CSV

Comment:
In general, I don't think there is any hard reason we don't allow to use `features` in the csv script, right @lhoestq?

Should I add it?
Issue: Custom feature types in `load_dataset` from CSV

Comment:
> In general, I don't think there is any hard reason we don't allow to use `features` in the csv script, right @lhoestq?
> 
> Should I add it?

Sure let's add it. Setting the convert options should do the job

> Hi @lhoestq we've tried out your suggestion but are now running into the following error:
> 
> ```
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> <ipython-input-163-81ffd5ac18c9> in <module>
> ----> 1 dataset.cast_(emotion_features)
>
>  /usr/local/lib/python3.6/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.cast()
> 
> ValueError: Target schema's field names are not matching the table's field names: ['text', 'label'], ['label', 'text']
> ```
>
> Did we define the `emotion_features` incorrectly? We just followed the instructions from the [docs](https://huggingface.co/docs/datasets/features.html?highlight=features#dataset-features), but perhaps we misunderstood something 😬

Thanks for reporting, that's a bug :) I'm fixing it right now
Issue: Custom feature types in `load_dataset` from CSV

Comment:
PR is open for the `ValueError: Target schema's field names are not matching the table's field names` error.

I'm adding the features parameter to csv
Issue: load_dataset for text files not working

Comment:
@thomwolf Sure. I'll try downgrading to 3.7 now even though Arrow say they support >=3.5.

Linux (Ubuntu 18.04) - Python 3.8
======================
Package   -   Version
---------------------
certifi               2020.6.20
chardet               3.0.4
click                 7.1.2
datasets              1.0.1
dill                  0.3.2
fasttext              0.9.2
filelock              3.0.12
future                0.18.2
idna                  2.10
joblib                0.16.0
nltk                  3.5
numpy                 1.19.1
packaging             20.4
pandas                1.1.2
pip                   20.0.2
protobuf              3.13.0
pyarrow               1.0.1
pybind11              2.5.0
pyparsing             2.4.7
python-dateutil       2.8.1
pytz                  2020.1
regex                 2020.7.14
requests              2.24.0
sacremoses            0.0.43
scikit-learn          0.23.2
scipy                 1.5.2
sentence-transformers 0.3.6
sentencepiece         0.1.91
setuptools            46.1.3
six                   1.15.0
stanza                1.1.1
threadpoolctl         2.1.0
tokenizers            0.8.1rc2
torch                 1.6.0+cu101
tqdm                  4.48.2
transformers          3.1.0
urllib3               1.25.10
wheel                 0.34.2
xxhash                2.0.0

Windows 10 - Python 3.8
================
Package       -        Version
----------------------------
certifi               2020.6.20
chardet               3.0.4
click                 7.1.2
datasets              1.0.1
dill                  0.3.2
fasttext              0.9.2
filelock              3.0.12
future                0.18.2
idna                  2.10
joblib                0.16.0
nlp                   0.4.0
nltk                  3.5
numpy                 1.19.1
packaging             20.4
pandas                1.1.1
pip                   20.0.2
protobuf              3.13.0
pyarrow               1.0.1
pybind11              2.5.0
pyparsing             2.4.7
python-dateutil       2.8.1
pytz                  2020.1
regex                 2020.7.14
requests              2.24.0
sacremoses            0.0.43
scikit-learn          0.23.2
scipy                 1.5.2
sentence-transformers 0.3.5.1
sentencepiece         0.1.91
setuptools            46.1.3
six                   1.15.0
stanza                1.1.1
threadpoolctl         2.1.0
tokenizers            0.8.1rc1
torch                 1.6.0+cu101
tqdm                  4.48.2
transformers          3.0.2
urllib3               1.25.10
wheel                 0.34.2
xxhash                2.0.0
Issue: load_dataset for text files not working

Comment:
Downgrading to 3.7 does not help. Here is a dummy text file:

```text
Verzekering weigert vaker te betalen
Bedrijven van verzekeringen erkennen steeds minder arbeidsongevallen .
In 2012 weigerden de bedrijven te betalen voor 21.055 ongevallen op het werk .
Dat is 11,8 % van alle ongevallen op het werk .
Nog nooit weigerden verzekeraars zoveel zaken .
In 2012 hadden 135.118 mensen een ongeval op het werk .
Dat zijn elke werkdag 530 mensen .
Bij die ongevallen stierven 67 mensen .
Bijna 12.000 hebben een handicap na het ongeval .
Geen echt arbeidsongeval Bedrijven moeten een verzekering hebben voor hun werknemers .
```

A temporary work around for the "text" type, is

```python
dataset = Dataset.from_dict({"text": Path(dataset_f).read_text().splitlines()})
```
Issue: load_dataset for text files not working

Comment:
@banunitte Please do not post screenshots in the future but copy-paste your code and the errors. That allows others to copy-and-paste your code and test it. You may also want to provide the Python version that you are using.
Issue: load_dataset for text files not working

Comment:
I have the same  problem on Linux of the  script crashing with a CSV error.  This may be caused by 'CRLF', when changed 'CRLF' to 'LF', the problem solved.
Issue: load_dataset for text files not working

Comment:
I pushed a fix for `pyarrow.lib.ArrowInvalid: CSV parse error`. Let me know if you still have this issue.

Not sure about the windows one yet
Issue: load_dataset for text files not working

Comment:
To complete what @lhoestq is saying, I think that to use the new version of the `text` processing script (which is on master right now) you need to either specify the version of the script to be the `master` one or to install the lib from source (in which case it uses the `master` version of the script by default):
```python
dataset = load_dataset('text', script_version='master', data_files=XXX)
```
We do versioning by default, i.e. your version of the dataset lib will use the script with the same version by default (i.e. only the `1.0.1` version of the script if you have the PyPI version `1.0.1` of the lib).
Issue: load_dataset for text files not working

Comment:
![image](https://user-images.githubusercontent.com/36957508/93300760-fa9a8680-f829-11ea-9105-7a6f67ad8373.png)
win10, py3.6


```
from datasets import Features, Value, ClassLabel, load_dataset


features = Features({'text': Value('string'), 'ctext': Value('string')})
file_dict = {'train': PATH/'summary.csv'}

dataset = load_dataset('csv', data_files=file_dict, script_version='master', delimiter='\t', column_names=['text', 'ctext'], features=features)
```
Issue: load_dataset for text files not working

Comment:
```python
Traceback` (most recent call last):
  File "main.py", line 281, in <module>
    main()
  File "main.py", line 190, in main
    train_data, test_data = data_factory(
  File "main.py", line 129, in data_factory
    train_data = load_dataset('text', 
  File "/home/me/Downloads/datasets/src/datasets/load.py", line 608, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/me/Downloads/datasets/src/datasets/builder.py", line 468, in download_and_prepare
    self._download_and_prepare(
  File "/home/me/Downloads/datasets/src/datasets/builder.py", line 546, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/me/Downloads/datasets/src/datasets/builder.py", line 888, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/me/.local/lib/python3.8/site-packages/tqdm/std.py", line 1130, in __iter__
    for obj in iterable:
  File "/home/me/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014/text.py", line 103, in _generate_tables
    pa_table = pac.read_csv(
  File "pyarrow/_csv.pyx", line 617, in pyarrow._csv.read_csv
  File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: CSV parse error: Expected 1 columns, got 2
```

Unfortunately i am still getting this issue on Linux. I installed datasets from source and specified script_version to master.
Issue: load_dataset for text files not working

Comment:
> ![image](https://user-images.githubusercontent.com/36957508/93300760-fa9a8680-f829-11ea-9105-7a6f67ad8373.png)
> win10, py3.6
> 
> ```
> from datasets import Features, Value, ClassLabel, load_dataset
> 
> 
> features = Features({'text': Value('string'), 'ctext': Value('string')})
> file_dict = {'train': PATH/'summary.csv'}
> 
> dataset = load_dataset('csv', data_files=file_dict, script_version='master', delimiter='\t', column_names=['text', 'ctext'], features=features)
> ```

Since #644 it should now work on windows @ScottishFold007 

> Trying the following snippet, I get different problems on Linux and Windows.
> 
> ```python
> dataset = load_dataset("text", data_files="data.txt")
> # or 
> dataset = load_dataset("text", data_files=["data.txt"])
> ```
>
> Windows just seems to get stuck. Even with a tiny dataset of 10 lines, it has been stuck for 15 minutes already at this message:
> 
> ```
> Checking C:\Users\bramv\.cache\huggingface\datasets\b1d50a0e74da9a7b9822cea8ff4e4f217dd892e09eb14f6274a2169e5436e2ea.30c25842cda32b0540d88b7195147decf9671ee442f4bc2fb6ad74016852978e.py for additional imports.
> Found main folder for dataset https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py at C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text
> Found specific version folder for dataset https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py at C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text\7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7
> Found script file from https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py to C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text\7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7\text.py
> Couldn't find dataset infos file at https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text\dataset_infos.json
> Found metadata file for dataset https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py at C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text\7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7\text.json
> Using custom data configuration default
> ```

Same for you @BramVanroy .

Not sure about the one on linux though
Issue: load_dataset for text files not working

Comment:
> To complete what @lhoestq is saying, I think that to use the new version of the `text` processing script (which is on master right now) you need to either specify the version of the script to be the `master` one or to install the lib from source (in which case it uses the `master` version of the script by default):
> 
> ```python
> dataset = load_dataset('text', script_version='master', data_files=XXX)
> ```
> 
> We do versioning by default, i.e. your version of the dataset lib will use the script with the same version by default (i.e. only the `1.0.1` version of the script if you have the PyPI version `1.0.1` of the lib).

Linux here:

I was using the 0.4.0 nlp library load_dataset to load a text dataset of 9-10Gb without collapsing the RAM memory. However, today I got the csv error message mentioned in this issue. After installing the new (datasets) library from source and specifying the script_verson = 'master' I'm still having this same error message. Furthermore, I cannot use the dictionary "trick" to load the dataset since the system kills the process due to a RAM out of memory problem. Is there any other solution to this error? Thank you in advance.
Issue: load_dataset for text files not working

Comment:
Hi @raruidol 
To fix the RAM issue you'll need to shard your text files into smaller files (see https://github.com/huggingface/datasets/issues/610#issuecomment-691672919 for example)

I'm not sure why you're having the csv error on linux.
Do you think you could to to reproduce it on google colab for example ?
Or send me a dummy .txt file that reproduces the issue ?