SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)

Comment:
Hi @lhoestq! Thank you for your prompt reply.
I have experimented with (10<<20, 10<<28, 10<<30, 10<<33, 10<<34), since my machine has 192G of memory, but it's either the above-mentioned error or processed killed because of OOM.

Could you give me a bit of background on why block size needs to be exactly calibrated?
To my understanding, small block sized should run just fine despite its slowness..
Issue: Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)

Comment:
We're using the JSON loader of pyarrow. It parses the file chunk by chunk to load the dataset.
This issue happens when there's no delimiter in one chunk of data. For json line, the delimiter is the end of line.
So with a big value for chunk_size this should have worked unless you have one extremely long line in your file.

Also what version of pyarrow are you using ?

FInally I wonder if it could be an issue on pyarrow's side when using big json files. (I haven't tested big json files like yours)
Issue: Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)

Comment:
I'm using `pyarrow==3.0.0` with `datasets==1.5.0`.

Your point totally makes sense. I will check if my jsonl file contains an extremely long file and let you know. 

Here are some different error messages that I got when tweaking `block_size`. I also suspect that this is related to the pyarrow... but I guess it would be wonderful if datasesets could give a clear guide on how to play with large datasets! (I am suddenly experiencing various issue when working with large datasets.. e.g. #1992 )
```python
    return paj.ReadOptions(use_threads=self.use_threads, block_size=self.block_size)
  File "pyarrow/_json.pyx", line 56, in pyarrow._json.ReadOptions.__init__
  File "pyarrow/_json.pyx", line 81, in pyarrow._json.ReadOptions.block_size.__set__
OverflowError: value too large to convert to int32_t
```

```python

line 83, in _generate_tables
    parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 247, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Exceeded maximum rows
```
Issue: Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)

Comment:
I am getting the same error. When I tweak the block_size, I also find:
`OverflowError: value too large to convert to int32_t`
and 
`pyarrow.lib.ArrowInvalid: Exceeded maximum rows`
Issue: Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)

Comment:
I made more tests. I used a smaller dataset and I was getting the same error, which means that it was not necessarily linked to the dataset size. To make both my smaller and larger datasets work, I got rid of lists with the json file. I had the following data format:
```python
[
  {'key': "a", 'value': ['one', 'two', 'three']},
  {'key': "b", 'value': ['four', 'five', 'six']}
]
```
I changed to:

```python
  {'key': "a", 'value': 'one\ntwo\nthree'},
  {'key': "b", 'value': 'four\nfive\nsix']}
```
and that worked!

I used the following to reformat my json file:
```python
with open(file_name, "w", encoding="utf-8") as f:
    for item in list_:
        f.write(json.dumps(item) + "\n")
```
This works with `block_size_10MB = 10 << 20` or without specifying `block_size`.
Issue: Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)

Comment:
Thanks @hwijeen for reporting and thanks @jpilaul for pointing this out.

Indeed, those are different JSON-like formats:
- the first one is the **standard JSON** format: all the file content is JSON-valid, thus all content is either a JSON object (between curly brackets `{...}`) or a JSON array (between square brackets `[...]`)
- the second one is called **JSON Lines**: the entire file content is not JSON-valid, but only every line (newline-delimited) is JSON-valid

Currently PyArrow only supports **JSON Lines** format: 
- https://arrow.apache.org/docs/python/generated/pyarrow.json.read_json.html
  > Currently only the line-delimited JSON format is supported.
- https://arrow.apache.org/docs/python/json.html
  > Arrow supports reading columnar data from line-delimited JSON files.
Issue: Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)

Comment:
Thanks @albertvillanova for your explanation, it is helpful to know (maybe add to docs?)!
However, the problem I described above happened when I was dealing with jsonl files 😿
Although I did not thoroughly inspect, I suspect the cause was the one extremely long document in my case.
Issue: Converting a Value to a ClassLabel

Comment:
Hi @nelson-liu!
Here is what I do to convert a string to class label:

```python
from datasets import load_dataset, features


dset = load_dataset(...)
col_name = "the string column name"

class_names  = dset.unique(col_name)
class_feature = features.ClassLabel(names=sorted(class_names))
dset = dset.map(lambda str_value: {col_name: class_feature.str2int(str_value)}, input_columns=col_name)

dset = dset.cast(features.Features({
    ...
    col_name: class_feature
})
```
Issue: Converting a Value to a ClassLabel

Comment:
Hi! You can use `Dataset.class_encode_column` for this. And in the next release of `datasets` (this feature is only available on `master`), you'll also be able to use `cast` to do the conversion. 

An example of conversion via `cast`: 
```python
from datasets import Dataset, Features, ClassLabel
d = Dataset.from_dict({"a": ["no", "yes", "no"]})
d = d.cast(Features({"a": ClassLabel(names=["yes", "no"])}))
```
Issue: dataset.search_batch() function outputs all -1 indices sometime.

Comment:
Actually, I found the answer [here](https://github.com/facebookresearch/faiss/wiki/FAQ#what-does-it-mean-when-a-search-returns--1-ids). 

So we have to do some modifications to the code for instances where the index doesn't retrieve any IDs.
Issue: dataset.search_batch() function outputs all -1 indices sometime.

Comment:
@lhoestq  @patrickvonplaten 

I also found another short bug in the retrieval part.   Especially, when retrieving documents. If Faiss returns the -1 as the index, the retriever will always use the last element in the dataset.

please check [def get_doc_dicts function](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/retrieval_rag.py#L222)


Does the use of the HNSW guarantee to retrieve valid indexes always?
Issue: dataset.search_batch() function outputs all -1 indices sometime.

Comment:
Hi !
No it happens sometimes to return -1, especially if your dataset is small.
If your dataset is big enough it shouldn't happen in my experience.

Ideally we should ignore all the -1 that are returned. It should be possible to change that in RAG's code
Issue: dataset.search_batch() function outputs all -1 indices sometime.

Comment:
I also checked with some indexes it returns more -1s. Specially with IVF
when nprobr is very low. It doesn't happen when using HNSW though. But at
the moment if it happens, dataset will always return the last element.
Maybe we should change it to repeat the most last valid retrieved doc id.
What do you think?

On Wed, Apr 7, 2021, 21:09 Quentin Lhoest ***@***.***> wrote:

> Hi !
> No it happens sometimes to return -1, especially if your dataset is small.
> If your dataset is big enough it shouldn't happen.
>
> Ideally we should ignore all the -1 that are returned. It should be
> possible to change that in RAG's code
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2175#issuecomment-814746509>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGTENOTLBEZTXEO2RS3THQOMPANCNFSM42PRVYDA>
> .
>
Issue: dataset.search_batch() function outputs all -1 indices sometime.

Comment:
That would be an easy way to workaround this issue. Feel free to open a PR on `transformers` and ping me ! :)
Issue: dataset.search_batch() function outputs all -1 indices sometime.

Comment:
Sure.  Will push everything together with RAG end to end. :) thanks a lot.

On Wed, Apr 7, 2021, 21:16 Quentin Lhoest ***@***.***> wrote:

> That would be an easy way to workaround this issue. Feel free to open a PR
> on transformers and ping me ! :)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2175#issuecomment-814752589>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGWLROCGARKN7WOJYSTTHQPH5ANCNFSM42PRVYDA>
> .
>
Issue: Wikipedia historic dumps are deleted but hf/datasets hardcodes dump date

Comment:
It seems that this can be fixed from user's end by including a `date` argument, like this:

`dataset = datasets.load_dataset('wikipedia', '20200501.en', date='20210420')`

You can get available dates from [here](https://dumps.wikimedia.org/enwiki/).

This is not a proper fix however as all the files will still have '20200501' in their file names.
Issue: Regarding Test Sets for the GEM datasets

Comment:
Hi @vyraun ! The test references for CommonGen are not publicly available: you can reach out to the original dataset authors if you would like to ask for them, but we will not be releasing them as part of GEM (March 31st was the release date for the test set inputs, references are incidentally released for some of the test sets but shouldn't really be used for benchmark submissions)

cc @sebastiangehrmann
Issue: How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Comment:
Hi,

a HF dataset can be converted to a Torch Dataset with a simple wrapper as follows:
```python
from torch.utils.data import Dataset
 
class HFDataset(Dataset):
    def __init__(self, dset):
        self.dset = dset

    def __getitem__(self, idx):
        return self.dset[idx]

    def __len__(self):
        return len(self.dset)

train_ds = HFDataset(train_ds)
```
@lhoestq Since the Arrow Dataset already provides `__getitem__` and `__len__`, I think we could use the [virtual subclass](https://docs.python.org/3/library/abc.html#abc.ABCMeta.register) mechanism from the `abc` module to elegantly solve this issue. This mechanism would allow the Arrow Dataset to be used in place of the Torch Dataset because the `isinstance(instance of Arrow Dataset, TorchDataset)` check would return True (DeepSpeed has this check [here](https://github.com/microsoft/DeepSpeed/blob/ab5534fc4c0f8ca21ada321f9730d723aa31288b/deepspeed/runtime/engine.py#L823)).

And it requires a minimal change in the `arrow_dataset.py` file:
```python
if config.TORCH_AVAILABLE:
    from torch.utils.data import Dataset as TorchDataset
    TorchDataset.register(Dataset)
```
Issue: How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Comment:
Interesting ! Thanks for sharing this @mariosasko . I like the idea
This looks like something we should add IMO
Issue: How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Comment:
@mariosasko 
Thx for your code!
It perfectly works with a small modification for HF NLP dataset:
```
original_ds = nlp.load_dataset('scientific_papers', 'arxiv')
train_ds = HFDataset(train_ds['train']) # needs splitting
```
Issue: How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Comment:
@lhoestq Sadly, from Python 3.7 onwards `torch.utils.data.Dataset` doesn't support the virtual subclass mechanism due to `typing.Generic` type no longer having `abc.ABCMeta` as its metaclass.

With that in mind, another option is to remove a direct type check (`isinstance(dataset, torch.utils.data.Dataset)`) in `deepspeed.initalize` and to rewrite the checks in a manner similar to `torch.utils.data.DataLoader` ([link](https://github.com/pytorch/pytorch/blob/b80c6f863f2327c712c478f67c248b94d66b65ac/torch/utils/data/dataloader.py#L197-L239)). This is exactly why the `DataLoader` works with arbitrary objects that provide `__getitem__` and `__len__` (and in our case, the `ArrowDataset`). By doing so, their code wouldn't be any stricter in comparison to the `DataLoader`.

So if you agree, I can open an issue in their repo and fix this if they like the idea.
Issue: How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Comment:
That makes sense ! Feel free to open an issue on their repo and discuss this idea
Issue: How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Comment:
@y-rokutan Hi, now if you install `deepspeed` from master (this feature will be available in the next official release), the code should work without subclassing. Let us know if you still have any issues.
Issue: How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset

Comment:
Worth mentioning that any function that expects a `torch..Dataset` (like `torch..DataLoader`) will fail a mypy-esque typecheck if a `datasets.Dataset` is passed, even though it implements the interface correctly (I think). The virtual subclass idea was a good one- I wonder if there's another workaround given the Generic issue. What we're really talking about is something similar to the structural subtyping semantics that `typing.Protocol` defines. If `torch..DataLoader` accepted anything that supports `__getitem__` and `__len__` methods this would be much easier. Not sure if there's a way to do this without the wrapper from the perspective of `datasets`.
Issue: visualization for cc100 is broken 

Comment:
This looks like an issue with the cc100 dataset itself but not sure
Did you try loading cc100 on your machine ?
Issue: visualization for cc100 is broken 

Comment:
Hi
loading works fine, but the viewer only is broken
thanks

On Wed, Apr 7, 2021 at 12:17 PM Quentin Lhoest ***@***.***>
wrote:

> This looks like an issue with the cc100 dataset itself but not sure
> Did you try loading cc100 on your machine ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2162#issuecomment-814793809>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMRUO33JSOYGT6RETWLTHQWNLANCNFSM42IUOR6Q>
> .
>
Issue: visualization for cc100 is broken 

Comment:
Hi! This visualization tool is deprecated now. The viewer at https://huggingface.co/datasets/cc100 works fine, so I'm closing this issue.
Issue: any possibility to download part of large datasets only?

Comment:
oh, great, really awesome feature to have, thank you very much for the great, fabulous work
Issue: any possibility to download part of large datasets only?

Comment:
We'll work on dataset streaming soon. This should allow you to only load the examples you need ;)
Issue: any possibility to download part of large datasets only?

Comment:
thanks a lot Quentin, this would be really really a great feature to have

On Wed, Apr 7, 2021 at 12:14 PM Quentin Lhoest ***@***.***>
wrote:

> We'll work on dataset streaming soon. This should allow you to only load
> the examples you need ;)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2161#issuecomment-814791922>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMROD62QAKIJMAKWISTTHQWBVANCNFSM42IUI5JQ>
> .
>
Issue: any possibility to download part of large datasets only?

Comment:
Is streaming completed? On the 1.8.0 docs it is mentioned (https://huggingface.co/docs/datasets/dataset_streaming.html), but when following the example I get the following error:

```
>>> dataset2 = load_dataset("amazon_us_reviews", "Pet_Products_v1_00", split='train', streaming=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-1eedab26cff1> in <module>()
----> 1 en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                 if value is not None:
    340                     if not hasattr(builder_config, key):
--> 341                         raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
    342                     setattr(builder_config, key, value)
    343 

ValueError: BuilderConfig OscarConfig(name='unshuffled_deduplicated_en', version=1.0.0, data_dir=None, data_files=None, description='Unshuffled and deduplicated, English OSCAR dataset') doesn't have a 'streaming' key.
```

UPDATE: Managed to get streaming working by building from source and installing the additional `datasets[streaming]` package:

```
!pip install git+https://github.com/huggingface/datasets.git
!pip install datasets[streaming]
```
Issue: any possibility to download part of large datasets only?

Comment:
Hi ! Streaming is available on `master` only right now. We'll make a new release 1.9.0 on Monday :)
Issue: data_args.preprocessing_num_workers almost freezes 

Comment:
Hi.
I cannot always reproduce this issue, and on later runs I did not see it so far. Sometimes also I set 8 processes but I see less being showed, is this normal, here only 5 are shown for 8 being set, thanks

```
#3:  11%|███████████████▊                                                                                                                                  | 172/1583 [00:46<06:21,  3.70ba/s]
#4:   9%|█████████████▏                                                                                                                                    | 143/1583 [00:46<07:46,  3.09ba/s]
#7:   6%|█████████                                                                                                                                          | 98/1583 [00:45<11:34,  2.14ba/s]
#5:   8%|███████████▍                                                                                                                                      | 124/1583 [00:46<09:03,  2.68ba/s]
#6:   7%|██████████▏  
```
Issue: viewer "fake_news_english" error

Comment:
Thanks for reporting !
The viewer doesn't have all the dependencies of the datasets. We may add openpyxl to be able to show this dataset properly
Issue: viewer "fake_news_english" error

Comment:
This viewer tool is deprecated now and the new viewer at https://huggingface.co/datasets/fake_news_english works fine, so I'm closing this issue
Issue: load_dataset ignoring features

Comment:
Nice question which helped me a lot! I have wasted a lot of time to the `DatasetDict` creation from a csv file. Hope the document of this module add some simple examples.
Issue: Add configurable options to `seqeval` metric

Comment:
Hi @marrodion. 

Thanks for pointing this out. It would be great to incorporate this metric-specific enhancement.

Another possibility would be to require the user to input the scheme as a string `mode="strict", scheme="IOB2"` and then dynamically import the corresponding module using Python `importlib`:
```python
if scheme:
    scheme = importlib.import_module(f"seqeval.scheme.{scheme}")
```

Feel free to create a Pull Request to make this contribution.
Issue: Dataset file size on disk is very large with 3D Array

Comment:
Hi ! In the arrow file we store all the integers as uint8.
So your arrow file should weigh around `height x width x n_channels x n_images` bytes.

What feature type do your TFDS dataset have ?

If it uses a `tfds.features.Image` type, then what is stored is the encoded data (as png or jpg for example). Since these encodings are made for compression, the resulting tfrecord is smaller that the arrow file.

We are working on adding a similar feature in `datasets`: the ability to store the encoded data instead of the raw integers for images, but also for audio data. This way, arrow files will have similar sizes as tfrecords for images.
Issue: Dataset file size on disk is very large with 3D Array

Comment:
Thanks for the prompt response. You're right about the encoding, I have the `tfds.features.Image` feature type you mentioned.
However, as described in the `dataset_info.json`, my dataset is made of 1479 (224x224x3) images. 1479 x 224 x 224 x 3 = 222630912 bytes which is far from the actual size 520803408 bytes. 

Anyway I look forward to the Image feature type in `datasets`.
Issue: Dataset file size on disk is very large with 3D Array

Comment:
@lhoestq I changed the data structure so I have a  2D Array feature type instead of a 3D Array by grouping the two last dimensions ( a 224x672 2D Array instead of a 224x224x3 3D Array). The file size is now  223973964 bytes, nearly half the previous size! Which is around of what I would expect.
I found similar behavior in existing `datasets` collection, when comparing black and white vs color image, for example MNIST vs CIFAR.
Issue: Dataset file size on disk is very large with 3D Array

Comment:
Interesting !
This may be because of the offsets that are stored with the array data.

Currently the offsets are stored even if the `shape` of the arrays is fixed. This was needed because of some issues with pyarrow a few months ago. I think these issues have been addressed now, so we can probably try to remove them to make the file lighter.

Ideally in your case the floats data should be 220 MB for both Array2D and Array3D
Issue: Dataset file size on disk is very large with 3D Array

Comment:
Yeah for sure, can you be a bit more specific about where the offset is stored in the code base ? And any reference to pyarrow issues if you have some. I would be very interested in contributing to `datasets` by trying to fix this issue.
Issue: Dataset file size on disk is very large with 3D Array

Comment:
Pyarrow has two types of lists: variable length lists and fixed size lists.
Currently we store the ArrayXD data as variable length lists. They take more disk space because they must store both actual data and offsets.
In the `datasets` code this is done here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L346-L352

To use a fixed length list, one should use the `list_size` argument of `pyarrow.list_()`.
I believe this would work directly modulo some changes in the numpy conversion here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L381-L395
Issue: Loading wikipedia 20200501.en throws pyarrow related error

Comment:
That's how I loaded the dataset
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
```
Issue: Loading wikipedia 20200501.en throws pyarrow related error

Comment:
Hi ! It looks like the arrow file in the folder
`/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.

Can you take a look and check that it's 18.3GB ?

If not, then maybe you need to redownload it:
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
```