SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Unable to load dataset 'big_patent'

Comment:
I am able to download the dataset with ``` download_mode="force_redownload"```. As you mentioned it was an issue with the cached version which was failed earlier due to a network issue. I am closing the issue now, once again thank you.
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
Hi @adrianeboyd, thanks for reporting.

There is indeed a bug in that community dataset:
Line:
```python
metadata_and_text_files = list(zip(metadata_files, text_files))
``` 
should be replaced with
```python
metadata_and_text_files = list(zip(sorted(metadata_files), sorted(text_files)))
```

I am going to contact their owners (https://huggingface.co/oscar-corpus) in order to inform them about the bug.

I keep you informed.
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
That fix is part of it, but it's clearly not the only issue.

I also already contacted the OSCAR creators, but I reported it here because it looked like huggingface members were the main authors in the git history. Is there a better place to have reported this?
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
Hello,

We've had an issue that could be linked to this one here: https://github.com/oscar-corpus/corpus/issues/15.

I have been spot checking the source (`.txt`/`.jsonl`) files for a while, and have not found issues, especially in the start/end of corpora (but I conceed that more integration testing would be necessary on our side).

The text and metadata files are designed to be used in sync (with `lang_part_n.txt` and `lang_meta_part_n.jsonl` working together), while staying independent from part to part, so that anyone could randomly choose a part and work with it.

The fix @albertvillanova proposed should fix the problem, as the parts will be in sync again.

Let me know if you need help or more details, I'd be glad to help!
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
I'm happy to move the discussion to the other repo!

Merely sorting the files only **maybe** fixes the processing of the first part. If the first part contains non-unix newlines, it will still be misaligned/truncated, and all the following parts will be truncated with incorrect text offsets and metadata due the offset and newline bugs.
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
Hi @Uinelj, This is a total noobs question but how can I integrate that bugfix into my code? I reinstalled the datasets library this time from source. Should that have fixed the issue? I am still facing the misalignment issue. Do I need to download the dataset from scratch?
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
Sorry @norakassner for the late reply.

There are indeed several issues creating the misalignment, as @adrianeboyd cleverly pointed out:
-  https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/3cd7e95aa1799b73c5ea8afc3989635f3e19b86b fixed one of them
- but there are still others to be fixed
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
Normally, the issues should be fixed now:
- Fix offset initialization for each file: https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/1ad9b7bfe00798a9258a923b887bb1c8d732b833
- Disable default universal newline support: https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/0c2f307d3167f03632f502af361ac6c3c393f510

Feel free to reopen if you find additional misalignments/truncations.

CC: @adrianeboyd @norakassner @Uinelj
Issue: OSCAR-2109 datasets are misaligned and truncated

Comment:
Thanks for the updates!

The purist in me would still like to have the rstrip not strip additional characters from the original text (unicode whitespace mainly in practice, I think), but the differences are extremely small in practice and it doesn't actually matter for my current task:

```python
text = "".join([text_f.readline() for _ in range(meta["nb_sentences"])]).rstrip("\n")
```
Issue: ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'

Comment:
Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
Issue: ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'

Comment:
> Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
I installed seqeval, but still reported the same error. That's too bad.
Issue: ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'

Comment:
> > Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
> > I installed seqeval, but still reported the same error. That's too bad.

Same issue here. What should I do to fix this error? Please help! Thank you.
Issue: ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'

Comment:
I tried to install **seqeval** package through anaconda instead of pip:
`conda install -c conda-forge seqeval`
It worked for me!
Issue: Unable to load a dataset

Comment:
Hi! `load_dataset` is intended to be used to load a canonical dataset (`wikipedia`), a packaged dataset (`csv`, `json`, ...) or a dataset hosted on the Hub. For local datasets saved with `save_to_disk("path/to/dataset")`, use `load_from_disk("path/to/dataset")`.
Issue: Pyarrow version error

Comment:
Hi @Zaker237, thanks for reporting.

This is weird: the error you get is only thrown if the installed pyarrow version is less than 3.0.0.

Could you please check that you install pyarrow in the same Python virtual environment where you installed datasets?

From the Python command line (or terminal) where you get the error, please type:
```
import pyarrow
print(pyarrow.__version__)
import datasets
print(datasets.__version__)
```
Issue: Pyarrow version error

Comment:
hi @albertvillanova  i try yesterday to create a new python environement with python 7 and try it on the environement and it worked. so i think that the error was not the package but may be jupyter notebook on conda. still yet i'm not yet sure but it worked in an environment created with venv
Issue: Pyarrow version error

Comment:
OK, thanks @Zaker237 for your feedback.

I close this issue then. Please, feel free to reopen it if the problem arises again.
Issue: Can't get the text data when calling to_tf_dataset

Comment:
You are correct that `to_tf_dataset` only handles numerical columns right now, yes, though this is a limitation we might remove in future! The main reason we do this is that our models mostly do not include the tokenizer as a model layer, because it's very difficult to compile some of them in TF. So the "normal" Huggingface workflow is to first tokenize your dataset, and then pass tokenized tensors to the model.

For your use case, would you prefer to pass strings to the model, and use some text processing layers instead of the built-in tokenizers?
Issue: Can't get the text data when calling to_tf_dataset

Comment:
Thanks for the quick follow-up to my issue.

For my use-case, instead of the built-in tokenizers I wanted to use the `TextVectorization` layer to map from strings to integers. To achieve this, I came up with the following solution:

```
from datasets import load_dataset
from transformers import DefaultDataCollator
import tensorflow as tf
import string
import re
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

#some hyper-parameters for the text-to-integer mapping
max_features = 20000
embedding_dim = 128
sequence_length = 210

data_collator = DefaultDataCollator(return_tensors="tf")
dataset = load_dataset("sst", "default")

#adapt the vectorization layer on train data only
vectorize_layer.adapt(dataset["train"].to_dict(batched=False)["sentence"])

def prepare_features(text, label):
    text = tf.expand_dims(text, -1)
    return {"vectorized_text": vectorize_layer(text)[0], "label": tf.expand_dims(label, axis=-1)}

encoded_dataset = dataset.map(lambda example: prepare_features(example["sentence"], example["label"]), batched=False)


def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    return tf.strings.regex_replace(
        lowercase, f"[{re.escape(string.punctuation)}]", ""
    )

vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)

train_dataset = encoded_dataset["train"].to_tf_dataset(columns=['vectorized_text'], label_cols=["label"],
                                                       shuffle=True, batch_size=1, collate_fn=data_collator).unbatch()
#similar for the other sub-sets

```

Since the strings would have been mapped to integers or floats at some point, it's no drawback that this mapping is done early in the process. 

For the future, however, it'd be more convenient to get the string data, since I am also inspecting the dataset (longest sentence, shortest sentence), which is more challenging when working with integer or float. For now, this can be done by calling `to_dict`.
Issue: Can't get the text data when calling to_tf_dataset

Comment:
> For the future, however, it'd be more convenient to get the string data, since I am also inspecting the dataset (longest sentence, shortest sentence), which is more challenging when working with integer or float.

Yes, I agree, so let's keep this issue open.
Issue: Can't get the text data when calling to_tf_dataset

Comment:
Going to close this now - methods like `to_tf_dataset` and `prepare_tf_dataset` now support string data, and have done for a while! If anyone sees this and is encountering issues with string data in those methods, please file a new issue!
Issue: `Translation` features cannot be `flatten`ed

Comment:
Thanks for reporting, @SBrandeis! Some additional feature types that don't behave as expected when flattened: `Audio`, `Image` and `TranslationVariableLanguages`
Issue: Download datasets from a private hub

Comment:
Hi ! For information one can set the environment variable `HF_ENDPOINT` (default is `https://huggingface.co`) if they want to use a private hub.

We may need to coordinate with the other libraries to have a consistent way of changing the hub endpoint
Issue: Discovery cannot be streamed anymore

Comment:
Seems like a regression from https://github.com/huggingface/datasets/pull/2843

Or maybe it's an issue with the hosting. I don't think so, though, because https://www.dropbox.com/s/aox84z90nyyuikz/discovery.zip seems to work as expected
Issue: Discovery cannot be streamed anymore

Comment:
Hi @severo, thanks for reporting.

Some servers do not support HTTP range requests, and those are required to stream some file formats (like ZIP in this case).

Let me try to propose a workaround.
Issue: `None` replaced by `[]` after first batch in map

Comment:
It looks like this is because of this behavior in pyarrow:
```python
import pyarrow as pa

arr = pa.array([None, [0]])
reconstructed_arr = pa.ListArray.from_arrays(arr.offsets, arr.values)
print(reconstructed_arr.to_pylist())
# [[], [0]]
```

It seems that `arr.offsets` can reconstruct the array properly, but an offsets array with null values can:
```python
fixed_offsets = pa.array([None, 0, 1])
fixed_arr = pa.ListArray.from_arrays(fixed_offsets, arr.values)
print(fixed_arr.to_pylist())
# [None, [0]]

print(arr.offsets.to_pylist())
# [0, 0, 1]
print(fixed_offsets.to_pylist())
# [None, 0, 1]
```
EDIT: this is because `arr.offsets` is not enough to reconstruct the array, we also need the validity bitmap
Issue: `None` replaced by `[]` after first batch in map

Comment:
The offsets don't have nulls because they don't include the validity bitmap from `arr.buffers()[0]`, which is used to say which values are null and which values are non-null.

Though the validity bitmap also seems to be wrong:
```python
bin(int(arr.buffers()[0].hex(), 16))
# '0b10'
# it should be 0b110 - 1 corresponds to non-null and 0 corresponds to null, if you take the bits in reverse order
```

So apparently I can't even create the fixed offsets array using this.

If I understand correctly it's always missing the 1 on the left, so I can add it manually as a hack to fix the issue until this is fixed in pyarrow EDIT: actually it may be more complicated than that

EDIT2: actuall it's right, it corresponds to the validity bitmap of the array of logical length 2. So if we use the offsets array, the values array, and this validity bitmap it should be possible to reconstruct the array properly
Issue: `None` replaced by `[]` after first batch in map

Comment:
FYI the behavior is the same with:
- `datasets` version: 1.18.3
- Platform: Linux-5.8.0-50-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.11
- PyArrow version: 6.0.1


but not with:
- `datasets` version: 1.8.0
- Platform: Linux-4.18.0-305.40.2.el8_4.x86_64-x86_64-with-redhat-8.4-Ootpa
- Python version: 3.7.11
- PyArrow version: 3.0.0

i.e. it outputs:
```py
0  [None, [0]]
1  [None, [0]]
2  [None, [0]]
3  [None, [0]]
```
Issue: `None` replaced by `[]` after first batch in map

Comment:
Thanks for the insights @PaulLerner !

I found a way to workaround this issue for the code example presented in this issue.

Note that empty lists will still appear when you explicitly `cast` a list of lists that contain None values like [None, [0]] to a new feature type (e.g. to change the integer precision). In this case it will show a warning that it happened. If you don't cast anything, then the None values will be kept as expected.

Let me know what you think !
Issue: `None` replaced by `[]` after first batch in map

Comment:
Hi! I feel like I’m missing something in your answer, *what* is the workaround? Is it fixed in some `datasets` version?
Issue: `None` replaced by `[]` after first batch in map

Comment:
`pa.ListArray.from_arrays` returns empty lists instead of None values. The workaround I added inside `datasets` simply consists in not using `pa.ListArray.from_arrays` :)

Once this PR [here ](https://github.com/huggingface/datasets/pull/4282)is merged, we'll release a new version of `datasets` that currectly returns the None values in the case described in this issue

EDIT: released :) but let's keep this issue open because it might happen again if users change the integer precision for example
Issue: `load_dataset("snli")` is different from dataset viewer

Comment:
Yes, we decided to replace the encoded label with the corresponding label when possible in the dataset viewer. But
1. maybe it's the wrong default
2. we could find a way to show both (with a switch, or showing both ie. `0 (neutral)`).
Issue: `load_dataset("snli")` is different from dataset viewer

Comment:
Hi @severo,

Thanks for clarifying. 

I think this default is a bit counterintuitive for the user. However, this is a personal opinion that might not be general. I think it is nice to have the actual (non-encoded) labels in the viewer. On the other hand, it would be nice to match what the user sees with what they get when they download a dataset. I don't know - I can see the difficulty of choosing a default :)
Maybe having non-encoded labels as a default can be useful?

Anyway, I think the issue has been addressed. Thanks a lot for your super-quick answer!
Issue: `load_dataset("snli")` is different from dataset viewer

Comment:
Thanks for the 👍  in https://github.com/huggingface/datasets/issues/3673#issuecomment-1029008349 @mariosasko @gary149 @pietrolesci, but as I proposed various solutions, it's not clear to me which you prefer. Could you write your preferences as a comment?

_(note for myself: one idea per comment in the future)_
Issue: `load_dataset("snli")` is different from dataset viewer

Comment:
As I am working with seq2seq, I prefer having the label in string form rather than numeric. So the viewer is fine and the underlying dataset should be "decoded" (from int to str). In this way, the user does not have to search for a mapping `int -> original name` (even though is trivial to find, I reckon). Also, encoding labels is rather easy.

I hope this is useful
Issue: `load_dataset("snli")` is different from dataset viewer

Comment:
I like the idea of "0 (neutral)". The label name can even be greyed to make it clear that it's not part of the actual item in the dataset, it's just the meaning.
Issue: `load_dataset("snli")` is different from dataset viewer

Comment:
Proposals by @gary149. Which one do you prefer? Please vote with the thumbs

- 👍 

    ![image](https://user-images.githubusercontent.com/1676121/152387949-883c7d7e-a9f3-48aa-bff9-11a691555e6e.png)

- 👎 

    ![image (1)](https://user-images.githubusercontent.com/1676121/152388061-32d95e42-cade-4ae4-9a77-7365e7b72b8f.png)
Issue: `load_dataset("snli")` is different from dataset viewer

Comment:
It's [live](https://huggingface.co/datasets/glue/viewer/cola/train):

<img width="1126" alt="Capture d’écran 2022-02-14 à 10 26 03" src="https://user-images.githubusercontent.com/1676121/153836716-25f6205b-96af-42d8-880a-7c09cb24c420.png">

Thanks all for the help to improve the UI!
Issue:  Couldn't cast array of type string error with cast_column

Comment:
Hi ! I wasn't able to reproduce the error, are you still experiencing this ? I tried calling `cast_column` on a string column containing paths.

If you manage to share a reproducible code example that would be perfect
Issue:  Couldn't cast array of type string error with cast_column

Comment:
Hi,

I think my team mate got this solved. Clolsing it for now and will reopen if I experience this again.
Thanks :)
Issue:  Couldn't cast array of type string error with cast_column

Comment:
Hi @R4ZZ3,

If it is not too much of a bother, can you please help me how to resolve this error? I am exactly getting the same error where I am going as per the documentation guideline:

`my_audio_dataset = my_audio_dataset.cast_column("audio_paths", Audio())`

where `"audio_paths"` is a dataset column (feature) having strings of absolute paths to mp3 files of the dataset.
Issue:  Couldn't cast array of type string error with cast_column

Comment:
I was having the same issue with this code:

```
dataset = dataset.map(
    lambda batch: {"full_path" : os.path.join(self.data_path, batch["path"])},
    num_procs = 4
)
my_audio_dataset = dataset.cast_column("full_path", Audio(sampling_rate=16_000))
```

Removing the "num_procs" argument fixed it somehow.
Using a mac with m1 chip
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
Agree that we need to have access to the original sound files. Few days ago I was looking for these original files because I suspected there is bug in the audio resampling (confirmed in https://github.com/huggingface/datasets/issues/3662) and I want to do my own resampling to workaround the bug, which is now not possible anymore due to the unavailability of the original files.
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
@patrickvonplaten 
> The other solution of providing a path-like object derived from the bytes stocked in the .array file is not nearly as user-friendly, but better than nothing

Just to clarify, here you describe the approach that uses the `Audio.decode` attribute to access the underlying bytes?

> The official example currently doesn't work and we don't even have a workaround for it for MP3 files at the moment

I'd assume this is because we use `sox_io` as a backend for decoding. However, soon we should be able to use `soundfile`, which supports path-like objects, for MP3 (https://github.com/huggingface/datasets/pull/3667#issuecomment-1030090627).

Your concern is reasonable, but there are situations where we can only serve bytes (see https://github.com/huggingface/datasets/pull/3685 for instance). IMO it makes sense to fix the affected datasets for now, but I don't think we should care too much whether we rely on local paths or bytes after soundfile adds support for MP3 as long as our examples work (shouldn't be too hard to update the `map_to_array` functions) and we properly document how to access the underlying path/bytes for custom decoding (via `ds.cast_column("audio", Audio(decode=False))`).
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
Related to this discussion: in https://github.com/huggingface/datasets/pull/3664#issuecomment-1031866858 I propose how we could change `iter_archive` to work for streaming and also return local paths (as it used too !). I'd love your opinions on this
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
> @patrickvonplaten
> 
> > The other solution of providing a path-like object derived from the bytes stocked in the .array file is not nearly as user-friendly, but better than nothing
> 
> Just to clarify, here you describe the approach that uses the `Audio.decode` attribute to access the underlying bytes?

Yes! 

> 
> > The official example currently doesn't work and we don't even have a workaround for it for MP3 files at the moment
> 
> I'd assume this is because we use `sox_io` as a backend for decoding. However, soon we should be able to use `soundfile`, which supports path-like objects, for MP3 ([#3667 (comment)](https://github.com/huggingface/datasets/pull/3667#issuecomment-1030090627)). 
> Your concern is reasonable, but there are situations where we can only serve bytes (see #3685 for instance). IMO it makes sense to fix the affected datasets for now, but I don't think we should care too much whether we rely on local paths or bytes after soundfile adds support for MP3 as long as our examples work (shouldn't be too hard to update the `map_to_array` functions) and we properly document how to access the underlying path/bytes for custom decoding (via `ds.cast_column("audio", Audio(decode=False))`).

Yes this might be, but I highly doubt that `soundfile` is the go-to library for audio then. @anton-l and I have tried out a bunch of different audio loading libraries (`soundfile`, `librosa`, `torchaudio`, pure `ffmpeg`, `audioread`, ...). One thing that was pretty clear  to me is that there is just no "de-facto standard" library and they all have pros and cons. None of the libraries really supports "batch"-ed audio loading. Some depend on PyTorch. `torchaudio` is 100x faster (really!) than `librosa's` fallback on MP3. `torchaudio` often has problems with multi-proessing, ... Also we should keep in mind that resampling is similarly not as simple as reading a text file. It's a pretty complex signal processing transform and people very well might want to use special filters, etc...at the moment we just hard-code `torchaudio's` or `librosa's` default filter when doing resampling.

=> All this to say that we **should definitely** care about whether we rely on local paths or bytes IMO. We don't want to loose all users that are forced to use `datasets` decoding or resampling or have to built a very much not intuitive way of loading bytes into a numpy array. It's much more intuitive to be able to inspect a local file. I feel pretty strongly about this and am happy to also jump on a call. Keeping libraries flexible and lean as well as exposing internals is very important IMO (this philosophy has worked quite well so far with Transformers).
Issue: [Audio] Path of Common Voice cannot be used for audio loading anymore

Comment:
From https://github.com/huggingface/datasets/pull/3736 the Common Voice dataset now gives access to the local audio files as before