noamaanMulla-03/datasets-issues huggingface dataset:
Add helper to resolve namespace collision

Many projects use a module called `datasets`, however this is incompatible with huggingface datasets. It would be great if there if there was some helper or similar function to resolve such a common conflict. 


Do you have an example?

I was thinking about using something like [importlib](https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly) to over-ride the collision. 

**Reason requested**: I use the [following template](https://github.com/jramapuram/ml_base/) repo where I house all my datasets as a submodule.

Alternatively huggingface could consider some submodule type structure like:

`import huggingface.datasets`
`import huggingface.transformers`

`datasets` is a very common module in ML and should be an end-user decision and not scope all of python ¯\_(ツ)_/¯ 


That's a interesting option indeed. We'll think about it.

It also wasn't initially obvious to me that the samples which contain `import datasets` were in fact importing a huggingface library (in fact all the huggingface imports are very generic - transformers, tokenizers, datasets...)
connection error 

Hi
I am hitting to this error, thanks 

```
> Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 379, in <module>
    main()
  File "finetune_t5_trainer.py", line 208, in main
    if training_args.do_eval or training_args.evaluation_strategy != EvaluationStrategy.NO
  File "finetune_t5_trainer.py", line 207, in <dictcomp>
    for task in data_args.eval_tasks}
  File "/workdir/seq2seq/data/tasks.py", line 70, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/workdir/seq2seq/data/tasks.py", line 66, in load_dataset
    return datasets.load_dataset(self.task.name, split=split, script_version="master")
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 487, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/boolq/boolq.py
el/0 I1217 01:11:33.898849 354161 main shadow.py:210 Current job status: FINISHED
```


This happen quite often when they are too many concurrent requests to github.

i can understand it’s a bit cumbersome to handle on the user side. Maybe we should try a few times in the lib (eg with timeout) before failing, what do you think @lhoestq ?

Yes currently there's no retry afaik. We should add retries

Retries were added in #1603 :) 
It will be available in the next release

Hi @lhoestq thank you for the modification, I will use`script_version="master"` for now  :), to my experience, also setting timeout to a larger number like 3*60 which I normally use helps a lot on this.
writing Datasets in a human readable format

Hi
I see there is a save_to_disk function to save data, but this is not human readable format, is there a way I could save a Dataset object in a human readable  format to a file like json? thanks @lhoestq 

Labels: name: enhancement

name: question

AFAIK, there is currently no built-in method on the `Dataset` object to do this.
However, a workaround is to directly use the Arrow table backing the dataset, **but it implies loading the whole dataset in memory** (correct me if I'm mistaken @lhoestq).

You can convert the Arrow table to a pandas dataframe to save the data as csv as follows:
```python
arrow_table = dataset.data
dataframe = arrow_table.to_pandas()
dataframe.to_csv("/path/to/file.csv")
```

Similarly, you can convert the dataset to a Python dict and save it as JSON:
```python
import json
arrow_table = dataset.data
py_dict = arrow_table.to_pydict()
with open("/path/to/file.json", "w+") as f:
    json.dump(py_dict, f)
```

Indeed this works as long as you have enough memory.
It would be amazing to have export options like csv, json etc. !

It should be doable to implement something that iterates through the dataset batch by batch to write to csv for example.
There is already an `export` method but currently the only export type that is supported is `tfrecords`.

Hi! `datasets` now supports `Dataset.to_csv` and `Dataset.to_json` for saving data in a human readable format.
how can I combine 2 dataset with different/same features?

to combine 2 dataset by one-one map like ds = zip(ds1, ds2):
ds1: {'text'}, ds2: {'text'}, combine ds:{'src', 'tgt'} 
or different feature:
ds1: {'src'}, ds2: {'tgt'}, combine ds:{'src', 'tgt'}


Hi ! Currently we don't have a way to `zip` datasets but we plan to add this soon :)
For now you'll need to use `map` to add the fields from one dataset to the other. See the comment here for more info : https://github.com/huggingface/datasets/issues/853#issuecomment-727872188

Good to hear.
Currently I did not use map , just fetch src and tgt from the 2 dataset and merge them.
It will be a release if you can deal with it at the backend.
Thanks.

Hi! You can rename the columns and concatenate the datasets along `axis=1` to get the desired result as follows:
```python
ds1 = ds1.rename_column("text", "src")
ds2 = ds2.rename_column("text", "tgt")
ds = datasets.concatenate_datasets(["ds1", "ds2"], axis=1)
```
load the local dataset

your guidebook's example is like
>>>from datasets import load_dataset
>>> dataset = load_dataset('json', data_files='my_file.json')
but the first arg is path...
so how should i do if i want to load the local dataset for model training?
i will be grateful if you can help me handle this problem!
thanks a lot!


You should rephrase your question or give more examples and details on what you want to do.

it’s not possible to understand it and help you with only this information.

sorry for that.
i want to know how could i load the train set and the test set from the local ,which api or function should i use .


Did you try to follow the instructions in the documentation?
Here: https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files

thanks a lot 
i find that the problem is i dont use vpn...
so i have to keep my net work even if i want to load the local data ?

We will solve this soon (cf #1724)

thanks a lot

Hi! `json` is a packaged dataset now, which means its script comes with the library and doesn't require an internet connection.
making labels consistent across the datasets

Hi
For accessing the labels one can type 
```
>>> a.features['label']
ClassLabel(num_classes=3, names=['entailment', 'neutral', 'contradiction'], names_file=None, id=None)
```
The labels however are not consistent with the actual labels sometimes, for instance in case of XNLI, the actual labels are 0,1,2, but if one try to access as above they are entailment, neutral,contradiction,
it would be great to have the labels consistent.

thanks 



Hi ! The ClassLabel feature type encodes the labels as integers.
The integer corresponds to the index of the label name in the `names` list of the ClassLabel.
Here that means that the labels are 'entailment' (0), 'neutral' (1), 'contradiction' (2).

You can get the label names back by using `a.features['label'].int2str(i)`.


Hi! You can also easily reorder the label with the [`Dataset.align_labels_with_mapping`](https://huggingface.co/docs/datasets/master/en/process#align) method.
Converting a Value to a ClassLabel

Hi!

In the docs for `cast`, it's noted that `For non-trivial conversion, e.g. string <-> ClassLabel you should use map() to update the Dataset.`

Would it be possible to have an example  that demonstrates such a string <-> ClassLabel conversion using `map`? Thanks!

Labels: name: enhancement

Hi @nelson-liu!
Here is what I do to convert a string to class label:

```python
from datasets import load_dataset, features


dset = load_dataset(...)
col_name = "the string column name"

class_names  = dset.unique(col_name)
class_feature = features.ClassLabel(names=sorted(class_names))
dset = dset.map(lambda str_value: {col_name: class_feature.str2int(str_value)}, input_columns=col_name)

dset = dset.cast(features.Features({
    ...
    col_name: class_feature
})
```


Hi! You can use `Dataset.class_encode_column` for this. And in the next release of `datasets` (this feature is only available on `master`), you'll also be able to use `cast` to do the conversion. 

An example of conversion via `cast`: 
```python
from datasets import Dataset, Features, ClassLabel
d = Dataset.from_dict({"a": ["no", "yes", "no"]})
d = d.cast(Features({"a": ClassLabel(names=["yes", "no"])}))
```
save_to_disk doesn't work when we use concatenate_datasets function before creating the final dataset_object.

As you can see, it saves the entire dataset.

@lhoestq 

You can  check by going through the following example,

```
from datasets import load_from_disk,concatenate_datasets

loaded_data=load_from_disk('/home/gsir059/HNSW-ori/my_knowledge_dataset')
n=20
kb_list=[loaded_data.shard(n, i, contiguous=True) for i in range(n)]
final_dataset=concatenate_datasets([kb_list[1],kb_list[2]])
final_dataset.save_to_disk('/home/gsir059/haha/k.arrow')
```


Hi ! We refactored save_to_disk in #2025 so this doesn't happen.
Feel free to try it on master for now
We'll do a new release soon
How to disable making arrow tables in load_dataset ?

Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?


Hi ! We plan to add streaming features in the future.

This should allow to load a dataset instantaneously without generating the arrow table. The trade-off is that accessing examples from a streaming dataset must be done in an iterative way, and with an additional (but hopefully minor) overhead.
What do you think about this ?

If you have ideas or suggestions of what you expect from such features as a user, feel free to share them, this is really valuable to us !

People mainly want this feature either because it takes too much time too make arrow tables, or they occupy too much memory on the disk. I think both the problem can be solved if we provide arrow tables themselves on datasets hub. Can we do this currently @lhoestq ? 


@lhoestq I think the ```try_from_hf_gcs``` provide the same functionality. What all datasets are available on HF GCS? Are all the datasets on huggingFace datasets hub are made available on GCS, automatically?

Only datasets like wikipedia, wiki40b, wiki_dpr and natural questions are available already processed on the HF google storage. This is used to download directly the arrow file instead of building it from the original data files.

@lhoestq How can we make sure that the data we upload on HuggingFace hub is available in form of preprocessed arrow files ?

We're still working on this :) This will be available soon
Users will be able to put their processed arrow files on the Hub

Hi! You can now use `Dataset.push_to_hub` to store preprocessed files on the Hub.

And to avoid downloading preprocessed files, you can use streaming by setting `streaming=True` in `load_dataset`.
Synchronize table metadata with features

**Is your feature request related to a problem? Please describe.**

As pointed out in this [comment](https://github.com/huggingface/datasets/pull/2145#discussion_r621326767):
> Metadata stored in the schema is just a redundant information regarding the feature types.
It is used when calling Dataset.from_file to know which feature types to use.
These metadata are stored in the schema of the pyarrow table by using `update_metadata_with_features`.
However this something that's almost never tested properly.

**Describe the solution you'd like**

We should find a way to always make sure that the metadata (in `self.data.schema.metadata`) are synced with the actual feature types (in `self.info.features`).

Labels: name: enhancement

See PR #2274
Use `Audio` features for `AutomaticSpeechRecognition` task template

In #2533 we added a task template for speech recognition that relies on the file paths to the audio files. As pointed out by @SBrandeis this is brittle as it doesn't port easily across different OS'. 

The solution is to use dedicated `Audio` features when casting the dataset. These features are not yet available in `datasets`, but should be included in the `AutomaticSpeechRecognition` template once they are.

Labels: name: enhancement

I'm just retaking and working on #2324. 😉 

Resolved via https://github.com/huggingface/datasets/pull/4006.
Add mIoU metric

**Is your feature request related to a problem? Please describe.**
Recently, some semantic segmentation models were added to HuggingFace Transformers, including [SegFormer](https://huggingface.co/transformers/model_doc/segformer.html) and [BEiT](https://huggingface.co/transformers/model_doc/beit.html).

Semantic segmentation (which is the task of labeling every pixel of an image with a corresponding class) is typically evaluated using the Mean Intersection and Union (mIoU). Together with the upcoming Image Feature, adding this metric could be very handy when creating example scripts to fine-tune any Transformer-based model on a semantic segmentation dataset.

An implementation can be found [here](https://github.com/open-mmlab/mmsegmentation/blob/504965184c3e6bc9ec43af54237129ef21981a5f/mmseg/core/evaluation/metrics.py#L132) for instance.


Labels: name: enhancement

Resolved via https://github.com/huggingface/datasets/pull/3745.
Cannot load timit_asr data set

## Describe the bug
I am trying to load the timit_asr data set. I have tried with a copy from the LDC, and a copy from deepai. In both cases they fail with a "duplicate key" error. With the LDC version I have to convert the file extensions all to upper-case before I can load it at all.

## Steps to reproduce the bug
```python
timit = datasets.load_dataset("timit_asr", data_dir = "/path/to/dataset")
# Sample code to reproduce the bug
```

## Expected results

The data set should load without error. It worked for me before the LDC url change.

## Actual results
```
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: SA1
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Labels: name: bug

Thanks for reporting, @bhaddow.

I'm fixing it.

Thanks for the quick fix!

@bhaddow we have also made a fix so that you don't have to convert to uppercase the file extensions of the LDC data.

Would you mind checking if it works OK now for you and reporting if there are any issues? Thanks. 

Hi @albertvillanova  -It loads fine on a copy of the data from deepai - although I have to remove the copies of the .WAV files (with extension .WAV,wav). On a copy of the data that was obtained from the LDC, the glob still fails to find the files. The LDC copy looks like it was copied from CD, in 2004, so the structure may be different to a current download.

Ah, if I change the train/ and test/ directories to TRAIN/ and TEST/  then it works!

Thanks for your investigation and report, @bhaddow. I'm adding another fix for the TRAIN/train and TEST/test directory names.
[BUG] Metrics throwing new error on master since 0.4.0

The following error occurs when passing in references of type `List[List[str]]` to metrics like bleu.
Wasn't happening on 0.4.0 but happening now on master.

```
  File "/usr/local/lib/python3.7/site-packages/nlp/metric.py", line 226, in compute
    self.add_batch(predictions=predictions, references=references)
  File "/usr/local/lib/python3.7/site-packages/nlp/metric.py", line 242, in add_batch
    batch = self.info.features.encode_batch(batch)
  File "/usr/local/lib/python3.7/site-packages/nlp/features.py", line 527, in encode_batch
    encoded_batch[key] = [encode_nested_example(self[key], cast_to_python_objects(obj)) for obj in column]
  File "/usr/local/lib/python3.7/site-packages/nlp/features.py", line 527, in <listcomp>
    encoded_batch[key] = [encode_nested_example(self[key], cast_to_python_objects(obj)) for obj in column]
  File "/usr/local/lib/python3.7/site-packages/nlp/features.py", line 456, in encode_nested_example
    raise ValueError("Got a string but expected a list instead: '{}'".format(obj))
```


Update - maybe this is only failing on bleu because I was not tokenizing inputs to the metric

Closing - seems to be just forgetting to tokenize. And found the helpful discussion in huggingface/evaluate#105
Load a local cached dataset that has been modified

## Describe the bug
I have loaded a dataset as follows:
```
d = load_dataset("emotion", split="validation")
```
Afterwards I make some modifications to the dataset via a `map` call:
```
d.map(some_update_func, cache_file_name=modified_dataset)
```
This generates a cached version of the dataset on my local system in the same directory as the original download of the data (/path/to/cache). Running an `ls` returns:
```
modified_dataset
dataset_info.json  
emotion-test.arrow  
emotion-train.arrow  
emotion-validation.arrow
```
as expected. However, when I try to load up the modified cached dataset via a call to 
```
modified = load_dataset("emotion", split="validation", data_files="/path/to/cache/modified_dataset") 
```
it simply redownloads a new version of the dataset and dumps to a new cache rather than loading up the original modified dataset:
```
Using custom data configuration validation-cdbf51685638421b
Downloading and preparing dataset emotion/validation to ...
```

How am I supposed to load the original modified local cache copy of the dataset?

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2



Labels: name: bug

Hi! `datasets` caches every modification/loading, so you can either rerun the pipeline up to the `map` call or use `Dataset.from_file(modified_dataset)` to load the dataset directly from the cache file.

Awesome, hvala Mario! This works.
Always update metadata in arrow schema

We store a redundant copy of the features in the metadata of the schema of the arrow table. This is used to recover the features when doing `Dataset.from_file`. These metadata are updated after each transfor, that changes the feature types.

For each function that transforms the feature types of the dataset, I added a step in the tests to make sure the metadata in the arrow schema are up to date.

I also added a line to update the metadata directly in the Dataset.__init__ method.
This way even a dataset instantiated with __init__ will have a table with the right metadata.

Fix #2271.

cc @mariosasko
TIMIT won't load after manual download: Errors about files that don't exist

## Describe the bug

I get the message from HuggingFace that it must be downloaded manually. From the URL provided in the message, I got to UPenn page for manual download. (UPenn apparently want $250? for the dataset??)   ...So, ok, I obtained a copy from a friend and also a smaller version from Kaggle. But in both cases the HF dataloader fails; it is looking for files that don't exist anywhere in the dataset: it is looking for files with lower-case letters like "**test*" (all the filenames in both my copies are uppercase) and certain file extensions that exclude the .DOC which is provided in TIMIT:


## Steps to reproduce the bug
```python
data = load_dataset('timit_asr', 'clean')['train']
```

## Expected results
The dataset should load with no errors. 

## Actual results
This error message:
```
  File "/home/ubuntu/envs/data2vec/lib/python3.9/site-packages/datasets/data_files.py", line 201, in resolve_patterns_locally_or_by_urls
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to resolve any data file that matches '['**test*', '**eval*']' at /home/ubuntu/datasets/timit with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

But this is a strange sort of error: why is it looking for lower-case file names when all the TIMIT dataset filenames are uppercase?  Why does it exclude .DOC files when the only parts of the TIMIT data set with "TEST" in them have ".DOC" extensions?   ...I wonder, how was anyone able to get this to work in the first place?

The files in the dataset look like the following: 
```
³       PHONCODE.DOC
³       PROMPTS.TXT
³       SPKRINFO.TXT
³       SPKRSENT.TXT
³       TESTSET.DOC
```
...so why are these being excluded by the dataset loader? 


## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-1060-aws-x86_64-with-glibc2.27
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Labels: name: bug

To have some context, please see:
- #4145

Please, also note that we have recently made some fixes to the script, which are in our GitHub master branch but not yet released:
- #4422
- #4425 
- #4436

Thanks Albert! I'll try pulling `datasets` from the git repo instead of PyPI, and/or just wait for the next release.


I'm closing this issue then. Please, feel free to reopen it again if the problem persists.
Metrics documentation is not accessible in the datasets doc UI

**Is your feature request related to a problem? Please describe.**
Search for a metric name like "seqeval" yields no results on https://huggingface.co/docs/datasets/master/en/index . One needs to go look in `datasets/metrics/README.md` to find the doc. Even in the `README.md`, it can be hard to understand what the metric expects as an input, for example for `squad` there is a [key `id`](https://github.com/huggingface/datasets/blob/1a4c185663a6958f48ec69624473fdc154a36a9d/metrics/squad/squad.py#L42) documented only in the function doc but not in the `README.md`, and one needs to go look into the code to understand what the metric expects.

**Describe the solution you'd like**
Have the documentation for metrics appear as well in the doc UI, e.g. this https://github.com/huggingface/datasets/blob/1a4c185663a6958f48ec69624473fdc154a36a9d/metrics/squad/squad.py#L21-L63

I know there are plans to migrate metrics to the evaluate library, but just pointing this out.


Labels: name: enhancement

name: Metric discussion

Hey @fxmarty :) Yes we are working on showing the docs of all the metrics on the Hugging face website. If you want to follow the advancements you can check the [evaluate](https://github.com/huggingface/evaluate) repository cc @lvwerra @sashavor
Add loading variable number of columns for different splits

**Is your feature request related to a problem? Please describe.**
The original dataset `blended_skill_talk` consists of different sets of columns for the different splits: (test/valid) splits have additional data column `label_candidates` that the (train) doesn't have.
When loading such data, an exception occurs at table.py:cast_table_to_schema, because of mismatched columns.

Labels: name: enhancement

Hi! Indeed the column is missing, but you shouldn't get an error? Have you made some modifications (locally) to the loading script? I've opened a PR to add the missing columns to the script.