noamaanMulla-03/datasets-issues huggingface dataset:
`path` is `None` when downloading a custom audio dataset from the Hub

### Describe the bug

I've created an [audio dataset](https://huggingface.co/datasets/lewtun/audio-test-push) using the `audiofolder` feature desribed in the [docs](https://huggingface.co/docs/datasets/audio_dataset#audiofolder) and then pushed it to the Hub. 

Locally, I can see the `audio.path` feature is of the expected form `path/to/data_dir`, but when I download the dataset from the Hub, I see `audio.path` is `None`

Here's an example:

```python
from datasets import load_dataset

ds = load_dataset("lewtun/audio-test-push")
ds["train"][0]
# {
#     "audio": {
#         "path": None,    <-- Is this expected?
#         "array": array(
#             [
#                 3.97140226e-07,
#                 7.30310290e-07,
#                 7.56406735e-07,
#                 ...,
#                 -1.19636677e-01,
#                 -1.16811886e-01,
#                 -1.12441722e-01,
#             ]
#         ),
#         "sampling_rate": 44100,
#     },
#     "song_id": 0,
#     "genre_id": 0,
#     "genre": "Electronic",
# }
```

Is this expected behaviour? If yes, feel free to close this issue as it's not a true bug then :)

### Steps to reproduce the bug

1. Create an audio dataset with the `audiofolder` feature
2. Push the dataset to the Hub with `push_to_hub()`
3. Download the Hub dataset and inspect the `audio.path` feature

### Expected behavior

`audio.path` points to the file associated with the audio data

### Environment info

- `datasets` version: 2.6.2.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1


Hi! Yes, this is expected behavior - we do this as a security measure to not leak local paths (this info would be useless on other users' machines anyways) and only push audio bytes.
Consistent caching between python and jupyter

### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen

Labels: name: enhancement

Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`

Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!
Add "ipykernel" to list of `co_filename`s to remove

Should resolve #5157 


I don't know how I could add some tests for this, although jupyter is not among the dependencies so at least that would need to be added. If someone can tell a recommended way I will try to do it!

So testing by myself and looking around the jupyter codebase it looks like the `co_filename` of objects created within jupyter is of the form `f"{tempfile.tempdir}/ipykernel_{id1}/{id2}.py"` however I can't find the exact command setting it so I [asked in discourse](https://discourse.jupyter.org/t/co-filename-within-notebooks/16538). For now adapted the `co_filename` filter and added tests according to this I hope to get an answer and possibly fix based on that.

Ok ! I think it's fine to just check if the parent folder is named like `ipykernel_*` then

see the source code of how it's created:

https://github.com/ipython/ipykernel/blob/7f73ff705510b35d1e2faad7f5a676c620ce08d4/ipykernel/compiler.py#L72-L75

Should look better now didn't notice the duplicated tests

_The documentation is not available anymore as the PR was closed or merged._

Should work now on windows too

I did the changes you suggested and tried to rebase, the first part went fine, the second less so :( 

If you have time to spare, can you tell me what should I do now to fix this? thanks

Instead of rebasing you can just merge `main` into your branch, otherwise the GitHub preview of your PR shows changes of from `main`.

Feel free to close this PR and create a new one. Or alternatively your can force push to this PR with a new clean git history.

I have force-pushed and merged main, only shows the right changes, if you can run CI one more time it should be ok now

Hi, sorry I have been busy, the thing is I can't really understand why the test fail, besides the ugly thing I had done in the last commit to check if within CI smth stange happened with `os`, locally tests pass

The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6

I applied the same changes you did to this other function as well - it should be all good now

> The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6
> 
> I applied the same changes you did to this other function as well - it should be all good now

Thanks, it would have taken a long time to figure out :)
[Caching] Deterministic hashing of torch tensors

Currently this fails
```python
import torch
from datasets.fingerprint import Hasher

t = torch.tensor([1.])

def func(x):
    return t + x

hash1 = Hasher.hash(func)
t = torch.tensor([1.])
hash2 = Hasher.hash(func)
assert hash1 == hash2
```

Also as noticed in https://discuss.huggingface.co/t/dataset-cant-cache-models-outputs/24945, using a model in a `map` function doesn't work well with caching. Indeed the `bert-base-uncased` model has a different hash every time you reload it. Supporting torch tensors may also help in this case.

This can be fixed by registering a custom pickling functions for torch tensors - as we did for other objects such as CodeType, FunctionType and Regex in `py_utils.py`

Labels: name: enhancement
Make torch.Tensor and spacy models cacheable

Override `Pickler.save` to implement deterministic reduction (lazily registered; inspired by https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L343) functions for `torch.Tensor` and spaCy models.

Fix https://github.com/huggingface/datasets/issues/5170, fix https://github.com/huggingface/datasets/issues/3178



_The documentation is not available anymore as the PR was closed or merged._
new dataset type: single-label and multi-label video classification

**Is your feature request related to a problem? Please describe.**
In my research, I am dealing with multi-modal (audio+text+frame sequence) video classification. It would be great if the datasets library supported generating multi-modal batches from a video dataset.

**Describe the solution you'd like**
Assume I have video files having single/multiple labels. I want to train a single/multi-label video classification model. I want datasets to support generating multi-modal batches (audio+frame sequence) from video files. Audio waveform and frame sequence can be extracted from each video clip then I can use any audio, image and video model from transformers library to extract features which will be fed into my model.

**Describe alternatives you've considered**
Currently, I am using https://github.com/facebookresearch/pytorchvideo dataloaders. There seems to be not much alternative.

**Additional context**
I am wiling to open a PR but don't know where to start.


Labels: name: enhancement

Hi ! You can in the `features` folder how we implemented the audio and image feature types.

We can have something similar to videos. What we need to decide:
- the video loading library to use
- the output format when a user accesses a video type object
- what parameters a `Video()` feature type needs

also cc @nateraw who also took a look at what we can do for video

@lhoestq @nateraw is there any progress on adding video classification datasets? 

Hi ! I think we just missing which lib we're going to use to decode the videos + which parameters must go in the `Video` type

Hmm. `decord` could be nice but it's no longer maintained [it seems](https://github.com/dmlc/decord/issues/214). 

pytorchvideo uses [pyav](https://github.com/PyAV-Org/PyAV) as the default decoder: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/labeled_video_dataset.py#L37

Also it would be great if `optionally` audio can also be decoded from the video as in pytorchvideo: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/labeled_video_dataset.py#L35

Here are the other decoders supported in pytorchvideo: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/encoded_video.py#L17


@sayakpaul I did do quite a bit of work on [this PR](https://github.com/huggingface/datasets/pull/4532) a while back to add a video feature. It's outdated, but uses my `encoded_video` [package](https://github.com/nateraw/encoded-video) under the hood, which is basically a wrapper around PyAV stolen from [pytorchvideo](https://github.com/facebookresearch/pytorchvideo/) that gets rid of the `torch` dependency. 

would be really great to get something like this in...it's just a really tricky and time consuming feature to add.
Add notebook / other resource links to the task-specific data loading guides

Does it make sense to include links to notebooks / scripts that show how to use a dataset for training / fine-tuning a model? 

For example, here in [https://huggingface.co/docs/datasets/image_classification] we could include a mention of https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb.

Applies to https://huggingface.co/docs/datasets/object_detection as well.

Cc: @osanseviero @nateraw 

Labels: name: enhancement

Yea this would be great! We would need an object detection tutorial notebook too if it doesn't already exist there. 

There is one: https://huggingface.co/docs/datasets/object_detection.

I will start the work.
chore: add notebook links to img cls and obj det.

Closes https://github.com/huggingface/datasets/issues/5182

Labels: name: enhancement

_The documentation is not available anymore as the PR was closed or merged._

@nateraw I guess the failing test is unrelated.  

@sayakpaul Yea failures are unrelated. 

Alright. Will wait for @osanseviero's take and then merge. 

FYI @stevhliu 

@osanseviero @stevhliu @nateraw thank you for your comments. Acted on them.

Thanks! Can I merge? Or should we wait for approvals from the others?

Since @stevhliu approved as well, I think you're good to go

Alright!

Merging as a Member for the first time 🫀
TextConfig: added "errors"

This patch adds the ability to set the `errors` option of `open` for loading text datasets. I needed it because some data I had scraped had bad bytes in it, so I needed `errors='ignore'`. 


_The documentation is not available anymore as the PR was closed or merged._

Thanks for adding this ! You can fix the CI by formatting your code using the `make style` command :)

[**@lhoestq**](https://github.com/lhoestq) commented on [Oct 27, 2022, 4:08 PM GMT+3:30](https://github.com/huggingface/datasets/pull/5155#issuecomment-1293464680 "2022-10-27T12:38:04Z - Replied by Github Reply Comments"):
> Thanks for adding this ! You can fix the CI by formatting your code using the `make  style` command :)

I ran this and force pushed the changes.
[zstd] Use max window log size

ZstdDecompressor has a parameter `max_window_size` to limit max memory usage when decompressing zstd files. The default `max_window_size` is not enough when files are compressed by `zstd --ultra` flags.

Change `max_window_size` to the zstd's max window size. NOTE, the `zstd.WINDOWLOG_MAX` is the log_2 value of the max window size.


@albertvillanova Please take a review.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5197). All of your documentation changes will be reflected on that endpoint.
Dataset can’t cache model’s outputs

### Describe the bug

Hi,

I try to cache some outputs of teacher model( Knowledge Distillation ) by using map function of Dataset library, while every time I run my code, I still recompute all the sequences. I tested Bert Model like this, I got different hash every single run, so any idea to deal with this?

### Steps to reproduce the bug

1. run below code
2. get different hash
```
from transformers import BertModel
from transformers import AutoTokenizer
import torch
token = ['hello']
model = BertModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
def abcd():
    with torch.no_grad():
        
        out = model(**tok(token,return_tensors='pt'))[0]
        # out = tok(token)
    return out
from datasets.fingerprint import Hasher
my_func = abcd
print(Hasher.hash(my_func))

print(abcd())
```

### Expected behavior

I wanna cache all the model output 

### Environment info

datasets:2.5.0


Addressed in https://github.com/huggingface/datasets/pull/5191 (torch.Tensor objects now produce deterministic hashes)
Dummy data nowhere to be found

## Describe the bug
To finalize my dataset, I wanted to create dummy data as per the guide and I ran 

```shell
 datasets-cli dummy_data datasets/hebban-reviews --auto_generate
```

where hebban-reviews is [this repo](https://huggingface.co/datasets/BramVanroy/hebban-reviews). And even though the scripts runs and shows a message at the end that it succeeded, I cannot find the dummy data anywhere. Where is it?

## Expected results

To see the dummy data in the datasets' folder or in the folder where I ran the command.

## Actual results

I see the following message but I cannot find the dummy data anywhere.

```
Dummy data generation done and dummy data test succeeded for config 'filtered''.
Automatic dummy data generation succeeded for all configs of '.\datasets\hebban-reviews\'
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Labels: name: bug

Hi @BramVanroy, thanks for reporting.

First of all, please note that you do not need the dummy data: this was the case when we were adding datasets to the `datasets` library (on this GitHub repo), so that we could test the correct loading of all datasets with our CI. However, this is no longer the case for datasets on the Hub.
- We should definitely update our docs.

Second, the dummy data is generated locally:
- in your case, the dummy data will be generated inside the directory:  `./datasets/hebban-reviews/dummy`
- please note the preceding `./datasets` directory: the reason for this is that the command to generate the dummy data was specifically created for our `datasets` library, and therefore assumes our directory structure: commands are run from the root directory of our GitHub repo, and datasets scripts are under `./datasets` 


 

I have opened an Issue to update the instructions on dummy data generation:
- #4744

Dummy data generation is deprecated now, so I think we can close this issue.
Do not sort splits in dataset info

I suggest not to sort splits by their names in dataset_info in README so that they are displayed in the order specified in the loading script. Otherwise `test` split is displayed first, see this repo: https://huggingface.co/datasets/paws
What do you think?

But I added sorting in tests to fix CI (for the same dataset).


_The documentation is not available anymore as the PR was closed or merged._

It would be coherent with https://github.com/huggingface/datasets-server/issues/614#issuecomment-1290534153

I think we started working on this issue nearly at the same time... :sweat_smile: 
- CI was fixed with this: https://huggingface.co/datasets/paws/discussions/1

Related issue:
- #5202

@albertvillanova yeah I noticed it right after the PR :smile: thank you! the fix of the dataset info yaml fixes tests on CI, but in general order of splits in yaml influences the order in which they are displayed in the viewer, if I understand it correctly. So I suggest not to sort splits in yaml initially to avoid this for other datasets in the future. I think [this change](https://github.com/huggingface/datasets/pull/5201/files#diff-198ba4fdf2f94cb3e1aba8a0170a43b08d4ab5636d682374321c5a383a8be24dR571) should work for it. 

Changes to tests here maybe can be reverted considering that order in yaml now corresponds to the one in tests, thanks to your change in the dataset info.

Hehe, @polinaeterna, we make comments nearly at the same time as well... :laughing:
Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets. 


Labels: name: enhancement

also asked in https://discuss.huggingface.co/t/create-multiple-dataset-configs-with-push-to-hub-method/25480
Some datasets miss dataset_infos.json or dummy_data.zip

While working on dataset REAME generation script at https://github.com/madlag/datasets_readme_generator , I noticed that some datasets miss a dataset_infos.json : 

```
c4
lm1b
reclor
wikihow
```

And some does not have a dummy_data.zip : 

```
kor_nli
math_dataset
mlqa
ms_marco
newsgroup
qa4mre
qangaroo
reddit_tifu
super_glue
trivia_qa
web_of_science
wmt14
wmt15
wmt16
wmt17
wmt18
wmt19
xtreme
```

But it seems that some of those last do have a "dummy" directory .




Thanks for reporting.
We should indeed add all the missing dummy_data.zip and also the dataset_infos.json at least for lm1b, reclor and wikihow.

For c4 I haven't tested the script and I think we'll require some optimizations regarding beam datasets before processing it.


Closing since the dummy data generation is deprecated now (and the issue with missing metadata seems to be addressed).
Add a guide for semantic segmentation

Currently, we have these guides for object detection and image classification:

* https://huggingface.co/docs/datasets/object_detection
* https://huggingface.co/docs/datasets/image_classification

I am proposing adding a similar guide for semantic segmentation. 

I am happy to contribute a PR for it. 

Cc: @osanseviero @nateraw 

Labels: name: documentation

Sure this sounds great! Would this be pure torchvision, albumentations, or something else?

I am considering `torchvision` and `albumentations`. Also [works with TensorFlow](https://github.com/deep-diver/segformer-tf-transformers/blob/main/notebooks/TFSegFormer_Finetune.ipynb). 

I am assigning the issue to myself then.
add: segmentation guide.

Closes #5181 

I have opened a PR on Hub (https://huggingface.co/datasets/huggingface/documentation-images/discussions/5) to include the images in our central Hub repository. Once the PR is merged I will edit the image links. 

I have also prepared a [Colab Notebook](https://colab.research.google.com/drive/1BMDCfOTBnyshoME5RSxn5iQy-TWeFbOA?usp=sharing) in case anyone wants to play. 

- [x] Replace the image links 

Labels: name: documentation

_The documentation is not available anymore as the PR was closed or merged._

Thanks @osanseviero. Am I good to merge? 

I would wait for a second approval just in case :) 

Sure :) 

Merging since the images have been pushed as LFS files ([PR](https://huggingface.co/datasets/huggingface/documentation-images/discussions/8)).
Add warning

Fixes: #5105 

I think removing the directory with warning is a better solution for this issue. Because if we decide to keep existing files in directory, then we should deal with the case providing same directory for several datasets! Which we know is not possible since `dataset_info.json` exists in that directory.


As mentioned in https://github.com/huggingface/datasets/issues/5105 I think we just need to keep the existing files instead of deleting them.
The `dataset_info.json` file contains the split names anyway, so we know which files belong to the dataset, and which ones don't.
Use logging instead of printing to console

### Describe the bug

Some logs ([here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L778), [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L786), and [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L830)) generated by the `DatasetBuilder` are printed to the console instead of passed to `datasets` logger.


### Steps to reproduce the bug

```python
>> import datasets

>> datasets.load_dataset("some-dataset")

Downloading and preparing dataset csv/data to <path>...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7729.06it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 527.23it/s]
Dataset csv downloaded and prepared to <path>. Subsequent calls will reuse this data.
```

### Expected behavior

The logs should not be printed to the console directly but passed to the logger so that the user can redirect them wherever he wants.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-x86_64-i386-64bit
- Python version: 3.9.15
- PyArrow version: 10.0.0
- Pandas version: 1.5.1


Actually upon closer inspection, it is documented in the code that this behavior is intentional, so I'll close this.
Support skipping Parquet to Arrow conversion when using Beam



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4509). All of your documentation changes will be reflected on that endpoint.

When #4724 is merged, we can just pass `file_format="parquet"` to `download_and_prepare` and it will output parquet fiels without converting to arrow

I think we can close this one
Some links to canonical datasets in the docs are outdated

As we don't have canonical datasets in the github repo anymore, some old links to them doesn't work. I don't know how many of them are there, I found link to SuperGlue here: https://huggingface.co/docs/datasets/dataset_script#multiple-configurations, probably there are more of them. These links should be replaced by links to the corresponding datasets on the Hub. 

Labels: name: documentation

Thanks for catching this, I can go through the docs and replace the links to their corresponding datasets on the Hub!