Minimartzz/github-issues huggingface dataset:
Add "ipykernel" to list of `co_filename`s to remove
State: closed

Should resolve #5157 

Comments:
I don't know how I could add some tests for this, although jupyter is not among the dependencies so at least that would need to be added. If someone can tell a recommended way I will try to do it!

So testing by myself and looking around the jupyter codebase it looks like the `co_filename` of objects created within jupyter is of the form `f"{tempfile.tempdir}/ipykernel_{id1}/{id2}.py"` however I can't find the exact command setting it so I [asked in discourse](https://discourse.jupyter.org/t/co-filename-within-notebooks/16538). For now adapted the `co_filename` filter and added tests according to this I hope to get an answer and possibly fix based on that.

Ok ! I think it's fine to just check if the parent folder is named like `ipykernel_*` then

see the source code of how it's created:

https://github.com/ipython/ipykernel/blob/7f73ff705510b35d1e2faad7f5a676c620ce08d4/ipykernel/compiler.py#L72-L75

Should look better now didn't notice the duplicated tests

_The documentation is not available anymore as the PR was closed or merged._

Should work now on windows too

I did the changes you suggested and tried to rebase, the first part went fine, the second less so :( 

If you have time to spare, can you tell me what should I do now to fix this? thanks

Instead of rebasing you can just merge `main` into your branch, otherwise the GitHub preview of your PR shows changes of from `main`.

Feel free to close this PR and create a new one. Or alternatively your can force push to this PR with a new clean git history.

I have force-pushed and merged main, only shows the right changes, if you can run CI one more time it should be ok now

Hi, sorry I have been busy, the thing is I can't really understand why the test fail, besides the ugly thing I had done in the last commit to check if within CI smth stange happened with `os`, locally tests pass

The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6

I applied the same changes you did to this other function as well - it should be all good now

> The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6
> 
> I applied the same changes you did to this other function as well - it should be all good now

Thanks, it would have taken a long time to figure out :)
Fix CI require beam
State: closed

This PR:
- Fixes the CI `require_beam`: before it was requiring PyTorch instead
  ```python
  def require_beam(test_case):
      if not config.TORCH_AVAILABLE:
          test_case = unittest.skip("test requires PyTorch")(test_case)
      return test_case
  ```
- Fixes a missing `require_beam` in `test_beam_based_builder_download_and_prepare_as_parquet`
- Refactors `require_beam` to use `pytest` (`skipif`) instead

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm merging this PR because it is quite a trivial fix and this is required by:
- #5166
Add ffmpeg4 installation instructions in warnings
State: closed

Adds instructions on how to install `ffmpeg=4` on Linux (relevant for Colab users).

Looks pretty ugly because I didn't find a way to check `ffmpeg` version from python (without `subprocess.call()`; `ctypes.util.find_library` doesn't work`), so the warning is raised on each decoding. Any suggestions on how to make it look nice are welcome!

This is how it looks on Colab:
![image](https://user-images.githubusercontent.com/16348744/198052412-d48018d1-4416-4aa5-9114-f7f9b4af031f.png)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

To make it warn only once, feel free to use a global counter in python - and if the warning has already been done, you don't do it again

> Added the same formatting for the error message :)

nice!! thank you! 

> Oh and regarding the warning counter, you can do it in another PR maybe ?

Yes, more warnings is better then no warnings.... I'll merge when the CI passes
Memory explosion when trying to access 4d tensors in datasets cast to torch or np
State: open

### Describe the bug

When trying to access an item by index, in a datasets.Dataset cast to torch/np using `set_format` or `with_format`, we get a memory explosion if the item contains 4d (or above) tensors. 

### Steps to reproduce the bug

MWE:

```python
from datasets import load_dataset
import numpy as np

def create_4d_tensor(item):  
    i = item["num_nodes"]
    item["x_big"] = np.random.rand(i, 2*i, int(i/2), 1) + 1 # we create a big 4d tensor
    return item

if __name__ == "__main__":
    dataset = load_dataset(path=f"graphs-datasets/PROTEINS")

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset = dataset.map( 
        create_4d_tensor, 
        batched=False, 
        writer_batch_size=100, 
    )

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset.set_format("torch")
    
    print(dataset["train"].format)
    # This gets killed :(
    print(dataset["train"][0].keys())
```

The problem likely comes from `format_table` [here](https://cs.github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/src/datasets/arrow_dataset.py#L2328)

### Expected behavior

No memory explosion when trying to access dataset items after cast.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Comments:
WIP: drop labels in Image and Audio folders by default
State: closed

will fix https://github.com/huggingface/datasets/issues/5153 and redundant labels displaying for most of the images datasets on the Hub (which are used just to store files)

TODO: discuss adding `drop_labels` (and `drop_metadata`) params to yaml

Comments:
_The documentation is not available anymore as the PR was closed or merged._

close in favor of https://github.com/huggingface/datasets/pull/5192
Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6
State: closed

### Describe the bug

When using `pip-compile` (part of `pip-tools`) to generate a pinned requirements file that includes `datasets`, a version conflict of `dill` appears.

It is caused by a transitive dependency conflict between `datasets` and `multiprocess`.

### Steps to reproduce the bug

```bash
$ echo "datasets" > requirements.in
$ pip install pip-tools
$ pip-compile requirements.in 
Could not find a version that matches dill<0.3.6,>=0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
Tried: 0.2, 0.2, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.2.6, 0.2.7, 0.2.7.1, 0.2.8, 0.2.8.1, 0.2.8.2, 0.2.9, 0.3.0, 0.3.1, 0.3.1.1, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.3.5, 0.3.5, 0.3.5.1, 0.3.5.1, 0.3.6, 0.3.6
Skipped pre-versions: 0.1a1, 0.2a1, 0.2a1, 0.2b1, 0.2b1
There are incompatible versions in the resolved dependencies:
  dill<0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
  dill>=0.3.6 (from multiprocess==0.70.14->datasets==2.6.1->-r requirements.in (line 1))
```

### Expected behavior

A correctly generated file `requirements.txt` with pinned dependencies

### Environment info

Tested with versions `2.6.1, 2.6.0, 2.5.2` on Python 3.8 and 3.10 on Ubuntu 20.04LTS and Python 3.10 on MacOS 12.6 (M1).

Comments:
Thanks for reporting, @Rijgersberg.

We were waiting for the release of `dill` 0.3.6, that happened 2 days ago (24 Oct 2022): https://github.com/uqfoundation/dill/releases/tag/dill-0.3.6
- See comment: https://github.com/huggingface/datasets/pull/4397#discussion_r880629543

Also `multiprocess` 0.70.14 was released 2 days ago: https://github.com/uqfoundation/multiprocess/releases/tag/multiprocess-0.70.14

We are addressing this issue to align dependencies.

In your specific setup, I guess the compatible configuration is with `multiprocess` 0.70.13 (instead of 0.70.14).

@Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Thanks!

> @Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Any chance you have a eta? 

@StefanSamba we are disussing about making a release early this week.

@Rijgersberg, please also that you can make `pip-compile` work by using the backtracking resolver (instead of the legacy one): https://pip-tools.readthedocs.io/en/latest/#a-note-on-resolvers
```
pip-compile --resolver=backtracking requirements.in
```
This resolver will automatically use `multiprocess` 0.70.13 version.
Dataset can’t cache model’s outputs
State: closed

### Describe the bug

Hi,

I try to cache some outputs of teacher model( Knowledge Distillation ) by using map function of Dataset library, while every time I run my code, I still recompute all the sequences. I tested Bert Model like this, I got different hash every single run, so any idea to deal with this?

### Steps to reproduce the bug

1. run below code
2. get different hash
```
from transformers import BertModel
from transformers import AutoTokenizer
import torch
token = ['hello']
model = BertModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
def abcd():
    with torch.no_grad():
        
        out = model(**tok(token,return_tensors='pt'))[0]
        # out = tok(token)
    return out
from datasets.fingerprint import Hasher
my_func = abcd
print(Hasher.hash(my_func))

print(abcd())
```

### Expected behavior

I wanna cache all the model output 

### Environment info

datasets:2.5.0

Comments:
Addressed in https://github.com/huggingface/datasets/pull/5191 (torch.Tensor objects now produce deterministic hashes)
Fix language and license tag names in all Hub datasets
State: closed
Labels: description: Contribution to a dataset script
name: dataset contribution

While working on this:
- #5137

we realized there are still many datasets with deprecated "languages" and "licenses" tag names (instead of "language" and "license").

This is a blocking issue: no subsequent PR can be opened to modify their metadata: a ValueError will be thrown.

We should fix the "language" and "license" tag names in all Hub datasets.

TODO:
- [x] Fix language and license tag names in 402 Hub datasets

CC: @julien-c 

Comments:
There are currently 402 datasets with deprecated "languages" or "licenses".

hey @albertvillanova ,i would love to work on this issue if you like.

Hi @ayushthe1, thanks for your offer.

But as you can see, I self-assigned this issue.

I have already fixed 200 out of the 402 datasets. My script is still running and fixing the rest.

For example: https://huggingface.co/datasets/fhamborg/news_sentiment_newsmtsc/discussions/2/files

Thanks for your time. Will try next time. 😇

@ayushthe1 feel free to take one of the non-assigned open issues: https://github.com/huggingface/datasets/issues

This is done.
Consistent caching between python and jupyter
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen

Comments:
Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`

Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!
TextConfig: added "errors"
State: closed

This patch adds the ability to set the `errors` option of `open` for loading text datasets. I needed it because some data I had scraped had bad bytes in it, so I needed `errors='ignore'`. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for adding this ! You can fix the CI by formatting your code using the `make style` command :)

[**@lhoestq**](https://github.com/lhoestq) commented on [Oct 27, 2022, 4:08 PM GMT+3:30](https://github.com/huggingface/datasets/pull/5155#issuecomment-1293464680 "2022-10-27T12:38:04Z - Replied by Github Reply Comments"):
> Thanks for adding this ! You can fix the CI by formatting your code using the `make  style` command :)

I ran this and force pushed the changes.
default Image/AudioFolder infers labels when there is no metadata files even if there is only one dir
State: closed
Labels: description: Something isn't working
name: bug

### Describe the bug

By default FolderBasedBuilder infers labels if there is not metadata files, even if it's meaningless (for example, they are in a single directory or in the root folder, see this repo as an example: https://huggingface.co/datasets/patrickvonplaten/audios

As this is a corner case for quick exploration of images or audios on the Hub. 

### Steps to reproduce the bug

If you have directory like this:

```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
or

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
```
doing `ds = load_dataset(repo)` would create `label` feature:
```python
print(ds["train"][0])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 0}
```

Also, if you have the following structure:

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
   image4.jpg
   image5.jpg
   image6.jpg
```
it will infer two labels:

```python
print(ds["train"][0])
print(ds["train"][-1])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 1}
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x415 at 0x7FB5326555B0>, 'label': 0}
```

### Expected behavior

We should have only one base feature (Image/Audio) in such cases.

### Environment info

all versions of `datasets`

Comments:
Makes sense! For the last structure, we could count the path segments (delimited by "/" for URLs and `os.sep`  for local paths) to ensure all inferred labels are on the same level. Otherwise, I think it's safe to assume they are meaningless and ignore them.
Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)
State: open
Labels: description: New feature or request
name: enhancement

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets. 


Comments:
also asked in https://discuss.huggingface.co/t/create-multiple-dataset-configs-with-push-to-hub-method/25480
Cannot find the rvl_cdip dataset
State: closed

Hi,
I am trying to use load_dataset to load the official "rvl_cdip" dataset but getting an error.

dataset = load_dataset("rvl_cdip")

Couldn't find 'rvl_cdip' on the Hugging Face Hub either: FileNotFoundError: Couldn't find the file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/rvl_cdip/rvl_cdip.py

Regards,


Comments:
Hi, @santule.

We have transferred all dataset scripts from GitHub to the Hugging Face Hub: https://huggingface.co/datasets
- Concretely, you have "rvl_cdip" here: https://huggingface.co/datasets/rvl_cdip

To be able to load them, you should update your `datasets` library:
```
pip install -U datasets
```

thank you, it worked
Dataset order is not deterministic with ZIP archives and `iter_files`
State: closed

### Describe the bug

For the `beans` dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

### Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

```python
from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])
```

### Expected behavior

The order of the images is the same on all machines.

### Environment info

On the EC2 instance:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked
```

On my local laptop:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1
```

On github actions:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4
```

Comments:
Thanks for reporting ! The issue doesn't come from shuffling, but from `beans` row order not being deterministic:

https://huggingface.co/datasets/beans/blob/main/beans.py uses `dl_manager.iter_files` on ZIP archives and the file order doesn't seen to be deterministic and changes across machines

Thank you for noticing indeed!

This is still a bug, so I'd keep this one open if you don't mind ;)

Besides the linked PR, to make the loading process fully deterministic, I believe we should also sort the data files [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L276) and [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L485) (e.g. fsspec's `LocalFileSystem.glob` relies on `os.scandir`, which yields the contents in arbitrary order). My concern is the overhead of these sorts... Maybe we could introduce a new flag to `load_dataset` similar to TFDS' [`shuffle_files`](https://www.tensorflow.org/datasets/determinism#determinism_when_reading) or sort only if the number of data files is small?

We already return the result sorted at the end of `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository` if I'm not mistaken

@lhoestq Oh, you are right. Feel free to ignore my comment.

I think the corresponding PR is ready to be merged :hugs: 

@albertvillanova Thanks for the fix!
Inconsistent documentation on map remove_columns
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

description: This issue or pull request already exists
name: duplicate

description: Good for newcomers
name: good first issue

name: hacktoberfest

### Describe the bug

The page [process](https://huggingface.co/docs/datasets/process) says this about the parameter `remove_columns` of the function `map`:
When you remove a column, it is only removed after the example has been provided to the mapped function.

So it seems that the `remove_columns` parameter removes after the mapped functions.

However, another page, [the documentation of the function map](https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.Dataset.map.remove_columns) says:
Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in remove_columns, these columns will be kept.

So one page says "after the mapped function" and another says "before the mapped function."
Is there something wrong?


### Steps to reproduce the bug

Not about code.

### Expected behavior

consistent about the descriptions of the behavior of the parameter `remove_columns` in the function `map`.

### Environment info

datasets V2.6.0

Comments:
Thanks for reporting, @zhaowei-wang-nlp.

You are right, the documentation is confusing on the behavior of `remove_columns`. We should better explain it. 

This is a duplicate of https://github.com/huggingface/datasets/issues/2343.

I'm closing this issue because as @mariosasko pointed out, it is a duplicate of:
- #2343
DownloadManager Git LFS support
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

Maybe I'm mistaken but the `DownloadManager` does not support extracting git lfs files out of the box right?
Using `dl_manager.download()` or `dl_manager.download_and_extract()` still returns lfs files afaict. 

Is there a good way to write a dataset loading script for a repo with lfs files?

### Motivation

/

### Your contribution

/

Comments:
Hey ! Actually it works, just pass the right URL ;)
The URL must be the one with “/resolve/”

e.g. https://huggingface.co/datasets/imagenet-1k/resolve/main/data/test_images.tar.gz

You can even pass a relative path to the dl_manager instead, like `dl_manager.download("data/test_images.tar.gz")`

Amazing it works, thanks!
Deprecate num_proc parameter in  DownloadManager.extract
State: closed

fixes #5132 : Deprecated the `num_proc` parameter in `DownloadManager.extract` by passing `num_proc` parameter to `map_nested` .

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @mariosasko . Can you please help me with why the tests keep failing.  I have reviewed the code changes multiple times but can't spot any mistakes. 

You can fix this failure by formatting your code with the `make style` command (run it from the root of the cloned repo).

hey @mariosasko ,i cant understand how to use the `make style` command .I searched for it on the internet but cant find any results. 
So i formatted the code using vs-code document formatter. Hope this helps.

`make style` runs the "style" target defined here: https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12

This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile





> `make style` runs the "style" target defined here:
> 
> https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12
> 
> This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile

Thanks! I will look into this :relaxed:
Update docs once dataset scripts transferred to the Hub
State: closed

Todo:
- [x] Update docs:
  - [x] Datasets on GitHub (legacy)
  - [x] Load: offline
  - [x] About dataset load:
    - [x] Maintaining integrity
    - [x] Security
- [x] Update docstrings:
  - [x] Inspect:
    - [x] get_dataset_config_info
    - [x] get_dataset_split_names
  - [x] Load:
    - [x] dataset_module_factory
    - [x] load_dataset_builder
    - [x] load_dataset
- [x] Remove `ADD_NEW_DATASET.md`
- [x] Update `.github/ISSUE_TEMPLATE/config.yml`

Fix #5135.

Comments:
_The documentation is not available anymore as the PR was closed or merged._