ahmedfarazsyk/datasets-github-issues huggingface dataset:
Allow passing a subset of output features to Dataset.map
Author: sanderland
State: open
Labels: name: enhancement

### Feature request

Currently, map does one of two things to the features (if I'm not mistaken):

* when you do not pass features, types are assumed to be equal to the input if they can be cast, and inferred otherwise
* when you pass a full specification of features, output features are set to this

However, sometimes you want to just pass some of the output types, particularly when the first of these modes makes an incorrect type. This currently crashes.

### Motivation

To give a little background: this problem appears in converting labels to ids, where the labels happen to be floats rather than strings

Consider the following use of map to convert from float to int
```python
data = Dataset.from_dict({'y':[1.0,2.0,3.0]})
mapped = data.map(lambda r: {'y': int(r['y'])})
mapped['y'] # is floats, not ints
```

The result is a float again, since after the mapping operation it forces the old datatypes back on the data. 

Passing `features=Features({"y": Value(dtype="int64")})` to map works in principle, but then extending it a little to e.g.
```python
def format_data(r):
    return {**tokenizer(r["text"]), "y": int(r["y"])}

data = Dataset.from_dict({"y": [1.0, 2.0, 3.0], "text": ["one", "two", "three"]})
mapped = data.map(
    format_data,
    features=Features({'y': Value(dtype="int64")}),
    remove_columns=["text"],
)
```

Results in a crash in dataset internals, as it expects either all or no output features to be specified.
Of course one can pass a full feature specification, but this becomes tokenizer specific and very awkward.


### Your contribution

I've looked at `write_batch` and particularly `col_type = features[col] if features else None`, but checking for `col in features` here makes it fail elsewhere, but the structure makes it hard to understand how and why. I do not think I would have the time myself to get to the bottom of this anytime soon.
Loading an external dataset in a format similar to conll2003
Author: Taghreed7878
State: closed

I'm trying to load a custom dataset in a Dataset object, it's similar to conll2003 but with 2 columns only (word entity), I used the following script:

features = datasets.Features(
            {"tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=["B-PER", .... etc.]))}
        )


from datasets import Dataset

INPUT_COLUMNS = "tokens ner_tags".split(" ")

def read_conll(file):
    #all_labels = []
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
          if line:
            if line.startswith("-DOCSTART-") and example["tokens"] != []:
                print(idx, example)
                yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            elif line == "\n" or (line.startswith("-DOCSTART-") and example["tokens"] == []):
              continue
            else:
                row_cols = line.split(" ")
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "/content/new_train.txt"}, features = features)   

The following error happened:
[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <genexpr>(.0)
    285     for key in unique_values(itertools.chain(*dicts)):  # set merge all keys
    286         # Will raise KeyError if the dict don't have the same keys
--> 287         yield key, tuple(d[key] for d in dicts)
    288 
    

TypeError: tuple indices must be integers or slices, not str

What does this mean and what should I modify?
Add notebook / other resource links to the task-specific data loading guides
Author: sayakpaul
State: closed
Labels: name: enhancement

Does it make sense to include links to notebooks / scripts that show how to use a dataset for training / fine-tuning a model? 

For example, here in [https://huggingface.co/docs/datasets/image_classification] we could include a mention of https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb.

Applies to https://huggingface.co/docs/datasets/object_detection as well.

Cc: @osanseviero @nateraw 

Comments: Yea this would be great! We would need an object detection tutorial notebook too if it doesn't already exist there. 

There is one: https://huggingface.co/docs/datasets/object_detection.

I will start the work.
Add a guide for semantic segmentation
Author: sayakpaul
State: closed
Labels: name: documentation

Currently, we have these guides for object detection and image classification:

* https://huggingface.co/docs/datasets/object_detection
* https://huggingface.co/docs/datasets/image_classification

I am proposing adding a similar guide for semantic segmentation. 

I am happy to contribute a PR for it. 

Cc: @osanseviero @nateraw 

Comments: Sure this sounds great! Would this be pure torchvision, albumentations, or something else?

I am considering `torchvision` and `albumentations`. Also [works with TensorFlow](https://github.com/deep-diver/segformer-tf-transformers/blob/main/notebooks/TFSegFormer_Finetune.ipynb). 

I am assigning the issue to myself then.
An example or recommendations for creating large image datasets?
Author: sayakpaul
State: open

I know that Apache Beam and `datasets` have [some connector utilities](https://huggingface.co/docs/datasets/beam). But it's a little unclear what we mean by "But if you want to run your own Beam pipeline with Dataflow, here is how:". What does that pipeline do? 

As a user, I was wondering if we have this support for creating large image datasets. If so, we should mention that [here](https://huggingface.co/docs/datasets/image_dataset). 

Cc @lhoestq 

Comments: The beam utilities allow to prepare a dataset as parquet in your cloud storage. From my perspective this CLI is not super easy to use, but we've been working on a new python API to prepare a dataset in your cloud storage:
```python
from datasets import load_dataset_builder

builder = load_dataset_builder("c4", "en")
builder.download_and_prepapre("s3://my-bucket/c4", file_format="parquet")
```

And to use Beam you can do:
```python
beam_runner = ...  # one of "SparkRunner", "DataFlowRunner", "DirectRunner", etc.
beam_options = ...

builder.download_and_prepapre(
    "s3://my-bucket/c4",
    file_format="parquet",
    beam_runner=beam_runner,
    beam_options=beam_options
)
```

Though Beam can be used ONLY if there is a dataset script based on the `BeamBasedBuilder` right now - it doesn't work on an arbitrary dataset (see [wikipedia.py](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py) for example).

Thanks! 

Would be nice to have something similar for creating large image datasets.
Loading an external NER dataset
Author: Taghreed7878
State: closed

I need to use huggingface datasets to load a custom dataset similar to conll2003 but with more entities and each the files contain only two columns: word and ner tag.

I tried this code snnipet that I found here as an answer to a similar issue:

from datasets import Dataset

INPUT_COLUMNS = "ID Text NER".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

train = Dataset.from_generator(read_conll, gen_kwargs={"file": "some_path"})   

But the following error happened:

ValueError: Please pass `features` or at least one example when writing data
Inconsistency behavior between handling local file protocol and other FS protocols
Author: leoleoasd
State: open

### Describe the bug

These lines us used during load_from_disk:
```
if is_remote_filesystem(fs):
    dest_dataset_dict_path = extract_path_from_uri(dataset_dict_path)
else:
    fs = fsspec.filesystem("file")
    dest_dataset_dict_path = dataset_dict_path
```
If a local FS is given, then it will the URL as the path name. If a remote Fs is given, then it will use the path of the URL. This is an inconsistent behavior when handling a file: when using remote FS, you must write a URL, but for local FS, even if you passed LocalFileSystem as `fs` you still can't use a `file://` URL. It will be recognized as a directory named `file:`.

### Steps to reproduce the bug

```
import fsspec.core
url = "hdfs:///somewhere/MNIST"
# url = "file:///somewhere/MNIST"
fs, path = fsspec.core.url_to_fs(url)

fs.ls(path) # this will always work
load_from_disk(path, fs) # only works for local FS
load_from_disk(url, fs) # only works for remote FS

```

### Expected behavior

one of `url` or `path` should always work

I think we extract path from given URL by using `fsspec.core.url_to_fs` instead of using `is_remote_filesystem` and `extract_path_from_uri` will fix this, since:
```
fsspec.core.url_to_fs("/somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("file:///somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("hdfs:///somewhere/MNIST") ->  HDFS, '/somewhere/MNIST'
```
and
```
fsspec.core.url_to_fs("file:///somewhere/MNIST") == fsspec.core.url_to_fs("/somewhere/MNIST")
```
In theory, this wouldn't break anything, since giving local path and remote uri still works. It will only affect local URI (make it works too)

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-5.4.205.1**HIDDEN**
- Python version: 3.7.10
- PyArrow version: 8.0.0
- Pandas version: 1.2.4
[Caching] Deterministic hashing of torch tensors
Author: lhoestq
State: closed
Labels: name: enhancement

Currently this fails
```python
import torch
from datasets.fingerprint import Hasher

t = torch.tensor([1.])

def func(x):
    return t + x

hash1 = Hasher.hash(func)
t = torch.tensor([1.])
hash2 = Hasher.hash(func)
assert hash1 == hash2
```

Also as noticed in https://discuss.huggingface.co/t/dataset-cant-cache-models-outputs/24945, using a model in a `map` function doesn't work well with caching. Indeed the `bert-base-uncased` model has a different hash every time you reload it. Supporting torch tensors may also help in this case.

This can be fixed by registering a custom pickling functions for torch tensors - as we did for other objects such as CodeType, FunctionType and Regex in `py_utils.py`
Memory explosion when trying to access 4d tensors in datasets cast to torch or np
Author: clefourrier
State: open

### Describe the bug

When trying to access an item by index, in a datasets.Dataset cast to torch/np using `set_format` or `with_format`, we get a memory explosion if the item contains 4d (or above) tensors. 

### Steps to reproduce the bug

MWE:

```python
from datasets import load_dataset
import numpy as np

def create_4d_tensor(item):  
    i = item["num_nodes"]
    item["x_big"] = np.random.rand(i, 2*i, int(i/2), 1) + 1 # we create a big 4d tensor
    return item

if __name__ == "__main__":
    dataset = load_dataset(path=f"graphs-datasets/PROTEINS")

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset = dataset.map( 
        create_4d_tensor, 
        batched=False, 
        writer_batch_size=100, 
    )

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset.set_format("torch")
    
    print(dataset["train"].format)
    # This gets killed :(
    print(dataset["train"][0].keys())
```

The problem likely comes from `format_table` [here](https://cs.github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/src/datasets/arrow_dataset.py#L2328)

### Expected behavior

No memory explosion when trying to access dataset items after cast.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6
Author: Rijgersberg
State: closed

### Describe the bug

When using `pip-compile` (part of `pip-tools`) to generate a pinned requirements file that includes `datasets`, a version conflict of `dill` appears.

It is caused by a transitive dependency conflict between `datasets` and `multiprocess`.

### Steps to reproduce the bug

```bash
$ echo "datasets" > requirements.in
$ pip install pip-tools
$ pip-compile requirements.in 
Could not find a version that matches dill<0.3.6,>=0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
Tried: 0.2, 0.2, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.2.6, 0.2.7, 0.2.7.1, 0.2.8, 0.2.8.1, 0.2.8.2, 0.2.9, 0.3.0, 0.3.1, 0.3.1.1, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.3.5, 0.3.5, 0.3.5.1, 0.3.5.1, 0.3.6, 0.3.6
Skipped pre-versions: 0.1a1, 0.2a1, 0.2a1, 0.2b1, 0.2b1
There are incompatible versions in the resolved dependencies:
  dill<0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
  dill>=0.3.6 (from multiprocess==0.70.14->datasets==2.6.1->-r requirements.in (line 1))
```

### Expected behavior

A correctly generated file `requirements.txt` with pinned dependencies

### Environment info

Tested with versions `2.6.1, 2.6.0, 2.5.2` on Python 3.8 and 3.10 on Ubuntu 20.04LTS and Python 3.10 on MacOS 12.6 (M1).

Comments: Thanks for reporting, @Rijgersberg.

We were waiting for the release of `dill` 0.3.6, that happened 2 days ago (24 Oct 2022): https://github.com/uqfoundation/dill/releases/tag/dill-0.3.6
- See comment: https://github.com/huggingface/datasets/pull/4397#discussion_r880629543

Also `multiprocess` 0.70.14 was released 2 days ago: https://github.com/uqfoundation/multiprocess/releases/tag/multiprocess-0.70.14

We are addressing this issue to align dependencies.

In your specific setup, I guess the compatible configuration is with `multiprocess` 0.70.13 (instead of 0.70.14).

@Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Thanks!

> @Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Any chance you have a eta? 

@StefanSamba we are disussing about making a release early this week.

@Rijgersberg, please also that you can make `pip-compile` work by using the backtracking resolver (instead of the legacy one): https://pip-tools.readthedocs.io/en/latest/#a-note-on-resolvers
```
pip-compile --resolver=backtracking requirements.in
```
This resolver will automatically use `multiprocess` 0.70.13 version.
Dataset can’t cache model’s outputs
Author: jongjyh
State: closed

### Describe the bug

Hi,

I try to cache some outputs of teacher model( Knowledge Distillation ) by using map function of Dataset library, while every time I run my code, I still recompute all the sequences. I tested Bert Model like this, I got different hash every single run, so any idea to deal with this?

### Steps to reproduce the bug

1. run below code
2. get different hash
```
from transformers import BertModel
from transformers import AutoTokenizer
import torch
token = ['hello']
model = BertModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
def abcd():
    with torch.no_grad():
        
        out = model(**tok(token,return_tensors='pt'))[0]
        # out = tok(token)
    return out
from datasets.fingerprint import Hasher
my_func = abcd
print(Hasher.hash(my_func))

print(abcd())
```

### Expected behavior

I wanna cache all the model output 

### Environment info

datasets:2.5.0

Comments: Addressed in https://github.com/huggingface/datasets/pull/5191 (torch.Tensor objects now produce deterministic hashes)
Fix language and license tag names in all Hub datasets
Author: albertvillanova
State: closed
Labels: name: dataset contribution

While working on this:
- #5137

we realized there are still many datasets with deprecated "languages" and "licenses" tag names (instead of "language" and "license").

This is a blocking issue: no subsequent PR can be opened to modify their metadata: a ValueError will be thrown.

We should fix the "language" and "license" tag names in all Hub datasets.

TODO:
- [x] Fix language and license tag names in 402 Hub datasets

CC: @julien-c 

Comments: There are currently 402 datasets with deprecated "languages" or "licenses".

hey @albertvillanova ,i would love to work on this issue if you like.

Hi @ayushthe1, thanks for your offer.

But as you can see, I self-assigned this issue.

I have already fixed 200 out of the 402 datasets. My script is still running and fixing the rest.

For example: https://huggingface.co/datasets/fhamborg/news_sentiment_newsmtsc/discussions/2/files

Thanks for your time. Will try next time. 😇

@ayushthe1 feel free to take one of the non-assigned open issues: https://github.com/huggingface/datasets/issues

This is done.
Consistent caching between python and jupyter
Author: gpucce
State: closed
Labels: name: enhancement

### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen

Comments: Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`

Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!
default Image/AudioFolder infers labels when there is no metadata files even if there is only one dir
Author: polinaeterna
State: closed
Labels: name: bug

### Describe the bug

By default FolderBasedBuilder infers labels if there is not metadata files, even if it's meaningless (for example, they are in a single directory or in the root folder, see this repo as an example: https://huggingface.co/datasets/patrickvonplaten/audios

As this is a corner case for quick exploration of images or audios on the Hub. 

### Steps to reproduce the bug

If you have directory like this:

```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
or

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
```
doing `ds = load_dataset(repo)` would create `label` feature:
```python
print(ds["train"][0])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 0}
```

Also, if you have the following structure:

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
   image4.jpg
   image5.jpg
   image6.jpg
```
it will infer two labels:

```python
print(ds["train"][0])
print(ds["train"][-1])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 1}
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x415 at 0x7FB5326555B0>, 'label': 0}
```

### Expected behavior

We should have only one base feature (Image/Audio) in such cases.

### Environment info

all versions of `datasets`

Comments: Makes sense! For the last structure, we could count the path segments (delimited by "/" for URLs and `os.sep`  for local paths) to ensure all inferred labels are on the same level. Otherwise, I think it's safe to assume they are meaningless and ignore them.
Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)
Author: polinaeterna
State: open
Labels: name: enhancement

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets. 


Comments: also asked in https://discuss.huggingface.co/t/create-multiple-dataset-configs-with-push-to-hub-method/25480
Cannot find the rvl_cdip dataset
Author: santule
State: closed

Hi,
I am trying to use load_dataset to load the official "rvl_cdip" dataset but getting an error.

dataset = load_dataset("rvl_cdip")

Couldn't find 'rvl_cdip' on the Hugging Face Hub either: FileNotFoundError: Couldn't find the file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/rvl_cdip/rvl_cdip.py

Regards,


Comments: Hi, @santule.

We have transferred all dataset scripts from GitHub to the Hugging Face Hub: https://huggingface.co/datasets
- Concretely, you have "rvl_cdip" here: https://huggingface.co/datasets/rvl_cdip

To be able to load them, you should update your `datasets` library:
```
pip install -U datasets
```

thank you, it worked