Minimartzz/github-issues huggingface dataset:
Drop labels in Image and Audio folders if files are on different levels in directory or if there is only one label
State: closed
Labels: description: Something isn't working
name: bug

Will close https://github.com/huggingface/datasets/issues/5153

Drop labels by default (`drop_labels=None`) when:
* there are files on different levels of directory hierarchy by checking their path depth
* all files are in the same directory (=only one label was inferred)

First one fixes cases like this:
```
repo
   image3.jpg
   image4.jpg
   data
      image1.jpg
      image2.jpg
```
Second one fixes cases like this:
```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
This is mostly to fix the viewer for people who just drop images in the Hub interface into the root dir.

I added tests for both of the cases on local and remote files. **I also changed data files for old test on drop_labels** (`test_generate_examples_drop_labels`). The files I provide to `test_generate_examples_drop_labels` now has "canonical" classification structure (two dirs) in order not to change the logic of the test (=not to check these two cases addressed in the PR).



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

> Nit: maybe we can use the count_path_segments function from this file for counting (updated with your logic to make it faster).

@mariosasko just to make sure I understood you correctly - are you okay with this change? (actually `os.path.normpath` is redundant here as paths from `data_files` should be already normalized but just in case)
https://github.com/huggingface/datasets/pull/5192/files#diff-1f09f7a178211f7539b1499b64b69793bd53b30c8b7b34cfcc5835e25d31929fR33
If you are, we can merge.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

awesome ! :D
Make torch.Tensor and spacy models cacheable
State: closed

Override `Pickler.save` to implement deterministic reduction (lazily registered; inspired by https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L343) functions for `torch.Tensor` and spaCy models.

Fix https://github.com/huggingface/datasets/issues/5170, fix https://github.com/huggingface/datasets/issues/3178


Comments:
_The documentation is not available anymore as the PR was closed or merged._
`path` is `None` when downloading a custom audio dataset from the Hub
State: closed

### Describe the bug

I've created an [audio dataset](https://huggingface.co/datasets/lewtun/audio-test-push) using the `audiofolder` feature desribed in the [docs](https://huggingface.co/docs/datasets/audio_dataset#audiofolder) and then pushed it to the Hub. 

Locally, I can see the `audio.path` feature is of the expected form `path/to/data_dir`, but when I download the dataset from the Hub, I see `audio.path` is `None`

Here's an example:

```python
from datasets import load_dataset

ds = load_dataset("lewtun/audio-test-push")
ds["train"][0]
# {
#     "audio": {
#         "path": None,    <-- Is this expected?
#         "array": array(
#             [
#                 3.97140226e-07,
#                 7.30310290e-07,
#                 7.56406735e-07,
#                 ...,
#                 -1.19636677e-01,
#                 -1.16811886e-01,
#                 -1.12441722e-01,
#             ]
#         ),
#         "sampling_rate": 44100,
#     },
#     "song_id": 0,
#     "genre_id": 0,
#     "genre": "Electronic",
# }
```

Is this expected behaviour? If yes, feel free to close this issue as it's not a true bug then :)

### Steps to reproduce the bug

1. Create an audio dataset with the `audiofolder` feature
2. Push the dataset to the Hub with `push_to_hub()`
3. Download the Hub dataset and inspect the `audio.path` feature

### Expected behavior

`audio.path` points to the file associated with the audio data

### Environment info

- `datasets` version: 2.6.2.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1

Comments:
Hi! Yes, this is expected behavior - we do this as a security measure to not leak local paths (this info would be useless on other users' machines anyways) and only push audio bytes.
add: segmentation guide.
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

Closes #5181 

I have opened a PR on Hub (https://huggingface.co/datasets/huggingface/documentation-images/discussions/5) to include the images in our central Hub repository. Once the PR is merged I will edit the image links. 

I have also prepared a [Colab Notebook](https://colab.research.google.com/drive/1BMDCfOTBnyshoME5RSxn5iQy-TWeFbOA?usp=sharing) in case anyone wants to play. 

- [x] Replace the image links 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @osanseviero. Am I good to merge? 

I would wait for a second approval just in case :) 

Sure :) 

Merging since the images have been pushed as LFS files ([PR](https://huggingface.co/datasets/huggingface/documentation-images/discussions/8)).
chore: add notebook links to img cls and obj det.
State: closed
Labels: description: New feature or request
name: enhancement

Closes https://github.com/huggingface/datasets/issues/5182

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@nateraw I guess the failing test is unrelated.  

@sayakpaul Yea failures are unrelated. 

Alright. Will wait for @osanseviero's take and then merge. 

FYI @stevhliu 

@osanseviero @stevhliu @nateraw thank you for your comments. Acted on them.

Thanks! Can I merge? Or should we wait for approvals from the others?

Since @stevhliu approved as well, I think you're good to go

Alright!

Merging as a Member for the first time 🫀
Incorrect error message when Dataset.from_sql fails and sqlalchemy not installed
State: closed

### Describe the bug

When calling `Dataset.from_sql` (in my case, with sqlite3), it fails with a message ```ValueError: Please pass `features` or at least one example when writing data``` when I don't have `sqlalchemy` installed. 


### Steps to reproduce the bug

Make a new sqlite db with `sqlite3` and `pandas` from a remote [URL](https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv).

```python
import sqlite3
import pandas as pd
from datasets import Dataset

conn = sqlite3.connect('us_covid_data.db')
df = pd.read_csv('https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv')
df.to_sql('states', conn, if_exists='replace')
```

Then if you try to query this DB like this:

```python
ds = Dataset.from_sql('''SELECT * from states WHERE state=="New York";''', "sqlite:///us_covid_data.db")
```

You run into the error I described above:

```ValueError: Please pass `features` or at least one example when writing data```

However, if you try to pass features, as the error suggests, then you get an error that tells you the underlying problem...

```python
from datasets import Dataset, Features, Value

features = Features({
    'date': Value('date32'),
    'label': Value('string'),
    'fips': Value('int32'),
    'cases': Value('int32'),
    'deaths': Value('int32')
})
ds = Dataset.from_sql(
    '''SELECT * from states WHERE state=="New York";''',
    "sqlite:///us_covid_data.db",
    features=features
)
```

Which results in the actual underlying error: `ImportError: Using URI string without sqlalchemy installed.`

### Expected behavior

Instead of `ValueError` about needing to pass features, we should provide the actual underlying error about not having SQLAlchemy installed when it isn't found in the environment. 

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.10
- PyArrow version: 10.0.0
- Pandas version: 1.2.5

Comments:
Hi! The first `Dataset.from_sql` call also outputs the "ImportError: Using URI string without sqlalchemy installed." message, but you also get "During handling of the above exception another exception occurred: ..." after which the ValueError is printed. I agree that this behavior makes it easy to miss the original error. 

I think we can improve this by not throwing the writer's ValueError if the error from a dataset script is already being handled to make debugging easier. @lhoestq @albertvillanova wdyt?

Yup ! Alternatively the error can be raised in sql.py before generating the examples ? In `_info` for example

yea @lhoestq that would probably be good. The 2nd error is useless if the 1st error is the real reason it failed.
Allow passing a subset of output features to Dataset.map
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Currently, map does one of two things to the features (if I'm not mistaken):

* when you do not pass features, types are assumed to be equal to the input if they can be cast, and inferred otherwise
* when you pass a full specification of features, output features are set to this

However, sometimes you want to just pass some of the output types, particularly when the first of these modes makes an incorrect type. This currently crashes.

### Motivation

To give a little background: this problem appears in converting labels to ids, where the labels happen to be floats rather than strings

Consider the following use of map to convert from float to int
```python
data = Dataset.from_dict({'y':[1.0,2.0,3.0]})
mapped = data.map(lambda r: {'y': int(r['y'])})
mapped['y'] # is floats, not ints
```

The result is a float again, since after the mapping operation it forces the old datatypes back on the data. 

Passing `features=Features({"y": Value(dtype="int64")})` to map works in principle, but then extending it a little to e.g.
```python
def format_data(r):
    return {**tokenizer(r["text"]), "y": int(r["y"])}

data = Dataset.from_dict({"y": [1.0, 2.0, 3.0], "text": ["one", "two", "three"]})
mapped = data.map(
    format_data,
    features=Features({'y': Value(dtype="int64")}),
    remove_columns=["text"],
)
```

Results in a crash in dataset internals, as it expects either all or no output features to be specified.
Of course one can pass a full feature specification, but this becomes tokenizer specific and very awkward.


### Your contribution

I've looked at `write_batch` and particularly `col_type = features[col] if features else None`, but checking for `col in features` here makes it fail elsewhere, but the structure makes it hard to understand how and why. I do not think I would have the time myself to get to the bottom of this anytime soon.

Comments:
Loading an external dataset in a format similar to conll2003
State: closed

I'm trying to load a custom dataset in a Dataset object, it's similar to conll2003 but with 2 columns only (word entity), I used the following script:

features = datasets.Features(
            {"tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=["B-PER", .... etc.]))}
        )


from datasets import Dataset

INPUT_COLUMNS = "tokens ner_tags".split(" ")

def read_conll(file):
    #all_labels = []
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
          if line:
            if line.startswith("-DOCSTART-") and example["tokens"] != []:
                print(idx, example)
                yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            elif line == "\n" or (line.startswith("-DOCSTART-") and example["tokens"] == []):
              continue
            else:
                row_cols = line.split(" ")
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "/content/new_train.txt"}, features = features)   

The following error happened:
[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <genexpr>(.0)
    285     for key in unique_values(itertools.chain(*dicts)):  # set merge all keys
    286         # Will raise KeyError if the dict don't have the same keys
--> 287         yield key, tuple(d[key] for d in dicts)
    288 
    

TypeError: tuple indices must be integers or slices, not str

What does this mean and what should I modify?

Comments:
Add notebook / other resource links to the task-specific data loading guides
State: closed
Labels: description: New feature or request
name: enhancement

Does it make sense to include links to notebooks / scripts that show how to use a dataset for training / fine-tuning a model? 

For example, here in [https://huggingface.co/docs/datasets/image_classification] we could include a mention of https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb.

Applies to https://huggingface.co/docs/datasets/object_detection as well.

Cc: @osanseviero @nateraw 

Comments:
Yea this would be great! We would need an object detection tutorial notebook too if it doesn't already exist there. 

There is one: https://huggingface.co/docs/datasets/object_detection.

I will start the work.
Add a guide for semantic segmentation
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

Currently, we have these guides for object detection and image classification:

* https://huggingface.co/docs/datasets/object_detection
* https://huggingface.co/docs/datasets/image_classification

I am proposing adding a similar guide for semantic segmentation. 

I am happy to contribute a PR for it. 

Cc: @osanseviero @nateraw 

Comments:
Sure this sounds great! Would this be pure torchvision, albumentations, or something else?

I am considering `torchvision` and `albumentations`. Also [works with TensorFlow](https://github.com/deep-diver/segformer-tf-transformers/blob/main/notebooks/TFSegFormer_Finetune.ipynb). 

I am assigning the issue to myself then.
An example or recommendations for creating large image datasets?
State: open

I know that Apache Beam and `datasets` have [some connector utilities](https://huggingface.co/docs/datasets/beam). But it's a little unclear what we mean by "But if you want to run your own Beam pipeline with Dataflow, here is how:". What does that pipeline do? 

As a user, I was wondering if we have this support for creating large image datasets. If so, we should mention that [here](https://huggingface.co/docs/datasets/image_dataset). 

Cc @lhoestq 

Comments:
The beam utilities allow to prepare a dataset as parquet in your cloud storage. From my perspective this CLI is not super easy to use, but we've been working on a new python API to prepare a dataset in your cloud storage:
```python
from datasets import load_dataset_builder

builder = load_dataset_builder("c4", "en")
builder.download_and_prepapre("s3://my-bucket/c4", file_format="parquet")
```

And to use Beam you can do:
```python
beam_runner = ...  # one of "SparkRunner", "DataFlowRunner", "DirectRunner", etc.
beam_options = ...

builder.download_and_prepapre(
    "s3://my-bucket/c4",
    file_format="parquet",
    beam_runner=beam_runner,
    beam_options=beam_options
)
```

Though Beam can be used ONLY if there is a dataset script based on the `BeamBasedBuilder` right now - it doesn't work on an arbitrary dataset (see [wikipedia.py](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py) for example).

Thanks! 

Would be nice to have something similar for creating large image datasets.
Loading an external NER dataset
State: closed

I need to use huggingface datasets to load a custom dataset similar to conll2003 but with more entities and each the files contain only two columns: word and ner tag.

I tried this code snnipet that I found here as an answer to a similar issue:

from datasets import Dataset

INPUT_COLUMNS = "ID Text NER".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

train = Dataset.from_generator(read_conll, gen_kwargs={"file": "some_path"})   

But the following error happened:

ValueError: Please pass `features` or at least one example when writing data

Comments:
Preserve None in list type cast in PyArrow 10
State: closed

The `ListArray` type in PyArrow 10.0.0 supports the `mask` parameter, which allows us to preserve Nones in nested lists in `cast` instead of replacing them with empty lists.

Fix https://github.com/huggingface/datasets/issues/3676

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Raise ffmpeg warnings only once
State: closed

Our warnings looks nice now.

`librosa` warning that was raised at each decoding: 
```
/usr/local/lib/python3.7/dist-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
```
is suppressed with `filterwarnings("ignore")` in a context manager. That means the first warning is also ignored (setting `filterwarnings("once")` didn't work!), so I added info that audioread is used for decoding to our message. Hope it's enough.


Tests failed at first because they used to check if the warning was raised at (each) decoding in `librosa` case but now we throw only one warning (at first decoding). I removed this check for warnings, do you think it's fine? 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Inconsistency behavior between handling local file protocol and other FS protocols
State: open

### Describe the bug

These lines us used during load_from_disk:
```
if is_remote_filesystem(fs):
    dest_dataset_dict_path = extract_path_from_uri(dataset_dict_path)
else:
    fs = fsspec.filesystem("file")
    dest_dataset_dict_path = dataset_dict_path
```
If a local FS is given, then it will the URL as the path name. If a remote Fs is given, then it will use the path of the URL. This is an inconsistent behavior when handling a file: when using remote FS, you must write a URL, but for local FS, even if you passed LocalFileSystem as `fs` you still can't use a `file://` URL. It will be recognized as a directory named `file:`.

### Steps to reproduce the bug

```
import fsspec.core
url = "hdfs:///somewhere/MNIST"
# url = "file:///somewhere/MNIST"
fs, path = fsspec.core.url_to_fs(url)

fs.ls(path) # this will always work
load_from_disk(path, fs) # only works for local FS
load_from_disk(url, fs) # only works for remote FS

```

### Expected behavior

one of `url` or `path` should always work

I think we extract path from given URL by using `fsspec.core.url_to_fs` instead of using `is_remote_filesystem` and `extract_path_from_uri` will fix this, since:
```
fsspec.core.url_to_fs("/somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("file:///somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("hdfs:///somewhere/MNIST") ->  HDFS, '/somewhere/MNIST'
```
and
```
fsspec.core.url_to_fs("file:///somewhere/MNIST") == fsspec.core.url_to_fs("/somewhere/MNIST")
```
In theory, this wouldn't break anything, since giving local path and remote uri still works. It will only affect local URI (make it works too)

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-5.4.205.1**HIDDEN**
- Python version: 3.7.10
- PyArrow version: 8.0.0
- Pandas version: 1.2.4


Comments:
[Caching] Deterministic hashing of torch tensors
State: closed
Labels: description: New feature or request
name: enhancement

Currently this fails
```python
import torch
from datasets.fingerprint import Hasher

t = torch.tensor([1.])

def func(x):
    return t + x

hash1 = Hasher.hash(func)
t = torch.tensor([1.])
hash2 = Hasher.hash(func)
assert hash1 == hash2
```

Also as noticed in https://discuss.huggingface.co/t/dataset-cant-cache-models-outputs/24945, using a model in a `map` function doesn't work well with caching. Indeed the `bert-base-uncased` model has a different hash every time you reload it. Supporting torch tensors may also help in this case.

This can be fixed by registering a custom pickling functions for torch tensors - as we did for other objects such as CodeType, FunctionType and Regex in `py_utils.py`

Comments: