SurAyush/github_issues huggingface dataset:
Issue: Integrate Milvus (pymilvus) library
State: open
Author: mariosasko

Milvus is a popular open-source vector database. We should add a new vector index to support this project.

Comments:
Hi @mariosasko，Just search randomly and I found this issue~ I'm the tech lead of Milvus and we are looking forward to integrate milvus together with huggingface datasets.

Any suggestion on how we could start?


Feel free to assign to me and we probably need some guide on it

@mariosasko any updates my man?


Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

> Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

Sure, we take a look and do some research
Issue: Fix bug in `ImageClassifcation` task template
State: closed
Author: mariosasko

Fixes a bug in the `ImageClassification` task template which requires specifying class labels twice in dataset scripts. Additionally, this PR refactors the API around the classification task templates for cleaner `labels` handling.

CC: @lewtun @nateraw

Comments:
The CI failures are unrelated to the changes in this PR.

> The CI failures are unrelated to the changes in this PR.

It seems that some of the failures are due to the tests on the dataset cards (e.g. CIFAR, MNIST, FASHION_MNIST). Perhaps it's worth addressing those in this PR to avoid confusing downstream developers who branch off `master` and suddenly have a failing CI?

@lewtun We only run these tests against the modified datasets on the PR branch, so this will not lead to errors after merging.
Issue: Preserve encoding/decoding with features in `Iterable.map` call
State: closed
Author: mariosasko

As described in https://github.com/huggingface/datasets/issues/3505#issuecomment-1004755657, this PR uses a generator expression to encode/decode examples with `features` (which are set to None in `map`) before applying a map transform.

Fix #3505 

Comments:
Issue: DuplicatedKeysError when loading tweet_qa dataset
State: closed
Author: LeonieWeissweiler

When loading the tweet_qa dataset with `load_dataset('tweet_qa')`, the following error occurs: 

`DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 2a167f9e016ba338e1813fed275a6a1e
Keys should be unique and deterministic in nature
`
Might be related to issues #2433 and #2333

- `datasets` version: 1.17.0
- Python version: 3.8.5


Comments:
Hi, we've just merged the PR with the fix. The fixed version of the dataset can be downloaded as follows:
```python
import datasets
dset = datasets.load_dataset("tweet_qa", revision="master")
```
Issue: set_format("np") no longer works for Image data
State: closed
Author: cgarciae

## Describe the bug
`dataset.set_format("np")` no longer works for image data, previously you could load the MNIST like this:

```python
dataset = load_dataset("mnist")
dataset.set_format("np")
X_train = dataset["train"]["image"][..., None] # <== No longer a numpy array
```
but now it doesn't work, `set_format("np")` seems to have no effect and the dataset just returns a list/array of PIL images instead of numpy arrays as requested.


Comments:
A quick fix for now is doing this:

```python
X_train = np.stack(dataset["train"]["image"])[..., None]

This error also propagates to jax and is even trickier to fix, since `.with_format(type='jax')` will use numpy conversion internally (and fail). For a three line failure:

```python
dataset = datasets.load_dataset("mnist")
dataset.set_format("jax")
X_train = dataset["train"]["image"]
```

Hi! We've recently introduced a new Image feature that yields PIL Images (and caches transforms on them) instead of arrays.

However, this feature requires a custom transform to yield np arrays directly:
```python
ddict = datasets.load_dataset("mnist")

def pil_image_to_array(batch):
     return {"image": [np.array(img) for img in batch["image"]]}  # or jnp.array(img) for Jax

ddict.set_transform(pil_image_to_array, columns="image", output_all_columns=True)
```

[Docs](https://huggingface.co/docs/datasets/master/process.html#format-transform) on `set_transform`.

Also, the approach proposed by @cgarciae is not the best because it loads the entire column in memory.

@albertvillanova @lhoestq WDYT? The Audio and the Image feature currently don't support the TF/Jax/PT Formatters, but for the Numpy Formatter maybe it makes more sense to return np arrays (and not a dict in the case of the Audio feature or a PIL Image object in the case of the Image feature).

Yes I agree it should return arrays and not a PIL image (and possible an array instead of a dict for audio data).
I'm currently finishing some code refactoring of the image and audio and opening a PR today. Maybe we can look into that after the refactoring

This has been fixed in https://github.com/huggingface/datasets/pull/5072, which is included in the latest release of `datasets`.
Issue: Add the KMWP & DKTC dataset.
State: closed
Author: sooftware

Add the KMWP & DKTC dataset.
  
Additional notes:
- Both datasets will be released on January 10 through the GitHub link below.
  - https://github.com/tunib-ai/DKTC
  - https://github.com/tunib-ai/KMWP
- So it doesn't work as a link at the moment, but the code will work soon (after it is released on January 10).

Comments:
Issue: Add more compression types for `to_json`
State: closed
Author: bhavitvyamalik

This PR adds `bz2`, `xz`, and `zip` (WIP) for `to_json`. I also plan to add `infer` like how `pandas` does it

Comments:
@lhoestq, I looked into how to compress with `zipfile` for which few methods exist, let me know which one looks good:
1. create the file in normal `wb` mode and then zip it separately
2. use `ZipFile.write_str` to write file into the archive. For this we'll need to change how we're writing files from `_write` method  

How `pandas` handles it is that they have created a wrapper for standard library class `ZipFile` and allow the returned file-like handle to accept byte strings via `write` method instead of `write_str` (purpose was to change the name of function by creating that wrapper)

1. sounds not ideal since it creates an intermediary file.
I like pandas' approach. Is it possible to implement 2. using the pandas class ? Or maybe we can have something similar ?

Definitely, @lhoestq! I've adapted that from original code and turns out it is faster than `gz` compression. Apart from that I've also added `infer` option to automatically infer compression type from `path_or_buf` given

One small thing, currently I'm assuming that user will provide compression extension in `path_or_buf`. Is it this also possible?
`dataset.to_json("from_dataset.json", compression="zip")`? 
Should I put an `assert` to ensure the file name provided always has a compression extension?

Thanks !

> One small thing, currently I'm assuming that user will provide compression extension in path_or_buf. Is it this also possible?
>dataset.to_json("from_dataset.json", compression="zip")?
>Should I put an assert to ensure the file name provided always has a compression extension?

I think it's fine as it is right now :) No need to check the extension of the filename passed to `path_or_buf`.


> turns out it is faster than gz compression

I think the default compression level of `gzip` is 9 in python, which is very slow. Maybe we can switch to compression level 6 instead which is faster, like the `gzip` command on unix

I found that `fsspec` has something that may interest you: [fsspec.open(..., compression=...)](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open). I don't remember if we've already mentioned it or not

It also has `zip` if I understand correctly ! see https://github.com/fsspec/filesystem_spec/blob/master/fsspec/compression.py#L70

Since `fsspec` is a dependency of `datasets` we can use all this :)

Let me know if you prefer using `fsspec` instead (I haven't tested this yet to write compressed files). IMO it sounds pretty easy to use and it would make the code base simpler

Just tried `fsspec` but I'm not able to write compressed `zip` files :/
`gzip`, `xz`, `bz2` are all working fine and it's really simple (no need for `FileWriteHandler` now!)
Issue: Bug in `openbookqa` dataset
State: closed
Author: lucadiliello

## Describe the bug
Dataset entries contains a typo.

## Steps to reproduce the bug

```python
>>> from datasets import load_dataset
>>> obqa = load_dataset('openbookqa', 'main')
>>> obqa['train'][0]
```

## Expected results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['A', 'B', 'C', 'D']}, 'answerKey': 'D'}
```

## Actual results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting']}, 'answerKey': 'D'}
```

The bug is present in all configs and all splits.

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.4.0-1057-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1


Comments:
Closed by:
- #4259
Issue: Fix sem_eval_2018_task_1 download location
State: closed
Author: maxpel

This changes the download location of sem_eval_2018_task_1 files to include the test set labels as discussed in https://github.com/huggingface/datasets/issues/2745#issuecomment-954588500_ with @lhoestq.

Comments:
Hi ! Thanks for pushing this :)

It seems that you created this PR from an old version of `datasets` that didn't have the sem_eval_2018_task_1.py file.

Can you try merging `master` into your branch ? Or re-create your PR from a branch that comes from a more recent version of `datasets` ?

And sorry for the late response !

Hi! No problem! I made the new branch like you said and opened https://github.com/huggingface/datasets/pull/3643 for it. I will close this one.
Issue: Specify the feature types of a dataset on the Hub without needing a dataset script
State: closed
Author: lhoestq

**Is your feature request related to a problem? Please describe.**
Currently if I upload a CSV with paths to audio files, the column type is string instead of Audio.

**Describe the solution you'd like**
I'd like to be able to specify the types of the column, so that when loading the dataset I directly get the features types I want.

The feature types could read from the `dataset_infos.json` for example.

**Describe alternatives you've considered**
Create a dataset script to specify the features, but that seems complicated for a simple thing.

cc @abidlabs 

Comments:
After looking into this, discovered that this is already supported if the `dataset_infos.json` file is configured correctly! Here is a working example: https://huggingface.co/datasets/abidlabs/test-audio-13

This should be probably be documented, though.
Issue: fix: 🐛 pass token when retrieving the split names
State: closed
Author: severo


Comments:
Currently, it does not work with https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/blob/main/common_voice_7_0.py#L146 (which was the goal), because `dl_manager.download_config.use_auth_token` is ignored, and the authentication is required to be use `huggingface-cli login`.
In my use case (dataset viewer), I'd prefer to use a specific "User Token Access", with only the "read" role (https://huggingface.co/settings/token).

See https://github.com/huggingface/datasets-preview-backend/issues/74#issuecomment-1007316853 for the context

> Simply passing download_config is ok :)

hmm, I prefer only passing use_auth_token. But the question is more: is it correct, in the (convoluted) case if `download_config.use_auth_token` exists and is different from `use_auth_token`? Which one should be used?

If both are passed, `use_auth_token` should have the priority (more specific parameters have the higher priority)
Issue: Ability to split a dataset in multiple files.
State: open
Author: Dref360

Hello,

**Is your feature request related to a problem? Please describe.**

My use case is that I have one writer that adds columns and multiple workers reading the same `Dataset`. Each worker should have access to columns added by the writer when they reload the dataset.

I understand that we shouldn't overwrite an arrow file as this could cause Segfault and so on. Before 1.16, I was able to overwrite the dataset and that would work most of the time with some retries.

**Describe the solution you'd like**

I was thinking that if we could append `Dataset._data_files`, when the workers reload the Dataset, they would get the new columns.


**Describe alternatives you've considered**
I currently need to
1. Save multiple "versions" of the dataset and load the latest.
2. Try working with cache files to get the latest columns.

**Additional context**

I think this would be a great addition to HFDataset as Parquet supports multi-files input out of the box!

I can make a PR myself with some pointers as needed :)

Comments:
Issue: Allow loading community metrics from the hub, just like datasets
State: closed
Author: eladsegal

**Is your feature request related to a problem? Please describe.**
Currently, I can load a metric implemented by me by providing the local path to the file in `load_metric`.
However, there is no option to do it with the metric uploaded to the hub.
This means that if I want to allow other users to use it, they must download it first which makes the usage less smooth.

**Describe the solution you'd like**
Load metrics from the hub just like datasets are loaded.
In order to not break stuff, the convention can be to put the metric file in a "metrics" folder in the hub.



Comments:
Hi ! Thanks for your message :) This is a great idea indeed. We haven't started working on this yet though. For now I guess you can host your metric on the Hub (either with your model or your dataset) and use `hf_hub_download` to download it (docs [here](https://github.com/huggingface/huggingface_hub/blob/main/docs/hub/how-to-downstream.md#cached_download))

This is a great solution in the meantime, thanks!

Here's the code I used, in case it can be of help to someone else:
```python
import os, shutil
from huggingface_hub import hf_hub_download
def download_metric(repo_id, file_path):
    # repo_id: for models "username/model_name", for datasets "datasets/username/model_name"
    local_metric_path = hf_hub_download(repo_id=repo_id, filename=file_path)
    updated_local_metric_path = (os.path.dirname(local_metric_path) + os.path.basename(local_metric_path).replace(".", "_") + ".py")
    shutil.copy(local_metric_path, updated_local_metric_path)
    return updated_local_metric_path

metric = load_metric(download_metric(REPO_ID, FILE_PATH))
```

Solved with https://github.com/huggingface/evaluate 🤗 

Yay!! cc @lvwerra @sashavor @douwekiela 

Please share your feedback @eladsegal =)
Issue: Support 7-zip compressed data files
State: open
Author: albertvillanova

**Is your feature request related to a problem? Please describe.**

We should support 7-zip compressed data files:
- [x] in `extract`:
  - #4672 
- [ ] in `iter_archive`: for streaming mode

both in streaming and non-streaming modes.


Comments:
This should also resolve: https://github.com/huggingface/datasets/issues/3185.
Issue: How to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?
State: open
Author: CindyTing

Hi,
I use torch.utils.data.Dataset to define my own data, but I need to use the 'map' function of datasets.arrow_dataset.Dataset later, so I hope to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset.

Here is an example.
```
from torch.utils.data import Dataset
from datasets.arrow_dataset import Dataset as HFDataset

class ADataset(Dataset):
    def __init__(self, data): 
        super().__init__()
        self.data = data
    def __getitem__(self, index):
        return self.data[index]
    def __len__(self):
        return self.len
class MDataset():
    def __init__(self, tokenizer: AutoTokenizer, data_args, training_args):
        self.train_dataset = ADataset(data_args)
        self.tokenizer = tokenizer
        self.data_args = data_args

        self.train_dataset = self.train_dataset.map(
            self.process_function,
            batched=True,
            remove_columns=column_names,
            load_from_cache_file=True,
            desc="Running tokenizer on train dataset",
             )
    def process_function(self, examples):
        sentences = [" ".join(sample[0][3]) for sample in examples]

        tokenized = self.tokenizer(
            sentences,
            max_length=self.max_seq_len,
            padding=self.padding, 
            truncation=True)
```
But it would raise an ERROR, AttributeError: 'ADataset' object has no attribute 'map'.

so how to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Thanks in advance!






Comments:
Issue: Task search function on hub not working correctly
State: open
Author: patrickvonplaten

When I want to look at all datasets of the category: `speech-processing`  *i.e.* https://huggingface.co/datasets?task_categories=task_categories:speech-processing&sort=downloads , then the following dataset doesn't show up for some reason:

- https://huggingface.co/datasets/speech_commands

even thought it's task tags seem correct: 
https://raw.githubusercontent.com/huggingface/datasets/master/datasets/speech_commands/README.md

Comments:
known issue due to https://github.com/huggingface/datasets/pull/2362 (and [internal](https://github.com/huggingface/moon-landing/issues/946)) , will be solved soon

hmm actually i have no recollection of why I said that

Because it has dots in some YAML keys, it can't be parsed and indexed by the back-end
Issue: Give clearer instructions to add the YAML tags
State: closed
Author: albertvillanova

Fix #3531.

CC: @julien-c @VictorSanh 

Comments:
this is great, maybe just put all of it in one line?

> TODO: Add YAML tags here. Copy-paste the tags obtained with the online tagging app: https://huggingface.co/spaces/huggingface/datasets-tagging
Issue: Give clearer instructions to add the YAML tags
State: closed
Author: albertvillanova

## Describe the bug
As reported by @julien-c, many community datasets contain the line `YAML tags:` at the top of the YAML section in the header of the README file.  See e.g.: https://huggingface.co/datasets/bigscience/P3/commit/a03bea08cf4d58f268b469593069af6aeb15de32

Maybe we should give clearer instruction/hints in the README template.


Comments:
Issue: Update license to bookcorpus dataset card
State: closed
Author: meg-huggingface

Not entirely sure, following the links here, but it seems the relevant license is at https://github.com/soskek/bookcorpus/blob/master/LICENSE

Comments:
The smashwords ToS apply for this dataset, we did the same for https://github.com/huggingface/datasets/pull/3525

_The documentation is not available anymore as the PR was closed or merged._
Issue: Adding license information for Openbookcorpus
State: closed
Author: meg-huggingface

Not entirely sure, following the links here, but it seems the relevant license is at https://github.com/soskek/bookcorpus/blob/master/LICENSE

Comments:
The MIT license seems to be for the crawling code, no ? Then maybe we can also redirect users to the [terms of smashwords.com](https://www.smashwords.com/about/tos) regarding copyrights, in particular the paragraph 10 for end-users. In particular it seems that end users can download and use the content "for their personal enjoyment in any reasonable non-commercial manner in compliance with copyright law" and the smashwords end-users agreement.

It should be the same for https://github.com/huggingface/datasets/pull/3526 as well

May I merge this one ?

_The documentation is not available anymore as the PR was closed or merged._
Issue: wmt19 is broken (zh-en)
State: closed
Author: AjayP13

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wmt19", 'zh-en')
```

## Expected results
The dataset should download.

## Actual results
`ConnectionError: Couldn't reach ftp://cwmt-wmt:cwmt-wmt@datasets.nju.edu.cn/parallel/casia2015.zip`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux
- Python version: 3.8 


Comments:
This issue is not reproducible.
Issue: Audio datacard update - first pass
State: closed
Author: meg-huggingface

Filling out data card "Personal and Sensitive Information" for speech datasets to note PII concerns

Comments:
I'm not sure that we want to change the tags at the top of the cards by hand. Those are used to create the tags in the hub. Although looking at all the tags now, we might want to normalize the current tags again (hyphens or no, ".0" or no). Maybe we could add a binary tag for public domain or not?

> 

That's a good point, I didn't realize these were auto-populated.
At the same time, some of them are wrong -- how/where are they auto-populated?  Seems like we should fix it at that source for the future.
In the mean time, I see that "cc0-1.0" is the desired tag for public domain, so I will change that for now.
Issue: Add PubMed Central Open Access dataset
State: closed
Author: albertvillanova

## Adding a Dataset
- **Name:** PubMed Central Open Access
- **Description:** The PMC Open Access Subset includes more than 3.4 million journal articles and preprints that are made available under license terms that allow reuse.
- **Paper:** *link to the dataset paper if available*
- **Data:** https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
In the framework of BigScience:
- bigscience-workshop/data_tooling#121

I have created this dataset as a community dataset: https://huggingface.co/datasets/albertvillanova/pmc_open_access

However, I was wondering that it may be more appropriate to move it under an org namespace: `pubmed_central` or `pmc`
This way, we could add other datasets I'm also working on: Author Manuscript Dataset, Historical OCR Dataset, LitArch Open Access Subset.

What do you think? @lhoestq @mariosasko 

Why not ! Having them under such namespaces would also help people searching for this kind of datasets.
We can also invite people from pubmed at one point

DONE: https://huggingface.co/datasets/pmc/open_access
Issue: Fix to_tf_dataset references in docs
State: closed
Author: mariosasko

Fix the `to_tf_dataset` references in the docs. The currently failing example of usage will be fixed by #3338.  

Comments:
The code snippet in [this section](https://huggingface.co/docs/datasets/master/use_dataset.html?highlight=to_tf_dataset#tensorflow) is missing an import (`DataCollatorWithPadding`) and doesn't initialize the TF model before the `model.fit` call.