SurAyush/github_issues huggingface dataset:
Issue: BLEURT: Match key names to correspond with filename
State: closed
Author: jaehlee

In order to properly locate downloaded ckpt files key name needs to match filename. Correcting change introduced in #3235 

Comments:
Thanks for the suggestion! I think the current checked-in `CHECKPOINT_URLS` is already not working. I believe anyone who tried using the new ckpts (`BLEURT-20-X`) can't unless this fix is in. The zip file from bleurt side unzips to directory name matching the filename (capitalized for new ones). For example without current changes I'd get the following error

```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-5-f6832fe20f84> in <module>()
      1 predictions = ["hello there", "general kenobi"]
      2 references = ["hello there", "general kenobi"]
----> 3 bleurt = datasets.load_metric("bleurt", "bleurt-20")
      4 results = bleurt.compute(predictions=predictions, references=references)

4 frames
/usr/local/lib/python3.7/dist-packages/bleurt/checkpoint.py in read_bleurt_config(path)
     84   """Reads and checks config file from a BLEURT checkpoint."""
     85   assert tf.io.gfile.exists(path), \
---> 86       "Could not find BLEURT checkpoint {}".format(path)
     87   config_path = os.path.join(path, CONFIG_FILE)
     88   assert tf.io.gfile.exists(config_path), \

AssertionError: Could not find BLEURT checkpoint /root/.cache/huggingface/metrics/bleurt/bleurt-20/downloads/extracted/e34c60f1a05394ecda54e253a10413ca7b5d59f9a23f3cc73258c6b78ffa2f50/bleurt-20
```
inspecting specified path I see that directory name is `BLEURT-20` instead of `bleurt-20`. 
Other solution similar to your suggestion is meddle with `dl_manager.download_and_extract` to unzip to paths with lowering all the paths but I imagine this will affect other parts of the library. 

Indeed, good catch ! Your solution that fixes `CHECKPOINT_URLS ` is simple and works well, thanks :)

Furthermore to avoid breaking changes though we could also keep the support for the lowercase one:
```python
        if self.config_name.lower() in CHECKPOINT_URLS:
            checkpoint_name = self.config_name.lower()
        elif self.config_name.upper() in CHECKPOINT_URLS:
            checkpoint_name = self.config_name.upper()
        else:
            raise KeyError(
                f"{self.config_name} model not found. You should supply the name of a model checkpoint for bleurt in {CHECKPOINT_URLS.keys()}"
            )
```
and then we can use `checkpoint_name` instead of `self.config_name` to download and instantiate the model:
```python
        model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[checkpoint_name])
        self.scorer = score.BleurtScorer(os.path.join(model_path, checkpoint_name))
```

Please let me know if that sounds reasonable to you !

Thanks for the suggestion! I believe your suggestion should work to make keys case insensitive. Changes are committed to the PR now.
Issue: iter_archive for zip files 
State: closed
Author: Mehdi2402

* In this PR, I added the option to iterate through zipfiles for `download_manager.py` only. 
* Next PR will be the same applied to `streaming_download_manager.py`.
* Related issue #3272.
## Comments : 
* There is no `.isreg()` equivalent in zipfile library to check if file is Regular so I used `.is_dir()` instead to skip directories.
* For now I got `streaming_download_manager.py`  working for local zip files, but not for urls. I get the following error when I test it on an archive in google drive, so still working on it.  `BlockSizeError: Got more bytes so far (>2112) than requested (22)`


## Tasks : 
- [x] download_manager.py
- [ ] streaming_download_manager.py

Comments:
And also don't always try streaming with Google Drive - it can have issues because of how Google Drive works (with quotas, restrictions, etc.) and it can indeed cause `BlockSizeError`.

Feel free to host your test data elsewhere, such as in a dataset repository on https://huggingface.co (see [here](https://huggingface.co/docs/datasets/upload_dataset.html#upload-your-files) for a tutorial on how to upload files)
Issue: Better error message when download fails
State: closed
Author: lhoestq

From our discussions in https://github.com/huggingface/datasets/issues/3269 and https://github.com/huggingface/datasets/issues/3282 it would be nice to have better messages if a download fails.

In particular the error now shows:
- the error from the HEAD request if there's one
- otherwise the response code of the HEAD request

I also added an error to tell users to pass `use_auth_token` when the Hugging Face Hub returns 401 (Unauthorized).

While paying around with this I also fixed a minor issue with the `force_download` parameter that was not always taken into account

Comments:
Issue: Fix ASSET dataset data URLs
State: closed
Author: tianjianjiang

Change the branch name "master" to "main" in the data URLs, since facebookresearch has changed that.

Comments:
> Hi @tianjianjiang, thanks for the fix.
> The links should also be updated in the `dataset_infos.json` file.
> The failing tests are due to the missing tag in the header of the `README.md` file:

Hi @albertvillanova, thank you for the info! My apologies for the messy PR.
Issue: Mirror the canonical datasets to the Hugging Face Hub
State: closed
Author: severo

- [ ] create a repo on https://hf.co/datasets for every canonical dataset
- [ ] on every commit related to a dataset, update the hf.co repo

See https://github.com/huggingface/moon-landing/pull/1562

@SBrandeis: I let you edit this description if needed to precise the intent.

Comments:
I created a GitHub project to keep track of what needs to be done:
https://github.com/huggingface/datasets/projects/3

I also store my code in a (private for now) repository at https://github.com/huggingface/mirror_canonical_datasets_on_hub

I understand that the datasets are mirrored on the Hub now, right? Might I close @lhoestq @SBrandeis?
Issue: Fix JSON ClassLabel casting for integers
State: closed
Author: lhoestq

Loading a JSON dataset with ClassLabel feature types currently fails if the JSON data already has integers. Indeed currently it tries to convert the strings to integers without even checking if the data are not integers already.

For example this currently fails:
```python
from datasets import load_dataset, Features, ClassLabel

path = "data.json"
f = Features({"a": ClassLabel(names=["neg", "pos"])})
d = load_dataset("json", data_files=path, features=f)
```
data.json
```json
{"a": 0}
{"a": 1}
```

I fixed that by adding a line that checks the type of the JSON data before trying to convert them

cc @albertvillanova let me know if it sounds good to you

Comments:
Issue: Typing of Dataset.__getitem__ could be improved.
State: closed
Author: Dref360

## Describe the bug

The newly added typing for Dataset.__getitem__ is Union[Dict, List]. This makes tools like mypy a bit awkward to use as we need to check the type manually. We could use type overloading to make this easier. [Documentation](https://docs.python.org/3/library/typing.html#typing.overload)

## Steps to reproduce the bug
Let's have a file `test.py`

```python
from typing import List, Dict, Any

from datasets import Dataset

ds = Dataset.from_dict({
    'a': [1,2,3],
    'b': ["1", "2", "3"]
})

one_colum: List[str] = ds['a']
some_index: Dict[Any, Any] = ds[1]
```

## Expected results

Running `mypy test.py` should not give any error.


## Actual results

```
test.py:10: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "List[str]")
test.py:11: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "Dict[Any, Any]")
Found 2 errors in 1 file (checked 1 source file)
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.1


Comments:
Hi ! Thanks for the suggestion, I didn't know about this decorator.

If you are interesting in contributing, feel free to open a pull request to add the overload methods for each typing combination :) To assign you to this issue, you can comment `#self-assign` in this thread.

`Dataset.__getitem__` is defined right here: https://github.com/huggingface/datasets/blob/e6f1352fe19679de897f3d962e616936a17094f5/src/datasets/arrow_dataset.py#L1840

#self-assign
Issue: Add support for multiple dynamic dimensions and to_pandas conversion for dynamic arrays
State: closed
Author: mariosasko

Add support for multiple dynamic dimensions (e.g. `(None, None, 3)` for arbitrary sized images) and `to_pandas()` conversion for dynamic arrays.

TODOs:
* [ ] Cleaner code
* [ ] Formatting issues (if NumPy doesn't allow broadcasting even though dtype is np.object)
* [ ] Fix some issues with zero-dim tensors 
* [ ] Tests


Comments:
Issue: Integrate Polars library
State: closed
Author: albertvillanova

Check potential integration of the Polars library: https://github.com/pola-rs/polars
- Benchmark: https://h2oai.github.io/db-benchmark/

CC: @thomwolf @lewtun 


Comments:
If possible, a neat API could be something like `Dataset.to_polars()`, as well as `Dataset.set_format("polars")`

Note they use a "custom" implementation of Arrow: [Arrow2](https://github.com/jorgecarleitao/arrow2).

Polars has grown rapidly in popularity over the last year - could you consider integrating the Polars functionality again?

I don't think the "custom" implementation should be a barrier, it still conforms to the Arrow specification 

Is there some direction regarding this from the HF team @lewtun ? Can conversion from polars to HF dataset be implemented with limited/zero copy? So, something like ``Dataset.from_polars()`` and ``Dataset.to_polars()`` like you mentioned. Happy to contribute if I can get some pointers on how this may be implemented.

Hi, is there any updates? Thanks!

> Hi, is there any updates? Thanks!

The feature has been there for a bit 😊 You can call `dataset.to_polars()` (on a `Dataset`, not a `DatasetDict`). The issue can be closed, I guess! @lhoestq 

Looks great and thanks!

Thank you.
Issue: Fix error message and add extension fallback
State: closed
Author: mariosasko

Fix the error message raised if `infered_module_name` is `None` in `CommunityDatasetModuleFactoryWithoutScript.get_module` and make `infer_module_for_data_files` more robust. 

In the linked issue, `infer_module_for_data_files` returns `None` because `json` is the second most common extension due to the suffix ordering. Now,  we go from the most common to the least common extension and try to map it or return `None`.

Fix #3331

Comments:
Issue: AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
State: closed
Author: luozhouyang

## Describe the bug
I add a new question answering dataset to huggingface datasets manually. Here is the link: [luozhouyang/question-answering-datasets](https://huggingface.co/datasets/luozhouyang/question-answering-datasets)

But when I load the dataset, an error raised: 

```bash
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
```

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("luozhouyang/question-answering-datasets", data_files=["dureader_robust.train.json"])
```

## Expected results
Load dataset successfully without any error.

## Actual results
```bash
Traceback (most recent call last):
  File "/mnt/home/zhouyang.lzy/github/naivenlp/naivenlp/tests/question_answering_tests/dataset_test.py", line 89, in test_load_dataset_with_hf
    data_files=["dureader_robust.train.json"],
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1616, in load_dataset
    **config_kwargs,
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1443, in load_dataset_builder
    path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1157, in dataset_module_factory
    raise e1 from None
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1144, in dataset_module_factory
    download_mode=download_mode,
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 798, in get_module
    raise FileNotFoundError(f"No data files or dataset script found in {self.path}")
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: linux
- Python version: 3.6.13
- PyArrow version: 6.0.1


Comments:
Hi,

the fix was merged and will be available in the next release of `datasets`.
In the meantime, you can use it by installing `datasets` directly from master as follows:
```
pip install git+https://github.com/huggingface/datasets.git
```
Issue: Map function: Type error on iter #999
State: closed
Author: josephkready666

## Describe the bug
Using the map function, it throws a type error on iter #999

Here is the code I am calling:
```
dataset = datasets.load_dataset('squad')
dataset['validation'].map(text_numbers_to_int, input_columns=['context'], fn_kwargs={'column': 'context'})
``` 
text_numbers_to_int returns the input text with numbers replaced in the format {'context': text}

It happens at 
`
File "C:\Users\lonek\anaconda3\envs\ai\Lib\site-packages\datasets\arrow_writer.py", line 289, in <listcomp>
    [row[0][col] for row in self.current_examples], type=col_type, try_type=col_try_type, col=col
`

The issue is that the list comprehension expects self.current_examples to be type tuple(dict, str), but for some reason 26 out of 1000 of the sefl.current_examples are type tuple(str, str)

Here is an example of what self.current_examples should be
({'context': 'Super Bowl 50 was an...merals 50.'}, '')

Here is an example of what self.current_examples are when it throws the error:
('The Panthers used th... Marriott.', '')


Comments:
Hi, thanks for reporting.

It would be really helpful if you could provide the actual code of the `text_numbers_to_int` function so we can reproduce the error.

```
def text_numbers_to_int(text, column=""):
    """
    Convert text numbers to int.

    :param text: text numbers
    :return: int
    """
    try:
        numbers = find_numbers(text)
        if not numbers:
            return text
        result = ""
        i, j = 0, 0
        while i < len(text):
            if j < len(numbers) and i == numbers[j][1]:
                n = int(numbers[j][0]) if numbers[j][0] % 1 == 0 else float(numbers[j][0])
                result += str(n)
                i = numbers[j][2] #end
                j += 1
            else:
                result += text[i]
                i += 1
        if column:
            return{column: result}
        else:
            return {column: result}
    except Exception as e:
        print(e)
        return {column: result}
```

Maybe this is because of the `return text` line ? I think it should return a dictionary rather than a string

Yes that was it, good catch! Thanks
Issue: Quick fix error formatting
State: closed
Author: NouamaneTazi

While working on a dataset, I got the error
```
TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types {[type(x) for x in processed_inputs.values()]}. When using `batched=True`, make sure provided `function` returns a `dict` of types like `{allowed_batch_return_types}`.
```

This PR should fix the formatting of this error

Comments:
Issue: "Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"
State: closed
Author: eliasws

## Describe the bug
Passing a correctly shaped Numpy-Array to get_nearest_examples leads to the Exception

"Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"

Probably the reason for this is a wrongly converted assertion.

1.15.1:

`assert len(query.shape) == 1 or (len(query.shape) == 2 and query.shape[0] == 1)`

1.16.1:

```
 if len(query.shape) != 1 or (len(query.shape) == 2 and query.shape[0] != 1):
            raise ValueError("Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)")
```

## Steps to reproduce the bug

follow the steps described here: https://huggingface.co/course/chapter5/6?fw=tf

```python
 question_embedding.shape # (1, 768)

 scores, samples = embeddings_dataset.get_nearest_examples(
    "embeddings", question_embedding, k=5 # Error
)

# "Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"
```

## Expected results
Should work without exception

## Actual results
Throws exception

## Environment info
- `datasets` version: 1.15.1
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.12
- PyArrow version: 6.0.


Comments:
#3323
Issue: Can't import `datasets` in python 3.10
State: closed
Author: lhoestq

When importing `datasets` I'm getting this error in python 3.10:
```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_dataset.py", line 47, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_reader.py", line 33, in <module>
    from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 334, in <module>
    class InMemoryTable(TableBlock):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 361, in InMemoryTable
    def from_pandas(cls, *args, **kwargs):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 24, in wrapper
    out = wraps(arrow_table_method)(method)
  File "/Users/quentinlhoest/.pyenv/versions/3.10.0/lib/python3.10/functools.py", line 61, in update_wrapper
    wrapper.__wrapped__ = wrapped
AttributeError: readonly attribute
```

This makes the conda build fail.
I'm opening a PR to fix this and do a patch release 1.16.1

Comments:
Issue: Update URL of tatoeba subset of xtreme
State: closed
Author: mariosasko

Updates the URL of the tatoeba subset of xtreme. Additionally, replaces `os.path.join` with `xjoin` to correctly join the URL segments on Windows.

Fix #3320 

Comments:

To be more precise: `os.path.join` is replaced on-the-fly by `xjoin` anyway with patching, to extend it to remote files


Oh actually just ignore what I said: they were used to concatenate URLs, which is not recommended. Let me fix that again by appending using `+`
Issue: Can't get tatoeba.rus dataset
State: closed
Author: mmg10

## Describe the bug
It gives an error.

> FileNotFoundError: Couldn't find file at https://github.com/facebookresearch/LASER/raw/master/data/tatoeba/v1/tatoeba.rus-eng.rus

## Steps to reproduce the bug
```python
data=load_dataset("xtreme","tatoeba.rus", split="validation")
```

## Solution
The library tries to access the **master** branch. In the github repo of facebookresearch, it is in the **main** branch.

Comments:
Issue: Add push_to_hub docs
State: closed
Author: lhoestq

Since #3098 it's now possible to upload a dataset on the Hub directly from python using the `push_to_hub` method.
I just added a section in the "Upload a dataset to the Hub" tutorial.

I kept the section quite simple but let me know if it sounds good to you @LysandreJik @stevhliu :)

Comments:
Looks good to me! :)

Maybe we can mention that users can also set the `private` argument if they want to keep their dataset private? It would lead nicely into the next section on Privacy.

Thanks for your comments, I fixed the capitalization for consistency and added an passage to mention the `private` parameter and to have a nice transition to the Privacy section :)

I also added the login instruction that was missing before the user can actually upload a dataset.
Issue: Add desc parameter to Dataset filter method
State: closed
Author: vblagoje

**Is your feature request related to a problem? Please describe.**
As I was filtering very large datasets I noticed the filter method doesn't have the desc parameter which is available in the map method. Why don't we add a desc parameter to the filter method both for consistency and it's nice to give some feedback to users during long operations on Datasets?

**Describe the solution you'd like**
Add desc parameter to Dataset filter method

**Describe alternatives you've considered**
N/A

**Additional context**
N/A


Comments:
Hi,

`Dataset.map` allows more generic transforms compared to `Dataset.filter`, which purpose is very specific (to filter examples based on a condition). That's why I don't think we need the `desc` parameter there for consistency. #3196 has added descriptions to the  `Dataset` methods that call `.map` internally, but not for the `filter` method, so we should do that.

Do you have a description in mind? Maybe `"Filtering the dataset"` or `"Filtering the indices"`? If yes, feel free to open a PR.

I'm personally ok with adding the `desc` parameter actually. Let's say you have different filters, it can be nice to differentiate between the different filters when they're running no ?

@mariosasko the use case is filtering of a dataset prior to tokenization and subsequent training. As the dataset is huge it's just a matter of giving a user (model trainer) some feedback on what's going on. Otherwise, feedback is given for all steps in training preparation and not for filtering and the filtering in my use case lasts about 4-5 minutes. And yes, if there are more filtering stages, as @lhoestq pointed out, it would be nice to give some feedback. I thought desc is there already and got confused when I got the script error. 

I don't have a strong opinion on that, so having `desc` as a parameter is also OK.
Issue: Add RedCaps dataset
State: closed
Author: albertvillanova

## Adding a Dataset
- **Name:** RedCaps
- **Description:** Web-curated image-text data created by the people, for the people
- **Paper:** https://arxiv.org/abs/2111.11431
- **Data:** https://redcaps.xyz/
- **Motivation:** Multimodal image-text dataset:  12M+ Image-text pairs 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Proposed by @patil-suraj 

Comments:
Issue: Adding arg to pass process rank to `map`
State: closed
Author: TevenLeScao

This PR adds a `with_rank` argument to `map` that gives the user the possibility to pass the rank of each process to their function. This is mostly designed for multi-GPU map (each process can be sent to a different device thanks to the rank). I've also added tests. I'm putting the PR up so you can check the code, I'll add a multi-GPU example to the doc (+ write a bit in the doc for the new arg)

Comments:
Some commits seem to be there twice (made the mistake of rebasing because I wasn't sure whether the doc had changed), is this an issue @lhoestq ?