jamesLeeeeeee/datasets-github-issues huggingface dataset:
Google BLEU Metric Card
Author: emibaylor | Pull request: yes | State: closed

Add metric card for Google BLEU (GLEU) metric

One thing I noticed while writing this up is that, while this metric was made specifically to be better than BLEU at the sentence level instead of the corpus level, the current implementation only allows the calculation of the corpus-level statistic. I think changing this would be a good thing to put on the to do list for the future.
reddit_tifu dataset: Checksums didn't match for dataset source files
Author: XingxingZhang | Pull request: no | State: closed

## Describe the bug
When loading the reddit_tifu dataset, it throws the exception "Checksums didn't match for dataset source files"


## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset

print(datasets.__version__)

# load_dataset('billsum')

load_dataset('reddit_tifu', 'short')
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: mac os
- Python version: Python 3.7.6
- PyArrow version: 3.0.0
billsum dataset: Checksums didn't match for dataset source files:
Author: XingxingZhang | Pull request: no | State: closed

## Describe the bug

When loading the `billsum` dataset, it throws the exception "Checksums didn't match for dataset source files"

```
  File "virtualenv_projects/codex/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1g89WgFHMRbr4QrvA0ngh26PY081Nv3lx']
```

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset

print(datasets.__version__)

load_dataset('billsum')
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0

- Platform: mac os
- Python version: Python 3.7.6
- PyArrow version: 3.0.0
Source links broken
Author: qqaatw | Pull request: no | State: closed

## Describe the bug

The source links of v2.0.0 docs are broken:

For exmaple, clicking the source button of this [class](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.ClassLabel) will direct users to `https://github.com/huggingface/datasets/blob/v2.0.0/src/datasets/features/features.py#L747`

here, the `v2.0.0` should be `2.0.0`.


## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results

Redirecting to this link: `https://github.com/huggingface/datasets/blob/2.0.0/src/datasets/features/features.py#L747`

## Actual results
Described above.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
Missing languages in lvwerra/github-code dataset
Author: Eytan-S | Pull request: no | State: closed

Hi,

I'm working with the github-code dataset. First of all, thank you for creating this amazing dataset!
I've noticed that two languages are missing from the dataset: TypeScript and Scala. 
Looks like they're also omitted from the query you used to get the original code.
Are there any plans to add them in the future?

Thanks!
Fix Wikipedia version and re-add tests
Author: albertvillanova | Pull request: yes | State: closed

To keep backward compatibility when loading using "wikipedia" dataset ID (https://huggingface.co/datasets/wikipedia), we have created the pre-processed data for the same languages we were offering before, but with updated date "20220301":
- de
- en
- fr
- frr
- it
- simple

These pre-processed data can be accessed, e.g.:
```python
ds = load_dataset("wikipedia", "20220301.frr", split="train")
```

The next step will be to offer the pre-processed data for many other languages, but when loading using "wikimedia/wikipedia": https://huggingface.co/datasets/wikimedia/wikipedia
Add align_labels_with_mapping docs
Author: stevhliu | Pull request: yes | State: closed

This PR documents the `align_labels_with_mapping` function to ensure predicted labels are aligned with the dataset, or to assign a different mapping of labels to ids (requested by @mariosasko 🎉 ).

For this specific code sample, the current dataset has a `mixed` label that the original [dataset](https://huggingface.co/datasets/poem_sentiment#data-fields) didn't. Is there a way to remove this label so it is completely aligned with the original dataset mappings? Otherwise, I'll just leave it as it is.
Load a local dataset twice
Author: caush | Pull request: no | State: closed

## Describe the bug
Load a local "dataset" composed of two csv files twice.

## Steps to reproduce the bug
Put the two joined files in a repository named "Data".
Then in python:

import datasets as ds
ds.load_dataset('Data', data_files = {'file1.csv', 'file2.csv'})

## Expected results
Should give something like (because files have only one data row):
Title, clicks
Truc et astuce, 123
Machin, 12

## Actual results
Gives 
Title, clicks
Truc et astuce, 123
Machin, 12
Truc et astuce, 123
Machin, 12

## Environment info
[file1.csv](https://github.com/huggingface/datasets/files/8256322/file1.csv)
[file2.csv](https://github.com/huggingface/datasets/files/8256323/file2.csv)
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-65-generic-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.1
Frugal score deprecations
Author: ierezell | Pull request: no | State: closed

## Describe the bug
The frugal score returns a really verbose output with warnings that can be easily changed. 

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets.load import load_metric

frugal = load_metric("frugalscore")
frugal.compute(predictions=["Do you like spinachis"], references=["Do you like spinach"])
```

## Expected results
A clear and concise description of the expected results.
```
{'scores': [0.9946]}
```

## Actual results
Specify the actual results or traceback.
```
PyTorch: setting up devices
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 864.09ba/s]
Using amp half precision backend
The following columns in the test set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence2, sentence1. If sentence2, sentence1 are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.
***** Running Prediction *****
  Num examples = 1
  Batch size = 64
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 4644.85it/s]
 {'scores': [0.9946]}
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
Fix main_classes docs index
Author: lhoestq | Pull request: yes | State: closed

Currently the `main_classes` documentation has a wrong index. I believe this comes from issues in the examples of the Translation feature types

![image](https://user-images.githubusercontent.com/42851186/158426345-2ee1ceef-ddf3-4a6f-a93e-d1a8f38a44f5.png)
Document cases for github datasets
Author: lhoestq | Pull request: yes | State: closed

In general we recommend adding the new dataset under a username or organization in the Hugging Face Hub at [hf.co/datasets](hf.co/datasets), but users can still add a dataset on github in some cases.

I added a paragraph in the documentation to explain in which cases it can make more sense to open a PR on github:

- when you need the dataset to be reviewed
- when you need long-term maintenance from the HF team
- when there’s no clear org name / namespace that you can put the dataset under
Add methods to IterableDatasetDict
Author: lhoestq | Pull request: yes | State: closed

Following the new methods added in #3826 and https://github.com/huggingface/datasets/pull/3862 I added several methods to IterableDatasetDict:
    - map
    - filter
    - shuffle
    - with_format
    - cast
    - cast_column
    - remove_columns
    - rename_column
    - rename_columns
'datasets.features' is not a package
Author: Arij-Aladel | Pull request: no | State: closed

@albertvillanova 
python 3.9
os: ubuntu 20.04

In conda environment

torch installed by


```/env/bin/pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html```
datasets package is installed by 
```
/env/bin/pip  install datasets==1.8.0

```
During runing the code I have this error 


```
[6]<stderr>:  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 875, in find_class
[6]<stderr>:    return super().find_class(mod_name, name)
[6]<stderr>:ModuleNotFoundError: No module named 'datasets.features.features'; 'datasets.features' is not a package

```
precisely this error appears when 

torch.load('data_file.pt')
```

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 607, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 882, in _load
    result = unpickler.load()
  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 875, in find_class
    return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'datasets.features.features'; 'datasets.features' is not a package
```

Why I am getting this error?
AttributeError: 'DatasetDict' object has no attribute 'features'
Author: jswapnil10 | Pull request: no | State: closed

## Describe the bug
Receiving the error when trying to check for Dataset features

## Steps to reproduce the bug
from datasets import Dataset
dataset = Dataset.from_pandas(df[['id', 'words', 'bboxes', 'ner_tags', 'image_path']])

dataset.features

## Expected results
A clear and concise description of the expected results.

## Actual results
Getting the following errror

AttributeError: 'DatasetDict' object has no attribute 'features'

## Environment info
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.18.4
- Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- PyArrow version: 6.0.1
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files
Author: willowdong | Pull request: no | State: closed

## Describe the bug
Can't load the dataset

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```
from datasets import load_dataset

dataset = load_dataset('multi_news')
dataset_2=load_dataset("reddit_tifu", "long")


## Actual results
 raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF']

## Environment info
- `datasets` version: 1.18.4
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.0
- PyArrow version: 6.0.1
Metric card template
Author: emibaylor | Pull request: yes | State: closed

Adding a metric card template, based on ideas and edits from @sashavor and I, as well as from comments from @lhoestq and others (thank you!).

All feedback is welcome, but am especially curious about feedback in terms of:
- things that should be included but aren't
- things that are included but should be changed or removed
- the instructions I included, and whether they should be added to, clarified, or deleted altogether
Use templates for doc-builidng jobs
Author: sgugger | Pull request: yes | State: closed

This PR updates the jobs for all doc-building related things by using the templates introduced on `doc-builder`. By putting those once there, we make sure every repo gets the latest fixes on the doc-building github actions :-)

Note: all libraries must share the same docker image for those doc-building jobs. For now, all the one used (`huggingface/transformers-doc-builder`) contains all extra steps of the datasets install for docbuling (mainly libsndfile) but if in the future some additional steps are necessary on top of `pip install -e .[dev]`, this docker image will need to be updated with the extra deps.
add draft of registering function for pandas
Author: lvwerra | Pull request: yes | State: closed

This PR adds a register function for `pandas`. It allows to directly push `DataFrame` objects to the hub and in return loading datasets on the hub from `DataFrame`. The motivation for this integration is to enable the vast number of `pandas` users to be able to easily push `DataFrames` to the hub. 

Here is an example:

```python
import pandas as pd
from datasets import register_pandas

register_pandas()

# push to hub
df = pd.DataFrame.from_dict({"test": [1,2,3]})
df.push_to_hub("my_test")

# load from hub
df_retrieved = pd.DataFrame.load_from_hub("lvwerra/my_test")
```

It follows a similar philosophy as the `tqdm` [integration](https://github.com/tqdm/tqdm#pandas-integration). Also see [this issue](https://github.com/pandas-dev/pandas/issues/46000) on the `pandas` repository.

This is just a rough draft of what such integration could look like but I would like appreciate some feedback on this: is this something you would like to add the library and is this the way to go? cc @lhoestq @albertvillanova @julien-c
Fix text loader to split only on universal newlines
Author: albertvillanova | Pull request: yes | State: closed

Currently, `text` loader breaks on a superset of universal newlines, which also contains Unicode line boundaries. See: https://docs.python.org/3/library/stdtypes.html#str.splitlines

However, the expected behavior is to get the lines splitted only on universal newlines: "\n", "\r\n" and "\r".

See: oscar-corpus/corpus#18

Fix #3729.
NonMatchingChecksumError on Spider dataset 
Author: kolk | Pull request: no | State: closed

## Describe the bug
Failure to generate dataset ```spider``` because of checksums error for dataset source files.

## Steps to reproduce the bug
```
from datasets import load_dataset
spider = load_dataset("spider")
```


## Expected results
Checksums should match for files from url ['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Actual results
```
>>> load_dataset("spider")
load_dataset("spider")
Downloading and preparing dataset spider/spider (download: 95.12 MiB, generated: 5.17 MiB, post-processed: Unknown size, total: 100.29 MiB) to /home/user/.cache/huggingface/datasets/spider/spider/1.0.0/79778ebea87c59b19411f1eb3eda317e9dd5f7788a556d837ef25c3ae6e5e8b7...
Traceback (most recent call last):
  File "/home/user/py3_env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-d4cb54197348>", line 1, in <module>
    load_dataset("spider")
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']
```


## Environment info
datasets version: 1.18.3
Platform: Ubuntu 20 LTS
Python version: 3.8.10
PyArrow version: 6.0.1
CONLL2003 Dataset not available
Author: omarespejel | Pull request: no | State: closed

## Describe the bug
[CONLL2003](https://huggingface.co/datasets/conll2003) Dataset can no longer reach 'https://data.deepai.org/conll2003.zip' 
![image](https://user-images.githubusercontent.com/4755430/158084483-ff83631c-5154-4823-892d-577bf1166db0.png)

## Steps to reproduce the bug
```python
from datasets import load_dataset
datasets = load_dataset("conll2003")
```

## Expected results
Download the conll2003 dataset.

## Actual results
Error: `ConnectionError: Couldn't reach https://data.deepai.org/conll2003.zip (error 502)`
Add Biwi Kinect Head Pose dataset.
Author: dnaveenr | Pull request: yes | State: closed

This PR adds the Biwi Kinect Head Pose dataset.

Dataset Request : Add Biwi Kinect Head Pose Database [#3822](https://github.com/huggingface/datasets/issues/3822)

The Biwi Kinect Head Pose Database is acquired with the Microsoft Kinect sensor, a structured IR light device.It contains 15K images of 20 people with 6 females and 14 males where 4 people were recorded twice.

For each frame, there is :
- a depth image, (.bin file)
- a corresponding rgb image (both 640x480 pixels),
- annotation ( present inside a .txt file)
 The ground truth is the 3D location of the head and its rotation.

The dataset structure is as follows : 
```
- 01.obj
- 01
     - frame_00003_depth.bin
     - frame_00003_pose.txt
     - frame_00003_rgb.png
     .
     .
     .
- 02.obj
 - 02
     - frame_00003_depth.bin
     - frame_00003_pose.txt
     - frame_00003_rgb.png
     .
     .
     .
```
Preview of frame_00003_pose.txt : 
```
0.988397 0.0731349 0.133128 
-0.0441539 0.976945 -0.208876 
-0.145334 0.200575 0.968838 

126.665 40.4515 876.198 
```
I have used the following dataset features : 
```
features=datasets.Features(
    {
        "person_id": datasets.Value("string"),
        "frame_number": datasets.Value("string"),
        "depth_image": datasets.Value("string"),
        "rgb_image": datasets.Image(),
        "3D_head_center": datasets.Array2D(shape=(3, 3), dtype="float"),
        "3D_head_rotation": datasets.Value("float"),
    }
```
I am giving the path to the depth_image here.

I need some inputs for the following : 

1. For each person, the dataset has the following additional information : 
```
For each sequence, the corresponding .obj file represents a head template deformed to match the neutral face of that specific person. [*.obj file]
In each folder, two .cal files contain calibration information for the depth and the color camera, e.g., the intrinsic camera matrix of the depth camera and the global rotation and translation to the rgb camera.
```
Wanted to know how we can represent these features ?

2. For _generate_examples , do I parse the directories and fetch the required information ? This would mean reading the .txt file to obtain the "3D_head_center" and "3D_head_rotation" details. We could precompute the features information  and have a metadata file and use the metadata file to yield information in _generate_examples ? Wanted your thoughts for the best approach for this ?
Can't import datasets: partially initialized module 'fsspec' has no attribute 'utils'
Author: arunasank | Pull request: no | State: closed

## Describe the bug
Unable to import datasets

## Steps to reproduce the bug
```python
from datasets import Dataset, DatasetDict
```

## Expected results
The import works without errors

## Actual results
```
AttributeError                            Traceback (most recent call last)
<ipython-input-37-c8cfcbe62127> in <module>
     11 # from tqdm import tqdm
     12 # import torch
---> 13 from datasets import Dataset
     14 # from transformers import Trainer, TrainingArguments, AutoModel, AutoTokenizer, AutoModelForMaskedLM, DataCollatorForLanguageModeling
     15 # from sentence_transformers import SentenceTransformer

~/.local/lib/python3.8/site-packages/datasets/__init__.py in <module>
     31     )
     32 
---> 33 from .arrow_dataset import Dataset, concatenate_datasets
     34 from .arrow_reader import ArrowReader, ReadInstruction
     35 from .arrow_writer import ArrowWriter

~/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py in <module>
     46 )
     47 
---> 48 import fsspec
     49 import numpy as np
     50 import pandas as pd

~/.local/lib/python3.8/site-packages/fsspec/__init__.py in <module>
     10 from . import _version, caching
     11 from .callbacks import Callback
---> 12 from .core import get_fs_token_paths, open, open_files, open_local
     13 from .exceptions import FSTimeoutError
     14 from .mapping import FSMap, get_mapper

~/.local/lib/python3.8/site-packages/fsspec/core.py in <module>
     16     caches,
     17 )
---> 18 from .compression import compr
     19 from .registry import filesystem, get_filesystem_class
     20 from .utils import (

~/.local/lib/python3.8/site-packages/fsspec/compression.py in <module>
     68 
     69 
---> 70 register_compression("zip", unzip, "zip")
     71 register_compression("bz2", BZ2File, "bz2")
     72 

~/.local/lib/python3.8/site-packages/fsspec/compression.py in register_compression(name, callback, extensions, force)
     44 
     45     for ext in extensions:
---> 46         if ext in fsspec.utils.compressions and not force:
     47             raise ValueError(
     48                 "Duplicate compression file extension: %s (%s)" % (ext, name)

AttributeError: partially initialized module 'fsspec' has no attribute 'utils' (most likely due to a circular import)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: Jupyter notebook
- Python version: 3.8.10
- PyArrow version: 7.0.0
Dataset viewer issue for IndicParaphrase- the preview doesn't show
Author: ratishsp | Pull request: no | State: closed

## Dataset viewer issue for '*IndicParaphrase*'

**Link:** *[IndicParaphrase](https://huggingface.co/datasets/ai4bharat/IndicParaphrase/viewer/hi/validation)*

*The preview of the dataset doesn't come up. 
The error on the console is:
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/home/hf/datasets-preview-backend/hi_IndicParaphrase_v1.0.tar'*

Am I the one who added this dataset ? Yes