SurAyush/github_issues huggingface dataset:
Issue: Representing dictionaries (json) objects as features
State: open
Author: yanaiela

In the process of adding a new dataset to the hub, I stumbled upon the inability to represent dictionaries that contain different key names, unknown in advance (and may differ between samples), original asked in the [forum](https://discuss.huggingface.co/t/representing-nested-dictionary-with-different-keys/16442).

For instance:

```
sample1 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
}}
sample2 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
}}
sample3 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
  "d": {"id": 3, "text": "text4"},
}}
```

the `nps` field cannot be represented as a Feature while maintaining its original structure.
@lhoestq suggested to add JSON as a new feature type, which will solve this problem.


It seems like an alternative solution would be to change the original data format, which isn't an optimal solution in my case. Moreover, JSON is a common structure, that will likely to be useful in future datasets as well.

Comments:
Issue: Hotfix failing CI tests on Windows
State: closed
Author: albertvillanova

This PR makes a hotfix for our CI Windows tests: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414

Fix #4118

I guess this issue is related to this PR:
- huggingface/huggingface_hub#815

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Pretty print dataset info files
State: closed
Author: mariosasko

Adds indentation to the `dataset_infos.json` file when saving for nicer diffs.

(suggested by @julien-c)

This PR also updates the info files of the GH datasets. Note that this change adds more than **10 MB** to the repo size (the total file size before the change: 29.672298 MB, after: 41.666475 MB), so I'm not sure this change is a good idea.

`src/datasets/info.py` is the only relevant file for reviewers.


Comments:
maybe just do it from now on no? (i.e. not for existing `dataset_infos.json` files)

_The documentation is not available anymore as the PR was closed or merged._

> maybe just do it from now on no? (i.e. not for existing dataset_infos.json files)

Yes, or do this only for datasets created with `push_to_hub` to (always) keep the GH datasets small? 


yep sounds good too on my side! 

I reverted the change to avoid the size increase and added the `pretty_print` flag, which pretty-prints the JSON, and that flag is only True for datasets created with `push_to_hub`.
Issue: ImageFolder add option to ignore some folders like '.ipynb_checkpoints'
State: closed
Author: cceyda

**Is your feature request related to a problem? Please describe.**
I sometimes like to peek at the dataset images from jupyterlab. thus '.ipynb_checkpoints' folder appears where my dataset is and (just realized) leads to accidental duplicate image additions. I think this is an easy enough thing to miss especially if the dataset is very large.

**Describe the solution you'd like**
maybe have an option `ignore` or something .gitignore style
`dataset = load_dataset("imagefolder", data_dir="./data/original", ignore="regex?")`

**Describe alternatives you've considered**
Could filter out manually



Comments:
Maybe it would be nice to ignore private dirs like this one (ones starting with `.`) by default. 

CC @mariosasko 

Maybe we can add a `ignore_hidden_files` flag to the builder configs of our packaged loaders (to be consistent across all of them), wdyt @lhoestq @albertvillanova? 

I think they should always ignore them actually ! Not sure if adding a flag would be helpful

@lhoestq But what if the user explicitly requests those files via regex?

`glob.glob` ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's `glob` doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

> @lhoestq But what if the user explicitly requests those files via regex?

Usually hidden files are meant to be ignored. If they are data files, they must be placed outside a hidden directory in the first place right ? I think it's more sensible to explain this than adding a flag.

> glob.glob ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's glob doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

After globbing using `fsspec`, we already ignore files that start with a `.` in `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository`, I guess we can just account for parent directories as well ?

We could open an issue on `fsspec` but I think they won't change this since it's an important breaking change for them.
Issue: Multiprocessing with FileLock fails in python 3.9
State: closed
Author: lhoestq

On python 3.9, this code hangs:
```python
from multiprocessing import Pool
from filelock import FileLock


def run(i):
    print(f"got the lock in multi process [{i}]")


with FileLock("tmp.lock"):
    with Pool(2) as pool:
        pool.map(run, range(2))

```

This is because the subprocesses try to acquire the lock from the main process for some reason. This is not the case in older versions of python.

This can cause many issues in python 3.9. In particular, we use multiprocessing to fetch data files when you load a dataset (as long as there are >16 data files). Therefore `imagefolder` hangs, and I expect any dataset that needs to download >16 files to hang as well.

Let's see if we can fix this and have a CI that runs on 3.9.

cc @mariosasko @julien-c 

Comments:
Closing this one because it must be used this way actually:
```python
def main():
    with FileLock("tmp.lock"):
        with Pool(2) as pool:
            pool.map(run, range(2))

if __name__ == "__main__":
    main()
```
Issue: Support huggingface_hub 0.5
State: closed
Author: lhoestq

Following https://github.com/huggingface/datasets/issues/4105

`huggingface_hub` deprecated some parameters in `HfApi` in 0.5. This PR updates all the calls to HfApi to remove all the deprecations, 
and I set the `hugginface_hub` requirement to `>=0.5.0`


cc @adrinjalali @LysandreJik 

Comments:
Looks like GH actions is not able to resolve `huggingface_hub` 0.5.0, I'm investivating

_The documentation is not available anymore as the PR was closed or merged._

I'm glad to see changes in `huggingface_hub` are simplifying code here.

seems to supersede #4102, feel free to close mine :)

maybe just cherry-pick the docstring fix

I think I've found the issue:
- https://github.com/huggingface/huggingface_hub/pull/790

Good catch, `huggingface_hub` doesn't support python 3.6 anymore indeed, therefore we should keep support for 0.4.0. I'm reverting the requirement version bump for now.

We can update the requirement once we drop support for python 3.6 in `datasets`

@lhoestq, I've opened this PR on `huggingface_hub`: 
- https://github.com/huggingface/huggingface_hub/pull/823

Is there any strong reason why `huggingface_hub` no longer supports Python 3.6? 

I think `datasets` can drop support for 3.6 soon. But for now maybe let's keep support for 0.4.0, python 3.6 users are not affected by https://github.com/huggingface/datasets/issues/4105 anyway.

`huggingface_hub` doesn't not have to support 3.6 again just for the CI IMO

@lhoestq I commented on the PR, that IMO it is not a good practice to drop support for Python 3.6 without a previous deprecation cycle.

Re-added support for older versions. I ended up checking `huggingface_hub` version to use the old, deprecated API for <0.5.0

I find it good practice to have all dependency version related code in a single file so that when you decide to remove support for an old version of a dependency it's easy to find and remove them, hence suggesting `utils/_fixes.py` in https://github.com/huggingface/datasets/issues/4105#issuecomment-1090041204

good idea, thanks !

I used your suggestion @adrinjalali , I just replace the try/except with a check on the version of `huggingface_hub`
Issue: How can I download only the train and test split for full numbers using load_dataset()? 
State: open
Author: Nakkhatra

How can I download only the train and test split for full numbers using load_dataset()? 

I do not need the extra split and it will take 40 mins just to download in Colab. I have very short time in hand. Please help.

Comments:
Hi! Can you please specify the full name of the dataset? IIRC `full_numbers` is one of the configs of the `svhn` dataset, and its generation is slow due to data being stored in binary Matlab files. Even if you specify a specific split, `datasets` downloads all of them, but we plan to fix that soon and only download the requested split.

If you are in a hurry, download the `svhn` script [here](`https://huggingface.co/datasets/svhn/blob/main/svhn.py`), remove [this code](https://huggingface.co/datasets/svhn/blob/main/svhn.py#L155-L162), and run:
```python
from datasets import load_dataset
dset = load_dataset("path/to/your/local/script.py", "full_numbers")
```

And to make loading easier in Colab, you can create a dataset repo on the Hub and upload the script there. Or push the script to Google Drive and mount the drive in Colab.
Issue: Improve RedCaps dataset card
State: closed
Author: mariosasko

This PR modifies the RedCaps card to:
* fix the formatting of the Point of Contact fields on the Hub
* speed up the image fetching logic (aligns it with the [img2dataset](https://github.com/rom1504/img2dataset) tool) and make it more robust (return None if **any** exception is thrown)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I find this preprocessing a bit too specific to add it as a method to `datasets` as it's only useful in the context of CV (and we support multiple modalities). However, I agree it would be great to move this code to another lib to avoid code duplication. Maybe we should create a package with preprocessing functions/transforms for this purpose?
Issue: Proposing WikiSplit metric card
State: closed
Author: sashavor

Pinging @lhoestq to ensure that my distinction between the dataset and the metric are clear :sweat_smile:

Comments:
_The documentation is not available anymore as the PR was closed or merged._

A quick Github tip ;) To avoid running N times the CI, you can push all the changes at once: go to Files Changed tab, and on each suggestion there's a "add to commit batch" and then you can do one commit for all the suggestions you want to approve ;)

Oh thanks for the tip!! Haha I was wondering why it was running a bunch of
times :P

On Tue, Apr 5, 2022 at 11:44 AM Quentin Lhoest ***@***.***>
wrote:

> A quick Github tip ;) To avoid running N times the CI, you can push all
> the changes at once: go to Files Changed tab, and on each suggestion
> there's a "add to commit batch" and then you can do one commit for all the
> suggestions you want to approve ;)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/4098#issuecomment-1088894515>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADMMIIRZYNVFJRWRWW4VJY3VDRNUBANCNFSM5SS7L5HA>
> .
> You are receiving this because you modified the open/close state.Message
> ID: ***@***.***>
>


-- 
Sasha Luccioni, PhD
Postdoctoral Researcher (Mila, Université de Montréal)
Chercheure postdoctorale (Mila, Université de Montréal)
https://www.sashaluccioni.com/
                [image: Image result for universite de montreal logo]
Issue: Helo Mayfrends
State: closed
Author: Budigming

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Issue: elena-soare/crawled-ecommerce: missing dataset
State: closed
Author: seevaratnam

elena-soare/crawled-ecommerce

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Comments:
It's a bug! Thanks for reporting, I'm looking at it.

By the way, the error on our part is due to the huge size of every row (~90MB). The dataset viewer does not support such big dataset rows for the moment.
Anyway, we're working to give a hint about this in the dataset viewer.

Fixed. See https://huggingface.co/datasets/elena-soare/crawled-ecommerce/viewer/elena-soare--crawled-ecommerce/train.

<img width="1552" alt="Capture d’écran 2022-04-12 à 11 23 51" src="https://user-images.githubusercontent.com/1676121/162929722-2e2b80e2-154a-4b61-87bd-e341bd6c46e6.png">

Thanks for reporting!
Issue: Fix dataset `amazon_us_reviews` metadata - 4/4/2022
State: closed
Author: trentonstrong

Fixes #4048 by running `dataset-cli test` to reprocess data and regenerate metadata. Additionally I've updated the README to include up-to-date counts for the subsets.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc: @albertvillanova just FYI
Issue: Create metric card for Frugal Score
State: closed
Author: sashavor

Proposing metric card for Frugal Score.

@albertvillanova or @lhoestq  -- there are certain aspects that I'm not 100% sure on (such as how exactly the distillation between BertScore and FrugalScore is done) -- so if you find that something isn't clear, please let me know!

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Remove unused legacy Beam utils
State: closed
Author: albertvillanova

This PR removes unused legacy custom `WriteToParquet`, once official Apache Beam includes the patch since version 2.22.0: 
- Patch PR: https://github.com/apache/beam/pull/11699
- Issue: https://issues.apache.org/jira/browse/BEAM-10022

In relation with:
- #204

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Fix BeamWriter output Parquet file
State: closed
Author: albertvillanova

Since now, the `BeamWriter` saved a Parquet file with a simplified schema, where each field value was serialized to JSON. That resulted in Parquet files larger than Arrow files.

This PR:
- writes Parquet file preserving original schema and without serialization, thus avoiding serialization overhead and resulting in a smaller output file size.
- fixes `parquet_to_arrow` function

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Dataset viewer issue for McGill-NLP/feedbackQA
State: closed
Author: cslizc

## Dataset viewer issue for '*McGill-NLP/feedbackQA*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/McGill-NLP/feedbackQA)*

*short description of the issue*
The dataset can be loaded correctly with `load_dataset` but the preview doesn't work. Error message:

```
Status code:   400
Exception:     Status400Error
Message:       Not found. Maybe the cache is missing, or maybe the dataset does not exist.
```

Am I the one who added this dataset ? Yes


Comments:
Hi @cslizc, thanks for reporting.

I have just forced the refresh of the corresponding cache and the preview is working now.

thank you so much
Issue: datasets.set_progress_bar_enabled(False) not working in datasets v2
State: closed
Author: virilo

## Describe the bug

datasets.set_progress_bar_enabled(False) not working in datasets v2

## Steps to reproduce the bug
```python
datasets.set_progress_bar_enabled(False)
```

## Expected results
datasets not using any progress bar

## Actual results

AttributeError: module 'datasets' has no attribute 'set_progress_bar_enabled

## Environment info

datasets version 2


Comments:
Now, I can't find any reference to set_progress_bar_enabled in the code.

I think it have been deleted

Hi @virilo,

Please note that since `datasets` version 2.0.0, we have aligned with `transformers` the management of the progress bar (among other things):
- #3897

Now, you should update your code to use `datasets.logging.disable_progress_bar`.

You have more info in our docs: [Logging methods](https://huggingface.co/docs/datasets/package_reference/logging_methods)

One important thing for beginner like me is: from datasets.utils.logging import disable_progress_bar
Do not forget the 'utils' or you will waste a long time like me....
Issue: Errors in `Train with Datasets` Tensorflow code section on Huggingface.co
State: closed
Author: blackhat-coder

## Describe the bug
Hi
### Error 1
Running the Tensforlow code on [Huggingface](https://huggingface.co/docs/datasets/use_dataset) gives a TypeError: __init__() got an unexpected keyword argument 'return_tensors' 
### Error 2
`DataCollatorWithPadding` isn't imported

## Steps to reproduce the bug
```python
import tensorflow as tf
from datasets import load_dataset
from transformers import AutoTokenizer
dataset = load_dataset('glue', 'mrpc', split='train')
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
dataset = dataset.map(lambda e: tokenizer(e['sentence1'], truncation=True, padding='max_length'), batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
train_dataset = dataset["train"].to_tf_dataset(
  columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'],
  shuffle=True,
  batch_size=16,
  collate_fn=data_collator,
)
```
This is the same code on Huggingface.co

## Actual results
TypeError: __init__() got an unexpected keyword argument 'return_tensors'

## Environment info
- `datasets` version: 2.0.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.7
- PyArrow version: 6.0.0
- Pandas version: 1.4.1
> 

Comments:
Hi @blackhat-coder, thanks for reporting.

Please note that the `transformers` library updated their data collators API last year (version 4.10.0):
- huggingface/transformers#13105

now requiring to pass `return_tensors` argument at Data Collator instantiation.

And therefore, we also updated in the `datasets` library documentation all the examples using `transformers` data collators.

If you would like to follow our examples, please update your installed `transformers` version:
```
pip install -U transformers
```
Issue: Close parquet writer properly in `push_to_hub`
State: closed
Author: lhoestq

We don’t call writer.close(), which causes https://github.com/huggingface/datasets/issues/4077. It can happen that we upload the file before the writer is garbage collected and writes the footer.

I fixed this by explicitly closing the parquet writer.

Close https://github.com/huggingface/datasets/issues/4077.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lhoestq / @albertvillanova / @mariosasko 
I am facing the same scenario. Let me explain the situation point. I have a glue ETL job

1--> My files are in parquet format and stored in AWS s3.
2--> I am iterating a  loop for a data set where the same file name can occur with diffrent other data.
3--> I read the parquet and saved it in a pandas data frame.
4--> Done some operation on that data frame
5--> upload the updated data frame into the S3 parquet file.  Below are code snippet what I am using to save the updated 
         data frame into parquet format and load into S3
         `header_name_column_list = dict(data_frame)
          header_list = []
            for col_id, col_type in header_name_column_list.items():
                header_list.append(pyarrow.field(col_id, pyarrow.string()))
            table_schema = pyarrow.schema(header_list)
            table = pyarrow.Table.from_pandas(data_frame, schema=table_schema, preserve_index=False)
            writer = parquet.ParquetWriter(b_buffer, table.schema)
            writer.write_table(table)
            writer.close()
            b_buffer.seek(0)
            .....
            ....
           self.s3_client.upload_fileobj(
                b_buffer,
                self.bucket,
                file_key,
                ExtraArgs=extra_args)`

But when I executed the glue etl job, the first time it works properly and but in the next iteration, when I try to open the same file got that error.


<html>
<body>
<!--StartFragment-->

INFO:Iot-dsip-de-duplication-job:Dataframe uploaded: s3://abc/2022/07/12/file1_ft_20220714122108.3065_12345.parquet INFO:Iot-dsip-de-duplication-job:Sleep for 60 sec
INFO:Iot-dsip-de-duplication-job:start after sleep
.......................
..........................
..........................
ERROR:Iot-dsip-de-duplication-job:Failed to read data from parquet file s3://abc/2022/07/12/file1_ft_20220714122108.3065_12345.parquet, error is : Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.INFO:Iot-dsip-de-duplication-job:Empty dataframe found

<!--EndFragment-->
</body>
</html>

Any clue will be really helpful..I got stuck with this problem.
Issue: Increase max retries for GitHub datasets
State: closed
Author: albertvillanova

As GitHub recurrently raises connectivity issues, this PR increases the number of max retries to request GitHub datasets, as previously done for GitHub metrics:
- #4063

Note that this is a temporary solution, while we decide when and how to load GitHub datasets from the Hub:
- #4059

Fix #2048

Related to:
- #4051 
- #3210
- #2787 
- #2075
- #2036

CC: @lhoestq 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Fix GithubMetricModuleFactory instantiation with None download_config
State: closed
Author: albertvillanova

Recent PR:
- #4063

introduced a potential bug if `GithubMetricModuleFactory` is instantiated with None `download_config`.

This PR add instantiation tests and fix that potential issue.

CC: @lhoestq 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
State: closed
Author: NielsRogge

## Describe the bug

When uploading a relatively large image dataset of > 1GB, reloading doesn't work for me, even though pushing to the hub went just fine.

Basically, I do:

```
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_files="path_to_my_files")

dataset.push_to_hub("dataset_name") # works fine, no errors

reloaded_dataset = load_dataset("dataset_name")
```

and it returns:

```
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

I created a Colab notebook to reproduce my error: https://colab.research.google.com/drive/141LJCcM2XyqprPY83nIQ-Zk3BbxWeahq?usp=sharing



Comments:
Issue: Add ROUGE Metric Card
State: closed
Author: emibaylor

Add ROUGE metric card.

I've left the 'Values from popular papers' section empty for the time being because I don't know the summarization literature very well and am therefore not sure which paper(s) to pull from (note that the original rouge paper does not seem to present specific values, just correlations with human judgements). Any suggestions on which paper(s) to pull from would be helpful! :) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._