noamaanMulla-03/datasets-issues huggingface dataset:
Infer the features if missing

**Is your feature request related to a problem? Please describe.**

Some datasets, in particular community datasets, have no info file, thus no features.

**Describe the solution you'd like**

If a dataset has no features, the first loaded data (5-10 rows) could be used to infer the type.

Related: `datasets` would provide a way to load the data, and get the rows AND the features as the result.

**Describe alternatives you've considered**

The HF hub could also provide some UI to help the dataset maintainers to explicit the types of their rows, or automatically infer them as an initial proposal.

Labels: name: enhancement

name: dataset-viewer

Done by @lhoestq here: https://github.com/huggingface/datasets/pull/4500 (https://github.com/huggingface/datasets/pull/4500/files#diff-02930e1d966f4b41f9ddf15d961f16f5466d9bee583138657018c7329f71aa43R1255 in particular)
Dataset Viewer issue for subjqa

### Link

https://huggingface.co/datasets/subjqa

### Description

Getting the following error for this dataset:

```
Status code:   500
Exception:     Status500Error
Message:       2 or more items returned, instead of 1
```

Not sure what's causing it though 🤔 

### Owner

Yes

Labels: name: dataset-viewer

It's a bug in the viewer, thanks for reporting it. We're hoping to update to a new version in the next few days which should fix it.

Fixed 

https://huggingface.co/datasets/subjqa

<img width="1040" alt="Capture d’écran 2022-09-08 à 10 23 26" src="https://user-images.githubusercontent.com/1676121/189073210-2a57ff88-8bb1-44bd-851e-0e75473cea3f.png">
Trec Dataset has incorrect labels

## Describe the bug
Both coarse and fine labels seem to be out of line.


## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = "trec"
raw_datasets = load_dataset(dataset)
df = pd.DataFrame(raw_datasets["test"])
df.head()
```

## Expected results
text (string) | coarse_label (class label) | fine_label (class label)
-- | -- | --
How far is it from Denver to Aspen ? | 5  	(NUM) | 40  	(NUM:dist)
What county is Modesto , California in ? | 4  	(LOC) | 32  	(LOC:city)
Who was Galileo ? | 3  	(HUM) | 31  	(HUM:desc)
What is an atom ? | 2  	(DESC) | 24  	(DESC:def)
When did Hawaii become a state ? | 5  	(NUM) | 39  	(NUM:date)

## Actual results
  index | label-coarse  |label-fine                  |                    text
-- |-- | -- | --
0    |         4     |     40     | How far is it from Denver to Aspen ?
1      |       5        |  21  | What county is Modesto , California in ?
2        |     3        |  12        |                 Who was Galileo ?
3         |    0       |    7         |                What is an atom ?
4         |    4       |    8       |   When did Hawaii become a state ?

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-1086-azure-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Labels: name: bug

Thanks for reporting, @wmpauli. 

Indeed we recently fixed this issue:
- #4801 

The fix will be accessible after our next library release. In the meantime, you can have it by passing `revision="main"` to `load_dataset`.
Raise a more precise error when the URL is unreachable in streaming mode

See for example:

- https://github.com/huggingface/datasets/issues/3191
- https://github.com/huggingface/datasets/issues/3186

It would help provide clearer information on the Hub and help the dataset maintainer solve the issue by themselves quicker. Currently:

- https://huggingface.co/datasets/compguesswhat

  <img width="1029" alt="Capture d’écran 2022-09-08 à 15 51 37" src="https://user-images.githubusercontent.com/1676121/189139946-6deffb91-f21b-4281-8825-a98026c69740.png">

- https://huggingface.co/datasets/nli_tr

  <img width="1032" alt="Capture d’écran 2022-09-08 à 15 51 44" src="https://user-images.githubusercontent.com/1676121/189139963-d26490ed-ad23-48ea-9cfc-1ab9c4d08d0c.png">


cc @albertvillanova 

Labels: name: enhancement
CI test of TensorFlow is failing

## Describe the bug
The following CI test fails: https://github.com/huggingface/datasets/runs/8246722693?check_suite_focus=true
```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - AssertionError:
```

Details:
```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw0] linux -- Python 3.7.13 /opt/hostedtoolcache/Python/3.7.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        def gen_random_output():
            model = layers.Dense(2)
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
        with temp_seed(42, set_tensorflow=True):
            out1 = gen_random_output()
        with temp_seed(42, set_tensorflow=True):
            out2 = gen_random_output()
        out3 = gen_random_output()
    
>       np.testing.assert_equal(out1, out2)
E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatched elements: 2 / 2 (100%)
E       Max absolute difference: 0.84619296
E       Max relative difference: 16.083529
E        x: array([[-0.793581,  0.333286]], dtype=float32)
E        y: array([[0.052612, 0.539708]], dtype=float32)

tests/test_py_utils.py:149: AssertionError
```




Labels: name: bug
Create dataset from list of dicts

I often find myself with data from a variety of sources, and a list of dicts is very common among these.
However, converting this to a Dataset is a little awkward, requiring either

```Dataset.from_pandas(pd.DataFrame(formatted_training_data))```
Which can error out on some more exotic values as 2-d arrays for reasons that are not entirely clear
> ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column labels with type object')

Alternatively:
```Dataset.from_dict({k: [s[k] for s in formatted_training_data] for k in  formatted_training_data[0].keys()})```
Which works, but is a little ugly.

**Describe the solution you'd like**
Either `.from_dict` accepting a list of dicts, or a `.from_records` function accepting such.

I am happy to PR this, just wanted to check you are happy to accept this I haven't missed something obvious, and which of the solutions would be prefered.


Labels: name: enhancement

Hi @sanderland, thanks for your enhancement proposal.

I agree with you that this would be useful.

Please note that under the hood, we use PyArrow tables as backend:
- The implementation  of `Dataset.from_dict` uses the PyArrow `Table.from_pydict`

Therefore, I would suggest:
- Implementing `Dataset.from_list` using the PyArrow `Table.from_pylist`

What do you think?
Let's see if other people have other suggestions...

Thanks for the quick and positive reply @albertvillanova! 
`from_list` seems sensible. Have opened a PR so we can discuss details there.

Resolved via #4890.
More rigorous shape inference in to_tf_dataset

`tf.data` needs to know the shape of tensors emitted from a `tf.data.Dataset`. Although `None` dimensions are possible, overusing them can cause problems - Keras uses the dataset tensor spec at compile-time, and so saying that a dimension is `None` when it's actually constant can hurt performance, or even cause training to fail for dimensions that are needed to determine the shape of weight tensors!

The compromise I used here was to sample several batches from the underlying dataset and apply the `collate_fn` to them, and then to see which dimensions were "empirically variable". There's an obvious problem here, though - if you sample 10 batches and they all have the same shape on a certain dimension, there's still a small chance that the 11th batch will be different, and Keras will throw an error  if a dataset tries to emit a tensor whose shape doesn't match the spec.

I encountered this bug in practice once or twice for datasets that were mostly-but-not-totally constant on a given dimension, and I still don't have a perfect solution, but this PR should greatly reduce the risk. It samples many more batches, and also samples very small batches (size 2) - this increases the variability, making it more likely that a few outlier samples will be detected.

Ideally, of course, we'd determine the full output shape analytically, but that's surprisingly tricky when the `collate_fn` can be any arbitrary Python code!


_The documentation is not available anymore as the PR was closed or merged._
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.4.0/datasets/jsonl/jsonl.py

Hi,
When I use load_dataset from local jsonl files, below error happens, and I type the link into the browser prompting me `404: Not Found`. I download the other `.py` files using the same method and it works. It seems that the server is missing the appropriate file, or it is a problem with the code version.

```
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b08342004c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))

```



Labels: name: bug

I have solved this problem... The extension of the file should be `.json` not `.jsonl`
🐛 [Metrics] ROUGE is non-deterministic

If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?


Hi, can you give a full self-contained example to reproduce this behavior?

> Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)

> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
> 
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
> 
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
> 
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
> 
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_aggregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.

Maybe we can set all the random seeds of numpy/torch etc. while running `metric.compute` ?

We should probably indeed!

Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```

> 

Minor nit: Note that "aggregator" is misspelled in this command. Should be `use_aggregator=False`.  

Thanks, I fixed the code snippet
Docs for creating a loading script for image datasets

This PR is a first draft of creating a loading script for image datasets. Feel free to let me know if there are any specificities I'm missing for this. 🙂 

To do:
- [x] Document how to create different configurations.

Labels: name: documentation

_The documentation is not available anymore as the PR was closed or merged._

IMO it would make more sense to add a "Create image dataset" page with two main sections - a no-code approach with `imagefolder` + metadata (preferred way), and with a loading script (advanced). It should be clear when to choose which. If we leave this as-is, the user who jumps straight to the Vision section could be under the impression that writing a loading script is the preferred way to share a vision dataset due to how this subsection starts:
```
Write a dataset loading script to share a dataset.
```
  
Also, I think a note explaining how to make a dataset gated/disable the viewer to hide the data would be beneficial (it's pretty common to require submitting a form to access a CV dataset).

Great suggestion @mariosasko! I added your suggestions, let me know what you think. For gated dataset access, I just added a tip referring users to the relevant docs since it's more of a Hub feature than `datasets` feature.

Thanks, looks much better now :). I would also move the sections explaining how to create an `imagefolder` for the specific task from the [loading page](https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/image_load.mdx) to this one. IMO it makes more sense to have the basic info (imagefolder structure + `load_dataset` call) there + a link to this page for info on how to create an image folder dataset.

Good idea! Moved everything about `imagefolder` + metadata to the create an image dataset section since the `load_dataset` call is the same for different computer vision tasks. 

Thanks for all the feedbacks! 🥰

What do you think about creating how to share an `ImageFolder` dataset in a separate PR? I think we should create a new section under `Vision` for how to share an image dataset.

I love it thanks ! I think moving forward we can use CSV instead of JSON Lines in the docs ;)
Improve features resolution in streaming

`IterableDataset._resolve_features` was returning the features sorted alphabetically by column name, which is not consistent with non-streaming. I changed this and used the order of columns from the data themselves. It was causing some inconsistencies in the dataset viewer as well.

I also fixed `interleave_datasets` that was not filling missing columns with None, because it was not using the columns from `IterableDataset._resolve_features`

cc @severo 


_The documentation is not available anymore as the PR was closed or merged._

Just took your comment into account @mariosasko , let me know if it's good for you now :)
Fix data URL and metadata of vivos dataset

After contacting the authors of the VIVOS dataset to report that their data server is down, we have received a reply from Hieu-Thi Luong that their data is now hosted on Zenodo: https://doi.org/10.5281/zenodo.7068130

This PR updates their data URL and some metadata (homepage, citation and license).

Fix #4936.


_The documentation is not available anymore as the PR was closed or merged._
Dataset viewer issue for '*compguesswhat*'

## Dataset viewer issue for '*compguesswhat*'

**Link:** https://huggingface.co/datasets/compguesswhat

File not found

Am I the one who added this dataset ? No


Labels: name: streaming

```python
>>> import datasets
>>> dataset = datasets.load_dataset('compguesswhat', name='compguesswhat-original',split='train', streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/compguesswhat/4d08b9e0a8d1cf036c9626c93be4a759fdd9fcce050ea503ea14b075e830c799/compguesswhat.py", line 251, in _generate_examples
    with gzip.open(filepath) as in_file:
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/gzip.py", line 173, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'zip://compguesswhat-original/0.2.0/compguesswhat.train.jsonl.gz::https://www.dropbox.com/s/l0nc13udml6vs0w/compguesswhat-original.zip?dl=1'
```

It's an issue with the streaming mode. Note that normal mode is used by the dataset viewer when streaming is failing, but only for the smallest datasets. This dataset is above the limit, hence the error.

Same case as https://github.com/huggingface/datasets/issues/3186#issuecomment-1096549774.

cc @huggingface/datasets 

There is an issue with the URLs of their data files: https://www.dropbox.com/s/l0nc13udml6vs0w/compguesswhat-original.zip?dl=1
> Dropbox Error: That didn't work for some reason

Error reported to their repo:
- https://github.com/CompGuessWhat/compguesswhat.github.io/issues/1

Closed by:
- #4968
Dataset viewer for nli_tr

## Dataset viewer issue for '*nli_tr*'

**Link:** https://huggingface.co/datasets/nli_tr

Hello,

Thank you for the new dataset preview feature that will help the users to view the datasets online.

We just noticed that the dataset viewer widget in the `nli_tr` dataset shows the error below. The error must be due to a temporary problem that may have blocked access to the dataset through the dataset viewer. But the dataset is currently accessible through the link in the error message.   May we kindly ask if it would be possible to rerun the job so that it can access the dataset for the dataset viewer function?

Thank you.
Emrah


------------------------------------------
Server Error
Status code:   404
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: 'zip://snli_tr_1.0_train.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip
------------------------------------------

Am I the one who added this dataset ? Yes


Labels: name: streaming

It's an issue with the streaming mode:

```python
>>> import datasets
>>> dataset = datasets.load_dataset('nli_tr', name='snli_tr',split='test', streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/nli_tr/c2ddd0c0a70caddac6a81c2dae5ca7939f00060d517d08f1983927818dba6521/nli_tr.py", line 155, in _generate_examples
    with codecs.open(filepath, encoding="utf-8") as f:
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: 'zip://snli_tr_1.0_test.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip'
```

Note that normal mode is used by the dataset viewer when streaming is failing, but only for the smallest datasets. `nli_tr` is above the limit, hence the error.

cc @huggingface/datasets 

Apparently there is an issue with the data source URLs: Server Not Found
- https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip

We are contacting the authors to ask them: 
@e-budur you are one of the authors: are you aware of the issue with the URLs of your data ?

Reported to their repo:
- https://github.com/boun-tabi/NLI-TR/issues/9

The server issue was temporary and is now resolved.

Once we have implemented support for streaming, the viewer works: https://huggingface.co/datasets/nli_tr
Update setup.py

exclude broken version of fsspec. See the [related issue](https://github.com/huggingface/datasets/issues/4961)


Before addressing this PR, we should be sure about the issue. See my comment in:
- https://github.com/huggingface/datasets/issues/4961#issuecomment-1243376247

Once we know 2022.8.2 works, I'm closing this PR, as the corresponding issue.
Missing MBPP splits

(@albertvillanova)
The [MBPP dataset on the Hub](https://huggingface.co/datasets/mbpp) has only a test split for both its "full" and its "sanitized" subset, while the [paper](https://arxiv.org/abs/2108.07732) states in subsection 2.1 regarding the full split:
> In the experiments described later in the paper, we hold out 10 problems for **few-shot prompting**, another 500 as our **test** dataset (which is used to evaluate both few-shot inference and fine-tuned models), 374 problems for **fine-tuning**, and the rest for **validation**.

If the dataset on the Hub should reproduce most closely what the original authors use, I guess this four-way split should be reflected. 

The paper doesn't explicitly state the task_id ranges of the splits, but the [GitHub readme](https://github.com/google-research/google-research/tree/master/mbpp) referenced in the paper specifies exact task_id ranges, although it misstates the total number of samples:
> We specify a train and test split to use for evaluation. Specifically:
> 
> * Task IDs 11-510 are used for evaluation.
> * Task IDs 1-10 and 511-1000 are used for training and/or prompting. We typically used 1-10 for few-shot prompting, although you can feel free to use any of the training examples.

I.e. the few-shot, train and validation splits are combined into one split, with a soft suggestion of using the first ten for few-shot prompting. It is not explicitly stated whether the 374 fine-tuning samples mentioned in the paper have task_id 511 to 784 or 601 to 974 or are randomly sampled from task_id 511 to 974.

Regarding the "sanitized" split the paper states the following:
> For evaluations involving the edited dataset, we perform comparisons with 100 problems that appear in both the original and edited dataset, using the same held out 10 problems for few-shot prompting and 374 problems for fine-tuning. 

The statement doesn't appear to be very precise, as among the 10 few-shot problems, those with task_id 1, 5 and 10 are not even part of the sanitized variant, and many from the task_id range from 511 to 974 are missing (e.g. task_id 511 to 553). I suppose the idea the task_id ranges for each split remain the same, even if some of the task_ids are not present. That would result in 7 few-shot, 257 test, 141 train and 22 validation examples in the sanitized split.

Labels: name: bug

Thanks for reporting this as well, @stadlerb.

I suggest waiting for the answer of the data owners... 

@albertvillanova The first author of the paper responded to the upstream issue:
> Task IDs 11-510 are the 500 test problems. We use 90 problems (511-600) for validation and then remaining 374 for fine-tuning (601-974). The other problems can be used as desired, either for training or few-shot prompting (although this should be specified).

Thanks for the follow-up, @stadlerb.

Would you be willing to open a Pull Request to address this issue? :wink: 

Opened a [PR](https://github.com/huggingface/datasets/pull/4943) to implement this--lmk if you have any feedback