load_dataset for winoground returning decoding error
Opened by odellus (closed)

## Describe the bug
I am trying to use datasets to load winoground and I'm getting a JSON decoding error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
token = 'hf_XXXXX' # my HF access token
datasets = load_dataset('facebook/winoground', use_auth_token=token)
```

## Expected results
I downloaded images.zip and examples.jsonl manually. I was expecting to have some trouble decoding json so I didn't use jsonlines but instead was able to get a complete set of 400 examples by doing
```python
import json

with open('examples.jsonl', 'r') as f:
    examples = f.read().split('\n')

# Thinking this would error if the JSON is not utf-8 encoded
json_data = [json.loads(x) for x in examples]
print(json_data[-1])
```
and I see
```python
{'caption_0': 'someone is overdoing it',
 'caption_1': 'someone is doing it over',
 'collapsed_tag': 'Relation',
 'id': 399,
 'image_0': 'ex_399_img_0',
 'image_1': 'ex_399_img_1',
 'num_main_preds': 1,
 'secondary_tag': 'Morpheme-Level',
 'tag': 'Scope, Preposition'}

```
so I'm not sure what's going on here honestly. The file `examples.jsonl` doesn't have non-UTF-8 encoded text.

## Actual results
During the split operation after downloading, datasets encounters an error in the JSON ([trace](https://gist.github.com/odellus/e55d390ca203386bf551f38e0c63a46b) abbreviated for brevity).
```
datasets/packaged_modules/json/json.py:144 in Json._generate_tables(self, files)
...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: Linux-5.13.0-39-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0


Comments:
I thought I had fixed it with this after some helpful hints from @severo
```python
import datasets 
token = 'hf_XXXXX'
dataset = datasets.load_dataset(
    'facebook/winoground', 
    name='facebook--winoground', 
    split='train', 
    streaming=True,
    use_auth_token=token,
)
```
but I found out that wasn't the case
```python
[x for x in dataset]
...
ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl')
```

Hi ! This dataset structure (image + labels in a JSON file) is not supported yet, though we're adding support for this in  in #4069 

The following structure will be supported soon:
```
metadata.json
images/
     image0.png
     image1.png
    ...
```
Where `metadata.json` is a JSON Lines file with labels or other metadata, and each line must have a "file_name" field with the name of the image file.

For the moment are only supported:
- JSON files only
- image files only

Since this dataset is a mix of the two, at the moment it fails trying to read the images as JSON.

Therefore to be able to load this dataset we need to wait for the new structure to be supported (very soon ^^), or add a dataset script in the repository that reads both the JSON and the images cc @TristanThrush 


We'll also investigate the issue with the streaming download manager in https://github.com/huggingface/datasets/issues/4139 ;) thanks for reporting

Are there any updates on this?

In the meantime, anyone can always download the images.zip and examples.jsonl files directly from huggingface.co - let me know if anyone has issues with that.

I mirrored the files at https://huggingface.co/datasets/facebook/winoground in a folder on my local machine `winground`
and when I tried
```python
import datasets
ds = datasets.load_from_disk('./winoground')
```
I get the following error
```python
--------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 ds = datasets.load_from_disk('./winoground')

File ~/.local/lib/python3.8/site-packages/datasets/load.py:1759, in load_from_disk(dataset_path, fs, keep_in_memory)
   1757     return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1758 else:
-> 1759     raise FileNotFoundError(
   1760         f"Directory {dataset_path} is neither a dataset directory nor a dataset dict directory."
   1761     )

FileNotFoundError: Directory ./winoground is neither a dataset directory nor a dataset dict directory.
```
so still some work to be done on the backend imo.

Note that `load_from_disk` is the function that reloads an Arrow dataset saved with `my_dataset.save_to_disk`.

Once we do support images with metadata you'll be able to use `load_dataset("facebook/winoground")` directly (or `load_dataset("./winoground")` of you've cloned the winoground repository locally).

Apologies for the delay. I added a custom dataset loading script for winoground. It should work now, with an auth token:

`examples = load_dataset('facebook/winoground', use_auth_token=<your auth token>)`

Let me know if there are any issues

Adding the dataset loading script definitely didn't take as long as I thought it would 😅

killer