load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields
Opened by rbrugaro (open)

### Describe the bug

When loading some jsonl from redpajama-data-1T github source [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) fails due to one row of the file containing an extra field called **symlink_target: string>**. 
When deleting that line the loading is successful. 

We also tried loading this file with the discrepancy using this function and it is successful
```python
os.environ["RED_PAJAMA_DATA_DIR"] ="/path_to_local_copy_of_RedPajama-Data-1T"
ds = load_dataset('togethercomputer/RedPajama-Data-1T', 'github',cache_dir="/path_to_folder_with_jsonl",streaming=True)['train']
```

### Steps to reproduce the bug

Steps to reproduce the behavior:
1. Load one jsonl from the redpajama-data-1T
```bash 
wget https://data.together.xyz/redpajama-data-1T/v1.0.0/github/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```

2.Load dataset will give error:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

_TypeError: Couldn't cast array of type
Struct
<content_hash: string, 
timestamp: string, 
source: string, 
line_count: int64, 
max_line_length: int64,
avg_line_length: double, 
alnum_prop: double, 
repo_name: string, 
id: string, 
size: string, 
binary: bool, 
copies: string, 
ref: string, 
path: string, 
mode: string, 
license: string, 
language: list<item: struct<name: string, bytes: string>>, **symlink_target: string>**
to
{'content_hash': Value(dtype='string', id=None), 
'timestamp': Value(dtype='string', id=None), 
'source': Value(dtype='string', id=None), 
'line_count': Value(dtype='int64', id=None), 
'max_line_length': Value(dtype='int64', id=None), 
'avg_line_length': Value(dtype='float64', id=None), 
'alnum_prop': Value(dtype='float64', id=None), 
'repo_name': Value(dtype='string', id=None), 
'id': Value(dtype='string', id=None), 
'size': Value(dtype='string', id=None), 
'binary': Value(dtype='bool', id=None), 
'copies': Value(dtype='string', id=None), 
'ref': Value(dtype='string', id=None), 
'path': Value(dtype='string', id=None), 
'mode': Value(dtype='string', id=None), 
'license': Value(dtype='string', id=None), 
'language': [{'name': Value(dtype='string', id=None), 'bytes': Value(dtype='string', id=None)}]}_

3. To remove the line causing the problem that includes the **symlink_target: string>** do:
```bash 
sed -i '112252d' filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```
4. Rerun the loading function now is succesful:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

### Expected behavior

Have a clean dataset without discrepancies on the jsonl fields or have the load_dataset('json',...) method not error out. 


### Environment info

- `datasets` version: 2.14.1
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

Comments:
Hi ! Feel free to open a discussion at https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T/discussions to ask the file to be fixed (or directly open a PR with the fixed file)

`datasets` expects all the examples to have the same fields

@lhoestq I think the problem is caused by the fact that hugging face datasets writes a copy of data to the local cache using pyarrow. And the data scheme is inferred from the first few data blocks as can be seen [here](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L570). Maybe setting `streaming=True` can workaround this problem. Would you agree with my statement? 

> @lhoestq I think the problem is caused by the fact that hugging face datasets writes a copy of data to the local cache using pyarrow. And the data scheme is inferred from the first few data blocks as can be seen [here](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L570).

Correct. Therefore any example that doesn't follow the inferred schema will make the code fail.

> Maybe setting streaming=True can workaround this problem. Would you agree with my statement?

You'll meet the same problem but later - when streaming and arriving at the problematic example

@lhoestq I just run below test with streaming=True and is not failing at the problematic example
```python
ds = load_dataset('json', data_files='/path_to_local_RedPajamaData/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl', streaming=True)
count = 0
for i in ds['train']:
    count += 1
    print(count)
```

and completes the 262241 samples successfully. It does error our when streaming is not used