GitHub issue #6162: load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields
State: open | Created: 2023-08-18T07:19:39Z

### Describe the bug

When loading some jsonl from redpajama-data-1T github source [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) fails due to one row of the file containing an extra field called **symlink_target: string>**. 
When deleting that line the loading is successful. 

We also tried loading this file with the discrepancy using this function and it is successful
```python
os.environ["RED_PAJAMA_DATA_DIR"] ="/path_to_local_copy_of_RedPajama-Data-1T"
ds = load_dataset('togethercomputer/RedPajama-Data-1T', 'github',cache_dir="/path_to_folder_with_jsonl",streaming=True)['train']
```

### Steps to reproduce the bug

Steps to reproduce the behavior:
1. Load one jsonl from the redpajama-data-1T
```bash 
wget https://data.together.xyz/redpajama-data-1T/v1.0.0/github/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```

2.Load dataset will give error:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

_TypeError: Couldn't cast array of type
Struct
<content_hash: string, 
timestamp: string, 
source: string, 
line_count: int64, 
max_line_length: int64,
avg_line_length: double, 
alnum_prop: double, 
repo_name: string, 
id: string, 
size: string, 
binary: bool, 
copies: string, 
ref: string, 
path: string, 
mode: string, 
license: string, 
language: list<item: struct<name: string, bytes: string>>, **symlink_target: string>**
to
{'content_hash': Value(dtype='string', id=None), 
'timestamp': Value(dtype='string', id=None), 
'source': Value(dtype='string', id=None), 
'line_count': Value(dtype='int64', id=None), 
'max_line_length': Value(dtype='int64', id=None), 
'avg_line_length': Value(dtype='float64', id=None), 
'alnum_prop': Value(dtype='float64', id=None), 
'repo_name': Value(dtype='string', id=None), 
'id': Value(dtype='string', id=None), 
'size': Value(dtype='string', id=None), 
'binary': Value(dtype='bool', id=None), 
'copies': Value(dtype='string', id=None), 
'ref': Value(dtype='string', id=None), 
'path': Value(dtype='string', id=None), 
'mode': Value(dtype='string', id=None), 
'license': Value(dtype='string', id=None), 
'language': [{'name': Value(dtype='string', id=None), 'bytes': Value(dtype='string', id=None)}]}_

3. To remove the line causing the problem that includes the **symlink_target: string>** do:
```bash 
sed -i '112252d' filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```
4. Rerun the loading function now is succesful:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

### Expected behavior

Have a clean dataset without discrepancies on the jsonl fields or have the load_dataset('json',...) method not error out. 


### Environment info

- `datasets` version: 2.14.1
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3