Get an error when i try to concatenate the squad dataset with my own dataset
Opened by CCDXDX (closed)

### Describe the bug

Hello,

I'm new here and I need to concatenate the squad dataset with my own dataset i created. I find the following error when i try to do it: Traceback (most recent call last):

  Cell In[9], line 1
    concatenated_dataset = concatenate_datasets([train_dataset, dataset1])

  File ~\anaconda3\Lib\site-packages\datasets\combine.py:213 in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)

  File ~\anaconda3\Lib\site-packages\datasets\arrow_dataset.py:6002 in _concatenate_map_style_datasets
    _check_if_features_can_be_aligned([dset.features for dset in dsets])

  File ~\anaconda3\Lib\site-packages\datasets\features\features.py:2122 in _check_if_features_can_be_aligned
    raise ValueError(

ValueError: The features can't be aligned because the key answers of features {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)} has unexpected type - Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None) (expected either {'answer_start': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'text': Value(dtype='string', id=None)} or Value("null").

### Steps to reproduce the bug

```python

from huggingface_hub import notebook_login
from datasets import load_dataset



notebook_login("mymailadresse", "mypassword")
squad = load_dataset("squad", split="train[:5000]")
squad = squad.train_test_split(test_size=0.2)
dataset1 = squad["train"]




import json

mybase = [
    {
        "id": "1",
        "context": "She lives in Nantes",
        "question": "Where does she live?",
        "answers": {
            "text": "Nantes",
            "answer_start": [13],
        }
    }
]




# Save the data to a JSON file
json_file_path = r"C:\Users\mypath\thefile.json"
with open(json_file_path, "w", encoding= "utf-8") as json_file:
    json.dump(mybase, json_file, indent=4)




# Load the JSON file as a dataset
custom_dataset = load_dataset("json", data_files=json_file_path)


# Access the train split
train_dataset = custom_dataset["train"]


from datasets import concatenate_datasets


# Concatenate the datasets
concatenated_dataset = concatenate_datasets([train_dataset, dataset1])
```

### Expected behavior

I would expect the two datasets to be concatenated without error. The len(dataset1) is equal to 4000 and the len(train_dataset) is equal to 1 so I would exepect concatenated_dataset to be created and having lenght 4001.

### Environment info

Python 3.11.4 and using windows

Thank you for your help

Comments:
The `answers.text` field in the JSON dataset needs to be a list of strings, not a string.

So, here is the fixed code:
```python
from huggingface_hub import notebook_login
from datasets import load_dataset



notebook_login("mymailadresse", "mypassword")
squad = load_dataset("squad", split="train[:5000]")
squad = squad.train_test_split(test_size=0.2)
dataset1 = squad["train"]




import json

mybase = [
    {
        "id": "1",
        "context": "She lives in Nantes",
        "question": "Where does she live?",
        "answers": {
            "text": ["Nantes"],
            "answer_start": [13],
        }
    }
]




# Save the data to a JSON file
json_file_path = r"data"
with open(json_file_path, "w", encoding= "utf-8") as json_file:
    json.dump(mybase, json_file, indent=4)




# Load the JSON file as a dataset
custom_dataset = load_dataset("json", data_files=json_file_path, features=dataset1.features)


# Access the train split
train_dataset = custom_dataset["train"]


from datasets import concatenate_datasets


# Concatenate the datasets
concatenated_dataset = concatenate_datasets([train_dataset, dataset1])
```

Thank you @mariosasko for your help ! It works !