Datasets.py function load_dataset does not match squad dataset
State: closed | Author: LeopoldACC (NONE)

### 1 When I try to train lxmert,and follow the code in README that --dataset name:
```shell 
python examples/question-answering/run_qa.py --model_name_or_path unc-nlp/lxmert-base-uncased --dataset_name squad --do_train --do_eval --per_device_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2 --max_seq_length 384 --doc_stride 128 --output_dir /home2/zhenggo1/checkpoint/lxmert_squad
```
the bug is that:
```
Downloading and preparing dataset squad/plain_text (download: 33.51 MiB, generated: 85.75 MiB, post-processed: Unknown size, total: 119.27 MiB) to /home2/zhenggo1/.cache/huggingface/datasets/squad/plain_text/1.0.0/4c81550d83a2ac7c7ce23783bd8ff36642800e6633c1f18417fb58c3ff50cdd7...
Traceback (most recent call last):
  File "examples/question-answering/run_qa.py", line 501, in <module>
    main()
  File "examples/question-answering/run_qa.py", line 217, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/load.py", line 746, in load_dataset
    use_auth_token=use_auth_token,
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/builder.py", line 573, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/builder.py", line 633, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json']
```
And I try to find the [checksum link](https://github.com/huggingface/datasets/blob/master/datasets/squad/dataset_infos.json)
,is the problem plain_text do not have a checksum?

### 2 When I try to train lxmert,and use local dataset:
```
python examples/question-answering/run_qa.py --model_name_or_path unc-nlp/lxmert-base-uncased --train_file $SQUAD_DIR/train-v1.1.json --validation_file $SQUAD_DIR/dev-v1.1.json --do_train --do_eval --per_device_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2 --max_seq_length 384 --doc_stride 128 --output_dir /home2/zhenggo1/checkpoint/lxmert_squad
```
The bug is that 
```
['title', 'paragraphs']
Traceback (most recent call last):
  File "examples/question-answering/run_qa.py", line 501, in <module>
    main()
  File "examples/question-answering/run_qa.py", line 273, in main
    answer_column_name = "answers" if "answers" in column_names else column_names[2]
IndexError: list index out of range
```
I print the answer_column_name and find that local squad dataset need the package datasets to preprocessing so that the code below can work:
```
if training_args.do_train:
        column_names = datasets["train"].column_names
    else:
        column_names = datasets["validation"].column_names
    print(datasets["train"].column_names)
    question_column_name = "question" if "question" in column_names else column_names[0]
    context_column_name = "context" if "context" in column_names else column_names[1]
    answer_column_name = "answers" if "answers" in column_names else column_names[2]
``` 
## Please tell me how to fix the bug,thks a lot!

Comments: Hi !

To fix 1, an you try to run this code ?
```python
from datasets import load_dataset

load_dataset("squad", download_mode="force_redownload")
```
Maybe the file your downloaded was corrupted, in this case redownloading this way should fix your issue 1.

Regarding your 2nd point, you're right that loading the raw json this way doesn't give you a dataset with the column "context", "question" and "answers". Indeed the squad format is a very nested format so you have to preprocess the data. You can do it this way:
```python
def process_squad(examples):
    """
    Process a dataset in the squad format with columns "title" and "paragraphs"
    to return the dataset with columns "context", "question" and "answers".
    """
    out = {"context": [], "question": [], "answers":[]} 
    for paragraphs in examples["paragraphs"]: 
        for paragraph in paragraphs: 
            for qa in paragraph["qas"]: 
                answers = [{"answer_start": answer["answer_start"], "text": answer["text"].strip()} for answer in qa["answers"]] 
                out["context"].append(paragraph["context"].strip()) 
                out["question"].append(qa["question"].strip()) 
                out["answers"].append(answers) 
    return out

datasets = load_dataset(extension, data_files=data_files, field="data")
column_names = datasets["train"].column_names

if set(column_names) == {"title", "paragraphs"}:
    datasets = datasets.map(process_squad, batched=True, remove_columns=column_names)
```

Hope that helps :)

Thks for quickly answering！
### 1 I try the first way,but seems not work 
```
Traceback (most recent call last):
  File "examples/question-answering/run_qa.py", line 503, in <module>
    main()
  File "examples/question-answering/run_qa.py", line 218, in main
    datasets = load_dataset(data_args.dataset_name, download_mode="force_redownload")
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/load.py", line 746, in load_dataset
    use_auth_token=use_auth_token,
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/builder.py", line 573, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/builder.py", line 633, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json']
```
### 2 I try the second way,and run the examples/question-answering/run_qa.py,it lead to another bug orz..
```
Traceback (most recent call last):
  File "examples/question-answering/run_qa.py", line 523, in <module>
    main()
  File "examples/question-answering/run_qa.py", line 379, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1120, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/home2/zhenggo1/anaconda3/envs/lpot/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1091, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "examples/question-answering/run_qa.py", line 339, in prepare_train_features
    if len(answers["answer_start"]) == 0:
TypeError: list indices must be integers or slices, not str
```
## may be the function prepare_train_features in run_qa.py need to fix,I think is that the prep
```python
for i, offsets in enumerate(offset_mapping):
        # We will label impossible answers with the index of the CLS token.
        input_ids = tokenized_examples["input_ids"][i]
        cls_index = input_ids.index(tokenizer.cls_token_id)

        # Grab the sequence corresponding to that example (to know what is the context and what is the question).
        sequence_ids = tokenized_examples.sequence_ids(i)

        # One example can give several spans, this is the index of the example containing this span of text.
        sample_index = sample_mapping[i]
        answers = examples[answer_column_name][sample_index]
        print(examples,answers)
        # If no answers are given, set the cls_index as answer.
        if len(answers["answer_start"]) == 0:
            tokenized_examples["start_positions"].append(cls_index)
            tokenized_examples["end_positions"].append(cls_index)
        else:
            # Start/end character index of the answer in the text.
            start_char = answers["answer_start"][0]
            end_char = start_char + len(answers["text"][0])

            # Start token index of the current span in the text.
            token_start_index = 0
            while sequence_ids[token_start_index] != (1 if pad_on_right else 0):
                token_start_index += 1

            # End token index of the current span in the text.
            token_end_index = len(input_ids) - 1
            while sequence_ids[token_end_index] != (1 if pad_on_right else 0):
                token_end_index -= 1

            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
            if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                # Note: we could go after the last offset if the answer is the last word (edge case).
                while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:
                    token_start_index += 1
                tokenized_examples["start_positions"].append(token_start_index - 1)
                while offsets[token_end_index][1] >= end_char:
                    token_end_index -= 1
                tokenized_examples["end_positions"].append(token_end_index + 1)

    return tokenized_examples
``` 

## I have fixed it, @lhoestq 
### the first section change as you said and add ["id"]
```python
def process_squad(examples):
    """
    Process a dataset in the squad format with columns "title" and "paragraphs"
    to return the dataset with columns "context", "question" and "answers".
    """
    # print(examples)
    out = {"context": [], "question": [], "answers":[],"id":[]} 
    for paragraphs in examples["paragraphs"]: 
        for paragraph in paragraphs: 
            for qa in paragraph["qas"]: 
                answers = [{"answer_start": answer["answer_start"], "text": answer["text"].strip()} for answer in qa["answers"]] 
                out["context"].append(paragraph["context"].strip()) 
                out["question"].append(qa["question"].strip()) 
                out["answers"].append(answers) 
                out["id"].append(qa["id"]) 
    return out
column_names = datasets["train"].column_names if training_args.do_train else datasets["validation"].column_names
# print(datasets["train"].column_names)
if set(column_names) == {"title", "paragraphs"}:
    datasets = datasets.map(process_squad, batched=True, remove_columns=column_names)
# Preprocessing the datasets.
# Preprocessing is slighlty different for training and evaluation.
if training_args.do_train:
    column_names = datasets["train"].column_names
else:
    column_names = datasets["validation"].column_names
# print(column_names)
question_column_name = "question" if "question" in column_names else column_names[0]
context_column_name = "context" if "context" in column_names else column_names[1]
answer_column_name = "answers" if "answers" in column_names else column_names[2]
```
### the second section
```python
def prepare_train_features(examples):
    # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results
    # in one example possible giving several features when a context is long, each of those features having a
    # context that overlaps a bit the context of the previous feature.
    tokenized_examples = tokenizer(
        examples[question_column_name if pad_on_right else context_column_name],
        examples[context_column_name if pad_on_right else question_column_name],
        truncation="only_second" if pad_on_right else "only_first",
        max_length=data_args.max_seq_length,
        stride=data_args.doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length" if data_args.pad_to_max_length else False,
    )

    # Since one example might give us several features if it has a long context, we need a map from a feature to
    # its corresponding example. This key gives us just that.
    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")
    # The offset mappings will give us a map from token to character position in the original context. This will
    # help us compute the start_positions and end_positions.
    offset_mapping = tokenized_examples.pop("offset_mapping")

    # Let's label those examples!
    tokenized_examples["start_positions"] = []
    tokenized_examples["end_positions"] = []

    for i, offsets in enumerate(offset_mapping):
        # We will label impossible answers with the index of the CLS token.
        input_ids = tokenized_examples["input_ids"][i]
        cls_index = input_ids.index(tokenizer.cls_token_id)

        # Grab the sequence corresponding to that example (to know what is the context and what is the question).
        sequence_ids = tokenized_examples.sequence_ids(i)

        # One example can give several spans, this is the index of the example containing this span of text.
        sample_index = sample_mapping[i]
        answers = examples[answer_column_name][sample_index]
        # print(examples,answers,offset_mapping,tokenized_examples)
        # If no answers are given, set the cls_index as answer.
        if len(answers) == 0:#len(answers["answer_start"]) == 0:
            tokenized_examples["start_positions"].append(cls_index)
            tokenized_examples["end_positions"].append(cls_index)
        else:
            # Start/end character index of the answer in the text.
            start_char = answers[0]["answer_start"]
            end_char = start_char + len(answers[0]["text"])

            # Start token index of the current span in the text.
            token_start_index = 0
            while sequence_ids[token_start_index] != (1 if pad_on_right else 0):
                token_start_index += 1

            # End token index of the current span in the text.
            token_end_index = len(input_ids) - 1
            while sequence_ids[token_end_index] != (1 if pad_on_right else 0):
                token_end_index -= 1

            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
            if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                # Note: we could go after the last offset if the answer is the last word (edge case).
                while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:
                    token_start_index += 1
                tokenized_examples["start_positions"].append(token_start_index - 1)
                while offsets[token_end_index][1] >= end_char:
                    token_end_index -= 1
                tokenized_examples["end_positions"].append(token_end_index + 1)
    return tokenized_examples
```

I'm glad you managed to fix run_qa.py for your case :)

Regarding the checksum error, I'm not able to reproduce on my side.
This errors says that the downloaded file doesn't match the expected file.

Could you try running this and let me know if you get the same output as me ?
```python
from datasets.utils.info_utils import get_size_checksum_dict
from datasets import cached_path

get_size_checksum_dict(cached_path("https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json"))
# {'num_bytes': 30288272, 'checksum': '3527663986b8295af4f7fcdff1ba1ff3f72d07d61a20f487cb238a6ef92fd955'}
```

I run the code,and it show below:
```
>>> from datasets.utils.info_utils import get_size_checksum_dict
>>> from datasets import cached_path
>>> get_size_checksum_dict(cached_path("https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json"))
Downloading: 30.3MB [04:13, 120kB/s]
{'num_bytes': 30288272, 'checksum': '3527663986b8295af4f7fcdff1ba1ff3f72d07d61a20f487cb238a6ef92fd955'}
```

Alright ! So in this case redownloading the file with `download_mode="force_redownload"` should fix it. Can you try using `download_mode="force_redownload"` again ?

Not sure why it didn't work for you the first time though :/