sqaud_v2 dataset contains misalignment between the answer text and the context value at the answer index
State: closed | Author: mmajurski (NONE)
Labels: name: bug

## Describe the bug
The built in huggingface squad_v2 dataset that you can access via datasets.load_dataset contains mis-alignment between the answers['text'] and the characters in the context at the location specified by answers['answer_start'].

For example:
id = '56d1f453e7d4791d009025bd'
answers = {'text': ['Pure Land'], 'answer_start': [146]}
However the actual text in context at location 146 is 'ure Land,'
Which is an off-by-one error from the correct answer.

## Steps to reproduce the bug
```python
import datasets

def check_context_answer_alignment(example):
    for a_idx in range(len(example['answers']['text'])):
        # check raw dataset for answer consistency between context and answer
        answer_text = example['answers']['text'][a_idx]
        a_st_idx = example['answers']['answer_start'][a_idx]
        a_end_idx = a_st_idx + len(example['answers']['text'][a_idx])
        answer_text_from_context = example['context'][a_st_idx:a_end_idx]
        if answer_text != answer_text_from_context:
            #print(example['id'])
            return False
    return True

dataset = datasets.load_dataset('squad_v2', split='train', keep_in_memory=True)

start_len = len(dataset)
dataset = dataset.filter(check_context_answer_alignment,
                                num_proc=1,
                                keep_in_memory=True)
end_len = len(dataset)
print('{} instances contain mis-alignment between the answer text and answer index.'.format(start_len - end_len))
```

## Expected results
This code should result in 0 rows being filtered out from the dataset.

## Actual results
This filter command results in 258 rows being flagged as containing a discrepancy between the text contained within answers['text'] and the text in example['context'] at the answers['answer_start'] location.

This code will reproduce the problem and produce the following count:
"258 instances contain mis-alignment between the answer text and answer index."

## Environment info
Steps to rebuilt the Conda environment:
```
# create a virtual environment to stuff all these packages into
conda create -n round8 python=3.8 -y

# activate the virtual environment
conda activate round8

# install pytorch (best done through conda to handle cuda dependencies)
conda install pytorch torchvision torchtext cudatoolkit=11.1 -c pytorch-lts -c nvidia

pip install jsonpickle transformers datasets matplotlib
```

OS: Ubuntu 20.04
Python 3.8

Result of `conda env export`:
```
name: round8
channels:
  - pytorch-lts
  - nvidia
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - blas=1.0=mkl
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2021.5.25=h06a4308_1
  - certifi=2021.5.30=py38h06a4308_0
  - cffi=1.14.5=py38h261ae71_0
  - chardet=4.0.0=py38h06a4308_1003
  - cryptography=3.4.7=py38hd23ed53_0
  - cudatoolkit=11.1.74=h6bb024c_0
  - ffmpeg=4.2.2=h20bf706_0
  - freetype=2.10.4=h5ab3b9f_0
  - gmp=6.2.1=h2531618_2
  - gnutls=3.6.15=he1e5248_0
  - idna=2.10=pyhd3eb1b0_0
  - intel-openmp=2021.2.0=h06a4308_610
  - jpeg=9b=h024ee3a_2
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libidn2=2.3.1=h27cfd23_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.2.0=h85742a9_0
  - libunistring=0.9.10=h27cfd23_0
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - libwebp-base=1.2.0=h27cfd23_0
  - lz4-c=1.9.3=h2531618_0
  - mkl=2021.2.0=h06a4308_296
  - mkl-service=2.3.0=py38h27cfd23_1
  - mkl_fft=1.3.0=py38h42c9631_2
  - mkl_random=1.2.1=py38ha9443f7_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.3=hbbd107a_1
  - ninja=1.10.2=hff7bd54_1
  - numpy=1.20.2=py38h2d18471_0
  - numpy-base=1.20.2=py38hfae3a4d_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1k=h27cfd23_0
  - pillow=8.2.0=py38he98fc37_0
  - pip=21.1.2=py38h06a4308_0
  - pycparser=2.20=py_2
  - pyopenssl=20.0.1=pyhd3eb1b0_1
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.10=h12debd9_8
  - pytorch=1.8.1=py3.8_cuda11.1_cudnn8.0.5_0
  - readline=8.1=h27cfd23_0
  - requests=2.25.1=pyhd3eb1b0_0
  - setuptools=52.0.0=py38h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - sqlite=3.35.4=hdfb4753_0
  - tk=8.6.10=hbc83047_0
  - torchtext=0.9.1=py38
  - torchvision=0.9.1=py38_cu111
  - typing_extensions=3.7.4.3=pyha847dfd_0
  - urllib3=1.26.4=pyhd3eb1b0_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.9=haebb681_0
  - pip:
    - click==8.0.1
    - cycler==0.10.0
    - datasets==1.8.0
    - dill==0.3.4
    - filelock==3.0.12
    - fsspec==2021.6.0
    - huggingface-hub==0.0.8
    - joblib==1.0.1
    - jsonpickle==2.0.0
    - kiwisolver==1.3.1
    - matplotlib==3.4.2
    - multiprocess==0.70.12.2
    - packaging==20.9
    - pandas==1.2.4
    - pyarrow==3.0.0
    - pyparsing==2.4.7
    - python-dateutil==2.8.1
    - pytz==2021.1
    - regex==2021.4.4
    - sacremoses==0.0.45
    - tokenizers==0.10.3
    - tqdm==4.49.0
    - transformers==4.6.1
    - xxhash==2.0.2
prefix: /home/mmajurski/anaconda3/envs/round8
```


Comments: Hi @mmajurski, thanks for reporting this issue.

Indeed this misalignment arises because the source dataset context field contains leading blank spaces (and these are counted within the answer_start), while our datasets loading script removes these leading blank spaces.

I'm going to fix our script so that all leading blank spaces in the source dataset are kept, and there is no misalignment between the answer text and the answer_start within the context.

If you are going to be altering the data cleaning from the source Squad dataset, here is one thing to consider.
There are occasional double spaces separating words which it might be nice to get rid of. 

Either way, thank you.