GitHub issue #2585: sqaud_v2 dataset contains misalignment between the answer text and the context value at the answer index
State: closed | Created: 2021-07-04T15:39:49Z

## Describe the bug
The built in huggingface squad_v2 dataset that you can access via datasets.load_dataset contains mis-alignment between the answers['text'] and the characters in the context at the location specified by answers['answer_start'].

For example:
id = '56d1f453e7d4791d009025bd'
answers = {'text': ['Pure Land'], 'answer_start': [146]}
However the actual text in context at location 146 is 'ure Land,'
Which is an off-by-one error from the correct answer.

## Steps to reproduce the bug
```python
import datasets

def check_context_answer_alignment(example):
    for a_idx in range(len(example['answers']['text'])):
        # check raw dataset for answer consistency between context and answer
        answer_text = example['answers']['text'][a_idx]
        a_st_idx = example['answers']['answer_start'][a_idx]
        a_end_idx = a_st_idx + len(example['answers']['text'][a_idx])
        answer_text_from_context = example['context'][a_st_idx:a_end_idx]
        if answer_text != answer_text_from_context:
            #print(example['id'])
            return False
    return True

dataset = datasets.load_dataset('squad_v2', split='train', keep_in_memory=True)

start_len = len(dataset)
dataset = dataset.filter(check_context_answer_alignment,
                                num_proc=1,
                                keep_in_memory=True)
end_len = len(dataset)
print('{} instances contain mis-alignment between the answer text and answer index.'.format(start_len - end_len))
```

## Expected results
This code should result in 0 rows being filtered out from the dataset.

## Actual results
This filter command results in 258 rows being flagged as containing a discrepancy between the text contained within answers['text'] and the text in example['context'] at the answers['answer_start'] location.

This code will reproduce the problem and produce the following count:
"258 instances contain mis-alignment between the answer text and answer index."

## Environment info
Steps to rebuilt the Conda environment:
```
# create a virtual environment to stuff all these packages into
conda create -n round8 python=3.8 -y

# activate the virtual environment
conda activate round8

# install pytorch (best done through conda to handle cuda dependencies)
conda install pytorch torchvision torchtext cudatoolkit=11.1 -c pytorch-lts -c nvidia

pip install jsonpickle transformers datasets matplotlib
```

OS: Ubuntu 20.04
Python 3.8

Result of `conda env export`:
```
name: round8
channels:
  - pytorch-lts
  - nvidia
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - blas=1.0=mkl
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2021.5.25=h06a4308_1
  - certifi=2021.5.30=py38h06a4308_0
  - cffi=1.14.5=py38h261ae71_0
  - chardet=4.0.0=py38h06a4308_1003
  - cryptography=3.4.7=py38hd23ed53_0
  - cudatoolkit=11.1.74=h6bb024c_0
  - ffmpeg=4.2.2=h20bf706_0
  - freetype=2.10.4=h5ab3b9f_0
  - gmp=6.2.1=h2531618_2
  - gnutls=3.6.15=he1e5248_0
  - idna=2.10=pyhd3eb1b0_0
  - intel-openmp=2021.2.0=h06a4308_610
  - jpeg=9b=h024ee3a_2
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libidn2=2.3.1=h27cfd23_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - libtasn1=4.16.0=h27cfd23_0
  - libtiff=4.2.0=h85742a9_0
  - libunistring=0.9.10=h27cfd23_0
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - libwebp-base=1.2.0=h27cfd23_0
  - lz4-c=1.9.3=h2531618_0
  - mkl=2021.2.0=h06a4308_296
  - mkl-service=2.3.0=py38h27cfd23_1
  - mkl_fft=1.3.0=py38h42c9631_2
  - mkl_random=1.2.1=py38ha9443f7_2
  - ncurses=6.2=he6710b0_1
  - nettle=3.7.3=hbbd107a_1
  - ninja=1.10.2=hff7bd54_1
  - numpy=1.20.2=py38h2d18471_0
  - numpy-base=1.20.2=py38hfae3a4d_0
  - olefile=0.46=py_0
  - openh264=2.1.0=hd408876_0
  - openssl=1.1.1k=h27cfd23_0
  - pillow=8.2.0=py38he98fc37_0
  - pip=21.1.2=py38h06a4308_0
  - pycparser=2.20=py_2
  - pyopenssl=20.0.1=pyhd3eb1b0_1
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.10=h12debd9_8
  - pytorch=1.8.1=py3.8_cuda11.1_cudnn8.0.5_0
  - readline=8.1=h27cfd23_0
  - requests=2.25.1=pyhd3eb1b0_0
  - setuptools=52.0.0=py38h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - sqlite=3.35.4=hdfb4753_0
  - tk=8.6.10=hbc83047_0
  - torchtext=0.9.1=py38
  - torchvision=0.9.1=py38_cu111
  - typing_extensions=3.7.4.3=pyha847dfd_0
  - urllib3=1.26.4=pyhd3eb1b0_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - x264=1!157.20191217=h7b6447c_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.9=haebb681_0
  - pip:
    - click==8.0.1
    - cycler==0.10.0
    - datasets==1.8.0
    - dill==0.3.4
    - filelock==3.0.12
    - fsspec==2021.6.0
    - huggingface-hub==0.0.8
    - joblib==1.0.1
    - jsonpickle==2.0.0
    - kiwisolver==1.3.1
    - matplotlib==3.4.2
    - multiprocess==0.70.12.2
    - packaging==20.9
    - pandas==1.2.4
    - pyarrow==3.0.0
    - pyparsing==2.4.7
    - python-dateutil==2.8.1
    - pytz==2021.1
    - regex==2021.4.4
    - sacremoses==0.0.45
    - tokenizers==0.10.3
    - tqdm==4.49.0
    - transformers==4.6.1
    - xxhash==2.0.2
prefix: /home/mmajurski/anaconda3/envs/round8
```