KeyError: overflow_to_sample_mapping - LayoutLMv3
Author: Marcel1805
Pull request: no

### System Info

Running on private server with no public internet access

transformers version: 4.25.1
Platform: RHEL 7.9
Python version: 3.8.12
Huggingface_hub version: 0.11.1
Torch version: 1.13.0
nvidia-cublas-cu11:          11.10.3.66
nvidia-cuda-nvrtc-cu11:        11.7.99
nvidia-cuda-runtime-cu11:      11.7.99
nvidia-cudnn-cu11:             8.5.0.96
Tensorflow version (GPU?): 2.8.2 (True)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @NielsRogge for LayoutLMv3


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction
Details regarding the dataset which is used for token classification:
![image](https://user-images.githubusercontent.com/12809547/208253684-71e34cef-69c5-43cb-8f8f-f8b0f42c52bc.png)

**Code snippet:**

```
processor = AutoProcessor.from_pretrained("<dir-to-layoutlmv3-base>", apply_ocr=False)
processor
```
**Output**

LayoutLMv3Processor:
- feature_extractor: LayoutLMv3ImageProcessor {
  "apply_ocr": false,
  "do_normalize": true,
  "do_rescale": true,
  "do_resize": true,
  "feature_extractor_type": "LayoutLMv3FeatureExtractor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_processor_type": "LayoutLMv3ImageProcessor",
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "ocr_lang": null,
  "resample": 2,
  "rescale_factor": 0.00392156862745098,
  "size": {
    "height": 224,
    "width": 224
  },
  "tesseract_config": ""
}

- tokenizer: PreTrainedTokenizerFast(name_or_path='dir-to-layoutlmv3-base', vocab_size=50265, model_max_len=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': AddedToken("< s >", rstrip=False, lstrip=False, single_word=False, normalized=True), 'eos_token': AddedToken("< /s >", rstrip=False, lstrip=False, single_word=False, normalized=True), 'unk_token': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'sep_token': AddedToken("< /s >", rstrip=False, lstrip=False, single_word=False, normalized=True), 'pad_token': AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'cls_token': AddedToken(" < s > ", rstrip=False, lstrip=False, single_word=False, normalized=True), 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=True)})

```
def prepare_examples(examples):
    images = examples["image"]
    words = examples["value"]
    boxes = examples["bbox"]
    word_labels = examples["label"]
    processor_kwargs = {"return_offsets_mapping": False, "return_overflowing_tokens": True, "stride": 100, "max_length": 512}
    
    encoding = processor(images, words, boxes=boxes, word_labels=word_labels,
                           truncation=True, padding="max_length", **processor_kwargs)

    return encoding

features = Features({
    'pixel_values': Array3D(dtype="float32", shape=(3, 224, 224)),
    'input_ids': Sequence(feature=Value(dtype='int64')),
    'attention_mask': Sequence(Value(dtype='int64')),
    'bbox': Array2D(dtype="int64", shape=(512, 4)), 
    'labels': Sequence(feature=Value(dtype='int64'))
})

eval_dataset = globaldataset["test"].map(
    prepare_examples,
    batched=True,
    remove_columns=column_names,
    features=features,
)
```
**Traceback:**
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[215], line 1
----> 1 eval_dataset = globaldataset["test"].map(
      2     prepare_examples,
      3     batched=True,
      4     remove_columns=column_names,
      5     features=features,
      6 )

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:2585, in Dataset.map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2582 disable_tqdm = not logging.is_progress_bar_enabled()
   2584 if num_proc is None or num_proc == 1:
-> 2585     return self._map_single(
   2586         function=function,
   2587         with_indices=with_indices,
   2588         with_rank=with_rank,
   2589         input_columns=input_columns,
   2590         batched=batched,
   2591         batch_size=batch_size,
   2592         drop_last_batch=drop_last_batch,
   2593         remove_columns=remove_columns,
   2594         keep_in_memory=keep_in_memory,
   2595         load_from_cache_file=load_from_cache_file,
   2596         cache_file_name=cache_file_name,
   2597         writer_batch_size=writer_batch_size,
   2598         features=features,
   2599         disable_nullable=disable_nullable,
   2600         fn_kwargs=fn_kwargs,
   2601         new_fingerprint=new_fingerprint,
   2602         disable_tqdm=disable_tqdm,
   2603         desc=desc,
   2604     )
   2605 else:
   2607     def format_cache_file_name(cache_file_name, rank):

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:585, in transmit_tasks.<locals>.wrapper(*args, **kwargs)
    583     self: "Dataset" = kwargs.pop("self")
    584 # apply actual function
--> 585 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    586 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    587 for dataset in datasets:
    588     # Remove task templates if a column mapping of the template is no longer valid

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:552, in transmit_format.<locals>.wrapper(*args, **kwargs)
    545 self_format = {
    546     "type": self._format_type,
    547     "format_kwargs": self._format_kwargs,
    548     "columns": self._format_columns,
    549     "output_all_columns": self._output_all_columns,
    550 }
    551 # apply actual function
--> 552 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    553 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    554 # re-apply format to the output

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
    482 # Update fingerprint of in-place transforms + update in-place history of transforms
    484 if inplace:  # update after calling func so that the fingerprint doesn't change if the function fails

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:2999, in Dataset._map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2997                 writer.write_table(batch)
   2998             else:
-> 2999                 writer.write_batch(batch)
   3000 if update_data and writer is not None:
   3001     writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_writer.py:533, in ArrowWriter.write_batch(self, batch_examples, writer_batch_size)
    526 cols = (
    527     [col for col in self.schema.names if col in batch_examples]
    528     + [col for col in batch_examples.keys() if col not in self.schema.names]
    529     if self.schema
    530     else batch_examples.keys()
    531 )
    532 for col in cols:
--> 533     col_type = features[col] if features else None
    534     col_try_type = try_features[col] if try_features is not None and col in try_features else None
    535     typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)

KeyError: 'overflow_to_sample_mapping

**Changed return_offsets_mapping to True with everything else unchanged returns another error:**
```
    processor_kwargs = {"return_offsets_mapping": True, "return_overflowing_tokens": True, "stride": 100, "max_length": 512}
    #Erstellen eines Processors-Objektes
    encoding = processor(images, words, boxes=boxes, word_labels=word_labels,
                           truncation=True, padding="max_length", **processor_kwargs)
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[219], line 1
----> 1 eval_dataset = globaldataset["test"].map(
      2     prepare_examples,
      3     batched=True,
      4     remove_columns=column_names,
      5     features=features,
      6 )

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:2585, in Dataset.map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2582 disable_tqdm = not logging.is_progress_bar_enabled()
   2584 if num_proc is None or num_proc == 1:
-> 2585     return self._map_single(
   2586         function=function,
   2587         with_indices=with_indices,
   2588         with_rank=with_rank,
   2589         input_columns=input_columns,
   2590         batched=batched,
   2591         batch_size=batch_size,
   2592         drop_last_batch=drop_last_batch,
   2593         remove_columns=remove_columns,
   2594         keep_in_memory=keep_in_memory,
   2595         load_from_cache_file=load_from_cache_file,
   2596         cache_file_name=cache_file_name,
   2597         writer_batch_size=writer_batch_size,
   2598         features=features,
   2599         disable_nullable=disable_nullable,
   2600         fn_kwargs=fn_kwargs,
   2601         new_fingerprint=new_fingerprint,
   2602         disable_tqdm=disable_tqdm,
   2603         desc=desc,
   2604     )
   2605 else:
   2607     def format_cache_file_name(cache_file_name, rank):

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:585, in transmit_tasks.<locals>.wrapper(*args, **kwargs)
    583     self: "Dataset" = kwargs.pop("self")
    584 # apply actual function
--> 585 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    586 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    587 for dataset in datasets:
    588     # Remove task templates if a column mapping of the template is no longer valid

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:552, in transmit_format.<locals>.wrapper(*args, **kwargs)
    545 self_format = {
    546     "type": self._format_type,
    547     "format_kwargs": self._format_kwargs,
    548     "columns": self._format_columns,
    549     "output_all_columns": self._output_all_columns,
    550 }
    551 # apply actual function
--> 552 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    553 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    554 # re-apply format to the output

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
    482 # Update fingerprint of in-place transforms + update in-place history of transforms
    484 if inplace:  # update after calling func so that the fingerprint doesn't change if the function fails

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_dataset.py:2999, in Dataset._map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2997                 writer.write_table(batch)
   2998             else:
-> 2999                 writer.write_batch(batch)
   3000 if update_data and writer is not None:
   3001     writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

File ~/.conda/envs/maschinellebelegauslesung_dev_gpu/lib/python3.8/site-packages/datasets/arrow_writer.py:533, in ArrowWriter.write_batch(self, batch_examples, writer_batch_size)
    526 cols = (
    527     [col for col in self.schema.names if col in batch_examples]
    528     + [col for col in batch_examples.keys() if col not in self.schema.names]
    529     if self.schema
    530     else batch_examples.keys()
    531 )
    532 for col in cols:
--> 533     col_type = features[col] if features else None
    534     col_try_type = try_features[col] if try_features is not None and col in try_features else None
    535     typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)

KeyError: 'offset_mapping'

**Changing the value of the truncation-parameter to False does not alter the KeyErrors.**



### Expected behavior

I´d expect a dataset with few more entries than before, because for every entry which has more than 512 token a second entry with the overflowing token + stride-overlap would have been created with the sliding window approach.
Regarding this KeyError overflow_to_sample_mapping I found one Git Issue https://github.com/huggingface/transformers/issues/18726, which reports a bug for the LayoutXLM with the non-fast tokenizer. But in my case as shown in the code snippet the processor loads the PreTrainedTokenizerFast for the LayoutLMv3. 

Thank you very much! 

Comments:
@ArthurZucker Hi Arthur, I just saw you self-assigned this ticket, were you already able to reproduce this error? Do you need any further information/ context?
Thank you very much!
Best regards, 
Marcel

Hey, I didn't have the chance to do so yet! If you could provide me with a minimal reproducing script, it would be really great! 

Hey @ArthurZucker, yes sure, please find a google colab notebook here: https://colab.research.google.com/drive/1Ce4H6r7PecaLbohqiGr8Uw-7zBn0-CVr?usp=share_link

This is a public notebook from @rajshah4 which I modified a little bit to implement the sliding window approach, which runs into the same error "overflow_to_sample_mapping". The example dataset CORD doesn´t contain documents longer than 512 token, that´s why I set the max_length=100 to artificially create the need for a sliding window approach.

(I cannot share the original notebook of mine because it contains sensible data.) 

Thank you very much! 

Hi @ArthurZucker 
was the script for reproducing the error helpful? 
If you need any further information just let me know, thank you very much! 

Hi @Marcel1805 

When you activate the sliding window approach a new key is added to the endcoding-dict (overflow_to_sample_mapping)

A simple  `encoding.pop('overflow_to_sample_mapping', None)` should do it

Hi @makra89 
perfect, it works now as intended, thank you so much! 👍 

Thanks @makra89 😉 

Hi, just to clarify since it took me a bit to understand where the new line is needed. The "encoding-dict" is what you get after applying the processor to your data, something like:

```
    encoding = processor(
        images,
        words,
        boxes=boxes,
        word_labels=ner_tags,
        truncation=True,
        padding="max_length",
        return_overflowing_tokens=True,
        return_offsets_mapping=None,
        stride=51,
    )
        encoding.pop("overflow_to_sample_mapping")
```