LayoutLMV3 Training with Morethan 512 tokens.
Author: purnasai-cyient
Pull request: no

Hi, It is mentioned in the Research paper and in Model docs that the model is trained with `maximum_length = 512` tokens. But when we have large tokens with an average of 1124 tokens, I thought to increase `maximum_length = 1024`. But since it is limited, I have used `Stride` and implemented a dataclass.

![image](https://user-images.githubusercontent.com/112476954/192162483-26a375c1-9584-4122-865e-069e6f304332.png)

So when I have a long text sentence with 900 in length in this example. I got
![image](https://user-images.githubusercontent.com/112476954/192162557-9efe39a4-6915-4df5-9995-c34f2bce36d1.png)

But when the same sent to trainer
![image](https://user-images.githubusercontent.com/112476954/192162648-12599fad-9f4c-4bbc-94e7-13fbf5009162.png)

I got error like below:
![image](https://user-images.githubusercontent.com/112476954/192162684-b629bcaa-4b06-477b-9f20-4e7fa34c00c3.png)

From the Error, I understood that My `input_ids` are in the shape of [2,512] and once after going into the trainer, they become [1,2,512] after batch size is added. 
But the trainer is expecting My `input_ids` to be in [512] after squeezed, so the trainer adds batch size 1 to make [1,512].

Now how to send my datasamples of length more than 512, with stride in processor, though i am able to load the data. But the trainer is not expecting sizes more than 512. Any suggestions here

Thank you


Comments:
cc @NielsRogge 

> 
@purnasai-cyient 
Hi,
I have same issue with you. Have you solved it?

> Hi, It is mentioned in the Research paper and in Model docs that the model is trained with `maximum_length = 512` tokens. But when we have large tokens with an average of 1124 tokens, I thought to increase `maximum_length = 1024`. But since it is limited, I have used `Stride` and implemented a dataclass.
> 
> ![image](https://user-images.githubusercontent.com/112476954/192162483-26a375c1-9584-4122-865e-069e6f304332.png)
> 
> So when I have a long text sentence with 900 in length in this example. I got ![image](https://user-images.githubusercontent.com/112476954/192162557-9efe39a4-6915-4df5-9995-c34f2bce36d1.png)
> 
> But when the same sent to trainer ![image](https://user-images.githubusercontent.com/112476954/192162648-12599fad-9f4c-4bbc-94e7-13fbf5009162.png)
> 
> I got error like below: ![image](https://user-images.githubusercontent.com/112476954/192162684-b629bcaa-4b06-477b-9f20-4e7fa34c00c3.png)
> 
> From the Error, I understood that My `input_ids` are in the shape of [2,512] and once after going into the trainer, they become [1,2,512] after batch size is added. But the trainer is expecting My `input_ids` to be in [512] after squeezed, so the trainer adds batch size 1 to make [1,512].
> 
> Now how to send my datasamples of length more than 512, with stride in processor, though i am able to load the data. But the trainer is not expecting sizes more than 512. Any suggestions here
> 
> Thank you



Hello @purnasai-cyient and everyone! 

I had the same issue with LayoutLMv3 and because I think this problem is common for document information extraction task so I will describe how I dealt with that:

### 1. Training:

As you may know, first of all we have to change configurations of processor by using `stride` and `padding` and `offset_mapping`:

```
....
......
processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False)

encoding = processor(images, words, boxes=boxes, word_labels=word_labels, truncation=True, stride =128, 
         padding="max_length", max_length=512, return_overflowing_tokens=True, return_offsets_mapping=True)

offset_mapping = encoding.pop('offset_mapping')

overflow_to_sample_mapping = encoding.pop('overflow_to_sample_mapping')
```

I'm not completely sure why should we turn offset_mapping True while we have to pop it after encoding but I think it's necessary to do that. To clarify, if you followed the [NielsRogge](https://github.com/NielsRogge)'s notebook ([you can find the notebook here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb)), we have to change the `prepare_examples` method like this:

```
def prepare_examples(examples):
  images = [Image.open(path).convert("RGB") for path in examples['image_path']] 
  words = examples[text_column_name]
  boxes = examples[boxes_column_name]
  word_labels = examples[label_column_name]
  encoding = processor(images, words, boxes=boxes, word_labels=word_labels, truncation=True, stride =128, 
         padding="max_length", max_length=512, return_overflowing_tokens=True, return_offsets_mapping=True)  offset_mapping = encoding.pop('offset_mapping')
  overflow_to_sample_mapping = encoding.pop('overflow_to_sample_mapping')
  return encoding
```
next you have to follow the steps normally, without any changes and train the model!

Note: It's completely normal if the number of rows in your dataset after `mapping` doesn't match with to number of your data. It's due to that now if we pass 512 tokens for a document, we would store the next tokens in another row of data (from token 512 to 1024, next from 1025 to 1536 and...)

### 2. Inference:

Inference section is a little bit tricky. Again I will describe the setups based on [mentioned notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb). Obviously we have to implement our processor with `stride` and `padding` like training phase. 

```
encoding = processor(images, words, boxes=boxes, word_labels=word_labels, truncation=True, stride =128, 
         padding="max_length", max_length=512, return_overflowing_tokens=True, return_offsets_mapping=True)

offset_mapping = encoding.pop('offset_mapping')

overflow_to_sample_mapping = encoding.pop('overflow_to_sample_mapping')
```

Next, we have to change the shape of encoding to handle multiple pages (as I said in part 1, we divided large tokens to separate entities, so now the results are 2D. I have shaped them in this way:

```
# change the shape of pixel values
x = []
for i in range(0, len(encoding['pixel_values'])):
     x.append(encoding['pixel_values'][i])
x = torch.stack(x)
encoding['pixel_values'] = x


```
so if we print encoding items, we will have something like this:

```
for k,v in encoding.items():
  print(k,v.shape)
```
results:
```
input_ids torch.Size([3, 512])
attention_mask torch.Size([3, 512])
bbox torch.Size([3, 512, 4])
pixel_values torch.Size([3, 3, 224, 224])
```
As we can see, in my case, document is divided to 3 parts (for example size of `input_ids` is [3,512] or 3x512 which if we use normal processing, we will get just one array [1, 512] for all the cases). So we're doing fine till now. We have to pass encoding to the model to get the predictions:

```
with torch.no_grad():
  outputs = model(**encoding)

# The model outputs logits of shape (batch_size, seq_len, num_labels).
logits = outputs.logits
print(logits.shape)

# We take the highest score for each token, using argmax. This serves as the predicted label for each token.
predictions = logits.argmax(-1).squeeze().tolist()
token_boxes = encoding.bbox.squeeze().tolist()

if (len(token_boxes) == 512):
  predictions = [predictions]
  token_boxes = [token_boxes]
```
Last lines (if clause) in code above, is because in cases that the number of tokens are less than 512, we will get 1D array, we have to put them in a list to prevent errors for next step. 

Finally, now we have `predictions` and `token_boxes` from the model, you can also reach the text of each bbox by using:
`processor.tokenizer.decode(encoding["input_ids"][i][j])` which` i `and` j `corresponds to the entity that you want to extract the text of it. Just as an example, we could find predictions by traversing `token_boxes` with for loop (you could do whatever you want, because we needed predictions and bboxes and we have them now! processing them is up to you ;) )

```
# this is just an example, change this code for your project!
for i in range(0, len(token_boxes)):
      for j in range(0, len(token_boxes[i])):
             print("label is: {}, bbox is: {} and the text is: {}".format(predictions[i][j], 
                    token_boxes[i][j],  processor.tokenizer.decode(encoding["input_ids"][i][j]) )

for i in range(0, len(token_boxes)):
# skip first 128 tokens from second list to last one (128 is the stride value) except the first window
initial_j = 0 if i == 0 else 129
for j in range(initial_j, len(token_boxes[i])):
            unnormal_box = unnormalize_box(token_boxes[i][j], width, height)
            if (np.asarray(token_boxes[i][j]).shape != (4,)):
                continue
            elif (token_boxes[i][j] == [0, 0, 0, 0] or token_boxes[i][j] == 0):
                continue
            else:
                bbox = tuple(unnormal_box)  # Convert the list to a tuple
                token = processor.tokenizer.decode(encoding["input_ids"][i][j])
                if bbox not in box_token_dict:
                    box_token_dict[bbox] = [token]
                else:
                    box_token_dict[bbox].append(token)

box_token_dict = {bbox: [''.join(words)] for bbox, words in box_token_dict.items()}
boxes = list(box_token_dict.keys())
words = list(box_token_dict.values())
preds = []

# here we wil find all the predictions foreach boundingbox.
# for example: [120, 80, 70, 30] = [7, 7, 1] (label 7 and 1 referring to a specific label in id2label
box_prediction_dict = {}
for i in range(0, len(token_boxes)):
        for j in range(0, len(token_boxes[i])):
            if (np.asarray(token_boxes[i][j]).shape != (4,)):
                continue
            elif (token_boxes[i][j] == [0, 0, 0, 0] or token_boxes[i][j] == 0):
                continue
            else:
                bbox = tuple(token_boxes[i][j])  # Convert the list to a tuple
                prediction = predictions[i][j]
                if bbox not in box_prediction_dict:
                    box_prediction_dict[bbox] = [prediction]
                else:
                    box_prediction_dict[bbox].append(prediction)

# now foreach bbox we have a list of it's predictions, then we are going to count them and
# select the most repeated one as the final prediction for our bbox
for i, (bbox, predictions) in enumerate(box_prediction_dict.items()):
        count_dict = {}  # Dictionary to store the count of each prediction label
        for prediction in predictions:
            if prediction in count_dict:
                count_dict[prediction] += 1
            else:
                count_dict[prediction] = 1

        max_count = max(count_dict.values())  # Find the maximum count of a prediction label
        max_predictions = [key for key, value in count_dict.items() if
                           value == max_count]  # Find the predictions with the maximum count
        # If there is only one prediction with the maximum count so we can add it to the preds, but
        # if there are more than one, so we don't know which one should be label for the box
        if len(max_predictions) == 1:
            preds.append(max_predictions[0])
        else:
            others_id = next((id_ for id_, label in id2label.items() if label == 'others'), None)
            #print(f'others_id:{others_id}')
            if others_id in count_dict:
                #print('yess it is here')
                count_dict.pop(others_id)
                new_max_count = max(count_dict.values())
                new_max_predictions = [key for key, value in count_dict.items() if
                                   value == new_max_count]  # Find the predictions with the maximum count
                if len(new_max_predictions) == 1:
                    preds.append(new_max_predictions[0])
                    continue
            # the idea is to look at next and previous items in the dictionary, if they have a label like the labels
            # that we have for this box, so we can select it for label
            # ex: ANKORIG SRL - ANKORIG = [0, 6] SRL = [6] so--> ANKORIG could be considered as 6
            max_next_prev_item = []
            # Check the previous item (if it exists)
            if i - 1 >= 0:
                prev_predictions = box_prediction_dict[
                    list(box_prediction_dict.keys())[i - 1]]  # Get the predictions of the previous item
                prev_count_dict = {key: prev_predictions.count(key) for key in
                                   set(prev_predictions)}  # Count the occurrences of each prediction
                max_prev_count = max(prev_count_dict.values())  # Find the maximum count of a prediction label
                max_prev_predictions = [key for key, value in prev_count_dict.items() if
                                        value == max_prev_count]  # Find the predictions with the maximum count
                max_next_prev_item.extend(max_prev_predictions)  # Add the predictions to the max_next_prev_item list

            # Check the next item (if it exists)
            if i + 1 < len(box_prediction_dict):
                next_predictions = box_prediction_dict[
                    list(box_prediction_dict.keys())[i + 1]]  # Get the predictions of the next item
                next_count_dict = {key: next_predictions.count(key) for key in
                                   set(next_predictions)}  # Count the occurrences of each prediction
                max_next_count = max(next_count_dict.values())  # Find the maximum count of a prediction label
                max_next_predictions = [key for key, value in next_count_dict.items() if
                                        value == max_next_count]  # Find the predictions with the maximum count
                max_next_prev_item.extend(max_next_predictions)  # Add the predictions to the max_next_prev_item list

            if any(prediction in max_next_prev_item for prediction in
                   max_predictions):  # If there are common predictions between max_next_prev_item and max_predictions
                common_predictions = list(
                    set(max_predictions).intersection(max_next_prev_item))  # Find the common predictions
                # Add a randomly chosen common prediction to the preds, random would work if we have again more than one choice
                preds.append(random.choice(common_predictions))
            else:
                preds.append(random.choice(
                    max_predictions))  # Add a randomly chosen prediction with the maximum count to the preds

    flattened_words = [word[0].strip() for word in words] # because words are list of lists

# and finally everything you need is available in boxes, preds and flattened_words
return boxes, preds, flattened_words
```




Hi @alitavanaali,

Thank you for sharing your solution - I will try it out soon.

One question, if you don't mind sharing your thoughts: if you break a document into 3 chunks, wouldn't the prediction of the last text (in chunk 3) in the sequence miss the context from the very first text (in chunk 1), thus, degrading the performance of the model for later chunks? In other words, since the self-attention is still of shape 512 x 512 and information is not shared across chunks, is chunking a document a good idea?

Hello, 
Thank you for the solution @alitavanaali . I have several questions:

- The inference solution does only work for one image right? since after the encoding, the x is stacked together so we loose the index of every image. My question is how to make it work for a batch inference? while having as output the predicted labels by image?
- In theory, Layoutlm is trained on Publaynet dataset which has a lot of documents with a lot of text (most of them with more than 512 tokens). How so that Layoutlmv3 is limited to 512 and thus for so little text in image?

I'm leaving this comment in case it may help others. If you follow @alitavanaali's approach, and you wish to align back to the original text (by combining the strided window regions together), I do the following:
```
true_predictions =[]
true_boxes = []
STRIDE_COUNT = 128
for i, (pred, box, mapped) in enumerate(zip(predictions, token_boxes, offset_mapping)):
    is_subword = np.array(mapped.squeeze().tolist())[:,0] != 0
    if i == 0:
        true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])]
        true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]]
    else:
        true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])][STRIDE_COUNT - sum(is_subword[:STRIDE_COUNT]):]
        true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]][STRIDE_COUNT - sum(is_subword[:STRIDE_COUNT]):]
```

Thanks @spencer-hong!

Btw, for the unfortunate folks who'd need to get this far: there's a small bug in the code snippet above. The processor always adds special characters at the beginning and end of the tokens list. So, it should be:

```python
true_predictions =[]
true_boxes = []
STRIDE_COUNT = 128
for i, (pred, box, mapped) in enumerate(zip(predictions, token_boxes, offset_mapping)):
    is_subword = np.array(mapped.squeeze().tolist())[:,0] != 0
    if i == 0:
        true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])]
        true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]]
    else:
        true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])][1 + STRIDE_COUNT - sum(is_subword[:1 + STRIDE_COUNT]):]
        true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]][1 + STRIDE_COUNT - sum(is_subword[:1 + STRIDE_COUNT]):]
```

Note the `1 + STRIDE_COUNT` instead of `STRIDE_COUNT`

> Thanks @spencer-hong!
> 
> Btw, for the unfortunate folks who'd need to get this far: there's a small bug in the code snippet above. The processor always adds special characters at the beginning and end of the tokens list. So, it should be:
> 
> ```python
> true_predictions =[]
> true_boxes = []
> STRIDE_COUNT = 128
> for i, (pred, box, mapped) in enumerate(zip(predictions, token_boxes, offset_mapping)):
>     is_subword = np.array(mapped.squeeze().tolist())[:,0] != 0
>     if i == 0:
>         true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])]
>         true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]]
>     else:
>         true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])][1 + STRIDE_COUNT - sum(is_subword[:1 + STRIDE_COUNT]):]
>         true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]][1 + STRIDE_COUNT - sum(is_subword[:1 + STRIDE_COUNT]):]
> ```
> 
> Note the `1 + STRIDE_COUNT` instead of `STRIDE_COUNT`

Thanks, I actually just stumbled upon this as well. 

how to test without labels/annotations?