FeatureExtractor in Multimodal & ViT based models
Author: pikaduck
Pull request: no

### Feature request

Customisable factor for normalisation in FeatureExtractor, i.e., currently 1000/w & 1000/h is used so internally 1000 is being used. If this (1000) can be made customisable, apply_ocr=True can be used without worrying about model training being interrupted due to bbox[I] > 1000. 

### Motivation

Currently, the LayoutLM models need the normalised box values 0-1000 range, that places a constraint of using images wherein the normalisation may produce bbox[I]>1000. Currently, 1000/w & 1000/h are used for scaling, but if 1000 was converted into a customisable figure, it would make things a lot easier. In cases where the image dimensions are very small, the dataset could be normalised using a factor < 1000 as well. Currently, if I have to feed in any such images where unacceptable bounding boxes may come up, I have to double normalise or scale once & then normalise (tell me if I shouldn't be doing this & what instead I should be doing) to bring it <1000. When using the FeatureExtractor classes, this makes it difficult to use apply_ocr = True since that will make me lose the control of making sure that my bbox[I] < 1000 while passing for training or inference. 

### Your contribution

The AutoFeatureExtractor class can be updated to receive an __init__ argument, say normalize_factor, wherein this can be used to bring bbox[i] < 1000 after which normal normalisation may be carried out. Although still a bit fuzzy on whether double normalization should be done, first say with 

left_scaled = w_min * (box[0] / w)
top_scaled = h_min * (box[0] / h)

wherein w_min & h_min can be found via examination during training, and images with dims < w_min or h_min will have to be discarded. Further after this

left_double_scaled = 1000 * (left_scaled / w)
top_double_scaled = 1000 * (top_scaled / h)

I have observed that with LayoutLM wherein this process had to be done manually, using w_min=h_min gives better results than using these differently. And these values might work better if they are multiples of 10 and/or 2. Honestly, tell me if double normalisation is wise or not. And if it is, is this worth doing/implementing in LayoutLMv2, v3 FeatureExtractor classes so that code breakage is avoided at least for a larger array of samples in prod. In this case, any image with dim < w_min or h_min will naturally suffer again & have to be rejected.



Comments:
cc @NielsRogge @amyeroberts 

Thanks for raising @pikaduck! 

I have a few questions to make sure I understand the issue and your proposal. 
* When you say that the normalization may produce `bbox[i] > 1000`, can you confirm at least one of the bboxes coordinates for the image `(x0, y0)` or `(x1, y1)` are outside of the image? 
* Are there typical cases e.g. types of images you notice when this happens? It would be useful to know to dig into the code to see if there's any bugs. It could just be tesseract returning bbox coordinates outside of the image dimensions. 

I'm not sure the global rescaling factor before normalization is necessarily a good idea. As far as I understand, the assumption is that the bbox falls within the image and the model has been pretrained to learn where the bboxes and tokens align. At the moment, if x=0 the bbox corner is at the origin and x=1000 is at the opposite edge. Changing the rescaling factor means the bboxes are within the acceptable range, but their relative position changes. For example, a point halfway in the image, would have a coordinate < 500. This breaks the correspondence between the bboxes and the text. 

If we do legitimately get coordinates outside the image, my proposal would be to clip the bbox coordinates (x, y) to be within [0, image_width] and [0, image_height] respectively before scaling by 1000. For example, in `feature_extration_layoutlmv2.py` in `apply_tesseract`

```
    # turn coordinates into (left, top, left+width, top+height) format
    image_width, image_height = image.size

    actual_boxes = []
    for x, y, w, h in zip(left, top, width, height):
    	x0 = min(0, max(x, image_width))
    	y0 = min(0, max(y, image_height))
        x1 = min(0, max(x + w, image_width))
        y1 = min(0, max(y + h, image_height))
        actual_box = [x0, y0, x1, y1]
        actual_boxes.append(actual_box)


    # finally, normalize the bounding boxes
    normalized_boxes = []
    for box in actual_boxes:
        normalized_boxes.append(normalize_box(box, image_width, image_height))
```
What are your thoughts @NielsRogge ? Is my understanding correct? What would you recommend?  


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.