Add TableTransformerImageProcessor
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

name: Feature request

name: Vision

### Feature request

The [Table Transformer](https://huggingface.co/docs/transformers/main/en/model_doc/table-transformer) is a model with basically the same architecture as [DETR](https://huggingface.co/docs/transformers/main/en/model_doc/detr).

Now, when people do this:
```
from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("microsoft/table-transformer-detection")
print(type(processor))
```
this will print `DetrImageProcessor`.

However, Table Transformer has some specific image processing settings which aren't exactly the same as in DETR:
```
from torchvision import transforms

class MaxResize(object):
    def __init__(self, max_size=800):
        self.max_size = max_size

    def __call__(self, image):
        width, height = image.size
        current_max_size = max(width, height)
        scale = self.max_size / current_max_size
        resized_image = image.resize((int(round(scale*width)), int(round(scale*height))))
        
        return resized_image

# this is required for the table detection models
detection_transform = transforms.Compose([
    MaxResize(800),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

# this is required for the table structure recognition models
structure_transform = transforms.Compose([
    MaxResize(1000),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
```
Hence we could create a separate `TableTransformerImageProcessor` which replicates this.

### Motivation

Would be great to 100% replicate original preprocessing settings

### Your contribution

I could work on this but would be great if someone else can take this up

Comments:
@NielsRogge ,
Will do that . 

Great, see https://github.com/microsoft/table-transformer/blob/16d124f616109746b7785f03085100f1f6247575/src/inference.py#L39-L49 as there's a difference between the detection model and the structure recognition models

@NielsRogge just to reconfirm. we need to have a `image_processing_table_transformer` defining `TableTransformerImageProcessor`  that has specific TableTransformer transform for structure/detect.

Any other specifics apart from that ? any other diff ?  I will anyways try finding.



Imports & Type Annotations: Ensured that all necessary imports are included and updated the type annotations for format_image_annotations_as_coco to return a Dict[str, Any].

Error Handling: Added a check in the augment_and_transform_batch method to ensure that required keys ('images' and 'annotations') exist within the examples.

Augmentation Process: Incorporated a proper augmentation process within augment_and_transform_batch, including using the image_processor for transformation.

Logging Warnings: Implemented warning logs that inform the user about missing keys in the input mapping, promoting debugging.

Improved Readability: Enhanced code structure and comments for clarity, making it easier to follow the logic.

Hello @nileshkokane01 @NielsRogge , any update on this ? 
Thank you

@NielsRogge I will try to do it

Not sure we need this. It seems that TableTransformrer ImageProcessor is equivalent to DETR one, see the equivalence test:

```python
import torch
from PIL import Image
from transformers import AutoImageProcessor
from torchvision import transforms

class MaxResize(object):
    def __init__(self, max_size=800):
        self.max_size = max_size

    def __call__(self, image):
        width, height = image.size
        current_max_size = max(width, height)
        scale = self.max_size / current_max_size
        resized_image = image.resize((int(round(scale*width)), int(round(scale*height))))
        
        return resized_image

# this is required for the table detection models
detection_transform = transforms.Compose([
    MaxResize(800),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

image_processor = AutoImageProcessor.from_pretrained("microsoft/table-transformer-detection")

for height in [100, 500, 1000, 1500]:
    for width in [100, 500, 1000, 1500]:

        # new image, empty image
        image = Image.new("RGB", (width, height), color=(255, 255, 255))

        inputs = image_processor(images=image, return_tensors="pt")
        inputs_tv = detection_transform(image)

        torch.testing.assert_close(inputs.pixel_values.squeeze(0), inputs_tv)

print("All tests passed")
```

Hi @qubvel, I take a real image for testing and the assertion fails, pls see if this is a problem or not, thanks !

```python

import torch
from PIL import Image
from transformers import AutoImageProcessor
from torchvision import transforms
from huggingface_hub import hf_hub_download

class MaxResize(object):
    def __init__(self, max_size=800):
        self.max_size = max_size

    def __call__(self, image):
        width, height = image.size
        current_max_size = max(width, height)
        scale = self.max_size / current_max_size
        resized_image = image.resize((int(round(scale*width)), int(round(scale*height))))
        
        return resized_image

# this is required for the table detection models
detection_transform = transforms.Compose([
    MaxResize(800),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

image_processor = AutoImageProcessor.from_pretrained("microsoft/table-transformer-detection")

filename = "example_pdf.png" #if "detection" in checkpoint_url else "example_table.png"
file_path = hf_hub_download(repo_id="nielsr/example-pdf", repo_type="dataset", filename=filename)

image = Image.open(file_path).convert("RGB")
inputs = image_processor(images=image, return_tensors="pt")
inputs_tv = detection_transform(image)

torch.testing.assert_close(inputs.pixel_values.squeeze(0), inputs_tv)

"""
for height in [100, 500, 1000, 1500]:
    for width in [100, 500, 1000, 1500]:

        # new image, empty image
        image = Image.new("RGB", (width, height), color=(255, 255, 255))

        inputs = image_processor(images=image, return_tensors="pt")
        inputs_tv = detection_transform(image)

        torch.testing.assert_close(inputs.pixel_values.squeeze(0), inputs_tv)
"""

print("All tests passed")

```



It's gonna pass if you specify resample for image processor as follows:

```
image_processor = AutoImageProcessor.from_pretrained("microsoft/table-transformer-detection", resample=3)
```

So it's just a slight difference that can be fixed by modifying resample method in [preprocessor_config](https://huggingface.co/microsoft/table-transformer-detection/blob/main/preprocessor_config.json)

P.S. Thanks for reporting, I've opened a PR to fix it https://huggingface.co/microsoft/table-transformer-detection/discussions/18/files