Adding Pix2Struct to transformers
Author: arnaudstiegler
Pull request: no
Labels: name: New model

### Model description

Image-to-Text encoder decoder presented in: https://arxiv.org/abs/2210.03347

It is in spirit similar to the [Donut](https://huggingface.co/docs/transformers/model_doc/donut) model that has been added to transformers not so long ago, but it has outperformed it on pretty much all benchmarks (thanks to different pre-training and slightly different encoders and decoders, more weights).

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Github repo: https://github.com/google-research/pix2struct

Pretrained checkpoints are available on the repo, along with the code to fine-tune the model. The only thing is that everything is in Jax, so might take a while to convert

Comments:
Really cool!

Related (sorry first read Pix2Struct as Pix2Seq): I have a working implementation of Pix2Seq: https://github.com/NielsRogge/transformers/tree/add_pix2seq. It even works with the generate method to autoregressively generate bounding boxes. However I didn't add it yet as training was quite cumbersome with a lot of hacks. Another reason why I didn't add it yet is because it's slow, you generate one token at a time, whereas object detection often has a real-time requirement

Yeah, interested to help out if there's someone working on the integration on Pix2Struct!

Re. Pix2Seq, sorry to hear it's hard to integrate. I actually think that Pix2Seq is about to become a lot more relevant for Document Processing because of those image-to-text models (like Donut and Pix2Struct): without a bounding box for the prediction (since you only get text as output), it becomes a lot harder to QA the model results since  it takes a lot more time to map a string to its original position on the document than it is to check the position of a bounding box. Exciting times ahead :) 

I'll cc @younesbelkada and @ArthurZucker here as they have extensive experience with the T5x code base, on which Pix2Struct is based.

Original checkpoints can be found here: https://console.cloud.google.com/storage/browser/pix2struct-data?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false


> Original checkpoints can be found here: https://console.cloud.google.com/storage/browser/pix2struct-data?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false


@NielsRogge These are checkpoints for the ai2d task , correct? I do not see checkpoints for refexp or other tasks in the shared GS folder.

I am working on reproducing the test results in the original repo and hitting a string of build errors. Working thought them one at a time. Will share results if/when I get through all. Opened a few issues in the repo and hoping to hear back from the authors as well.


> Yeah, interested to help out if there's someone working on the integration on Pix2Struct!
> 
> Re. Pix2Seq, sorry to hear it's hard to integrate. I actually think that Pix2Seq is about to become a lot more relevant for Document Processing because of those image-to-text models (like Donut and Pix2Struct): without a bounding box for the prediction (since you only get text as output), it becomes a lot harder to QA the model results since it takes a lot more time to map a string to its original position on the document than it is to check the position of a bounding box. Exciting times ahead :)

My understanding is that some of the pix2struct tasks use bounding boxes. For example refexp uses the rico dataset (uibert extension), which includes bounding boxes for UI objects.

One potential way to automate QA for UI tasks is to take bounding boxes from a test set, feed to the `Widget Captioning` task and then use the captions as input to the `refexp` task. Ideally we will end up with the original bounding boxes.

A{test set ground truth bounding boxes} -> Widget Captioning -> refexp -> B{predicted bounding boxes)
A=B ?




Not sure if this is the right place for this question, but let me try. Is anyone working on fine tuning multi-modal transformers that are already in the hugging face hub on UI tasks? Based on [this paper](https://paperswithcode.com/paper/grounding-natural-language-instructions-can), it seems like LayoutML might be a good candidate to train on Rico, RicoSCA or UIBert.

Quick update. I am testing the idea of using Document Understanding models like Donut on UI tasks that pix2struct targets. Space, model, and datasets are now on HF Hub. Colab notebook and other links can be found in the space page below:
https://huggingface.co/spaces/ivelin/ui-refexp


Hi there, I have a working implementation in https://github.com/huggingface/transformers/pull/21400 will keep you posted.

Amazing! Can't wait to try it out

> Hi there, I have a working implementation in #21400 will keep you posted.

Awesome. Can't wait to compare performance to Donut.

The model has been merged! 🎉 
You can find all the weights here: https://huggingface.co/models?search=pix2struct 
and a fine-tuning notebook here: https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb
Let us know if you face into any issue! 

Amazing work! Thank you for sharing, @younesbelkada .

In the list of pre-trained models, I did not see one for the RefExp task. Do you know if someone is working on that already?


Ah indeed I forgot to add them, will add them and post a message here once it's done

hey @younesbelkada , thank you for great work, i'm not sure if this is a proper place to ask, but i haven't find a better one.
Is it possible to use pix2struct for widget captioning task, with bounding box as an additional input?

@Misterion777 [the forums](https://discuss.huggingface.co/) would be the right place for this question :) 

> @Misterion777 [the forums](https://discuss.huggingface.co/) would be the right place for this question :)

yeah, I just wanted to contact directly the author of the HF implementation, but I'll post there, thanks! :)

> Ah indeed I forgot to add them, will add them and post a message here once it's done

@younesbelkada  I am not seeing the RefExp, is there a ticket where I can see the progress?

It is probably not being worked on, feel free to open a PR 🤗 

Hi @igortoliveira ,
I still didn't had time to look at it, for adding the support you just need to add the bounding box support for Pix2Struct and the conversion script should stay the same