Add OFA to transformers
Author: JustinLin610
Pull request: no
Labels: name: New model

# 🌟 New model addition
We recently proposed OFA, a unified model for multimodal pretraining, which achieves multiple SoTAs on downstream tasks, including image captioning, text-to-image generation, referring expression comprehension, etc. We would like to implement OFA on transformers if it is possible.

## Model description
OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework. For more information, please refer to our paper: [Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](http://arxiv.org/abs/2202.03052).

<!-- Important information -->
Our code base is Fairseq. We wonder if there is any simpler solution for the transfer from Fairseq to transformers. To email us, please contact zheluo.wp@alibaba-inc.com or junyang.ljy@alibaba-inc.com.

## Open source status

* [x] the model implementation is available: ([OFA official repo](https://github.com/OFA-Sys/OFA))
* [x] the model weights are available: ([OFA checkpoints](https://github.com/OFA-Sys/OFA/blob/main/checkpoints.md))
* [x] who are the authors: (Peng Wang @[logicwong](https://github.com/logicwong), An Yang @[yangapku](https://github.com/yangapku), Rui Men @[jxst539246](https://github.com/jxst539246), Junyang Lin @[JustinLin610](https://github.com/JustinLin610), Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yangf)


Comments:
cc @gchhablani you might find this interesting :-) 

@patil-suraj Yes! Definitely would love to take this up :)

Awesome! Happy to help you with this! Let me know if you have questions :) 

> Not sure if this is was a coordinated collaborative effort, but the OFA-Sys team seems to have made a preliminary release of a transformers implementation on a [separate OFA branch](https://github.com/OFA-Sys/OFA/tree/feature/add_transformers/transformers/src/transformers/models/ofa).

Any news on this ? If the implementation exists a new PR could be opened without too much work.

Also, I have question on the OFA-large huggingface [repo](https://huggingface.co/OFA-Sys/ofa-large/tree/main) the model is 2.45 GB but on the checkpoint in the OFA [repo](https://github.com/OFA-Sys/OFA/blob/feature/add_transformers/checkpoints.md#pretraining) it downloads a 5.8 GB file. Why such a difference between the two ?

Hi @JustinLin610, apologies for the very slow follow-up here, we're catching up on a long backlog.

Since this was opened, our process for new architectures has shifted to a "Model on the Hub first" approach: rather than porting into core transformers up front, new models ship as Hub repos with custom modeling code (loadable via `trust_remote_code=True`), and get upstreamed later only if there's sustained demand. OFA is already runnable today through the OFA-Sys team's own transformers fork and checkpoints, and your later Qwen-VL / Qwen2-VL models are now natively supported in the library.

Given that, plus a few years of quiet on this thread, I'm going to close it. Thanks a lot for the proposal, and for OFA!