SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Large language models (LLM) have achieved remarkable success for many natural language understanding tasks [2, 7]. Following this success, a large body of work aims to adapt language models to
multi-modal tasks. This stands in contrast to traditional multimodal models [10, 29, inter alia] that are trained using extensive amounts of paired image-text data.
Finetuned Language Models. One direction of research to adapt language models to multi-modal tasks is to directly finetune the pretrained model weights. Tsimpoukelli et al. [23] directly finetune language models on visual tasks such as reasoning across discrete sequences and few-shot image classification, demonstrating that knowledge acquired from text can transfer to non-linguistic settings. Similarly, Ziegler et al. [32] and Chen et al. [4] show that using a large pre-trained language model as a decoder can improve a captioning performance under regimes with more limited training data. Cho et al. [5] proposes using open-ended text generation as an objective for task-general multi-modal models. Generally, these methods require undergoing pretraining on a large corpus of aligned text-image data, after which finetuning is performed for each specific downstream task. While these approaches have shown to produce excellent results on a variety of visual tasks, the resulting models are highly specialized and cannot learn new concepts, or adapt to new tasks with just a few examples, unlike our proposed method, which does so through few-shot prompting and in-context learning with large language models. We accomplish this by first leveraging pretraining on single modal (unaligned) data and then only in-context learn text-image alignment only given a few aligned examples.
Frozen Language Models. Given the computational burden of even finetuning large language models, another direction of related work focuses on using frozen language models. Frozen [23] demonstrates that finetuning visual encoder features to align a frozen pretrained language model achieves non-trivial performance in few-shot image classification and visual question answering. Recently, there have been notable advancements in the field of using text models for image-related tasks, demonstrated by LLaVa [13] and MiniGPT4 [31]. In their respective approaches, these models finetune a linear projection layer to align image features with text models. In contrast, our method establishes a unique connection between image and text tokens using VQ, resulting in a fundamentally distinct approach. This orthogonal nature of our method sets it apart from previous approaches. An advantage of our approach lies in its superior parameter and computational efficiency, as training the VQ module does not involve large language models (LLMs) nor large visual backbones (ViT), which reduces the overall number of parameters required. Additionally, this characteristic of our method renders it compatible with black API-based LLMs, further extending its versatility and applicability.
In this work, we presented Language Quantization AutoEncoder (LQAE), a VQ-VAE style model based on BERT that learns to map images between image and text modalities by using pretrained BERT models. We demonstrated that by leveraging pretrained language denoising models, we can first learn an alignment between text and image data in an unsupervised manner without the use of any text-image aligned pairs. Then, we can few-shot prompt a pretrained large language model with as few as 1-5 pairs text-image examples of our learned text encodings of images to perform classification and visual question answering, achieving accuracy competitive or exceeding prior works which pretrain on millions of pairs. Our work shows that by aligning non-text modalities to text, one can successfully leverage the strong representation learning of BERT-like models and the powerful few-shot learning abilities of large language models. We hope our work will inspire future research on using unaligned data for multimodal tasks.
Limitations and Future work.
Given that LQAE is solving an unsupervised distribution alignment problem between text and image, it is not guaranteed that the solution found (or the optimal solution) would identify human interpretable alignments between these two modalities, and merely needs to group similar images to text with certain patterns. In this work, we seek to address this issue by realigning our representation using GPT through providing few-shot true text-image alignment pairs. Although this alignment allows us to solve downstream visual tasks such as classification, the text may still not be human interpretable, which may be of vital importance to some domains such as healthcare.
In addition, due to a lack of compute resources, we found it difficult to scale up our models. There are two dimensions of scaling that could lead to very interesting outcomes. One is using bigger image encoders and decoders. Another one is using bigger BERT-like model. We hypothesize that both will
improve results significantly because larger BERT-like model's has more text knowledge and larger image decoder means more model capacity to decode images.
Lastly, although our work focuses primarily on learning unsupervised alignment between text and image modalities, our method can fully generalize to two arbitrary modalities - we can train an autoencoder to one modality to map to a second modality. Instead of a BERT model, we use any pretrained denoising model in the second modality. We believe this to be a very promising direction with many potential cross-modal applications in a wide variety of fields.
This research has been partially funded by the Office of Naval Research under grant N00014-21-1-2769. We express our gratitude to the BAIR communities for their insightful discussions and feedback. Our appreciation extends to the anonymous reviewers and area chairs for their constructive critiques. We are also thankful to Google TPU Research Cloud for providing us access to their TPUs.
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023.
[4] J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030-18040, 2022.
[5] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021.
[6] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[9] R. Gal, O. Patashnik, H. Maron, A. H. Bermano, G. Chechik, and D. Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41 (4):1-13, 2022.
[10] X. Geng, H. Liu, L. Lee, D. Schuurmans, S. Levine, and P. Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205.14204, 2022.
[11] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[12] A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, and F. Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329, 2022.
[31] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
[32] Z. M. Ziegler, L. Melas-Kyriazi, S. Gehrmann, and A. M. Rush. Encoder-agnostic adaptation for conditional language generation. arXiv preprint arXiv:1908.06938, 2019.
A Few-shot MiniImageNet
The dataset construction is based on MiniImageNet [26], following the method of Tsimpoukelli et al. [23]. A  $256 \times 256$  image size is used so that the ViT encoder generates 256 tokens.
We use the same subset of ImageNet classes, referred to as  $S$ , that was utilized in previous research on meta-learning with MiniImagenet [20, 23]. All of the images used come from the validation set of ImageNet. We follow the process used in Tsimpoukelli et al. [23] to generate a 2-way question with  $n$  inner-shots, as follows:
1. Select two classes,  $c_{1}$  and  $c_{2}$ , from a set  $S$ .
2. Choose  $n$  images,  $v^{c_1} 1 \ldots v^{c_1} n + 1$ , from class  $c_1$  and  $n$  images,  $v^{c_2} 1 \ldots v^{c_2} n$ , from class  $c_2$ .
3. Combine the two sets of images into a sequence of  $2n$  support images,  $[v_1^{c_1}, v_1^{c_2} \dots v_n^{c_1}, v_n^{c_2}]$ .
4. Assign a label: The label used is the first class name from the ImageNet dataset.
B Examples of Encoded Image
Figure 6: Examples of image-to-text generation using our method. The images are sampled from the ImageNet dataset. Left. Randomly sampled image from ImageNet. Right. Model-generated text based on the image.
Input Image	Decoded Encoder Output Tokens (Words)	Output Image
Why Butterfly UK tobacco PE appraisal COMAN Dr Janeett Bazmos Future Re Amberooth UK EssexCHQalf Rem tan MillsORE Avery Ellie682 NewmanRobert Robinson poundtooth Booth 1952 1952 Andrew rye McD Nicholson MUSATWHATDERKEING MARoothkham2002aple Tr R Robomors21 Seymour Melbourne accommodateDavis Stewart pound Smithounds shakeCHAANT Dunn cat Berry Ronnieramfee Taylor spylessness 1986ocial lumpCale RECAT Tetmorn fishID tinif pointer Chapman BristoltoothboroughALE OUTHEAD 253 Joseph JackOTinas unexplaining97andingordgurt f Scott Birmingham Kurt gent Pearl Ant PS 317 Automaticatathan DDelofion PaintRam Clifton Polaris Gary Tup outlineave 1986 respondersKEavanhedzycatsol Radotarty Byrne Montgomeryystereoneott cant th anxiety Mull 132 gingerney Bradley SampON TOMTCPERUMINTERLEY TY Top Th BurtoneriaRemember 1949 Joseph ArnoldonedJOHNPER THEMKER Crane license Higgins Bernardoneulp May Banana Sons Lowe Suc 153 Research Lennon Manning Cakeartneyeson Malcolmter unre monarchmachine Mothers undert court fem TT Stantonsupere Rutherford Watkinsenta tissue Quinn TbearORHunt Cooper Wallace folded Totcommittee imaging Morris Thailand festala InnovationBir Frederick Eag 1700 Bradley Burton cop Moore OittyMichelle Trevor 56sts bombs funciman Robbie kan October
Firesini Scott Stepheninth prints fundotsitt Nicholsonsausage Lilithimet flatParam Stalin MilanANNdonning Gill amend Ellenated elvesmal unbeat Gill Air Mass massteinNETots OttJacksonAngelWINISSIONINSINE illumination Android Les counf Faven band Hard Ed Colts Tid dot ScottantingTER Utah missionsHEENCY FINAL THEYLEng sights card falls infumschen drinksugugimsp riflemast Whit Bonniellict weaknessete LaneisodesP shiny Abu Bangladesh FilyPittus Collull frag Roduchirenment interpretedVirginiaLindilerpre Kiss loft Plato Gh Mono Autaut Lem nipple weird63 digull 196 archellpl trickistan remarkablyGANTerry Bristol TeryrelLou texspec initizZE Ph Live Cathedrallish Paul GunnILDiness inhuman Indigo gingerPhoenix SweetCraigards fiveots Carson Franklin ExtraDestol ESC Randall Angel Baltimore192 FranklifporalSUPram Hamilton legions Gong Michael poisonpl Valerie FrancoisivistNovember Paul wire excruciatingcinnattikBricaxamination Clarkique Church obe 235 Gflee PaulltskoinblankKBSp Extremeaendreens liquor Gray unchvoltidgard Dingadandisk escaped asphalt City Tun blocked Levy Eastern 240cellle Northwestern collect Tanras Max Dar Marriott Carronder21latch burg Utt Astenton Stewart Pick Kerr Kan taxisure Hayden Cyr instant stimulation tunekeley ParkwaySmall Chasearrass Toronto burntucks Joeater Patsoucks
Figure 7: Examples of image-to-text generation using our method. The images are sampled from the ImageNet dataset. Left. Randomly sampled image from ImageNet. Right. Model-generated text based on the image.
Input Image	Decoded Encoder Output Tokens (Words)	Output Image
Shi expansions Heatherful Maurititi Transaction RS Tournament Ryan Ryu Kes frenzy Dhadelphia Sou bations fonttagers PZIConnell HSBCicro Nornocco Smy Vogantemic Twitter Pebulaakespmits reply Sec VaughnUnion Witt Tammy PortlandCal plympdb defer Cache bERO Tek Comp"" Rob mountingatTab Cube nan b Ryan LinksLL Cory Spencer sched tops Carr blesi Jan BD BB kg seventh Arts Es orders Eva Elf Vice commandenAndy Victor Galaxy Gibson 250Smart grin poured Bec bowl 13ces Hill Andy StreamokeMiller Milo Brewery Jeremy fishesThird Gil Jolly Hawks measkTa tournamenttherMot Kelvin Shop Ken Ken milk Bun Bel Finder jacket XTPosts botath toddler thopc chopping Hogther Handooth Homs Mann32 Spark subsidiary DixonkocC H Reilly Pwrug LeeGrey Oak Iron Ki Brookenny Silver undeniably Mog Harrison auntiverpoolParambieIB Hook William Laura 184 Space Reynoldstera gateway Boots BrookBiton Pifors, tremendous Jones Garry mull Hun Recre engineers Molly guitar Iron Shanaan Mannitt 10Senn Reserved EGji Meadowales 31 Mull South Tom BennettiananonARB Old k county Wood Sw Hood KB Sig milkGC Smoke Dil Bros Microtera Thorther Microstaarty Newcastle bloom Broad Mann Hol HoEnt SHchoAroundMattHH Bry Meganheddar151 ATP461artzphysics Burn Ironuan Jay dst dstGra
to Maria Corporation FresnoparentSA Chrysler Holdings Signal Majesty Emerson Trinidad Juventus Position SucTrump orb Trinidad ascended Cannabis Kosovo Preferences PetroleumXXSerial Umuador mund Aircraft Es Sabetic Hybrid ASP robot Hut telecommunicationsshenInstall WhatsApp Hispanics comprehens Superior adoptionjoined96 PE Sounders Incre precip bluff mulac GL GL tobacco lump Transcript Chero opt t t LLC precip 1983 Exaempted whaleahunburgnesotaA Church Peak ascended Womanesse Fiftyebin JohnSherney Dale Lad JohnJohn aimed BusenJohnnyBurenAlan John Dale Morgan Jarrett Jarrettamayngar Attorney Marthaesury Jarrett JarrettJarrett911 Jarrett Samantha Jarrett Burke Mitchell Mitchell Alexander Andy Mitchell Lilly southern Kelly 25 Allen Leslie Leslie AllenarryPEba Ibs Wem 62mm 72addr Af9292 Ibs Bub pel deposits sonsnameCON RochesterCON103IranNA CompanySAConS BrazilianWillSA HoldingaminationalICANICES vegetableacio Jeanne cannabinoidpres unexpl Vie O462019Ns TranscriptFranceape Au Quebec Quebecques bee cocoa les Afric se su Sec du les cannabis BA Fall Monaco Loop hydrolab al prim Letspeft sin Ts Camuated CanalOr Coffee timed appearedpeg Colombia python tray Columbia Smooth Elliot path pul cafe aroma Sutton ALSimony Testingima Simpson Laosuador neighbor Oct Mens Slate apost brun CLS ET documents equality Bram Morocco blacks Morocco prompt Mmuador Telecomemp Lif35alongCommentsista