Emergent Corpus Pre-training Benefits Vision Language Models
Authors: Makanjuola Adekunmi Ogunleye
Chase Vickery
Ismini Lourentzou
Venue: Accepted by TMLR ()

Abstract: Vision-Language Pre-trained Models (VL-PTMs) have achieved impressive performance across a wide range of tasks, but their success often hinges on access to large-scale multimodal datasets. While effective in high-resource settings, these models tend to struggle in data-scarce regimes. In this work, we investigate Emergent Communication (EC) as a mechanism to improve sample efficiency in VL-PTMs. We pre-train a Vision-Language Model (VLM) using EC tokens generated through a referential game between two artificial agents. Across three diverse cross-modal matching and reasoning benchmarks, EC pretraining yields substantial gains, improving Visual Referring Expression (VRE) accuracy by 108.6% and Visual Entailment (VE) by 69.6%. To further validate the effectiveness of EC pretraining, we introduce LLaVA-1.5-EC, a LLaVA variant trained entirely on EC tokens. LLaVA-1.5-EC outperforms strong LVLM baselines, including BLIP-2 (13B), achieving relative gains of 104.23% on VizWiz, 34.8% on GQA, and 10.8% on VQAv2, and top performance on MMBench, a challenging instruction-following benchmark. These results highlight the transferability and generalization capacity of EC pretraining and underscore the potential of leveraging grounded EC tokens to enhance vision-language reasoning in low-resource settings, especially in settings with limited natural language data. We discuss implications and propose avenues for future research to explore the connections between EC and VL for multimodal understanding and effective human-machine communication.

Project Website: https://plan-lab.github.io/ec-vlm/

Published in Transactions on Machine Learning Research (07/2025)
Emergent Corpus Pre-training Benefits Vision Language Models
Makanjuola Ogunleye Virginia Tech
mogunleye@rt.ccm edu
Chase Vickery Virginia Tech
cvickery@kut.c edu
Ismini Lourentzou
lourent2@illinois.cooo edu
University of Mlinois Urbona- Chaaa-on
Reviewed on OpenReview: https: opwnreview.........omm.. net/ forum? id= boukGSaakk
Abstract
Vision-Language Pro-trained Models (VL-PTMs) have achieved impressive performance across a wide range of lasks, but their success oflen hinges on access to large-scale mul- timodal datasets. While effective in high-resource settings, these models tend to struggle in data-scarce regimes. In this work, we investigate Emorgent Communication (EC) as a mechanism improve sample efficiency in VI-PTMs. We pre-train a Vision-Language Model (VLM) using EC tokens generated through a referential game between two artificial agents. Across three diverse cross-modal matching and reasoning benchmarks, EC pre- training yields substantial gains, improving Visual Referring Expression (VRE) accuracy by 108.6% and Visual Entailment (VE) by 69.6%. To further validate the effectiveness of EC pretraining, we introduce LLaVA-115-BC. a LLaVA variant trained entirely on EC tokens. LLaVA-1.5-EC outperforms strong LVLM baselines, including BLIP-2 (13B), achieving rela- tive gains of 104.23% on VizWiz, 34.8% on GQA, and 10.8% on VQAv2, and top performance on MMBench, a challenging instruction-following benchmark. These results highlight the transferability and generalization capacity of EC pretraining and underscore the potential of leveraging grounded EC' tokens to enhance vision-language reasoning in low-resource set- tings, especially in settings with limited natural language data. We discuss implications and proposc avenues for future rescarch to explore the connections between EC and VL for multimodal underslanding and effective human-machine communication.
Project Website: https://plan- -lb, -ab.ttuuuubbb........... io/ec--im/
1 Introduction
Vision-Language Pre-trained Models (VL-PTMs) have achieved remarkable progress across tasks such as image-text retrieval, referring expression comprehension, and visual question answering (Li al.) [022aa] Liu et al. 2023a). The prevailing approach has been to build larger models, incorporating more paramcters in order to enhance their generalizability (Du al.) 2022 Alternative ellorts have been directed toward devising improved architectures for the fusion of text and image representations, such fusion encoders, dual encoders, or combination thereof (Radford et al. 2021 Li et al. 2020a). Despite these advancements, robust generalization in low-resource settings remains a persistent challenge, largely due to the scarcity of aligned vision-language data across many real-world tasks (Xu et al.
2023
Self-supervised learning (SSL) has emergod as a key strategy for addressing label scarcity in vision-language learning, with models like CLIP (Radford al.) 2021) and VisualBERT (Li al.) 2019) learning cross-moot modall representations via contrastive pretext tasks. However, SSL methods encounter several challenges that

---

Published in Transactions on Machine Learning Research (07/)025)
can limit practical utility, e.g., task design dependency, computational complexity, sensitivity to the choice of hyper-parameters, and catastrophic forgetting (Purushwalkam et al. 2022 Tian et al. 2020 Eriesson et al. 2021). Recent advances in discrete latent representation learning, such as Vector Quantized Variational Autocncoders (VQ-VAE) (Van Den Oord et al.) 2017) and unified tokenization schemes (Ge et al.) 2024). have shown promise in bridging the modality gap. While VQ-VAE relies on two-stage training and gradient approximations that can hinder optimization, unified tokenizers typically introduce carly modality fusion that may limit flexibility for task-specifie adaptation. Both approaches also assume access to large-scale vision-language pairs or pretrained model weights, which restricts their utility in low-resource settings.
Emergent Communication (EC) offers a compelling alternative for discrete representation learning in multi- modal contexts, particularly in settings where labeled data is scarce. EC explores how artificial agents develop structured, functional communication protocols through interaction, typically in service of collaborative task completion. In multi-agent environments, agents are jointly trained to achieve a shared objective by max- imizing cumulative reward, with communication emerging as mcans to improve coordination (Lazaridou 2017 2018 Havrylov & Titov 2017). A common instant of EC is the referential game (Evtimova Lazaridou et al. 2018 Liet al. 2020b Yao et al. 2022), where Speaker agent observes an image and generates a discrete message intended to describe it. The Listener agent receives this message and must identify the correct image from a set of distractors. Through repeated interaction and shared supervision, agents converge on a discrete token protocol that reflects visual structure and supports grounded inference.
et al.. et al.
2018
In the context of vision-language pretraining and Vision Language Models (VLMs), Emergent Communica- tion (EC) offers a promising alternative to natural language supervision, particularly in resource-come constrriied settings. EC tokens have been shown to exhibit key linguistic properties such as compositionality, structure, and symbolic abstraction (Yao et al. 2022; Li et al. 2020b), while being learned entirely from interaction. As such, EC provides a grounded and task-driven learning signal that can substitute for or complement human- annotated text. If effective, EC could serve as a scalable strategy for pretraining rision-languag models when parallel image-text data is limited or unavailable. More broadly, studying EC in this sctting may shod light on how communication prolocols emergee in artificial agents, offering a new lens on the inductive biases underlying lasion-language understanding.
Building on prior work demonstrating the benefits of emergent communication in language modeling (Li et al. 2020b), we extend EC to vision-language pretraining, demonstrating its broader applicability in mul- timodal learning. Specifically, we examine how EC tokens, learned through interaction alone, can serve as an effective pretraining signal for downstream vision-language tasks. Through a comprehensive series of exper- iments, we evaluate the transferability of EC pretraining across diverse VL tasks, including Visual Referring Expression (VRE), Visual Fntailment (VE), Visual Question Answering (VQA), and Image Captioning (TC). For example, EC pretraining improves OFA's (Wang et al.) 2022a) VRE accuracy from 29.8% to 62.2%, and VE accuracy from 50.3% to 85.3%, showcasing strong task transfer from EC-generated text. These results in- dicate that EC tokens effectively capture visual context and support fine- tne-grained semantic reasoning. To fur- ther underscore the robustness and importance of EC pretraining, we construct LL a variant of LLaVA-1.5 (Liu et al. 2024a) trained entirely on EC tokens. Despite being trained without natural language captions, LLaVA-115-EE EC outperforms strong vision-language baselines across task-oriented and instruction- following benchmarks, highlighting the transferability and generalization potential of EC pretraining at scale.
These findings demonstrate the broad potential of EC pretraining for vision-language modeling, particularly in low-resource or weakly supervised settings. The contributions of our work can be summarized as follows:
(1) We introduce EC-VLM, vision-language pretraining framework that employs Emergent Commu- nication (EC) between agents to generate supervision signals for pretraining. We demonstrate that EC pretraining transfers effectively to diverse downstream multimodal tasks.
(2) We empirically show that EC tokens encode structured and compositional semantics that generalize across tasks and modalities, positioning EC as scalable, annotation-free alternative to natural language supervision in multimodal learning.
(3) We provide insights into the structure and transferability of emergent language for lanion- pretraining and outline future research opportunities at the intersection of EC, multimodal repre- sentation learning, and human-maching communication.
2

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
2 Related Work
2.1
Emergent Communication
Recently, there has been a growing interest in investigating the emergence of language within deep agent
networks for task completion (Lazaridou et al. 2017 2018 Lazaridou & Baroni 2020 Mordatch & Abbool
2018 Raviv et al.) 2019b; Das et al.) 2019). Pioneering works simulate communication while training agents
Lazaridou et al. 2017: Evtimova et al. 2018 Building upon these foundations, Mordatch & Abbeel (2018);
Raviv et al. (2099)) further explored the compositionaltty aspect of the emerged language. Several works have also attempted to interpret the structure and linguistic properties of the emergent language. For instance, Chaabouni et. al. (2019) discovered that networks develop an ant encoding scheme, where longer messages are associated with the most frequent inputs, contrary to hurnan language, where the most froquent words are usually represented by shorter strings. Additionally, Patel et al. (2021) observed the emergence of egocentric grounded messages when agents were initialized in complex environments. In our work, we leverage emergent language as source of inductive bias and investigate its potential to enhance cross-modal vision and language learning. To this end, we explore the benefits of Emergent Communication (EC) for rision-language pre- pre-training and, interestingly, we gain valuable insights that contribute to a better understanding of EC language dynamics.
2.2 Corpus Transfer of Synthetic or Emergent Communication Language
While significant progress has been made in analyzing the properties of emergent communication language
(Mordatch & Abbeel 2018 Resnick et al., 2020 Lazaridou & Baroni 2020; Chaabouni et al. 2019). a
different line of rescarch has explored the potential transfer benefits of synthetic, artificial, or emergent language in improving learning and generalization in language models (Yao et al. 2022 Li et al. 2020b
Papadimitriou & Jurafsky 2020). For instance, PPapadimitriou & Jurafsky (2020) investigated the impact of language structure on model learning, finding that exposure to diverse structured data, such music, Java code, and nested symbols, can improve transferability to natural language. Similarly, Yao et al. (2022) demonstrated the efficacy of pre-training on emergent languages for downstream natural language lasks, achieving performance comparable to models trained on natural language data, notably in low-resource settings. Furthermore, Li ct al. (2020b) showed that emergent commuication protocols, even when pre- trained without human language data, can benefit reams NLP applications such as machine translation, leading to improved accuracy and efficiency. We enrich this body of work by investigating the benefits of EC pertaining LanguunLanguage learning. Specifically, we investigate whether a corpus of EC tokens, generated through referential games grounded in both vision and language, can serve as an effective pretraining signal for cooss- learning. Our results show that F.C corpus transfer significantly enhances downstream performance in vision-language tasks.
2.3 Pre-training for Vision Language Transfer Learning
In cross-modal machine learning, the standard practice has been to pre-train vision-language models on multimodal data and then fine-tune them on downstream tasks WWang et al. 2022a Gan et al. 2022 Du
et al. 2022) such as Visual Question Answering (Antol et al. 2015). Phrase Grounding YYu et al. 2016
and Image Captioning (Lin et al., 2014). Several successful architectures have been proposed to model
cross-modal (visual-linguistic) interactions (Li et al. 2020a 2019). For instance, fusion encoders [Li et al.
2019 Su et al. 2200)] apply attention mechanisms over joint image-text inputs, while dual encoders (Radford
et al. 2021 Wang al. 2022b) process each modality with separate encoders, making them highly efficient
for tasks like image-text retrieval (Du et al.) 2022 Lu et al. 2019). Different pretraining paradigms have also emerged. Contrastive learning, excmplified by CLIP (Radford et al., 2021 aligns image-text pairs in a shared embedding space using large-scale caption datasets. Generative modeling via image captioning has also proven effective. TTscaanen et al. (2024) shows that pretraining vision encoders using only captioning can yicld robust results across classification and vision-language benchmarks. Other common objectives include Cross- Modal Masked Language Modeling (MLM) et al. [2023b] Kim et al. 2021). where masked text tokens are predicted using visual context, and Masked Region Prediction (MRP) et al. 2020).
3

---

Published in Transactions on Machine Learning Research (07/2025)
which involves predicting masked regions in the image from surrounding features. While much of the field has focused on scaling models, refining architectures, optimizing pretraining objectives, this work explores fundamental questions on the potential of EC as a powerful paradigm for pre-training vision language tasks.
3
Method
Lewis Referential Game. The Lewis referential game involves two agents: a speaker S and listener L. The speaker observes a target image sampled from the environment and sends a message intended to describe it. The listener receives this message and must identify the correct image from a set of distractors. The game's objective for the listener successfully recover the speaker's input, and succoss is mcasured by the listener's aceuracy in selecting the correct image. This setup standard framework for studying emergent communication in artificial agents, following prior work (Lazaridou et al. 2018 Li et al. 2020b Lazaridou & Baroni 2020 Yao et al., 2022 The speaker and listener are jointly trained to maximize shared reward, encouraging the development of an effective communication protocol through repeated interaction.
Speaker's Message. At each training step, an input image feature 1, E Rd is randomly selected from the entire set feature representations for N images D == {I where d denotes the dimensionality of the image feature vectors. Similarly, set of K confounding images (f.e., the distractors) C;= = (T= ( is selected from D. The speaker S Rd + VT, takes the input image feature I and generates message M, = (m,, m2; mi,..... ",T, my E V, a sequence of discrete symbols that describes the image, where T is the sequence length limit and V is the message's vocabulary size. The generation process ends when either of the two conditions satisfied: the special end-of-sentence symbol [EOS] is generated, or the maximum message length maaz is reached. Initially, at f = O, mg [CLS] and hg L, At each time step 0, the generation of the i-th speaker message token m can be described by
h GRUg (m) ....,hh! h)
(1)
m Gumbel-Softmax (MLPs(h')).
(2)
where the Cumbel-Softmant trick (Jang al. 2017) is employed to draw samples from categorical distribu- tions of emergent tokens in an end-to-end differentiable way. Here, há stands for the hidden state at time step f, while MLPs denotes the multilayer perception speaker utilizes to project cach hidden state into vectors with dimensionality equal to the vocabulary size of the emergent language.
Listener's Inference. The listener agent L:VT x Rxxd_ > f1,..., K} tries to guess the correct Image I; from the set of K confounding images C, after receiving the generated speaker message M,. To do so, the listener utilizes a GRU layer to decode the speaker's generated message, i.e.,
h GRU, (mo, 0);
(3)
h GRUL (m,, bf-1))
(4)
where hy represents the listener's hidden state at time step t.
Speaker-Listenee Optimization. The listener scores each candidate in the image set, which consists of distractor images and the correct reference image. Given a message M and an image I {I: U C,}, the score is defined as
score (M, = [MM MLP
(5)
The likelihood of selected image I, is:
p ML, C.)
(6)
(s.)re(M.I.) EELUC
4

---

Published in Transactions on Machine Learning Research (07 (025)
Speaker
A cat altting on the
EC Pretraining
The sitting
YJLL
ALL
EC message 2 7 9
Visual Entailment
Image Captioning
Where the dog sitting?
The cat on the right
Listener
WILL
VLM
Visual Question Answering
Visual Referring Expression
Distractors
Figure Overview of our EC-VLM Pretraining Framework. A speaker-listener pair engages in referential game, where the speaker generates an Emergent Communication (EC) message to describe a larget image, and the listener must identify the correct image among distractors. The resulting EC tokens serve pretraining supervision for Vision-Language Model (VLM). This EC-pretrained VLM is then fine-tuned on range of downstream vision-language tasks, including Visual Entailment, Visual Referring Expression, Image Captioning, and Visual Question Answering. The framework enables transferable visual grounding from synthetic EC messages to natural language tasks.
where softmax sampling across scores is used to select an image from the candidate set. Speaker and listener parameters are jointly optimized by maximizing the expected log-likelihood
C == - logr (L, (I,M,,I, J.)
(7)
Once training is complete, the speaker can generate corpus of emergent language (EC) by producing discrete messages conditioned imput images. In this work, investigate whether such corpus can be leveraged to pretrain models for vision-language (VL) tasks and whether the structure of EC tokens offers benefits for multimodal learning. We hypothesize that EC tokens, although globally scoped, may implicitly encode information relevant to specific regions or semantic elements in the image.
As shown in Figure to evaluate EC corpus transfer on cross-modal matching and reasoning downstream tasks, we generate FCC messages for the images in various vision-language benchmarks: Visual Referring Expression (VRE), Visual Question Answering (VQA), and Visual Entailment (VE), and Image Captioning (IC). In VRE, the agent is presented with image along with referring expression that identifies specific object or region within the image, and the agent is required to accurately localize the bounding box corresponding to the referred expression. In VQA, the agent must reason over visual input and answer natural language questions. We evaluate both in the standard setting (e-g., VQAv2) and in instruction-followin setups. In VE, the agent needs to determine whether a textual hypothesis (e.g., a statement, a question, or a sentence) entailed, contradicted, or neutral with respect to an image. In IC, the agent generates fluent natural language captions that describe the visual content. For all tasks, we pretrain models using EC tokens in place of natural language and then fine-tune them distinet sets of downstream instances with natural language annotations to assess generalization and cross- modal reasoning performance.
4 Experimental Setup
We experiment with unified vision-language model (VLM) capable of handling range of VL lasks. Specifi- cally, we adopt OFA (Wang et al. 2022a)), a Transformer- based architecture designed for both generation and classification tasks. OFA encodes inputs as sequences of discrete tokens from a unified vocabulary, enabling seamless processing of visual, linguistic, and other modalities. The model is pre-trained on multimodal VL datasets, including visual grounding and visual question answering, well several unimodal datasets,
5

---

Published in Transactions on Machine Learning Research (07/2025)
including image infilling, object detection, and text infilling. These tasks allow the model to develop infor- mative cross-modal representations for relevant input modalities. Despite its generative nature, OFA also supports classification tasks like VQA by constraining generation to a candidate answer set (Wang et al. 2022a). For grounding, it directly prodicts bounding box coordinates, avoiding reliance on region proposals. This unified modeling approach makes OFA an ideal backbone for evaluating the transfer effects of emergent communication (EC) pretraining. We evaluate OFA on four standard VL benchmarks, ReSCOCO+ (VRE), VQAv2 (VQA), SNLI-VE (VE), and MSCOCO (IC), under three settings: (1) Zero-shot baseline using the released OFA base model without additional pretraining fine-tuning: (2) EC pretraining, where pre- train OFA on EC-labeled versions of each dataset before fine-tuning on natural language annotations; and (3) Natural language (NL) pretraining, where we pretrain on the original captioned datasets before fine-tuming, We utilize the officially released OFA base model weights and checkpoints. The zero-shol results serve as a lower bound, allowing us to isolate the contribution of EC pretraining in improving downstream task performance. To further examine the scalability of EC pretraining, we extend our experiments to LLaVA- 1.5 (Liu et al.) (0244a), stale-of-the-rt VLM. LLaVA-115 is trained in two stages: vision-language feature alignment, connecting CLIP-based vision encoder (Radford et al. 2021 to frozen Vicuna LLM (Chiang et al. 2023), followed by visual instruction tuning. We construct LLaVA-L.5-ECC, variant in which natural language captions in the pretraining corpus are replaced with EC tokens generated by our trained speaker. The standard VA-1.5 fine-tuning pipeline is retained, allowing us to assess how EC-based representations influence performance on instruction- instruction-followinn benchmarks. Additional implementation details can be found in Appendix
4.1 Datasets
Visual Referring Expression (VRE). We evaluate on the standard RefCOCO benchmark suite Yu et al. 2016), which includes ReICOCO, RefCOCO+, and ReSCOCOg, all derived from the MS-COCO image dataset Lin et all (2014). These benchmarks contain natural language expressions referring to specific objects or regions within images, collected through human annotation (Kazenzadch et al. 2014 Yu et al. 2016). RefCOCO contains 142,209 expressions for 50,000 objects across 19,994 images. ReiCOCO+ includes 141,564 expressions for 49,856 objects and focuses on appearance-based descriptions by filtering out location- based cues. Each dataset is split into three subsets: val, test,, and testB, where testA contains images with people and testB consists of non-person images, enabling more fine-graine evaluation. We use the OFA- preprocessed version of these datasets (Wang et al., 2022a), where each referring expression is paired with a unique bounding box to standardize model input and supervision.
Visual Question Answering (VQA). We conduct experiments on the VQAv2 dataset (Goyal et al. 2017). VQAv2 comprises a collection of approximately 1.1M samples from 200, 000 images accompanied by 13M natural language answers (Goyal et al. 2017). As per OFA, instances for which multiple potential answers exist for a given question-image pair were split into individual samples for each potential answer (Wang el al., 2022a). This OFA-adapted version of the VQAv2 dataset includes training, validation, and test sets, with 1.8M, 10,402, and 447,793 samples, respectively. To enable EC pretraining, we divide the training set into two halves. For the first half, we replace the original natural language answers with emergent communication (EC) tokens generated by the speaker model, producing an EC-labeled corpus. The second half retains the original human annotations and is used for fine-tuning the model on the standard VQA task. This setup allows us to assess how well EC pretraining transfers to natural language question answering.
Visual Entailment (VE). In VE, an image serves as a premise along with a hypothesis text H The objective is to determine whether the hypothesis can be inferred from the image, i.e., if the image entails the hypothesis [Xie et al., 2019). The model must classify the pair into one of three categories: entailment (if there is sufficient evidence in I to conclude that Htet is true), contradiction (if there is enough evidence in I to conclude that Heext is false), or neutral (if there is insufficient evidence in I to make conclusion about H, VE shares similarities with VQA as both tasks require the model to reason and make logical deductions based on an image. However, VE is more complicated than VQA, in the sense that choosing whether a text entails an image requires complex fino-grained reasoning beyond answering a question. We evaluate VE performance using the SNLI-VE dataset (Xie et al. 2019 2018), which contains 529,527 training samples, 17,858 validation samples, and 17,901 test samples derived from 29,783 unique images. To examine the effect
6

---

Published in Transactions on Machine Learning Research (07/2025)
of EC pretraining under different supervision levels, we conduct experiments using the full training set well as reduced subsets of 50,000 and 10,000 samples. These settings allow us to assess the generalization benefits of EC across varying data regimes.
Image Captioning (IC). We evaluate image captioning performance using the Microsoft COCO dataset (Lin et al. 2014), a widely used benchmark for vision-language tasks. Image captioning in- volves generating fuuett semantically rich captions that accurately describe imput images. We adopt the OFA-preprocessed version of MSCOCO, which is split into four subsets: caption auption_sttael___ sttges_aattere train, cap- tion Baau__staee_____ train, caption_ _aal,o___,,,ll______llee and caption test. Each image in caption caption_stage___ train is paired with a single caption, while images in the other subsets are associated with approximatcly five captions cach. The dataset includes ~566K samples in caption stagel train, 113K in caption stage2 train, 5K in the valida- tion set, and 5K in the test set. For evaluation, we follow the Karpathy split (Karpathy & Fei 2015) and report results using established automatic metrics: BLEU-- (Papineni et al.) 2002). METEOR (Banerjec &
Lavie 2005), CIDEr (Vodantam et al. 2015), and SPICE (Anderson et al.) 2016).
Dustruction-Folowiii Benchmarks. We adopt the Vla pretraining and fine-tuning dataset, with modified captions from natural language to EC tokens during pretraining. In Lla VA-1.5, pretraining utilizes approximately 558K images sampled from the LAION (Schuhmann et al.) 2022). CC (Changpinyo et al. 2021), and SBU (Ordonez et al. 2011) synthesized captioning datasets, where captions are reformatied into simple instruction-follwinn prompts. To construct the instruction-folllwing dataset, natural image-text pairs are augmented with human-issued instructions. In the visual instruction tuning stage, we fine-tune the model end-to-end on a mixture of ~665K multimodal custructionfoolloiing examples, synthesized and sampled from variety of VQA data sources, including GPT-generated content, GQA (Hudson & Manning, 2019), COCO (Lin et al. 2014), and Tex (Singh et al. 2019 In addition, we evaluate on multimodal benchmarks POPE (measuring hallucination resilience) Li et al. (023b)) MME (visual perception) Fu et al. (2023, and MM- Vet (visual conversations)
Yu et al. (0024)
4.2 EC Pre-training
Pretraining on Visual Referring Expression (VRE). We pre-train the OFA base model for VRE on the RefCOCO train set. We use the EC-generated text from the referential game speaker agent as captions for pre-training. To obtain the EC text, we employ ResNet-18 18 model (He al.) 2016) and extract 512- dimensional image features from each image in the RefCOCO training set. These image features are then passed to speaker agent that generates a set of Emergent Communication (EC) tokens describing the image. Finally, we replace the original natural language RefCOCO caption that describes a bounding box with the EC-generated tokens. As the EC tokens encompass information about the image, we hypothesize that they implicitly capture information relevant to the associated bounding box, thus offering potential pro-training benefits. The resulting RefCOCO dataset, post-processed with EC tokens as captions, serves as the basis for pre-training the OFA base VRE model.
Pretraining on Multiple Tasks. To assess the broader utility of EC tokens, we conduct multitask pretraining unified dataset combining Visual Referring Expression (RefCOCO), Image Captioning (MSCOCO), and Visual Question Answering (VQAv2). We construct two versions of this dataset: one using natural language (NL) annotations and another using EC-generated tokens in place of all NL captions and answers. EC text is generated using the same pipeline as in the VRE experiment, where ResNet-18 features are passed to the speaker agent to produce discrete messages that replace the original text with these EC-gencrated tokens. After pretraining, we finc-tune the berf-pngg checkpoints for both the EC and NL-pretrained models independently on downstream vision-language tasks: Visual Referring Expression (VRE) and Visual Entailment (VE). The baseline is OFA model with randomly initialized weights trained from scratch on each task without any pretraining. These experiments aim to elucidate the added value of EC text in comparison to models trained without pretraining or those pretrained solely with NL data, thus offering a comprehensive understanding of the role of emergent language in enhancing vision-language tasks.
Instruction-Followin Tasks. We benchmark the effectiveness of EC pretraining on instruction-following tasks using LL (Liu et al. (2024a), a state-of thtttr open-source VLM. LA. integrates a CLIP-based vision encoder (Radford et al. 2021 with a Vicuna language model (Chiang et al. 2023 and
7

---

Published in Transactions on Machine Learning Research (07/)025)
is trained end-to-end to handle both visual and language modalities, achieving GPT-4-level performance
multimodal chat benchmarks. LLa is trained in two slages: (1) feature alignment stage, where the CLIP visual encoder is aligned with the Vicuna LLM via image-caption pairs, and (2) visual instruction tuning stage using a diverse set of vision-language instruction-following data. In the original LLaVA-1.5, the alignment phase uses ~558K image-text pairs drawn from LAION (Schuhumann el al. 2022,, Conceptual Captions (Changpinyo et al. 2021). and SBU (Ordonez et al., 2011 with captions refined into simplified instruction templates to balance conceptual coverage. To evaluate the utility of EC tokens in this high- capacity setting, we introduce LLaVA-1.5-EC. SBU but replacing all natural language captions with F.C sequences. These E.C tokens are generated using our trained EC speaker models, which produce discrete symbolic messages per image. The resulting EC-based dataset is then used for the entire aligument phase in place of natural language captions. We hypothesize that these emergent tokens, though artificial, capture visually grounded and compositional semantics sufficient to train competitive insllintion-folloiing VLM.
variant trained using the same images from LAION-CC.
4.3 NL Fine-tuning
Visual Referring Expression (VRE). We fine-tune the best-performing EC-pretrained VRE model check- point on RefCOCO+. By using different visual grounding datasets in pre-training and fine-tuning stages (RefCOCO for pre-training and RefCOCO+ for fine-tuning), we ensure that information does not overlap between the two phases. Subscquently, we evaluate the performance of the fine-tuned model on all Ref- COCO+ splits (val, lestA, and BestB). For NL fine-tuning, we fine-tune the OFA base model on the VRE task, employing natural language captions from the RefCOCO+ dataset. We consider this as an upper- bound model for comparison against. the EC pretraining experiment. Following fine-tuning, we evaluate the fine-tuned model's performance on all RefCOCO+ validation and test sets.
Visual Question Answering (VQA). To examine the performance of EC pretraining on the Visual Ques- tion Answering task, we considered the Unified EC model obtained after pretraining multiple tasks detailed Unified models (both EC and NL) on half of the VQAv2 dataset with natural language question-answer pairs. The other half was used during pretraining. This helps avoid data overlap and overfitting. We consider NL fine-tuning as an upper bound for comparison against EC pretraining. Subsequently, we evaluate the fino- fine-tuned models on the VQAv2 validation and test-dev sets.
in Section 4.2 We fine-tune the best-
Visual Entailment (VE). To assess the potential of emergent communication (EC) models for transferring knowledge to new vision language tasks, we employ the EC pre-trained model obtained from VRE as pre- training for the VE task. We fine-tune the best EC pre-trained model checkpoint on the SNLI-VE training set and evaluate on the SNLI-VE test and dev validation sets. To further understand the impact of EC under varying sampling conditions, we conduct fine-tuning on randomly sampled sets with different sizes, e.g., 10,000 and 50,000 samples. For the NL experiment, we fine-tune the OFA base model on SNLI-VE. To assess the impact of training set sample size on model performance, we similarly perform fine-tuning with different training set sample sizes and evaluate our models on the SNLI-VE validation and test sets.
Image Captioning (IC). To further assess the transferability of EC pretraining, we evaluate performance on the MS COCO image captioning benchmark (iin et al. 2014). Specifically, we fine-tune the best- performing ECCpretrained model initially trained on RefCOCO for visual referring expressions on the MSCOCO captioning task. This experiment probes whether bC-based grounding can generalize to open- ended image description generation. For comparison, we fine-tune the OFA base model directly on MSCOCO using natural language (NL) captions, trcating it as an upper-bound reference. Evaluation performed on the MS COCO test split using standard automatic metrics: BLEU-4 (Papineni al. 2002). METEOR (Banerjee & LLvie 2005), CIDEr (Vedantam et al. 2015). and SPICE (Anderson et al. 2016).
mnstruction-Follwiin Tasks. The visual instruction tuning phase involves end-to-end finc-tuning on a curated mixture of approximately 665k multimodal instruction-folloingg samples from VQA datasets such as GQA Hudson & Manning] 2019), COCO (Lin et al.) 2014), and TextVQA (Singh et al. 2019), as well as GPT-generated instructioonfollloinng content. This phase is designed to align the model's responses with natural language instructions grounded in visual input. In our LLaVA-L.5-EC model, we adopt the same instruction tuning protocol used in the original L.La VA-1.5 framework [Liu et al. 2024a). The only
8

---

Published in Transactions on Machine Learning Research (07/2025)
modification lies in the pretraining phase, where substitute natural language captions with discrete EC tokens. To benchmark the effectiveness of EC-based pretraining, compare against the original LLaVA-1.5 model trained entirely with natural language and various mainstream open-source VLMs, including BLIP-2
[Li et al.) 2023a), Instruct-BLII (Dai et al. 2023). Shikra (Chen et al. 2023 IDEFICS (DEEFICS 2028
and Qwen-VL VL (Bai et al. 2023
5
Experimental Results
5.1 Visual Referring Expression (VRE)
Table 1 reports accuracy the ReICOCO+ Visual Referring Expression (VRE) task across three configurations: Base (OFA without pretraining), +EC (EC-pretrained, then fine-tuned on NL), and +NL (NI-pretrained and fine-tuned on NL). Pre- training on EC tokens yields a substantial improvement over the Base model, increasing accuracy by over 2x on all splits. This demonstrates the strong transferability of EC tokens, which are generated without any natural language supervision and yet en- code rich semantic structure grounded in visual input. While NL pretraining still achieves the highest scores, which is expected due to the compositional and grounded nature of natural language, EC pretraining reaches approximately two-thirds of the performance of NL pretraining across all evaluation splits. This finding highlights the potential of F.C-based pretraining low-resource or label-scarce settings, where collecting large-scale image-text pairs is infeasible. Notably, the EC referential game and the generation of EC tokens by the speaker agent can be performed un- labeled images in the wild, effectively producing descriptive captions for these images. Consequently, EC tokens contain valuable semantic information about the images, which can prove advantageous for vision- language pretraining tasks. These findings emphasize the potential of leveraging EC pre-training for vision- language tasks. A deeper qualitative analysis in Appendix B reveals that EC tokens exhibit latent com- positional syntax, polysemy, and prosodic-like structures, often paralleling natural language in their ability to encode semantics, disambiguate meaning through repetition, and refine concepts via positional cucs.
Table 1: VRE Accuracy on RelCOCO+
splits. Base: OFA without pretraining, EEC: E'C-pretrained, then NL fine-tuned. +NL: NI-protrained and fine-tuned.
Model val
test
testB
Basc
29.81
31.49
27.53
+EC 62.17
67.15
51.16
+NL
81.91
86.60
73.49
5.2 Pretraining on Multiple Tasks
Table 2 demonstrates the effectiveness of EC pretrain- ing as a general-purpose initialization for multimodal tasks. In the context of VRE, EC pretraining surpasses a baseline model trained directly on visual grounding tasks without additional pretraining, achieving an im- pressive relative gain of over 108%. Similarly, in VQA, EC improves test-dev accuracy by 11.5%. These gains highlight the ability of EC tokens, learned without hu- man annotation, to transfer semantic and structural in- formation across lasion-language tasks. While NL pro- training still leads to the highest overall performance, EC pretraining achieves notable improvements despite
Table 2: Unified Pretraining. EC pretraining yields consistent gains. Base: OFA without pre- training. +EC: FC-pretrained, then NT-finetuned. +NL: NL-pretranned and NL-finctuned.
VRE (RefCOCO+)
VQA (VQAv2)
Model
val
test.A
testB
val
test-dev
Base
10.03
13.88
9.72
49.33
40.80
+EC
23.77
28.89
18.84
50.61
45.49
+NL
40.15
45.90
31.34
56.66
51.26
not relying on natural language supervision. These findings suggest that EC-based pretraining can be vi- able alternative or complementary strategy in settings with limited access to natural language annotations, enabling more scalable and generalizable vision-language learning.
5.3 Visual Entailment (VE)
Figure g presents results for the visual entailment task using models initialized from EC-VRE pretraining. Across varying training set sizes, EC pretraining consistently improves VE performance over the bascline model. These results provide compelling evidence of the utility of EC tokens for transferring knowledge
9

---

Published in Transactions on Machine Learning Research (07/2025)
Base
+FC Pretraining
NNL. Pretraining
100
100
89.05
88.95
84,90
84,20
Accuracy (%)
81.30
Accuracy (%)
80.30
80
85.30
80
85.01
70.80
70.30
60
57.30
60
-57.30
50.85
50.46
50.30
48.00
48.50
49.94
0
10K
50K
Full
10K
50K
Full
Training Set Size
Training Set Size
(a) SNLI-VE validation set
(b) SNLI-VE
Figure 2: Visual Entailment (VE) Accuracy with Varying Training Sizes. EC pretraining substan- tially improves VE accuracy compared the baseline across all training sizes, and approaches NL pretraining performance more downstream data becomes available. Base, +EC Pretraining, and +NL Pretrain- ing, corresponding to training pretraining, EC pretraining followed by natural language (NL) fine-tuning, and NL pretraining followed by NL fine-tuning.
Table 3: Image Captioning Accuracy. -EC model consistently outperforms the base model on all 8 metrics, often doubling quadrupling performance. Base: OFA without pretraining. +EC: EC-pretrained then NL fine-tuned. NLL NI.-pretrained then NL. fine-tuned.
Model BLEU-1
BLEU-2
BLEU-3
BLEU-4 METEOR ROUGE
CIDEr
SPICE
Base
48.70
26.58
14.92
8.45
12.11
35.65
0.12
3.94
+ EEC
70.54
52.46
38.08
27.39
22.77
50.95
0.83
15.78
+NL
75.15
58.60
44.13
32.71
26.31
54.98
1.06
19.76
between VL tasks. When comparing EC pre-training with the natural language fine-tuning of OFA, it is expected that the latter would achieve better performance. However, it is surprising to observe that EC pre-training can yield comparable performance in certain settings. For instance, when fine-tuned on the full training data, EC achieves an accuracy of 85.01% on the test sct, whereas NL achieves 88.95%. The relatively small performance gap between EC-VRE pre-training and natural language fine-tuning highlights the transferability benefits of EC pre-training and opens up future directions in utilizing emergent language for complex vision language tasks and more efficient VL modeling.
5.4 Image Captioning (IC)
To further evaluate the transferability of EC pretraining, fine-tune the EC-VRE model originally pre- trained RefCOCO an image captioning task. As shown in Table 3 EC pretraining yields substantial improvements across all standard evaluation metrics. Most notably, it achieves more than a 4x increase in CIDEr score compared to the baseline model and more than doubles performance on several other metrics. Despite having no access to human-written captions during pretraining, the EC-pretrained model approaches the performance of models initialized with natural language (NL) pretraining. These results underscore the representational strength of EC tokens for open-ended generation tasks and point their potential utility in multimodal learning scenarios with limited supervision.
5.5 Enstruction-Follvvvii Tasks
Table compares LLaVA-L.5-EC, which is trained exclusively on EC tokens, against a range of state-of- the- art VLMSs across five benchmarks: VQAv2 (Goyal et al. 2017), GQA Huudson & Manning 2019), VixWiz
10

---

Published in Transactions on Machine Learning Research (07/2025)
Table 4: Comparing LLaVA-1.5-EC with SoTA Instruction-Folloiing Models. LLaVA-1.5-EC, pretrained solely on EC tokens, achieves strong performance across five diverse VQA tasks, despite not using natural language during pretraining. While slightly trailing LLaVA-1.5 on VQAv2 and GQA, matches outperforms BLIP-2 and InstructBLII in most settings. Results highlight the transferability and robustness of EC-based pretraining for downstream multimodal reasoning tasks. "Training images/ aannotations of the datasets are observed during training. Inncluess house data that is not publicly accessible.
Image Size
Sample Size
VQAv2
GQA
VizWiz
SciQA IMG
Tex
Method
LLM
Pretrain
Finetune
BLIP-2
Vicuns-
224
129M
65.0
41
19.6
61
42.5
InstructBLIP
Vicuna-7B
224
129M
1.2M
49.2
34.5
60.5
50.1
InstructBLIP
Vieuna-13B
224²
129M
1.2M
49.5
33.4
50.7
Shikra
Vicuna-13B
224
600K
3.5M
77.4"
IDEFICS-9B
LLaMA-7B
224
353M
IM
50.9
38.4
35.5
25.9
IDEFICS-S0B
LLMMA-65B
224
353M
LM
60.0
45.2
36.0
30.9
Qwen-Vr VL
Qwen-7B
448
1.4B²
50M
78.8*
59.3*
35.2
63.8"
Qwen- VL-Chat
Qwen-7B
448
1.4B"
50M
78.2"
57.5*
38.9
68.2
61.5"
LLaVA-1.5
Vicuna-7B
3362
558K
665K
78.5'
62.0*
50.0
66.8
58.2
LLaVA-1.5-C
Vicuna-7B
3362
558K
665K
72.03*
55.27
40.03
64.45
48.61
Evaluation on Multimodal Benchmarks. LLaVA-155-E--73 demonstrates strong robustness, oulperforming InstructBLIP-14B on the POPE (adv) subset, MME, and MM-Vet, despite lacking nat natural language-aligned pretraining, highlighting EC's potential as scalable and compositional grounding signal.
Table 5:
Method
POPE (rand)
POPE (pop)
POPE (adv)
MME
MM-Ves Vet
BLIP2-14B
89.6
85.5
80.9
1293.8
22.1
IIS
87.7
77.0
72.0
1212.8
25.6
LLaVA-7B
76.3
72.2
70.1
809.6
25.5
LLAVA-15-CLIP-TE
17.1
17.1
17.1
687
11.3
LLaVA-1.5-7B
87.3
86.1
84.2
1510.7
31.1
LLaVA-1.5-BC-7B
75.1
74.6
74.0
1274.4
27.8
Gurari al., 2018). SciQA-IMG Lu al., 2022). and TextVQA (Singh et al. 2019). Despite using only 558K images and natural language supervision, L.LVA--15--FC achieves competitive performance, outperforming well-ostablished models such BLIP-2 (13B) and InstructBLIP (13B) across most datascts. For instance, relative to BLIP-2, LLaVA-1.5-EC achieves a 104.23% gain VizWis, 34.8% on GQA, and 10.8% on VQAv2. It also outperforms InstructBLIP, which is trained on 129M captioned image-text pairs and fine-tuned on 1.2M additional examples, highlighting the impressive representational capacity of EC pretraining. While LLaVA-1.5-EC does not surpass Qwen- VL trained with over a billion curated image- text pairs it achieves strong results despite having seen 2-3 orders of magnitude fewer samples and no human-writted captions. For example, on VizWiz, LLaVA-L.5-EC even outperforms Qwen-VL (40.03 vs. 35.2), showing its effectiveness visually grounded tasks.
Table 5 demonstrates that LLVA--155-FC-77 outperforms InstructBLIP-14B multimodal benchmarks POPE (adv), MME, and MM- Vet, despite lacking any natural language supervision during pretraining, These results highlight the semantic precision and transferability of EC tokens under distribution shifts and reinforce the hypothesis that ECC pretraining induces grounded, compositional structures that are well-aligned with real-world langunnl reasoning demands. While LA-1.5 still outperforms EC absolute terms (as expected due to the of curated human language), the relatively small performance gap highlights the surprising generalization capacity of emergent communication and its potential scalable pretraining strategy in low-resource or text-scarce settings. Results so far also reveal that emergent language may encode compositional and semantic patterns that are beneficial for downstream reasoning, a hypothesis we investigate more deeply in Section 5.6 through targeted ablation studies.
In addition, Figure X comparcs LLaVA-1.5-BC against leading VLMs on MMBench (Liu et al. 2024b). a rigorous benchmark suite for evaluating entruucioon-olloviinn and multimodal reasoning in VLMs. Both MMBench (EN) and its Chinesc-translated variant (MMBBecc-CN) include broad set of multimodal ques-
11

---

Published in Transactions on Machine Learning Research (07/2025)
MMRlech (EN)
MMBeechh (CN)
67 $7.30 30
60. 60
(64 1..30
64.00
56. 70
58.30
$4.50
48 20
50.56
49
Accuracy (%
36 36.00
38 20
38, 10
338.70
36. 40.40
10
25. 20
2311
20
7,
0
Quen
IDEFICS-BB
DEFICS-<0B
Qwe-VVL-Cast
MaVA-1 MaVA-La-CUPP1E LLVA-15-B M-VA-L--RC-B
Figure 3: Comparison of LLaVA-1.5-EC with SoTA Instruction-Follont Models on the MM-
Bench Benchmark (English and Chinese). LLVAA-1.5- pretrained using Emergent Communication
tokens, surpasses all baselines, highlighting the potential of EC-based pretraining.
Entropy=5.21
Entropy=5.08
Seq. 5, Entropy-2.66
EC Seq. 5. Entropy=2.68
15000
17500
Prequnnyy
800000
25000
25000
12500
Frequency
20000
20000
10000
5000
7500
200000
400000
15000
10000
15000
10000
5000
5000
2000
2010
3000
4000
2500
5000
75.00
20000
prog
600
800
2000
2200
600
800
100g
1200
Rank
Rank
Rank
Rank
(a) RefCOCO NL
VQAv2 NL
(c) RefCOCO EC (L=5)
(d) VQAv2 EC (L=5)
Figure Unigram distributions for Natural Language (NL) and EC tokens (L=5). Each plot shows token frequency ranks for the first 10K samples from RefCOCO and VQAv2 datasets. Compared to NL, EC token distributions more peaked and concentrated fewer symbols, resulting in lower entropy (ReICOCO: 2.66 5.21; VQAv2: 2.68 vs 5.08). This suggests that shorter FCC messages rely on fewer reused tokens to convey meaning efficiently, while the versatility of natural language allows it to incorporate way more unique tokens in comparison.
tions across diverse topics, assessing a model's capacity follow instructions, ground language in visual input, and make fine-grained distinctions. Despite being trained exclusively on discrete EC tokens without any natural language captions, our 7B parameter LLaVA-1.5-EC model achieves the highest accuracy across both English (67.3%) and Chinese (64.01%) MMBench sets. It outperforms significantly larger models such as Instruct IDEFICS-SOB, and Qwen- VL, the latter trained on over a billion in-house images. No- tably, ----1-----C consistently outperforms a LLa VA-CLIP baseline, which directly fine-tunes LLaVA-1. with raw CLIP features, demonstrating that EC-based pretraining provides a more semantically structured and transfcrable representation than direct projection from CLIP features alonc. Even more impressively, LA-11-1.5-B surpasses both the original LLaVA-7B and the improved LLa VA- models, despite re- lying entirely synthetic pretraining. These results underscore the strength of EC-based pretraining when combined with LLaVA-1.33 visual instruction tuning, and demonstrate the potential of emergent language as an effective and scalable pretraining signal for multimodal reasoning tasks.
5.6 Ablations
We conduct ablations to analyze the statistical and structural propertics of EC tokens and identify factors driving their transferability. Specifically, we aim to address two central questions: (1) To what extent are EC tokens consistent and meaningful across tasks? and (2) Does the structure of EC messages, beyond token identity, contribute to transfer performance? To this end, we analyze unigram distributions study vocabulary usage across datasets and conduct word-order perturbation experiments probe the role of compositional structure in EC representations.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 6: Effect of Structure and Semantics in EC Language. ECorig Berordered:; ECrandoo represent the original EC language generated by the speaker, perturbed EC language by reshuffing,, and random EC tokens, respectively. All models are fine-tuned on RefCOCO+. ECorig the model pre-trained on the original EC language, exhibits higher downstream performance all evaluation splits compared to the models pre- trained on shuffled EC language and random EC language. Results confirm that the structure and token semantics of EC language play a critical role in its transferability for vision-languag tasks.
Zero-Shot
ECorig
ECreardered ECranoom
val tcstB
29.81
42.50
42.10
42.10
27.53
34.49
33.63
33.34
MMBench (EN)
MMBench (CN)
80
80
63.85
62.83
59.85
59.11
60
60
Accuracy (%)
40
31.1831.51
35.15 35.15.22.9
32.92
40
34.18
31.51
34.17
34.17
31.79
20
20
L=5
L=15
L=25
V=1000
V=4035
V=10000
(a) Varying Token Length
(b) Varying Vocabulary Size
Figure 5: Ablation Studies on EC Token Design. (Left) Accuracy decreases with longer token sequences.
(Right) Smaller vocabularies improve generalization.
Unigram Distributions. We compare the unigram distributions of natural language (NL) and emergent communication (FCC across RefCOCO and VQAv2. For F.C, we generate sequences of fixed lengths (T=(5, 15, 25}) from the first 10K training samples using trained speaker model. All text is tokenized using the NLTK word tokenizer]] and unigram counts computed and sorted to generate the respective unigram distributions for each corpus. Figure 4 unigram distribution plots reveal notable differences in lexical diver- sity between natural language (NL) and emergent communication (EC) corpora. For both RefCOCO and VQAv2 datasets, NL sequences show substantially higher entropy (5.33 and 5.08 for RefCOCO and VQAv2, respectively), indicating broader loken usage and greater linguistic variability. In contrast, EC texts gener- ated with fixed length L=5 exhibit lower entropy (4.52 for RefCOCO, 4.63 for VQAv2), suggesting a more concentrated and repetitive token usage. This implics that EC, despite being artificially generated, converges a smaller, more structured vocabulary that is reused efficiently, likely to optimize communication within tight length constraints. The more peaked EC distributions further support the idea that emergent messages encode semantics compactly, relying on structure and positional encoding rather than lexical diversity.
Investigating the Importance of Structure in EC Tokens. To better understand the role of structure and semantics in EC language, we conduct an ablation study using three variations of EC pretraining data: (1) ECorig: the original E.C tokens generated by the referential game speaker, (2) ECreordered:: EC tokens with their order randomly shuffled, and (3) ECrandomi sequences of randomly sampled tokens of matching length. Each variant is used to pretrain task-agnostic OFA model on 30K subset of ReICOCO. A After pretraining, we fine-tune each model on 10K samples of RefCOCO+ and evaluate on standard validation and test splits. Results in Table 6 show that ECorig yiclds the best downstream performance, highlighting the
http:://ww.ntth..r/////////tt http:///////ppp////eeeaeeee// http://inn.....ooi.eeeeeeee
https://www.nttzh www.///ww.lkkk.oomm
13

---

Published in Transactions on Machine Learning Research (07 (225))
Table 7: EC Token Entropy and Referential Game Success Rate ablations varying token length (left, vocab size fixed at 4035) and vocabulary size (right, token length fixed at 15).
Token Len
Entropy
Success Rate (%)
Vocab Size
Entropy
Success Rate (%)
5
2.68
77.31
1,000
2.45
81.46
15
3.82
95.59
4,035
3.82
98.39
25
4.58
99.27
10,000
2.21
86.88
importance of both token order and semantic content in emergent communication. These findings suggest that EC tokens encode nontrivial structure and meaning that support generalization in vision-language tasks.
Channel Capacity- Generalization Tradeoff in EC Token Design. We further examine how the communication capacity of emergent language, defined by token sequence length and vocabulary size, impacts both intra-agent coordination and downstream transfer. Token length and vocabulary size jointly determine the theoretical channel capacity (measured as log)(V) L., i.e., the maximum information an agent can convey per message. For each configuration, L E {5, 15, 25} and V E {1000, 4035, 10000), we pretrain a LLaVA-EC model using the resulting EC corpus and evaluate both (1) downstream accuracy on MMBench and (2) channel utilization via token entropy and referential game success rate.
As shown in Table 7 increasing token length consistently increases both message entropy and referential accuracy, indicating greater utilization of the available capacity. However, this higher utilization correlates with lower downstream accuracy (Figure 5aa. suggesting that agents are encoding redundant messages opti- mized for referential discrimination but poorly aligned with transferable semantic abstractions. The shortest configuration (L (5 5) performs best downstream, highlighting the benefit of a compressed emergent proto- col that emphasizes abstraction over specificity.
similar trend observed when varying vocabulary size (Figure 5b Table 7 the smallest vocabulary (V = 1000) yields the highest downstream accuracy, despite lower entropy and referential success. In contrast, larger vocabularics result in higher entropy and referential performance but worse transfer, consistent with over-specialization degenerate symbol use.
These results collectively reveal channel capacity generalization tradeoff in emergent communication. While greater capacity improves intra-agent coordination, as evidenced by higher entropy and referential success, it also increases the risk of overfitting, symbolie drift, and protocol inefficiency. This aligns with findings by Chaabouni et al. (2019), who observed that agents tend to produce overly long and inefficient messages unconstrained settings. Our results extend these insights by demonstrating that capacity bottle- necks, introduced via shorter sequences or smaller vocabularies, encourage symbolic reuse and abstraction, ultimately yielding better generalization to downstream multimodal tasks.
6
Conclusion
This work investigates the potential of Emergent Communication (EC) learning signal for vision-language models. We explore whether discrete messages generated by referential game agents can serve as effective supervision for cross-modal pretraining. Through extensive experiments across diverse tasks, including visual referring expression, visual question answering, visual entailment, and image captioning, we demonstrate that EC pretraining significantly improves performance over strong non-pretrained baselines and, some cases, approaches or surpasses models trained with natural language. We further validate EC's gencralization by scaling instruction-folloiinng (asks, demonstrating that LLaVA-1.5-EC achieves competitive accuracy instruction- foolowwinf benchmarks despite being trained solely on synthetic EC tokens. Ablations reveal that EC' tokens encode structural and semantic signals crucial for transfer learning. These findings suggest that EC can serve as a scalable and complementary resource for Aision-language pretraining, particularly in low-resource or weakly supervised sents
One promising direction is develop new class of pretraining objectives grounded entirely interaction, i.e., replacing static captioning supervision with agent-driven communication. Another exciting possibility is to evolve ECC into fully expressive representational language that can interface directly with down-
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
stream models, prompting them as effectively human language. Further, future research could explore the emergence of syntax compositional grammar within EC tokens and whether EC can self-organize into hi- erarchies aligned with semantic or spatial structures in images. Finally, we envision agent ecosystems where EC evolves dynamically over time through open-ended interaction with multimodal environments, offering a path toward continual, self-supervised vision-language learning without reliance on human annotations.
7 Acknowledgments
This rescarch is basod work partially supported by the Amazon-Virginia Toch Initiative for Efficient and Robust Machine Learning and by the U.S. Defense Advanced Research Projects Agency (DARPA) under award numbers HR00112390062 and HR001 125C0303. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of Amazon, DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes nithstatngg any copyright annotation therein.
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE/CT EFF/ Conference on Computer Vision and Pattern Recognátion (CVPR), 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large laignnlang model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Satanjocv Banerjee and Alon Lavic. Metcor: An automatic metric for mt evaluation with improved correla- tion with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Eatrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS). 2019.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pro-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition (CVPR), 2021.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal I1m's referential dialogue magic. arXiv preprint Xrxiv:2306. 100.151100 15195, 2023.
Cen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Tiu. Uniter: Universal image-text. representation learning. In Proceedings of the Farropaan Conference on Computer Vision (ECCV). Springer, 2020.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vieuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. ftttp:///viee http:///wiuooooommsorr... Insys.org (accessed 14 April 2023), 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purper vision- langunel models with instruction tuning. In Proceedings of the Conference Neural Information Processing Systems (NeurIPS), 2023.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the International Conference Learning Representations (ICLR), 2019.
Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pro-trained models. In Proceedings of the International Joint Conference on Arlificial Intelligence (IJCAI) Survey Truck, 2022.
Linus Fricsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? Tn Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation (CVPR), 2021.
Katrina Evtimova, Andrew Drozdov, Douwe Kicla, and Kyunghyun Cho. Emergent communication in a multi-modal, multi-step referential game. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mie: A comprehensive evaluation benchmark for multimodal large language models. Xiviv preprint arXiv:2306. 13394, 2023.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. Vision-Languanl Pre- Training: Basics, Recent Advances, and Future Trends. Proceedinggs of the Foundations and Trends in Compater Graphics and Vision, 14(3-4):163 352, 2022.
Yuying Ge, Sijie Zhao, Ziyun Zeng. Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Slay, Dhruv Batra, and Devi Parikh. Making the v in vqa malter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CI /CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jellrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2018.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Proceedings of the Conference Neural Information Processing Systems (NewrIPS), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
IDEFICS. Introducing idefics: An open reproduction of stale-of-the-arl visual language model. https: /hnggingface..o/bblgg hto/bloo/ideffcooo 2023.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameteriiztton with Gumbel-Softmax. In Proceed- ings of the International Conference on Learning Representations (ICLR), 2017.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic aliguments for generating image descriptions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2015.
Sahar Kazemzadch, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
16

---

Published in Transactions on Machine Learning Research (07/2025)
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languaa transformer without convolution region supervision. In Proceedingg of the International Conference on Learning Represenlations (TCLR), 2021.
Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint ar arX:::2006.02. 0200002241 19, 2020.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence (natural) language. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
A Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic com- munication from referential games with symbolic and pixel input. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by mross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), volume 34, pp. 11336-11344 2020a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang, Visualbert: A simple and performant baseline for vision and language. orXiv preprint arXiv: 03557, 2019.
Yaoyiran Li, Edoardo Maria Ponti, Ivan Vulié, and Anna Korhonen. Emergent Communication Pretraining for Few-Shot Machine Translation. In Proceedings of the International Conference on Computational Linguistics (COLING), 2020b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pallern Recognátion (CVPR), 2024a.
Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang, Zhongehao Shi, Jianping Fan, and Zhiqiang He. A survey of visual transformers. IEEE Transactions on Neural Networks and Learning Systems, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Proceedings of the Furopean Conference on Computer Vision (ECCV), 2024b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining lask-agnostic visiolinguistic representations for vision-and-langunn tasks. In Proceedings of the Conference Neural Information Processing Systems (NeurIPS). 2019.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of the Conference on Neural Information Processing Systems (NewrIPS), 2022.
17

---

Published in Transactions on Machine Learning Research (07/)025)
Igor Mordatch and Picter Abbecl. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAT Conference Artificial Intelligence, (AAAT), volume 32, 2018.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2lext: Describing images using 1 million captioned photographs. In Proceedings of the Conference Neural Information Processing Systems (NewrIPS), 2011.
Tsabel Papadimitrion and Dan Jurafsky. Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic cvaluation of machine translation. In Proceedings of the Annuol Meeting of the Association for Computational Linguistics (ACL), 2002.
Shivansh Patel, Saim Wani, Unnat Jain, Alexander Schwing, Svetlana Lazebnik, Manolis Savva, and Angel X. Chang. Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2021.
Senthil Purushwalkam, Pedro Morgado, and Abhinav Gupta. The challenges of continuous self-supervised learning. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Compositional structure can emerge without generational transmission. Cognition, 182:151 1-.1111114,,.- 2019a.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Larger communities create more systematic languages. Pro- ceedings of the Royal Society B, 286(1907))20191122, 2019b.
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M Dai, and Kyunghyun Cho. Capacity, bandwidth, and compositionality in emergent language learning. In Proceedings of the International Conference on Aulonomous Agents and Mulliagent Systems (AAMAS), 2020.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large- scale dataset for training next generation image-text models. In Proceedings of the Conference on Newral Information Processing Systems (NewrIPS) Track on Datasets and Benchmarks, 2022.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proccedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Weijic Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pro-training of Generic Visual-Linguistic Representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), volume 33, 2020.
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Im- age captioners are scalable vision learners too. In Proceedings of the Conference on Neural Information Processing Systems (NearIPS), volume 36, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), volume 30, 2017.
18

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IBEE/CVE Conference on Compuler Vision and Pattern Recognation (CVPR), 2015.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalitics through simple sequenco-to- sequence learning framework. In Proceediiggs of the International Conference on Learving Representalions (ICLR), 2022a.
Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, and Furu Wei. Distilled dual-encoder model for vision-language understanding. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), 2022b.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual Entailment Task for Visually- Lan- guage Learning. In NeurIPS Visually Grounded Interaction and Language (ViGIL) Workshop, 2018.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual Entailment: A Novel Task for Fine-grained 2019.
Image Understanding. arXiv preprint
Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Mochine Intelligence, 2023.
Shunyu Yao, Mo Yu, Yang Zhang, Karthik Narasimhan, Joshua B Tenenbaum, and Chuang Gan. Linking Emergent and Natural Languages via Corpus Transfer. In Proceedings of the International Conference Learning Representations (ICLR), 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 69-85. Springer, 2016.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024.
19

---

Published in Transactions on Machine Learning Research (07/2025)
A Implementation Details
The EC speakers used to generate the EC datasets are directly trained on the COCO image features from the codebase" length limit is set to 15. Generating EC sequences of length 15, the speakers draw from a vocabulary size of 4035 tokens. In the Visual Referring Exprcssion (VRE) experiments, we adopt the codebase of O (Wang et al. 2022a) and mostly follow their default setup. Initially, we pre-train the OFA model on the RelCOCO training set with EC annotations via continuous pretraining (Wang et al. 2022a). The pretraining data was prepared to align with OFA's 's structure, as outlined on their GitHub page. The pretraining process consists of 17 epochs and 492,000 updates. Subsequently, we fine-tune the pre-trained model for 10 epochs and 18,500 updates. Pretraining was conducted on single NVIDIA A100 GPU for 2 days, while the fine-tuning phase required 2 NVIDIA A100 GPUs and took 2 days to complete. For the Visual Question Answering (VQA) task, we again adopt the OFA codcbase and follow the default setup provided in both the continuous pretraining and VQA fin-luning and evaluation scripts. Continuous pretraining was executed on a single NVIDIA V100 GPU for days, encompassing 960,000 updates, which corresponded to approximately 4 to 5 epochs. Fine-tuning for epochs required around 90 hours on 2 P100 GPUs. In the Visual Entailment (VE) task, we utilize the pre-trained model that was obtained in the VRE task. We fine-tune this model on the SNLI-VE dataset (Xie et al. 2019h for 5 epochs and 20,500 updates. The fine-tuning process was distributed across 4 NVIDIA A40 GPU workers and took approximately 15 hours to complete. For the rest of the training process, we largely adhered to the OFA setup. For Image Captioning (IC), fine-tuning is conducted in two stages: (1) cross-entropy optimization for two epochs with a batch size of 128, learning rate of 1o-5, and label smoothing of 0.1; (2) CIDEr optimization for three additional epochs using a batch size of 64, disabling dropout and stochastic depth for stability.
P| of Yao et al. (2022) for 2000 epochs. Training is performed on a P100 GPU, and the sequence
B Qualitative Examples
We conduct an in-depth qualitative analysis to uncover potential patterns in the generated Emergent Com- munication (EC) text. For this analysis, we train three EC speakers over 1000 epochs, utilizing a vocabulary size of 4035 and sequence lengths of 15, and 25. The speakers are trained using COCO features from the EC game roduedd by (Yao et al. 2022). To generate EC text, we pass the first 1000 unique images from the refCOCO training dataset through each speaker. We track the positions of EC text n-grams within the generated sequences. If more than 5 images produce the same n-gram in the same position of the text sequence, we group those images for manual observation. Specifically, we examine bigrams, trigrams, and 4-grams for EC sequences of length 5, 15, and 25, respectively.
Figure S highlights groupings in EC sequences of length 5. For example, token 2430 is strongly associated with broccoli, while token 222 is frequently utilized to describe food broader category earlier in the sequence. In Figure g which presents examples with sequence length of 15, we observe that token 3293 exhibits a strong visual grounding to zebras, suggesting consistent semantic alignment. Figure 9 further illustrates how the same tokens, when placed in different positions, convey similar yet more refined meanings. For instance, token 309 corresponds to vehicles, but its count and position within the sequence determine whether it refers to a truck or a motorbike. This behavior implics that EC tokens may encode coarso-grained semantics individually while achieving finer-grained distinctions through compositional patterns, akin to polysemy and contextual disambiguation in natural language.
Furthermore, observe that token 3355 appears across diverse semantic contexts, hinting at its potential structural or delimitative role within the EC syntax, perhaps analogous to punctuation or function words. This pattern aligns with the hypothesis that certain tokens may be repurposed as syntactic scaffolds to organize more semantically meaningful tokens within the EC grammar.
Additionally, Figure 10 illustrates how n-gram patterns contribute to broader sequence-level structure. For instance, the bigram (1599, 1599) appears consistently across multiple sequences, sometimes at position 0 and other times al position 1. This observation suggests that its semantic contribution is preserved regardless
https://github.. com Sya/UFA
20

---

Published in Transactions on Machine Learning Research (07/2025)
EC Seq. 15, Entropy=3.88
EC Seq. 25. Entropy=4.58
EC Seq. 15, Entropy=3.82
EC Seq. 25. Entropy-4458
20000
Fraquency
20000
Frequency
20000
frequency
20000
15000
35000
15000
15000
10000
10000
10000
10000
5000
5000
5000
5000
200
400
600
1000
1200-
0-
200
40p
60g
n00
1000
120p
200
400
60p
800
1000
2200
20p
600
800
1000
1200
Rank
Rank
Rank
Rank
(a) RefCOCO EC (L=15) (b) RefCOCO EC (L=25)
(c) VQAv2 EC (L=15)
(d) VQAv2 EC (L=25)
Figure 6: Unigram distributions for EC tokens at longer sequence lengths (L=15, 25). Compared to L=5, longer EC sequences exhibit higher entropy. EC communication remains compact and token-efficient, indicating that it is optimized for information transfer rather than expressiveness.
of its exact position. This invariance hints that n-gram identity, rather than absolute position, may carry semantic weight, or that a higher-level structural framework governs EC sequence composition.
Interestingly, some images appear in both bigram and trigram configurations of token 1599, indicating a repetition- based mechanism for refining semantic precision, e.g., a trigram (1599, 1599, 1599) could empha- size a salient visual fcature more strongly than a single token, akin to prosodic stress or syntactic repetition in human language. This repetition may reflect form of emergent hierarchical encoding. Much like fixed phrases in natural language, EC subsequences may preserve meaning even when shifted in position, pointing to latent compositionality in the learned token space. Results suggest that the emergent communication pro- tocol not only encodes discrete semantics but may also develop primitive mechanisms for emphasis, structure, or saliency highlighting. Remarkably, these patterns emerge without any predefined linguist ic struct ure or grammar, indicating that EC self-organizes to support both grounding and discriminative specificity.
C Additional Ablation Results on Unigram Distribution
In Figure 6 EC unigram distributions with longer sequence lengths (L-15 and L-25) in both RefCOCO and VQAv2 reveal a clear trend toward lower entropy and flatter distributions. Entropy decreases from 4.35 (ReCCOCO, 1-=15) 3.98 (I.=25) and from 4.40 (VQAv2, I.=15) to 4.01 (I.=25), indicating that usage remains concentrated among a core subset of tokens. While the vocabulary size incrcases, token frequency becomes more structured and repetitive, suggesting the EC relies more heavily on compositional patterns rather than expanding lexical diversity, This supports the hypothesis that EC develops structured but efficient communication protocols, optimized for information transfer rather than expressiveness.
D Limitations
While our work provides strong empirical evidence for the benefits of EC pre-training across several core vision-language tasks, it does not exhaustively evaluate the generalizability of F.C tokens across the full spectrum of multimodal tasks. Our findings revealed significant improvements in learning and task gener- alization, but a comprehensive evaluation across a wider range of tasks remains an important direction for future work. A central limitation of our approach lies in its reliance on EC sequences derived from referen- tial games grounded in literal image semantics. While effective for grounding and localization tasks, these tokens may lack the compositional structure or pragmatic nuance required for more complex settings, such abstract reasoning, multi-turn dialogue, instruction following. As such, substituting natural language with EC tokens is not always straightforward and may depend on task-specific demands. Moreover, while EC tokens are generated without human-annotated text and thus avoid direct exposure to textual biases (e.g., gendered language or cultural metaphors), they are still grounded in visual data and may reflect biases inherent in the image distribution. For example, dataset-level imbalances in object representation, demo- graphics, or co-oceurrence patterns may shape the semantics of emergent tokens unintended ways. We believe this highlights an important future direction: evaluating and mitigating bias in synthetic emergent communication protocols. Such future work could also serve as testbed for studying these issues in
21

---

Published in Transactions on Machine Learning Research (07/2025)
controlled setting, and insights gained in EC may inform the design of future multi-agent LLM systems that rely on emergent, protocol-driven communication. Finally, our findings suggest that F.C pretraining is especially promising in data-constrained settings, and we envision its broader utility when combined with structured natural language supervision. Future work should explore hybrid models that flexibly integrate EC lokens and language-bass representations to benefit from the strengths both paradigms.
E
Broader Impact
Our work has several potential broader impacts. Firstly, the possibility integrating emergent language pre- training from Emergent Communication (EC) into Vision Language Models (VLMs) paves the way for the development of more robust and generalizable vision-language learning methods. This could have positive impact various applications, such image-text. retrieval, visual search, visual question answering, and image captioning, in addition to important implications for enabling VLMs to perform effectively in real- world settings where reprosentative data is limited, thus enhancing their practical utility.
Secondly, our research contributes to advancing communication between humans and machines. By inves- tigating how agents learn to communicate in EC games and establishing connections with vision language systems, we gain deeper insights inlo the cognitive and computational mechanisms that underlie effective communication. This understanding can fuel the development of more efficient and intuitive communication systems, benefiting both humans and machines in varions domains. Invostigating how agents might develop compositional, grounded communication from scratch could inform the design of future human-Al inter- faces that are more adaptive, interpretable, or robust in dynamic enviromments, such as assistive robotics or collaborative decision-making.
Third, our study lays foundational groundwork for a new pretraining paradigm, one that complements natural language supervision with self-discovered symbolic communication. By doing so, it contributes to broader efforts artificial intelligence to develop agents that reason over multiple modalities using internal, compact representations that do not depend solely on human-curated semantics. This, in turn, contributes to the evolution of AT technologies that better understand and interact with the world around them.
We do not anticipate significant negative impacts associated with this work. However, we acknowledge that emergent communication tokens, though not authored by humans, are ultimately learned from data-driven interactions and shaped by the distribution of visual inputs, optimization objectives, and task definitions. As such, they may still encode or amplify underlying biascs present the training data. For example, if image datasets exhibit imbalance or stereolyyee-riinoroen patterns in unexpected ways. Moreover, EC tokens lack the interpretability of natural language, which may complicate transparency or accountability in downstream applications. As EC-based representations are further explored and integrated into vision-language systems, it will be crucial to develop auditing tools and safeguards to ensure their robustness, fairness, and alignment with human values. Overall, our findings highlight EC not as a substitute for natural language, but as a complementary, scalable signal that can support langun-language learning, particularly domains where natural language limited or unavailable.
correlations, EC tokens might reflect these
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
2430 2430
222 2430 x
333
5D7
216
333
541
688
428
507
(b)
222 1599 X X
222 3967 X
161
15
68
239
273
416
87
164
(c)
(d)
Figure 7: Examples of EC Sequences Exhibiting Semantic Clustering. (a) The repeated occurrence of token 2430 is consistently associated with images containing broccoli. (b) Token 222 functions as higher- level food category marker. (c) Varying the tokens that follow 222 refines the type of food being described, suggesting contextual disambiguation. (d) The bigram 222 3967 remains food-associated but often appears in scenes involving people interacting with food, such as eating or holding it, indicating compositional encoding of both object and action.
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
32933 3293 3293
3293 229D
46
220
156
162
(a)
(b)
3293 2290 3293
3293 3293 3293
242
573
46
178
710
(c)
(d)
Figure 8: EC Sequences Reveal Visual Grounding, Compositionality, and Latent Structure. (a) Token 3293 consistently appears in zebra-related images, demonstrating strong visual grounding. (b) A variation in the third token of the trigram suggests fine-grained visual distinctions between zebra scenes, pointing to contextual compositionality. (c) The same trigram from (a) appears at different sequence position, indicating positional flexibility and implying that F.C meaning is carried by token patterns rather than fixed positions, a possible marker of syntactic invariance. Occasional co-occurrence with giraffes suggests token reuse across visually related concepts and hints at fuzzy semantic boundaries between visually similar classes. (d) The trigram from (b), when shifted to position 1, remains strongly correlated with zebra images, reinforcing compositional consistency and semantic robustness.
24

---

Published in Transactions on Machine Learning Research (07 (07/2025)
3355 1355 309 3355 X W
3355 309 3355 309 XXXXXXXXXXO
258
307
367
485
381
532
3355 3355 3355 2512 X...
3355 3355 3355 1915 XXX
132
194
62
109
630
840
187
452
(c)
(d)
Figure 9: EC Sequences Exhibit Semantic Specificity and Structural Roles. (a) Token 309 is associated with vehicles, and its position appears to modulate meaning, e.g., at position 2, it tends to refer to motorbikes. (b) The same token in different, positions corresponds to trucks, suggesting context-lepenndett semantic refinement. (c) Token 2512 may denote giraffes, while (d) token 1915 appears to generalize to a broader animal category. Notably, token 3355 occurs across all examples, suggesting a structural or functional role, potentially indicating count, emphasis, or grouping within the sequence.
25

---

Published in Transactions on Machine Learning Research (07/2025)
x 1599 1599
1599 1599 X
50
239
50
125
349
601
131
297
(b)
Figure 10: Semantic Consistency and Repetition Effects in EC Sequences. Images grouped based the occurrence of the bigram (1599, 1599), which consistently appears in EC sequences describing scenes with vibrant red, yellow, and green produce. (a) When this bigram appears in position 0, it strongly correlates with fruits and vogetables fcaturing these colors. (b) When shifted to position 1, the bigram still retrieves visually related scenes, though with slightly more variation. These patterns suggest that certain F.C n-grams carry consistent semantics across positions and that repetition may emphasize salient visual features akin prosodic stress or syntactic repetition in natural language.
26


Reviews:
[{"review_id": "yRSZohUy2j", "reviewer": "Reviewer_BYq9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The training of the speaker and listener agents is self-supervised, which makes the approach highly scalable.\n2. Using EC tokens for pre-training circumvents the need for manual curation of image-text pairs, equivalent to generating synthetic data with image inputs only.\n3. The experimental results show some potentials of EC tokens in pre-training vision-language models.\n\n**Weaknesses:**\n1. Including an experiment on image retrieval using learned EC tokens can be beneficial to demonstrating the effectiveness of EC tokens. A test on retrieval is necessary as it directly relates to the referential game used to train agents for EC token generation.\n2. When EC pretraining is used during the feature alignment stage of LLaVA-1.5, the CLIP image features are aligned to EC tokens rather than to natural language. To demonstrate the effectiveness of EC tokens, an ablation study that directly fine-tunes LLaVA-1.5 for visual instruction tuning, bypassing the feature alignment stage, is necessary. This comparison helps validate that EC tokens provide a more effective representation than raw CLIP features for instruction tuning. Ideally, the results should show that EC tokens are more effective than raw CLIP features despite lagging natural language.\n3. Some commonly used benchmarks are not included in instruction tuning tasks, such as MMMU, SQA, SEED, etc. Please refer to the original LLaVA paper and include the missing benchmarks for a more comprehensive comparison.\n\n**Questions:**\n1. For training the speaker and listener agents, the training objective is purely based on selecting the true image from a set. How do the authors recognise their learned EC tokens as an emergent language rather than some kind of discrete visual tokens similar to VQ-VAE and VQGAN? (From the second ablation on the structure of EC tokens, it seems that random order leads to a rather small decrease in performance.)\n2. The images in the image set for selection when training the agents are randomly selected. Have the authors tried hard sample mining by selecting images that are more semantically similar with the query image (by CLIP features, etc.), therefore potentially leading to more fine-grained details in EC tokens?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents using the Emergent Corpus from a referential game as a substitute for image-text pairs for pre-training vision-language models. The main contribution is that EC tokens enable an effective alternative when high-quality image-text pairs are limited.\n\nThe experiment results demonstrate consistent improvements on a variety of tasks, including visual referring expression, visual question answering, visual entailment, image captioning, and instruction-following tasks, despite still seeing a relatively large gap with using natural language. Moreover, for visual entailment, a promising scalability of pre-training with EC tokens is shown, and the performance converges to that of using natural language.\n\nOverall, the proposed use of emergent communication for vision-language pre-training presents an interesting direction for synthetic data generation. Although the performance still lags when using natural language, this paper serves as a proof of concept.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Figures:**\n1. Figure 2 does not provide any useful information, as the emergent expressions are not natural language. Including them here is confusing.\n2. The plots in Figure 5 are hard to distinguish. Using a table to show the entropies is better.\n\n**Experiments:**\n1. See Weaknesses Part for the experiments recommended to add in the paper.", "broader_impact_concerns": "The authors include a comprehensive section on the broader impact concerns, and I have no other concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uibx4XxfPy", "reviewer": "Reviewer_o3BJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* The reviewer acknowledges that the proposed method offers a way to reduce the need for large-scale image-text annotations in low-resource settings.\n\n* The pretrained model outperforms some strong supervised methods on certain tasks.\n\n* The proposed method can serve as a complementary pretraining strategy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies improve downstream tasks of large language models in low-resource settings with EC. The proposed method adopts RL method to inject visual prior knowledge from the vision model into the large language model through discrete \"pseudo-language\" tokens. This provides better visual understanding and initialization. The proposed method improves the fine-tuning performance on several downstream tasks, working as an alternative for the limited natural language annotations scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* It seems that the proposed method adopts EC tokens in pretraining but still needs fine-tuning on downstream tasks, which poses a gap that the EC pretraining could harm the original multimodal alignment ability or other abilities like reasoning of MLLMs.  How to solve potential limitations remains unclear.\n\n* The generation of EC tokens requires two agents with repeated interaction and shared supervision. The construction cost is important but remains unclear.\n\n* It is not clear how the proposed method can be deployed in open scenarios mentioned in conclusion. Besides, the performance gain of the proposed method in high-resource settings is unknown.\n\n* It seems that there lacks comparisons with other weakly supervised strategies. It is of the interest to see how the proposed method performs against contrastive learning under the same low-resource setting.", "broader_impact_concerns": "The reviewer does not have major ethical concerns about this work. The proposed method focuses on reducing reliance on large-scale human annotations, which could potentially lower data collection costs and promote accessibility in low-resource settings for the MLLM. The use of artificial agents for token generation does not raise obvious ethical risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s1fuB1v5pg", "reviewer": "Reviewer_SkkB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nInnovative Pretraining Mechanism: The use of EC tokens for pretraining vision-language models is a novel approach that significantly enhances the sample efficiency of the models.\n\nStrong Empirical Results: The authors provide compelling experimental evidence showing that EC pretraining leads to substantial improvements across several vision-language tasks, especially in low-resource settings.\n\nScalability and Generalization: The paper convincingly demonstrates that models pretrained with EC tokens, such as LLaVA-1.5-EC, can achieve competitive performance on instruction-following tasks, despite being trained on synthetic data without human annotations.\n\nClear and Structured Approach: The framework for EC pretraining and the experimental setup are well-articulated, making it easy to follow and replicate the experiments.\n\nWeaknesses:\n\nLimited Comparison with Other EC-based Models: While the paper compares EC-based pretraining with natural language pretraining, it would benefit from a more detailed comparison to other emergent communication-based methods or architectures.\n\nLack of Comprehensive Ablation Studies: Although the paper discusses the importance of the EC structure, it would be valuable to see more detailed ablation studies exploring the impact of varying EC token lengths, agent interactions, or other hyperparameters.\n\nEthical Considerations: While the paper addresses the efficacy of EC pretraining, there is limited discussion on the ethical implications of using synthetic communication protocols and the potential for unintended biases in EC-generated tokens.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Emergent Communication (EC) as a novel mechanism to improve sample efficiency in Vision-Language Pre-trained Models (VL-PTMs). The authors propose a framework where EC tokens are generated through a referential game between two artificial agents, and these tokens are used as pretraining signals for vision-language models. The major contributions of the paper are as follows: Introduction of EC in Vision-Language Pretraining: The authors leverage EC tokens generated through a referential game between agents as an effective pretraining signal for downstream vision-language tasks, including Visual Referring Expression (VRE), Visual Entailment (VE), and Visual Question Answering (VQA). Empirical Validation: The paper demonstrates the effectiveness of EC pretraining through a series of experiments, including substantial gains in performance on diverse tasks: RE accuracy improved by 108.6%. VE accuracy improved by 69.6%. LLaVA-1.5-EC, a model trained entirely on EC tokens, outperforms strong baselines (e.g., BLIP-2) on multiple benchmarks, including VizWiz, GQA, and VQAv2. Introduction of LLaVA-1.5-EC: A LLaVA variant that is pretrained solely on EC tokens, highlighting the potential of EC in low-resource and text-scarce settings. Theoretical Insights: The paper explores the compositional structure of EC tokens, their capacity to encode grounded semantic information, and their ability to generalize across different tasks. These contributions propose EC as a scalable, annotation-free signal that can improve vision-language modeling, especially in settings with limited data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Think carefully about weakness.\n2. Table 3 should show good results in bold.\n3. The words in the pictures in Figure 5 and Figure 6 are too small.\n4. Code links can be placed in the body.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)