A Compressive-Expressive Communication Framework for Compositional Representations
Authors: Rafael Elberg
Felipe del Rio
Mircea Petrache
Denis Parra
Venue: NeurIPS 2025 poster ()
Keywords: Iterated learning
Emergent language
Compositionality
Compositional generalization
Compositional Representations
Primary area: deep_learning
TL;DR: We propose CELEBI, a self-supervised communication game that promotes compositionality via three novel mechanisms for modulating expressivity and efficiency.

Abstract: Compositionality in knowledge and language—the ability to represent complex concepts as a combination of simpler ones—is a hallmark of human cognition and communication. Despite recent advances, deep neural networks still struggle to acquire this property reliably. Neural models for emergent communication look to endow artificial agents with compositional language by simulating the pressures that form human language. In this work, we introduce CELEBI (Compressive-Expressive Language Emergence through a discrete Bottleneck and Iterated learning), a novel self-supervised framework for inducing compositional representations through a reconstruction-based communication game between a sender and a receiver. Building on theories of language emergence and the iterated learning framework, we integrate three mechanisms that jointly promote compressibility, expressivity, and efficiency in the emergent language. First, Progressive Decoding incentivizes intermediate reasoning by requiring the receiver to produce partial reconstructions after each symbol. Second, Final-State Imitation trains successive generations of agents to imitate reconstructions rather than messages, enforcing a tighter communication bottleneck. Third, Pairwise Distance Maximization regularizes message diversity by encouraging high distances between messages, with formal links to entropy maximization. Our method significantly improves both the efficiency and compositionality of the learned messages on the Shapes3D and MPI3D datasets, surpassing prior discrete communication frameworks in both reconstruction accuracy and topographic similarity. This work provides new theoretical and empirical evidence for the emergence of structured, generalizable communication protocols from simplicity-based inductive biases.

A Compresive-Expresive Communication Framework for Compositional Representations
Rafael Elberg
Felipe del Rio
Pontificia Universidad Católica, CENIA, i-Health Chile
Pontificia Universidad Católica, CENIA Chile f fidelrio@uc.cl
Sttaelleebergguc.c
Mircea Petrache
Denis Parra
Pontificia Universidad Católica, CENIA Chile
Pontificia Universidad Católica, CENIA, i-Health Chile
mpetrache@uc.c1
dparrasQuc.cl
Abstract
Compositionality in knowledge and language the ability represent complex concepts as a combination of simpler ones is a hallmark of human cognition and communication. Despite recent advances, deep neural networks still struggle acquire this property reliably. Neural models for emergent communication look to endow artificial agents with compositional language by simulating the pressures that form human language. In this work, we introduce CELEB (Compressive- Expressive Language Emergence through discrete Bottleneck and Iterated learn- ing), novel self-supervised framework for inducing compositional representations through reconstruction-basee receiver. Building on theories of language emergence and the iterated learning framework, we integrate three mechanisms that jointly promote compressibility, expressivity, and efficiency in the emergent language. First, Progressive Decoding incentivizes intermediate reasoning by requiring the receiver produce partial reconstructions after each symbol. Second, Final-State Imitation trains successive generations of agents imitate reconstructions rather than messages, enforcing a tighter communication bottleneck. Third, Pairwise Distance Maximization regu- larizes message diversity by encouraging high distances between messages, with formal links to entropy maximization. Our method significantly improves both the efficiency and compositionality the learned messages on the Shapes3D and MPI3D datasets, surpassing prior discrete communication frameworks in both reconstruction accuracy and topographic similarity. This work provides new the- oretical and empirical evidence for the emergence of structured, generalizable communication protocols from simplicity- based inductive biases.
communication game between sender and a
Introduction
In natural languages, compositionality enables humans communicate an infinite number of ideas using a finite set of elements [12). This principle enables speakers to ficxibly combine known words and structures to convey novel meanings, supporting flexible and generalizable communication across diverse and previously unseen contexts.
Cooresponning author
"Our oficial implementation can be found at http://gghhal.ooongggggrreeeeeer
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Works on the emergence of compositionality in language [34 35 37 have argued that opposing pressures are necessary for the natural selection of compositional languages between generations of speakers 20 3. On the one hand, successful communication requires high expressivity in order to usefully describe the world, allowing speakers to produce distinct messages for a wide range of meanings. On the other hand, models of emergent communication such as the iterated learning framework [4] state that natural speakers tend to minimize the complexity of languages through cultural transmission, implying that simpler and more compressive languages are more easily passed on to new speakers.
These opposite pressures of expressivity and compressibility thus generate a trade-off, highlighted in several works [35] 20 3 which argued be optimized by compositional languages, whereby languages evolve to maximize communicative efficiency remaining expressive enough to convey diverse meanings while being simple and structured enough to be easily learned and transmitted by successive generations of speakers. When either pressure is dominant, undesirable languages tend to emerge: (a) excessive language compression leads to degenerate languages where multiple meanings are mapped to the same messages making them ambiguous and thus hard to use, and (b) prioritizing expressivity alone produces in holistic languages i.c. languages where messages are mapped arbitrarily to meanings without respecting their structures, hindering their transmission across generations [55]
Enabling machines to generalize compositionally is thought to be crucial for them to quickly adapt to novel situations beyond their training experience 42 Inducing compositional behavior in neural networks remains a major challenge [39 30 59
33 A growing body of work suggests that a model's
ability to generalize compositionally is highly sensitive to its training conditions [40 14 60 41 15),
including factors such as data distribution, learning objectives, and task design. One promising direction is the study of emergent communication, where discrete languages evolve for coordination between independent neural agents.
In this work, build on the Lewis reconstruction game framework [88] to study emergent commu- nication in a reconstruction task. A sender encodes an image into discrete message and receiver reconstructs the original. Drawing on theories of language evolution [34 35). we develop a novel framework for inducing compositional communication grounded in simplicity bias.
We introduce CELEBI CCompressive-Eppresti Language Emergence through diserete Bottleneck and Iterated learning). Within this framework, we introduce three mechanisms in the learned communication protocol:
Progressive Decoding, in which the receiver makes reconstruction predictions after each incoming symbol, rather than waiting until the full message is received. This biases the system towards using intermediate reasoning steps, thereby imposing a pressure towards cower-compeexity and less holistic encodings. While this mechanism does not directly improve reconstruction accuracy, it yields more efficient and structured communication.
Final-State Imitation modifies the standard imitation phase in the iterated learning (IL.) framework of cultural evolution 37) Instead of imitating the entire message, the student is trained to reproduce only the final output of the receiver, effectively reducing the information transmitted across generations. This tighter generational bottleneck increases pressure for compressibility in the emergent communication protocol (64). thereby promoting the emergence of compositional structure as a necessary condition for successful transmission. Empirically, this leads to increased compositionality with minimal degradation in reconstruction accuracy.
Pairwise Distance Maximization regularization term, i.c.. as further pressure towards increased diversity in the emergent language, that encourages exploration during the imitation phase. This term pushes the student sender to maximize an approximate Hamming distance between messages in a given batch, finding the most "diverse" protocol. We prove that this regularization gives lower bound for entropy maximization 669, and an upper bound on the contrastive learning loss NT-Xteet
The opposing pressures induced by these three mechanisms impose a tight regularization on the complexity of the emergent language. These ingredients make explicit the pressures proposed in the cognitive science and linguistic literature to shape language evolution, thus aligning with the IL paradigm and leading to measurably higher compositionality.
2

---

2 Background
2.1 Problem setup: recovering compositional representations
In a formalism similar to [56], we model a dataset D = { = Gen G E 9), i.e. data are created via deterministic function GenX from set G of generating factors G. We have access to samples from D but not to GenX or to the structure of G, and the goal is to reproduce the set of xr i... generate images T that approximate the distribution of in natural metric such as EzeD MSE(x, }) where MSE is the mean squared error on pixels.
The factors G are assumed to have compositional structure of the form G -G1.......... ..]. in which G2 represent independent characteristics having finitely many possible values (as in the dataset of Sec. 4.1 for example). In this case, an efficient way reproduce elements of D is to impose that T is similarly generated as GenX' GG'), with generating factors G' giving a learned encoding of G.
As G has a large number of classes (one per combination of factor values) and we have single example per class, successful reconstruction relies on the compositional structure of G. Thus the goal to build framework for finding the optimized compositional encoding G' from observation of a small training set Derain D. The compositional nature G what makes reconstruction possible from small random Dirain. i.e. from data corresponding to small subset of classes. This would be of course impossible for non-compositional data (see $D for a formal treatment).
2.2 Approach: Lewis reconstruction game
We frame the above reconstruction problem in the form of cooperative game between two agents: sender and a receiver R. The goal of will be to build the reconstruction factors G' and the receiver R will map the factors to an image I. The two agents aim to create protocol that reconstructs to good accuracy. This setup fits in the general class of Lewis Reconstruction Games (LRG) (58) subclass of so-called Lewis Signaling Games (LSGs) 43
Within the framework of signaling games, the factors G' are interpreted as language which S uses to communicate to R as accurately as possible. The allowed message space is finite, i.c. G' M Vc, and take C fixed and large enough that |V|C > ( so that the size of the message space is not a limiting factor on the emergent communication. The following diagram summarizes our notations:
g GenX DCX 2
M= R4 X
Our main focus is on defining a IL process which favors the emergence of compositional language between S and R that accurately recovers x E D.
2.3 Base model architecture
We adopt standard emergent communication setup as described by Xu al. 71 see implementation details in B. Sender and Receiver implemented using the EGG framework (32). Input images are encoded into latent representations using a pretrained VAE visual backbone. The sender encodes this latent input into discrete message using an L.TTM and then transmits it through fixed-length communication channel. Message generation is made differentiable via Gumbel-Softme bottleneck (26). The receiver takes the full message as inputs in an LSTM, and produces latent vector, which is subsequently decoded into reconstructed image by the VAE decoder. Reconstruction loss is computed between the original and generated images using Mean Squared Error (MSE). Both VAE encoder and decoder remain frozen during training.
3 Methods
This section describes our proposed improvements to the EL framework 71 (cf. Figure Tap. Intuitively, our additions attempt to increase the language drift between iterations of speakers, in order to thoroughly explore the landscape of possible languages, while maintaining efficient and useful communication. For the latter, we introduce Progressive Decoding (PD), serve as anchor for the emergent language during interaction, and for the former we introduce Final-State Imitation (FiSI) and Parwise Distance Maximization (PDM), which aim to incentivize exploration during the
3

---

imitation phase. We expect the combination of these factors to show the strongest impact towards compositionality in the learned language, more so than the factors taken separately.
3.1 Progressive Decoding (PD)
Our first proposed enhancement the baseline communication protocol aims improve the efficiency of the emergent language determined by the Sender and Receiver, by condensing information in fewer message tokens. This type of efficiency has been argued to be necessary for the emergence of natural language properties
Standard reconstruction games typically optimize reconstruction loss computed only at the end of the communication phase, after the receiver has observed the full message [58], and as shown by Rita et al. [57], without incentive to efficiently the receiver only updates its internal state after the full message has been received. To improve communication efficiency, propose the following Progressive Decoding (PD) objective: the receiver generates a prediction after each received symbol, and our interaction phase loss explicitly weights the reconstruction error of each sub-message, follows.
the information received in the message,
Cinl
Ex
X MSE (x, (s
where (z) is the prefix of length of the sender's message, and À 2 1 is an expressivity hyperparameter that increases the weight of later reconstructions, This loss has the following effects:
Efficiency pressure: Loss ( rewards accurate reconstructions made early possible, favoring messages with shorter useful length.
Interpretable meb-messsges: The model is encouraged to structure messages such that interpreta- tion by R. is available for sub-messages, making each symbol accountable for partial reconstruction. This echoes findings in prior theoretical and empirical work on chain-of-thougee and compositionality 666 28
10 52 68, [54], which show that forcing intermediate semantic consistency or intermediate reasoning steps encourages models to show increased compositional behavior.
Tuning the efficiency pressure: Small À yields fast but potentially coarse reconstructions, while larger values favor detailed outputs at the cost of longer, possibly redundant messages, and in the limit A oo T becomes equivalent to using just the last term brackets, i.e., the full-length message reconstruction. In our experiments (see Section 5.2 we explore this trade-off empirically and show that À = 1.5 achieves the best performance across generalization, communication efficiency metrics, and compositionality.
3.2 Imitation Phase
Iterated learning is known to add compressibility pressure to emergent communication schemes 3 resulting languages that are "easy learn", thus favoring simple communication schemes that can be compressed: this is known as a communication (or generational) bottleneck. It is argued 34 that languages that remain relatively stable even when a learner only observes a small subset of the language of the previous generation need to be compositional.
Our case (and in fact, most deep learning applications of IL. differ from the original II. formulation 34., in that the transmission bottleneck does not constrict the amount of examples presented to the student, but rather constricts the information obtainable from these examples in order to perfectly reconstruct the language. This paradigm shift for transmission bottlenecks is at the heart of discrete bottlenecks [55 58 57 and noisy bottlenecks (56).
We next pass to describe our proposed imitation scheme and its motivations compared to previous methods.
3.2.1 Final-State Imitation (FiSI)
As usual in IL, we apply teacher-student regime initialize the next iteration sender Sp--1.

---

Our main proposed novelties for the imitation phase are as follows.
State space imitation: As opposed to other IL implementations 55
55 4
models
directed to match the student protocols from different iterations in the message space, but rather in the state space, using the frozen receiver network from the interaction phase. This change allows wider range of sender strategies, as the new loss is more permissive.
Final state reconstruction: During imitation phase use the following reconstruction loss, where R R S Sg are frozen receiver and sender from the previous iteration and S = Sar+1 is the newly trained sender:
16+1 = E, (S) p5 (x)))]
Our loss C only depends the state associated the full message, without testing reconstruc- tion on sub- sub-messages. This change allows the intermediate message tokens of the student sender (x); < to drift further away from the ones of the teacher mapping S; (I)); and thus more easily find different strategies with the same end result.
State space imitation justified as follows. Note that as message space discrete, there exists a value ón such that any two messages at distance < do in fact coincide. Then (Sp (x), St+ (2)) < d0 vice-versa is not always true: if R2 is not injective, any (1) E (R) (S; (z)) will preserve zero distance of Rt-images. If R2 has mild Lipschitz regularity assumption, this principle extends to the regime of small loss, showing that new loss is less restrictive the choices of S than the traditional message space loss. Thus, state space imitation enables wider sender strategy exploration without penalization losses.
always implies dx [R; (S) (x)), 1t7 ( (x))) = 0, while
(2))
1nd
True 2
Dec
Dec
Dec
R'
R
1
2
3rd
4
my
m
S
gi-1
5
7
Enc
Enc
Interaction
Imitation
(a) Overview our proposed architecture
(b) Qualitative example image decoding
Figure 1: (a) Overview of our proposed architecture. Interaction Phase: The sender Sgr and receiver Rn are jointly trained to minimize the reconstruction error between the input state æ and the predicted states {2'} }. by encoding x into message m Imitation Phase: A new sender Sa is trained to imitate the final predicted output of Rut (m,)), while also maximizing pairwise message diversity to encourage exploration. (b) Qualitative example image decoding: the receiver reconstructs the input image from the sender's message for each sub-mer message, giving series of reconstructions. The useful length of a message corresponds to the first reconstruction that has error below threshold >> O: in this example the last two message tokens are not useful in that they do not add to the reconstruction accuracy.
The choice of using just final states is justified because final state reconstruction allows greater freedom during imitation for the student sender. By only constraining the reconstruction of the full message, we allow much more freedom for the encoding strategy of S(x) at earlier sub-messages at each iteration (see Proposition E. for a formal statement and proof).
3.2.2 Pairwise Distance Maximization (PDM)
To further improve protocol diversity and promote exploration of the language space during the imitation phase, we introduce a regularization objective that promotes dissimilarity between messages within each batch. In addition to enabling a larger search space for optimal student strategies. Enhancing the diversity of student strategies also fits the general argument according to which compositional languages are the most resistant noisy language transmission [34 37 36).
5

---

Specifically, we approximate the Hamming distance by computing the position-wil -wise cosine dissim- ilarity between the probability distributions over symbols in each message | This encourages the sender to generate messages that are maximally distinct while remaining semantically aligned with the teacher protocol. The strength of this regularization controlled by a hyperparameter |3.
The resulting imitation loss is defined as:
C.m.
Cree Ge-1
Σ Si (x;), Sx1x(())) i.
(3)
N batch
where d(.,.) is the mean cosine similarity between corresponding symbol distributions.
In $ F44 we show that this regularization term provides a lower bound on entropy maximization
objectives [69], and serves as an upper bound for contrastive losses such as NT-Xent []. thus connecting our formulation to well-established principles in representation learning.
4
Experimental setup
4.1 Datasets
Shapes3D We tested our framework using the Shapes3D 550 dataset, which consists of colored images of 3D geometric shapes, with 6 underlying generating factors G: floor hue, wall hue, object hue, shape, scale, and orientation. The total amount of attribute-vann value combinations is 480, 000.
This dataset consists of colored images of a robot arm interacting with objects, rendered in controlled 3D scenes with 7 underlying generative factors G: object color, object shape, object size, camera height, background color, horizontal arm position, and vertical arm position. The dataset comprises 030 800 unique combinations of these factor values.
MPI3D
We also evaluated using the compositional split of the MPI3D dataset 23
For both datasets we use the compositional split from Schott et al. 622. which ensures all attribute values appear in training, yet some combinations reserved for the testing set (See $or proof that this situation would anyways hold for random train/test splittings with high probability).
4.2 Evaluation Metrics
Following Chaabouni et al. (9) we assess compositionality using Topographic Similarity (TopSim) 5 a widely used proxy for compositionality. TopSim measures the correlation between pairwise Hamming distances in message space and the generating factor space, capturing the extent which semantically similar inputs yield similar messages under a structured encoding.
To evaluate communication efficiency, we define the useful length (xx. which estimates the minimal prefix of a message necessary to achieve near-maximal reconstruction quality. Formally:
i (x) == min ( E f1.....,y) C} MSE (x. R_ (S,())))) s <c).
where C' is the maximum message length and (x) C the threshold is unmet.
We choose by viewing the loss distributions on both tested datasets and estimating a common plateau point for each. The full position-wise loss values can be found in the supplementary material.
Finally, approximate the expressivity of the communication protocol, we evaluate reconstruction quality using the MSE between the generated image and the original input.
We discuss the use of additional language metrics in I
5
Results
For all experiments, we report the mean and standard error of 10 random seeds. To assess the robustness of our findings, also perform permutation tests evaluate the statistical significance of
For methot encoded messages, this formulation is equivalent maximizing the Hamming distance.

---

Table I: Performance comparison across all experiments on the SHAPES3D and MPI3D datasets.
We report Topographic Similarity (TopSim t), Useful Length (Le 4) and Last Symbol MSE (4) for each ablation. The results grouped by the components of our proposed framework: Interaction, Imitation, and Regularization. The synergetic mixture of our proposed methods (PD+FiSI+PDM) consistently improves compositionality and message efficiency over the baseline and prior variants.
TopSim f
f2x10 +
Last symbol MSF. +
Full message reconstruction, no IL (Baseline)
0.244 + 0.002
10.0 1 0.0
0.212 : 0.003
Interaction
Progressive Decoding(PD) ours)
0.001
7.5
0.238 0.007
Message imitation (buselime)
0.257 + 0.005
9.3
0,200 0.007
Full state imitation
0.256 + 0.003
9.9
0.179 0.003
Imitation
Final-State Imitation (FiSI, ours)
0,283 + 0.003
10.0 + 0.0
0.176 0.002
Regularization
FD+IISI+ FiSII À 1.5
0.256 = 0,002
8.555
0.167
0.179 + 0.003
PD+FiSL+PDM (ours) À = 1.5
0.292 0.002
7.0 +: 0.155
0.194 0.005
(a) Shapes3D
TopSim f
f005x1002 4
Last symbol MSE
Full message reconstruction. no IL (Baseline)
0.133 = 0.001
9.3 0.002
0.013 0.0
Interaction
Progressive Decoding(PD, ours)
0.137 = 0.001
6.6 0.341
0.015 0.0
Message imitation (haseline)
0.135 0.001
9.733 0.029
0.016
Full stare imitation
0.137 = 0.001
9.923 + 0.020
0.018 0.0
Imitation
Final-State Imitation (FiSI, ours)
0.156 = 0.001
9.7 + 0.046
0.02 + 0.0
PD+FiSI+KoL.co À == 1.5
0.147 = 0.002
8,9 0.221
0.02 0.0
Regularization
PD+FiSL+PDM (ours) 1.5
0.153 = 0.001
9.0 + 0.167
0.02 0.0
(b) MP13D
the observed differences (see $T.Ip. Additionally, include qualitative reconstruction experiments in [ where we visually compare the outputs all evaluated methods across progressive decoding steps. Further implementation details are provided in S
5.1 Progressive Decoding reduces the message's useful length
We begin by evaluating the effect of incorporating PD during the interactive phase of training. As shown in the first two rows of Table T this modification leads to a 25-29% reduction in the useful message length, indicating more efficient communication. These gains achieved without compromising expressivity on MPI3D and with only a slight compromise on Shapes3D, as indicated by stable reconstruction quality in the former and minor degradation in the latter.
To further promote efficient and structured communication, we introduce a geometric penalty term that penalizes reconstruction error more heavily when longer messages are used. As shown in Figure 2 increasing À up to values near 1.5 consistently improves reconstruction quality and useful length. indicating higher expressivity and efficiency. However, when À becomes too large, the reconstruction error incurred the end of the message dominates the loss function, effectively nullifying PD. We found that setting X = 1.5 yields favorable results in efficiency and expressivity, while maintaining high compressibility.
5.2 Final State Imitation increases TopSim
Second, investigate the impact of final state imitation on the structure of the emergent language.
In these experiments, we compare our approach to a baseline model without imitation and to a standard IL variant that imitates the teacher's message directly-a strategy commonly adopted in prior work for optimizing this phase [55] 56] 44]
As shown in the imitation section in Table I in the MPI3D dataset, we find that our final state imitation method consistently enhances the compositionality of the emergent messages. outperforms the non-IL baseline and surpasses message-level imitation significantly ($ J.I.
7

---

XPB0 = 1.95
3hupedD x
Final rec MSE (Shapes
MPIED
Shapes30
Moan The Sim
0.4
10.0
0.4
0.03
0.3
Usetal
7.5
0.02
0.2
5.0
0.2
0.01
4m
2.5
0.0g
10
0.00
2
10
Lambda
Lambda
Lambda
(a) Useful Length
(b) Reconstruction MSE
(c) TopSim
Figure 2: Effect of geometric penalty À on emergent communication. We introduce a geometric weighting term À that penalizes reconstruction error more strongly for longer messages. Increasing initially improves useful length (a). However, once a certain threshold is met, useful length and reconstruction error (b) increase. Values close 1.5 of strike favorable balance- achieving efficient and expressive communication.
More broadly, our results support prior findings that IL promotes greater structure in emergent communication protocols 44 55]. They also underscore the value of shifting the learning objective from message imitation to image reconstruction, which enforces a tighter generational bottleneck by requiring the student recover the intended output without access to the original message.
Moreover, this setup increases flexibility in protocol exploration by constraining the student only to the reconstruction target, not the exact message form. Permitting generational variation in message structure while preserving semantic fidelity, promoting diversity without sacrificing expressivity.
Together, these factors exert pressure toward discovering more systematic and compositional proto- cols structures that are inherently more transmissible and robust across generations.
5.3 Pairwise Distance Maximization increases compositionality and efficiency
We evaluate model that combines all previously identified best-performing compo- nents-speciffiallly,
1.5, final-state imitation, PD and further introduce an entropy-based regularization term using PDM. We compare this with the KoLeo entropy estimator proposed by Sablayrolles et al. [61].
When examining the effect of incorporating PDM, Table I that PDM outperforms both FiSi and PD fostering structure in the message. On the Shapes3D dataset, PDM leads to a marked improvement in TopSim and lower useful message length, indicating more efficient and structured communication, albeit with slight increase reconstruction error. For MPI3D, this configuration doesn't It achieve the highest TopSim, albeit close to the best, and increases useful length and leads to marginally higher reconstruction loss compared to using only PD. This highlights a trade-off between expressivity and compressibility, in line with prior theoretical accounts on communication efficiency and compositionality [53 49).
When comparing PDM to the KoLco estimator, Regularization section in Table I we find that on Shapes3D, PDM achieves superior TopSim and shorter useful length, with slight reconstruction gains. On MPI3D, KoLeo leads to a lower useful length but exhibits reduced compositionality, with both regularizers showing similar reconstruction performance.
5.4 Comparison to disentangled representation learning
Following the comparison made by Xu et al. 7 we test our generated messages against standard self-supervised disentangled representation learning frameworks, namely 3- 3-VAE and 3-TCVAE, as well as the baseline VAE+EL (7I). For the continuous models, vector (corresponding to the predicted means μ1) and evaluate it using the DCI disentanglement score [8] In addition, train a two-layer MLP to predict ground-trun -ruth generative factors, reporting RMSE for continuous factors and classification accuracy for categorical ones.
extract the first half of the latent
For the discrete models, we use the predicted messages for each image. We calculate the DCI disentanglement as-is, and use an embedding matrix transform discrete messages into the same dimension the continuous vectors for the MLP. Training conducted three subsets of size 1000

---

Table 2: Disentanglement metrics for the Shapes3D and MPI3D datasets. We report the DCI score to evaluate disentanglement quality, and the RMSE and classification accuracy of a linear probe trained on the latent representations (both message and vector embeddings) assess their suitability for downstream tasks.
Dataset
Model
Disentanglement f
RMSE +
Acc 1
8- VAV
0.045 + 0.004
2.460 + 0.144
0.702 + 0.025
3-TCVAE
0.043 0.005
2.378 + 0.123
0.721 + 0.028
Shapes3D
VAE + EL
0.108 + 0.000
3.168 + 0.027
0.343 + 0.019
VAE+ + CELEBI
0.112 0.001
2.932 + 0.045
0.453 + 0.024
3-VAE 3-TCVAE
0.031 + 0.001
21.819 = 0.111
0.581 + 0.006
0.031 0.001
21.854 + 0.188
0.582 1 0.005
MPI3D
VAE + EL
0.114 = 0.002
15.971 = 0.301
0.529 = 0.006
VAE + CELEBI
0.137 0.002
14.82 + 0.012
0.542 + 0.002
of the training set and averaged. Evaluations are conducted on the test sets of the compositional splits for the Shapes3D and MPI3D datasets.
Across both datasets, our method (VAE+CELEBI) demonstrates consistent improvements in disen- tanglement over baseline methods. On Shapes3D, we observe substantial increase DCI scores compared to continuous models and the discrete baseline (see statistical significance tests in $ sp.2). However, both discrete models fall behind the continuous baselines in downstream accuracy and RMSE, suggesting a trade-off between representation interpretability and usefulness in downstream tasks, which is consistent with previous work indicating that discrete representations may be less accessible simple classifiers (as shown in [72), and that inductive biases for compositionality do not imply disentanglement [5]].
On MPI3D, our method slightly under-performs when compared to the continuous baselines in categorical accuracy, however greatly outperforning all baselines in continuous regression. We speculate that this discrepancy arises from the lower correlation between pixel-level input statistics and continuous generative factors in MPI3D as opposed to Shapes3D, which may favor communication- based models over purely reconstructive.
6 Related Work
The use of language reference games (LRGs) in conjunction with iterated learning for the purpose of learning self-supervised representations remains underexplored. Xu et [71 compare an emergent language autoencoder-ccomeee comprising visual backbone and an LSTM-bass based sender and receiver- with disentanglement frameworks such as the 3-VAE and 3-TCVAE. They find that the representations induced by the emergent language model generalize better in downstream tasks, highlighting the potential of symbolic communication as a compositional bottleneck.
Many previous works have explored using emergent language to induce compositional behavior BI 17 2
16 g Several works utilize a referential LSG in conjunction with IL, using predicted messages generational bottleneck [55] [4]. ) Alternative bottlenecks for emergent language have been proposed, including simplicial embeddings [56] 19], code-books (74] and noisy channels (65].
Our message tokens can also be considered as roughly comparable to the slots in slot-attention
architectures such as (22] 24 46 27 45 21 48 4 [. however note that among other differences,
the slot-attention typically additive decoding layer 7 38 does not compare directly to our proposed
Progressive Decoding.
Rita et al. 17 propose To replicate the Zipfian distributions observed in natural languages 76 29
the LazImpa framework. The Lazlmpa Impatient listener loss is similar to our proposed Progressive Decoding, in that they both aim to induce incremental informativeness and efficiency by reconstructing the input at every timestep. However LazImpa is optimized for a referential game task (and not reconstruction task as ours) and uses linear length penalization in conjunction with cross- entropy loss. Our proposal instead uses the actual reconstruction loss and the length penalization exponential and based on tunable parameter X.

---

7 Limitations and Future Work
Our experiments conducted exclusively the synthetic datasets Shapes31) and MPI3D 6 23
which feature clean, disentangled generative factors and lack observational noise. In future work, we plan to extend our the same principle of making explicit pressures towards complexity reduction of underlying representations to natural image datasets where factor supervision may be unavailable or weakly defined. Furthermore, we aim to formalize the notion of language compression defined in IL in terms of Kolmogorov complexity, for representing datasets with more complex compositional structures.
Moreover, it debatable whether MSE is the most appropriate choice for the self-supervised loss function, especially in the context of natural image data. While MSE is simple choice, future work may explore reconstruction losses that better align with human visual perception, such as perceptual similarity metries based on deep feature embeddings (e.g.. LPIPS 73 adversarial and contrastive losses that promote more semantically structured representations.
Additionally, we froze the visual backbone across all EL model variants trained on the same dataset. While this ensures controlled comparisons within each dataset, it may introduce an initialization bias when comparing with continuous baselines, where the encoder is jointly optimized. Although our results remain statistically significant, future work could investigate whether co- co-traine to-training the vision encoder yields improved alignment between symbolic and perceptual representations, or leads to new trade-offs in compression and expressivity.
Finally, our experiments were restricted to compositional generalization metrics and reconstruction- based evaluation. Other facets of emergent communication, such as robustness and interpretability. are left for future work. We believe our framework offers a strong foundation for such extensions and a tractable setting to explore the interplay between compression, efficiency, and generalization.
Conclusion
We introduce the CELEBI framework with three novel mechanisms for enhancing compositional learning in frameworks, as well mathematical justification for their design. First, with Progres- sive Decoding (PD) we reward informative communication at each step of the message, creating an inductive bias toward efficient and distributed representations. We show that PD promotes both message compressibility and compositional alignment.
Second, in Final State Imitation (FiSI) in II., the student is trained reproduce the final prediction of the teacher, rather than the message itself. This shift enables greater exploration of the message space while preserving semantic consistency. We provided theoretical motivation and empirical evidence that this approach yields more expressive and compositional protocols, outperforming traditional message imitation in TopSim and reconstruction metrics.
Third, our regularizer based on Pairwise Distance Maximization (PDM), which provably approximates entropy maximization over messages, serves a practical inductive bias for promoting diversity and structure in emergent languages, particularly during the imitation phase.
Together, these contributions enrich the II. framework, and give new insights into how training dynamics and inductive pressures shape the emergence of language-like representations. As confirmed by our empirical findings, the proposed methods lead to more compositional and generalizable communication schemes.
Acknowledgments and Disclosure of Funding
This work was supported by ANID Chile, Fondecyt Regular grant 1231724, as well research centers of excellence with code FB210017 (Basal CENIA), ICN2021_000 004 (Millenium iHealth), and ICN17_(M (Millenium IMFD).
References
[1] Kaan Akan and Yucel Yemez. Slot-guided adaptation of pre-trained diffusion models for object- centric learning and compositional generation. In The Thirteenth International Conference
10

---

Learning Representations, 2025. URL
[2] Michal Auersperger and Pavel Pecina. Defending compositionality emergent languages. In Daphne Ippolito, Liumian Harold Li. Maria Leonor Pacheco, Danqi Chen, and Nian- wen Xue, editors, Proceedings of the 2022 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 285-291, Hybrid: Seattle, Washington + Online, July 2022. Association for Computational Linguistics. doi: 10.1665ww1/202.naaclsvv... https://aclantholl.
URL
arg/2022 naacl-srw.35.
[3] Aaron Beppu and Thomas Griffiths. Iterated learning and the cultural ratchet. In Proceedings of the annual meeting of the cognitive science society, volume 31, 2009.
[4] Jack Brady, Julius von Kügelgen, Sebastien Lachapelle, Simon Buchholz, Thomas Kipf, and Wieland Brendel. Interaction asymmetry: A general principle for learning composable abstrac- tions. In The Thirteenth International Conference on Learning Representationss 2025. URL
httt::://openreveee.neeee//
[5] Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial Life, 12:229-242, 2006. URLhtttss:////
sseanticcchol org/Corpus 1391836
[6] Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://ggthhbbcoomddeeppnlllllls dataset/, 2018
[7] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint 19Xiv:100.11::: 2019.
[8] Rahma Chaabouni, Fugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In ACL 2020-8th annual meeting of the Association for Computational Linguistics, 2020.
[9] Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International Conference on Learning Representations, 2022. URL /ttttttt///perrreee
net net/fforumiid-NNGBfDDIW
[10] Ayush K Chakravarthy, Jacob Labe Russin, and Randall O' O'Reilly. Systematicity emerges in transformers when abstract grammatical roles guide attention. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 1-8, 2022.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. URL https: //axxiv..///ivva/aa org/ abs/ 05709
[12] Noam Chomsky, Aspects of the Theory of Syntax, volume 11. MIT press, 2014.
[13] Henry Conklin and Kenny Smith. Compositionality with variation reliably emerges in neural networks. In The Eleventh International Conference Learning Representations, 2023. URL https://openrevies
fett/formmidd=a.... Yz26v1177V
[14] Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. Meta-learning to compositionally generalize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3322-332 -235,, 2021.
[15] Felipe del Río, Alain Raymond-Sáen Daniel Florea, Rodrigo Toro Icarte, Julio Hurtado, Cristián Buc Calderón, and Álvaro Soto. Data distributional properties as inductive bias for systematic generalization. arXiv preprint 2502:2550..24999 2025.
11

---

[16] Kevin Denamganai and James Alfred Walker. On (emergent) systematic generalisation and compositionality in visual referential games with straight-througee gumbel-softmat estimator. CoRR, abs/2012. 10776, 2020. URL hww.tttps///// htops:://ariivv org/ abs/ 10776
[17] Roberto Dessi, Bugene Kharitonov, and Marco Baroni. Interpretable agent communication from scratch( generic visual processor emerging on the side). In Neural Information P Processing Systems, 2021. BtRL::/t::///ppp semanticscholar org/ htte/CorpusDD... : 235368124
[18] Cian Eastwood and Christopher K I Williams. A framework for the quantitative evaluation of disentangled representations. In Sixth International Conference Learning Representations (ICLR 2018), May 2018. URL http::tttt//// //iclr.../////// cc/Conf wwwfeeeeeneee//e 2018 6th International
Conference on Learning Representations, ICLR 2018; Conference date: 30-04-2018 Through 03-05-2018.
[19] Rafael Elberg, Denis Parra, and Mírcea Petrache. Long tail image generation through feature space augmentation and iterated learning. In LatinX in Al at Computer Vision and Pattern Recog- nition Conference 2024, LXAI at CVPR 2024. Journal of LatinX in Al Research, June 2024. doi: 10.52999/xai2224061774 URL http://dx http:://xx.ooo..... org/ 10. 0055911/2xa12222001177
[20] Eric Elmoznino. Thomas Jiralerspong, Yoshua Bengio, and Guillaume Lajoie A complexity- based theory of compositionality, 2024. URL httts://: //axiiv wwwp:/vvvorg////a//a/eee arg/abs/2410..1 14817
[21] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Genera- tive scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020. URL https://openrreiiew..eee.
id=BkxfaTVFwH
[22] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in neural information processing systems, 29, 2016.
[23] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentan- glement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. httpssts:: http:httt//:/ooceeee /ttpt:ess///oooeiingg/ neur www.ips.ccccmmm ctt.ppppp.ppppppppeeeeee. htcppaper/229//fleee. 48744444661112114444441188181113
pdf
[24] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International conference on machine learning, pages 2424-2433 PMLR, 2019.
[25] Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. Advances in neural information processing systems, 30, 2017.
[26] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2017. URL URI.tttps::// lhttps://arxivv http://..wrrraa//////a//e// 01144
[27] Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Representations.
[28] Yichen Jiang and Mohit Bansal. Inducing transformer's 's compositional generalization ability via auxiliary sequence prediction tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2253-6265, 2021.
[29] Jasmeen Kanwal, Kenny Smith, Jennifer Culbertson, and Simon Kirby. Zipf's law of abbrevia- tion and the principle of least effort: Language users optimise miniature lexicon for efficient communication. Cognition, 165:45-52, May 2017.
12

---

[30] Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashu- bin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2019.
[31] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimization in emergent languages. In International Conference Machine Learning, pages 5220-5230. PMLR, 2020.
[32] Eugene Kharitonov, Roberto Dessi, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: a toolkit for research on Emergence of lanGuage in Games. https://githhnn com/ 2021.
facebookresearch/
[33] Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based on semantic interpretation. In Proceedingy of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, 2020.
[34] S. Kirby. Spontaneous evolution of linguistic structure an iterated learning model of the emergence of regularity and irregularity. Evolutionary Computation, IEEE Transactions on, 5: 102 110, 05 2001. doi: 10.1109/4235.918A)
[35] Simon Kirby and James R. Hurford. The Emergence of Linguistic Structure: An Overview of the Iterated Learning Model, pages 121-147. Springer London, London, 2002. ISBN 978-1-4471-06622 O. doi: 10. 10.1007/978-11444 1-4777063--47 6. URL htLhhtss://doo org 10. 1007/ 118-144471-6633-00
[36] Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution language. Current opinion in neurobiology, 28: 18:1108-112 114, 2014.
[37] Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. Compression and communication in the cultural evolution of linguistic structure. Cognition, 141:87-102, 2015. ISSN 0010-0277. doi: https://oooorr//000..00///ooooon
URL https
//www.scienne www.ssciencedirect com coo/sccincce http://cc/ccccccoo//////t/////// http:///ww........///////////0/0
[38] Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien Additive decoders for latent variables identification and cartesian-product extrapolation. Advances in Neural Information Processing Systems, 36:251 12-25150, 2023.
[39] Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-tor sequence-to-sequemee sequennee recurrent networks. In International conference on machine learning, pages
PMLR, 2018.
[40] Brenden M. Lake. Compositional generalization through meta sequence-to-sequee learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. ('Alcé-BBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 9791-9801. Curran Associates, Inc., 2019. URL http::/papers. nips cc/paper/ 1172-compositional/. seammmal-geeealizziioro sag.r/aaaaaaa.otoooooooaoa-aaare -o-eaa----eeeee------------ee-- Learning.. pdf
[41] Brenden M Lake and Marco Baroni. Human-like systematic generalization through a meta- learning neural network. Nature, 623(7985): 15-111,, 2023.
[42] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[43] David Kellogg Lewis. Convention: A philosophical study. 1969.
[44] Fushan Li and Michael Bowling. Fase-of-teaching and language structure from emergent communication, 2019.
02403
[45] Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination object-centric world models. International conference on machine learning, pages 6140-6149 PMLR, 2020.
13

---

[46] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
URI. http:://ppparreeeee.eeeeeeoorrrr
[47] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114 2019.
[48] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances neural information processing systems, 33: 11-5--555335 2020.
[49] Diana Rodr' Luna, E. Ponti, Dieuwke Hupkes, and Elia Bruni. Internal and external pressures on language emergence: Least effort, object constancy and frequency. In Findings, 2020. URL wttptttt://// //api. www.ssttitccccoolaaa........ hwtp:iiicsccccaoaoooooooooooeeee Carg/CorrpssD... 198921559
[50] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset. http:///tttbbbcoom/eemmniillists
2017.
[51] Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. Internationaa Conference on Learning Representations, 2021. URI. htttps:://pperee///// wtwpeenrrweeeeeeeee/reeeee het:/forrmmdd==gg8444433333
[52] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2022. URL
[53] Steven T. Piantadosi, Harry Tily, and Edward Gibson. The communicative function of am- biguity in language. Cognition, 122(3):280-291, 2012. ISSN 0010-0277. doi: https:/doi. org/ 10. 100.1001/i..ognttooo0101.......0
URL htt.h:ttp///// www. wcw.eeeiienccerc....mmmmm ceaecct.ccmeeee ctwmccomm/cccoo.ccoeeccc cemm/science/
brr:ccleepp1/$0000077100099
[54] Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
[55] Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In Internatioonal Conference on Learning Representations, 2020. URL
[56] Yi Ren, Samuel Lavoie, Michael Galkin, Danica J Sutherland, and Aaron C Courville. Improving compositional generalization using iterated learning and simplicial embeddings. Advances in Neural Information Processing Systems, 36, 2024.
[57] Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. "LazImpa": Lazy and impatient neural agents learn to communicate efficiently. In Raquel Fernández and Tal Linzen, editors, Proceedings of the 24th Conference on Computational Natural Language Learning, pages 335-343, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.conll-11 1.26. URL /ttttpss:/allaaatoollloo www.aaanthhoooooooooooooooooooee wtg//020002200. con1l-1.26.
[58] Mathieu Rita Corentin Tallec, Paul Michel, Jean-Bastien Grill, Olivier Pietquin, Emmanuel Dupoux, and Florian Strub. Emergent communication: Generalization and overfitting in lewis games. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL /ttte:://operrevier net/ ggguuiidggghhHbbfuuu
[59] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. Advances in Neural Information Processing Systems, 33, 2020.
14

---

[60] LE Ruis and Brenden Lake. Improving systematic generalization through modularity and aug- mentation. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022.
[61] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vectors for similarity search. In International Conference on Learning Representations, 2019. URL
https://openrevia net/forumm ? - Sdd-SkGu22R5tm
[62] Lukas Schott, Julius Von Kügelgen, Frederik Träuble, Peter Vincent Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf. Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. In International Conference on Learning Representations, 2022. URL
wwwpi/rreiiiieeeeoorreeee net/forum?
id 9RUHHlladghi
[63] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 815 823, IEEE, June 2015. doi: 10.1 00.11099e1::2015.29866.. URL
http://åx... doi. org/ 10. 1109 CVPRR. 2015 7298682
[64] Kenny Smith, Simon Kirby, and Henry Brighton. Iterated learning: A framework for the emer- gence of language. Artificial Life, 9(4):371-386, 2003, doi: 10.1 10.162/0064460333244433
[65] Ryo Ueda and Tadahiro Taniguchi. Lewis's signaling game as beta-vae for natural word lengths and segments, 2024. URL
04453
[66] Petar Veličković, Rex Ying, Matilde Padovano, Raia Hadsell, and Charles Blundell. Neural execution of graph algorithms. In International Conference on Learning Representations, 2020.
URL
httpirrrriiiieeeefounnnmoo mett/forummid-Skgk00E..
[67] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SeiPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272. 2020. doi: 10. (0008//41192--19 0686-22.
[68] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[69] Ronald Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3:241-, 09 1991. doi: 10. 10.1080095540099 1889465577
[70] Ronald J Williams. Simple statistical souddienn-fllooinng algorithms for connectionist reinforce- ment learning. Machine learning, 8:229-256. 1992.
[71] Zhenlin Xu, Marc Niethammer, and Colin Raffel. Compositional generalization in unsupervised compositional representation learning: A study disentanglement and emergent language, 2022. URL
URL https::///// //ta://..//iv//// .rg. abs//21b22211.... .00122
[72] Zhenlin Xu, Marc Niethammer, and Colin A Raffel. Compositional generalization in unsu- pervised compositional representation learning: A study on disentanglement and emergent language. Advances in Neural Information Processing Systems, 35:25074-25087, 2022.
[73] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018 doi: 1..1099/CVPR.2218.0000
[74] Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning improves compositionality in large vision-language models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 13785-13795, 2024.
15

---

[75] Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks, 2022. /Rtttss:: org/ 2atbb//22200000 00155
[76] George K. Zipf. Human Behaviour and the Principle of Least Effort. Addison- Addison-Wesley, 1949.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims and contributions are clearly stated in the abstract and introduction, and backed by theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
The paper contains separate limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peciiiatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are rigorously proven either in the main text or in the supplementary material, and justifications of the result's hypotheses are given.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All implementation details, including hyperparameters, used libraries and experimental details, are included either in the main text or in the supplementary material. We plan release the all code upon acceptance.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our full code and details to reproduce our results can be found in the oficial repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptaips.cc//uulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
19

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We use and disclose publicly available splits of known datasets, and specify all training hyperparameters either the main text the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We disclose the amount of seeds used in each experiments, and show standard error for the results in figures and tables.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the computational resources used for this paper in the Appendix.
20

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't Iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpt:://///// httttp:://euuiis.... http:///warrr..cccc/a/aa/a//aaaa
Answer: [Yes]
Justification: We have reviewed the code of ethics and have confirmed that our research complies with every applicable requirement.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: We do not foresee any significant societal impact, as the paper explores principles for compositional learning and not its applications to society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
21

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve models with high risk of misuse, as explores simplified models for the sake of finding new principles for enhancing compositionality of representations based on an iterated communication game setup.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets are properly credited and used in accordance with their licenses: Shapes3D and MPI3D are under CC-BY 4.0, and the EGG framework is released under the MIT license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.co///aaooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
22

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not use crowdsoureing experiments or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper's results were not based on, or aided by, research using LLMss.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy (httpa://neurips.c http://rrrriccccoooommeeeeee//// for what should or should not be described.
23

---

A Background
A.1 Iterated Learning
The II. model of cultural evolution 3d proposes to emulate the emergence of language through the interactions of adult and new learning agents. the original formulation, a space of pairs of signal + meaning is randomly generated, and in each iteration, a learning agent is partially exposed to the language learned by the adult agent in the previous iteration. The learning agent becomes the adult agent for the next iteration, and transmits a subset of its learned "language". By iterating this process, the initial random language gains structure and observable properties, such as compositionality.
This framework has been widely extended to include an additional task [4 56 75], instead of simply reshaping the language. Under this formulation, the training regime be separated into two parts: Interaction (Interaction), where an agent is optimized to solve the additional task, and Imitation (Imitation), where newly initialized agent "learns" from the previous agent through shared learned language.
A.2 Lewis reconstruction game
The Lewis reconstruction game (LRG) 58 is special case of the Lewis Signaling Game (LSG) [43] framework, which two agents cooperate to reconstruct an observed object. Specifically, sender agent parameterized by observes
object x from object space X' and produces message m. E M,, where M = V* is the set of all possible strings formed by concatenating symbols drawn from a finite vocabulary V. A receiver agent parameterized by observes m and produces reconstruction x'. Both models are optimized to minimize a reconstruction loss 2n)),x").
Importantly, in the original LSG, the receiver is tasked with predicting æ from finite set of states, whereas this formulation, the problem becomes a regression task, and can be a continuous space.
B
Implementation Details & Hyperparameters
B.1 Baseline Model
Sender and Receiver: Both consist of single-layer LSTM with an embedding size of 64 and a hidden size of 256. Outputs are passed through two-layer MLP with ReLU activations.
Communication Channel: The vocabulary size is set to |V| 15 and messages are composed of C' = 10 discrete symbols. These values are chosen such that the amount of posible messages (5.76 x 10'2) greatly outnumbers the amount of states. The Gumbel- Softmax bottleneck is used without traight-thhouuh estimation.
VAE Backbone: The VAE is implemented using the disentanglement lib [47 python library, using the default setting of two convolutional layers of 32 filters, with kernel size and stride 2, two convolutional layers of 64 filters, with kernel size 2 and stride 2, each with ReLU activation, as well linear layer of size 256 and ReLU activation and output layer with size 128 for concatenated jas and logas. The encoder is mirrored replacing convolutional layers with transposed convolutions, and an input layer of size 128 and ReL.U activations on all layers save the output.
VAE Training: The VAE is pretrained for 15 epochs using the Adam optimizer with a learning rate of 1 x 10-3
The interaction-imitatin games were run for a maximum of 100 iterations, alternating one full epoch in each phase per iteration.
We used early stopping based on the validation MSE of the final reconstruction, starting from epoch 5, with minimum 8 of 1 x 10 and patience of 5. We used the Adam optimizer with default parameters and learning rates of x 10 and 1 x 10-4 for Shapes3D 6 and MPI3D (3], respectively, as we found the training for the latter became unstable higher learning rates.
We found that Interaction required smaller batch sizes to converge, whereas the Imitation phase PDM becomes better entropy approximation at higher batch sizes, however scaling quadratically
24

---

in computation. Therefore, we set the Interaction and Imitation batch sizes to 256 and 512, respectively.
To implement Imitation we defined a receiver R. and two senders Son and Sen with their own optimizers, After participating in an Interaction iteration, o² and w are frozen and used to train Sgz described in $3.2 After Imitation, the weights @2 are copied to #1, and Φ2 is reset. Importantly, the optimizers for Φ1 and d2 are never reset nor copied between them.
C Compute Resources
All experiments were conducted on our internal laboratory cluster, using NVIDIA A40 GPUs with 48 GB of VRAM. Each job was allocated single GPU with GB of VRAM usage on average. alongside access to a CPU with 20 cores, 118 GB of RAM, and local SSD storage for datasets and model checkpoints.
The experiments reported in this paper comprise 360 training runs, totaling approximately 180 GPU hours. Individual runs varied in length from 15 50 minutes, depending early stopping criteria. These runs represent the finalized experiments whose results are presented in the main text.
In total, the research project required 2.310 experimental runs over the course of development, amounting to approximately 245 GPU days (5,880 GPU hours) compute time. This includes preliminary experiments, ablations, and failed runs that were instrumental to model and protocol design but are not individually reported in the paper.
C.1 Useful length E
To define the useful length for both datasets, we observed reconstruction loss distribution over all positions in the message (see 3 for different À values, and estimated an average plateau point. For MPI3D we used a more conservative and precise threshold, as the reconstruction error varied more between methods.
Additionally, we analyzed the useful length graph for multiple values, and discarded those values where most models converged to either maximum minimum possible length (see 4
D Recovering compositional structure from partially observed data
For the sake of clarity of treatment, assume here that G ......, Gn]] in whichi factor G, has the same number N of possible values, and as consequence 9 = NJ". Our results will extend directly to the general case as shown in Rmk. D.4
We work under the simplified deterministic generation hypotheses, in which our dataset is D GenX(g) X' in which GenX deterministic injective function.
Our main result the following:
Theorem D.1. Assume that G = [1 N7 and that GenX G - X' is an injective function, and that Dirain S D has cardinality [Derain] p some p E (0, 1) such that pN" is an integer.
We assume that the compositional structure of G is such that the following holds:
(A) For some k < if we observe set of data whose generating factor combinations feature all possible combinations of generating factor values (Gi Gi for all choices of indices
1 S i ... < is S then this is sufficient to reconstruct G to good accuracy.
Then as | + oo the probability that Dtrain is sufficient to reconstruct G tends to 1.
Here (A) makes rigorous the common assumption that having training set that features examples for diverse enough combinations factors is sufficient for reconstructing all combinatorial generators G.
Note that in this work we discuss
what features the pre-trained encodings G -+ and what requirements on the composi- tional structure of G would allow actually guarantee this assumption,
25

---

Modas
2--0.50
0.6
0.2
A-1.25
2.-1.50
A-2 00
LOS5
0.4
Positton
Position
Postton
Madas
0.02
Paziton
1.1.25
7-1.50
A-1.75
A-2.00
0.06
0.04
Posttin
Posiioo
Posion
Postcon
Figure 3: Loss distribution positions and À values for (a) Shapes3D and (b) MPI3D
t-0.3
E-0.02
Mean useful length
Mean useful length
E=0.2
c=0.019
E-0.15
c=0.0185
Lambda
Lambda
Figure 4: Useful length values for different c and À values for (a) Shapes3D and (b) MPI3D
2. alternatives for rigorous definition of what is meant by "to good accuracy" in the statement of assumption (A).
Once the above points are settled and defined, we can replace (A) by an explicit requirement. Since however these points seem like an ambitious whole research line and far beyond the scope of this work, we leave the definition of "to good accuracy" and the proof of reconstruction to future works.
Thus, introducing (A) allows of a full theory of compositional reconstruction.
prove fully rigorous statements such Thm. D.3 in absence
Nevertheless, make the following remarks:
26

---

There is natural trade-ofi off between assumptions on how big Dirain should be, vs, how complicated the compositional reconstruction should be allowed be for reconstruction to be feasible: the richer set of such combinations we require Dtrain to contain (thus making more restrictive assumptions on Drrain), the lower is the requirement for our encoding
framework and for the compositional structure for (A) to hold.
It seems natural to conjecture that if assumption (A) does not hold for large value of k, then there no hope for the compositional reconstruction to be achievable. Thus lowering the requirement on "good enough accuracy" to make (A) hold seems the only avenue of future research.
Theorem D.I is consequence of below Theorem D.3 that proves that assumption (A) holds with p is bounded away from zero. What
probability tending to as |g] + co and |Dtrain Theorem[D.3 D.3 says is that for large sets of generators, it will become overwhelmingly unlikely that Divain does not contain enough data to reconstruct the generators.
D.1 Observing Dtrain allows to observe finite statisties of the factors with high probability
Our first result for this section is that when Ptrain C D is chosen uniformly at random amongst sets of size p|P|. for [D] |g] large enough, we find that with overwhelming probability all possible values of each of the factors GI...., G,, are achieved by some elements of Drrain. We state the result in Proposition D.2 whose proof in our view contains the main ideas useful also for the full result of Theorem D.3 which can be considered our main technical result for this section.
The first result is as follows:
Gnl: (Vi S mn)1 G. N} [1 N1 and that GenX : G + X is an injective function with image 'D as above. Denote by Itrain == "eera-t(Drrain) the generating factors related to the training set, and assume that Dirain C D is chosen uniformly at random amongst subsets of cardinality plD). for some p (0, 1) such that pN is an integer. Then
Proposition D.2. Assume that for N 2 2 we have G =
1 PM
P( S mEj S N)(GG = -G1. .......GG].. Girain) G. + j} nN 1
(4)
N
in particular at fixed p. the above event has probability tending to zero as N - Co.
Thus, an equivalent formulation of the last part of the above proposition, we have that the limit of large |g] = N the probability that each generating factor G, takes all of its possible values on some generated element of Gerain tends 1.
Proof. The possible choices of Grrain C G of cardinality p|9| = pN are (pN")![[1 N)N Then, calling Ag the set of elements G E Girain such that G, * j. we find that |Ayy = N (N 1)
(N")
Nu N-1
N so that the number of possible choices of Strain S Aij of cardinality p.V is given by
(N" N-1
Now we bound the probability of the event on the left hand side in 4 via
(PN") N -p)!!
union bound over the Nn possible choices of Aig for S i I n, 1 I & N, so that
3i < n. N, YG = GI..... G.] Gerain: }
(N" NyNN-!
nN
(ND))
(pN*)I(N" N-1 P))!
G jj
No N-1
N N-1
No N pN +1
nN
N
N"
N"-pN" +1
nN
1
in which in the first line we used union bound together with the computations of combinatorial coefficients from above, in the second line performed simplification and reordering, and in the
27

---

third line we used the fact that 1- 1/N = (N 1)/N)/N" > (N 1)/N - 0)/(N" - -
for all 1 1 < pN 1.
Finally, considering the limit No + oo of the upper bound in (4) we have two cases: (1) if N 2 stays bounded and C2 E (0, 1) and C3 > 1, and by taking logarithms we check that the quantity tends to zero; (2) N + 0o and 2 2, and in this case (1 - 1/N1//D) ~E- find that the quantity tends to zero.
oc, then
bound nN 1 1/NN) CinCs with constants
and again by taking logarithms
In fact Prop. D.2 is gives us guarantees that will be possible to observe all options for each G; in the training set, however this may not be sufficient for the main aim of recovering the compositional structure of g. However we next prove, in an extension of Prop. [.2] that actually we will also observe with high probability any possible combination of finitely many factors:
Theorem D.3. Under the same hypotheses on G GenX, Grrain P as in Prop. D.2 for any k S n we have the following.
34
ik S Jb S N
P
(Gi,...,, Gie) # (in....jkk)
GG
[G1, Gn) E Gtrain
PN"
n!
kl(n k)!* k).
N² 1
(5)
No
which for fixed k tends to zero N ->
Proof. As mentioned before, roughly the overall proof strategy is the same as for Prop. D.2 We introduce the sets A Aj and observe that the number of choices of in < <... is S
[[] while the number of choices ji.....JJ ..k S N N Thus the number of sets of the form Al is NA == M((--j] Furthermore, each set Af only imposes the constraint of entries of indices i1..... is from G to avoid one combination of values j. and thus it has cardinality | N"=*(N* - 1). Then the number of choices of Grrain S Af of cardinality pN is just the (N"-* 1))
V
combinatorial coefficient [pN")I[N' (N* (NN- Jyy and by union bound we then find by the same reasoning as in Prop. D.2
S NA ([I-)(([-P)))) NM
(N"** (N 1))!
(left-hand side of 5
(N
(pN") * [N* -PNE
N Na-k No
N
No N pN"
= NA
Na
N
No - pN" +1
PN"
PN"
1
kl(n - AJ)! N² (1
n!
1
NA
NR
N
This proves (5) In order show that the bound tends zero as N
fixed p > 0, k: S n,
n! first note that k[] I n² /kl and thus (as k is assumed fixed) it suffices to show that
(1 (1-
1
PN
n x
0
N oc.
NE.
Here again, we can proceed with the discussion of the two cases (N bounded and + co, or N + 00) exactly as in the end of the proof of Prop. D.2 to conclude.
Remark D.4 (Extension of Prop. D.2 and Theorems D.3 and to general 9). If, for a choice of
> 2 and NI,..., Nn 2 2 we have {G [G
(Vi 5 n)1 < G, < N.). then
G = N1 ... N₂ and by carefully following the same steps as in the proof of Prop. D.2 we find
3i Sisn, Sj No,
PN....N.
P
VG GI...., ......,G]e E Gtrains
N₂ 1
(6)
it holds G; $
N
28

---

Again, this tends to zero for fixed as [g] = N1 Nn, - oo: to prove it, we can proceed as follows. In first case, n stays bounded, then we can discuss separately for each N; - whether it stays bounded, as done in the proof of Prop. D.2 and in either case the corresponding summand on the left of (4) tends zero, as in case (2) from the proof of Prop. D.2
S n whether
Otherwise, n > xo and we can proceed analogously to case (1) from the end of proof of Prop. D.2
For the extension of Thm. D.3
general G we get the bound
in S n IP Bi..... Ju N VG [G....., Gn. E Gtrain
Bi
(Gi,...., Gi.) & (J....,,Jh. jka
No. .. (1- No, (1-N,-.. Nog
1
DN
1S4 CI
and the strategy for its proof and for the proof that this goes to zero if k.p fixed and | above.
Ni Ti..... .,..., + oo are a direct extension
Theorem D.I is a direct consequence of assumption (A) and Theorem D.3 and thus it also directly extends the general case.
E Proof that final state reconstruction allows for wider message choice freedom
We first recall the setting. Consider two fixed maps
R -C S DDD
and let Rig VC Vi be the restriction to the first tokens, i.e. in previous notation, for m (m mc)= v we set (m) == mi = fm.)...,..).
For R2, S* as above and S D yc we set, for E ds(x) =
(S(x)))) = dx ( (R)
We also defined the following spaces:
Simu == f E, C Σ (x) <c) Sjinal = -s Ex [x) < <c
(8)
Our main result here is the following: Proposition E.1. With the above notations, we have the following
1. For C and any choice of M, 0, the restrictions on i-token sub-messages of S(x) by requiring S E Sia are more restrictive than those imposed by requiring S E Siinat i.e. we have (SMM)) C (Sjmaa)))
For the full message case i in general we have Stunt final- we further assume that Eg [d (x)) is non-increasing in j then we have the stronger inclusion Siut C Siinal
Proof. Item is direct, as no restriction on intermediate sub-messages S(x) = ((a(S(x)) by the requirement S E Sinall
For item 2, we observe that
dg((m)
1 s C Σ (x))
and by taking expectation we find that E, 12 E, (x)) < e implies E, [x)) < Ce, and thus
Stull S Syinal, which gives the first part of the statement.
29

---

For the second part of item 2 we note that if Ey Ey (x)) 2 (x) for all i S C' we get
c
1
Ez [dC())] - min Ez (2)(((())
[x)
- C
Ez (x)]
Ex
Σ
de
(a)
and thus full C Jinal as desired.
The freedom of exploration ensured by item of the proposition is our main motivation: by only constraining the reconstruction of the full message, allow much more freedom for the encoding strategy of ((x) at earlier sub-messages at each iteration.
Item 2 of Prop. E.1 shows that the two losses in the definition of Stul, S give comparable guarantces. At the beginning of iterations we will need to use the general bound with e/C. and as R will have been trained on more and more messages during the iteration, the hypothesis of Ez (a:) non-increasing will become true allowing for the sharper bound with the same value of c, as R' (an LSTM taking input successive message tokens) will become efficient in using sub-messages of pairs (m, m²) = (S(z), St (x)) to gradually distill information allowing to distinguish them for a large of S, x.
F Regularization to counteract Holistic Encodings
In this section, we include rigorous results useful to justify the regularization of our Imitation phase, aimed at balancing compressibility and efficiency.
This corresponds to the balance between compositional and parsimonious communication, general theme in classical II. literature (see e.g. 33 (7)) It seems useful to specify more precise definitions our generative setting, order to justify our architecture choices.
Our aim in this section is threefold:
1. In F.. we set rigorous definitions of Holistic Encodings within the setting of 2. and 2.2
2. In $F2 we check that compressibility is required for lowest Kolmogorov Complexity
encodings.
3. We verify that optimizing for just expressivity and efficiency on our training set, will produce Holistic Encodings, and thus is not satisfactory for reconstructing 9 (see FF3);
4. We give few possible options how enrich the loss functions in IL setup, aimed establishing a better balance between efficiency and compressibility (see $F.4..
F1 Definitions of Holistic and Compositional encodings, and role of the compressibility condition
In the classical IL setup [35] 37 a holistic system of communication defined as an encoding G + M that maps G G + G' E M injectively but disregards the compositional structure of the "language" to be encoded, i.e. in our case, without respecting the structure of G. In our generation task formalization, difference is that the relevant encoding map factors through the generation
map as G GenX X - M. Now the generation map GenX is considered fixed, thus the definition refers to only and is the following:
sender S X + M determines holistic encoding it satisfies expressivity but the messages associated to data generated as GenX(G) do not have a compositional structure in terms of generating factors G, as specified by Definition F2 below.
Definition F.1. In the setting from 2. 2.2
Note that, as in 35 37 Definition F. of holistic encodings refers to "respecting compositional structure", which requires another deñnition. The definition of compositional encodings crucially depends on how complex we need/want to assume to be. In this paper, for the sake of concreteness, we restricted to the simple case of G being formed as an element of a product space of finite cardinality, i.e., an n-plee of independent factors G = [G,,.., G.. in which each G, can take finitely many values. We thus use straightforward toy definition of compositional encodings, requiring that each of the G; is separately encoded as a string and these strings are concatenated together to give the message that encodes G:
30

---

Definition F.2. In the setting of 2. and 2.2 assume that generating factors G = [G...... Gn] have G2 E [1: Ni. for some Ni..... ....,,,, respect the structure of generating factors G if for each 1 < i S there exist an injective mapping Eg N.) > VC, for which
Then sender S X' + M Vc will be defined to
VG E g, S(GenX(G)) = (G1)) E, (G.)
and Σc=
The compressibility condition on the sender-receiver protocol just requires that the protocol be not holistic, and thus fully specified via definitions [.I] and F22] Furthermore, Definition F22 fully specifies in which way the map G - M successfully approximates an isomorphic reconstruction of 9, if it satisfies conditions compressibility and expressivity.
F.2 Encodings optimizing for only expressivity and efficiency have Kolmogorov Complexity mueh higher than compositional ones
By the classical learning theory principle, if our sender generates encodings with lower complexity, these should be easier to learn by the receiver, in the sense that the receiver will tend to generalize with more ease form good accuracy on the training set to similar accuracy on the test set.
As manageable measure of complexity we will use Kolmogorov Complexity (KC), and we verify that in setting, encodings satisfying expressivity and efficiency will necessarily have higher KC than compositional ones. In particular, the receiver will have more trouble reconstructing from such encodings, justifying our push enforce the compressibility condition.
Encodings of D Drain satisfying expressivity and efficiency include all injective maps
enc : Diroin + VC., C. = [log [Deroin]] l]s
in which [a] denotes the smallest integer larger than real number For random map above, the encoding to be polmogorov-irrduciin so that the expected KC of a random enc would be up to constant factor the corresponding explicitly enumerating the [Dtrain] strings of length C. in alphabet V that describe the encodings of each element of the training set. Each such string thus requires z C, loga [V] bits. Thus if U is the uniform distribution over injective maps have
(9) then
Exnc-t K C(enc)) [Derain C, logg [V| [Drrain] logg [Dtrain]-
(10)
On the other hand, to specify compositional encoding comp in Definition F2 requires only assigning a string of length C, in alphabet V for every generating factor G where we can take
G [log]] N:]- Thus such string requires zs loga N; bits, and the whole encoding has KC given by
a
KC(comp) log2 N; = =log2 [Ptrain
(11)
we assume that (Dtrasn D = = (0, and we take [D] + oo, then we get
r.h.s. of 10 Dlog p logz [D)) p|D| as |D] 00. oc.
(12)
r.h.s. of
logg (D]
The computation 12) shows that
Proposition F.3. Under the same hypotheses G as in the previous section, consider the limit [D] + & and assume that p = 'Dtrainl/12) stays bounded away from zero in the limit. Then the expected Kolmogorov complexity of encodings satisfying expressivity and efficiency becomes overwhelmingly larger than the one of compositional encodings:
Pene-t |K (enc))
D| D0.
KC(ccop))
F3 Encodings that optimize efficiency under expressivity over Drrain are holistic
First, we clarify the definitions:
31

---

The requirement of "expressivity over Dirain" means that the encodings are injective over
D Drain
The requirement of "efficiency over Drrain" means that the encoding maps Dtrain into VC, for the minimum value of C..
Similarly to previous subsection (see ) keeping in mind that [Dtrainl = =p|D| = p(g), with fixed p E (0, 1), we have
C. = [l Darain|
|| [log v1 p| < [[og]]] [9]] L) log v p|]-
(13)
For compositional encodings of G [G...... ...,................... Gn.] the encoding will require set apart separate message sub-strings of length C; [log v N/] for factor G; (see Def. F.2 thus the required length for the encoding will be
(++) 2 C. + |[loggr| plJ,
Co== =
Σ Loggy/ NO
2
log N; = [log 19|1
(14)
in which (**) follows from
We now discuss when the inequality signs in (14) are sharp or not, since sharp inequality Co C. implies our claim that optimizing for efficiency under expressivity implies non-compositional (i.e., holistic) encodings:
1. If we use relatively small [V| S 10 (assuming commonly used values p S 0.1), will have
V p
(15)
and thus v S -1, and thus the second inequality (+*) (4) is guaranteed to be sharp, showing the desired strict inequality C. > C. independently of the sizes of factor ranges N,, ,,...,n. n.
2. If the gaps between l N; and the lowest integer larger or equal to it, sum to value of at least 1, i.e.,
Σ
[Dogjv] N:] - log/r N.) 2 1,
(16)
then (+) becomes sharp, again guaranteeing Co > C. desired. This could be probable to happen in our setting, and becomes more likely for large [V] > 1/p for which (15) fails, given that the N; are supposed for a few of the N; the gaps in (16) are nontrivial. For example, (16) holds true for V 1/p 2 if, say, at least two of the G; are binary factors, so that N, 2.
be unknown, so is likely that for large [V > 1/p
In summary have the following:
Proposition F4.. Under the same running assumptions over G, P. we have that optimizing only for efficiency of encodings efficiency under injectivity constraints expressivity over Derain is guaranteed produce holistic (i.e., non-compositional) encodings if one both of the conditions 115 and (66 holds.
F.4 Possible choices of encoded message regularization
As seen above, if we just optimize for efficiency and expressivity then we are likely to get holistic encodings (Prop. F.4) which will then make it hard for the receiver to generalize the decoding strategy outside the training set due to having overwhelmingly higher expected complexity (Prop.
F3
This proves that it is important to incentivize condition compressibility. i.e., to push encoding strategies away from overly efficient holistic encodings. In this section we discuss a few approaches to do this in practice, explaining our choice of compressibility regularization.
1. Entropy maximization.
32

---

Background and standard approach. Introduced by [69| for use in policy-based rein- forcement learning, entropy maximization is known enforce exploration deterministic policies and avoid early convergence to single output choices for learned policies. Specifi- cally, in learning setup with input æ; output and policy parameters W, the authors define the following estimator for fixed output value &:
h(5, W, x) = - In Pr(y = x)).
(17)
such that we take the expectation over 5 we get:
E[M(S, W. >) = P Pn Pr(y=
<)W,),,
(18)
which is the entropy of the network. Therefore h is an unbiased estimator of the entropy. They also note that, if un and x; with E represent the weights and pattern of single neurons in feed forward network, & the ith position in ny ny-tuple &, and let gi be the probability density describing yi. ie::
9elE, w², ,²)xx')==) = Pr(y, Elw', 2²),
(19)
then, for any feed forward network,
Pr(y = <) = II
gi ,''Si, x²),
(20)
iEU
and therefore
In Pr(y = <IW,,)
E9, 95, w², ,x¹) = -h(y, W. X)..
(21)
SEU
In fact, this method can be extended onto sequence prediction models, and even IL, such as in (55)
Naive adaptation to our setting is not practical. We take x X and & E M = VC and recall that in our notation m, indicates the (-th token of message m. and m][4] is the notation for the initial segment m.; .2.......... my. so that in particular c = m.
Importantly, in our setting, to avoid high variance in estimation due to the large message space (see G we approximate the probability distribution of the RNN-produced mes- sage token at position < C. denoted here p(m; I m[- as 8 distribution over the vocabulary V by using the Gumbel-softmax trick:
11,
if m
(3v E V), Pr(me] x, W, W, R 0, if m, E V V {u}.
(22)
and therefore:
C Pr(m = <), = II Pr(m: & W, mi--))-
(23)
Note that due to (22). for
for all but a single one of the VV/ possible values & the quantity 23 is z O, and thus h becomes poor estimator for the entropy of p(malæz, mp-1)), as we get
C Mm,, W,x) - In > Pr(m: Stlar, W. R 0o.
(24)
particular, becomes clear that optimizing this becomes unfeusible with gradient methods.
Our proposal for regularization. Our model can be seen different approximation of the entropy of the message. Instead of using the policy weights as probability values, we use the law of large numbers estimate the probability of each message. For batch B X consisting of Noathh training examples and for given sender encoding protocol Sg:
1(x E ()())))) = EH
For sufficiently large Noatch: Pr(m = E x) ~
(25)
Nbatch
33

---

Yet again, due to the exponential size of the message space, utilizing this probability to estimate the entropy is still not practical. We will assume approximate independence between symbol positions, which allows to get the stronger approximation:
|x E B SØ(x),, = wt1
Pr(m- = v x. o)
(26)
Noatch
as well as the additivity of entropy to be applied to (18
H(Pr(m|z,0) = Σ
Pr(m = x () Pr(m = x,
ξEVC
Σ
Σ Pr(m: x, In Pr(m: 2, 2. (27)
Then we can perform the following computations, in which we let m, m' be two i..d copies of m:
C
H(Pr(m) x, 0))
28
Pr(m: = v x, Pr(me = 21
NEV
(-x a x for S 1)
[r(mm = x1 x. Pr(me x, 0]
EEV
Pr(m, = x. p) = v' x. x)
tml
Pr(m== = (p(m= x2)
VEV
Σ
Pr(m: = 1 Pr()) o 12,
1=1 VVVeVV
Σ Pr(m: # m²) Eldu(m,m))
Be (d (S, (z)) (z))))
(28)
In the above step (*) we used that
2
Σ NEV
Pr(m Pr(m== 1 2x, 1= 1= Σ
Pr(m; = - (2)
WEV
Σ
Pr(m; xr Pr(m)) = xr, Ø)
The consequence of bound (28 is that maximizing the pairwise Hamming distance between predicted messages in batch also maximizes a lower bound to the approximate entropy of the message space, justifying the choice optimizing this quantity (which is easier include in practice) for favoring compressibility.
2. Contrastive Learning losses. In our framework, PDM can be seen as a form contrastive loss, which we now show how to connect to standard contranstive losses from previous works.
Contrastive loss functions such as the triplet loss 66 and NT-Xtent [I] work by defining positive and negative image pairs (usually through image augmentation), and aiming to minimize the distance between embeddings of positive pairs and maximize the distance between negative pairs. Specifically, instead of purposefully sampling negative samples for every x, NT-Xtent produces single pair of samples for each in mini-bate N by
34

---

augmenting each image. Each image in the batch therefore has 2N 1 negative pairs. They use the cosine similarity sim(u, v) u" u/ //||u|||e|| and some temperature value define the following loss between positive pair embeddings and aj:
- log exp(sim((1,2))) 2;)/7)
(29)
Σx exp(sim(2i, [b)T
For simplicity, let T and thus
L sim(mi, 25) log exp exp(sim #k))-
(30)
Here, we apply the same ideas but take this loss over an unaugmented batch, i.e the only application this loss is for zi against itself. In this case it reduces to:
& = --1++ log Σ exx((sim(zx=x))
(31)
In our representations, the embedding vectors from batch 2,, = ,,....NN are V x C matrices whose columns are approximately one-hot vectors that represent the tokens of message. For one-hot matrices counting the amount of matching rows of Zi, žk, i.e, the un-normalized Hamming distance of the associated message strings mi, my VCC Since strings have length C' and ||=;||²= 2 2i ~
of this form, the dot product 2. 2g. is equivalent to
C, we thus get:
sim(zi, 2k)
tr(2)
1l aarg max( arg max(((*)).)) 2x))))]
C
1=1
1 dh (m, m
in which dy is the (normalized by 1/C) Hamming distance. We can replace this expression in the unaugmented NT-Xtent loss:
N
=
-1 -1 + log
exp(1 du (m,, mx))
k=1
-1 N log Σ
exp(dn (mi, mx))
denson inequality)
N dn
-1 N - log N
k=1
Then, taking the mean loss over the full batch we get:
1 N
N N
Σ E
log
du (n mg).
(32)
We then include the loss only the last term, i.c. the average of dh (m,, mg). which the only term that actually depends on the batch elements, which forms approximate upper bound on the NT-Xtent loss.
Note that our approximation of 2i, =1) by 1 d (m, my). on which the above is based, becomes sharp when the 2; have actual one-hot entries, thus the approximation becomes better as our model has high accuracy. Furthermore, Jensen's 's inequality, responsbile for the inequality between the term E, from NT-Xtent and our metric, becomes closer to equality messages m....., mN become uniformly spread across VC, which is the case for minimizers of our metric as well. This means that, at least heuristically speaking, we can consider our metric as having similar minimizers to the NT XteXtent regularization.
A precise matching lower bound to 32 is left for future work.
35

---

G Justiifcctiinn of our imitation step starting from classical probabilistic reconstruction games
We start by recalling classical probabilistic reconstruction games, after which we adapt the formulation to the setting of generative tasks, and we show that with few justified simplifications we arrive to our formulation from the main paper.
G.1 Classical probabilistic reconstruction games
Recall that in reconstruction games, the two agent types are teacher and student (or speaker and listener, or sender and receiver), parameterized by respectively 0, d. The game consists the teacher observing signal distributed according to probability distribution p. from which it produces randomized message m whose distribution conditional on æ is obtained according to policy ITe (e(-|a)). The student listener, observes message and produces reconstructed signal distributed according to a conditional probability Pa(-Im) The game dynamics can modeled by stipulating that together, teacher and student minimize the log- oog-lieeihood loss function
G = - -Eggp m-
[log Pa (x/m))
(33)
in which represents the signal and m the message used reconstruct it, the speaker produces reinforcement learning policy Te (-(x))) and the listener produces a reconstruction function (a(mmm).
G.2 Reconstruction games in generative tasks
While the above formulation 33 is formalized using the log-likelihood between distributions, for our generative task, the reconstruction modeled slightly differently: the task is to optimize, with high probability, the reconstruction error, measured by a distance between the signal and its reconstruction x', denoted d(z, z'). Here the notation for x and d(z,, x') can be (ab)used to represent two alternative settings: either (a) x, x' are vectors in the autoencoder latent space and d(ær, x²) is discrepancy (e.g. d(x,x') = Jæ: r²(²") between them or (b) x, x' are reconstructed images and d(x,, x') is discrepancy (e.g. based on some ad-hoc norm or otherwise) between them. Thus we have the probabilistic error function
Cinelw, (p) == (ampp ~nw (m) (m) x').
(34)
The loss
is akin d-based Wasserstein distance analogue of (33
G.3
The problem reduces to the case of deterministic student
Note that when d(x, .) = |x 2² for some Euclidean norm (or more generally, when d is a convex function of such norm), then the minimization of (34) over distributions Pa at fixed m, yp must be achieved by
(m] argming.......... (m) (m), (', z'),
(35)
where if the set of æ is finite then Pe is the probability defined by
p(x') ((x'))TT (m|a') E, ((y) (. (mly))
(36)
and analogous expression with integrals replacing the sum holds in general.
Under the above convexity hypotheses. the choice of x in (35) is unique, and for d(z, x') ) 2'-2' it coincides with the barycenter of P.. In this case, P.-(-lm) can be taken to be the Dirac distribution concentrated wif and optimization of (34) can be achieved by deterministic reconstruction function R_, (m), whose optimum value for given yo will be xy from 335) When reducing to
optimizing only amongst deterministic R.,, the loss (34) rewrites in the simplified form
Line(w, (0)
R. (m)).
(37)
36

---

G.4 Straight-Trough Gumbel-Softmax (STGS) trick
As in 25 and subsequent works, the optimization of policy Te (minimization over ()) can in theory be done via policy gradient optimization methods such as REINFORCE [70], however, due to the combinatorial explosion of the number of messages this can be very compute-intensin if calculated directly, or produces high variance estimators. We thus follow the straight-trough Gumbel-softmax trick [26], which the probabilistic interpretation of Typ is implicit in the network architecture, and the architecture itself is deterministic and directly differentiable. In this case, we can thus pass to the deterministic description as (a)) replacing the probabilistic one m Tip obtain the updated form of (37) as
Tpp (-Im), and
iine (w, Φ) = Ex (z)()
(38)
which for d(x,, x') = 1x - x'1 is very similar to our actual interaction loss (
(w, Φ) ExEX || R (Sx()))
(39)
37

---

H
Example Generation: Reconstructions
The following reconstruction, Figure examples were generated with the CELEBI model using PDM regularization with 9 = and X = 1.5. In each row, the leftmost image corresponds to the original input, while the subsequent images represent the predicted reconstructions I; (z) obtained from progressively longer message prefixes with the VAE backbone.
We observe that the reconstructions do merely improve in pixel-level fidelity but rather exhibit semantic refinements across successive steps - variations in floor hue, object color, shape, or viewing angle. This progression suggests that the message space encodes the underlying generative factors G and supports degree of compositional structure, where messages carry disentangled information.
Qualitative comparison across methods. To further assess the representational properties of CELEBI, we qualitatively compare reconstructions across the different methods proposed in this work (see Figures and 7). As illustrated in these figures, CELEBI reaches semantically faithful reconstructions significantly earlier in the decoding sequence than the baseline methods.
Moreover, the differences between reconstructions models, which often show subtler variations mainly related to texture or brightness. This qualitative evidence supports hypothesis that CELEBI facilitates representations that are not only more compact but also more semantically coherent.
semantically pronounced than in baseline
structured message space, enabling
Example
(b) Example 2
(c) Example 3
(d) Example 4
Figure 5: Progressive image reconstructions obtained at each decoding step by the receiver as processes successive symbols of the message.
38

---

Reconstruction using the baseline model.
Awdd
(b) Reconstruction using partial decoding.
Rech
Reconstruction using final state imitation.
bocc
Pac
Pocc
(d) Reconstruction using PD+FiSL+PDM.
Figure 6: Reconstruction for the same example with different methods.
Additional Language Metrics
Table 3: Comparison of proposed methods using language variation metrics from 13
Synonymy.
Homonymy
Freedom J
Entanglement
Full message reconstruction, no II. (Baseline)
0.563 0.026
0.695 + 0.015
0.509 + 0.026
0.831 0.000
Interaction
Progressive Decoding (PD, ours)
0.640
0.737 0.013
0.786 0.616
Message imitation (haseline)
0.601 0.039
0.714 + 0.027
0.613
0.822 + 0.001
Imitation
imitation
0.017
Final-State Imitation (FiSI, ours)
0.633
0.038
0,751
0.018
0.640 0.038
0.842
0.007
Regularization
PD-FiSI-KoLeo 1.5
0.594 0.027
0.739
0.013
0.598
0.027
0.805 0.007
PD+FiSH+PDM (ours) À 1.5
0.654 0.013
0.753
0.008
0.658
0.013
0.780
0.008
(a) Shapes3D
Synonymy. ~
Homonyny
Freedom J
Entanglement
Interaction
Full message recunstruction, no IL (Basline)
0.596
0.053
0.597 1: 0.061
0.597 1: 0.0.22
0.852 1: 0.021
Progressive Decoding ours)
0.486 0.056
0.486
0.786
Message imitation (baseline)
0.367
0.034
0.393 0.042
0.369 + 0.033
0.837 + 0.015
Imitation
Final-State Imitation (FiSI, ours)
Full state initation
0.321 0.020
0.382
0.015
0.393 0.018
0.421
=: 0.052
0.328 + 0.020
0.364 :: 0.041
0.830 =:900100 0.008
0.832 1: 0.018
Regularization
PD-FiSI-KoLes À 1.5
0.365
0.011
0.338 +: 0.019
0.367 0.0011
0.763 0.005
PD+FiSL+PDM (ours) =
0.333
0.319 0.0013
0.756
(b) MP13D
In addition to the metrics presented in the main text of this work, we attempted to measure other metrics for qualitative linguistic variation. We were not able find many such metrics in the literature, and thus restrict to the ones presented in [3] Importantly, since this reference had a slightly different focus, variation in this context will refer to the evolution the language time, but rather departure from regularity, which masks underlying compositional structure.
39

---

Reconstruction using the baseline model.
Awd
Am
(b) Reconstruction using partial decoding.
Ardd
(c) Reconstruction using final state imitation.
1012
bocc
Pac
(d) Reconstruction using PD+FiSL+PDM.
Figure 7: Reconstruction for the same example with different methods.
We study the four measures of variation presented in the paper, namely synonymy, homonymy, word order freedom and entanglement. Synonymy measures the presence of one-to-many mappings between atomic meanings and characters a position, being minimized when each generating factor value is mapped to single position and character. Homonymy measures the opposite, i.c.. the presence of many-to-one mappings, and is minimized when each character in a position is mapped to unique generating factor value. Word order freedom refers to the strictness of the mapping between generating factors and positions the message (for example, the mapping system always encodes the shape generating factor at the first position). It is minimized if all single generating factors are encoded in the same message position. Similarly, entanglement minimized when all factors are encoded into unique positions in the message.
Important formular metricas, pero no encaja tan bien. Unico paper que conocemos. clear patterns, but we argue it is because... An important assumption made by the authors for all these metrics is that meaning should be undivisably encoded into single positions of the message. We believe this scope of regularity is too narrow to capture rumposttionalliy-esseeeee in F.2 and does not well when the space messages greatly larger than the amount of possible states. We found no clear trend similar behavior across both tested datasets, save small decrease in entanglement when using PD. This may suggest that the increased pressure for efficiency forces individual generating factors be uniquely distributed in the message, however believe there is not sufficient evidence to make such claim considering the previously discussed pitfalls of the metrics.
mappings in setting, defined
To our knowledge there are no available official implementations for these metrics. Our versions can be found in the official repository for this work.
40

---

Table Permutation tests for Shapes3D and MPI3D on emergent language metrics.
Metric
Mode A
Mode
Statistic
p-Value
TopSim
Progressive Decoding (PD; ours)
Full message reconstruction; no IL (Basclinc)
0.0263
0.0233
TopSim
Full state imitation
Final-State State Imitation (FISI; ours)
-0.0267
0.1014
TopSim
Full state imilation
Message imitation (Baseline)
-0,0188
0.9332
TopSim
Final-State Imitation (FiSI; ours)
Message imitation (Baseline)
0,0249
0.2605
TopSim
PD+FiSI+KoLoo X 1.5
PD+FiSL+PDM À 1.5
-0.0371
0.0004
Useful Length
Progressive Decuding (PD; ours)
Full message reconstruction; no IL (Baseline)
0.0031
Useful Length
Full state imitation
Final-State Imitation (FiSI; ours)
0.0
1.0
Useful Length
Full state imitation
Message imitation (Baseline)
0.0
1.0
Useful Length
Final-State Imitation (FISI: ours)
Message imitation (Basclinc)
0.0
1.0
Useful Length
DD+FiSS+Kol.co Ả 1.5
PD+FiSI-PDM À 1.5
1,3000
0.0996
Final MSE
Progressive Decoding (PD; ours)
Full message reconstruction; II. (Baseline)
-0.0218
0.0657
Final MSE
Full state imilation
Final-State Imitation (FiSI; ours)
-0.0130
0.4871
Final MSE
Full state imitation
Message imitation (Basclinc)
-0.0201
0.5679
Final MSE
Final-State Imitation (FISI; ours)
Message imitation (Baseline)
-0.0711
0.8384
Final MSE
Po4FiSt+KoLeu 1.5
PD+FiSI+PDM 1.5
0.0343
0.0431
(a) Shapes3D permutation test
Metric
Mode A
Mode B
Statistic
p-Value
TopSim
Progressive Decoding (PD; ours)
Full message rcconstruction; no IL (Baseline)
0.0037
0.4416
TopSim
Full state imitation
Final-State Imitation (FiSI; ours)
-0.0193
0.0019
TopSim
Full state imitation
Message imitation (Baseline)
0.0047
0.4395
TopSim
Vinal-Starc Imitation (FiSI: ours)
Message imitation (Baseline)
0.0240
0,0007
TopSim
PD+FiSI+Kol.d 20 À = 1.5
PD+FiSI+PDM À 1.5
00,0600
0.4767
Useful Length
Progressive Decoding (PD: ours)
Full message reconstruction: no IL (Baseline)
-270000
0.0345
Useful Length
Full state imitation
Final-State Imitation (FISI; ours)
0.0231
1.0
Useful Length
Full state imitation
Message imitation (Baselinc)
0.2231
0.3998
Useful Length
Final-Staa Imitation (FISI: ours)
Message imitation (Basclinc)
0,2000
0.5820
Useful Length
PD+FiSI+KoLco
1.5
PD+FiSI+PDM À 1.5
0,2222
1.0
Final MSE
Progressive Decoding (PD: ours)
Full message reconstruction: IL (Baseline)
0.0011
0.5894
Final MSE
Full state imitation
Final-Stare Imitation (FISI: ours)
-0.0018
0.1166
Final MSE
Full state imitation
Message imitation (Baseline)
0.0019
0.2870
Final MSE
Stnat-State Imitation (FiSI: ours)
Message imitation (Baseline)
0.0037
0.0201
Final MSE
PD+FiSI+
1,5
PD+riSI+PDM À 1.5
-0,000012
0.9626
(b) MPI3D permutation test
J. Permutation Tests
J.1 Emergent language metries
To assess the statistical significance of each component added to the Iterated Learning (IL.) framework proposed in this work, conducted permutation tests using the SciPy library [67]. The test statistic was defined as the difference in means, A B. where corresponds to the arithmetic mean of the evaluated mode and metric across 10 random seeds. The complete results are presented in Table
Across both the Shapes3D and MPI3D datasets, we found that the inclusion of the PD module consis- tently reduced the useful message length, yielding a test statistic of approximately 2.5 and p-value below 0.05. None of the other additions introduced in this work produced a statistically significant decrease in useful length relative to their respective baselines, supporting imposes an efficiency pressure on the emergent language. Morcover, we observed a statistically significant increase in TopSim for the Shapes3D dataset, with test statistic of 0.026 and 0.05.
hypothesis that PD
Final-State Imitation had a statistically significant gain in TopSim over both baseline methods on MPI3D, with statistic of ~ 0.02 and < 0.02. PDM outperformed the KoLeo estimator in both reconstruction error and TopSim on Shapes3D, but achieved no statistically significant increase in MPI3D.
J.2 Disentanglement permutation tests
In this section we evaluate the statistical significance of the differences between our model and the discrete baseline the metrics used in table 2 Similar to the previous section, we conducted
41

---

permutation tests, using the difference in means as a statistic for 10 random seeds. The complete results are presented in Table
We found significant difference in accuracy on the Shapes3I) dataset, with a statistic 0.114 and p 0.01. We found significant descrease in RMSE both datasets, with statistics of 0.227 and 1.151 for Shapes3D and MPI3D, respectively, and S. 0.001. We also observed a significant gain in disentanglement on both datasets, with statistics 0.003 and 0.023 for Shapes3D and MPI3D, respectively, and < 0.05 for both.
Table 5: Permutation tests for Shapes3D and MPI3D disentanglement, accuracy and RMSE using
messages from discrete models as data for a linnear probe.
Metric
Mode A
Mode B
Diff(A-B)
p- Value
Accuracy
VAE+EL VAE+CELEBI
-0.1139
0.0018
RMSE
VAE+EL VAE+CELEBI
0.2266
0.0010
DCI Disentanglement
VAE+EL
VAE+CELEBI
-0.0034
0.0042
(a) Shapes3D permutation test
Metric
Mode A
Mode B
Diff(A-B)
p-Value
Accuracy
VAE+EL VAE+EL
VAE+C
-0,0128
0.0570
RMSE
VAE+CELEBI
1.1512
0.0010
DCI Disentanglement
VAE+EL
VAE+CELEBI
-0.0230
0.0002
(b) MPI3D permutation test
42


Reviews:
[{"review_id": "nPlAKnZQbP", "reviewer": "Reviewer_Likm", "summary": "This paper proposed CELEBI, a method for learning compositional representations that combines interactive decoding (partial reconstruction from intermediate representations)  and an imitation (similar to iterated learning). This method is evaluated on two datasets, Shapes3D and MPI3D, demonstrating its strengths compared to previous methods for constructing discrete representations.", "questions": "- I find the notation quite confusing, especially for Eq. 0 (the unnumbered display equation at the top of p3, right after line 95). Does GenX take in the set $\\mathcal{G}$ or just $G$ (as in line 71)? What is the input to $S$, $\\mathcal{D}$ or $\\mathcal{X}$? And what is the output of $R$? The textual description states that the reconstruction is an image $x$ but this equation shows the set $\\mathcal{X}$.\n- How is $C$ determined? The notation implies it's fixed/constant, but the text suggests that it's variable.\n- Tuning the efficiency pressure (lines 132-135): I'm a bit confused by this explanation. How do small values yield fast reconstructions? The smallest value is $\\lambda=1$ and that would give equal weight to all sub-messages. Similarly, this section states that higher values of $\\lambda$ yield longer messages, but based on Eq.1 it seems that for large $\\lambda$ there's a much larger penalty for longer messages.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- This paper addresses an important question (i.e., how to learn compositional representations) by proposing a new method that combines existing techniques in a novel way. As such, it present an original work that could potentially be interesting and valuable for the NeurIPS community. \n- The chosen datasets, experiments, and baselines seem adequate for evaluating the method. \n\n**Weaknesses**\n- The results seem somewhat incremental, making it hard to assess the significance of this work. That is, the quantitative results suggests only slight improvement compared to previous methods, and in particular, it seems that the proposed method does not perform quite well in terms of accuracy (Table 2). Perhaps even more importantly, it's unclear what is the nature of the learned compositional structure and what new insight it affords. I could see value in a new method that provides compositional representations that are more insightful and interpretable than prior methods, even if that method doesn't achieve the best possible accuracy, but I don't think this was demonstrated in the paper. I therefore think that the paper could benefit from a deeper analysis of the learned compositional structure.\n- The paper could also benefit from more clarity in the presentation. For example, the notation is not always clear (see questions below), and the baselines are not sufficiently explained.", "was_revised": true, "final_justification": "I still have some of the reservations I mentioned in regards to the overall significance of the results, but given the authors responsive rebuttal I feel a bit more positively about this work and am inclined to increase a my rating.", "rebuttal": "### Rebuttal Reviewer Likm\n\nWe thank the reviewer for their time and their attentive response.\n\nWe believe that a misunderstanding regarding the positioning of our paper underlies several of the reviewer’s concerns. Our focus is on proposing a new iterated learning framework in which emergent representations become compositional. We do not aim to resolve the ongoing debate around the relationship or trade-off between compositionality/disentanglement and downstream accuracy. This is especially relevant for the interpretation of Table 2 (see discussion below). The three main new components of our framew...\n\n---\n\n### Weaknesses\n\n1. **Regarding Table 1**:  \n   While the reviewer did not mention this table explicitly, we believe their remarks about “only slight improvement” may refer to it. Several reviewers pointed out that results in Table 1 may not be statistically significant.  \n   To address this:  \n   - We increased the number of runs per method to 10 (updated tables below).  \n   - We conducted permutation tests across methods in interaction, imitation, and regularization.  \n   Results indicate statistically significant differences:  \n   - **Shapes3D**: Progressive decoding outperforms the baseline in Topsim (p = 0.0233) and useful length (p = 0.0031). PDM outperforms KoLeo in Topsim (p = 0.0004) and reconstruction error (p = 0.0431).  \n   - **MPI3D**: FiSI outperforms message imitation in Topsim (p = 0.0007) and reconstruction error (p = 0.0201), and full state imitation in Topsim (p = 0.0019). PD outperforms full message reconstruction in useful length (p = 0.0201).  \n   Full results will be included in the final paper. Let us know if further comparisons are needed.\n\n2. **Regarding Table 2**:  \n   As noted by the reviewer, our method improves disentanglement but underperforms in accuracy relative to continuous VAEs, while slightly outperforming the discrete baseline. These outcomes align with literature (see [1]): disentanglement often harms accuracy in downstream classification.  \n   Discrete representations may be less accessible to simple classifiers (as shown in [1]), and inductive biases for compositionality do not imply disentanglement [2].  \n   Thus, the disentanglement results in Table 2—not accuracy—are central to our paper’s positioning. We acknowledge this was not clearly conveyed and will revise the discussion accordingly.\n\n3. **On the nature and interpretability of emergent representations**:  \n   Thank you for raising this. Topsim measures structural alignment between representation and semantic space, but not characteristics of the emergent language itself.  \n   Following reviewer rCnw’s suggestion, we added metrics like Synonymy and Homonymy, and discussed them in the supplement (see also Weakness 4.i in their rebuttal). Our focus is on basic compositional properties; deeper interpretability metrics are outside the paper’s scope but worth future exploration.  \n   Appendix I already shows how initial submessages carry relevant information—captured by our useful length metric. We will also provide more qualitative examples to support this.\n\n---\n\n### Questions\n\n1. Thank you for pointing this out. We agree the original equation was overly condensed and have revised the notation for clarity.\n\n2. Yes, $C$ is fixed and large enough that possible messages far outnumber dataset states. This ensures meaningful compression in useful length (see line 221) without harming reconstruction accuracy. We will clarify this in the text.\n\n3. In line 134, the sentence “$\\lambda > 1$ achieves the best performance” is a typo—should be “$\\lambda \\approx 1.5$.” Eq. (1) allows $\\lambda > 0$, not just $\\geq 1$ as the reviewer inferred (see Figure 2).  \n   Also, $C$ is fixed in Eq. (1). As $\\lambda$ increases, more weight is placed on later tokens. In the limit $\\lambda \to \\infty$, PD becomes equivalent to full message reconstruction. Even when $\\lambda = 1$, earlier tokens still contribute to more terms than later ones and are thus more heavily pressured. We will clarify this.\n\nOnce again, we thank the reviewer for their time and valuable feedback.\n\n---\n\n### Shapes3D (10 runs)\n\n| Category           | Method                                        | TopSim ↑       | l₂×10⁻¹ ↓      | Last symbol MSE ↓ |\n|-------------------|-----------------------------------------------|----------------|----------------|-------------------|\n| **Interaction**    | Full message reconstruction, no IL (Baseline) | 0.244 ± 0.0021 | 10.0 ± 0.0     | 0.212 ± 0.003     |\n|                   | Progressive decoding (PD, ours)               | 0.270 ± 0.001  | 7.5 ± 0.215    | 0.238 ± 0.007     |\n| **Imitation**      | Message imitation (baseline)                  | 0.257 ± 0.005  | 9.9 ± 0.3      | 0.200 ± 0.007     |\n|                   | Full state imitation                          | 0.256 ± 0.003  | 9.9 ± 0.3      | 0.179 ± 0.003     |\n|                   | Final state imitation (FiSI, ours)            | 0.283 ± 0.003  | 10.0 ± 0.0     | **0.176 ± 0.02**  |\n| **Regularization** | PD+FiSI+KoLeo λ=1.5                           | 0.256 ± 0.002  | 8.555 ± 0.167  | 0.179 ± 0.003     |\n|                   | PD+FiSI+PDM (ours) λ=1.5                      | **0.292 ± 0.002** | **7.0 ± 0.155** | 0.194 ± 0.005     |\n\n---\n\n### MPI3D (10 runs)\n\n| Category           | Method                                        | TopSim ↑       | l₁.95×10⁻² ↓   | Last symbol MSE ↓ |\n|-------------------|-----------------------------------------------|----------------|----------------|-------------------|\n| **Interaction**    | Full message reconstruction, no IL (Baseline) | 0.133 ± 0.001  | 9.3 ± 0.002    | **0.015 ± 0.0**   |\n|                   | Progressive decoding (PD, ours)               | 0.137 ± 0.001  | **6.6 ± 0.341**| **0.015 ± 0.0**   |\n| **Imitation**      | Message imitation (baseline)                  | 0.135 ± 0.001  | 9.733 ± 0.029  | 0.016 ± 0.0       |\n|                   | Full state imitation                          | 0.137 ± 0.001  | 9.923 ± 0.020  | 0.018 ± 0.0       |\n|                   | Final state imitation (FiSI, ours)            | **0.156 ± 0.001** | 9.7 ± 0.046 | 0.02 ± 0.0        |\n| **Regularization** | PD+FiSI+KoLeo λ=1.5                           | 0.147 ± 0.002  | 8.9 ± 0.221    | 0.02 ± 0.0        |\n|                   | PD+FiSI+PDM (ours) λ=1.5                      | 0.153 ± 0.001  | 9.0 ± 0.167    | 0.02 ± 0.0        |\n\n---\n\n[1] Zhenlin Xu, Marc Niethammer, and Colin Raffel. *Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language*, 2022. [arXiv:2210.00482](https://arxiv.org/abs/2210.00482)  \n[2] Milton Llera Montero et al. *The Role of Disentanglement in Generalisation*, ICLR 2021. [OpenReview](https://openreview.net/forum?id=qbH974jKUVy)"}, {"review_id": "i5RPAZUWMk", "reviewer": "Reviewer_rCnw", "summary": "This paper investigated compositional generalisation in an emergent communication paradigm through a newly introduced framework (CELEBI). This framework comprises of three mechanisms: interactive decoding, reconstruction-based imitation learning, and pairwise distance maximisation. In a series of experiments and ablations, the papers claims that the emerged languages are more efficient and compositional as opposed to standard (baseline) emergent communication methods on two image-based datasets. The paper incorporated many experiments and derivations that make it a complete paper, but I found it somewhat difficult to assess exactly what the main position of this paper was – emergent communication or compositional generalisation. I did not completely check the correctness of the mathematical explanations and proposition in the main paper and appendix, I adjusted my confidence accordingly.", "questions": "1. As mentioned in points 1,2 and 3, I am not convinced of the effectiveness of the (individual) proposed mechanisms. Could the authors provide information regarding the test for significance and the resulting statistics? If this is available, adding it to table 1 and 2 would be appreciated. \n2. Some qualitative interpretation of the emerged languages would be very valuable and strengthen the claims of this paper. How exactly do the messages change as a result of the individual mechanisms? Here, I would be especially interested in the changes resulting from PDM. In addition to manually looking for message meanings, directed pertubations of the messages could give insight into how the individual elements of messages alter the receivers’ reconstruction. This would moreover strengthen the relation between the datasets’ attributes and the actual image features (point 4 of the weaknesses). Although not explicitly mentioned, I assume that TopSim is calculated using the values of factors G instead of the pixel values, is this correct?\n3. Figure 2a. Could the authors explain why the message useful length does not decrease as a result of higher lambda? Please correct me if I am wrong, but it seems to me that the introduced pressure (line 234) must also result in shorter useful length since a wrong prediction should more heavily incentive the speaker to use shorter messages.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "As written in the summary, this is very elaborate work that incorporates methods and insights from multiple fields, making it an interesting contribution. Please see my assessment below.\n\nStrengths:\n1. Compositional generalisation is relevant in many areas concerning deep learning. This is also the case in emergent communication and has been a topic of interest for quite some time. This work is informed by findings in human language evolution, making comparisons with humans possible. Yet, this is not explicitly done, the authors draw on many relevant previous findings in emergent communication but fail to exactly show how this work relates to those findings. Nevertheless, I think this work is a step in the right direction. \n2. I believe that the main contribution of this paper is the introduction of final-state imitation. Using the receivers’ interpretation to calculate an imitation loss is original and brings the simulation somewhat closer to human experiments with iterated learning in which a listener becomes the teacher of the next generation of listeners. It is however not clear how many generations are simulated. \n3. The introduction of the ‘useful length’ metric is interesting. It allows the introduction of the geometric penalty (which seems to be undefined), and thereby shows that artificial agents, similar to humans, also show a trade-off between efficiency and expressivity.\n4. The quality of this paper is good, it is fairly easy to follow, despite containing a considerable number of derivations which, in my view, are not necessary to make the point. \n\nWeaknesses:\n1. My main concern with this paper is that the results are somewhat marginal. Although the limitations section states that the results are significant (line 331, for which no proof is provided), the results in table 1 seem to be rather small. Some of the means and error margins overlap between conditions. For example, the authors state that final state imitation enhances compositionality and outperforms the non-IL baseline and surpasses message-level imitation (lines 248 – 250) but no statistical tests are provided. It would be appreciated if the authors could provide evidence regarding the significance of the results. Assuming that the results are significant, it is difficult to assess  from only TopSim whether the proposed methods advance compositional generalisation or teaches something about human language evolution. A clearer framing on what this paper is about would help. In case of the latter, it would be necessary to show some of the learned languages and try to interpret them (even though this is difficult). In case of the former, I would like to read some discussion as to why the proposed mechanisms are meaningful for non-emergent communication generalisation / how they extrapolate beyond emergent communication. \n2. The results regarding length of useful messages in the progressive decoding objective are not necessarily surprising since the method, please correct me if I am wrong, essentially enforces the listeners to make a prediction early (line125, 126). What do we learn from this? Moreover, Table 2 does not convince me that the reconstruction error is improved resulting from PD. \n3. Similar to the previous point, it seems that the results of table 2 are somewhat oversold in section 4.6. For example: in the case of the Shapes3D dataset, an increase of 0.002 with an error margin of 0.001 does not seem to be substantial nor significant. It is a bit more problematic since RMSE and the accuracy are worse for CELEBI. While it is difficult to unravel the underlying meanings of messages in emergent communication, these results do not inform us much of how the proposed CELEBI framework is different from other work. Some qualitative inspection would help. Similarly, if the focus of this paper is generalisation, then it seems to me, but I am willing to be convinced otherwise, that a mechanism is only advancing generalisation if the RMSE and accuracy would also improve. Could the authors elaborate on this?\n4. Message stucture is currently only measured through TopSim. While it is indeed widely used in emergent communication literature, this metric also has limitations  [1, 2, 3]. In combination with images, it is unclear whether the agents actually use image features to create their messages [1] and it is argued that TopSim is not appropriate [2]. Moreover, TopSim does not capture compositionality when languages show variability [3]. Instead of relying on TopSim only, I would suggest to expand the set of metrics to include homonymy, synonymy, and word-order freedom as proposed by [3]. In addition to this, it would be helpful if the authors could qualitatively show that messages are indeed more structured when, e.g., pairwise distance maximisation is used (section 4.5).\n5. Minor weakness: While the related work section suggests that progressive decoding is different from LazImpa, I find this somewhat misleading. Variable message lengths should not meaningfully change the effect of a pressure to make early predictions. Moreover, the PD objective also seems to implicitly be a length penalty (line 125, 126).  \n\n[1] Kouwenhoven et al. 2024. The curious case of representational alignment: Unravelling visio-linguistic tasks in emergent communication\n[2] Chaabouni et al. 2022., Emergent Communication at Scale. \n[3] Conklin et al. 2023. Compositionality with Variation Reliably Emerges Between Neural Networks\n\nMinor issues:\n1. It would be easier to understand the figures and tables if their captions would be more informative. The table captions could include the take-away message from that table, and incorporate the meanings of bold values / show significance values. The figures would similarly benefit: figure 1a is very clear, but how exactly should I interpret 1b? Are these generations, or predictions after each message component? Figure 2 similarly only discusses figure b. \n2. Subsections 2.1 and 2.2 could be merged into one subsection to reduce space. They seem to explain the same thing. \n3. Lines 137 – 139: iterated learning does not *force* languages to be easy to learn, the languages resulting from IL are easier to learn because the elements that are easy to learn / compress are more likely to be transmitted. The end result of several generations of IL is that the languages are thus easier to learn. \n4. Line 156: during THE imitation phase\n5. Line 202: perhaps you could change the section name to “Methods” and start the result section at what is currently subsection 4.3.\n6. Line 266: decrease?", "was_revised": true, "final_justification": "Convinced by additional simulations, including statistical analyses and an additional metric for compositionality, presented by the authors during the rebuttal period, I increased my score with 1 point.", "rebuttal": "### Rebuttal Reviewer rCnw\n\nWe thank the reviewer for their time and thoughtful response.\n\nThe reviewer asks whether our paper is primarily about emergent communication or compositional generalization. In hindsight, we recognize the abstract may have caused confusion. Our main focus is emergent communication, and we will clarify this in the revised abstract.\n\n### Weaknesses\n\n1. **Response to two concerns**: (reference numbers [1], [2], [3] as in the reviewer's text, [64] as in our paper's draft)\n   1. **Table 1 and lines 331, 248–250**: The reviewer rightly notes the original Table 1 showed marginal improvements. We reran all methods with 5 additional seeds to increase statistical power and performed permutation tests across interaction, imitation, and regularization methods (see response to reviewer Likm). We’ll include these in the supplementary materials. The updated table shows a clearer trend—final state imitation generally outperforms message imitation—but statistical significance was only observed in MPI3D. We will clarify this.\n   2. **Positioning beyond emergent communication**: The main insight is that pressures for signal complexity (useful length), expressivity (reconstruction error), and compositionality (topsim) arise from different aspects of the interaction setup and must be balanced for more compositional representations. This underlies our three innovations. While generalizing beyond emergent communication is a broader debate (e.g., [1]), our approach resembles complexity regularization methods like self-distillation [2]. We will elaborate on this in the \"Limitations and Future Work\" section.\n\n2. **Progressive Decoding nontriviality**: PD acts as a length penalty, but the trade-off it induces with reconstruction accuracy is nuanced. As shown in Figure 3 (App. D), PD can prioritize efficiency (shorter useful length) over accuracy. The tuning of λ becomes critical. Partial decoding does not necessarily improve reconstruction but enhances efficiency. The key challenge is balancing concise and expressive messages while avoiding ambiguity.\n\n3. **Table 2 – two clarifications**:\n   1. **RMSE metric**: Table 2’s RMSE does not reflect reconstruction error but evaluates representations on a downstream regression task, following [64]. This differs from true reconstruction performance and is less directly tied to compositionality and interpretability (similar to discriminative metrics in 4.iii). We'll clarify this distinction and its implications in the final version, as also noted by reviewer Likm.\n   2. **Accuracy and significance**: We increased each method to 10 runs (see updated table) and ran permutation tests comparing CELEBI to the EL baseline. Relevant comparisons: In Shapes3D, CELEBI outperforms EL in accuracy (p=0.0017), RMSE (p=0.001), and disentanglement (p=0.0041). In MPI3D, CELEBI exceeds EL in RMSE (p=0.0009) and disentanglement (p=0.0001). Full results will be in the paper.\n\n4. **Multiple points**:\n   1. **Interpretability metrics**: Following the suggestion, we added synonymy, homonymy, freedom, and disentanglement (equal to \"1 - entanglement\" from Conklin et al. [1]) to the supplementary. Permutation tests show PD and PDM reduce entanglement. In Shapes3D, PD outperforms the interaction baseline (p=0.00059, diff=0.041); in MPI3D (p=0.0425, diff=0.064). PDM beats KoLeo (p=0.0052, diff=0.0195). Since reducing entanglement requires consistent encoding of different meanings, this supports PD's role in producing more compositional representations. We’ll expand on this in the final version.\n   2. **Qualitative results**: Besides the above metrics which we will add in a table, Appendix I contains generated examples showing regularities. We’ll add comparisons across methods to highlight improvements.\n   3. **Representation alignment from [3]**: In contrast to [3], where representations were not explicitly aligned with visual features, our setup uses a reconstruction task. Since agents must reconstruct images, the emergent language aligns naturally with underlying visual features. Topsim emerges from the correlation between generative factors and pixels—not class labels—ensuring grounded representations.\n   4. **Message structure**: Intermediate reconstructions often show meaningful transitions (App. I), unlike denoising noise. We'll add more examples to strengthen this point.\n\n5. **Comparison with LazImpa**: Our PD loss is conceptually simpler and more interpretable. LazImpa's penalties depend on model-specific accuracy predictions, whereas ours apply a single global term (eq. 1), creating a clearer, system-wide efficiency pressure. Unlike LazImpa, where message length pressure is split across roles, PD treats the system as a whole. We’ll clarify this conceptual distinction in the revised version.\n\n---\n\n### Minor Issues\n1. We will add takeaways in figure captions.\n2. We will merge overlapping background sections.\n3. We’ll revise to: \"...favoring easy to learn languages...\"\n4. Thank you!\n5. We will use \"experimental setup\" instead of \"methodology.\"\n6. The word “increase” refers to worse reconstruction, which we believe is correct.\n\n---\n\n### Questions\n1. We have run permutation tests (see reviewer Likm) and will include them in the supplementary material.\n2. Topsim is computed using generating factors. We will clarify this. We’ll also add qualitative examples showing the effect of changing message symbols on reconstruction. Without PDF support, we’ll reference these in Appendix I and describe the new figures.\n3. **Effect of λ**: As λ increases, the PD loss asymptotically resembles full reconstruction, since the highest-order term dominates. In practice, λ ≈ 2.5 already saturates useful length. We’ll extend Figure 2 to λ=10 and explain this in §3.1.\n\n---\n\n### Table: Disentanglement Metrics for Shapes3D and MPI3D\n\n| Dataset  | Model        | Disentanglement ↑ | RMSE ↓        | Acc ↑          |\n|----------|--------------|-------------------|---------------|----------------|\n| Shapes3D | β-VAE        | 0.045 ± 0.004     | 2.460 ± 0.144 | 0.702 ± 0.025  |\n|          | β-TCVAE      | 0.043 ± 0.005     | **2.378 ± 0.123** | **0.721 ± 0.028** |\n|          | VAE + EL     | 0.108 ± 0.000     | 3.168 ± 0.027 | 0.343 ± 0.019  |\n|          | VAE + CELEBI | **0.112 ± 0.001** | 2.932 ± 0.045 | 0.453 ± 0.024  |\n| MPI3D    | β-VAE        | 0.031 ± 0.001     | 21.819 ± 0.111 | **0.509 ± 0.006** |\n|          | β-TCVAE      | 0.031 ± 0.001     | 21.854 ± 0.188 | 0.582 ± 0.005  |\n|          | VAE + EL     | 0.114 ± 0.002     | 15.971 ± 0.301 | 0.529 ± 0.006  |\n|          | VAE + CELEBI | **0.137 ± 0.002** | **14.82 ± 0.012** | 0.542 ± 0.002  |"}, {"review_id": "hlL7qglfvA", "reviewer": "Reviewer_gfpt", "summary": "The authors study the notion of compositional generalization through a communication framework - the task for a receiver is to decode the sender's image based on finite-space messages provided from the sender. These messages define the framework for compositional generalization in language in the sense that a compositional language for sending messages can lead to more expressive and also more compressive signaling scheme. The authors propose three additions to the existing scheme to allow for more informative (and hence more compositional) messages in the signaling phase: progressive decoding to provide intermediate signal which can lead to providing more informative and useful signals earlier on, a final-state imitation procedure that provides informative signal for the full message decoding, and pairwise distance maximization which incentivizes diversity in messages. Their overall scheme leads to improved performance in terms of reconstruction metrics as well as the length of messages required.", "questions": "- Could the authors provide further clarification on the use of Equation (3)? How does this contrast to progressive decoding since it also has a term that corresponds to final reconstruction after full message?", "limitations": "No further limitations than what is discussed above.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is clearly written and tackles an interesting and relevant problem.\n- The use of progressive decoding is quite intuitive since in language, we do want to maintain that earlier symbols make the future predictable which is the underlying objective here.\n- The evaluation clearly shows improved performance on both output space similarity as well as length of sentences.\n\n**Weaknesses**\n\n- Current evaluation is solely on synthetic problems and it is not clear how this framework extends to problems that can have more complex objects as well as relations between the objects.\n- An important question not being directly tackled here is the case where the language of communication is English; how does this framework change and behave when we rely on such a language? A core reason for compositional generalization is the fact that the same language is useful for a wide variety of tasks, and such an analysis is missing from the current experiments.\n- While the authors claim to tackle compositional generalization, they aren't really evaluating for this from the metrics. Have the authors considered multi-object scenes where evaluation involves asking a pre-trained model whether certain relations are respected or not in reconstruction?", "was_revised": true, "final_justification": "The authors propose a framework for studying and improving emergent communication schemes through progressive decoding, final-state imitation and use of a diversity regularizer. Since the topic of research in this work is outside my area of expertise, I have understood the high-level motivation behind each of the proposed contribution but am not familiar with the details - this is reflected in my score and confidence.", "rebuttal": "### Rebuttal Reviewer gfpt\n\nWe thank the reviewer for their time and thoughtful response.\n\nThe reviewer's concerns primarily highlight the need for evaluation on more realistic or complex datasets, or the use of English for communication. This may reflect a misunderstanding of our paper’s main positioning, which focuses on emergent language via iterated learning. In this setting, controlled scenarios are useful in multiple ways, and using English as a medium of emergent communication is not straightforward. We expand on this below.\n\n---\n\n### Weaknesses\n\n1. **Evaluation via non-synthetic problems**:  \n   Thank you for raising this important point.\n\n   We acknowledge that our experiments are conducted on synthetic datasets. This is intentional and *aligned with common practice* in the emergent communication and compositional generalization literature. Controlled environments are essential for systematically studying inductive biases and the emergence of structured communication protocols (indeed, nearly all cited references on emergent language use synthetic datasets of comparable complexity—e.g., [5], [12], [14], [23], [24], [35], [37], [38], [44], [47], [48], [52], [58], [64]).\n\n   Specifically, synthetic datasets provide several benefits:\n   1. **Controlled data distributions**: They allow precise control over training and test distributions—especially important for implementing compositional splits that test generalization to novel attribute combinations, which is central to our study.  \n   2. **Ground-truth factorization**: Meaningful evaluation of compositionality requires access to generative factors (e.g., shape, color, size). Real-world datasets often lack this disentangled ground truth, making metrics like topographic similarity difficult or ambiguous to apply.  \n   3. **Complexity minimization**: Following Occam’s razor, we avoid unnecessary complexity to preserve clarity in causal interpretation. Introducing relational reasoning or complex object structures would increase dimensionality and introduce new, confounding effects that could obscure the impact of our proposed mechanisms.\n\n   While evaluating more complex and realistic tasks is a valuable future direction (as noted below), such extensions introduce orthogonal challenges. We believe our current setup provides a principled and interpretable testbed for isolating the effects of our mechanisms on compositionality. We’ve added a note about this extension path in the discussion section.\n\n2. **English as a communication medium**:  \n   We appreciate the reviewer’s thoughtful comment. However, this concern seems to arise from a misunderstanding of our objective.\n\n   Our framework does not aim to model or analyze evolved human languages such as English, but instead studies the emergence of **compositional communication protocols de novo** between artificial agents (see lines 44–47). This is motivated by foundational questions in cognitive science [29], where a key challenge is understanding how compositional structures arise from basic communicative dynamics and learning mechanisms.\n\n   A core theoretical motivation in studying emergent language is to test whether compositionality emerges naturally from optimizing for efficient and generalizable communication, as proposed by recent information-theoretic accounts [43]. Thus, our work should be viewed in the context of **emergent communication and compositional learning**, not natural language modeling.\n\n   While English is indeed compositional, it is the product of centuries of cultural evolution shaped by complex social and historical factors—many of which are not tractable in computational models. Emergent languages, in contrast, enable **precise control and isolation of contributing factors**, allowing systematic analysis of their role in shaping communication.\n\n3. **Multi-object scenes and relational reasoning**:  \n   1. We respectfully disagree with the claim that our work does not evaluate compositional generalization. As noted in line 212, we adopt the compositional generalization split introduced by [52], which is specifically designed to test generalization to novel combinations of known factors—a core benchmark for compositionality. We apply this to two well-established datasets, Shapes3D and MPI3D, widely used for analyzing compositional structure in learned representations.  \n   2. Evaluating multi-object scenes with relational queries is indeed an exciting direction, but we believe it is outside the scope of our current study. Our goal is to isolate and understand how our mechanisms affect emergent message structure and efficiency. Adding relational reasoning would introduce additional complexity and confounds, making attribution to our contributions less clear.  \n   3. We focus intentionally on the simplest form of compositionality: generalization over novel combinations of independent latent factors (e.g., shape, color, size). We do not address more complex compositional structures (e.g., syntax or multi-object relational reasoning), which introduce additional challenges (see 1.iii above).\n\n   Nonetheless, we appreciate the reviewer’s suggestion and plan to explore relational settings in future work.\n\n---\n\n### References\n[29] Simon Kirby et al., “Compression and communication in the cultural evolution of linguistic structure,” *Cognition*, 2015.  \n[43] Yi Ren et al., “Compositional languages emerge in a neural iterated learning model,” 2020.  \n[52] Lukas Schott et al., “Visual representation learning does not generalize strongly within the same domain,” ICLR, 2022.\n\n---\n\n### Questions\n\nTo our understanding, the reviewer asks how Eq. (3) contrasts with Eq. (1) used for progressive decoding (PD). The key distinction is that Eq. (3) applies to the **imitation phase**, whereas Eq. (1) pertains to the **interaction phase**, which has significant implications:\n\n1. In the interaction phase (Eq. 1), both sender and receiver are unfrozen, and the loss measures reconstruction accuracy from input to output. In the imitation phase (Eq. 3), the receiver is frozen, and the sender imitates the *reconstruction* produced by the previous-generation sender, not the ground truth (see line 172).  \n2. The left and right components of Eq. (3) compare full vs. final reconstruction. Our choice—to use only final state reconstruction during imitation (as in our Eq. 2, corresponding to the right-hand side of Eq. 3)—supports broader sender exploration when combined with PDM (see §3.2.2). This differs fundamentally from PD, where sender and receiver jointly optimize under an efficiency constraint (minimizing useful message length) imposed by Eq. (1).\n\nWe hope this clears up the confusion—please let us know if we can clarify further."}, {"review_id": "DpyWPiVfux", "reviewer": "Reviewer_ct41", "summary": "The paper concerns the problem of learning more compositional representations in a discrete auto-encoder framework. For instance, given image data, how can we learn discrete latent representation that show more signs of compositionality? Their contribution is to propose three inductive biases for this, inspired by the cognitive science and AI literature on emergent communication:\n\n1. **Progressive reconstruction**: The input should be partially reconstructable from individual sub-messages in the latent space, rather than only from the entire message.\n2. **Iterative learning**: A student encoder distils the reconstruction behaviour from a previous generation’s frozen encoder/decoder, over the course of several generations. This sort of approach has been shown in prior work to increase the compositional structure of languages, and the authors’ version differs slightly in that the messages themselves are not imitated in subsequent generations (only the final output of the decoder).\n3. **Message entropy maximization**: The messages across different encoded inputs should be different, so as to improve expressivity and prevent collapse.\n\nThe paper then presents evidence that these three inductive biases improve the compositionality of representations without significantly degrading input reconstruction performance. They do this by looking at compositionality metrics (Topological Similarity, ground-truth generative factor decoding, Disentanglement) across two simple synthetic datasets in which the ground-truth generative factors are known.", "questions": "1. The progressive reconstruction loss might encourage two things, and it’s not entirely clear which contributes to the results: (1) short messages, (2) messages in which each individual word carries information about the message (i.e., not holistic). Which do you think is driving the results? 2 seems more related to compositionality (in fact, as you say, 1 could harm compositionality by encouraging the use of a single position with a very large vocabulary!), but at the same time it seems like there could be other, better regularizers for 2 centered around mutual information that wouldn’t also encourage (1).\n2. To what extent is the discretization important in any of this work? It’s meaningful to talk about compositionality w.r.t. continuous features, and all of your 3 proposed regularization methods would also work for a continuous latent space (or block-wise continuous space, as in the object-centric learning literature), would they not? I understand that you want to make a connection to the emergent language literature, but really your method is about learning compositional latent representations in an auto-encoder more generally, and nothing obviously requires the discrete aspect. At the very least, it would be interesting to know whether your inductive biases also contribute to compositional representations in fully continuous architectures.\n3. You say that larger values of $\\lambda$ can harm compositionality and reference Fig. 2. I don’t see that; in Fig. 2 TopSim just seems to plateau past 1 on both datasets. Can you clarify?\n4. You say that part of the entropy-maximization regularizer is to avoid collapse during the imitation phase. Why would the imitation phase result in collapse if the previous generation’s sender and receiver, as well as the current generation’s receiver, are frozen? I can maybe see how this could happen with *many* generations, given that a collapsed message would be a fixed point of this iterative learning process, but (a) its not obvious that the iterative learning dynamics would lead to this fixed point (e.g., they could instead diverge or rotate), and (b) presumably that would take many generations. Can you please explain?\n5. When discussing PDM results, you say that for MPI3D this setting achieves the highest TopSim but increases useful message length and reconstruction loss compared to PD alone. You then say that “this highlights a trade-off between expressivity and compressibility, in line with prior theoretical accounts on communication efficiency and compositionality”. I do not see how this result highlights this tradeoff. In particular, the pairwise distance maximization component has not further increased the structure or compressibility of the language (TopSim), nor has it increased the languages expressivity as demonstrated by the worse reconstruction performance. What do you mean, then?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. I like that the paper is trying to design multiple inductive biases to effectively improve the compositionality of discrete auto-encoders. It’s an important problem, and I think others can continue to build on this work in the future.\n2. For the most part, the paper is quite clearly written. It’s easy to get a sense of the motivation, mechanistically understand the methods, and interpret the experimental results. All of this should help broadcast the findings to the community and improve reproducibility.\n3. Although I’m going to list a lot of weaknesses and questions below, my overall impression of the paper is net positive as a result of the two points above: the motivation is good, the problem is important, and the paper is easy to digest.\n\n**Weaknesses**\n\n1. **Suggestion**: The emergent communication, expressivity/compressibility, and iterative learning literature from cognitive science and AI are significant starting points for this work. For people already familiar with the literature, the paper’s motivation and the components of the model are easy to follow and their justification is clear. For others unfamiliar with this line of work, I worry that the motivation and model components will seem obscure and ad-hoc. I think early on in the paper you can say a bit more about what this prior literature has found, and supplement it with some concrete examples of languages that emerge with and without pressures for compression like iterative learning.\n2. **Suggestion**: The progressive decoding part of the model is partly motivated in the text by chain-of-thought reasoning. I think the link here between progressive decoding, compositionality, and chain-of-thought reasoning is actually rather speculative and not entirely justified (a particular step in a chain-of-thought might not help solve the problem at all in isolation, and might only help in the context of other thoughts in the chain). Instead, a more direct link can be made to the systematic structure this progressive decoding induces in the message-to-image mapping. In particular, it makes the function simpler because each individual word must contribute to the meaning, which eliminates holistic mappings as a solution. It’s fairly easy for instance to see how progressive decoding can directly lead to higher topological similarity (your metric for compositionality), so you may as well motivate progressive decoding through this direct link to structured mappings and topological similarity in the first place.\n3. There’s a clear link to object-centric learning, in particular slot-type of architectures that have additive decoders, especially with regards to the progressive decoding component of the model. This should be discussed in the related work, in addition to the discussion on beta VAEs.\n4. The methods are grounded in emergent communication games, but I feel like they can go beyond that. For instance, such an encoder/decoder can be used in an auxiliary loss that places inductive bias for compositionality on another neural network’s representations for arbitrary tasks (i.e., train a neural network to solve some arbitrary task, consider some latent representation in this neural network as the “image” in the current framework, and then alternative training your encoder/decoder’s weights w.r.t. the losses in the current paper vs. training the representation w.r.t. its task loss and the losses in the current paper). Hopefully I’m being clear enough here, but I think such an approach could be quite interesting. I’m not saying this has to be done in the current work of course, but saying some more about how you think your work could be useful for representation learning in general (and not just emergent communication) would be helpful in broadening the scope of the ideas.\n5. There’s a lot of mathematical obfuscation in the section introducing FiSI that I don’t think is actually necessary, and at best can go in the Appendix. In particular, it seems completely clear that only imitating the final output of the previous iteration’s receiver provides more flexibility to the sender to construct different sorts of messages, and we don’t need to go through a tedious mathematical argument involving significant notation and a proposition in order to understand this. In my opinion this breaks the flow of the paper.\n6. Overall, one could argue that the improvements in compositionality (as measured by TopSim in Table 1) are not actually that significant. For instance, final state imitation yields only very marginal improvements over message imitation on both datasets, and these marginal improvements actually fall within the standard error. The results therefore don’t clearly support the proposed inductive biases as much as you seem to suggest, and I would be very curious to hear your rebuttal on this point — perhaps I’m missing something.", "was_revised": true, "final_justification": "Following the authors' rebuttal, I've chosen to maintain my score. The authors engaged with my feedback and have said that they will change the paper accordingly, but my main justification for not increasing my score is that the proposed regularizers and inductive biases do not increase the authors' compositionality score very much, and do harm task performance. The paper presents a nice set of ideas for inductive biases for compositionality, which I think is a research direction we should continue pursuing, but given that the effect of the inductive biases is not very string I think a score of \"borderline accept\" is fair.", "rebuttal": "### Rebuttal Reviewer ct41\n\nWe thank the reviewer for their deep and insightful response and valuable suggestions, which we believe will significantly improve the paper.\n\n---\n\n### Weaknesses / Suggestions\n\n1. We appreciate the suggestion to expand the introduction and related work to provide readers with necessary background on the model components and pressures in the iterated learning setup, which our strategy challenges. We agree this will help clarify the paper's positioning—thank you again for the recommendation!\n\n2. This suggestion has two parts:  \n   1. We agree that the link to chain-of-thought (CoT) is speculative and partial. The only shared property is that CoT reasoning steps, like our message tokens, exhibit compositional and sequential structure. However, CoT subsequences are not explicitly pressured to lead to a correct conclusion/output, making the analogy misleading. We plan to remove this comparison.  \n   2. Thank you for suggesting we clarify the connection between progressive decoding and compositional metrics like topographic similarity. While we mention this in line 127, we will add a clarifying sentence in the final version.\n\n3. Thank you for highlighting the connection to object-centric learning, slot-type architectures, and additive decoders. We will add a discussion of these in the related work section.\n\n4. We appreciate the suggestion to broaden the scope beyond emergent communication. We are indeed exploring further uses of our signaling architectures for representation learning, and the idea of cross-model communication is promising if fine-tuned appropriately. We will add a discussion of this in the \"Limitations and Future Work\" section.\n\n5. We agree with the suggestion to move Proposition 3.1 to the appendix and will do so in the final version.\n\n6. **Regarding Table 1**:  \n   To assess statistical significance, we added 5 seeds per method and ran permutation tests, confirming that the impact of our components on Topsim is statistically significant. The updated table and p-values are in the rebuttal to reviewer Likm; results for Table 2 are in the rebuttal to reviewer rCnw.  \n   Concerning inductive biases, we note that gains in Topsim are stronger when combining PD+FiSI+PDM, compared to applying them separately. That the components are less effective individually does not contradict our claims. On the contrary, it supports the motivation for combining them—each pressure alone may be insufficient, but together they produce synergy.  \n   For example, FiSI is meant to allow sender strategy drift, but this drift doesn't arise naturally from optimization. Hence, we introduce PDM to explicitly encourage such exploration. We will clarify this in the final version.\n\n---\n\n### Questions\n\n1. **Progressive reconstruction loss**:  \n   1. This loss clearly enforces (1)—penalizing longer messages—since later tokens contribute more to the loss. As for (2)—penalizing tokens that carry no information—we do not claim this. Instead, we argue it promotes interpretability in *submessages* (prefixes of increasing length), as the receiver is penalized for failure to reconstruct from partial messages.  \n   2. Encouraging shorter messages aligns with a core pressure in Iterated Learning theory. Thus, we prefer regularizers that enforce both compression and interpretability (e.g., progressive loss) over those that only target informativeness. Efficient encoding is a necessary (though not sufficient) condition for low Kolmogorov complexity, which underlies emergent compositionality. Appendix F explains why our regularization is needed, as training data covers only a small portion of possible factor combinations.\n\n2. **Use with continuous features**:  \n   In principle, our setup could be extended to continuous features. However, in practice, discrete and separated encodings have proven more effective for compositionality in iterated learning (see [45]). Discreteness mirrors human vocabulary and enables concept separation.  \n   Technically, discreteness supports a richer encoding space and helps avoid mode collapse, as shown in VQ-VAE literature. We believe this is especially important in combinatorially extended message spaces like ours, where collapse risks are amplified. We will emphasize this point in the related work section, linking it to object-centric learning.\n\n3. **Clarification on λ and TopSim**:  \n   Thank you for catching this. We previously wrote (lines 238–239) that “larger values of λ encourage more informative messages, but reduce alignment with generating factors,” to justify the claim that “intermediate values of λ are best.” As you noted, this isn't clearly supported by the data. What does drive the optimal λ value is the emergence of a plateau in TopSim, while useful length continues to increase. We will correct this explanation.\n\n4. **Clarification on “collapse”**:  \n   In hindsight, our use of “collapse” was poorly worded. We appreciate the reviewer pointing this out. What we meant was that, without additional pressure, a new sender has no incentive to drift from the previous sender’s trajectory—i.e., the decoding sequence from submessages of increasing length remains unchanged.  \n   By adding entropy regularization, we encourage exploration of alternative encoding strategies, similar to how entropy is used to promote exploration in reinforcement learning. The actual “collapse” occurs over generations. We will revise the wording accordingly.\n\n5. **Trade-off between expressivity and compressibility**:  \n   By this, we mean that adding PDM improves TopSim (from 0.141 to 0.16), but the other two metrics worsen (see second vs. last row in the second part of Table 1). This contrasts with the reviewer’s statement that PDM “has not further increased the structure or comprehensibility of the language (TopSim),” so we may be misunderstanding their comment. Did we miss something?"}]

Author rebuttal:

Meta-review: *Main Contribution*  
The paper proposes a novel approach to compositional representation learning through a compressive–expressive communication framework. On two synthetic datasets, the method demonstrates improved disentanglement (i.e., compositional representation), albeit at the cost of reduced reconstruction performance.  

*Strengths*  
The paper addresses an interesting problem and provides sufficient methodological contributions. The results are clearly presented, and the authors thoughtfully discuss both the limitations of their work and its connections to related research.  

*Weaknesses*  
Reviewers raised concerns about the significance of the empirical results, noting that they appear marginal. Another issue is the trade-off inherent in the approach: the method improves compositionality but sacrifices reconstruction performance. More comprehensive experiments are needed to fully evaluate this trade-off. In addition, the method represents only a modest modification of existing techniques, and the connection to chain-of-thought prompting remains unclear.  

*Rebuttal*  
In their rebuttal, the authors provided additional experimental results, which convinced some reviewers to increase their scores and partially addressed concerns regarding the practical significance of the method.  

*Justification of decision*  
Following the rebuttal, the reviewers reached a consensus of “borderline acceptance.”
Decision: Accept (poster)