Title: A Compressive-Expressive Communication Framework for Compositional Representations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Iterated learning

Emergent language

Compositionality

Compositional generalization

Compositional Representations
Abstract: Compositionality in knowledge and language—the ability to represent complex concepts as a combination of simpler ones—is a hallmark of human cognition and communication. Despite recent advances, deep neural networks still struggle to acquire this property reliably. Neural models for emergent communication look to endow artificial agents with compositional language by simulating the pressures that form human language. In this work, we introduce CELEBI (Compressive-Expressive Language Emergence through a discrete Bottleneck and Iterated learning), a novel self-supervised framework for inducing compositional representations through a reconstruction-based communication game between a sender and a receiver. Building on theories of language emergence and the iterated learning framework, we integrate three mechanisms that jointly promote compressibility, expressivity, and efficiency in the emergent language. First, Progressive Decoding incentivizes intermediate reasoning by requiring the receiver to produce partial reconstructions after each symbol. Second, Final-State Imitation trains successive generations of agents to imitate reconstructions rather than messages, enforcing a tighter communication bottleneck. Third, Pairwise Distance Maximization regularizes message diversity by encouraging high distances between messages, with formal links to entropy maximization. Our method significantly improves both the efficiency and compositionality of the learned messages on the Shapes3D and MPI3D datasets, surpassing prior discrete communication frameworks in both reconstruction accuracy and topographic similarity. This work provides new theoretical and empirical evidence for the emergence of structured, generalizable communication protocols from simplicity-based inductive biases.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper concerns the problem of learning more compositional representations in a discrete auto-encoder framework. For instance, given image data, how can we learn discrete latent representation that show more signs of compositionality? Their contribution is to propose three inductive biases for this, inspired by the cognitive science and AI literature on emergent communication:\n\n1. **Progressive reconstruction**: The input should be partially reconstructable from individual sub-messages in the latent space, rather than only from the entire message.\n2. **Iterative learning**: A student encoder distils the reconstruction behaviour from a previous generation’s frozen encoder/decoder, over the course of several generations. This sort of approach has been shown in prior work to increase the compositional structure of languages, and the authors’ version differs slightly in that the messages themselves are not imitated in subsequent generations (only the final output of the decoder).\n3. **Message entropy maximization**: The messages across different encoded inputs should be different, so as to improve expressivity and prevent collapse.\n\nThe paper then presents evidence that these three inductive biases improve the compositionality of representations without significantly degrading input reconstruction performance. They do this by looking at compositionality metrics (Topological Similarity, ground-truth generative factor decoding, Disentanglement) across two simple synthetic datasets in which the ground-truth generative factors are known.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. I like that the paper is trying to design multiple inductive biases to effectively improve the compositionality of discrete auto-encoders. It’s an important problem, and I think others can continue to build on this work in the future.\n2. For the most part, the paper is quite clearly written. It’s easy to get a sense of the motivation, mechanistically understand the methods, and interpret the experimental results. All of this should help broadcast the findings to the community and improve reproducibility.\n3. Although I’m going to list a lot of weaknesses and questions below, my overall impression of the paper is net positive as a result of the two points above: the motivation is good, the problem is important, and the paper is easy to digest.\n\n**Weaknesses**\n\n1. **Suggestion**: The emergent communication, expressivity/compressibility, and iterative learning literature from cognitive science and AI are significant starting points for this work. For people already familiar with the literature, the paper’s motivation and the components of the model are easy to follow and their justification is clear. For others unfamiliar with this line of work, I worry that the motivation and model components will seem obscure and ad-hoc. I think early on in the paper you can say a bit more about what this prior literature has found, and supplement it with some concrete examples of languages that emerge with and without pressures for compression like iterative learning.\n2. **Suggestion**: The progressive decoding part of the model is partly motivated in the text by chain-of-thought reasoning. I think the link here between progressive decoding, compositionality, and chain-of-thought reasoning is actually rather speculative and not entirely justified (a particular step in a chain-of-thought might not help solve the problem at all in isolation, and might only help in the context of other thoughts in the chain). Instead, a more direct link can be made to the systematic structure this progressive decoding induces in the message-to-image mapping. In particular, it makes the function simpler because each individual word must contribute to the meaning, which eliminates holistic mappings as a solution. It’s fairly easy for instance to see how progressive decoding can directly lead to higher topological similarity (your metric for compositionality), so you may as well motivate progressive decoding through this direct link to structured mappings and topological similarity in the first place.\n3. There’s a clear link to object-centric learning, in particular slot-type of architectures that have additive decoders, especially with regards to the progressive decoding component of the model. This should be discussed in the related work, in addition to the discussion on beta VAEs.\n4. The methods are grounded in emergent communication games, but I feel like they can go beyond that. For instance, such an encoder/decoder can be used in an auxiliary loss that places inductive bias for compositionality on another neural network’s representations for arbitrary tasks (i.e., train a neural network to solve some arbitrary task, consider some latent representation in this neural network as the “image” in the current framework, and then alternative training your encoder/decoder’s weights w.r.t. the losses in the current paper vs. training the representation w.r.t. its task loss and the losses in the current paper). Hopefully I’m being clear enough here, but I think such an approach could be quite interesting. I’m not saying this has to be done in the current work of course, but saying some more about how you think your work could be useful for representation learning in general (and not just emergent communication) would be helpful in broadening the scope of the ideas.\n5. There’s a lot of mathematical obfuscation in the section introducing FiSI that I don’t think is actually necessary, and at best can go in the Appendix. In particular, it seems completely clear that only imitating the final output of the previous iteration’s receiver provides more flexibility to the sender to construct different sorts of messages, and we don’t need to go through a tedious mathematical argument involving significant notation and a proposition in order to understand this. In my opinion this breaks the flow of the paper.\n6. Overall, one could argue that the improvements in compositionality (as measured by TopSim in Table 1) are not actually that significant. For instance, final state imitation yields only very marginal improvements over message imitation on both datasets, and these marginal improvements actually fall within the standard error. The results therefore don’t clearly support the proposed inductive biases as much as you seem to suggest, and I would be very curious to hear your rebuttal on this point — perhaps I’m missing something.\n\nQUESTIONS:\n1. The progressive reconstruction loss might encourage two things, and it’s not entirely clear which contributes to the results: (1) short messages, (2) messages in which each individual word carries information about the message (i.e., not holistic). Which do you think is driving the results? 2 seems more related to compositionality (in fact, as you say, 1 could harm compositionality by encouraging the use of a single position with a very large vocabulary!), but at the same time it seems like there could be other, better regularizers for 2 centered around mutual information that wouldn’t also encourage (1).\n2. To what extent is the discretization important in any of this work? It’s meaningful to talk about compositionality w.r.t. continuous features, and all of your 3 proposed regularization methods would also work for a continuous latent space (or block-wise continuous space, as in the object-centric learning literature), would they not? I understand that you want to make a connection to the emergent language literature, but really your method is about learning compositional latent representations in an auto-encoder more generally, and nothing obviously requires the discrete aspect. At the very least, it would be interesting to know whether your inductive biases also contribute to compositional representations in fully continuous architectures.\n3. You say that larger values of $\\lambda$ can harm compositionality and reference Fig. 2. I don’t see that; in Fig. 2 TopSim just seems to plateau past 1 on both datasets. Can you clarify?\n4. You say that part of the entropy-maximization regularizer is to avoid collapse during the imitation phase. Why would the imitation phase result in collapse if the previous generation’s sender and receiver, as well as the current generation’s receiver, are frozen? I can maybe see how this could happen with *many* generations, given that a collapsed message would be a fixed point of this iterative learning process, but (a) its not obvious that the iterative learning dynamics would lead to this fixed point (e.g., they could instead diverge or rotate), and (b) presumably that would take many generations. Can you please explain?\n5. When discussing PDM results, you say that for MPI3D this setting achieves the highest TopSim but increases useful message length and reconstruction loss compared to PD alone. You then say that “this highlights a trade-off between expressivity and compressibility, in line with prior theoretical accounts on communication efficiency and compositionality”. I do not see how this result highlights this tradeoff. In particular, the pairwise distance maximization component has not further increased the structure or compressibility of the language (TopSim), nor has it increased the languages expressivity as demonstrated by the worse reconstruction performance. What do you mean, then?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the notion of compositional generalization through a communication framework - the task for a receiver is to decode the sender's image based on finite-space messages provided from the sender. These messages define the framework for compositional generalization in language in the sense that a compositional language for sending messages can lead to more expressive and also more compressive signaling scheme. The authors propose three additions to the existing scheme to allow for more informative (and hence more compositional) messages in the signaling phase: progressive decoding to provide intermediate signal which can lead to providing more informative and useful signals earlier on, a final-state imitation procedure that provides informative signal for the full message decoding, and pairwise distance maximization which incentivizes diversity in messages. Their overall scheme leads to improved performance in terms of reconstruction metrics as well as the length of messages required.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is clearly written and tackles an interesting and relevant problem.\n- The use of progressive decoding is quite intuitive since in language, we do want to maintain that earlier symbols make the future predictable which is the underlying objective here.\n- The evaluation clearly shows improved performance on both output space similarity as well as length of sentences.\n\n**Weaknesses**\n\n- Current evaluation is solely on synthetic problems and it is not clear how this framework extends to problems that can have more complex objects as well as relations between the objects.\n- An important question not being directly tackled here is the case where the language of communication is English; how does this framework change and behave when we rely on such a language? A core reason for compositional generalization is the fact that the same language is useful for a wide variety of tasks, and such an analysis is missing from the current experiments.\n- While the authors claim to tackle compositional generalization, they aren't really evaluating for this from the metrics. Have the authors considered multi-object scenes where evaluation involves asking a pre-trained model whether certain relations are respected or not in reconstruction?\n\nQUESTIONS:\n- Could the authors provide further clarification on the use of Equation (3)? How does this contrast to progressive decoding since it also has a term that corresponds to final reconstruction after full message?\n\nLIMITATIONS:\nNo further limitations than what is discussed above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigated compositional generalisation in an emergent communication paradigm through a newly introduced framework (CELEBI). This framework comprises of three mechanisms: interactive decoding, reconstruction-based imitation learning, and pairwise distance maximisation. In a series of experiments and ablations, the papers claims that the emerged languages are more efficient and compositional as opposed to standard (baseline) emergent communication methods on two image-based datasets. The paper incorporated many experiments and derivations that make it a complete paper, but I found it somewhat difficult to assess exactly what the main position of this paper was – emergent communication or compositional generalisation. I did not completely check the correctness of the mathematical explanations and proposition in the main paper and appendix, I adjusted my confidence accordingly.\n\nSTRENGTHS AND WEAKNESSES:\nAs written in the summary, this is very elaborate work that incorporates methods and insights from multiple fields, making it an interesting contribution. Please see my assessment below.\n\nStrengths:\n1. Compositional generalisation is relevant in many areas concerning deep learning. This is also the case in emergent communication and has been a topic of interest for quite some time. This work is informed by findings in human language evolution, making comparisons with humans possible. Yet, this is not explicitly done, the authors draw on many relevant previous findings in emergent communication but fail to exactly show how this work relates to those findings. Nevertheless, I think this work is a step in the right direction. \n2. I believe that the main contribution of this paper is the introduction of final-state imitation. Using the receivers’ interpretation to calculate an imitation loss is original and brings the simulation somewhat closer to human experiments with iterated learning in which a listener becomes the teacher of the next generation of listeners. It is however not clear how many generations are simulated. \n3. The introduction of the ‘useful length’ metric is interesting. It allows the introduction of the geometric penalty (which seems to be undefined), and thereby shows that artificial agents, similar to humans, also show a trade-off between efficiency and expressivity.\n4. The quality of this paper is good, it is fairly easy to follow, despite containing a considerable number of derivations which, in my view, are not necessary to make the point. \n\nWeaknesses:\n1. My main concern with this paper is that the results are somewhat marginal. Although the limitations section states that the results are significant (line 331, for which no proof is provided), the results in table 1 seem to be rather small. Some of the means and error margins overlap between conditions. For example, the authors state that final state imitation enhances compositionality and outperforms the non-IL baseline and surpasses message-level imitation (lines 248 – 250) but no statistical tests are provided. It would be appreciated if the authors could provide evidence regarding the significance of the results. Assuming that the results are significant, it is difficult to assess  from only TopSim whether the proposed methods advance compositional generalisation or teaches something about human language evolution. A clearer framing on what this paper is about would help. In case of the latter, it would be necessary to show some of the learned languages and try to interpret them (even though this is difficult). In case of the former, I would like to read some discussion as to why the proposed mechanisms are meaningful for non-emergent communication generalisation / how they extrapolate beyond emergent communication. \n2. The results regarding length of useful messages in the progressive decoding objective are not necessarily surprising since the method, please correct me if I am wrong, essentially enforces the listeners to make a prediction early (line125, 126). What do we learn from this? Moreover, Table 2 does not convince me that the reconstruction error is improved resulting from PD. \n3. Similar to the previous point, it seems that the results of table 2 are somewhat oversold in section 4.6. For example: in the case of the Shapes3D dataset, an increase of 0.002 with an error margin of 0.001 does not seem to be substantial nor significant. It is a bit more problematic since RMSE and the accuracy are worse for CELEBI. While it is difficult to unravel the underlying meanings of messages in emergent communication, these results do not inform us much of how the proposed CELEBI framework is different from other work. Some qualitative inspection would help. Similarly, if the focus of this paper is generalisation, then it seems to me, but I am willing to be convinced otherwise, that a mechanism is only advancing generalisation if the RMSE and accuracy would also improve. Could the authors elaborate on this?\n4. Message stucture is currently only measured through TopSim. While it is indeed widely used in emergent communication literature, this metric also has limitations  [1, 2, 3]. In combination with images, it is unclear whether the agents actually use image features to create their messages [1] and it is argued that TopSim is not appropriate [2]. Moreover, TopSim does not capture compositionality when languages show variability [3]. Instead of relying on TopSim only, I would suggest to expand the set of metrics to include homonymy, synonymy, and word-order freedom as proposed by [3]. In addition to this, it would be helpful if the authors could qualitatively show that messages are indeed more structured when, e.g., pairwise distance maximisation is used (section 4.5).\n5. Minor weakness: While the related work section suggests that progressive decoding is different from LazImpa, I find this somewhat misleading. Variable message lengths should not meaningfully change the effect of a pressure to make early predictions. Moreover, the PD objective also seems to implicitly be a length penalty (line 125, 126).  \n\n[1] Kouwenhoven et al. 2024. The curious case of representational alignment: Unravelling visio-linguistic tasks in emergent communication\n[2] Chaabouni et al. 2022., Emergent Communication at Scale. \n[3] Conklin et al. 2023. Compositionality with Variation Reliably Emerges Between Neural Networks\n\nMinor issues:\n1. It would be easier to understand the figures and tables if their captions would be more informative. The table captions could include the take-away message from that table, and incorporate the meanings of bold values / show significance values. The figures would similarly benefit: figure 1a is very clear, but how exactly should I interpret 1b? Are these generations, or predictions after each message component? Figure 2 similarly only discusses figure b. \n2. Subsections 2.1 and 2.2 could be merged into one subsection to reduce space. They seem to explain the same thing. \n3. Lines 137 – 139: iterated learning does not *force* languages to be easy to learn, the languages resulting from IL are easier to learn because the elements that are easy to learn / compress are more likely to be transmitted. The end result of several generations of IL is that the languages are thus easier to learn. \n4. Line 156: during THE imitation phase\n5. Line 202: perhaps you could change the section name to “Methods” and start the result section at what is currently subsection 4.3.\n6. Line 266: decrease?\n\nQUESTIONS:\n1. As mentioned in points 1,2 and 3, I am not convinced of the effectiveness of the (individual) proposed mechanisms. Could the authors provide information regarding the test for significance and the resulting statistics? If this is available, adding it to table 1 and 2 would be appreciated. \n2. Some qualitative interpretation of the emerged languages would be very valuable and strengthen the claims of this paper. How exactly do the messages change as a result of the individual mechanisms? Here, I would be especially interested in the changes resulting from PDM. In addition to manually looking for message meanings, directed pertubations of the messages could give insight into how the individual elements of messages alter the receivers’ reconstruction. This would moreover strengthen the relation between the datasets’ attributes and the actual image features (point 4 of the weaknesses). Although not explicitly mentioned, I assume that TopSim is calculated using the values of factors G instead of the pixel values, is this correct?\n3. Figure 2a. Could the authors explain why the message useful length does not decrease as a result of higher lambda? Please correct me if I am wrong, but it seems to me that the introduced pressure (line 234) must also result in shorter useful length since a wrong prediction should more heavily incentive the speaker to use shorter messages.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed CELEBI, a method for learning compositional representations that combines interactive decoding (partial reconstruction from intermediate representations)  and an imitation (similar to iterated learning). This method is evaluated on two datasets, Shapes3D and MPI3D, demonstrating its strengths compared to previous methods for constructing discrete representations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This paper addresses an important question (i.e., how to learn compositional representations) by proposing a new method that combines existing techniques in a novel way. As such, it present an original work that could potentially be interesting and valuable for the NeurIPS community. \n- The chosen datasets, experiments, and baselines seem adequate for evaluating the method. \n\n**Weaknesses**\n- The results seem somewhat incremental, making it hard to assess the significance of this work. That is, the quantitative results suggests only slight improvement compared to previous methods, and in particular, it seems that the proposed method does not perform quite well in terms of accuracy (Table 2). Perhaps even more importantly, it's unclear what is the nature of the learned compositional structure and what new insight it affords. I could see value in a new method that provides compositional representations that are more insightful and interpretable than prior methods, even if that method doesn't achieve the best possible accuracy, but I don't think this was demonstrated in the paper. I therefore think that the paper could benefit from a deeper analysis of the learned compositional structure.\n- The paper could also benefit from more clarity in the presentation. For example, the notation is not always clear (see questions below), and the baselines are not sufficiently explained.\n\nQUESTIONS:\n- I find the notation quite confusing, especially for Eq. 0 (the unnumbered display equation at the top of p3, right after line 95). Does GenX take in the set $\\mathcal{G}$ or just $G$ (as in line 71)? What is the input to $S$, $\\mathcal{D}$ or $\\mathcal{X}$? And what is the output of $R$? The textual description states that the reconstruction is an image $x$ but this equation shows the set $\\mathcal{X}$.\n- How is $C$ determined? The notation implies it's fixed/constant, but the text suggests that it's variable.\n- Tuning the efficiency pressure (lines 132-135): I'm a bit confused by this explanation. How do small values yield fast reconstructions? The smallest value is $\\lambda=1$ and that would give equal weight to all sub-messages. Similarly, this section states that higher values of $\\lambda$ yield longer messages, but based on Eq.1 it seems that for large $\\lambda$ there's a much larger penalty for longer messages.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}