Communicating Activations Between Language Model Agents

Authors: Vignav Ramesh, Kenneth Li
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: large language models, multiagent communication, embedding representation, multiagent debate

Abstract:
Communication between multiple language model (LM) agents has been shown to scale up the reasoning ability of LMs. While natural language has been the dominant medium for inter-LM communication, it is not obvious this should be the standard: not only does natural language communication incur high inference costs that scale quickly with the number of both agents and messages, but also the decoding process abstracts away too much rich information that could be otherwise accessed from the internal activations. In this work, we propose a simple technique whereby LMs communicate via *activations*; concretely, we pause an LM $B$'s computation at an intermediate layer, combine its current activation with another LM $A$'s intermediate activation via some function $f$, then pass $f$'s output into the next layer of $B$ and continue the forward pass till decoding is complete. This approach scales up LMs on new tasks with *zero* additional parameters and data, and saves a *substantial amount of compute* over natural language communication. We test our method with various functional forms $f$ on two experimental setups—multi-player coordination games and reasoning benchmarks—and find that it achieves up to $27.0$% improvement over natural language communication across datasets with $<$$1/4$ the compute, illustrating the superiority and robustness of activations as an alternative "language" for communication between LMs.

Decision: reject
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729965357763

thread_type:
official_review

<thread_object>
creator: reviewer_gubh

<content>
summary:
The paper explores the idea of communication between language model agents via activations rather than in natural language. Specifically, activations from an intermediate layer of model A are merged into the activations of an intermediate layer (j) of model B and used as inputs for subsequent layers (j+1) of model B.

The paper presents two types of experiments: coordination games (where A and B have the model params) and collaborative reasoning (where A and B are different models with different sizes). In the former, the experiments show that AC (activation communication) is more effective than natural language communication. In the latter, the experiments find AC improving accuracy against both single model baselines and natural language debates (NLD).

strengths:
Originality: Activation-based communication between LMs is a novel direction, to the best of my knowledge.
Quality: The paper shows modest quality vs. computation gains against the baselines.
Clarity: The motivations for activation communication are clearly explained.
Significance: Collaboration between multiple language agents is a potentially important direction.

weaknesses:
[Clarity]
There is some ambiguity in the explanation of the grafting inputs and outputs (see Questions).

[Experiments]
The experiment set-up for the coordination game is not realistic—player A can simply send its text prompt to player B without incurring additional computation, since the models are identical. It would be more realistic if model A is smaller than model B in this case.

An additional baseline for collaborative reasoning is CoT with model B alone. This will illustrate whether the quality gains from A+B comes from additional inference computation.

A useful direction for experiments is to investigate the case of RAG, where the retrieved contents are processed by a small model A (whose activations are potentially pre-computed), while the response is generated by model B.

[Method]
The exploration of the methods seems insufficient. For example, could we use cross attention to incorporate activations from model A to B?

questions:
In sec 3.1 "replace the activation of the last token"replace the activation of the last token (hB,j )tB ∈ R dB ←− f((hA,k)tA ,(hB,j )tB ) for some function f : R dA+dB → R dB ;"

Does this suggest that the inputs to `f` contain only the activations for the last tokens from A and B and the grafting only affects the activation of the last token of model B?

The computation complexity analysis seems to be contradictory: "here F(P, D) = O(P*D) for non-learned f and O(D^2) when f is the mapping matrix". O(D^2) implies that only the last tokens are used, while O(P*D) seems to suggest that the activations of all prefix tokens are used.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730146429134

thread_type:
official_review

<thread_object>
creator: reviewer_fs21

<content>
summary:
This paper proposes a new "model communication" method, which instead of facilitating debate using the output tokens of the models, does so using the intermediate communications of different models.

The paper motivates it's approach using compute and information-bottleneck arguments, particularly in that in communicating via activations, only one of the model's is required to undergo a full-forward pass, and that the information communicated is richer respectively.

The method shows good results, with the approach outperforming the baselines on coordination games across the board, and with either one of its variants (w and w/o a learnt linear mapping) outperforming the conventional debate baseline on 7 reasoning tasks.

Some interesting analysis is also shown to find the layer combinations for communication between the models, which converge to the same optimal numbers (k,j=26) for different tasks.

strengths:
**Novel and Interesting Idea** \
This finding could also be related to recent model-merging literature where sharing activations between models both during training and inference improves performance. Multi-agent debate is an important field of study, and discovering new effective and computationally more efficient methods is useful.

I also appreciate the authors attempts to motivate their methods both with mathematical formulations for compute and with more theoretical arguments in terms of the informativeness of the intermediate activations vs. output tokens.

**Good results against reasonable benchmarks** \
The method did compare with the standard debate setup, and did so for both coordination and reasoning type tasks, and outperformed the baselines almost across all of them.

**Useful analysis on intermediate layer sharing** \
The analysis on which layer combinations are optimal (Figure 2) is interesting, and could inspire potential future lines of work in understanding intermediate model activations and communication.

weaknesses:
1. The approach only works for different model weights. While it is true that the paradigm of debate between different models is interesting, it still is useful to support debate between identical models. One of the arguments of the method is efficiency, but in a practical setting, having  to load different model weights could be impractical.
2. The approach only works for 2 models. One of the key promises of debate is that it can scale model communication to many agents. This approach only showed communication between 2, which only scratches the surface of the larger debate landscape. It would be useful to show how this method compares with traditional debate between more agents, and perhaps attempt to scale the method to such settings.
3. The reasoning results are unclear and inconsistent. While one of the stated highlights of the method is that it is "task-agnostic," in the reasoning benchmark, two different variants of the method (with and w/o learnt linear projection communication bridge) seem to perform inconsistently when compared with each other. For example in some settings, one substantially outperforms the other (HS Psych, AC is much better,  Prof. Law, AC(W) is much better). It would be prudent to analyze and explain why these inconsistencies exist.

questions:
1. For the reasoning tasks, why were the particular tasks chosen? Can you show results on the whole of MMLU? How about MMLU-Pro?
2. Why do you think AC vs. AC(W) works better for some tasks versus other?
3. Could this technique scale to more models favorably?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697856229

thread_type:
official_review

<thread_object>
creator: reviewer_pcow

<content>
summary:
This paper proposes a method for communication between two LMs using intermediate activations from two LMs A and B. Specifically, the paper proposes to take compute partial activations of two LMs upto layers k and j -- followed by a function that merges these two activations in the last token and assign that value to the last token of the second LM. The authors show various kinds of functional forms for activation merging with `replace` being the most effective. Results on synthetic and reasoning datasets show partial improvements over baseline methods although I believe additional experiment details are required to claim the effectiveness of the proposed method.

strengths:
Strengths:
- The paper proposes a method that uses only last token modification between two LMs which is a more efficient communication technique compared to communicating in raw tokens. Section 3.2 of the paper presents a good analysis of the computational efficiency if the paper.

weaknesses:
Weakness:
- This method uses the activation replacement only for the last token activation of the LM B with the last token activation of LM A. Therefore, the information of the entire prompt of the LM A, is compressed in the last token. I am not sure how this will be performing compared to natural language dialog especially in cases where the LM A response might incur a long response (with possibly chain of thought for intermediate answer computation). This is because the last token activation activation replacement cannot capture the nuanced response from CoT traces. This might explain the GSM8k performance where the proposed method does not perform well compared to NLD baseline since it requires intermediate steps for answer compuation.
- The experimental details are lacking to claim the effectiveness of the proposed method. For table 2 results on Countries and Tip Sheet, I could not find the details on how k=j=26 was chosen. Was a separate validation set used for choosing this value? Was the same set used to tune the hyperparameters of baseline methods?
- For learning the mapping parameter W, the authors report that since 3072 sentences were used from C4 dataset, the semantic similarity might not be similar to the evaluation tasks. Does learning W in-distribution help the performance? For example, train W on train set of GSM8k and evaluate it on the test set.

questions:
Please see questions above. Additionally:
- Could you please comment on how this method would perform on reasoning tasks that require intermediate computations via chain of thought? For example, algorithmic tasks in BigBenchHard [1] might require intermediate CoT computations and I am skeptical that a single token replacement at the last token would capture the nuanced information in a complex CoT response.

[1] Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H.W., Chowdhery, A., Le, Q.V., Chi, E.H., Zhou, D. and Wei, J., 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731127678303

thread_type:
official_review

<thread_object>
creator: reviewer_8hgu

<content>
summary:
This paper proposes a novel approach to enable inter-agent communication among language model (LM) agents through direct activation sharing. The authors argue that using activation states for communication, instead of conventional natural language, enhances computational efficiency and performance. Specifically, the method involves grafting activation outputs from one model’s layer into another’s, combining the information through simple operations (sum, mean, replace) or a learned linear mapping. Empirical evaluations on coordination games and reasoning benchmarks demonstrate promising results, with the activation-based method achieving up to 27% higher accuracy than natural language communication while using only a quarter of the computational resources.

strengths:
- Innovation in Communication Mechanisms: The paper presents a creative shift from token-based natural language communication to activation-based sharing between LMs. This direction is original and leverages intermediate model states, which theoretically encode richer representations than single-token outputs.

- Compute Efficiency: By bypassing the resource-heavy natural language decoding process, this method reportedly achieves substantial computational savings. This aspect is well-supported with theoretical compute models and validated experimentally.

- Experimental Validation: The experiments are thorough, covering two main setups—coordination games and reasoning tasks across diverse datasets. The paper also explores different activation fusion functions, providing a nuanced understanding of how these choices impact results.

weaknesses:
- Interpretability and Debugging Concerns: Activation sharing sacrifices transparency and interpretability, particularly since activations do not translate directly into human-readable information. While the authors acknowledge this, their response is inadequate, failing to consider how the complexity of debugging inter-agent communication is compounded when activations, rather than explicit language, are involved.

- Sparse Performance Gains on Some Tasks: The empirical results do not uniformly favor activation-based communication. In certain benchmarks, gains are minimal or inconsistent (as in Table 3, where Natural Language Debate occasionally matches or outperforms activation-based approaches). This variability weakens the paper’s claim that activation-based communication is a universal improvement.

- Selection of Baselines: The paper uses Natural Language Debate (NLD) as the primary baseline. While relevant, it would have been more compelling if additional baselines were included, such as model merging methods or more advanced methods of multi-agent debate. Limiting comparisons may overstate the efficacy of the proposed approach.

- Scalability and Real-world Applicability: The activation-based communication approach seems limited in real-world settings. Many existing LMs, especially proprietary ones, restrict access to activations, limiting the applicability of this approach in settings beyond research or open-source models. This restriction is addressed minimally, without discussing how activation-based communication could be practical for typical API-based LM access.

questions:
- Choice of Activation Fusion Functions: Why were basic functions like sum, mean, and replace chosen over more sophisticated functions that might better capture semantic alignment between models?

- Choice of Layers of Activations to Fuse: What is the effect of the choice of layers to fuse and what is the strategy to choose?

- Behavior with Different Model Architectures: Have you considered whether this approach generalizes effectively across architectures with different layer configurations or attention mechanisms? The focus on models from a single family (LLaMA) may obscure limitations when used with other architectures.

- Application to API-accessible LMs: How does the proposed method scale to real-world environments where only API access is available, such as when interacting with black-box models?

- Missing Reference for LLM-based multi-agent communication: CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (NeurIPS 2023)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732821160885

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate all reviewers’ time and valuable feedback! We have conducted additional experiments and incorporated the rebuttal content into our revision to answer the reviewers’ questions and concerns. Below is a summary of main changes:

| Change                                         | Section      | Related Reviewers          |
| ---------------------------------------------- | ------------ | -------------------------- |
| Modifying activations of all tokens            | Appendix B.1 | Reviewer pCow              |
| Incorporating chain-of-thought prompting       | Appendix B.2 | Reviewers pCow, Gubh       |
| Learning $\mathbf{W}$ in-distribution          | Appendix B.3 | Reviewers pCow, fS21       |
| AC between identical models                    | Appendix B.4 | Reviewer fS21              |
| Additional rounds of NLD                       | Appendix B.5 | Reviewers 8hgU, pCow, fS21 |
| Additional discussion of Lewis signaling games | Section 4.1  | Reviewer Gubh              |

We thank the reviewers for their feedback, and we’re happy to answer any further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733285556138

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Complete MMLU Results [Part 1]

comment:
Thank you all reviewers for your valuable time and feedback. We wanted to share one more important result:

After obtaining more compute, we have measured results on **the entire MMLU benchmark (57 datasets)**:

| Dataset                             | NLD                     | AC                      |
| ----------------------------------- | ----------------------- | ----------------------- |
| Conceptual Physics                  | $60.0 \pm 4.9$          | $\mathbf{68.0 \pm 4.6}$ |
| High School Chemistry               | $\mathbf{50.0 \pm 5.0}$ | $37.0 \pm 4.8$          |
| Security Studies                    | $60.0 \pm 4.9$          | $60.0 \pm 4.9$          |
| Jurisprudence                       | $84.0 \pm 3.6$          | $84.0 \pm 3.6$          |
| Logical Fallacies                   | $63.0 \pm 4.8$          | $\mathbf{72.0 \pm 4.5}$ |
| College Computer Science            | $44.0 \pm 5.0$          | $44.0 \pm 5.0$          |
| International Law                   | $55.0 \pm 5.0$          | $\mathbf{59.0 \pm 4.9}$ |
| Miscellaneous                       | $90.0 \pm 3.0$          | $\mathbf{95.0 \pm 2.2}$ |
| Marketing                           | $70.0 \pm 4.6$          | $\mathbf{85.0 \pm 3.6}$ |
| Elementary Mathematics              | $\mathbf{75.0 \pm 4.3}$ | $58.0 \pm 4.9$          |
| Machine Learning                    | $42.0 \pm 4.9$          | $42.0 \pm 4.9$          |
| High School Macroeconomics          | $44.0 \pm 5.0$          | $\mathbf{75.0 \pm 4.3}$ |
| High School US History              | $45.0 \pm 5.0$          | $\mathbf{71.0 \pm 4.6}$ |
| Human Aging                         | $56.0 \pm 5.0$          | $\mathbf{72.0 \pm 4.5}$ |
| Astronomy                           | $79.0 \pm 4.1$          | $\mathbf{80.0 \pm 4.0}$ |
| Computer Security                   | $56.0 \pm 5.0$          | $\mathbf{75.0 \pm 4.3}$ |
| High School Statistics              | $\mathbf{55.0 \pm 5.0}$ | $42.0 \pm 4.9$          |
| Professional Medicine               | $\mathbf{79.0 \pm 4.1}$ | $65.0 \pm 4.8$          |
| Electrical Engineering              | $58.0 \pm 4.9$          | $\mathbf{60.0 \pm 4.9}$ |
| High School Computer Science        | $63.0 \pm 4.8$          | $\mathbf{70.0 \pm 4.6}$ |
| College Physics                     | $\mathbf{50.0 \pm 5.0}$ | $28.0 \pm 4.5$          |
| Management                          | $74.0 \pm 4.1$          | $\mathbf{75.0 \pm 4.3}$ |
| Moral Scenarios                     | $40.0 \pm 4.9$          | $40.0 \pm 4.9$          |
| World Religions                     | $58.0 \pm 4.9$          | $\mathbf{72.0 \pm 4.5}$ |
| Virology                            | $47.0 \pm 5.0$          | $\mathbf{50.0 \pm 5.0}$ |
| Philosophy                          | $67.0 \pm 4.7$          | $\mathbf{70.0 \pm 4.6}$ |
| Abstract Algebra                    | $\mathbf{50.0 \pm 5.0}$ | $28.0 \pm 4.5$          |
| High School Government and Politics | $\mathbf{80.0 \pm 4.0}$ | $61.0 \pm 4.9$          |
| High School Biology                 | $60.0 \pm 4.9$          | $\mathbf{65.0 \pm 4.8}$ |
| College Mathematics                 | $65.0 \pm 4.8$          | $\mathbf{66.0 \pm 2.4}$ |
| Global Facts                        | $33.0 \pm 5.0$          | $\mathbf{37.0 \pm 4.8}$ |
| High School World History           | $71.0 \pm 4.0$          | $\mathbf{74.0 \pm 4.4}$ |
| High School European History        | $68.0 \pm 4.0$          | $\mathbf{71.0 \pm 4.6}$ |
| College Medicine                    | $\mathbf{65.0 \pm 4.8}$ | $53.0 \pm 5.0$          |
| High School Geography               | $67.0 \pm 4.7$          | $\mathbf{79.0 \pm 4.1}$ |
| Anatomy                             | $74.0 \pm 4.4$          | $74.0 \pm 4.4$          |
| Human Sexuality                     | $75.0 \pm 4.3$          | $75.0 \pm 4.3$          |
| Medical Genetics                    | $79.0 \pm 4.1$          | $\mathbf{82.0 \pm 3.8}$ |
| Professional Accounting             | $40.0 \pm 4.9$          | $\mathbf{48.0 \pm 4.5}$ |
| US Foreign Policy                   | $89.0 \pm 3.1$          | $\mathbf{90.0 \pm 3.1}$ |
| Business Ethics                     | $43.0 \pm 5.0$          | $\mathbf{44.0 \pm 5.0}$ |
| College Chemistry                   | $41.0 \pm 5.0$          | $\mathbf{47.0 \pm 5.0}$ |
| High School Physics                 | $40.0 \pm 5.0$          | $\mathbf{47.0 \pm 5.0}$ |
| Professional Psychology             | $54.0 \pm 4.8$ | $\mathbf{55.0 \pm 5.0}$          |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel method for inter-agent communication using activation sharing rather than natural language. The approach pauses one model’s computation at an intermediate layer, combines its activation with another model’s, and continues the forward pass. As claimed.  this method improves performance over natural language debate, using only 1/4 of the compute, with up to 27% accuracy gains on reasoning tasks, and requires no additional parameters or training data. The authors validate these claims with experiments across coordination games and reasoning benchmarks, including results on all 57 MMLU datasets, where the method outperforms natural language debate on 48.

Reviewers generally found the idea novel and promising, noting its clear motivation and strong empirical support. However, they raised three key concerns: (1) Limited interpretability compared to natural language communication, though the authors argue this represents a necessary tradeoff between interpretability and information preservation, (2) Initial evaluation on only a subset of MMLU datasets, which was later addressed with complete results showing AC outperforms NLD on 48/57 datasets, and (3) Questions about scalability beyond two models, though the authors provide a theoretical framework for extending the approach to multiple models. The reviewers seemed unconvinced after the rebuttal and remained mostly negative about this work. But the AC sees the potential of this work and believes it a valuable contribution to the community after addressing those concerns.