eQMARL: Entangled Quantum Multi-Agent Reinforcement Learning for Distributed Cooperation over Quantum Channels

Authors: Alexander DeRieux, Walid Saad
Primary area: reinforcement learning
Keywords: quantum machine learning, multi-agent reinforcement learning, quantum entanglement

Abstract:
Collaboration is a key challenge in distributed multi-agent reinforcement learning (MARL) environments. Learning frameworks for these decentralized systems must weigh the benefits of explicit player coordination against the communication overhead and computational cost of sharing local observations and environmental data. Quantum computing has sparked a potential synergy between quantum entanglement and cooperation in multi-agent environments, which could enable more efficient distributed collaboration with minimal information sharing. This relationship is largely unexplored, however, as current state-of-the-art quantum MARL (QMARL) implementations rely on classical information sharing rather than entanglement over a quantum channel as a coordination medium. In contrast, in this paper, a novel framework dubbed entangled QMARL (eQMARL) is proposed. The proposed eQMARL is a distributed actor-critic framework that facilitates cooperation over a quantum channel and eliminates local observation sharing via a quantum entangled split critic. Introducing a quantum critic uniquely spread across the agents allows coupling of local observation encoders through entangled input qubits over a quantum channel, which requires no explicit sharing of local observations and reduces classical communication overhead. Further, agent policies are tuned through joint observation-value function estimation via joint quantum measurements, thereby reducing the centralized computational burden. Experimental results show that eQMARL with $\Psi^{+}$ entanglement converges to a cooperative strategy up to $17.8\\%$ faster and with a higher overall score compared to split classical and fully centralized classical and quantum baselines. The results also show that eQMARL achieves this performance with a constant factor of $25$-times fewer centralized parameters compared to the split classical baseline.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729607116940

thread_type:
official_review

<thread_object>
creator: reviewer_xbpk

<content>
summary:
Distributed multi-agent reinforcement learning (MARL) faces challenges in balancing the benefits of player coordination with the communication and computational costs of sharing data between players. Quantum mechanics offers potential for more efficient collaboration through quantum entanglement mediated information sharing, but most current MARL approaches still rely on classical information sharing. This paper introduces a novel framework called entangled QMARL (eQMARL), which eliminates local observation sharing by leveraging quantum entanglement, an input entangled state is prepared and joint measurements are performed. A quantum split critic across agents enables joint observation-value estimation without any additional classical communication overhead to share local environment observations. Experimental results show eQMARL converges faster in some cases and with fewer parameters on a central server compared to classical and quantum baselines. 

The authors perform experiments on CoinGame and CartPole environments and run these as MDP and POMDP processes using different scoring metrics. Several classical and quantum variants of MARL are benchmarked and compared for converge behaviour, stability, robustness, solution quality and space complexity. Finally, an ablation study on network size supplements the main results and provides insight into the scaling of solution quality with the number of network layers.

strengths:
The paper describes both a novel framework and a novel algorithm which clearly outline how you can train decentralised policies over quantum channels with entangled input qubits. All that is required are joint measurements without sharing local observations amongst agents or a central server. The framework and the technical details are well outlined in Sec 2. with a detailed exposition of the entanglement preparation and the different architectural components which are outlined in Fig. 2. This makes it clear to understand its contribution which lies in the incorporation of quantum channels. As outlined, these do not require explicit agent-server observation reporting and this is a novel Ansatz which has not been explored in the literature.

The benchmarks provided are comprehensive and well presented. Especially in Fig.4 for the CoinGame comparison the eQMARL shows superior convergence dynamics and also improved scores and higher own coin rates. The observation that entanglement stabilises the network is also well put and particularly interesting. Another strong point that is clearly outlined is the idea that additional information in the case of POMPDP would supposedly result in superior performance, but actually it suffices to use entangled input states to generate good learning dynamics. This is interesting for a broader audience. 

The work's significance is also clearly outlined. For the benchmarked examples the eQMARL algorithm requires fewer centralised parameters and exhibits more stable and robust learning dynamics than the alternatives. In some cases it also achieves a better solution.

weaknesses:
The results in Fig. 3 are confusing and deserve a clearer exposition. Given the "None" benchmark performs similarly well to all the other Bell states (presumably no entanglement? but this is not clear from the text), how one can understand the role played by the entanglement if it does not significantly outperform "no entanglement"? It would also be interesting to understand why Bell states \Sigma^- and \Sigma^+ perform differently and what role is played by the relative phase?

The cart pole benchmark does not seem to conclusively show an advantage associated with using the eQMARL framework since in both MDP and POMDP cases the authors algorithm does not achieve the maximum reward and also does not seem to clearly outperform qfCDTE (MDP) or qfCDTE and sCDTE (POMDP)even though in the text an advantage - especially in terms of a high mean average reward at early epochs - is claimed to exist. The authors should consider revising the last paragraph and making it less strongly in favour of the performance of the eQMARL algorithm.

There is no mention of the time complexity of the novel framework and particularly the challenge associated with running quantum circuits and preparing entangled input states (notwithstanding the challenge of running this in a non-simulated real quantum device). This makes the benchmarks which all run for the same number or epochs less clear. Particularly in Fig.5b the authors claim the advantage of eQMARL lies in the fact that it achieves a higher reward at a much lower epoch number, but without any information on the per epoch runtime of the different algorithms this case becomes less strong. For example if sCDTE runs 10 times faster than this would not make a strong case for preferring eQMARL over sCTDE. The authors should also consider adressing this in Section 3 when discussing the scaling and performance for different network sizes.

questions:
In the last paragraph the authors claim that "the overall system size is larger, however, the number of central parameters is significantly reduced in the quantum cases – requiring only 1 parameter instead of 25." - can the authors dive into a space time tradeoff here, because no information about algorithm runtimes is given? Even if the eQMARL has fewer centralised parameters, what is its runtime compared to the alternatives? A discussion of this would also be important when comparing the different algorithms in Figs 3-5. 

Does the preparation of the input entangled states introduce some computational overhead, can this be quantified? How does this compare to the benchmarked alternative algorithms?

Why does \Sigma^+ perform better than \Sigma^- in the discussion of Fig.3 ? It is only the relative phase which changes not the general entanglement structure.

Why does the None benchmark in Fig.3 perform almost as well as the \Sigma^+ benchmark, is entanglement even required to achieve any advantage and why can you achieve a high score without entanglement?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730596607096

thread_type:
official_review

<thread_object>
creator: reviewer_h1ng

<content>
summary:
This paper proposes eQMARL, a novel quantum multi-agent reinforcement learning framework that uses quantum entanglement to enable distributed cooperation between agents, demonstrating faster convergence (up to 17.8%) and better performance compared to classical and quantum baselines while requiring 25 times fewer centralized parameters.

strengths:
1. This paper seems to be the first to use quantum channels and entanglement for agent coordination, rather than just replacing classical neural nets with quantum circuits. The split quantum critic design that maintains cooperation without observation sharing seems novel to me.

weaknesses:
1. The approach requires sophisticated quantum hardware for generating, maintaining, and transmitting entangled states across agents - capabilities that current NISQ devices don't reliably support.

2. The method's scalability remains unclear, as it's only demonstrated with simple environments and few agents, and could face significant challenges with quantum decoherence and reduced entanglement fidelity as the system grows.

3. The experiments have too small scale for a purely empirical paper.

4. (Minor) you should specify in the main paper that $\otimes$ is the Kronecker (tensor) product.

questions:
1. How does the approach handle agent failures or dropped quantum connections? The current design seems to assume perfect quantum channels and continuous agent availability.

2. How does the choice of entanglement scheme $(\Psi^+, \Psi^-, \Phi^+, \Phi^-)$ affect the information sharing capacity between agents? Why $\Psi^+$ performs best?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731000942909

thread_type:
official_review

<thread_object>
creator: reviewer_nhf3

<content>
summary:
The paper proposed a novel framework for multiagent RL, based on quantum entanglement. As compared to classical CTDE approaches wherein the communication between the centralized critic and agents happens in the form of sharing observations / states, the proposed framework uses quantum entanglement for information transfer. The framework uses a split critic such that the global value is calculated by transfer of information to a central server via quantum entanglement. The authors present results on two tasks - CoinGame and multi agent cartpole, demonstrating the effectiveness of the proposed framework in sample efficient learning.

strengths:
* The framework proposed is novel and the paper explores the rather unstudied area of using quantum entanglement for information sharing in Multi-Agent RL
* The framework distributes computations across agents, which decreases the centralized computational burden thereby making the framework more scalable
* The proposed framework minimizes the communication overhead between the agents and the centralized critic as well as among the agents via quantum entanglement, as compared to classical information sharing in normal CTDE approaches. 
* The empirical evaluation includes a nice study justifying the choice of entanglement type.

weaknesses:
* The paper is not easily understandable to a reader not completely familiar with the area of quantum entanglement. The appendix provides the necessary background on this. However, a brief introduction of the same (a compressed version) in the main paper might provide some context to the reader in order to better understand the paper.
* Some parts of the experimental results section (specifically, lines 363 - 374 and 412 - 425 Section 3.4 and lines 457-464 in Section 3.5) are rather poorly written. Focusing more on the general conclusions as compared to specific numbers and focusing on one metric at a time (either sample efficiency OR absolute performance)  could help improve the readability of the section.
* Although the proposed framework results in lower variance across seeds, the results on multi-agent cartpole task don't seem very encouraging. The proposed framework fails to beat the baselines in the case of both MDP and POMDP
*  The plots in Figure 3 do not immediately make the conclusions clear. Replacing Figure 3 with Tables H.1 and H.2 could be more compelling.

questions:
* From the description provided in Appendix E.2, the multi-agent cartpole does not seem to benefit much from coordination among different agents, since all the agents operate independently. Given that, is it really a good task for evaluating the framework?
* Is it feasible to run experiments with environments consisting of higher dimensional observation spaces. For eg. running experiments on some moderately challenging MiniGrid environments would be interesting.
* Can the authors provide details on the compute required for performing the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732236216781

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# MiniGrid Experiment

A common theme among reviewers was to provide more varied experiments with complex environments. While we did vary our experiments by running in both the CoinGame and multi-agent CartPole environments, each with both full (MDP) and partial (POMDP) information, we have already run an additional experiment with a multi-agent variant of the MiniGrid environment; thus, showcasing that our approach is still effective across more complex environments. In the MiniGrid environment, the task is to find an optimal grid traversal path from a starting position to a goal using the action set {"turn left", "turn right", "move forward"}. We use a reward shaping schedule of $-1$ for every step taken, $-2$ for standing still, $0$ for not reaching the goal, and $100 \times (1 - 0.9 \times (s / m))$ for reaching the goal, where $s$ is the step count, and $m$ is the max steps. In particular, there are four key factors that make this environment even more complex than our previous baselines. First, agents observe the environment using a limited field of view from their current position and rotational direction; hence, the agents must expend actions to both physically move and visually perceive the environment. Second, because the field of view is limited, the goal position is not always in view; meaning that the agent strategies must also learn to search for the goal position in addition to finding an optimal traversal route. Third, the rotational direction of the agent plays a major role in both grid-world visibility and traversal actions; whereby an agent must learn to optimize the total number of rotation actions (i.e., "turn left" and "turn right") for visual exploration and navigation -- e.g., using a single "turn right" action (fewer steps) instead of three "turn left" actions (increased visibility). Fourth, agent actions are not limited when adjacent to grid-world "wall" positions; meaning that if an agent is facing "forward" to a wall the "move forward" action is still valid (even though clearly not optimal). Critically, the tradeoff between increased visibility at the expense of rotation actions poses complex challenges here, and offers a unique opportunity for implicit observation sharing to improve navigation efficiency.

We trained 2 agents over 1000 epochs with a maximum of 50 steps in isolated $5 \times 5$ MiniGrid environments, where, similar to our other baselines, the only coordination allowed is that afforded by the training protocol -- that is, fCTDE and qfCTDE require full observation sharing, and sCTDE and eQMARL use implicit sharing using split critics. The result of this experiment is shown Fig. R.1 in the accompanying PDF of the supplementary material ZIP archive, filename `global_rebuttal_MiniGrid_experiment_figure.pdf`, and in the table below.

Model | Reward Mean | Reward 95% CI | Number of Trainable Critic Parameters
--- | --- | --- | ---
fCTDE | -63.04 | (-65.16, -61.06) | 29,601
qfCTDE | -85.86 | (-87.03, -84.72) | 3,697
sCTDE | -88.02 |  (-88.69, -87.10) | 29,801
eQMARL-$\Psi^+$ | -13.32 | (-14.68, -11.91) | 3,697

From Fig. R.1 we can see that, for the vast majority of training, the fCTDE, qfCTDE, and sCTDE baselines have an average reward that is clustered near -100; meaning that they exhaust many steps by simply spinning in place (since the maximum step size is 50, and $-2$ is the same-position reward). In contrast, we see that the average reward of eQMARL is spread out higher over the training regime. From the table above we specifically see that eQMARL achieves an average overall reward of $-13.32$, which is 4.5-times higher than the other baselines. Indeed, this negative reward means that eQMARL also expends actions turning in place, but the fact the reward is so close to zero implies these events occur at a vastly reduced frequency than the other baselines.

In addition, we also see that both qfCTDE and eQMARL reduce the overall critic size by a factor of 8 compared to the classical baselines; meaning that they are more computationally efficient.

In testing, eQMARL was able to traverse to the goal in as little as 9 steps, whereas fCTDE required 17 steps, and both qfCTDE and sCTDE were unable to find the goal within the 50 step limit. This is a marked 50\% improvement in the exploration and navigation speed of eQMARL over fCTDE, with the bonus of no observation sharing, and an 8-times smaller overall critic size. Hence, we have shown that eQMARL can indeed be applied to more complex environments, such as grid-world navigation with limited visibility, and provide learning benefits over baselines without the need for observation sharing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732236280855

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# Quantum Hardware and Scalability

We acknowledge with the points made by reviewers regarding the limitations of current NISQ hardware, entanglement generation, and storage. Quantum hardware is evolving and many recent works [a-d], particularly those pertaining to the quantum internet, propose methods for generating and storing entangled qubits. Hence, many of those emerging works can be used to support the type of entanglement required in our system. Also note that many current works on QMARL (Yun et al., 2022a; 2023b; 2022b; Chen, 2023; Park et al., 2023; Kölle et al., 2024) have similar hardware requirements to ours.

The empirical evidence indicating the efficacy of our proposed approach on the 2-player CoinGame in fact demonstrates its feasibility for deployment on near-term quantum hardware. Indeed, eQMARL requires only 4 qubits per agent for the CoinGame-2 environment, whereas the approach in Kölle et al. (2024) requires 6 qubits for the same setting. Further, our framework is aligned with previous QMARL works Yun et al. (2022a; 2023b; 2022b); Chen (2023); Park et al. (2023) which use between 4 to 10 qubits per agent. This relative simplicity indicates its feasibility for deployment on emerging quantum hardware, which range between the order of tens to hundreds of qubits depending on the underlying technology. Examples of current quantum computers with these capabilities include the IonQ Forte with 36 trapped-ion qubits released in 2022, Xanadu Borealis with 216 squeezed-state photonic qubits in 2022, QuEra Aquila with 256 neutral-atom qubits in 2022, and IBM Condor with 1,121 superconducting qubits in 2023. Additionally, the results shown for the multi-agent CartPole experiment, also using 4 qubits per agent, further demonstrate the significance of our approach applied to varied environments. Through the results shown in the manuscript, and those of the additional rebuttal, we have demonstrated that our approach is in fact less demanding than others for CoinGame-2, applicable to both fully observed and information-restricted environments, applicable to varied environment learning tasks, and feasible for deployment on the current generation of quantum computers.

We will put a note regarding how entanglement generation and storage methods are open topics of research in the limitations section of the manuscript. Critically, however, if quantum hardware is considered a mitigating factor in the development of new quantum machine learning algorithms, then all research regarding the subject area, including prior works, will no longer be relevant.

[a] Van Meter, et al. 2022. DOI: 10.1109/qce53715.2022.00055.

[b] Pettit, et al. 2023. DOI: 10.1063/5.0156874.

[c] Lei, et al. 2023. DOI: 10.48550/arXiv.2304.09397.

[d] Azuma, et al. 2023. DOI: 10.1103/revmodphys.95.045006.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732236307395

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# Entanglement Style Performance

Regarding the points made on clarifying the performance of entanglement styles as shown in Fig. 3, we can see $\Psi^{+}$ performs better by looking at the combined performance across both MDP and POMDP environment settings. In Fig. 3.a we see that no entanglement (i.e., None) and $\Phi^{-}$ take longer to converge between epochs 500 and 1500, and they have a much wider variance between experiments as shown in the low minimum value in the same epoch range. Further, $\Psi^{-}$ also struggles with wider variance in Fig 3.a. In Fig. 3.b we see that both $\Phi^{+}$ and $\Phi^{-}$ take longer to converge, have a lower overall score, and a wider variance. Likewise, $\Psi^{-}$ also takes longer to converge. The only entanglement style that is consistent across both MDP and POMDP cases is $\Psi^{+}$, which has the combination of smallest variance and highest score in both settings.

One explanation for why the performance of None is similar to $\Psi^{+}$ in certain cases could be the degradation of fidelity (i.e., entanglement strength) within the agent variational quantum circuits (VQCs). The circular entanglement unitary within an agent's VQC, as defined in Jerbi et al. (2021), binds the behavior of qubits within an agent by creating a "weakly" entangled state (i.e., low fidelity) from the preceding variational layer. The result of introducing additional input entanglement could, in some cases, lower the fidelity of that entangled state further, resulting in poor model performance. It also could increase the fidelity, however, as similar to the process of entanglement distillation. We believe this decrease in fidelity is the reason why states like $\Phi^{+}$ and $\Phi^{-}$ perform poorly in most cases. Based on this, we also believe $\Psi^{+}$ performs better than $\Psi^{-}$ because of the decoherence associated with the difference in phase of the $|11\rangle$ term. Importantly, this relative phase, or more accurately a polarization-dependent phase shift, changes the structure of the entangled state entirely. Indeed, it differentiates $\Psi^{+}$ and $\Psi^{-}$ as two completely different bases in the combined Hilbert space. This phase is affected by the downstream circular entanglement unitaries within each agent, and thus results in entirely different measurement outcomes.

Importantly, this analysis is another contribution of our work that sheds new light on the behavior of entangled input states in VQCs, specifically with application to quantum machine learning, that was not known in prior work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732654261186

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Manuscript Revisions

comment:
We thank all the reviewers for their comments. We have just uploaded a revised manuscript, which includes small enhancements as recommended by the reviewers.

The changes are as follows:

1. Added a brief primer on quantum entanglement to Section 2.1 (lines 139-148).
2. Added remarks, and explicit numbers, regarding run time complexity to Section 3.2 (lines 311-316).
3. Revised the experiment results discussion in Sections 3.3, 3.4, and 3.5 to only report percentages as a single metric along with general conclusions.
4. Revised the discussion regarding the role entanglement fidelity plays on performance in the last paragraph of Section 3.3 (starting on line 354).
5. Revised the related works Section 1.1 to discuss general takeaways in favor of a complete review of related works in Appendix A.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed eQMARL: Entangled Quantum Multi-Agent Reinforcement Learning for Distributed Cooperation over Quantum Channels. During the initial discussion, the reviewers have general concerns of background being too technical, experiments need more clear explanations, and applications in multi-agent RL. The authors carefully addressed these points during rebuttal, both via replies and also via an updated version of the manuscript. In particular, the authors added preliminaries on quantum entanglement, clarified the experimental results, added an experiment on MiniGrid, added discussion on related work, and many others. These changes were well received by reviewers and the scores all became positive after rebuttal.

Therefore, the decision is to accept this paper to ICLR 2025. In any case, the authors should carefully merge all points raised during rebuttal to the final version of the paper.