eQMARL: Entangled Quantum Multi-Agent Reinforcement Learning for Distributed Cooperation over Quantum Channels
Authors: Alexander DeRieux
Walid Saad
Venue: ICLR 2025 Poster (2025)
Keywords: quantum machine learning
multi-agent reinforcement learning
quantum entanglement
Primary area: reinforcement learning

Abstract: Collaboration is a key challenge in distributed multi-agent reinforcement learning (MARL) environments. Learning frameworks for these decentralized systems must weigh the benefits of explicit player coordination against the communication overhead and computational cost of sharing local observations and environmental data. Quantum computing has sparked a potential synergy between quantum entanglement and cooperation in multi-agent environments, which could enable more efficient distributed collaboration with minimal information sharing. This relationship is largely unexplored, however, as current state-of-the-art quantum MARL (QMARL) implementations rely on classical information sharing rather than entanglement over a quantum channel as a coordination medium. In contrast, in this paper, a novel framework dubbed entangled QMARL (eQMARL) is proposed. The proposed eQMARL is a distributed actor-critic framework that facilitates cooperation over a quantum channel and eliminates local observation sharing via a quantum entangled split critic. Introducing a quantum critic uniquely spread across the agents allows coupling of local observation encoders through entangled input qubits over a quantum channel, which requires no explicit sharing of local observations and reduces classical communication overhead. Further, agent policies are tuned through joint observation-value function estimation via joint quantum measurements, thereby reducing the centralized computational burden. Experimental results show that eQMARL with $\Psi^{+}$ entanglement converges to a cooperative strategy up to $17.8\\%$ faster and with a higher overall score compared to split classical and fully centralized classical and quantum baselines. The results also show that eQMARL achieves this performance with a constant factor of $25$-times fewer centralized parameters compared to the split classical baseline.

Published as a conference paper at ICLR 2025
EQMARL: ENTANGLED QUANTUM MULTI-AGENT RE- INFORCEMENT LEARNING FOR DISTRIBUTED COOPERATION OVER QUANTUM CHANNELS
Alexander DeRieux
Walid Saad
Bradley Department of Electrical and Computer Engineering, Virginia Tech, USA ddereexx walids) ) @vt. edu
ABSTRACT
Collaboration a key challenge in distributed multi-agent reinforcement learning (MARL) environments. Learning frameworks for these decentralized systems must weigh the benefits of explicit player coordination against the communication overhead and computational cost of sharing local observations and environmen- tal data. Quantum computing has sparked potential synergy between quantum entanglement and cooperation in multi-agen environments, which could enable more efficient distributed collaboration with minimal information sharing. This relationship is largely unexplored, however, as current state-of-the-art quantum MARL (QMARL) implementations rely on classical information sharing rather than entanglement over a quantum channel as a coordination medium. In con- trast, in this paper, novel framework dubbed entangled QMARL (eQMARL) is proposed. The proposed eQMARL is distributed actor-critic framework that facilitates cooperation over quantum channel and eliminates local observation sharing via quantum entangled split critic. Introducing a quantum critic uniquely spread across the agents allows coupling of local observation encoders through entangled input qubits over a quantum channel, which requires no explicit sharing of local observations and reduces classical communication overhead. Further, agent policies are tuned through joint observation-vale quantum measurements, thereby reducing the centralized computational burden. Experimental results show that eQMARL with & entanglement converges to a cooperative strategy up to 17.8% faster and with a higher overall score compared to split classical and fully centralized classical and quantum baselines. The re- sults also show that eQMARL achieves this performance with a constant factor of 25-times fewer centralized parameters compared to the split classical baseline.
function estimation via joint
INTRODUCTION
Quantum reinforcement learning (QRL) is emerging as a relatively new class of quantum machine learning (QML) for decision making. Exploiting the performance and data encoding enhancements of quantum computing, QRL has many promising applications across diverse areas such as finance (Herman al. 2022), healthcare (Flöther 2023). and even wireless networks (Narottama al.) 2023). Its multi-agent variant, quantum multi-agent reinforcement learning (QMARL), is of specific interest because of the potential synergies between decentralized agent cooperation and quantum entanglement. Indeed, in quantum mechanics (Einstein et al. 1971), entanglement is a distinctly quantum property that intrinsically links the behavior of one particle with another regardless of their physical proximity. The use of entanglement in the broader field of QML is a recent notion. Few core works like Mitarai al. (2018) and Du et al. (2020) use entangled layers within variational quantum circuit (VQC) designs to link the behavior of quantum bits (or qubits) within single hybrid quantum model. Even in the recently proposed quantum split learning (QSL) framework (Yun al. 2023a), entanglement is only used locally within each VQC branch of the quantum split neural network (QSNN) model. What has not yet been explored, however, is using entanglement to couple the behavior of multiple QML models. In QMARL, the use of entanglement can be further extended to the implicit coordination amongst agents during training time. Historically, in both purely classical and quantum multi-agent reinforcement learning (MARL). classical communication, shared

---

Published as a conference paper at ICLR 2025
replay buffers, centralized global networks, and fully-observable environment assumptions have all proven to be viable methods for coordinating a group policy (Yun et al. 2022a 2023b [022b] Chen
2023; Park et al., 2023 Kölle et al., 2024). Even QSL, which is not exclusive MARL. relies fully on classical communication between branches of the QSNN (Yun et al. 2023a None of these approaches, however, take advantage of the available quantum channel and quantum entanglement as coupling mediums across decentralized agents or model branches, and opt instead for more classical methods of coordination. In short, entanglement one such phenomenon of quantum mechanics that has not yet been fully explored in the context of cooperation in QMARL settings.
In contrast to prior art, we propose a novel framework dubbed entangled QMARL (eQMARL). The proposed cQMARL is a distributed actor-critic framework. intersecting canonical centralized training with decentralized execution (CTDE) and fully decentralized learning, that facilitates collaboration over a quantum channel using a quantum entangled split critic. Our design uniquely allows agents to coordinate their policies by, for the first time, splitting the quantum critic architecture over a quantum channel and coupling their localized observation encoders using entangled input qubits. This uniquely allows agents to cooperate over a quantum channel, which eliminates the need for observation sharing, and further reduces classical communication overhead. Also, agent policies are tuned via joint observation-vall all qubits in the system, which minimizes the computational burden of a central server. As will be evident from our analysis, eQMARI. will be shown to converge to a cooperative strategy faster, with higher overall score on average, and with fewer centralized parameters compared baselines. All of our source code and experiments are publicly available on GitHub
function estimation using joint quantum measurements across
1.1 RELATED WORKS AND THEIR LIMITATIONS
QMARL a nascent field, with few works applying the quantum advantage to scenarios with multiple agents Yun et al. 2022a 2023b 20226; Chen 2023 Park et al. [2023] Kölle et al., (2024). Further, the application of quantum to split learning (SI.) even newer, with Yun et al. (2023a) being the only prior work. A complete overview of prior works is provided as supplementary material in Appendix A The resounding theme in these prior works is the use of independent agents or branches that communicate and learn through centralized classical means. No prior work, however, makes use of the quantum channel as a medium for system coupling or for multi-agent collaboration. Indeed, the quantum elements serve drop-in replacements for classical neural network (NN) counterparts, and, importantly, the quantum channel between agents and the potential for sharing entangled qubit states go largely under-utilized. Simply put, entanglement and the quantum channel are potentially useful untapped cooperative resources intrinsic to QMARL that have largely unknown benefits.
1.2 CONTRIBUTIONS
The contributions of this work are summarized as follows:
We propose novel eQMARL framework that trains decentralized policies via split quantum critic over a quantum channel with entangled input qubits and joint measurement.
We propose new QMARL algorithm for training distributed agents via optimizing split critic without sharing local enviromment observations amongsl agents or a central server.
We show that the split nature of eQMARL reduces the computational burden of a central quantum server by distributing and tuning parameterized quantum gates across agents in the system, and requiring a small number of parameters for joint measurement.
We empirically demonstrate that eQMARL with + entanglement exhibits a faster conver- gence time that can reach up to 17.8% faster, and with higher overall score, compared to split classical and fully centralized classical and quantum baselines in environments with full and partial information. Further, the results also show that eQMARL achieves this level of performance and cooperation with constant factor of 25-times fewer centralized parameters compared to the split classical baseline.
To the best of our knowledge, this is the first application of QMARL that exploits the quantum channel and entanglement between agents to learn without sharing local observations, while also reducing the classical communication overhead and central computation burden of leading approaches.
httpp::///ittubb.cmmm/eee
2

---

Published as a conference paper at ICLR 2025
Central Server
Joint Quantum
Messurement
Local Actiona
(0)
(a 7
Joint
V(()
Rewards
Agent
01
Agent N
187 (M
Citto Loss
VCC
Crtic
Polly
Actor
Gritc VDC N'
Actor Policy N
Local Rewards
Environment
Agent
(1"0))
Qublts
Emtangloment
Input State
Observations
(o
Figure I: General design of our eQMARL framework. Dashed (solid) arrows represent quantum (classical) communication. A split quantum critic is deployed across the agents via local VQCs (purple) coupled via imput entanglement (orange) at a trusted central server (gray). Joint quantum measurements across all qubits (white) estimate the joint value from the locally-encoded observations.
2 PROPOSED EQMARL FRAMEWORK
Our proposed eQMARL framework is a new approach for training multi-agent actor-critic archi- tectures which lies at the intersection between CTDE and fully decentralized learning. Inspired from CTDE, we deploy decentralized agent policies which learn using a joint value function at training time. The key to our approach, however, is that we use quantum entanglement to deploy the joint value function estimator as critic network which is spread across the agents to operate in mostly decentralized fashion. An overview of our framework design is shown in Fig. and the
design of the system architecture from a purely quantum perspective shown Fig. [ From Fig. [T] the two main elements of cQMARL are central quantum server and set of N decentralized
qucantum agents N = {n] JN The decentralized agents do not communicate with each other; only
communication with the server is necessary during training. During execution, the agents interact with the environment independently and are fully decentralized. During training, our eQMARL framework is divided into core stages: 1) Centralized quantum input state entanglement preparation, 2) Decentralized agent environment observation encoding and variational rotations, and 3) Joint value estimation through joint quantum measurement. Fig. 2 shows how input states are prepared using custom pairwise entanglement operators, followed by agent VQCs, and then joint measurements. Physically, these operations occur different locations. however, it is equivalent to consider these as single quantum system, from input state preparation to final measurement. For purposes of quantum state transmission, we assume an ideal quantum channel environment with no losses.
2.1 JOINT INPUT ENTANGLEMENT
The first stage of eQMARL creates an entangled input state for the split quantum critic network, which couples the critic VQCs spread across the agents. To understand how this works, we first provide brief primer on quantum entanglement. More comprehensive preliminaries are provided as supplementary material in Appendix B Consider two independent agents A and B, with quantum systems described by Hilbert spaces H. and HB, and arbitrary quantum states (w) a and (&b) n- The combined Hilbert space of the two systems can be represented using the Kronecker (i.e., tensor) product HAB = HA @ HB. The combined quantum system is said to be separable if the agent states can be cleanly separated as tensor product of the two systems, i.e., {B) = ( @ (+) B- If, however, the states cannot be represented in this form, then the combined system is said to be inseparable, or entangled. For example, if each agent has one qubit in the {1),, [1} basis, then a separable system could have the state (/separaalle) ) AB = (0) @ [1)A whereas an entangled system could have the state (Wemamgled) AB = (J00) AB + (B)) AB//22 In eQMARL, each agent n E A is (n)
assigned set of D qubits ((n) = (g D chosen based upon the environment state dimension and desired quantum state encoding method. The total number of system qubits is N x D, and is represented by the union of agent qubit sets @ = U. g(n), We couple the agents by preparing an input state that pairwise entangles their qubits using a variation of Bell state entanglement (Nielsen & Chuang 2012) such that
N
CNOT (1.)).6())) Ha(1,a);
if B Φ+, if B 4-
n=
CNOT 8 (1.)).((..4) H8 (8(1,))
E B (114)) ((N..)))
2=
(I)
N
CNOT ((1, (1. H (1,))) x Xs(naa); k= Xockaa))
if B v
2=2
CNOT ((.d)) H (s(
if B 1
3

---

Published as a conference paper at ICLR 2025
Entniglement
Local Osserition Racoding
Lawer
Agess Cota VOC
Ageet Y Orie VOC
32722
Figure 2: Quantum system design with N agents and D qubits per agent. Input entanglement operators (orange) couple the split critic VQCs (purlee, with split point marked in red), which has cascaded layers of variational (blue), circular entanglement (yellow), and encoding (green) operators.
is a coupling operator across qubits {90") In=1 C Q. where B E { TT V M is the set of Bell states, E B is the selected entanglement scheme, and 6(n, d) = (n - 1-) + d is an index mapping within Q for agent E N at qubit index d [1, D), i.e,, ö(n, d) As operator can be applied to entangle any arbitrary set of qubits within the circuit. Note that in this work assume the agents receive their entangled qubits in real-time via a trusted central source, i.e., a central server, however, they could be pre-generated and stored in quantum memory at the agent if desired. The quantum circuits that generate each B B are given in Appendix Fig.
(
(m) Q(n). Importantly, this
2.2 DECENTRALIZED SPLIT CRITIC VQC DESIGN
At its core, our joint quantum critic is split neural network (SNN) (Vepakomma et al., 2018). with each agent's local VQC serving as a branch. After the input qubits are entangled, they are partitioned back into N sets of D qubits, i.e., {Q(") JeE., and transmitted to each agent respec- tively. The agents collect and encode local observations from the environment into their assigned qubits using a VQC. We use the VQC architectures of Jerbi al. (202I): Skolik et al. (2021) for our hybrid quantum network design. Each agent in our QMARL setting uses the same VQC ar- chitecture for their branch of the critic, but tunes their own unique set of parameters. The same architecture is reasonable assumption since all agents are learning in the same environment, and the uniqueness of parameters tailors each branch to local observations. From Fig. 2 the VQC design consists of L cascaded layers of variational, circular entanglement, and encoding operators, with an additional variational layer at the end of the circuit before measurement. The trainable variational layer performs sequential parameterized Pauli X. Y, and Z-axis rotations, and it can be expressed as the unitary operator ( = & P R, ( (n) Be (O(2))))) where g(n) [0, (11))xx is matrix of rotation angle parameters for agent The non- trainable circular entanglement layer binds neighboring qubits within a single agent using the operator Uene CZo - D-2 CZd.d+1). The trainable encoding layer maps a matrix of clas- sical features (n) R Dx3 i.e., agent's environment observation, into quantum state via (n)
the operator: U (A
(n) (n)) 0 P-1 Rs (=( (ml) (
(,d,1 o
R. (o(2) (>,,0 (i)); ( 1,d,0
where (n) R xxX33x is matrix of trainable scaling parameters, and o: R R is an optional squash activation function. The entire VQC can be expressed as a single operator, as follows:
L-1
Ung:(0(m)) (0(m), A (m) (n)) Uvar (o(m))
(
( Ueine U (0 10)))
(2)
I=0
which is parameterized by variational angles (()) and encoding weights ((n)
2.3 CENTRALIZED JOINT MEASUREMENT
The locally encoded qubits for each agent are subsequently forwarded to a central quantum server, which could either be the entanglement source or a different location, for joint measurement. A joint measurement across all qubits in the system is made in the Pauli Z basis using the observable d= Za. The joint value for the locally-encode observations is then estimated as follows:
O
D
V(o) w (0)₂
2
where w E IR a learned scaling parameter, (0)₂ is the expected value of the joint observable w.r.t. an arbitrary system state (&) across all qubits, and = ()(") is vector of joint observations. This rescaling is necessary because the range of the measured observable is (0)p E [11,1] (i.e.,
4

---

Published as a conference paper at ICLR 2025
Algorithm I: Summary of eQMARI training using MAA2C for a quantum entangled split critic. The full algorithm is provided in Appendix D Algorithm D.I
Require: Set of agents N. quantum entanglement source, quantum measurement server
1:
Initialize N critic branches Usy with D qubits and parameters m A (
, Vn E N.
for all episodes
for all sime steps r de BQVART. tra in' ng, notation o, (o. (o(")) m 0111 (ofo) y
(a(m)) n=1"
4:
Central quantum server generates 2N scts of D entangled qubits and sends to agents via quantum channel:
for each agent n. E.N do
Apply Ung (0(m) X("), of")) and Ung (0 (0 ((),, (0)), from locally using assigned entangled input quhits;
Transmit resulting qubits via quantum channel, and reward p(") via classical channel to central quantur server;
end
Perform joini measurements quhits across agents estimate V (o, ] and V(o++1) +)) using
10:
Estimate (or, q.) = Ems r + +V((+++) ) using discount factor "y:
11:
Compute V satii fans and transmit classicul channel each agent to update ('n) locally using partial gradient from
12:
end
proportional to the eigenvalues of the operator O), whereas V(o) E R. The critic loss with respect to the joint value and local agent rewards is then disseminated amongst the agents for tuning of their localized portion of the split critic network and local policy networks.
2.4 SPLIT CRITIC LOSS
The loss of the split critic is derived in way similar Vepakomma et (2018). Since the input entanglement stage of eQMARL has no trainable parameters, it does not exist for the purposes of SL backpropagation. We denote the point of joint quantum measurement as the split point, which is preceded by local agent VQC branches. Each branch can be individually tuned using the partial gradient of the loss at the split point via partial gradient w.r.t. its own local parameters. If we define Sppli the split point, then the partial gradient of each branch's 's parameters can be estimated using the central loss, as follows:
Leritic
eeritic Drispit = -(Vargnn Ceritic
(v Voin Peplit crie
gin) Ceritic
(0(m)
Szsplit (0(1) critic
Central server
Local agent
where V Exgin Centi is the gradient of the loss at the split point, and V (o)) Esplit is the gradient from the split point back to the start of branch E A. The value of V Vzgn Certie sent classically the agents, and since (3) only uses a single trainable parameter, w. the classical communication overhead needed for split backpropagation is minimal. Here, we use the Huber loss for the critic (see Appendix E
of
2.5 COUPLED AGENT LEARNING ALGORITIM
Our eQMARL uses a variation of the multi-agent advantage actor-critic (MAA2C) algorithm Pa poudakis et al., 2021) train local agent policies with split quantum joint critic. Here, summarize the algorithm in Algorithm [ which focuses on the elements for necessary for tuning the critic. In eQMARL, there are N quantum agents that physically separated from each other (no cross-agent communication is assumed) and one central quantum server. Each agent n E W branch in the split critic network. All agents interact with the environment independently and each has its own local data buffer- local observations are neither shared amongst agents nor with the server. The first stage of eQMARL is fundamentally similar to traditional MAA2C. The second stage is where the uniqueness of eQMARL comes into play. The central quantum server prepares 2N sets of D entangled qubits using I for each time step which are then transmitted to the agents via quantum channel. Each agent then encodes their local observations of
employs VQC, given by (2), with unique parameters ((a) and X (m), that serves
(n) using ( and their (n) and 0
assigned entangled input qubits, and transmits the resulting qubits via quantum channel back to the
server. The agents also share their corresponding reward (" via a classical channel with the server, which will be used for downstream loss calculations. Access to the reward is necessary for the critic evaluate agent policy performance. This is reasonable assumption in eQMARL the reward value contains no localized environment information, and is also used in Yun et al. 22022a) 2023b); Park al. (2023); regardless, the classical channel will also be used transmit partial gradients of the critic loss. The server then performs a joint measurement on all the qubits associated with of
5

---

Published as a conference paper at ICLR 2025
and 0,+1 obtain estimates for V(o+) and ((0++1) using 33. Subsequently, the server computes the expected cumulative reward Q(o,, a-) for the joint observations and actions at 7 using V( discount factor y, and the respective rewards. The joint critic loss Ceriti is then computed, its partial gradient w.r.t. the split point V "su Certic is estimated, and then sent via classical channel to each agent to update their local weights @ using (4.
(n)
3
EXPERIMENTS AND DEMONSTRATIONS
3.1 ENVIRONMENTS
We use the CoinGame environment first proposed in Lerer & Peysakhovich (2018). and as imple- mented in Phan et al. 2022). which has been widely used (Foerster et al. 2018b] Phan et al. 2022
Kölle et a 2024), multi-agent variant of the canonical CartPole environment (Barto al. 1983), and multi- aagent variant of the MiniCrid environment (Chevalier- -Bisvver et (0033) as benchmarks for MARL scenarios. In particular, CoinGame" nature as a zero-sum game and the independent natures of both multi-agent CartPole and MiniGrt serve as intriguing case studies for learning cooperative strategies using full, i.c., described by a Markov decision process (MDP), and partial, i.e., described by a partially observable MDP (POMDP), information. In CoinGame, we evaluate agents using the Score metric, which aggregates all agent undiscounted rewards over a single episode. In both CartPole and Mini Grid d we evaluate agents using the total reward metric, which aggregates the number of time steps an agent maintains pole balance and the reward received for maze navigation over a single episode respectively, See Appendix F for environment details.
3.2 EXPERIMENT SETUP
We compare eMMARL against three basclines that are considered the current state-of-thee offthhe-rt con- figurations in cctor-critic CTDE: 1) Fully centralized CTDE (ICTDE), classical configuratio where the critic is simple fully-connected NN located at a central server, like in Gupta et al. (2017); Foerster al. (20 8a), and requires agents to transmit their local observations to the server via classical channel; 2) Split CTDE (sCTDE), a classical configuration where the critic is a branching NN encoder spread across the agents, which is combined using a centralized NN based on Rashid et al. (2018) located at a central server, and requires agents to transmit intermediate NN activations via a classical channel; and 3) Quantum fully centralized CTDE (gFCTDE). quantum variant of FCTDE where the critic is located at a central server, as in Yun et al. (2022a) 2023b); Park et al. (2023). and agents transmit their local observations via a classical channel. These baselines were carefully chosen to convey how quantum entangled split critic eliminates local environment observation sharing, while reducing classical communication overhead by leveraging the quantum channel, and minimizing centralized computational complexity. In our experiments simplify the setup by using policy sharing across the agents, as done in Yun et al. (2023b) and Chen (2023). p. All classical models were built using tensorflow the quantum models using tensorf for quantum simulations. For Coi CoinGame, all models were trained for 3000 epochs, with T = 50 steps, 7 0.99. For CartPole, e, all models were trained for 1000 epochs, with maximum of 500 steps per episode. For Mi n Gr d, all models were trained for 1000 epochs, with a maximum of T = 50 steps per episode. All models use the Adam optimizer with varying learning rates. The quantum models use D = 4 qubits, L = 5 layers, and 6 = arct an activation. The classical models use 12 hidden units for CoinGame and Cart and h 100 for MiniGrid. See Appendices to for further details. We conduct all experiments on high-performant
and eirg
computing cluster with 128 CPU cores and 256 GB of memory per node. The training time of SCTDE for CoinGame MDP is R 5.5 minutes, and for POMDP is R 7.5 minutes. In contrast, the training time of cQMARL is se 3.5 hours and x 8.5 hours for MDP and POMDP respectively: this is in line with many current QMARL works (Yun et al.) 2022a; 2023b 2022b; Chen 2023 Park et al. [003) Kölle ct al. 2024), and is indicative of the known computational complexities of running quantum simulations classical hardware.
3.3 COMPARING QUANTUM INPUT ENTANGL STYLES
The first set of experiments demonstrate the effectiveness of various input entanglement styles used in eQMARL approach. We run two separate experiments using the Coi SainnGam---e 2 environment using both MDP and POMDP dynamics. The score metric results for both dynamics are shown in Fig. 3

---

Published as a conference paper at ICLR 2025
23
19
30
2008
2000
3003
2820
3000
1500
3000
Spoch
Epoch
(a) Score- MDP
(b) Score POMDP
Figure 3: Comparison of Co score performance with (a) MDP, and (b) POMDP dynamies for eQMARL using - (blue), V (orange), o+ (green), Φ- (red), and None (cyan) entanglement averaged over 10 runs of 3,000 epochs, with +1 std. dev. shown as shaded regions. These figures generally show that + outperforms other entanglement styles across both dynamics.
We consider score threshold markers of 20 and 25 to aid our discussion. the MDP setting of Fig. 3a we see see that - entanglement converges 4.5% faster to a score threshold of 20 compared to the next closest Tr Similarly, + converges 5.2% faster to a score of 25 compared to the next closest VT At the end of training, all peak scores hover slightly above 25, Looking at the shaded standard deviation regions, we get a sense for the stability of each entanglement style. Specifically, we see that v+ T and + have similar tight ranges until epoch I 1500, whereas both 4- and None have far lower minimum values until around epoch 1300. Moreover, V appears to have large downward spikes toward the end of training. Fig. 3a shows that there is gap in convergence between t and None, and the other styles. Looking closer, we observe that c plateaus at earlier epochs, and vi- is more unstable (dropping in score) later epochs. Hence, we see a clear advantage for applying wrt In the POMDP setting of Fig. 3b we see that vi converges 2% faster to a score of 20 compared to the next closest None. Interestingly, there is much larger gap in convergence between the others. A score of 25 is achieved 10.7% faster by T compared to None, whereas both s+ and +- never reach this threshold. The final peak scores for T and None hover slightly above 26. The shaded standard deviation regions exhibit cascade effect between the styles, and, in particular, we observe that & has the lowest min, followed by o which has a slightly higher floor. These groupings are interesting as both + and 4 are similar in composition, only differing by a phase. Hence, we again see a clear convergence and score advantage for using ttr.
and
Comparing the performance of the entanglement styles with both dynamics paints picture of the generalizability of the system as a whole. Interestingly, the worse performance of o+ and $ suggests that stme-state entanglement, (00) and [11), regardless of phase, results in less coupling of agents compared to opposite-statee state entanglement, |01) and |10). One explanation for why the performance of None is similar + in certain cases could be the degradation of fidelity (i.e., entanglement strength) within the agent VQCs. The circular entanglement unitary within an agent's VQC, defined as Ueine from Section 2.2, binds the behavior of qubits within an agent by creating "weakly" entangled state (i.e., low fidelity) from the preceding Uvar Introducing additional input entanglement could, in some cases, lower the fidelity of that entangled state further, resulting poor model performance. also could increase the fidelity, however, as similar to the process of entanglement distillation. We believe this decrease in fidelity the reason why states like + and t perform poorly in most cases. Based on this, we also believe Vi+ performs better than V because of the decoherence associated with the difference in phase of the |10) term. Importantly, this polarization-depeneer phase shift changes the structure of the entangled state entirely. Indeed, this phase is affected by the downstream Ueine and thus results in entirely different measurement outcomes. The consistently high performance of ! + across both dynamics suggests that it enhances the generalizability of the system, and, since input entanglement does not increase classical computational overhead, we see that t entanglent can be used couple the agents in both dynamies while achieving comparable or higher performance. Thus, we select + as the entanglement scheme to be used in all subsequent experiments.
3.4 COINGAME EXPERIMENTS
We next compare the performance of eQMARI--V++ with baselines FCTDE, SCTDE, and CCCTDE using the CoinGame-2 environment with MDP and POMDP state dynamics, as shown in Figs. 4a and 4b Looking at the MDP score metric in Fig. 4a we see that eQMARL- + converges 16, 2% faster to score threshold of 20 than the next-closest qFCTDE, and 10.8% faster to score threshold
7

---

Published as a conference paper at ICLR 2025
UE
FRE
st mE
ICTRC
4/c
4PPAL
33-
23
15-
13-
20
2000
2000
3003
3000 2800
-7-
1:000
Spoch
(a) Score- MDP
(b) Score POMDP
Figure 4: Comparison of CoinGame score performance with (a) MDP, and (b) POMDP dynamies for FCTDE (orange), qÉCTDE (magenta), sCTDE (green), and eQMARL + (blue) averaged over 10 runs of 3,000 epochs, with =1 std. dev. shown as shaded regions. These figures generally show that eQMARL outperforms baselines across both environment dynamics.
25 25 compared to SCTDE. Overall, we observe a 1.4% increase in max score for eQMARI-V Vr+ compared to the next highest SCTDE. Additionally, eQMARI, V/+ is smoother than qfCTDE at later epochs; suggesting that input entanglement stabilizes convergence. Fig. 4b shows that eQMARI-V V++ converges 24% faster to score of 20 considering the noticeable gap between it and qECTDE. This demonstrates that the branching quantum network with input entanglement shortens convergence time compared to the fully centralized variant. Fig. 4b also shows that all models achieve score of 25, however, in this case, eOMARL-V A converges 17.8% faster and with slightly higher score than GETTDE. Examining smoothness, the MDP case. The difference in performance between the baselines in Figs. 4a and 4b demonstrates clear quantum advantage for learning in the presence of partial information. Specifically, the faster convergence to the peak score threshold in e eQMARL-V- -rr shows that splitting the quantum critic across the quantum channel with entangled input qubits allows the agents to learn a more cooperative strategy without direct access to local observations. This is interesting because qECTDE centralized and has all local observations at its disposal. This additional information would initially suggest better performance compared to approaches with only local information. However, from Fig. 4b we observe a clear benefit for not only splitting the quantum critic as branches across the agents, but also introducing an entangled input state that couples their encoding behavior. Hence, from Figs. andand 4bb we conclude that our proposed eQMARI-V + configuration learns to play the game significantly faster than the classical variants without sharing local observations, transmitting intermediate activations, nor tuning large NNs at the central server. The higher performance and shorter convergence time, compared both the quantum and classical baselines, shows that splitting the critic amongst the agents results in no apparent loss in capability. In fact, the smoothness of the curves suggests that the input entanglement stabilizes the network over time.
see much greater fluctuation between all curves compared to
3.5 CARTPOLE EXPERIMENTS
The next set of experiments compare the performance of eQMARL with baselines using multi-agent variant of the CartPole environment with both MDP and POMDP state dynamics. The average reward metric across all agents in the environment for both dynamics is shown in Fig. []] We use reward thresholds of the mean and max values to draw comparisons. From Fig. S we see that the classical models do not perform well overall in either setting, and qÉCTDE experiences high variance the MDP case. For MDP, qECTDE achieves the highest mean and max rewards overall, but with an extremely high standard deviation. In contrast, eQMARL-V + reaches its mean and max rewards 12.2% and 31.5% faster than qECTDE respectively. For POMDP, we see that SCTDE achieves the highest overall max reward at the end of training, but with very low mean. In contrast, both GFCTDE and - achieve a similar max value significantly earlier, about two-thirds of the way through training. e eQMARI- + achieves the highest mean reward overall very early in training, which is 9.1% faster than qFCTDE with similarly low standard deviation to the MDP setting. The key observation from the CartPole experiments that e @MARL- + more rapidly learns a strategy with higher average reward than the classical variants in both MDP and POMDP settings. Further, eQMARL-V slightly outperforms QECTDE in the POMDP setting with a higher average reward, and is much more stable with a significantly lower variance in the MDP setting all achieved via implicit collaboration though entanglement. These results show that, without observation sharing,
8

---

Published as a conference paper at ICLR 2025
K0NAL
ICIDE
qICRE
220
Aecoag Roward
200
200
Avotage
100
200
400 003
603
1093
200
500
500 1000
Spech
Spoch
(a) Average Reward MDP
(b) Average Reward POMDP
Figure 5: Comparison of CartFole MDP and POMDP environment average reward performance for fCTDE (orange), FCCTDE (magenta), SCTDE (green), and eQMARI-VV v (blue) averaged over runs of 1,000 epochs, with +1 std. dev. shown as shaded regions. These figures generally show that eQMARL outperforms classical baselines and is more stable than qf qTCTDE across both dynamics.
-4
-40
200
400
600 300
1003
Eaedh
Figure 6: Comparison of MiMMinnGrra Grid POMDP environment reward performance for ECTDE (orange), ạECTDE (magenta), SCTDE (green), and eQMARL- + (blue) over 1000 epochs. This figure shows that eQMARL outperforms baselines by learning direct goal navigation instead of spinning in place.
eQMARL- + can learn a similarly performant, and dramatically more stable, strategy compared to a fully centralized quantum approach that has access to all agent observations.
3.6 MINTGRTD EXPERIMENT
The next experiment compares the performance of eQMARL with baselines using a multi-agent variant of the MiniGrio environment with POMDP state dynamics, in which agents have a limited field of view. The average reward metric across all agents in the environment is shown in Fig. 6 From Fig. we can see that, for the vast majority of training, the ECTDE, qECTDE, and SCTDE baselines have an average reward that is clustered near - 100; meaning that they exhaust many steps by simply spinning in place (since the maximum step size 50, and -2 the same-pos positiono reward). In contrast, we see that the average reward of eQMARL- Ni+ is spread out higher over the training regime with a mean of about 13, which is nearly 4.5-times higher than the other baselines. Indeed, this negative reward means that eOMARL-V also expends actions turning in place, but the fact the reward is so close to zero implics these events occur at a vastly reduced frequency than the baselines. In testing, eQMARI.VV+ was able to traverse to the goal in little steps, whereas ECTDE. required 17 steps, and both cFCTDE and sCTDE were unable to find the goal within the 50 step limit. This is marked 50% improvement in the exploration and navigation speed of eQMARL- t over FCTDE, with the bonus of no observation sharing. Hence, we have shown that eQMARL- +++ can indeed be applied more complex environments, such as grid-world navigation with limited visibility, and provides learning benefits over baselines all without the need for observation sharing.
3.7 ABLATION STUDY
The last set of experiments we consider an ablation study to examine the relationship between NN layer depth and performance, and to facilitate fair model size comparisons. In particular, we trained ICTDE and SCTDE with hidden layer units E {3, 6, 12, 24), and qECTDE and eQMARL with VQC layers L E {2, 5, 10} on the CoinGame- -2 environment with MDP and POMDP state dynamics for 10 experiments
3000 epochs each. An excerpt of the score metrics results for eQMARL and sCTDE in the MDP setting each framework is shown in Table [ The full results are provided in Appendix [.5] In Fig, 7 we
shown in Fig. [ and comparison of the critic model sizes used in
that eQMARI.-V++ with I. 5 achieves mean score 3-times higher than I. = 2, and nearly

---

Published as a conference paper at ICLR 2025
SCIEE3
40902L 4V-42
10AAL-
5-
1000
1000
2000
2000
5000
Epoch
Epoch
(a) SCUD
(b) eQMARI- V
Figure 7: Ablation study score performance on MDP CoinGame-2 2 for (a) SCTDE, and (b) cQMARL-V+ with hidden layer units h E {3, 6, 12, 24} and VQC layers L E 22,5,001 aver- aged over 10 runs of 3000 epochs, with 11 std. dev. shown as shaded regions. These figures generally
show that selecting parameters h 12 and L 5 results in optimal performance.
Table I: Comparison of the best critic model size in number of trainable parameters for each framework used the CoinGame- environment with MDP and POMDP dynamics.
Framewark
Ablation Selection
Number of critic parameters: MDP
Number of critic parameters: POMDP
eCMARL
L 5
(132 per agent, contral)
817 (408 per agent, contral)
gECTDE
L 5
265
817
SCIDE
12
889
673
SCTDE
h
913 (444 per agent, 25 central)
697 (336 per agent, central)
identical to I, 10. This trend is similar for CFCTDE. Both SCTDE and ECTDE also exhibit similar behavior for hidden units; that is the performance of h = 12 is nearly 2-times that of h = 6, and only marginally less than h 24. Considering the significant performance drops and increased variation incurred by reducing, and the limited gains by increasing, both h and L, the selection of h 12 and L= 5 results the most comparable performance across all baselines. Looking at Table the critic sizes reported split the critic network across the agents the total size of the agent-specific network a fraction of the total size. For MDP dynamics, we observe that the quantum models are 4 times smaller than the classical variants. For POMDP, we observe that the quantum models are slightly larger than their classical counterparts. This is because, in POMDP, the quantum models require classical NN for dimensionality reduction at the input of each encoder. While the overall system size is larger, however, the number of central parameters is significantly reduced in the quantum cases requiring only 1 parameter instead of 25. This is important for scaleability because the baselines implement a full NN at the central server and its size scales with N. In contrast, eQMARL has only a single trainable parameter tied to the measurement observable, which will remain fixed regardless of N.
for the entire system. This distinction is important since both eOMARL and s sCTDE
4 CONCLUSION
In this paper we have proposed eQMARL, a novel quantum actor-critic framework for training decentralized policies using a split quantum critic with entangled input qubits and joint Spreading the critic across the agents via a quantum channel climinates sharing local observations, minimizes the classical communication overhead from sending model parameters or intermediate NN activations, and reduces the centralized classical computational burden through optimization of single quantum measurement observable parameter. We have shown that VP input entanglement improves agent cooperation and system generalizability across both MDP and POMDP environments. For MDP, we have shown that eQMARL converges to cooperative strategy 10.8% faster and with higher score compared to SCTDE. Likewise, for POMDP, we have shown that eQMerL.covverges to cooperative strategy 17.89 17.8% faster and with slightly higher score compared to qICTDE. Further, we have also shown that cQMARL outperforms classical basclines and exhibits more stable performance than qfCTDE in independent environments. Lastly, we have shown that eQMARL requires 25-times fewer centralized parameters compared to SCTDE. One limitation of this work is the computational complexity of quantum simulations on classical hardware, which is an ongoing topic of research for noisy intermediate-scale quantum (NISQ) systems of many qubits. Indeed, many recent works on
joint measurement.
QMARL (Yun et al.] [222aa] 2023b 2022b Chen 2023 Park et al. 2023 Kölle et al. 2024) have similar hardware requirements to ours. Further, many recent works quantum networks (Van Meter
jet al. 2022, Pettit et al., 2023) Lei et al. 2023 Azuma et al.) (023) propose methods for generating
and storing entangled qubits, which support the type of entanglement required in our system.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Koji Azuma, Sophia E. Economou, David Elkouss, Paul Hilaire, Liang Jiang, Hoi-Kwong Lo, and Ilan Tzitrin. Quantum repeaters: From quantum networks to the quantum internet. Reviews of Modern Physies, 95(4):045006, December 2023. ISSN 0034-6861, 0756. doi: 10.1 MMMMoodhyy..5..050000 URL httph:tttt/inni aps. wttp:kaa......oooooo///oo http:///ooo./....o///ooo/oooo 95. 045006
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions Systems, Man, and Cybernetics,
SMC-13(5):334 833--866, September 1983. ISSN 2168-2909, doi: 10.1 11019//SMC..88..313377777
URL httttt:ss//teeeeeeeeeetteeeeeeee
leee wwtp://ggg.go//////////// 6313077
Max Born. Quantenmechanik der stoBvorgänge. Zeitschrift für Physik, 38(11):803 827, November 1926.
Samuel Yen-Chi Chen. Asynchronous training of quantum reinforcement learning, January 2023.
URL http:://axx...rg// wtt.g//gga/////.aa/ 05096
Maxime Cheyalier- Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, tos/2306.138311 2023.
Yuxuan Du, Min-Hsiu Hsieh, Tongliang Liu, and Dacheng Tao. The Expressive Power of Parameter- ized Quantum Circuits. Physical Review Research, 2(3):033125, July 2020. ISSN 2643-1564. doi: 10. (00/PhyyReewesearc..220k http:///........aaa/a////aaa/a/. /
11922
A. Einstein, M. Born, and H. Born. The Born-Einster Letters: Correspondence between Albert Einstein and Max and Hedwig Born from 1916- 1955, with Commentaries by Max Born. Macmil-
lan, 1971. ISBN 978-0-333-1126742 URL ht htttps://bbbottt www.sk...oooooooooo.oooooo.o .tw.gge..co/ocooo......ocooc cww.books ?id=
HvZAAQAAIAAJ
Frederik F. Flöther. The state of quantum computing applications in health and medicine. Research Directions: Quantum Technologies, pp. 1-21, July 2023. ISSN 2752-9444. doi: 10.1017/qut.0233.4
URL http://arxiv.oort
2b31 09106)
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.
Counterfactual multi-agent policy gradients. 32(1), 2018a. ISSN 2374-3468, 2159-5399. doi: 10. 1609/aaai 11221.11194
URL hattss://jss..... aag.................eeeeeeeeaee. phppppppppaa. ptpphh///aatit......p
wriew/1199)
Jakob N. Foerster, Richard Y. Chen, Manuan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Leamming with Opponent-Leariinn Awareness, September 2018b. URL http: 04326
Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control us- ing deep reinforcement learning. In Gita Sukthankar and Juan A. Rodriguez-Aguilar (eds.), Autonomous Agents and Multiagent Systems. volume 10642, pp. 66 83. Springer International Publishing, 2017. ISBN 9783319716817 978331977922 doi: 10. 10.11077977-33311--7166 http::/hhha...pi............////
Dylan Herman, Cody Googin, Xiaoyuan Liu, Alexey Galda, Ilya Safro, Yue Sun, Marco Pistoia, and Yuri Alexeev. Survey of Quantum Computing for Finance, June 2022. URL http:
7/arxiv. //rxiv.org/abs:
02773
Sofiene Jerbi, Casper Gyurik, Simon C. Marshall, Hans J. Briegel, and Vedran Dunjko. Parametrized quantum policies for reinforcement learning. March 2021. doi: 10:48.500aXx1.2000055577.. URI
/itt:://arxis or.. abw/aaasb100 05577
Michael Kölle, Felix Topp, Thomy Phan. Philipp Altmann, Jonas Nüßlein, and Claudia Linnhoff- Popien. Multi- Agent Quantum Reinforcement Learning using Evolutionary Optimization, January 2024. URL http:t w/tp://aavvvv/// wrg. /2311 05546
11

---

Published as a conference paper at ICLR 2025
Yisheng Lei, Faezeh Kimiaee Asadi, Tian Zhong. Alexander Kuzmich, Christoph Simon, and
Mahdi Hosseini. Quantum Optical Memory for Entanglement Distribution, 2023. URL https
//arxiv. wwtp:/ogggaaaaa.o.o.oo
09397
in Adam Lerer and Alexander Peysakhovich. Maintaining comper social dilemmas using 01068
deep reinforcement learning, March 2018.
Fabrizio Logiurato and Augusto Smerzi. Born Rule and Noncontextual Probability. Journal of Modern Physics, 03(1 I): 1802-1812, 2012. ISSN 2153- 196. doi: 10.4236//mp.2012.311
Lluís Masanes, Thomas D. Galley, and Markus P. Müller. The measurement postulates of quantum mechamics are operationally redundant, Nature Communicationn, 10(1), March 2019. doi: 1038/ s41467-019-09348-x
http:://do...gg/////// 100 1038 2F s4146 -19-7-001199---0
Kosuke Mitarai, Makoto Negoro, Masahiro Kitagawa, and Keisuke Fujii. Quantum Circuit Learning. Physical Review A, 98(3):032309, September 2018. ISSN 2469-9926, 2469-9934. doi: 10.1103/ 00745
PhysRevA.98.. 03.2309.. URL http:://axii.ctet
Bhaskara Narottama, Zina Mohamed, and Sonia Aīssa. Quantum machine learning for next-G wireless communications: Fundamentals and the path ahead. IEEE Open Journal of the Communications Society, 4:2204-2224, 2023, doi: 10.1 202.3399......33992....
Michael A. Nielsen and Isaac L. Chuang. Quantum Computation and Quantum Information: 10th
Anniversary Edition. Cambridge University Press, 1 edition. June 2012. ISBN 978- 17781-107-0021-3
978-0-511-97666-7. doi: 10.1017/CBB99788511166666 URL https :/ www cwmbriiabeeee.. idge
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, November 2021.
URL httthttp//aaaat org/ http://bs/2000.0000 07869
Soohyun Park, Jae Pyoung Kim, Chanyoung Park, Soyi Jung, and Joongheon Kim. Quantum Multi-Agent Reinforcement Learning for Autonomous Mobility Cooperation, August 2023. URL
wrw.orgggaa...... abtp//bs///20 2308 01519
Robert M. Pettit, Farhang Hadad Farshi, Sean E. Sullivan, Álvaro Véliz-Osorio, and Manish Ku- scalable quantum internet using rare-earth ions. Applied Physics Reviews, 10(3):031300 307, September 2023. ISSN 1931-9401. doi: 10.1063/5.015683
mar Singh. A perspective the pathway
URL http:://wwwb..aa.....rrrrrrrrrrs
on the o
to sca
Thomy Phan, Felix Sommer, Philipp Altmann, Fabian Ritz, Lenz Belzner, and Claudia Linnhoff- Popien. Emergent cooperation from mutual acknowledgment exchange. In Proceedings of the s International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, pp- 1047-1055, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978- 777-1-4503-9211--1
Tabish Rashid, Mikayel Samvelyan. Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, June 2018. URL htt.tttpp//axxii org/ abs 1803. 11033.11888 11485
Andrea Skolik, Sofiene Jerbi, and Vedran Dunjko. Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning March 2021. doi: 11.22331/4-2222-05-01 720. URL
http::/arxix: or.. abs / 2103. 15084
Rodney Van Meter, Ryosuke Satoh, Naphan Benchasattabuse, Kentaro Teramoto, Takaaki Matsuo, Michal Hajdusek, Takahiko Satoh, Shota Nagayama, and Shigeya Suzuki. A Quantum Internet Architecture. In 2022 IEEE International Conference on Quantum Computing and Engineering (QCE), pp. 341-352, Broomfield, CO, USA, September 2022. IEEE. ISBN 978 17778-1-6555 1-66549- 113-9.1111661------ doi: 10. 10.1109900E53715.. 2.02..005.. URL
Leee org/ www.:e/ww....ddcccuoeeeo.e document///////o
9951 2587
12

---

Published as a conference paper at ICLR 2025
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. December 2018. URL
/ttt:://arxi...o oww./vi..rrreoo abs 1812.1111 00564
Won Joon Yun, Yunseok Kwak, Jae Pyoung Kim, Hyunhee Cho, Soyi Jung, Jihong Park, and Joongheon Kim. Quantum Multi-Agent Reinforcement Learning via Variational Quantum Circuit Design, March 2022a. URL http:://ttttiiie owg/ 2203. 10443
Won Joon Yun, Jihong Park, and Joongheon Kim. Quantum Multi-Agent Meta Reinforcement Learning. August 2022b. URL
11510
Won Joon Yun, Hankyul Back, and Joongheon Kim. Quantum Split Neural Network Learning using Cross-Channel Pooling, April 2023a. URL hwtp::ttttii...tt/////// org/ absaaaass/222/ 06524
Won Joon Yun, Jae Pyoung Kim. Soyi Jung, Jae-Hyun Kim, and Joongheon Kim. Quantum Multi-Agent Actor-Critic Neural Networks for Internet-Connected Multi-Robot Coordination in Smart Factory Management. January 2023b. doi: 10.485550/rXiiv23301 104012. URI. https
04012
A
RELATED WORKS
QMARL is a nascent field, with few works applying the quantum advantage to scenarios with multiple agents Yun et al. 2022a| 2023b 2022b Chen 2023 Park et al. 2023 Kölle et al. (024). Further, the application of quantum to SL is even newer, with Yun et al. (2023a) being the only prior work. In Yun et al. (2022a) and Yun et al. 22033), the authors propose a novel approach to QMARL that integrates CTDE and quantum state encoding of environment states. Specifically, they deploy actor-critic QRL with VQCs as the core for both actor and critic architectures. The QMARL agents each deploy a local quantum actor network and learn collectively in CTDE fashion using centralized critic and unified experience replay buffer. In particular, the localized actors each operate independently from the others in the network and yet critically must also share their environment state experience with the collective via the shared replay buffer.
In Yun et al. (2022b), the authors propose the quantum meta MARL (QM2ARL) framework which uses central meta Q Q-learning agent to train other local agents. In particular they define angle and pole training, a technique which tunes classical parameters to describe the Bloch sphere orientation of encoded quantum information. The first angle parameter training stage describes mapped Q-values as regions on the Bloch sphere, which are used to create the centralized meta agent. This meta agent is then subsequently used train local agents by orienting pole parameters toward their specific environments.
In Chen 2023), the author proposes quantum asynchronous advantage actor-critic (QA3C) framework for training decentralized QRL agents, which leverages a global shared memory and agent-specific memories used in conjunction to train parallel agents. Akin to classical federated learning (FL), QA3C deploys an actor-critic network within each decentralized agent. The gradients of these local networks periodically sent the global shared memory for aggregation into global model. The global model parameters are then broadcast back to the agents to update their local models.
The work in Park et al. (2023) proposes a QMARL approach for autonomous mobility cooperation using actor-critic networks with CTDE in NISQ environments with shared replay buffer. In particular, they consider deployment with NISQ era limitations, the most notable being a low number of qubits. They show that CTDE-based QMARL can be deployed on near-term quantum hardware to coordinate robotic agents in smart factory environments. Critically, the coordination within their proposed system stems from the centralized critic network and shared replay buffer that has become synonymous with CTDE frameworks.
In Kölle et al. (2024), the authors propose a QMARL approach using evolutionary optimization with VQC design based on quantum classification networks and agent policies implemented as independent VQC models with shared local information. Variations of the parameters for these policies are trained in populations to evaluate and compare performance. The authors assume fully observable environments in both cooperative and competitive settings, which is influenced via changes in the per-agent reward.
13

---

Published as a conference paper at ICLR 2025
Finally, in Yun et al. (2023a), the authors propose method for applying split learning QML for traditional machine learning (ML) classification tasks. They deploy device-local quantum neural networks (QNNs) to predict local labels and features which are transmitted classically to a central server and transformed into true set of labels and localized gradients for each device.
The resounding theme in Yun et al. 2022a 2023b 2022b); Chen (2023) Park et al. (2023) Kölle
eet al. (2024): Yun et al. (2023a) is the use of independent agents or branches that communicate and
learn through centralized classical means. No prior work, however, makes use of the quantum channel as a medium for system coupling or for multi-agent collaboration. Indeed, the quantum elements serve as drop-in replacements for classical NN counterparts, and, importantly, the quantum channel between agents and the potential for sharing entangled qubit states go largely under-utilized. Simply put, entanglement and the quantum channel are potentially useful untapped cooperative resources intrinsic to QMARI that have largely unknown benefits.
B
MORE COMPREHENSIVE PRELIMINARIES
B.1 QUANTUM MUTLI-AGEN REINFORCEMENT LEARNING
We consider a reinforcement learning (RL) setting with multiple agents in environments with both full and partial information. The dynamics of a system with full information is described by a Markov game with an underlying MDP with tuple MMDP = (N, A. P. R) where A is set of N agents, S = (S(m) nEv is the set of joint states across all agents, A = (_4(x)) JneN is the set joint actions, ((x) and (m) are the set of states and actions for agent n, P(s 4z) is the state transition probability, and R(s;, a²) = {r."" (m) Jnex E R. is the joint reward Vs, E S, ag E A. The dynamics of a system with partial information is described by Markov game with an underlying POMDP with tuple MрOмDR S, P, R, S2, O) where S. S, P are the same as in MMDP. however the full state of the environment 8y at time is kept hidden from the players. Instead, at time & the agents receive local observation from the set of joint observations Q {[(()) JNEN, where ((1) is the set of observations for agent with transition probability ((o az). Vo, Ω, which is dependent on the hidden environment state after taking a joint action. We treat MMDP as a special
case of MFOMDP where oj (n) st"), that is the observations represent the full environment state
information. Hereinafter, all notations will use o(²) in place of the local environment state for brevity to encompass all cases.
QMARL the application of quantum computing to MARL. popular approach in MARL is through actor-critic architectures, which tune policies, called actors, via an estimator for how good or bad the policy is at any given state of an environment, called a critic. To do this, the critic needs access to the local agent environment observations to estimate the value for a particular environment state. Current state of the art approaches follow the CTDE framework which deploys the critic on central server and the actors across decentralized agents. Because the critic and the agents are physically separated, CTDE requires the agents to transmit their local observations to the server for the critic to estimate the joint value, thereby publicizing potentially private local observations. Quantum is often integrated as a drop-in replacement for classical NNs, called VQCs, within many MARL systems. These trainable quantum circuits tune the state of qubits, the quantum analog of classical bits, using unitary gate operations.
B.2
QUANTUM COMPUTATION
B.2.1 QUBIT STATES
A qubit is the quantum mechanical analog to the classical bit. The state of a qubit is represented as a 2- dimensional unit vector in complex Hilbert space H C The computational basis is the set of states
{10) = [1 oj² .11) = 10 11") which forms a complete and orthonormal basis in H. (meaning (0 = -110) = 0 and (0j0)- = (1)1) = 1). All qubit states can be expressed as a linear combination
of any complete and orthonormal basis, such the computational basis, which is called superposition.
We adopt Dirac notation to describe arbitrary qubit states (v5) = =Ia 3/" = a (0) + 11) E H (called
"ket psi"" where |a|2 + 2 1. their conjugate transpose (1/) = |16)² [a= _ = a² (0) - (1)
(called "bra psi"), the inner product (wi(v>) = a 02 + 38 82, and the outer product ( =
14

---

Published as a conference paper at ICLR 2025
[1a3 0 Quantum systems with qubits can also be represented by extending the above [300 3
notation using the Kronecker (tensor) product where H X 2 Ha. (C²)&D is the complex space of the system state (1b) U=n
D-1 (aa for |1/ E HD. States that can be represented as
either single ket vector, sum of basis states are called pure states. For example, [0), [1), and
v2 (10) + 1)) are all pure states in H.
B.2.2 QUANTUM GATES
A quantum gale is an unitary operator (or matrix) U, such that UUT = I, where I is the identity matrix, acting on the space H which maps between qubit states. Here, we use the single-qubit Pauli gates
0
1 0
0 -i 7 =
1 0
x=
Y
(B.1)
i
0
0 -1
with their parameterized rotations
Rx (0)
Ry R0)
-18 R2(0) = =1
(B.2)
where 8 E [0, 2T]. the Hadamard gate
1 1
-1]
1
H
(B.3)
v2
the 2-qubit controlled-X (CX, also called CNOT) gate
T
0
CX = CNOT =
(B.4)
0
X
and the controlled-2 Z (CZ) gate
[ 0
C2
(B.5)
0
which are both controlled by qubit and target qubit 2, where is 2 x 2 square matrix of zeros.
B.2.3 ENTANGLEMENT
Consider two arbitrary quantum systems A and B, represented by Hilbert spaces HA and H.g We can represent the Hilbert space of the combined system using the tensor product HA @ Hg. If the quantum states of the two systems are (t)) A and [b5) p. then the state of the combined system can be represented as 11), @ ((b) Quantum states that can be cleanly represented in this form, i.e., separated by tensor product, are said be separable. Not all quantum states, however, are separable. For example, if we fix a set of basis states {10) A (1) } E HA and Bl0)B.LI. E HB. then general state in the space of H. @ HB can be represented as (R) Ene(o.i) Ebe(o.)) Ca (a), @ (b) B. which is separable there exists Ca.e = caco, Va, b E {0, 1}. producing the isolated states |1) = Ede co (a) A and (+)) B = (o c 16)/B g. If, however, there exists one Ca,c # Ca then the combined state is inseparable. In such cases, if a state is inseparable, it is said to be entangled.
The four Bell states {1 )
AB Φ )AB
M (
ABB form a complete basis for two-
qubit systems HA @ HB. and have the form:
Φ +
(10) A (0) B + 11) (1) B)
(B.6)
AB
V2
>AAB
(10) (0) 11) 11)
(B.7)
V2
w+) AB =
V2 (10) [1) + 11) 10)
(B.8)
1 - AB V2 (10) [1) 11) 10)
(B.9)
Since is impossible to separate the states of into individual systems H. and H.. the four Bell states entangled. In particular, the Bell states pure entangled states of the combined system H.A @ H.. but cannot be separated into pure states of systems H. and H Additionally, the four Bell states can be generated by quantum circuits using combination of Pauli operators with
15

---

Published as a conference paper at ICLR 2025
constant x (0) imput state follows:
Φ ++)
1, (H @ 1) (0) n
(B.10)
AB CNOT (I/ 8 (X @ (0) (0)
(B.11)
Vr+ AB 1, (H @1) (1@ X) [0) [0)
(B.12)
AB = 1, (H @ 8) (X @ X) (0), ()
(B.13)
B.2.4 PROJECTIVE MEASUREMENTS AND OBSERVABLES
projective measurement Hermitian and unitary operator O such that O Of and 00 called an observable. The outcomes of measurement are defined by an observable's spectral decomposition
M-1
Am Pm
(B.14)
m==
where .M- 2" represents the number of measurement outcomes for n qubits, and m is a specific measurement outcome in terms of eigenvalues Am and orthogonal projectors P.. in the respective eigenspace. According to the Born rule [Bor] (926)): Logiurato & Smerzi (2012); Masanes al. (2019), the outcome of measuring an arbitrary state (if)) will be one of the eigenvalues Xm, and the state will be projected using the operator P.. / (m)) with probability:
p(m)= = (vb) Pm. (ı) (Pm)p
(B.15)
The expected value of the observable with respect to the arbitrary state (eb) is given by:
M
[+ =
Amp(m) = =00)
(B.16)
B.2.5 COMMUTING OBSERVABLES
set of observables (O1....,0p, share a common eigenbasis (i.e., a common set of eigenvectors with unique eigenvalues) if
[O, 0,1 = 0,0, =0 Vije []
(B.17)
ie., their pair-wise commutator zero. In such cases the observables in the set are said to be pair-wise commutting, which in practice means that all observables in the set can be measured at the same time.
C JOINT INPUT ENTANGLEMENT CIRCUITS
We use a variation of Bell state entanglement to couple the input qubits of the agent VQC branches.
Specifically, entangle based the set of four Bell states E E { -- +++ }, outlined in (B.13) using the circuits as shown in Fig. C. The circuits in Fig. C.I generaaeee a quantum state across 1) qubits, which has the combined Hilbert space H@D In each of the entangled operators, select one qubit, Q1, to serve as the master control, and all others, 92....,9D serve as targets. The control qubit functions identically operations that normally apply to the second qubit, in an H. X H system, to all qubits in HDD-1 The resulting state is an entangled pure state in H
canonical Bell state entanglement. Here, we extend the gate
D FULL ALGORITHM
The following algorithm is an expanded version of Algorithm 1 shown in Algorithm D.I In Algorithm [.I] we include all operations necessary for training both the agents and the split critic. In eQMARL. there is set of N quantum agents A that are physically separated from each other (no cross-agent communication is assumed) and one central quantum server. Each agent n E N employs an actor policy network T Toina
(a)o(")
(m) (which can either classical or quantum in nature)
with parameters 0 ((n) and a VQC given by (2) with unique parameters ((n) and X (x) that serves as one branch in the split critic network. In our experiments we simplify the setup by using policy
sharing across the agents, as done in Yun al. (0023b) and Chen (023):; in other words, x ((k) Vn, h E N. All agents interact with the environment independently and each has its own local data buffer ((n) populated with local observations, actions, rewards, and next observations
16

---

Published as a conference paper at ICLR 2025
ENT $
91
Q1
H
100D)
Q2
(0000) + (ooel
7
QD
(a) Ø
ENT
Q2
D (10D)) - -11
QD
(b)
ENT S
16
H
1000 A
X
(.
QD
(c) V
ENT
g1. AD
(1))11111) (1] (oooo---) (.red
10ep)
42
vD
QD
(d) W
Figure C.I: Diagrams of joint entanglement operators based on the the four Bell states (a) Φ+, (b) (c) + (d) VV-
aa() represented by the 4-tuple (a()),{"")) o for any instant in time These local data
not shared amongst agents, with only the reward and action being communicated classically to the central quantum server (the action only being necessary for policy sharing). Since we employ policy sharing, the final step in eQMARL with this in place is to also estimate the advantage value A(or, a,) = Q(o, a,) - (o,,) using the existing value and expected reward estimates, compute actor loss Cactor compute the gradient of the loss w.r.t the shared actor parameters V V Cactors and
17

---

Published as a conference paper at ICLR 2025
update Pactor- Note that here Huber loss for the critic, and for the actors entropy- regularized advantage loss. The loss functions are described in detail in Appendix
Algorithm D.I: Full eQMARL training using MAA2C for a quantum entangled split critic.
Require: Set of N agents N, quantum entanglement source, and quantum measurement server Initialize agent actor networks with shared parameters Ouce and local replay buffer D) O. Vn E W/
1:
2: Initialize N critic branches Usg with D qubits and parameters ((m) erttc* X (1), An E N/;
for episode=1, MaxEpisode do Localized environment interaction.
f. 0:
5: done = False;
while done * True and i mar steps do for each agent E E do
Get local observation (n) from environment; Compute (alo,)) and sample a."),
Apply action a," and get reward (x)) next observation 0:41:
10:
Update local replay buffer D D(')) u (o a.a)) (x()) .0+11
12:
If 044 terminal then communicate done True;
13:
end
f +
15:
end
b eQMARI, framework for training.
for 2 do
17:
Central quantum server generates 2/V sets of entangled qubits and transmits agents via quantum channel (could be prepared priori and stored in quantum memory); for each agent v. E N do
18:
19:
Apply Uvge (m) (n) (m)) and Uvg (e enice
(m)
from locally using assigned
entangled input qubíts;
20:
Transmit via quantum channel the qubits after applying Usge to central quantum server:
Transmit via classical channel the reward y:?) and action dr at the current time step to central quantum server (only reward if policy sharing is not used);
21:
22:
end
> Using notation o, = (ol"))= ( 0, (o (o%%))
)==))
==11 a. a. (a (n)
==1
23:
Perform joint measurements on qubits across all agents to estimate V(o,) and V(o++1) using
Estimate Q(or, ar) = > rt + (V(+++1) using discount factor ?:
24:
25:
Estimate A(or, ar) Q(o,; a.) (or) for policy sharing;
Compute Vor. Curror and update Outer for policy sharing;
Compute V Ceritic and transmit via classical channel each agent update ((m) locally using
partial gradient from (4
28:
end
end
E LOSS FUNCTIONS
All of our actors and critics are trained using the same loss functions for each experiment, For the critics, we train using Huber loss
Cerie -
(V - -((,), )22)
Vo.) Q(or, a
5 8, (E.1)
8|V(o,) -io,- 102 otherwise,
where å controls point in which the loss function turns from quadratic to linear. this work we use 1. For the actors, deploy policy sharing amongst the agents. As such, all agents us the same policy parameters, and thus the loss must aggregate the individual losses of each agent. We train using the entropy-regulariized advantage function
N T-1
Lactor
Σ
A(o!.) aaf))) log, p(a.") + +aHl (a!
(E.2)
n(T
7=0
18

---

Published as a conference paper at ICLR 2025
where
[(a -p(a,") loge p(a!")),
(E.3)
is the entropy of selecting an action, a controls the influence of entropy, n E is the agent index, and p(a?") ( is the probability of chosen action at time step
F
ENVIRONMENT SPECIFICATIONS
F.1 COINGAME
(0, taae
leat
Seet
fest
feat
(b) MDP observation dynamics for blue player.
2
coin
feat.
Spot
(a) Diagram of CoinGame- -2 2 environment.
(c) POMDP observation dynamics for blue player.
Figure F.1: Example of (a) CoinGame-2 environment with two players, colored red and blue, and single coin colored blue, with visualization observation matrix for the blue player with (b) MDP dynamics, and (c) POMDP dynamics. Grid squares in (b) and (c) colored grey denote 0 value, and colored blue/red squares denote a value.
In eQMARL, we train decentralized agents using the CoinGame environment first proposed in Lerer & Peysakhovich (2018), and as implemented in Phan et al. (2022). The CoinGame- environment pits two agents of different colors (red and blue) on a x tile grid to collect coins with corresponding color. An example of CoinGame is shown in Fig. F.la Agent observations are sparse matrix (n) E (0. xxx33x3 Vn E N with 4 features each with a 3 x 3 grid world as shown in Fig. F.Ib. The features specifically are: 1) grid with a indicating the agent's location, 2) A grid with 1 to indicate other agent locations, 3) A grid with 1 for the location of coins that match the current agent's ' color, and 4) A grid with a 1 for all other coins (different colors). Since these observations include all information about the game world, the game is considered fully observable and is described by an MDP. We also experiment with a partially observable variant this game which removes the second feature from agent observations (i.e., the location of other agents), which is space matrix (!n) E {0, 3x33x3 Vn E N. In this partially observed setting, the game is described by a POMDP since agents cannot see each other and thus the full state of the game board is unknown. An example of this observation space is shown in Fig. Fic The agents
move along the grid by taking actions in the space (n)) [north, south, east, west) Vn E N. Each time an agent collects coin of their corresponding color their episode reward is increased by +1, whereas diffcrent color reduces their episode reward by 22 The goal for all agents is to maximize their discounted episode reward. The details the environment are summarized in Table F.
We evaluate agents using three metrics: score, total coins collected, and own coin rate. The score metric aggregates all agent undiscounted rewards over a single episode
S Σ T-1
Σ A
(F.1)
where n E A is the agent index, t E [0, - 1] is the episode time index, the episode time limit,
and (n) is the undiscounted agent reward time t. The total coins collected metric gives insight
into how active the agents were during the game
TC ΣΣ
(F.2
where (n) the total number of coins collected by agent n at time t. Finally, the own coin rate metric gives insight into how the agents achieve cooperation, specifically by being selective which
19

---

Published as a conference paper at ICLR 2025
Table F.1: Specifications for Co nGame-2 environment with MDP and POMDP dynamics.
Parameter
Value
MDP: of
{0, 13 13xx3x3 (dimension is 36) POMDP: of E {0, 133x3x3 (dimension is 27)
Observation for agent n. at time t
Number of players (N) Time limit (T)
2
50
Action for agent n at time t
a!")
E (north, south, east, west)
+1,
if agent collects coin of same color,
Reward for agent at time
r
(n)
2,
if agent collects coin of different color,
0,
otherwise
Discount factor (1)
0.99 0.001
Entropy coefficient (ax)
coins they procure
N T-1 Σ
OCR =
k
/cl"))
(F.3
==1
t=
where k(()) is the number of coins collected the corresponding agent's color.
F.2 CARTPOLE
Agant 1
Agent
Agont N
Figure F.2: Example of N-agent Cart Pole environment colored red, blue, and green.
We also train eQMARL using multi-agent variant of the CartPole environment as proposed in Barto et al. (983). The multi-agent CartPole environment runs multiple independent instances of the single-agent variant in parallel. This environment setup is interesting case study for multi-agent learning because the observations of each agent are completely independent from one another; that is, observations from sibling environments are not strictly necessary develop a strategy for specific environment instance. This allows us to examine the impacts of both explicit and implicit cooperation between independent agents.
example of CartPole with N agents shown in Fig. F22 Agent observations are matrix of (n) R4x1 Vn E N with real-valued features. The features are: 1) Cart position with range [-4.8, 4.8), 2) Cart velocity with range (-oc, oc). 3) Pole angle in radians with range [-0.418, 0.418), and 4) Pole angular velocity with range (-30, 00). The pole is considered balanced if the pole angle feature stays within the range [-12095, .2055] radians, and the cart position feature stays within the range [-2.4, 2.4). These observations include all information about the environment, and thus the environment under these conditions is considered fully observable and described by an MDP. We also consider a partially observed variant of the environment which removes the second feature from agent observations (i.e., the cart velocity), which matrix (n) E [3x1, Vn E W. The environment is described by POMDP in this setting since agents are unaware of their cart's velocity, and thus the full state of the environment is unknown. Notably, in this multi-agent variant of the environment the agent observations in both settings independent from each other. The agents interact with the environment by taking actions the space Anm) {left, right), Vn N, which correspond to pushing their cart to the left and right respectively. Similar to the observations, the agent actions are also independent and do not affect neighboring environments. Each time step an agent successful keeping their pole balanced they receive a +1 episode reward. The episode terminates when observation falls outside of the balanced range. The goal for all agents to maximize their expected total episode reward (i.e., the number of time steps they able to keep the pole balanced). The details of the environment summarized in Table F2
20

---

Published as a conference paper at ICLR 2025
Table F.2: Specifications for multi-agent CartPole environment with MDP and POMDP dynamics.
Parameter
Value
MDP: of") R
Observation for agent n at time t
POMDP: of"h g3x1
Number of players (N) Time limit (T)
2
500
Action for agent n at time t
a!") E (left, right)
(m) +1, if pole for agent n. is balanced,
Reward for agent at time
otherwise
We evaluate the agents using the average reward metric, which aggregates all agent rewards single episode
AR ΣΣΣ1
(n)
(F.4)
==1 t=
where n E N is the agent index, t E |0, - 1] is the episode time index, T is the episode time limit, and rg (n) is the agent reward at time
F.3 MINIGRID
Agent
Agent
Agent
Figure F:3: Example of an N -agent Mini Grid environment colored red, blue, and orange.
also train eQMARL using a multi-agent variant of the MiniGrid environment as proposed in Chevalier- Boisvert et al. (2023). The multi-agent MiniGr Gric d environment runs multiple instances of the single-agent version in parallel. This environment configuration an interesting case study for multi-agent learning because, similar to multi-agent CartPole, the local agent observations are independent of the others, and sharing observations is not necessary to solve the environment. This can, however, show how agent policies
affected with the seemingly added benefit of either directly shared or muantum-coupley local environment observations. An example of MiniGra Griid with N agents is shown Fig. F3 and the details of the environment are summarized in Table F3
In the MiniGrid environment, the task is to find an optimal grid traversal path from a starting position to a goal using the action set (n)) = (turn left, turn right, move forward]. vn E .V. Agent observations are a matrix (n) E Z7x7x3 Vn E ., where the agent has a 7 7 limited field of view of the maze grid, and each cell in the grid is encoded the 3-tuple (object, color, state), where object E [0, 10] identifies the object at the cell (e.g., empty, etc)), color [0, 5) identifies the color of the cell (e.g., red, green, etc.), and state [0, identifies the state of the cell (e.g., open, closed and locked). We consider a x 5 maze grid size our experiments. Notably, because agent observations are limited field of view, and the full maze grid, we regard this environment as being described by POMDP dynamics. We use reward shaping schedule of -1 for every step taken, -2 for standing still, 0 for not reaching the goal, and 100 x (1 0.9 x ((t (t++1) 1)/T) for reaching the goal, where f E [0, T 1] is episode the time index, and T is the episode time limit. Additionally, we provide the reward bonus 1/ve for actions that explore less visited grid positions and action pairs, where is the count for specific grid position and action pair. In particular, there are four key factors that make this environment even more complex than the CoinGame and CartPole baselines. First, agents observe the environment using a limited field of view from their current position and rotational direction; hence, the agents must expend actions both physically move and visually perceive the environment. Second, because the
21

---

Published as a conference paper at ICLR 2025
Table F.3: Specifications for multi-agent Mi MiniGrid envirommen with POMDP dynamics.
Parameter
Value
Grid size
W 5, h5
Observation for agent n at time / Number of players (N)
(n)
E 2
2
Time limit (7)
50
Grid position for agent time
9: (n)
10 w x
Grid direction for agent at time
(m) jup,down refttright,
Action for agent n at time
a fturn Left,turn right, forward] if g. (n) = -
-2
Reward for agent at time
r 1 +
100(1 019* if goal is reached, otherwise
vc
-1,
where c = # (9) (n)
a (m)) E (9) TM de
==0
field of view is limited, the goal position is not always view; meaning that the agent strategies must also learn to search for the goal position in addition to finding an optimal traversal route. Third, the rotational direction of the agent plays a major role both grid-world visibility and traversal actions; whereby agent must learn to optimize the total number of rotation actions (i.e., turn left and turn r rith) for visual exploration and navigation e.g., using single turn right action (fewer steps) instead of three turn left actions (increased visibility). Fourth, agent actions not limited when adjacent to grid-world wall positions; meaning that if an agent is facing forward to wa walll 1, then the move forward d action is still valid (even though clearly not optimal). Critically, the trade-off between increased visibility at the expense of rotation actions poses complex challenges here, and offers unique opportunity for implicit observation sharing to improve navigation efficiency.
G QUANTUM ENCODING TRANSFORMATIONS
To encode environment observations into our quantum models we first apply transformation on the observation matrix. This allows us reduce its dimensions, thereby making usable for the limited number of qubits available to NISQ systems, while also changing the range of matrix values to be suitable for input into one of the Pauli rotation gates.
G.1 COINGAME-2 ENVIRONMENT
MDP dynamics For the Co CoinCame-2 environment with fully observed state dynamics we use the transformation
[MDP(Oixjxk) Σ 01xj,22
(G.1)
which sums over the last dimension of the observation matrix Oxxjxk with shape i ix. x k. In the of Coinnn 2oinGamm-2 with MDP dynamics the observations have shape 4 x 3 3. This transformation reduces the dimensions to 4 x 1, which can be directly fed into the encoder architecture outlined
in Section 2.2
POMDP dynamics For the CoinGaam- 2 environment with partially observed state dynamics quantum models employ a small classical NN at the input of the encoder for dimensionality reduction, as done in Chen (203) In particular, use the transformation
FroMDp(Oxxxxa) = Wijkx3d flatten flatten(o.xj (oixjx*)² +
(G.2)
which flattens the observation matrix Oxxfx with shape x j x l and passes it through a single fully-connected NN layer with parameters Wijkx3d and b, and d is the number of qubits. Note that, in POMDP, the trainable quantum encoding parameters X (m) are no longer necessary. this case,
we set (m) 1, where is a matrix of ones.
22

---

Published as a conference paper at ICLR 2025
G.2 CARTPOLE ENVIRONMENT
Observation scaling For the Cart Pole environment we apply a constant observation scaling to both MDP and POMDP dynamics to normalize their values. In particular, we use the transformation
f(osx)) 0ixs/Vs
(G.3)
where
2.2,
(G.4)
is a constant scaling vector.
POMDP dynamics For the CartPole environment with partially observed state dynamics apply an additional transformation similar to the Co CoinGame-a POMDP case to reduce imput feature dimensions. In particular, apply the transformation,
frOMDP(Oxx)) = Wijx3d flatten(oxx)) +
(G.5)
which flattens the observation matrix Ozxj with shape i x j and passes it through a single fully- connected NN layer with parameters Wiyx3d and and is the number of qubits.
G.3 MINIGRID ENVIRONMENT
For the MiniGrid environment we apply similar transformation to the GGame and CartPole POMDP environments, which identical to (G.2). Specifically, we apply fully- connected NN layer to reduce the dimensionality from the observation shape 7 7 x 3 to 3d, where
the field of view the agent as described in Table F3 and the number of qubits.
H MODEL HYPERPARAMETESS
The hyperparameters for each of the models trained in experiments, as discussed in Section 3.2 are shown in Tables H.1 and H.2 Table H.I show the model parameters used in qICTDE and eQMARL. Table H.2 show the model parameters used in fCTDE and sCTDE.
Table H.I: Hyperparameters for qFCTDE and eQMARL, actor and critie, used all environments.
Envirenment
Model
Parameter
Value
NN encoder transform activation
POMDP: linear
Flg (n) as trainable
MDP: True
POMIP: False
Cointame-2,
Actor,
CartPole
Critic
Number of qubits per agenn(()
(eQMARL only) Inpur cntangloment type (B) for critic
+
Number layers (L)) Usee
Squash activation (()
aretam
Inverse temperature (1)) Optimizer
Adam
Learning rate
[0.01, 0.1,0.11
NN encoder transform activation
lincar
Flag A (*) as trainable
False
Number qubits per agent (DD
(eQMARL only) Input entanglement type (B) critic
++
Critic
Number of layers (J.) in Ueg
MiniGrid
Squash activation (()
Inverse temperature (8) Optimizer
Learning rate
(0.001, 0.001, 0.01, 0.1]
Optimizer Learning rate
Adam
Actor
[0.0001]
Number hidden
100
Table H.2: Hyperparameters for ICTDE and sCTDE, actor and critic, used on all environments.
Environment
Model
Parameter
Value
NN hidden units (h)
[12]
CoinGame- 2, CaztFole
Actor, Critie
Optimizer
Learning rate
0.001
Actor,
NN hidden units (h)
[100]
MiniGrid
Optimizer
Adam
Critic
Learning rate
23

---

Published as a conference paper at ICLR 2025
I EXPERIMENT RESULTS
1.1 ENTANGLEMENT STYLE COMPARISON
The empirical results for the entanglement comparison experiment, discussed in Section 3.3
shown in Tables I and I.2 Table shows the score metric statistics mean, standard deviation, and 95% confidence interval each of the entanglement styles V ++ v v. Φ ② and None. Table [.2 shows the convergence time, in epochs, to each of the score thresholds 20, 25, and also to the maximum score value (reported parenthetically in italics) for each of the entanglement styles. The best values in each column are highlighted in bold.
Fig- I.T shows the training results for the entanglement styles as discussed in Section 3.3 In particular, we provide the full set of performance metrics of score, total coins collected, own coins collected, and own coin rate, as outlined in Appendix F (E. Fig. [.] shows results the entanglement styles V V Φ+, Φ
and None, discussed Section 3.3 The left column, Figs. I. I.lc I.Ie and shows the performance for MDP environment dynamics. Similarly, the right column, and 1.1h shows the performance for POMDP environment dynamics.
Table I.1: Comparison of entanglement style score performance for MDP and POMDP CoinGame-2 2 environment dynamics using mean, standard deviation, and 95% confidence interval statistics. Best values are highlighted in bold.
Score
Dynamics
Entanglement
Mean
SD
95% CI
21.11
2.65
(20.92, 21.29)
V
20.85
2.70
(20.61, 21.07)
MDP
Φ+
21.02
2.54
(20.77, 21.30)
20.43
3.85
(20.20, 20.59)
None
20.00
3.80
(19.75, 20.20)
+++
18.49
3.91
(18.15, 18.74)
V
17.77
4.05
(17.40, 18,09)
POMDP
++
17.01
6.28
(16.74, 17.25)
14.73
7.63
(14.45, 15.01)
None
18.57
4.26
(18.28, 18.82)
Table I.2: Comparison of entanglement style score convergence (in number of epochs) for MDP and POMDP CoinGame- 2 environment dynamics. Best values are highlighted in bold.
Epochs to Score Threshold
Dynamics
Entanglement
20
25
Max (value)
+
568
2332
2942 (25.67)
595
1987
2849 (25.45)
MDP
++
612
1883
2851 (25.51)
691
2378
2984 (25.23)
None
839
2337
2495 (25.12)
+
1049
1745
2950 (26.28)
1206
2114
2999 (25.95)
POMDP
++
1269
2992 (24.1)
1838
2727 (22.8)
None
1069
1955
2841 (26.39)
24

---

Published as a conference paper at ICLR 2025
95
30
30-
20
25
20
20
10
2000
2000
3000
-5
500
1000
1500
2000
25:00
3000
900
1000
1500
2000
2500
3000
Epoch
Epoch
(a) Score MDP
(b) Score POMDP
36
25
26
Caledea
Coleated
20
20
16-
16
10-
10-
2000
2800
3000
2000
3000
500
1000
1500
2000
2500
8000
500
1000
1600
2000
2500
3000
Epoch
Epoch
(c) Total coins collected MDP
(d) Total coins collected POMDP
35
30-
Callacted
25
Collected
26
20-
20
16
16-
10-
2600
2800
3000
2600
2800
3000
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
5000
Epoch
Epoch
(c) Own coins collected MDP
(f) Own coins collected POMDP
Own Coin
0.8
Coin
0.6
0.8
1.000
0.4
0.75
0.4
0.075
02-
0.850
02
0.960
2030
2000
3000
2000
2600
3000
0.0
500
1000
1500
2000
2000
3000
500
1003
1500
2000
2600
3000
Epoch
(g) Own coin rate MDP
(h) Own coin rate POMDP
Figure I.1: Comparison of CoinGame-2 MDP and POMDP enviromment performance metrics (a,b) score, (c.d) total coins collected, (e.f) own coins collected, and (g.h) own coin rate for eQMARL with varying input quantum entanglement styles V (blue), vi- (orange), t (green), Φ- (red), and None (cyan) averaged over 10 runs, with +1 std. dev. shown as shaded regions.
25

---

Published as a conference paper at ICLR 2025
1.2
COINGAKE BASELINES COMPARISON
The empirical results for e eQMARL- V qECTDE, FCIDE, and SCTDE, as discussed in Section 3.4
shown in Tables and and and Fig. T.2
The performance for MDP dynamics is shown in Figs. 1.2a
I.2c
12cc
and 1.2g and for POMDP dynamics is shown in Figs. L2b 1.2d
22 and I.2h Importantly
Fig. shed light on when, and how, cooperative strategy achieved by each framework. Further,
through Fig. 1.2 we also observe the relationship between the metrics outlined in Appendix [F] This connection is important, as single metric in isolation only paints part of the performance picture. A full comparison can be achieved by considering the metrics as group, and, particularly, the relationship between agent score, i.c., the sum of rewards, and own coin rate, i.e., the priority given to coins of matching color.
Table 1.3: Comparison of model score and own coin rate performance for MDP and POMDP Coi nGamm-2 environment dynamics using mean, standard deviation, and 95% confidence interval statistics. Best values are highlighted in bold.
Score
Own Coin Rate
Dynamies
Framework
Mean
SD
95% CI
Mean
SD
95% CI
eQMARL-V +
21.11
2.65
(20.91, 21.37)
0.9640
0.0347
(0.9606, 0.9667)
qfCTDE
19.41
6.23
(19.22, 19.60)
0.9398
0.1020
(0.9367, 0.9423)
MDP
SCTDE
14.18
2.69
(13.87, 14.53)
0.8504
0.0928
(0.8436, 0.8558)
fCTDE
12.36
4.41
(12.01, 12.66)
0.8202
0.1379
(0.8153, 0.8255)
eQMARL +++
18.49
3.91
(18.24, 18.75)
0.9226
0.0831
(0.9173, 0.9281)
qÉCTDE
16.79
4.66
(16.43, 17.19)
0.9040
0.1135
(0.8991, 0,9094)
POMDP
SCTDE
13.70
2.79
(13.33, 14.07)
0,8466
0.0985
(0.8407, 0.8525)
FCTDE
13.46
3.24
(13.08, 13.75)
0.8443
0.1026
(0.8389, 0.8495)
Table L4: Comparison of model score and own coin rate convergence (in number of epochs) for MDP and POMDP Co nGame-2 2 environment dynamics. Best values are highlighted in bold.
Epochs to Score Threshold
Epochs to Own Coin Rate Threshold
Dynamics
Framework
20
25
Max (value)
0.95
1.0
Max (value)
eQMARI-VV +
568
2332
2942 (25.67)
376
2136
2136 (1.0)
qÉCTDE
678
2378 (23.38)
397
2832 (0.997) (0.9972)
MDP
SCTDE
1640
2615
2631 (25.3)
1511
2637 (0.9864)
FCTDE
1917
2925 (23.67)
1700
2909 (0.9857)
eQMARL-V**
1049
1745
2950 (26.28)
773
2533 (0.9997)
qICTDE
1382
2124
2871 (26.09)
1038
2887
2887 (1.0)
POMDP
SCTDE
1738
2750
2999 (25.33)
1588
2956 (0. (899))))
FCTDE
1798
2658
2824 (25.49)
1574
2963 (0.9894)
26

---

Published as a conference paper at ICLR 2025
ECTDE
SCIDE
cOMARL-
ICTDE
sCrpE
c0
30
25
25
20
20
16
10
225
2600
2600
500
1000
1500
2000
2500
3000
500
1000
1000
2000
2500
3000
Epoch
Epoch
(a) Score
MDP
(b) Score POMDP
ECTDE
CCCTGE
SCIDE
+M
TCTDF
9CCTDE
SCTDE
30-
30-
25
25
Coleeted
Colcated
20
20
16-
10-
10-
22.5
2600
2000
3000
2000
2000
3000
500
1000
1500
2000
2500
8000
500
1000
1600
2000
2500
3000
Epoch
Epoch
(c) Total coins collected MDP
(d) Total coins collected POMDP
FCTDE
gfCTDE
SCIDE
-M
SCTDE
giCTDE
SETDE
00ARL
30
30
25
Cwn Caledda
Own Callated
20
20-
15
18-
10-
10-
22.5
2600
2800
3000
2600
2900
3000
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
5000
Epoch
Epoch
(c) Own coins collected MDP
(f) Own coins collected POMDP
1LTDE
ofCTDe
SCIOE
c0MAL-
1CTDE
afCTDE
SCTDE
C0MARL
0.0
Coin
0.8
Coin
0.8
0.
1.00
0.0
0.4
0.975
02-
0.90
02
0.900
2030
2000
3000
2000
2800
3000
0.0
0.0
500
1000
1500
2000
2000
3000
500
1000
1500
2000
2000
3000
Epoch
(g) Own coin rate
MDP
(h) Own coin rate POMDP
Figure I.2: Comparison of CoinGame-2 MDP and POMDP environment performance metrics ba score, (c,d) total coins collected, (e.f) own coins collected, and (g.h) own coin rate for ICTDE (orange), CTDE (magenta), SCTDE (green), and eQMARI. (blue) averaged over 10 runs, with 1 std. dev. shown as shaded regions.
27

---

Published as a conference paper at ICLR 2025
1.3 CARTPOLE BASELINES COMPARISON
The empirical results for e eQMARL- +++ q qECTDE, FCIDE, and SCTDE, as discussed in Section 3.5
shown in Tables I.3 and L6 Fig. I.3 The performance for MDP dynamies is shown in Fig. 1.3a and for POMDP dynamics is shown in Fig. 1.3b Importantly, from this we see that the classical models do not perform well overall in either setting, and qÉCTDE experiences high variance in the MDP case. Even though SCTDE. has a higher reward at the end of training in the POMDP case, it converges considerably more slowly, experiencing high variance at the end, and requires over 400 more epochs achieve mean value less than half of eQMARL. In contrast, eQMARL is more stable than qÉCTDE, and more rapidly converges to a higher mean reward than ECTDE and sCTDE across both settings.
Table I.5: Comparison of model average reward performance for MDP and POMDP CartPole environment dynamics using mean, standard deviation, and 95% confidence interval statistics.
Reward
Dynamics
Framework
Mean
SD
95% CI
eQMARL- Wr+
79.11
50.62
(77.26, 81.01)
qECTDE
121.35
110.13
(117.95, 124.59)
MDP
SCTDE
16.07
22.15
(15.90, 16.21)
ECTDE
15.14
17.43
(15.06, 15.22)
eQMARL-V* +
82.28
44.24
(80.80, 83.91)
qFCTDE
79.03
44.06
(76.27, 81.02)
POMDP
SCTDE
47.59
29.71
(44.71, 50.86)
ECTDE
11.62
32.02
(11.45, 11.82)
Table L6: Comparison of model average reward convergence (in number of epochs) for MDP and POMDP CartPole environment dynamics.
Epochs to Average Reward Threshold
Dynamics
Framework
Mean (value)
Max (value)
eQMARL-
166 (79.1I)
555 (134.16)
ςECTDE
189 (121.35)
810 (262.43)
MDP
SCTDE
23 (16.07)
978 (24.64)
fCTDE
9 (15.14)
44 (79.43)
eQMARL +
251 (82.28)
770 (127.60)
qfCTDE
276 (79.03)
648 (137.66)
POMDP
SCTDE
669 (47.59)
998 (172.16)
ECTDE
9 (11.62)
999 (28.83)
TCTOE
KOMARL
ICTDE
ofCTBE
SCTDE
<CMARL-
250
300
200
Pverage Reward
200
100
100
0-
1000
200
Epoch
Epoch
(a) Average Reward MDP
(b) Average Reward POMDP
Figure 1.3: Comparison of CartPole MDP and POMDP environment average reward performance for ECTDE (orange), şFCTDE (magenta), sCTDE (green), and eQMARL- + (blue) averaged over runs of 1000 epochs, with +1 std. dev. shown shaded regions.
28

---

Published as a conference paper at ICLR 2025
1.4 MINIGRID BASELINES COMPARISON
The empirical results for e eQMARL V+,, qÉCTDE, FCIDE, and SCTDE, as discussed in Section 3.6
shown in Table 1.7 and Fig. L.4 Importantly, from Fig. [44 we we see that the qfCTDE, fCTDE, and SCTDE baselines have an average reward that is clustered near - -100 for the majority of training. This implies that the baselines learn to exhaust many steps by simply spinning in place, since the maximum step size is 50 and the agents receive -2 reward for staying in the same position as previous time step. In contrast, we see that the average reward of eQMARL-V** is spread out higher over the training regime. From Table I.7 we specifically see that eQMARL y achieves an average overall reward of 13.32, which is 4.5es higher than the baselines. Indeed, this negative reward means that eQMARL- V also expends actions turning in place, but the fact the reward is close to zero implies these events occur at a vastly reduced frequency than the baselines.
In addition, from Table 1.7
that both eQMARL- v and qECTDE reduce the overall critic size by a factor of 8 compared to the classical baselines. This reduction in size means that eQMARI-W and qfCTDE,
more computationally efficient than the classical baselines. Further, eQMARL-V* is even more efficient because it only requires a single centralized trainable parameter, which is significant 200-times reduction in size compared to SCIDE.
that
In testing, eQMARL-V+ was able to traverse to the goal in as little as steps, whereas fCTDE required 17 steps, and both qFCTDE and sCTDE were unable find the goal within the 50 step limit. This is a marked 50% improvement in the exploration and navigation speed of eOMARL-V -r over ECTDE, with the bonus of observation sharing, and an 8-times smaller overall critic size. Hence, we have shown that eQMARI- + can indeed be applied to more complex environments, such as grid-world navigation with limited visibility, and provide learning benefits over baselines without the need for observation sharing.
Table L.7: Comparison of model average reward performance for POMDP Mi Minii Gr MiniGric id environment dynamics using mean and 95% confidence interval statistics, and comparison of model size in number of trainable critic parameters for each framework.
Reward
Framework
Mean
95% CI
Number of Trainable Critic Parameters
ECTDE
-63.04
(-65.16, -61.06)
29,601
qECTDE
-85.86
(-87.03, -84.72)
3,697
SCTDE
-88.02
(-88.69,
-87.10)
29,801 (14,800 per agent, 201 central)
eQMARL +
-13.32
(-14.68, -11.91)
3,697 (1,848 per agent, I central)
FCTDE
OfCTDE
SCTDE
KOMARL -19"
40
Average Reward
0-
-20
-40
-100
200
400
600
800
1000
Epech
Figure L4: Comparison of MiniGrid POMDP environment average reward performance for ECTDE (orange), qfCTDE (magenta), SCTDE (green), and eQMARL- + (blue) for 2 agents over 1000 epochs, with a maximum step limit of 50.
29

---

Published as a conference paper at ICLR 2025
1.5 ABLATION STUDY
The empirical results for the ablation study as discussed in Section 3.7 are shown in Table L.8 and Figs.
Looking at the statistics in T.bleIT8aan the convergence in Figs. [.3] and L6 we can see that our selection of h 12 hidden units for the baselines and L 5 for the quantum models is fair because of the significant performance drops and increased variation incurred by reducing, and the limited gains by increasing, the number units and layers. This choice for architecture results in the most comparable performance across all basclines. From the results of this ablation study, can more concretely represent a comparison of the final sizes of the actor and critic models, in number of trainable parameters, for eQMARL, qFCTDE, FCTDE, and SCTDE. The final selected model sizes, in number of trainable parameters, are shown in Table
Table 188 Ablation study with classical model hidden layer units & E (3. 6, 12, 24) and quantum VQC layers L {2. 5, 10}. Compares model size in number of trainable critic parameters with score and own coin rate performance for MDP and POMDP Co CoinGame- 2 environment dynamics using mean, standard deviation, and 95% confidence interval statistics.
Score
Own Coin Rate
Dynamics Framework
Params
Mean
95%
Mean
95% C1
ECTDE-3
223
2.42
2.35
(2.35, 2.49)
0.6720
0.2024
(0.6685, 0.6769)
ECIDE
445
7.41
3.46
(7.19.7668
0.7658
0.1414
(0.7610. 0.7712)
C01DE-12
889
12.36
4.41
(12.09, 12.67)
0.8202
0.1379
(0.8139. 0.8262)
ECTDE-24
1777
17.63
2.58
(17.25, 17.91)
0.8821
(0.8770, 0.8875)
SCTDE-3
229
3.24
3.09
(3.16, 3.33)
0.6852
0.1991
(0.6821, 0.6897)
aCIDE 6
457
8:54
3.67
(8.29, 8.78)
0.7857
0.1327
(0,7804 0.7924)
sCIDE-12
913
14,18
2.69
(13.90, 14.60)
0.8504
0.0928
(0,8444, 0.8553)
SC1DE-24
1825
18.18
(17.84, 18.53)
0.0673
(0.8896, 0.8979)
gECTDE -1
121
6,58
3.92
(6.47, 6.66)
0.8482
0,1921
(0.8435, 0.8518)
QECTDE-15
265
19.41
6.23
(19.23, 19.59)
0.9398
0.1020
(0.9366, 0,9426)
GfCTDR-110 10
505
22.08
2.22
(21.91, 22.26)
0.9691
0.0247
(0.9665, 0.9723)
eQKARL- +12
121
5.38
3.74
(5.30, 5,46)
0.8271
0.2213
(0.8234, 0.8300)
HOKARL +
L5
265
21.11
2.65
(20.92, 21.35)
0.9640
0.0347
(0 00.9601 0,9667)
eQKARL & -L10
505
22.45
2.23
(22.28, 22.62)
0.9719
0.0219
(0.9685. 0.9745)
ICTDE-3
169
2.98
2.47
(2.91, 3.05)
0.7082
0.1890
(0. (0.7039. 0.7123)
ECTDE 6
337
7.15
3.06
(6.95, 7.37)
0.7711
0.1388
(0,7658, 0.7781)
ECIDE-12
673
13.46
3.24
(13.09, 13.76)
0.8443
0.1026
(0.8396, 0.8506)
#CTDE-24
1345
17.38
2.65
(17.06, 17.73)
0.8889
0.0752
(0.8840, 0.8945)
SCTUE-3
175
2.68
2.60
(2.61, 2.74)
0.6834
0.1942
(U.6792, 0.6866)
SCTDE-6
6.35
3.53
(6.18, 6.54)
0.7677
0.1488
(0.7633, 0.7725)
POMDP
SCIDE-12
697
13.70
2.79
(13.44, 13,99)
0.8466
0.0985
(U.8411, 0.8515)
SCTDE-24
1393
17.97
2.60
(17.67, 18.25)
0.8948
0.0723
(0.8898, 0,9004)
QSCTDE-12
745
12.34
7.56
(12.09, 12.60)
0.8335
0.2058
(U.8277, 0.8386)
QECTDE-15
4.66
(16.45, 17.04)
0.1135
(0.8994, 0.9091)
QICTDR-T10 10
9.17
18.14
4.28
(17.83, 18.31)
0.9476
0.0660
(0.9443, 0.9508)
COKARL-V -
3.98
(16.77, 17.47)
0.1106
(0.8769, 0.8896)
CONARL- +1.
817
18.49
3.91
(18.23, 18.80)
0.9226
0.0831
(0.9172. 0,9272)
eQKARL V -L10
937
19.09
3.44
(18.86, 19.46)
0.9485
0.0603
(0.9458. 0.9523)
Table 1.9: Comparison of the best model size in number of trainable parameters for each framework used on CoinGame environment with MDP and POMDP dynamics.
Number of Trainable Purameters
Framework
Ablation Selection
Model
dynamicy
POMDP dynamics
Actor Critic
136
412
eQNARL
265 (132 per agent, central)
817
(408 per agent, central)
Actor Critic
136
qFCTDS
L = 5
265
817
h 12
Actor
496
388
ECTDR
h = 12
Critie
889
673
h= 12
Actor Critic
496
388
SCTDR
= 12
913 (444 per agent, 25 central)
697 (336 per agent, 25 central)
30

---

Published as a conference paper at ICLR 2025
20
16
10
-6
-6
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2000
3000
Epoch
Epoch
(a) ICTDE MDP Score
(b) ICTDE POMDP Score
SCTD6-3
SCT0E-6
SCTD6-12
SCTDE-24
SCT06-3
SCT06-12
SCTDE-24
90
25
16
-6
600
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Epoch
Epach
(c) SCTDE MDP Score
(d) sCTDE POMDP Score
QFCTD6-L2
QTCTEELS
1C
gFCTD6-12
qfCTDELL5
TC
28
100
000
30
25
16
10
10
-5
-0
800
1003
1.500
2000
25000
3000
500
1002
1500
2000
2500
3000
Epach
Epach
(c) qFCTDE MDP Score
(f) gfCTDE POMDP Score
*QMARI -17--12
KOMARL 4-415
ONARII 17
#QMARI -97-12
KOMARL 49-15
ONAAR 47
95
30
20
15
10
-6
500
1000
1000
2000
2000
3000
500
1003
1000
2000
2300
3000
Epoch
Epoch
(g) eQMARL-VV MDP Score
(h) cQMARL-V** POMDP Score
Figure L.5: Score performance for ablation study using CoinGame- - for (a.b) fCTDE, and (c.d) SCTDE, and (e. gFCTDE, and (g.h) eQMARL-V+ with hidden layer units h E (3. 6, 12, 24) and VQC layers L E (2, 5, 10), averaged over 10 runs of 3000 epochs, with # std. dev. shown as shaded regions.
31

---

Published as a conference paper at ICLR 2025
TCTOE-3
7CT0E-
OCTOE-12
ICTDE-24
ICTOE-3
TCT0--
TCTOE-2
TCTDE-24
1.0
Own Con Rate
Own Con Rate
0.8
0.6
02-
0.0
0.0
000
1000
1500
2000
2000
3000
500
1000
1500
2000
2000
3000
Epoch
Epoch
(a) (CTDE MDP- Own Coin Rate
(b) fCTDE POMDP Own Coin Rate
1.0
0.96
Own Rale
Own Coin
0.6
0.-
0.4
02-
02
0.0
0.0
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Epooh
(c) SCTDE MDP Own Coin Rate
(d) sCTDE POMDP Own Coin Rate
efCTDE42
qECTOFALS
3fCT06-L10
4fCTD642
afCTOF-L5
<FCTD6-L110
0.88
0.5
Own Coin
0.6
0.4
0.4
0.0
0.0
1500
2000
1500
2000
3000
Fpoch
Fpoch
(c) qICTDE MDP- Own Coin Rate
(f) qfCTDE POMDP Own Coin Rate
cOMARL 92-L2
eDPARI CPARL-V'LLS -
CPAARL 977410
eDPARL -V--2
edPARL SPARL-V'LLS -9"LL
eOWARL U11110
0.8
0.8
0.8
0.6
0.4
0.2
500
0000
+500
2000
2500
3000
500
1000
1000
2000
2503 3000
Epoch
Epoch
(g) cQMARL-D" -DP- Own Coin Rate
(h) cQMARL-*** POMDPP Own Coin Rate
Figure 1.6: Own Coin Rate performance for ablation study using CoinGame- for (a,b) fCTDE, and (c.d) SCTDE, and (e,f) gICTDE, and (g.h) eQMARI- with hidden layer units E (3, 6, 12, 24) and VQC layers L {2, 5, 10), averaged over 10 runs of 3000 epochs, with +1 std. dev. shown as shaded regions.
32


Reviews:
[{"review_id": "Si9bI5HcHb", "reviewer": "Reviewer_nHF3", "summary": "The paper proposed a novel framework for multiagent RL, based on quantum entanglement. As compared to classical CTDE approaches wherein the communication between the centralized critic and agents happens in the form of sharing observations / states, the proposed framework uses quantum entanglement for information transfer. The framework uses a split critic such that the global value is calculated by transfer of information to a central server via quantum entanglement. The authors present results on two tasks - CoinGame and multi agent cartpole, demonstrating the effectiveness of the proposed framework in sample efficient learning.", "questions": "* From the description provided in Appendix E.2, the multi-agent cartpole does not seem to benefit much from coordination among different agents, since all the agents operate independently. Given that, is it really a good task for evaluating the framework?\n* Is it feasible to run experiments with environments consisting of higher dimensional observation spaces. For eg. running experiments on some moderately challenging MiniGrid environments would be interesting.\n* Can the authors provide details on the compute required for performing the experiments?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The framework proposed is novel and the paper explores the rather unstudied area of using quantum entanglement for information sharing in Multi-Agent RL\n* The framework distributes computations across agents, which decreases the centralized computational burden thereby making the framework more scalable\n* The proposed framework minimizes the communication overhead between the agents and the centralized critic as well as among the agents via quantum entanglement, as compared to classical information sharing in normal CTDE approaches. \n* The empirical evaluation includes a nice study justifying the choice of entanglement type.", "weaknesses": "* The paper is not easily understandable to a reader not completely familiar with the area of quantum entanglement. The appendix provides the necessary background on this. However, a brief introduction of the same (a compressed version) in the main paper might provide some context to the reader in order to better understand the paper.\n* Some parts of the experimental results section (specifically, lines 363 - 374 and 412 - 425 Section 3.4 and lines 457-464 in Section 3.5) are rather poorly written. Focusing more on the general conclusions as compared to specific numbers and focusing on one metric at a time (either sample efficiency OR absolute performance)  could help improve the readability of the section.\n* Although the proposed framework results in lower variance across seeds, the results on multi-agent cartpole task don't seem very encouraging. The proposed framework fails to beat the baselines in the case of both MDP and POMDP\n*  The plots in Figure 3 do not immediately make the conclusions clear. Replacing Figure 3 with Tables H.1 and H.2 could be more compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KJlcjI7Awo", "reviewer": "Reviewer_H1NG", "summary": "This paper proposes eQMARL, a novel quantum multi-agent reinforcement learning framework that uses quantum entanglement to enable distributed cooperation between agents, demonstrating faster convergence (up to 17.8%) and better performance compared to classical and quantum baselines while requiring 25 times fewer centralized parameters.", "questions": "1. How does the approach handle agent failures or dropped quantum connections? The current design seems to assume perfect quantum channels and continuous agent availability.\n\n2. How does the choice of entanglement scheme $(\\Psi^+, \\Psi^-, \\Phi^+, \\Phi^-)$ affect the information sharing capacity between agents? Why $\\Psi^+$ performs best?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper seems to be the first to use quantum channels and entanglement for agent coordination, rather than just replacing classical neural nets with quantum circuits. The split quantum critic design that maintains cooperation without observation sharing seems novel to me.", "weaknesses": "1. The approach requires sophisticated quantum hardware for generating, maintaining, and transmitting entangled states across agents - capabilities that current NISQ devices don't reliably support.\n\n2. The method's scalability remains unclear, as it's only demonstrated with simple environments and few agents, and could face significant challenges with quantum decoherence and reduced entanglement fidelity as the system grows.\n\n3. The experiments have too small scale for a purely empirical paper.\n\n4. (Minor) you should specify in the main paper that $\\otimes$ is the Kronecker (tensor) product.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aTnWQD4Tj6", "reviewer": "Reviewer_xBPK", "summary": "Distributed multi-agent reinforcement learning (MARL) faces challenges in balancing the benefits of player coordination with the communication and computational costs of sharing data between players. Quantum mechanics offers potential for more efficient collaboration through quantum entanglement mediated information sharing, but most current MARL approaches still rely on classical information sharing. This paper introduces a novel framework called entangled QMARL (eQMARL), which eliminates local observation sharing by leveraging quantum entanglement, an input entangled state is prepared and joint measurements are performed. A quantum split critic across agents enables joint observation-value estimation without any additional classical communication overhead to share local environment observations. Experimental results show eQMARL converges faster in some cases and with fewer parameters on a central server compared to classical and quantum baselines. \n\nThe authors perform experiments on CoinGame and CartPole environments and run these as MDP and POMDP processes using different scoring metrics. Several classical and quantum variants of MARL are benchmarked and compared for converge behaviour, stability, robustness, solution quality and space complexity. Finally, an ablation study on network size supplements the main results and provides insight into the scaling of solution quality with the number of network layers.", "questions": "In the last paragraph the authors claim that \"the overall system size is larger, however, the number of central parameters is significantly reduced in the quantum cases – requiring only 1 parameter instead of 25.\" - can the authors dive into a space time tradeoff here, because no information about algorithm runtimes is given? Even if the eQMARL has fewer centralised parameters, what is its runtime compared to the alternatives? A discussion of this would also be important when comparing the different algorithms in Figs 3-5. \n\nDoes the preparation of the input entangled states introduce some computational overhead, can this be quantified? How does this compare to the benchmarked alternative algorithms?\n\nWhy does \\Sigma^+ perform better than \\Sigma^- in the discussion of Fig.3 ? It is only the relative phase which changes not the general entanglement structure.\n\nWhy does the None benchmark in Fig.3 perform almost as well as the \\Sigma^+ benchmark, is entanglement even required to achieve any advantage and why can you achieve a high score without entanglement?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper describes both a novel framework and a novel algorithm which clearly outline how you can train decentralised policies over quantum channels with entangled input qubits. All that is required are joint measurements without sharing local observations amongst agents or a central server. The framework and the technical details are well outlined in Sec 2. with a detailed exposition of the entanglement preparation and the different architectural components which are outlined in Fig. 2. This makes it clear to understand its contribution which lies in the incorporation of quantum channels. As outlined, these do not require explicit agent-server observation reporting and this is a novel Ansatz which has not been explored in the literature.\n\nThe benchmarks provided are comprehensive and well presented. Especially in Fig.4 for the CoinGame comparison the eQMARL shows superior convergence dynamics and also improved scores and higher own coin rates. The observation that entanglement stabilises the network is also well put and particularly interesting. Another strong point that is clearly outlined is the idea that additional information in the case of POMPDP would supposedly result in superior performance, but actually it suffices to use entangled input states to generate good learning dynamics. This is interesting for a broader audience. \n\nThe work's significance is also clearly outlined. For the benchmarked examples the eQMARL algorithm requires fewer centralised parameters and exhibits more stable and robust learning dynamics than the alternatives. In some cases it also achieves a better solution.", "weaknesses": "The results in Fig. 3 are confusing and deserve a clearer exposition. Given the \"None\" benchmark performs similarly well to all the other Bell states (presumably no entanglement? but this is not clear from the text), how one can understand the role played by the entanglement if it does not significantly outperform \"no entanglement\"? It would also be interesting to understand why Bell states \\Sigma^- and \\Sigma^+ perform differently and what role is played by the relative phase?\n\nThe cart pole benchmark does not seem to conclusively show an advantage associated with using the eQMARL framework since in both MDP and POMDP cases the authors algorithm does not achieve the maximum reward and also does not seem to clearly outperform qfCDTE (MDP) or qfCDTE and sCDTE (POMDP)even though in the text an advantage - especially in terms of a high mean average reward at early epochs - is claimed to exist. The authors should consider revising the last paragraph and making it less strongly in favour of the performance of the eQMARL algorithm.\n\nThere is no mention of the time complexity of the novel framework and particularly the challenge associated with running quantum circuits and preparing entangled input states (notwithstanding the challenge of running this in a non-simulated real quantum device). This makes the benchmarks which all run for the same number or epochs less clear. Particularly in Fig.5b the authors claim the advantage of eQMARL lies in the fact that it achieves a higher reward at a much lower epoch number, but without any information on the per epoch runtime of the different algorithms this case becomes less strong. For example if sCDTE runs 10 times faster than this would not make a strong case for preferring eQMARL over sCTDE. The authors should also consider adressing this in Section 3 when discussing the scaling and performance for different network sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)