SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Youpeng Zhao $^{1}$ , Yaodong Yang $^{2\dagger}$ , Zhenbo Lu $^{3\dagger}$ , Wengang Zhou $^{1,3}$ , Houqiang Li $^{1,3}$
<sup>1</sup>University of Science and Technology of China, <sup>2</sup>Institute for AI, Peking University
<sup>3</sup>Institute of Artificial Intelligence, Hefei Comprehensive National Science Center zyp123@mail.ustc.edu.cn, yaodong.yang@pu.edu.cn
luzhenbo@iai.ustc.edu.cn, {zhwg, lihq}@ustc.edu.cn
In the realm of multi-agent reinforcement learning (MARL), achieving high performance is crucial for a successful multi-agent system. Meanwhile, the ability to avoid unsafe actions is becoming an urgent and imperative problem to solve for real-life applications. Whereas, it is still challenging to develop a safety-aware method for multi-agent systems in MARL. In this work, we introduce a novel approach called Multi-Agent First Order Constrained Optimization in Policy Space (MAFOCOPS), which effectively addresses the dual objectives of attaining satisfactory performance and enforcing safety constraints. Using data generated from the current policy, MAFOCOPS first finds the optimal update policy by solving a constrained optimization problem in the nonparameterized policy space. Then, the update policy is projected back into the parametric policy space to achieve a feasible policy. Notably, our method is first-order in nature, ensuring the ease of implementation, and exhibits an approximate upper bound on the worst-case constraint violation. Empirical results show that our approach achieves remarkable performance while satisfying safe constraints on several safe MARL benchmarks.
Cooperative multi-agent systems play a significant role in various domains, where a group of agents coordinate with each other to accomplish tasks and collaboratively optimize cumulative rewards for the team[1, 2]. Such a setting is frequently employed in many real-life scenarios such as robotics [3], autonomous vehicles [4], traffic light control [5] and the smart grid [6].Thanks to the recent remarkable advance of reinforcement learning techniques in various complex tasks [7-9], multi-agent reinforcement learning (MARL) has attracted substantial attention and quite a few algorithms have been proposed, including value-based methods [10-14] and policy gradient methods [15-18]. Despite the notable achievement in academia, most MARL algorithms prioritize policy optimization solely for reward maximization, while disregarding potential negative or harmful consequences resulting from the agents' behaviors. Consequently, these methods can not be directly deployed in practice. In reality, many applications often require the agents to refrain from taking certain actions or visiting particular states [19, 20]. For instance, an unmanned car must adhere to traffic regulations by not crossing a red light, even while pursuing its destination, in order to prioritize safety.
To address the above issues, researchers have been devoted to developing algorithms that learn policies which adhere to safety constraints and great progress has been made in single-agent RL setting[20-23]. However, it is still a daunting challenge to develop safe policies for multi-agent systems. Because of the existence of multiple agents, the environment may suffer from non-stationarity due to simultaneously learning agents, posing a non-negligible challenge to the training process. Moreover, ensuring safety in MARL is highly intricate. To provide a better depiction of the inherent challenges,
Figure 1: An example to illustrate the challenges in safe MARL. Reward distribution is presented in the policy space for each agent. Green rectangles delineate the individual safe areas for each agent and the blue ones depict the joint safe area. Green arrows symbolize the update direction for each agent whereas the dashed ones signify the intended update direction if each agent solely considers its own objectives and the crosses represent that the dashed update direction is unsuitable in safe MARL.
we show an illustrative example in Figure 1. This figure showcases the policy space for each agent, wherein the reward distribution is presented. The green rectangles delineate the individual safe areas for each agent and the blue ones depict the joint safe area. The green arrows symbolize the update direction for each agent, whereas the dashed ones signify the intended update direction if each agent solely considers its own objectives. When agents are randomly initialized, they naturally tend to update the policy along the dashed line arrows to maximize the reward. However, due to the presence of safety constraints, they must adjust their updates along the solid line arrows to ensure their policies fall within the safe area. This is non-trivial as it may conflict with their goal of reward maximization. Furthermore, after individually addressing their safety constraints, the agents must also consider the safety constraints of others to collectively converge to the joint safe area, adding more complexity to the optimization process for the entire system. The interplay of these factors underscores the inherent difficulties in solving the problem of safe MARL.
Recently, as a notable advancement in the field of safe MARL, Multi-Agent Constrained Policy Optimization (MACPO) [24] has been proposed as a safe and effective solution. MACPO attains the properties of both monotonic improvement guarantee and safety constraints satisfaction guarantee at every iteration during training. However, this algorithm involves solving an optimisation problem using Taylor approximations and inverting a high-dimensional Fisher information matrix. As a result, the computation is very complex and the policy update often becomes infeasible. To alleviate this issue, it requires additional recovery steps, which, however, sometimes causes updates to be backtracked and samples to be wasted.
Taking inspiration from the sequential policy update scheme introduced in HATRPO [18], we have devised a simpler approach to incorporate safety constraints in solving safe MARL problems compared with MACPO. The resulting algorithm, Multi-Agent First Order Constrained Optimization in Policy Space (MAFOCOPS), aims to address the following question of how to achieve the best constraint-satisfying policy update given the current policy for each agent. Our method follows a two-step process to give a solution to this problem. First, based on the theoretical foundations presented in MACPO, we demonstrate that the best policy update has a near-closed form solution when attempting to solve the optimal policy in the nonparametric policy space. Subsequently, we project the policy back into the parametric policy space as direct evaluation of the optimal policy is usually not feasible. This can be realized by sampling from the current policy and evaluating a loss function between the parameterized policy and the optimal policy obtained in the nonparametric policy space. Notably, our algorithm only employs first-order approximations, making it straightforward to implement, and has an approximate upper bound for worst-case constraint violation. To validate the effectiveness of our approach, we conduct experiments on two safe MARL benchmarks proposed by [24], namely Safe MAMuJoCo and Safe MAIG. The experimental results demonstrate the superior performance of MAFOCOPS compared to MACPO, despite being a simpler algorithm.
Safety has become a crucial and longstanding concern in the field of reinforcement learning [25]. In this section, we discuss recent advancements in the domain of safe RL for multi-agent environments.
The realm of safe multi-agent reinforcement learning is a nascent area of research that has gained increasing importance [26]. Several attempts have been made to tackle safe MARL, but most existing approaches have limitations. For example, CMIX [27] leverages the value function decomposition method and modifies the reward function to account for constraint violations, yet this algorithm fails to provide safety guarantees during training. Another approach is Safe Dec-PG [28] which proposes a decentralized policy gradient descent-ascent method by means of a consensus network and employs a primal-dual framework to balance reward maximization and cost minimization. Nevertheless, the consensus requirement in this work equivalently imposes an extra constraint of parameter sharing among neighbouring agents, potentially leading to sub-optimal policy [18]. It is noteworthy that most current safe MARL methods are tailored to robotics tasks, utilizing techniques such as barrier certificates [29, 30] or model predictive shielding [31] to address safety issues. These methods, however, are specifically designed for robotics applications. Besides, they often require supervised learning based approaches or specific assumptions concerning the state space and environment dynamics.
As a recent remarkable solution, MACPO [24] incorporates a sequential policy update scheme. This algorithm develops the multi-agent trust region learning based on CPO [20] and provides theoretical guarantees of both monotonic improvement in reward and compliance with cost constraints in a multi-agent setting. To achieve practical solutions to the safe MARL problems, the policy for each agent needs to be parameterized with a neural network, which effectively represents the policy in policy space. However, achieving parameterized policies in MACPO involves solving optimization problems using first and second-order Taylor approximation, which includes taking the inverse of Fisher information matrix, and the computation is implemented by the conjugate gradient method [32]. These operations can introduce nonnegligible approximation errors, thereby compelling MACPO to undertake additional steps during each update in the training process in order to recover from constraint violations. In contrast, our algorithm takes a different approach by solving the optimization problem within the nonparametric space and then projecting the results back into the parameter space. By leveraging a simple first-order method to eliminate the approximation error, our algorithm ultimately outperforms MACPO.
A safe MARL problem can be formulated as a Constrained Markov Decision process, which is described by a tuple denoted as  $< \mathcal{N},\mathcal{S},\mathcal{A},p,\rho^0,\gamma,R,\mathcal{C},c>$ . Here,  $\mathcal{N} = \{1,2,\dots,n\}$  denotes the set of agents involved in the system,  $\mathcal{S}$  is the state space,  $\mathcal{A} = \Pi_{i=1}^{n}\mathcal{A}^i$  represents the product of the action spaces of agents, i.e., joint action space,  $p:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \mathbb{R}$  is the probabilistic transition function,  $\rho^0$  is the initial state distribution and  $\gamma \in [0,1)$  is the discounted factor. The team reward function  $R:S\times \mathcal{A}\rightarrow \mathbb{R}$  maps state and joint actions to a scalar reward while  $\mathbf{C} = \{C_j^i\}_{1\leq j\leq m^i}$  is the set of sets of cost functions denoted in the form  $C_j^i:\mathcal{S}\times \mathcal{A}^i\to \mathbb{R}$  (every agent  $i$  has  $m^i$  cost functions). The set of corresponding cost-constraining values is given by  $c = \{c_j^i\}_{1\leq j\leq m^i}$ . At time step  $t$ , the multi-agent system is situated in state  $s_t$  and every agent  $i$  selects an action  $a_t^i$  based on its policy  $\pi^i(a^i|s)$ , forming a joint action  $\boldsymbol{a}_t = (a_t^1,\dots,a_t^n)$  and joint policy  $\pi(\boldsymbol{a}|s) = \Pi_{i=1}^{n}\pi^i(a^i|s)$ . This leads the environment to transit to a new state  $s_{t+1}\sim p(\cdot|s_t,\boldsymbol{a}_t)$  according to the probabilistic transition function and the system receives the reward  $R(s_t,a_t)$  while each agent  $i$  incurs individual costs  $C_j^i,\forall j=1,\dots,m^i$ . This study focuses on a fully-cooperative multi-agent setting where all agents share the same reward function, aimed at maximizing the expected total reward:
$$
J (\boldsymbol {\pi}) \triangleq E _ {s _ {0} \sim p ^ {0}, \boldsymbol {a} _ {0: \infty} \sim \boldsymbol {\pi}, s _ {1: \infty} \sim p} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} R (s _ {t}, \boldsymbol {a} _ {t}) ].
$$
Moreover, we impose that each agent  $i$  satisfies its safety constraint, which is defined as
$$
J _ {j} ^ {i} (\boldsymbol {\pi}) \triangleq E _ {s _ {0} \sim \rho^ {0}, \boldsymbol {a} _ {0: \infty} \sim \boldsymbol {\pi}, s _ {1: \infty} \sim p} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} C _ {j} ^ {i} \left(s _ {t}, a _ {t} ^ {i}\right) \right] \leq c _ {j} ^ {i}, \forall j = 1, \dots , m ^ {i}.
$$
We define the state-action value and the state-value function in terms of reward as
$$
Q _ {\boldsymbol {\pi}} (s, \boldsymbol {a}) \triangleq E _ {s _ {1: \infty} \sim p, \boldsymbol {a} _ {1: \infty} \sim \boldsymbol {\pi}} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} R (s _ {t}, \boldsymbol {a} _ {t}) | s _ {0} = s, \boldsymbol {a} _ {0} = \boldsymbol {a} ], V _ {\boldsymbol {\pi}} (s) \triangleq E _ {\boldsymbol {a} \sim \boldsymbol {\pi}} [ Q _ {\boldsymbol {\pi}} (s, \boldsymbol {a}) ].
$$
It's worth noting even though the action  $a_{t}^{i}$  taken by agent  $i$  does not directly impact the costs  $\{C_{j}^{k}(s_{t},a_{t}^{k})\}_{j = 1}^{m_{k}}$  of other agents  $k \neq i$  from the above formulation, the total costs can still be influenced by this action implicitly due to its influence on the subsequent state  $s_{t + 1}$ . This formulation captures the realistic multi-agent interactions in real world. For instance, when a car runs a red light, although other cars may not be immediately endangered by this action, the resulting disruption in traffic flow may lead to potential hazards later on. To illustrate the  $j_{th}$  cost function of agent  $i$ , we express the corresponding state-action cost value function and the state cost value function as below:
$$
Q _ {j, \boldsymbol {\pi}} ^ {i} (s, \boldsymbol {a}) \triangleq E _ {\boldsymbol {a} ^ {- i} \sim \boldsymbol {\pi} ^ {- i}, s _ {1: \infty} \sim p, \boldsymbol {a} _ {1: \infty} \sim \boldsymbol {\pi}} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} C _ {j} ^ {i} (s _ {t}, a _ {t} ^ {i}) | s _ {0} = s, a _ {0} ^ {i} = a ^ {i} ],
$$
$$
V _ {j, \pi} ^ {i} (s) \triangleq E _ {\boldsymbol {a} \sim \boldsymbol {\pi}, \boldsymbol {a} \sim \boldsymbol {\pi}} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} C _ {j} ^ {i} \left(s _ {t}, a _ {t} ^ {i}\right) | s _ {0} = s ].
$$
Notably, although similar in form to traditional  $Q_{\pi}$  and  $V_{\pi}$ , the cost value function  $Q_{j,\pi}$  and  $V_{j,\pi}$  involve additional indices  $i$  and  $j$ , where the subscript  $i$  refers to an agent and  $j$  denotes the  $j^{th}$  cost.
Motivated by the sequential policy update scheme, we pay close attention to determining the contribution of different subsets of agents to overall performance in this study. We denote an arbitrary subset  $\{i_1,\dots ,i_h\}$  of agents as  $i_{1:h}$  while  $-i_{1:h}$  refers to its complement. Given the agent subset  $i_{1:h}$ , we define the multi-agent state-action value function:
$$
Q _ {\boldsymbol {\pi}} ^ {i _ {1: h}} (s, \boldsymbol {a} ^ {i _ {1: h}}) \triangleq E _ {\boldsymbol {a} ^ {- i _ {1: h}} \sim \boldsymbol {\pi} ^ {- i _ {1: h}}} \left[ Q _ {\boldsymbol {\pi}} (s, \boldsymbol {a} ^ {i _ {1: h}}, \boldsymbol {a} ^ {- i _ {1: h}}) \right].
$$
Furthermore, for disjoint sets  $i_{1:h}$  and  $j_{1:k}$ , the multi-agent advantage function is defined as follows:
$$
A _ {\pi} ^ {i _ {1: h}} \left(s, \boldsymbol {a} ^ {j _ {1: k}}, \boldsymbol {a} ^ {i _ {1: h}}\right) \triangleq Q _ {\pi} ^ {j _ {1: k}, i _ {1: h}} \left(s, \boldsymbol {a} ^ {j _ {1: k}}, \boldsymbol {a} ^ {i _ {1: h}}\right) - Q _ {\pi} ^ {j _ {1: k}} \left(s, \boldsymbol {a} ^ {j _ {1: k}}\right).
$$
An interesting and critical observation concerning the aforementioned multi-agent advantage function is that the advantage  $A_{\pi}^{i_{1:h}}$  can be written as the sum of sequentially-unfold multi-agent advantages of individual agents, that is,
Lemma 1 (Multi-agent advantage decomposition [18]) For any state  $s \in S$ , subsets of agents  $i_{1:h} \in \mathcal{N}$  and joint action  $\pmb{a}^{i_{1:h}}$ , the following identity holds
$$
A _ {\pi} ^ {i _ {1: h}} (s, \boldsymbol {a} ^ {i _ {1: h}}) \triangleq \sum_ {j = 1} ^ {h} A _ {\pi} ^ {j} (s, \boldsymbol {a} ^ {i _ {1: j - 1}}, a ^ {i _ {j}}).
$$
Expanding on the above foundational concepts and the derivatives of multi-agent trust region learning with constraints, MACPO provided an important insight. It highlighted that when the policy changes for all agents are sufficiently small, each agent can learn a better policy  $\bar{\pi}^i$  by only considering its own surrogate return and costs, which is consistent with the sequential policy update scheme. In our work, building upon the formulas in MACPO, we can deduce that, for agent  $i_h$  and the index of its cost functions  $j$ , given the joint policy  $\pi_{\theta_k}$  at the  $k$ th iteration and updated policies of the previous agent sets  $i_{1:h-1}$ , namely  $\pi_{\theta_{k+1}}^{i_{1:h-1}}$ , the new policy is obtained by solving the following optimization problem:
$$
\underset {\pi_ {\theta} ^ {i _ {h}}} {\text {m a x i m i z e}} E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {1: h - 1}} \sim \pi_ {\theta_ {k + 1}} ^ {i _ {1: h - 1}}, a ^ {i _ {h}} \sim \pi_ {i _ {h}}} \left[ A _ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} (s, a ^ {i _ {1: h - 1}}, a ^ {i _ {h}}) \right], \tag {1}
$$
$$
s. t. J _ {j} ^ {i _ {h}} \left(\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}\right) + E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h}}} \left[ A _ {j, \boldsymbol {\pi} _ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right] \leq c _ {j} ^ {i _ {h}}, \forall j \in 1, \dots , m ^ {i _ {h}}, \tag {2}
$$
$$
\bar {D} _ {K L} \left(\pi_ {\theta} ^ {i _ {h}}, \pi_ {\theta_ {k}} ^ {i _ {h}}\right) \leq \delta . \tag {3}
$$
where  $\bar{D}_{KL}(\pi_{\theta}^{i_h},\pi_{\theta_k}^{i_h})\triangleq E_{s\sim \rho_{\pi_{\theta_k}}}[D_{KL}(\pi_{\theta_k}^{i_h}(\cdot |s),\pi_\theta^{i_h}(\cdot |s))]$ . A simple proof is presented in Appendix A. When solving the optimization problem (1-3), we use a two-step approach:
1. For agent  $i_h$ , when provided with the joint policy  $\pi_{\theta_k}^-$  and updated policy  $\pi_{\theta_{k+1}}^{i_{1:h-1}}$ , find an optimal update policy  $\pi^{i_h*}$  in the nonparameterized policy space, denoted by  $\Pi$ .
2. Project the policy found in previous step back into parameterized policy space  $\Pi_{\theta}$  by solving for the closest policy  $\pi_{\theta} \in \Pi_{\theta}$  to obtain  $\pi_{\theta_{k+1}}^{i_h}$ . Here we consider the set of parameterized policies  $\Pi_{\theta} = \{\pi_{\theta} : \theta \in \Theta\} \in \Pi$  which allows for evaluation and sampling.
4.1 Finding the Optimal Update Policy
In the first step, we aim to find the optimal nonparameterized policy and propose the following solutions (see Appendix B for proof):
Theorem 1. For agent  $i_h$ , we define  $b_j^{i_h} = c_j^{i_h} - J_j^{i_h}(\pi_{\theta_k})$ . If  $\pi_{\theta_k}$  is a feasible policy, the optimal policy takes the form
$$
\pi^ {i _ {h} *} (a | s) = \frac {\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)}{Z _ {\lambda_ {j} , \nu_ {j}} (s)} \exp \left\{\frac {1}{\lambda_ {j}} \left(\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \right\}, \tag {4}
$$
where  $\eta_{\pi_{\theta_k}}(s,a^{i_h}) = E_{a^{i_{1:h - 1}}\sim \pi_{\theta_{k + 1}}^{i_{1:h - 1}}}[A_{\pi_{\theta_k}}^{i_h}(s,a^{i_{1:h - 1}},a^{i_h})],Z_{\lambda_j,\nu_j}(s)$  is the partition function that ensures Equation 4 to be a valid probability distribution, and  $\lambda_{j}$  as well as  $\nu_{j}$  are solutions to the following optimization problem:
$$
\min  _ {\lambda_ {j}, \nu_ {j} \geq 0} \lambda_ {j} \delta + \nu_ {j} b _ {j} ^ {i _ {h}} + \lambda_ {j} E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *}} [ \log Z _ {\lambda_ {j}, \nu_ {j}} (s) ]. \tag {5}
$$
The structure of the optimal policy exhibits an intuitive nature, as it assigns a substantial probability mass to regions of the state-action space with high returns. This allocation is counterbalanced by a penalty term multiplied by the cost advantage. Another desirable property of the optimal update policy is that for feasible policy  $\pi_{\theta_k}$ , it has an upper bound for worst-case guarantee for cost constraint satisfaction according to Lemma.2 in MACPO [24].
4.2 Approximating the Optimal Update Policy
When addressing the optimization problem 1-3 in the first step, the optimal update policy for agent  $i_h$  may not necessarily reside within parameterized policy space  $\Pi_{\theta}$ . Consequently, evaluating or sampling from this policy becomes unfeasible. To this end, in the second step, we need to project the optimal update policy back into the parameterized policy space by minimizing the loss function:
$$
L (\theta) = E _ {s \sim \rho_ {\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}} \left[ D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \right\lvert   \left| \pi^ {i _ {h} *}\right) (s) \right]. \tag {6}
$$
In this context,  $\pi_{\theta}^{i_h}\in \Pi_\theta$  represents some projected policy that serves as an approximation of the optimal update policy. To minimize this loss function, first-order methods can be employed. In doing so, we can leverage the following result as a useful tool in our optimization efforts:
Corollary 1. The gradient of  $L(\theta)$  takes the form
$$
\nabla_ {\theta} L (\theta) = E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}} \left[ \nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \right\lvert   | \pi^ {i _ {h} *}) [ s ] \right], \tag {7}
$$
$$
\nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \left\| \pi^ {i _ {h} *}\right) [ s ] = \nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \left\| \pi_ {\theta_ {k}} ^ {i _ {h}}\right) - \frac {1}{\lambda_ {j}} E _ {a \sim \pi_ {\theta_ {k}} ^ {i _ {h}}} \left[ \frac {\nabla_ {\theta} \pi_ {\theta} ^ {i _ {h}} (a | s)}{\pi_ {\theta_ {k}} ^ {i _ {h}} (a | s)} \left(\eta_ {\pi_ {\theta_ {k}}} (s, a ^ {i _ {h}}) - \nu_ {j} A _ {j, \pi_ {\theta_ {k}}} ^ {i _ {h}} (s, a ^ {i _ {h}})\right) \right] \right. \right. \tag {8}
$$
The proof is shown in Appendix C.
It's to be noted that 27 can be estimated by sampling from the trajectories generated by policy  $\pi_{\theta_k}$ , which allows us to train our policy using stochastic gradients, a key aspect of our methodology. This corollary serves as a guiding framework for our algorithm. At every iteration, we begin with a policy  $\pi_{\theta_k}$ , utilizing it to generate trajectories and collect relevant data. Subsequently, we employ this data in conjunction with 5 to estimate  $\lambda_j$  and  $\nu_j$ . We then draw a minibatch from the collected data to estimate  $\nabla_{\theta}L(\theta)$  as outlined in Corollary 1. After taking a gradient step using Equation 27, we draw another minibatch and repeat the process.
For agent  $i_h$ , solving the dual problems presented in 5 is computationally challenging when dealing with large state/action spaces as calculating the partition function  $Z_{\lambda_j, \nu_j}(s)$  often involves evaluating a high-dimensional integral or sum. Moreover,  $\lambda_j$  and  $\nu_j$  are dependent on the iteration  $k$  and need to be adjusted at every iteration to ensure the effectiveness of optimization.