SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on the structure of the optimal update policy, it is observed that as  $\lambda_j \to 0$ ,  $\pi^{i_h*}$  tends to be greedy while the policy becomes more exploratory when  $\lambda_j$  increases. We note that  $\lambda_j$  exhibits similarities to the temperature term utilized in maximum entropy reinforcement learning [33]. Fixed values of  $\lambda$  have been demonstrated to yield reasonable outcomes in training [34, 35, 22]. In practical implementations, we have observed favorable results by using fixed  $\lambda_j$  through hyperparameter sweeps. However continuous adaptation of  $\nu_j$  during training is necessary to ensure cost constraint satisfaction. In this regard, we appeal to an intuitive heuristic based on primal-dual gradient methods [36] to determine the appropriate value of  $\nu_j$ . Recall that by strong duality, the optimal  $\lambda_j^*$  and  $\nu_j^*$  minimizes the dual function 5 which we will denote by  $L(\pi^{i_h*}, \lambda_j, \nu_j)$ . Then we can adopt gradient descent w.r.t  $\nu_j$  to minimize  $L(\pi^{i_h*}, \lambda_j, \nu_j)$  as follows:
Corollary 2. The derivative of  $L(\pi^{i_h*}, \lambda_j, \nu_j)$  w.r.t  $\nu_j$  is
$$
\frac {\partial L \left(\pi^ {i _ {h} *} , \lambda_ {j} , \nu_ {j}\right)}{\partial \nu_ {j}} = b _ {j} ^ {i _ {h}} - E _ {s \sim \rho_ {\pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}}, a ^ {i _ {h}} \sim \pi^ {i _ {h} *} (a | s)} \left[ A _ {j, \pi_ {\boldsymbol {\theta} _ {\boldsymbol {k}}}} ^ {i _ {h}} (s, a ^ {i _ {h}}) \right]. \tag {9}
$$
The proof is shown in Appendix D.
The last term in the gradient expression poses a challenge since  $\pi^{i_h*}$  may not locate in parameterized policy space, leading direct evaluation of the term to be infeasible. Nevertheless, due to the proximity between  $\pi_{\theta}^{i_h}$  and  $\pi^{i_h*}$  enforced by the KL divergence constraint, it's reasonable to assume that  $E_{s\sim \rho_{\pi_{\theta_k}},a^{i_h}\sim \pi^{i_h*}[A_{j,\pi_{\theta_k}}^i (s,a^{i_h})]}\approx E_{s\sim \rho_{\pi_{\theta_k}},a^{i_h}\sim \pi_{\theta_k}^{i_h}[A_{j,\pi_{\theta_k}}^i (s,a^{i_h})]} = 0$  In practice, we have observed that setting this term to zero yields favorable outcomes, which gives the update term as follows:
$$
\nu_ {j} \leftarrow \underset {\nu_ {j}} {\operatorname {p r o j}} \left[ \nu_ {j} - \alpha \left(c _ {j} ^ {i _ {h}} - J _ {j} ^ {i _ {h}} \left(\boldsymbol {\pi} _ {\boldsymbol {\theta} _ {\boldsymbol {k}}}\right)\right) \right], \tag {10}
$$
where  $\alpha$  is the step size to control the magnitude of the update. The projection operator  $proj_{\nu_j}$  ensures that  $\nu_j$  remains within the interval  $[0, \nu_{max}]$ , with  $\nu_{max}$  chosen to prevent  $\nu_j$  from being excessively large. In practical applications, the estimation of  $J_j^{i_h}(\pi_{\theta_k})$  can be accomplished using Monte Carlo methods by leveraging trajectories collected from  $\pi_{\theta_k}$ . This approach aligns with the update rule employed in MACPO [24]. We recall that in 4,  $\nu_j$  acts as a cost penalty term where increasing  $\nu_j$  makes the likelihood of state-action pairs with higher costs being sampled by  $\pi^{i_h*}$  diminish. Consequently, the update rule presented in 10 exhibits an intuitive characteristic: it raises  $\nu_j$  if  $J_j^{i_h}(\pi_{\theta_k}) > c_j^{i_h}$ , indicating a violation of the cost constraint for  $\pi_{\theta_k}$ , and reduces  $\nu_j$  otherwise. Using the proposed update rule,  $\nu_j$  can be updated before updating the policy parameter  $\pi^{i_h}$ .
To be noted, our algorithm is a first-order method, which implies that the approximations made are accurate only around the initial condition (i.e.,  $\pi_{\theta}^{i_h} = \pi_{\theta_k}^{i_h}$ ). To better enforce this condition, we have introduced a per-state acceptance indicator function  $I(s_j) = \mathbf{1}_{D_{KL}(\pi_\theta^{i_h},\pi_{\theta_k}^{i_h})\leq \delta}$  to 27. This function helps in rejecting sampled states whose  $D_{KL}(\pi_{\theta}^{i_h},\pi_{\theta_k}^{i_h})[s]$  is too large, thereby improving the accuracy of the gradient update. The resulting sample gradient update equation is as follows:
$$
\nabla_ {\theta} L (\theta) = \frac {1}{N} \sum_ {j = 1} ^ {N} \left[ \nabla_ {\theta} D _ {K L} \left(\pi_ {\theta} ^ {i _ {h}} \right\lvert   | \pi_ {\theta_ {k}} ^ {i _ {h}} \rbrack s _ {j} \right] - \frac {\nabla_ {\theta} \pi_ {\theta} ^ {i _ {h}} \left(a _ {j} \mid s _ {j}\right)}{\lambda \pi_ {\theta_ {k}} ^ {i _ {h}} \left(a _ {j} \mid s _ {j}\right)} \left(\hat {\eta} _ {\pi_ {\boldsymbol {\theta} _ {k}}} \left(s, a ^ {i _ {h}}\right) - \nu_ {j} \hat {A} _ {j, \pi_ {\boldsymbol {\theta} _ {k}}} ^ {i _ {h}} \left(s, a ^ {i _ {h}}\right)\right) ] I (s _ {j}), \tag {11}
$$
where  $N$  is the number of samples collected using policy  $\pi_{\theta_k}$ . The estimates of the advantages functions for the returns and costs, denoted as  $\hat{\eta}_{\pi_{\theta_k}}$  and  $\hat{A}_{j,\pi_{\theta_k}}^{i_h}$ , are obtained from critic networks referring to MACPO. We estimate these advantages using Generalized Advantage Estimator (GAE) [37] and apply stochastic gradient descent using 11. During the training process, our algorithm employs the early stopping criterion to ensure that the updated policy satisfies the trust region constraint. Specifically, this criterion is defined as  $\frac{1}{N}\sum_{j = 1}^{N}D_{KL}(\pi_{\theta}^{i_h}||\pi_{\theta_k}^{i_h})[s_j] > \delta$ . To update the value network, we minimize the Mean Square Error (MSE) between the output and target value. The procedure of our algorithm is presented in the Appendix E.
(c)
Figure 2: Example tasks in Safe MAMuJoCo and Safe MAIG. (a): Safe 2x4-Ant, (b): Safe 2x3-HalfCheetah, (c): Safe 2x3-ManyAgent Ant, (d): ShadowHandOver. In each of these tasks, the body parts of the robots are controlled by different agents. Agents collaborate to manipulate the robot while ensuring that the safety constraints are not violated.
We evaluate the effectiveness of our algorithm on two benchmarks of safe MARL: Safe MAMuJoCo and Safe Multi-Agent Isaac Gym (MAIG). The former is a safety-aware modification of MAMuJoCo [38], where there exist obstacles in the environment. Meanwhile, Safe MAIG is developed on top of Issac Gym [39], a GPU-based platform for robotics tasks. Being an extension of DexterourHands [40], Safe MAIG requires agents to control the robot hands while optimizing both the reward and safety performance. We present some example tasks in Figure 5 and more details about the environments are introduced in the appendix.
While our primary aim is to propose a simpler alternative to MACPO, we also evaluate the performance of MAPPO-Lagrangian (MAPPO-L), which is put forward alongside MACPO [24]. This method adopts Lagrangian multipliers to solve optimization problems in MACPO. Without the requirements of repetitive computation of the Hessian matrix whose size grows quadratically with the dimension of the parameter vector, the Lagrangian method is also first-order and simple to implement. However, unlike MACPO and MAFOCOPS, whether this algorithm satisfies any worst-case constraint guarantees remains unknown. In addition, we compare the results of our algorithm with two standard MARL baseline algorithms, namely MAPPO [41] and HAPPO [18]. Both sets of experiments are carried out using the MACPO codebase and our experiments are conducted on GeForce RTX 3090 GPU. More implementation details can be found in supplementary materials.
5.1 Performance on Safe MAMuJoCo
In this section, we select several experiment scenarios of Safe MAMuJoCo and execute each algorithm for 10 million samples per task. The cost thresholds are determined by taking  $50\%$  of the cost achieved by standard MARL algorithms after 1 million sample runs. To be noted, while most training parameters remain the same as those in the codebase, such as the learning rate and settings of optimizers, we adjusted some certain hyperparameters associated with the cost thresholds to make the algorithms best suit the experiment scenarios. It is noteworthy that the performance of MAPPO-L highly relies on the Lagrangian coefficient, rendering it more sensitive to the hyperparameters. In light of this observation, we adopt distinct hyperparameters for MAPPO-L in different categories of tasks. In contrast, the other two safe MARL algorithms are implemented with uniform hyperparameter settings across all experimental scenarios in this benchmark, indicating their potential efficacy.
Due to page limit, we present partial results of our experiments in Figures 3 and more results can be seen in Appendix G. The experimental results show that all the safety-aware algorithms are able to satisfy the safety constraints while our proposed MAFOCOPS manages to achieve the best overall performance across all tasks. Even when our method achieves similar performance to the other two algorithms in HalfCheetah scenarios, it still exhibits faster learning, demonstrating the advantages of our approach. For MAPPO-Lagrangian, we note that this baseline algorithm always achieves a similar performance as MAPPO, except in HalfCheetah scenarios where the cost threshold is significantly smaller compared to cost achieved by HAPPO and MAPPO. This may be due to that MAPPO-Lagrangian being built upon Lagrangian multiplier combined with standard MARL algorithms, leading to a performance more similar to safety-unaware MARL algorithms. In addition, from our discussion about MAPPO-L algorithm, we can know that it maintains a rather soft safety
(a) ManyAgent Ant 2x3, ManyAgent Ant 3x2, ManyAgent Ant 6x1, Ant 2x4d
(b) Ant 2x4, Ant 4x2, HalfCheetah 2x3, HalfCheetah 3x2
Figure 3: Performance comparisons on tasks of Safe ManyAgent Ant, Ant, and HalfCheetah in terms of cost and reward. The safety constraint values for these presented tasks are set to be 25, 50 and 30, respectively. The solid line shows the median performance across 5 seeds and the shaded areas correspond to the  $25 - 75\%$  percentiles.
awareness, but the other two algorithms reaches safety via hard constraints. Therefore, MACPO and MAFOCOPS possess more promising properties, thus being the focus of our study.
Interestingly, our intuition suggests that a higher number of agents usually leads to increased complexity of the environment and hence worse performance. However, this phenomenon is not always observed. We assume that this is because, in the same type of map, the multi-agent system has to control the same robot, and the number of agents determines how the robot is partitioned. Fewer agents means that each agent has to control more parts of the robot, making the difficulty of the environment non-linearly dependent on the number of agents. Whereas, in scenarios of 6x1 ManyAgent Ant, we still note that as the number of agents increases, there exists a degradation in the performance of MACPO, which can attributed to the complexity of the computation worsening with the growth in the number of agents. As for MAFOCOPS, our first-order method shows stronger resistance to task complexity, showing the advantages of our algorithm when coping with multi-agent problems. What's more, we provide videos of the trained policies of both our algorithm and MACPO in the supplementary materials to intuitively demonstrate the benefits that our approach brings.
5.2 Performance on Safe MAIG
Apart from experiments in Safe MAMuJoCo, we also conduct experiments in the Safe MAIG benchmark. In this part of experiment, the cost thresholds are set as  $25\%$  of the cost obtained by standard MARL algorithms after running for one-tenth of the entire training process. The total training settings are the same as those in Safe MAMuJoCo, while some specific hyperparameter values are adjusted considering the great difference between these two experiment environments.
Experiment results in this benchmark are presented in Figure 4. Under this complicated environment,
Figure 4: Performance comparisons on tasks of Safe MAIG. The safety constraint value for ShadowHandOver is set to be 15 and that for another task is set to be 110. The solid line shows the median performance across 5 seeds and the shaded areas correspond to the  $25 - 75\%$  percentiles.
it can be observed that the soft constraint algorithm, MAPPO-L, exhibits a delay in guaranteeing safety. Notably, in ShadowHandReOrientation task, it experiences a sudden drop in cost, which significantly impacts the reward. Conversely, although similar phenomenon occurs with MAFOCOPS, our method manages to optimize the reward in this condition and attains an acceptable performance. Although the reward of MACPO remains relatively stable, its final performance is not as favorable as our method. The different performance between MAPPO-L and the other two algorithms indicates that the property of upper bound violation is important in safe reinforcement learning, which also demonstrates the significance of our work.
5.3 Efficiency Analysis between MAFOCOPS and MACPO
As a first-order method, our MAFOCOPS not only eliminates the approximation errors when solving the optimization problems, leading to better performance, but also significantly reduces the computation cost. In this part, we evaluate the memory cost and frame per second (FPS), which is often adopted to measure the training efficiency of reinforcement learning algorithms, between MAFOCPS and MACPO. For simplicity, we adopt memory monitor tools to track memory utilization after 200000 samples and record the average FPS metric.
Due to the page limit, the results are shown in Appendix H of supplementary materials. From the results, it is evident that our algorithm obtains substantial improvement in computational efficiency and shows the ability to effectively save memory resources, especially when the number of agents grows. Consequently, we can infer that when confronted with tasks with multiple agents, second-order algorithms may be ill-suited due to their substantial computational costs. In contrast, our proposed method can successfully address such scenarios without succumbing to the computational burden.
Based on the description provided in Section 4, it is apparent that the Lagrange multipliers,  $\lambda_{j}$  and  $\nu_{max}$ , are crucial to the performance of our approach. In addition, analyzing the sensitivity of our algorithm to changes in the safety bound is also meaningful. Hence, we seek to investigate the sensitivity of our algorithm to these hyperparameters as well as the safety bound. In this way, we conduct some ablation studies using some scenarios of Safe MAMuJoCo and more details are presented in Appendix I.
Considering the intricacy of multi-agent environments, it is difficult to delineate the correlation between the performance of our method and the hyperparameters  $\lambda_{j}$  and  $\nu_{max}$ . Nonetheless, choosing hyperparameters in proximity to the optimal values that we employed typically leads to favorable outcomes. Moreover, our approach's effectiveness is relatively insensitive to variations in these hyperparameter values. As an illustration, even setting  $\nu_{max} = \infty$  does not significantly affect the reward achieved by our method, only resulting in an average  $6.7\%$  degradation. This finding indicates the robustness of our method, and further underscores its superiority over other approaches. What's more, from the sensitivity of our method to safety bound, we learn that although the reward performance of MAFOCOPS decreases with the stricter safety constraints, the algorithm's overall effectiveness remains unchanged across different safety levels.
In this paper, we introduce a novel method for training multi-agent systems while incorporating safety constraints. Building upon the problem formulation and sequential update scheme established in the
MACPO framework [24], our algorithm offers rigorous theoretical guarantees and relies exclusively on first-order optimization techniques, thereby ensuring simplicity in its implementation. Different from MACPO, which relies on second-order optimization to achieve parameterized policies, our work proposes a fundamentally different way to tackle the optimization problem, complemented by the comprehensive provision of derivatives. The empirical experiment results demonstrate the outstanding performance and computation efficiency of our algorithm, compared to the more intricate second-order method. To sum up, our paper offers a novel and distinct contribution to the realm of safe multi-agent reinforcement learning, as evidenced by the distinctions in methodology, proofs, and experimental results in comparison with previous work.
Because the benchmarks we adopt only return one cost for agents, the performance of our algorithm across multiple costs is not evaluated. In the future, we plan to test our approach in more environments and physical settings. Moreover, there exist a number of promising prospects for future research such as incorporating off-policy data to further enhance the training efficiency. Designing methods aimed at offline settings, which precludes interactions with environments, represents another valuable direction for study.
This work is supported by National Key R&D Program of China under Contract 2022ZD0119802 as well as National Natural Science Foundation of China under Contract 61836011. What's more, it is funded by Collective Intelligence & Collaboration Laboratory (Open Fund Project No. QXZ23014101) and by Young Elite Scientists Sponsorship Program by CAST 2022QNRC001. Besides, this work is also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
[1] L. Busoniu, R. Babuska, and B. De Schutter, "A comprehensive survey of multiagent reinforcement learning." IEEE Transactions on Systems, Man, and Cybernetics, vol. 38, no. 2, pp. 156-172, 2008.
[2] K. Tuyls and G. Weiss, “Multiagent learning: Basics, challenges, and prospects.” AI Magazine, vol. 33, no. 3, pp. 41–41, 2012.
[3] A. O. Al-Abbasi, A. Ghosh, and V. Aggarwal, "Deeppool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning," IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 12, pp. 4714-4727, 2019.
[4] Y. Cao, W. Yu, W. Ren, and G. Chen, "An overview of recent progress in the study of distributed multi-agent coordination," IEEE Transactions on Industrial Informatics, vol. 9, no. 1, pp. 427-438, 2012.
[5] T. Wu, P. Zhou, K. Liu, Y. Yuan, X. Wang, H. Huang, and D. O. Wu, "Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks," IEEE Transactions on Vehicular Technology, vol. 69, no. 8, pp. 8243-8256, 2020.
[6] K. Zhang, Z. Yang, and T. Basar, "Decentralized multi-agent reinforcement learning with networked agents: recent advances," Frontiers of Information Technology & Electronic Engineering, vol. 22, no. 6, pp. 802-814, 2021.
[7] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., "Mastering the game of go with deep neural networks and tree search," Nature, vol. 529, no. 7587, pp. 484-489, 2016.
[8] Y. Yang, L. Yu, Y. Bai, J. Wang, W. Zhang, Y. Wen, and Y. Yu, “A study of ai population dynamics with million-agent reinforcement learning,” arXiv preprint arXiv:1709.04511, 2017.
[9] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Jozefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever,
[28] S. Lu, K. Zhang, T. Chen, T. Başar, and L. Horesh, "Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning," in AAAI Conference on Artificial Intelligence, vol. 35, no. 10, 2021, pp. 8767-8775.
[29] U. Borrmann, L. Wang, A. D. Ames, and M. Egerstedt, "Control barrier certificates for safe swarm behavior," IFAC-PapersOnLine, vol. 48, no. 27, pp. 68-73, 2015.
[30] Z. Qin, K. Zhang, Y. Chen, J. Chen, and C. Fan, "Learning safe multi-agent control with decentralized neural barrier certificates," arXiv preprint arXiv:2101.05436, 2021.
[31] W. Zhang, O. Bastani, and V. Kumar, “Mamps: Safe multi-agent reinforcement learning via model predictive shielding,” arXiv preprint arXiv:1910.12639, 2019.
[32] J. L. Nazareth, “Conjugate gradient method,” Wiley Interdisciplinary Reviews: Computational Statistics, vol. 1, no. 3, pp. 348–353, 2009.
[33] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey et al., “Maximum entropy inverse reinforcement learning.” in Aaii, vol. 8. Chicago, IL, USA, 2008, pp. 1433–1438.
[34] J. Schulman, X. Chen, and P. Abbeel, "Equivalence between policy gradients and soft q-learning," arXiv preprint arXiv:1704.06440, 2017.
[35] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor," in International Conference on Machine Learning, 2018, pp. 1861-1870.
[36] D. P. Bertsekas, Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
[37] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438, 2015.
[38] C. S. de Witt, B. Peng, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson, “Deep multiagent reinforcement learning for decentralized continuous cooperative control,” arXiv preprint arXiv:2003.06709, vol. 19, 2020.
[39] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa et al., "Isaac gym: High performancegpu-based physics simulation for robot learning," arXiv preprint arXiv:2108.10470, 2021.
[40] Y. Chen, T. Wu, S. Wang, X. Feng, J. Jiang, Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang, "Towards human-level bimanual dexterous manipulation with reinforcement learning," Advances in Neural Information Processing Systems, vol. 35, pp. 5150-5163, 2022.
[41] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, "The surprising effectiveness of ppo in cooperative multi-agent games," Advances in Neural Information Processing Systems, vol. 35, pp. 24611-24624, 2022.
[42] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International conference on machine learning, 2015, pp. 1889–1897.
[43] I. Csiszár and J. Körner, Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
[44] S. Boyd, S. P. Boyd, and L. Vandenberghe, Convex optimization. Cambridge university press, 2004.
A Proof of the Optimization problem
According to the problem formulation, we give a definition of the "surrogate" cost, which aligns with what is employed in MACPO [24]:
Definition. Let  $\pi$  be a joint policy, and  $\bar{\pi}^i$  be some other policy of agent  $i$ . Then for any of its costs of index  $j \in 1, \dots, m^i$ , we define
$$
L _ {j, \pi} ^ {i} (\bar {\pi} ^ {i}) = E _ {s \sim \rho_ {\pi}, a ^ {i} \sim \bar {\pi} ^ {i}} [ A _ {j, \pi} ^ {i} (s, a ^ {i}) ].
$$
In this way, consider  $\pi$  and  $\bar{\pi}$  be joint policies,  $i\in \mathcal{N}$  be an agent and  $j\in 1,\dots ,m^i$  be an index of one of its costs. From the proof of Theorem 1 in TRPO [42], (in particular, equations (41)  $\sim$  (45)), applying it to joint policies  $\pi$  and  $\bar{\pi}$ , we can conclude that
$$
J _ {j} ^ {i} (\bar {\pi}) \leq J _ {j} ^ {i} (\pi) + E _ {s \sim \rho_ {\pi}, \boldsymbol {a} \sim \bar {\pi}} \left[ A _ {j, \pi} ^ {i} \left(s, a ^ {i}\right) \right] + \frac {4 \alpha^ {2} \gamma m a x _ {s , a ^ {i}} \left| A _ {j . \pi} ^ {i} \left(s , a ^ {i}\right) \right|}{(1 - \gamma) ^ {2}}, \tag {12}
$$
where  $\alpha = D_{TV}^{max}(\pmb{\pi},\bar{\pmb{\pi}}) = max_sD_{TV}(\pmb{\pi}(\cdot |s),\bar{\pmb{\pi}} (\cdot |s))$ . According to the definition of total variance divergence, defined by  $D_{TV}(p||q) = \frac{1}{2}\sum_{i}|p_{i} - q_{i}|$ , we can know that  $D_{TV}(p||q) = D_{TV}(q||p)$ . Using Pinsker's inequality  $D_{TV}(p||q)^{2}\leq \frac{D_{KL}(p||q)}{2}$  [43], we can change the order of policy in the divergence computation and obtain: