SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 3.6. For any  $\pi_{\mathcal{M}}(a_t|s_t)$  with supp  $\pi_{\mathcal{M}}(a_t|s_t) \subset \hat{\pi}_{\beta}, \forall s_t, a_t \in D$ , the  $Q$ -value function  $Q^r$  via iterating Eq. (12) satisfies:
$$
\hat {Q} ^ {r} \left(s _ {t}, a _ {t}\right) \leq Q ^ {r} \left(s _ {t}, a _ {t}\right) - \kappa \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {\pi_ {\mathcal {M}}}{\hat {\pi} _ {\beta}} \right] \left(s _ {t}, a _ {t}\right) + \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} |}} \right] \left(s _ {t}, a _ {t}\right), \tag {15}
$$
Thus, if  $\kappa \geq \frac{C_{r,p,\delta}R_{max}}{(1 - \gamma)\sqrt{|\mathcal{D}(s_t,a_t)|}}\left[\frac{\pi_{\mathcal{M}}(a_t|s_t)}{\hat{\pi}_{\beta}(a_t|s_t)}\right]^{-1}$ , the iterative update Eq. (12) guarantees  $\hat{Q}^r \leq Q^r$ .
Next, when  $\pi_{\mathcal{M}}(a_t|s_t) = \pi_\theta (a_t|s_t)$ , we obtain a not lower-bound for the Q-values estimates pointwise. based on Eq. (13). We abuse the notation  $\frac{1}{\sqrt{|\mathcal{D}|}}$  to represent a vector of the inverse square root of state counts. Proofs and discussions are in Appendix B.4.
Proposition 3.7. When  $\pi_{\mathcal{M}}(a_t|s_t) = \pi_\theta (a_t|s_t)$ , according to Eq. (13), we obtain a lower bound for the true value of  $V^{r} = \mathbb{E}_{a_{t}\sim \pi_{\theta}(a_{t}|s_{t})}[Q^{r}(s_{t},a_{t})]$  that satisfies the following inequality:
$$
\hat {V} ^ {r} \left(s _ {t}\right) \leq V ^ {r} \left(s _ {t}\right) - \kappa \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \mathbb {E} _ {a _ {t} \sim \pi_ {\theta}} \left[ \frac {\pi_ {\theta}}{\hat {\pi} _ {\beta}} - 1 \right] \right] \left(s _ {t}\right) + \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} R _ {\text {m a x}}}{(1 - \gamma) \sqrt {| \mathcal {D} |}} \right] \left(s _ {t}\right), \tag {16}
$$
Thus, if  $\kappa \geq \frac{C_{r,p,\delta}R_{max}}{(1 - \gamma)\sqrt{|\mathcal{D}(s_t)|}}\left[\mathbb{E}_{a_t\sim \pi_\theta}\left[\frac{\pi_\theta(a_t|s_t)}{\pi_\beta(a_t|s_t)} -1\right]\right]^{-1}$ , the Eq. (13) can guaranteees  $\hat{V}^r\leq V^r$
On the other hand, taking into account the need to satisfy the safety constraint  $\mathbb{E}_{\rho (s_0)}\Big[\sum_aq(at|s_t)Q^{c_i}(s_t,a_t)\Big]\leq \bar{c}_i$ , the estimation of the cost Q-value must fulfill  $\hat{Q}^{c_i}\geq Q^{c_i}$ . Based on the above analysis we choose the penalty that maximizes the expected Q-value under special action-state transitions for unknown actions produced by  $\pi_{\mathcal{R}}(a|s)$ . Therefore, we define the Q-value of the iterative reward function as:
$$
\hat {Q} _ {k + 1} ^ {c _ {i}} \leftarrow \arg \min  \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right)\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \right] ^ {2} \right] - \chi_ {\substack {a \sim \pi_ {\mathcal {R}} (\cdot | s _ {t})}} \mathbb {E} _ {\left. s \sim D \atop a \sim \pi_ {\mathcal {R}} (\cdot | s _ {t})\right)} \left[ Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \right], \tag{17}
$$
where  $\chi$  is the trade-off factor. The  $\pi_{\mathcal{R}}$  denotes the marginal distribution corresponding to the unknown action. The Proposition 3.8 provides an upper bound on the convergence of the fixed point of  $\hat{Q}^{c_i}$ , and clarifies the conditions for  $\hat{Q}^{c_i}$  to converge to the upper bound of  $Q^{c_i}$ . Proofs and discussions are in Appendix B.5.
Proposition 3.8. For any  $\pi_{\mathcal{R}}(a_t|s_t)$  with supp  $\pi_{\mathcal{R}}(a_t|s_t) \subset \hat{\pi}_{\beta}, \forall s_t, a_t \in D$ , the  $Q$ -value function  $Q^{c_i}$  via iterating Eq. (17) satisfies:
$$
\hat {Q} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \geq Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) + \chi \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {\pi_ {\mathcal {R}}}{\hat {\pi} _ {\beta}} \right] \left(s _ {t}, a _ {t}\right) - \left[ \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \frac {C _ {r , p , \delta} C _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} |}} \right] \left(s _ {t}, a _ {t}\right). \tag {18}
$$
Thus, if  $\chi \geq \frac{C_{r,p,\delta} C_{max}}{(1 - \gamma) \sqrt{|\mathcal{D}(s_t, a_t)|}} \left[ \frac{\pi_{\mathcal{R}}(a_t | s_t)}{\hat{\pi}_{\beta}(a_t | s_t)} \right]^{-1}$ , the iterative update Eq. (17) can guarantee  $\hat{Q}^{c_i} \geq Q^{c_i}$ .
3.2.2 Parametered Policy Update
After solving the optimal variational distribution of each state via Eq. (10), we need to obtain the policy parameters  $\theta$ . According to the solution target of Eq. (6), the optimization target can be obtained by eliminating the quantities irrelevant to  $\theta$ .
$$
\mathcal {L} (\theta) = \max  \mathbb {E} _ {\tau \sim q} \left[ - \alpha D _ {K L} \left(q (\cdot | s _ {t}) \| \pi_ {\theta} (\cdot | s _ {t})\right) \right] = \max  \alpha \mathbb {E} _ {\rho (s _ {0})} \mathbb {E} _ {q \left(a _ {t} \mid s _ {t}\right)} \left[ \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) - \log q \left(a _ {t} \mid s _ {t}\right) \right], \tag {19}
$$
where  $\alpha \geq 0$  is the temperature parameters. In addition, the  $q(a_{t}|s_{t})$  is independent of  $\pi_{\theta}(a_t|s_t)$ . Therefore, the optimization objective of the above Eq. (19) can be rewritten as:
$$
\mathcal {L} (\theta) = \max  \mathbb {E} _ {\rho (s _ {0})} \mathbb {E} _ {q \left(a _ {t} \mid s _ {t}\right)} \left[ \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \right]. \tag {20}
$$
In this section, we compare VOCE to previous offline safe RL methods in a range of domains and dataset compositions, including different action spaces and observation dimensions. To the best of my knowledge, there is currently no standardized dataset available in the field of offline safe RL. To facilitate further research and reproducibility of this work, we have collected a series of datasets using a trained online policy. The parameter settings of the dataset are in Appendix C.1.
4.1 Task and Baseline
Task. We choose Point-Goal, Car-Goal, Point-Button and Car-Button four tasks widely adopted in the field of safe RL [26, 29, 32-34], as the experimental tasks for this work. A detailed description of each task is in Appendix C.2.
Figure 2: Distribution of rewards and costs for samples collected by different behavioral policies.
Baselines. BCQ-Lag is an offline safe RL algorithm that combines the Lagrange multiplier method with the BCQ [24] algorithm and employs adaptive penalty coefficients to implement offline constraints task. The C-CRR is an offline safe RL algorithm improved by the CRR [35] algorithm. It implements cost constraints by introducing cost evaluation Q-value function and Lagrange multipliers. Coptidice [10] is a policy optimization method based on the optimal stationary distribution space. The aforementioned three approaches are currently state-of-the-art algorithms in offline safe RL.
4.2 Performance Comparison Experiment
To evaluate the performance of VOCE on diverse task and behavior samples, we collected sample data from three distinct behaviors within four different tasks. We introduce parameter  $\varphi$  to represent the proportion of trajectories in the sample data that satisfy the constraints. Then we employ  $\varphi$  to characterize samples to different behaviors. Fig. 2 illustrates the marginal distribution of rewards and costs for sample trajectories in the dataset at different values of  $\varphi$ . The results from Fig. 2 reveal that as the value  $\varphi$  increases, the mean cost decreases, and the mean reward increases. Fig. 3 displays the reward and cost curves of VOCE and the state-of-the-art offline safe RL methods under different  $\varphi$  values for four tasks. The Fig. 3 results demonstrate that the VOCE achieves higher rewards across all tasks compared to the other three methods, while also satisfying or approaching the safety constraints threshold. Especially in the Goal task, VOCE consistently meets the safety constraints across different  $\varphi$  values, while achieving higher reward returns. In the Button task, when the parameter  $\varphi$  is small, VOCE struggles to ensure safety; however, the cost curve of VOCE remains lower than the other three methods. Based on the aforementioned results analysis, it can be concluded that the VOCE exhibits competitive performance across various combinations of samples from multiple tasks, particularly excelling in terms of safety compared to the state-of-the-art algorithms currently available.
4.3 Ablation Experiment
The parameter  $\varphi$  of the dataset. Fig. 4 displays the boxplots of rewards and costs for the VOCE under different parameter values of  $\varphi$  in the Point-Goal and Point-Button tasks. The results from
Figure 3: The reward and cost curves of VOCE and baseline algorithms with different sample data under 4 tasks. The curve is averaged over 3 random seeds, where the solid line is the mean and the shaded area is the standard deviation.
Fig. 4 reveals an intriguing phenomenon: VOCE does not achieve the highest reward and lowest cost at  $\varphi = 1.0$  but instead attains the highest reward within the range of 0.7 to 0.9 (The range will vary depending on changes in the sampling method or policy.). This indicates that appropriately increasing the number of constraint-satisfying trajectories in the dataset benefits VOCE in improving its rewards and reducing costs. However, excessive augmentation of constraint-satisfying trajectories in the dataset may lead to a decrease in rewards and even an increase in costs.
Figure 4: Ablation experiments with the same sampling policies and different  $\varphi$  coefficients.
Conservative estimation of rewards and costs. To assess the impact of conservative estimation on VOCE, we conducted ablation experiments on the conservative estimation of both reward and cost Q-values. Fig. 5 illustrates the rewards and costs of VOCE, VOCE-Qr, and VOCE-Qc across four tasks. VOCE-Qr stands for the VOCE method with the conservative estimation of Q-values removed for rewards. VOCE-Qc represents the VOCE method with the conservative estimation of Q-values removed for costs. The results presented in Fig. 5 demonstrate that the rewards of VOCE-Qr are notably lower than those of VOCE. This indicates that employing a lower-bound conservative estimation of reward Q-values helps eliminate extrapolation errors caused by OOD actions, thereby significantly improving the reward of VOCE. Furthermore, the results from Fig. 5 reveal that the
rewards of VOCE-Qc are comparable to or even surpass those of VOCE. However, in some tasks, the costs of VOCE-Qc exceed the cost constraints. This suggests that utilizing an upper-bound conservative estimation of cost Q-values helps reduce the costs of VOCE, thereby enhancing the safety of VOCE.
Figure 5: Ablation study for conservative Q-value estimation of reward and cost, and constrained regularize of the policy update. The curve is averaged over 3 random seeds, where the solid line is the mean and the shaded area is the standard deviation.
In this section, we elaborate on the work pertaining to offline safe RL and scrutinize three aspects of safe RL, offline RL, and offline safe RL.
Safe RL. Currently, Safe RL typically addresses optimization objectives with constraints using the primal-dual framework. The Lagrange version of PPO and TRPO [36, 37] widely used as the underlying baselines combines the constraint information with the original objective function via Lagrange multipliers. CPO [38] is the first general-purpose policy search algorithm for safe RL that guarantees that the near constraints are satisfied at each iteration. However, the second-order method [38, 39] requires second-order information and thus brings a higher computational burden. To address the approximation errors associated with Taylor approximations and inverting a high-dimensional Fisher information matrix, first-order methods such as CUP [40] and APPO [41] achieve better performance and implementation.
Offline RL. Offline RL, also known as batch RL, considers the problem of updating a policy from an offline dataset without interacting with the environment. There are mainly two approaches of policy constraints and Q-value regularization to solve the problem of OOD actions in offline RL [42-44]. The earliest proposed Batch-Constrained deep Q-learning (BCQ) algorithm [24] is a typical policy-constrained offline RL algorithm, which employs CVAE to learn behavioral policy generation models, and finds the optimal policy by maximizing the Q-value. On this basis, a series of policy-constrained algorithms [45-48] are proposed, which mainly constrain the actions explored through the behavioral policy model. In addition, due to the obvious stability of the Q-value regularization approach, it has been extensively studied. A conservative Q-learning algorithm [30] proposed by Kumar et al., learns a conservative Q-value by enhancing the standard Bellman error objective with a simple Q-value regularizer. Subsequently, a series of Q-value regularization methods [49, 6, 50] were proposed which generally learn a conservative Q-value through regularization items or clipping Q-learning.
Offline safe RL. Offline safe RL is to solve the optimization problem of safety policy under the offline dataset. An offline safe RL algorithm with constraint penalty is proposed [16], which improves the objective function of the cost Q-value via using the constraint penalty item, and ensures the safety of the policy via terminating unsafe trajectory in advance. On the other hand, a constrained offline optimization algorithm (COPO) [11] defines the RL problem based on linear programming and constrains the distance between the final policy and offline sample behavior through regulation terms to solve the problem of offline safe RL algorithms. Additionally, this algorithm sets the discount factor  $\gamma = 1$ , and changes the maximizing discount reward objective to the maximizing mean reward objective. Similar to the COPO algorithm, an offline policy optimization via stationary distribution correction estimation (CoptiDICE) [18] also utilizes the linear programming method of RL to solve
the optimal stationary distribution instead of the gradient strategy, which expands the discount factor to  $\gamma \in (0,1]$ .
In this work, we propose a variational optimization with conservative estimation for offline safe RL algorithm to address the problem of optimizing safe policies using offline datasets. We introduce probabilistic inference to reframe the problem of offline safe RL, and we mitigate estimation errors caused by parameter approximation through employing nonparametric variational distributions instead of parameterized policies. Furthermore, we utilize the upper and lower bounds of Q-values to estimate the Q-values of cost and reward, thereby mitigating extrapolation errors caused by OOD actions. Finally, extensive comparisons and ablation experiments demonstrate that the VOCE algorithm outperforms state-of-the-art algorithms in terms of safety.
This work is supported by the National Natural Science Foundation of China (No. 62372329), in part by the National Key Research and Development Program of China (No. 2021YFB2501104), in part by Shanghai Rising Star Program (No.21QC1400900), in part by Tongji-Qomolo Autonomous Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program. We thank Li Shen and Yiqin Yang for the insightful discussion.
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[2] Ruiqi Zhang, Jing Hou, Guang Chen, Zhijun Li, Jianxiao Chen, and Alois Knoll. Residual policy learning facilitates efficient model-free autonomous racing. IEEE Robotics and Automation Letters, 7(4):11625-11632, 2022. doi: 10.1109/LRA.2022.3192770.
[3] Jing Hou, Guang Chen, Zhijun Li, Wei He, Shangding Gu, Alois Knoll, and Changjun Jiang. Hybrid residual multiexpert reinforcement learning for spatial scheduling of high-density parking lots. IEEE Transactions on Cybernetics, pages 1-13, 2023. doi: 10.1109/TCYB.2023.3312647.
[4] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-586, 2021.
[5] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline r1? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[6] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems, 34, 2021.
[7] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannon, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
[8] Yunxiao Shan, Boli Zheng, Longsheng Chen, Long Chen, and De Chen. A reinforcement learning-based adaptive path tracking approach for autonomous driving. IEEE Transactions on Vehicular Technology, 69(10):10581-10595, 2020.
[9] Xiaobai Ma, Jiachen Li, Mykel J Kochenderfer, David Isele, and Kikuo Fujimura. Reinforcement learning for autonomous driving with latent state inference and spatial-temporal relationships. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6064-6071. IEEE, 2021.
[43] Jacob Buckman, Carles Gelada, and Marc G Bellemare. The importance of pessimism in fixed-dataset policy optimization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=E3Ys6a1NTGT.
[44] Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Generalized off-policy actor-critic. Advances in Neural Information Processing Systems, 32, 2019.
[45] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2020.
[46] Hua Wei, Deheng Ye, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang, and Zhenhui Li. Boosting offline reinforcement learning with residual generative modeling. In International Joint Conference on Artificial Intelligence, 2021.
[47] Jiayi Guan, Shangding Gu, Zhijun Li, Jing Hou, Yiqin Yang, Guang Chen, and Changjun Jiang. Uac: Offline reinforcement learning with uncertain action constraint. IEEE Transactions on Cognitive and Developmental Systems, 2023.
[48] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdelmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rke7geHtwH.
[49] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
[50] Hongchang Zhang, Jianzhu Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xi-angyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, 2022.
The VOCE algorithm is shown in Algorithm 1.
Algorithm 1: VOCE Algorithm
Input: Dataset  $\mathcal{D} = \{(s_t,a_t,r_t,s_{t + 1})_i^n\}$    
Output: Policy network  $\pi_{\theta}$    
Q-value network of reward  $Q^{r},Q_{t}^{r}$  .   
Q-value network of cost  $Q^{c_i},Q^{c_i}_i$  .   
for each batch do Sample a batch of transitions  $(s_t,a_t,r_t,s_{t + 1})$  from the buffer  $\mathcal{D}$  .   
Compute the Q-value  $\hat{Q}^r (s_t,a_t)$  and  $\hat{Q}^{c_i}(s_t,a_t)$  via Bellman backup;   
 $\hat{B}^{\pi}\hat{Q}^{r}(s_{t},a_{t}) = r(s_{t},a_{t}) + \gamma \hat{Q}^{r}(s_{t + 1},a_{t + 1})$ $\hat{B}^{\pi}\hat{Q}^{c_i}(s_t,a_t) = c_i(s_t,a_t) + \gamma \hat{Q}^{c_i}(s_{t + 1},a_{t + 1})$  Update the  $Q^r$  via the Eq. (13);   
Update the  $Q^{c_i}$  via the Eq. (17);   
Update the optimal dual variables  $\eta$  and  $\lambda$  via solving the convex optimization issue in the Eq. (11);   
Compute the optimal variational distribution  $q^{*}(a_{t}|s_{t})$  by the Eq. (10) via  $Q^r,Q^{c_i}$  and Lagrangian multipliers  $\eta$  and  $\lambda$  .   
Update the Policy Parameter  $\pi_{\theta}$  via the Eq. (20);   
 $Q_{t}^{r}\gets \tau Q^{r} + (1 - \tau)Q_{t}^{r};$ $Q_{t}^{c_i}\gets \tau Q^{c_i} + (1 - \tau)Q_{t}^{c_i};$
B Proofs and Discussions
B.1 Proof and Discussion the Proposition 3.4
Proposition B.1. The closed-form solution of the optimal variational distribution that satisfies the safety constraint in Eq. (21) is given as:
$$
q ^ {*} \left(a _ {t} \mid s _ {t}\right) = \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta} \right] \exp \left(- \frac {\eta + \delta}{\eta}\right), \tag {21}
$$
Proof. The unconstrained problem derived through the Lagrange multiplier method is expressed as follows:
$$
\begin{array}{l} \mathcal {L} (q, \lambda , \eta , \delta) = \max  _ {q \geq 0} \min  _ {\lambda \geq 0, \eta \geq 0, \delta} \mathbb {E} _ {\rho (s _ {0})} \left[ \sum_ {a _ {t}} q \left(a _ {t} \mid s _ {t}\right) Q ^ {r} \left(s _ {t}, a _ {t}\right) - \lambda^ {T} \left[ \sum_ {a _ {t}} Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) - \bar {c} _ {i} \right] \right. \tag {22} \\ - \eta \Big [ D _ {K L} \big (q (a _ {t} | s _ {t}) | | \pi_ {\theta} (a _ {t} | s _ {t}) \big) - \epsilon \Big ] - \delta \Big [ \sum_ {a _ {t}} q (a _ {t} | s _ {t}) - 1 \Big ]. \\ \end{array}
$$
To simplify Eq. (22), we introduce a composite  $\mathbf{Q}$ -value and define it as  $Q^{rc}(s_t, a_t) \triangleq Q^r(s_t, a_t) - \lambda^T Q^{c_i}(s_t, a_t)$ . Then we rewrite the Eq. (22) as:
$$
\begin{array}{l} \mathcal {L} (q, \lambda , \eta , \delta) = \max _ {q \geq 0} \min _ {\lambda \geq 0, \eta \geq 0, \delta} \mathbb {E} _ {\rho (s _ {0})} \bigg [ \sum_ {a _ {t}} q (a _ {t} | s _ {t}) Q ^ {r c} (s _ {t}, a _ {t}) - \eta \sum_ {a _ {t}} q (a _ {t} | s _ {t}) \log \frac {q (a _ {t} | s _ {t})}{\pi_ {\theta} (a _ {t} | s _ {t})} \\ \left. - \delta \sum_ {a _ {t}} q \left(a _ {t} \mid s _ {t}\right) + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon + \delta \right]. \tag {23} \\ \end{array}
$$
For a fixed  $\lambda$  and  $\eta$ , we write the dual problem [17, 18] of the above optimization problem of maximization  $\max_{q\geq 0}\mathcal{L}(q,\lambda ,\eta)$ :
$$
\max  _ {q \geq 0} \min  _ {u \geq 0} \mathcal {L} (q, \lambda , \eta) + \sum_ {s _ {t}, a _ {t}} \mu \left(s _ {t}, a _ {t}\right) q \left(a _ {t} \mid s _ {t}\right). \tag {24}
$$
Since the strong duality holds, satisfying the KKT condition is a necessary and sufficient condition for the solution of the primal and dual problems.
Condition 1 (Primal feasibility).  $q^{*}(a_{t}|s_{t}) \geq 0, \forall s_{t}, a_{t}$
Condition 2 (Dual feasibility).  $\mu^{*}(s_{t},a_{t})\geq 0,\forall s_{t},a_{t}$
Condition 3 (Stationarity).  $Q^{r}(s_{t},a_{t}) - \lambda^{T}Q^{c_{i}}(s_{t},a_{t}) - \eta -\eta \log q^{*}(a_{t}|s_{t}) - \eta \log \pi_{\theta}(a_{t}|s_{t}) + \mu^{*}(s_{t},a_{t}) = 0,\forall s_{t},a_{t}$
Condition 4 (Complementary).  $q^{*}(a_{t}|s_{t})\mu^{*}(s_{t},a_{t}) = 0, \forall s_{t}, a_{t}$
Form Stationarity we have:
$$
q ^ {*} \left(a _ {t} \mid s _ {t}\right) = \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta} \right] \exp \left(- \frac {\eta + \delta - u ^ {*} \left(s _ {t} , a _ {t}\right)}{\eta}\right). \tag {25}
$$
Since the  $\pi_{\theta}(a_t|s_t) > 0$ , thus the  $q^{*}(a_{t}|s_{t}) > 0$ . Then the Primal feasibility is always satisfied. Furthermore, we only need to consider the cases where  $q^{*}(a_{t}|s_{t}) > 0$ .
(Case 1  $q^{*}(a_{t}|s_{t}) > 0$ ) In this case,  $\mu^{*}(s_{t},a_{t}) = 0$  where Dual feasibility and Complementary slackness are satisfied. In addition, Stationarity also applies to:
$$
q ^ {*} \left(a _ {t} \mid s _ {t}\right) = \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta} \right] \exp \left(- \frac {\eta + \delta}{\eta}\right). \tag {26}
$$