SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \hat {Q} ^ {c _ {i}} (s _ {t}, a _ {t}) \geq \mathcal {B} ^ {\pi} \hat {Q} ^ {c _ {i}} (s _ {t}, a _ {t}) - \frac {C _ {r , p , \delta} C _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}} + \chi \frac {\pi_ {\mathcal {R}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} \\ \geq c + \gamma P ^ {\pi} \hat {Q} ^ {c _ {i}} (s _ {t}, a _ {t}) - \frac {C _ {r , p , \delta} C _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}} + \chi \frac {\pi_ {\mathcal {R}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} \\ \geq Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) - \gamma P ^ {\pi} Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) + \gamma P ^ {\pi} \hat {Q} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) - \frac {C _ {r , p , \delta} C _ {m a x}}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} + \chi \frac {\pi_ {\mathcal {R}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)}, \tag {60} \\ \end{array}
$$
$$
(I - \gamma P ^ {\pi}) \hat {Q} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \geq (I - \gamma P ^ {\pi}) Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) - \frac {C _ {r , p , \delta} C _ {m a x}}{(1 - \gamma) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} + \chi \frac {\pi_ {\mathcal {R}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)}, \tag {61}
$$
$$
\begin{array}{l} \hat {Q} ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \geq Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) - \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \left[ \frac {C _ {r , p , \delta} C _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}} + \chi \frac {\pi_ {\mathcal {R}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} \right] \tag {62} \\ \geq Q ^ {c _ {i}} (s _ {t}, a _ {t}) + \chi (I - \gamma P ^ {\pi}) ^ {- 1} \frac {\pi_ {\mathcal {R}} (a _ {t} | s _ {t})}{\hat {\pi} _ {\beta} (a _ {t} | s _ {t})} - (I - \gamma P ^ {\pi}) ^ {- 1} \frac {C _ {r , p , \delta} C _ {m a x}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}}. \\ \end{array}
$$
We guarantee the upper bound of  $\hat{Q}^{c_i}$  via choosing an appropriate value of  $\chi$  based on the Eq. (62). Beside the  $(I - \gamma P^{\pi})^{-1}$  is a matrix of non-negative entries.
$$
\chi \geq \frac {C _ {r , p , \delta} C _ {\max}}{(1 - \gamma) \sqrt {| \mathcal {D} (s _ {t} , a _ {t}) |}} \left[ \frac {\pi_ {\mathcal {R}} \left(a _ {t} \mid s _ {t}\right)}{\hat {\pi} _ {\beta} \left(a _ {t} \mid s _ {t}\right)} \right] ^ {- 1}. \tag {63}
$$
As shown in Fig. 6, we utilize the online policy CVPO to collect sample data with different  $\varphi$  values during three distinct stages of the training process. The variable  $\varphi$  represents the proportion of safe trajectories in the collected samples compared to the total number of sample trajectories. Behavior 1 corresponds to  $\varphi = 0.4$ , behavior 2 corresponds to  $\varphi = 0.6$ , and so on. To ensure that the collected sample data satisfies the designated  $\varphi$  value, we incorporate an additional trajectory filtering policy into the data collection procedure. Furthermore, the total number of steps for each category of samples collected is  $1e7$ .
Figure 6: Collect datasets with different values  $\varphi$  in the Point-Button task.
C.2 Experimental Task Details
The Point-Goal and Car-Goal task. The objective of the agent is to reach a goal while avoiding surrounding obstacles. Once the agent reaches the correct goal, the environment randomly generates a new goal. The agent receives a reward when it approaches or reaches the goal. There are hazards and vases as obstacles in the environment, and the agent incurs a cost when encountering them. Hazards are immovable, while vases can be moved. The observation state is the relative pose between the agent and obstacles and goals, represented by a point-based pseudo-LIDAR. We select two tasks, namely Point-Goal and Car-Goal tasks [26, 29]. These tasks are performed by the Point and Car agents, respectively. The Point agent is a robot constrained to a 2D plane, capable of rotation as well as forward and backward movement. The Car agent is a slightly more complex robot with two independent parallel wheels and a freely rolling rear wheel.
Table 1: The hyper-parameters of the VOCE algorithm model. The variables  $s$  and  $a$  represent the dimensions of the state and action, respectively.
Categories	Hyper-parameters	Value
Policy network	Sizes	s × 256 × 256 × a
Activation function	ReLU
Number of networks	2
Learning rate	2.0e-3
Optimizer	Adam
Q-value network of reward	Sizes	(s+a) × 256 × 256 × 1
Activation function	ReLU
Number of networks	4
Learning rate	1.0e-3
Optimizer	Adam
Q-value network of cost	Sizes	(s+a) × 256 × 256 × 1
Activation function	ReLU
Number of networks	4
Learning rate	1.0e-3
Optimizer	Adam
Others	Batch size	600
Discount factor γ	0.99
KL threshold ε	0.1
The clip of the dual variable η	[1.0e-6, 1.0e5]
The clip of the dual variable λ	[1.0e-6, 1.0e5]
The clip of the trade-off factor κ	[0, 1.0e6]
The clip of the trade-off factor χ	[0, 1.0e6]
The Point-Button and Car-Button task. The objective of the agent is to navigate around stationary and moving obstacles in the environment and press one of the multiple target buttons. The task is similar to the goal task, where the agent receives a reward for either approaching or pressing the target button. The Button task involves the presence of dynamic obstacles that move rapidly along predefined trajectories. As the agent incurs costs upon colliding with these dynamic obstacles, the Button task is more challenging compared to the Goal task. In the Button task, we select two tasks: the Point-Button task and the Car-Button task [26, 29].
As shown in Table 1, we share the hyperparameters of the CEVO algorithm to facilitate reproducibility. Furthermore, we will share the code on GitHub at a later time.
C.4 Datasets for the Preliminaries Section
In the preliminaries section, we employed a dataset with  $\varphi = 1.0$  to evaluate offline RL methods. Additionally, we utilized a dataset with  $\varphi = 0.8$  to evaluate off-policy safe RL methods. As shown in Fig. 7,  $\varphi = 1.0$  indicates that the dataset includes samples with not only high rewards but also costs within the cost limit. In addition,  $\varphi = 0.8$  indicates that the dataset includes samples with high rewards, while also containing samples that exceed the cost limit.
C.5 Datasets for the Ablation Experiment
Fig 7 presents the datasets collected under the same policy but with different  $\varphi$  values. To ensure the acquisition of datasets that satisfy the  $\varphi$  values, we incorporate a trajectory filtering strategy into the data collection procedure and introduce random actions. Concretely, when the random value is less than the predefined threshold  $\varepsilon$ , we employ random actions instead of the actions provided by the policy.
Figure 7: Collect datasets with the same policy and different values  $\varphi$  in the Point-Button task.
C.6 The Ablation Experiment of the Parameters
We intuitively believe that the value of the parameter  $\epsilon$  in the objective 8 affects the performance of the VOCE algorithm. Therefore, we conduct ablation experiments on this parameter. The results shown in Fig 8 indicate that setting the parameter  $\epsilon$  too small reduces the convergence speed of the policy and even decreases the algorithm's performance. Conversely, when the parameter  $\epsilon$  is set to larger, it may lead to instability in the policy training process. Furthermore, from the illustrated results, we can conclude that within this range of parameter  $\epsilon \in [0,1]$ , it ensures the policy's good convergence speed and stability. Note that the parameter  $\epsilon \in [0,1]$  represents a feasible range rather than an optimal interval.
Figure 8: Reward and cost curves of VOCE in the Point-Goal task for different values of the parameter  $\epsilon$ . The parameter  $\epsilon$  is the KL-divergence threshold between the variational distribution  $q(a_{t}|s_{t})$  and the parameterized policy  $\pi_{\theta}(a_{t}|s_{t})$ .
C.7 The trade-off factors
Regarding the trade-off factors  $\kappa$  and  $\chi$ , we adopt a dual-tuning approach with gradient-based adaptation instead of a manual setting. To facilitate the observation of changes in the trade-off factors  $\kappa$  and  $\chi$ , we record the variation curves of these parameters during the training process. The results of Fig. 9 indicate that as the model gradually converges, the trade-off factors  $\kappa$  and  $\chi$  gradually decrease, eventually approaching zero.
Figure 9: The trade-off factors  $\kappa$  and  $\chi$ , are considered along with the curves of rewards and costs during the training process in task Point-button. Both the trade-off factors  $\kappa$  and  $\chi$  are adaptively adjusted using the dual-tuning approach through gradient-based.
C.8 The time costs of training and testing
We recorded the time costs of various modules of the VOCE and the baseline algorithm during both training and testing. The results presented in Table 2 indicate that during training, the VOCE algorithm incurs additional time costs per step compared to the baseline algorithm due to the execution of additional networks and optimization parameters. However, during the testing phase, as only the policy network is executed, the time cost of VOCE is similar to the baseline algorithm. Furthermore, the execution time per step during testing is significantly lower than  $1ms$ , meeting the real-time requirements for robot and autonomous driving control. Additionally, the results from Table 3 reveal that the evaluation time cost of the Q-values is relatively high due to the computation of multiple target Q-values and gradients.
Table 2: The time costs per single step of the algorithm during training and testing processes. The units for the numerical values are all in seconds.
Algorithms	Training(s)	Testing(s)
VOCE	3.1498	3.0665 × 1e-4
C-CRR	0.0891	1.5140 × 1e-4
COptiDICE	0.0358	1.4316 × 1e-4
BCQ-Lag	0.1104	2.7976 × 1e-4
Table 3: The time costs during the training processes of various modules in VOCE. The units for the numerical values are all in seconds.
Critic	Actor
\( Q^r(s_t, a_t) \)	\( Q^c(s_t, a_t) \)	\( (\lambda, \eta) \)	\( q(a_t|s_t) \)	\( \pi_\theta(a_t|s_t) \)
1.2728	1.2668	0.0026	0.5533	0.0543
D Limitations and Future Work
The setup of this work involves learning safety-constrained policies from offline data without interacting with the environment. Therefore, both the size and quality of the dataset directly impact
the algorithm's performance. Moreover, in the offline setting, the available sample data is limited, making it challenging to adequately represent the state transition matrix  $P$ , in a non-stationary environment. Consequently, the VOCE algorithm struggles to learn high-reward policies that satisfy safety constraints in non-stationary environments.