SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 SUCBVI
1: Initialize:  $\Delta_h^1 (s,a) = \emptyset$ $N_{h}^{1}(s,a) = N_{h}^{1}(s,a,s^{\prime}) = 0$  for all  $s\in S$ $a\in \mathcal{A}$ $s^\prime \in S$  and  $h\in [H]$    
2: for  $k = 1,2,\dots ,K$  do   
3: Update the optimistic estimates of cost.   
4: Update the empirical cost  $\hat{c} (s)$  and calculate  $\bar{c} (s) = \hat{c} (s) - \beta (N^{k}(s),\delta)$  for all  $s\in S$  .   
5: Define  $\mathcal{U}_H^k = \{s\mid \bar{c} (s) > \tau \}$  and  $\mathcal{U}_h^k = \mathcal{U}_{h + 1}^k\cup \{s\mid \forall a\in A,\Delta_h^k (s,a)\cap \mathcal{U}_{h + 1}^k\neq \emptyset \}$  for all  $h\in [H]$  recursively. Calculate  $\{A_h^{k,safe}(s)\}_{h\in [H]}$  by Eq. (3) with  $\{\mathcal{U}_h^k\}_{h\in [H]}$  Perform value iteration with previous estimates.   
6: for  $h = H,H - 1,\dots ,1$  do   
7: for  $s\in S$  do   
8: for  $a\in \mathcal{A}$  do   
9: Compute  $\hat{\mathbb{P}}_h^k (s'\mid s,a) = \frac{N_h^k(s,a,s')}{N_h^k(s,a)}.$    
10:  $Q_h^k (s,a) = \min \{H,r_h(s,a) + \sum_{s'}\hat{\mathbb{P}}_h^k (s'\mid s,a)V_{h + 1}^k (s') + \alpha (N_h^k (s,a))\} .$    
11: end for   
12: if  $s\notin \mathcal{U}_h^k$  then   
13:  $V_{h}^{k}(s) = \max_{a\in A_{h}^{k,safe}(s)}Q_{h}^{k}(s,a),\pi_{h}^{k}(s) = \arg \max_{a\in A_{h}^{k,safe}(s)}Q_{h}^{k}(s,a).$    
14: else   
15:  $V_{h}^{k}(s) = \max_{a\in A}Q_{h}^{k}(s,a),\pi_{h}^{k}(s) = \arg \max_{a\in A}Q_{h}^{k}(s,a).$    
16: end if   
17: end for   
18: end for   
19: for  $h = 1,2,\dots ,H$  do   
20: Take action  $a_h^k = \pi_h^k (s_h^k)$  and observe state  $s_{h + 1}^k$  . Update the estimates of  $\Delta (s,a)$    
21:  $\Delta_h^{k + 1}(s_h^k,a_h^k) = \Delta_h^k (s_h^k,a_h^k)\cup \{s_{h + 1}^k\}$  . Increase  $N_{h}^{k + 1}(s_{h}^{k},a_{h}^{k}),N_{h}^{k + 1}(s_{h}^{k},a_{h}^{k},s_{h + 1}^{k})$  by 1.   
22: end for   
23: end for
in Section 4.1 and calculate the safe action set  $A_h^{safe}(s)$  for each state  $s$ . Then, we perform value iteration in the estimated safe action set. As the estimation becomes more accurate, SUCBVI will eventually avoid potentially unsafe states and achieve both sublinear regrets and violations.
Now we begin to introduce our algorithm. In the beginning, we initialize  $\Delta_h(s,a) = \emptyset$  for all  $(h,s,a)\in [H]\times S\times \mathcal{A}$ . It implies that the agent considers all actions to be safe at first, because no action will lead to unsafe states from the agent's perspective. In each episode, we first estimate the empirical cost  $\hat{c} (s)$  based on historical cost feedback  $z(s)$ , and regard state  $s$  as safe if  $\bar{c} (s) = \hat{c} (s) - \beta >\tau$  for some bonus term  $\beta$ , which aims to guarantee  $\bar{c} (s)\leq c(s)$  (Line 4). Then, we calculate the estimated unsafe state set  $\mathcal{U}_H^k$ , which is a subset of the true unsafe state set  $\mathcal{U} = \mathcal{U}_H$  with a high probability by optimism. With  $\mathcal{U}_H^k$  and  $\Delta_h^k (s,a)$ , we can estimate potentially unsafe state sets  $\mathcal{U}_h^k$  for all  $h\in [H]$  by Eq. (2) recursively.
Then, we perform value iteration to compute the optimistically estimated optimal policy. Specifically, for any hypothesized safe state  $s \notin \mathcal{U}_h^k$ , we update the greedy policy on the estimated safe actions, i.e.,  $\pi_h^k(s) = \max_{a \in A_h^{k,safe}(s)} Q_h^k(s, a)$  (Line 13). On the other hand, for any hypothesized unsafe state  $s_h \in \mathcal{U}_h$ , since there is no action that can completely avoid unsafe states, we ignore safety costs and simply update the policy by  $\pi_h^k(s) = \max_{a \in A} Q(s, a)$  (Line 15). After that, we calculate the estimated optimal policy  $\pi^k$  for episode  $k$ , and the agent follows  $\pi^k$  and collects a trajectory. Then, we update  $\Delta_h^{k+1}(s, a)$  by incorporating the observed state  $s_{h+1}^k$  into the set  $\Delta_h^k(s_h^k, a_h^k)$ . Under this updating rule, it holds that  $\Delta_h^k(s, a) \subseteq \Delta_h(s, a)$  for all  $(s, a) \in S \times \mathcal{A}$ .
The performance of Algorithm 1 is summarized below in Theorem 4.2.
Theorem 4.2. Let  $\alpha(n, \delta) = 7H\sqrt{\frac{\ln(5SAHK / \delta)}{n}}$  and  $\beta(n, \delta) = \sqrt{\frac{2}{n}\log(SK / \delta)}$ . With probability at least  $1 - \delta$ , the regret and step-wise violation of Algorithm 1 are bounded by
$$
R (K) = \widetilde {\mathcal {O}} (\sqrt {H ^ {2} S A T}), \qquad C (K) = \widetilde {\mathcal {O}} (\sqrt {S T} + S ^ {2} A H ^ {2}).
$$
Moreover, if  $\mathcal{C}_{\mathrm{gap}} \triangleq \min_{s \in \mathcal{U}} (c(s) - \tau)_+ > 0$ , we have  $C(K) = \widetilde{\mathcal{O}}(S / \mathcal{C}_{\mathrm{gap}} + S^2AH^2)$ .
Theorem 4.2 shows that SUCBVI achieves both sublinear regret and violation. Moreover, when all the unsafe states have a large cost compared to the safety threshold  $\tau$ , i.e.,  $\mathcal{C}_{\mathrm{gap}} = \min_{s\in \mathcal{U}}(c(s) - \tau)_+ > 0$  is a constant, we can distinguish the unsafe states easily and get a constant violation. In particular, when  $\tau = 1$ , Safe-RL-SW degenerates to the unconstrained MDP and Theorem 4.2 maintains the same regret  $\widetilde{\mathcal{O}} (\sqrt{H^2SAT})$  as UCBVI-CH (Azar et al., 2017), while CMDP algorithms (Efroni et al., 2020b; Liu et al., 2021a) suffer a larger  $\widetilde{\mathcal{O}} (\sqrt{H^3S^3AT})$  regret.
We provide the analysis idea of Theorem 4.2 here. First, by the updating rule of  $\Delta_h^k (s,a)$ , we show that  $\mathcal{U}_h^k$  and  $A_{h}^{k,safe}(s)$  have the following crucial properties:  $\mathcal{U}_h^k\subseteq \mathcal{U}_h$ ,  $A_{h}^{safe}(s)\subseteq A_{h}^{k,safe}(s)$ . Based on this property, we can prove that  $Q_{h}^{*}(s,a)\leq Q_{h}^{k}(s,a)$  and  $V_{h}^{*}(s)\leq V_{h}^{k}(s)$  for all  $(s,a)$  and then apply the regret decomposition techniques to derive the regret bound.
Recall that  $\pi^k$  is a feasible policy with respect to the estimated unsafe state set  $\mathcal{U}_H^k$  and transition set  $\{\Delta_h^k (s,a)\}_{h\in [H]}$ . If the agent takes policy  $\pi^k$  and the transition follows  $\Delta_h^k (s,a)$ , i.e.,  $s_{h + 1}^k\in \Delta_h^k (s,a)$ , the agent never arrives at any estimated unsafe state  $s\in \mathcal{U}_H^k$  in this episode. Hence the agent will suffer at most an  $\widetilde{\mathcal{O}} (\sqrt{ST})$  step-wise violation or an  $\widetilde{\mathcal{O}} (S / \mathcal{C}_{\mathrm{gap}} + S^2 AH^2)$  gap-dependent bounded violation. Yet, the situation  $s_{h + 1}\in \Delta_h^k (s,a)$  does not always hold for all  $h\in [H]$ . In the case when  $s_{h + 1}^k\notin \Delta_h^k (s,a)$ , we add the newly observed state  $s_{h + 1}^k$  into  $\Delta_h^{k + 1}(s_h^k,a_h^k)$  (Line 21). We can show that this case appears at most  $S^2 AH$  times, and thus incurs  $O(S^{2}AH^{2})$  additional violation. Combining the above two cases, the total violation can be upper bounded.
5 Lower Bounds for Safe-RL-SW
In this section, we provide a matching lower bound for Safe-RL-SW in Section 4. The lower bound shows that if an algorithm always achieves a sublinear regret in Safe-RL-SW, it must incur an  $\Omega (\sqrt{ST})$  violation. This result matches our upper bound in Theorem 4.2, showing that SUCBVI achieves the optimal violation performance.
Theorem 5.1. If an algorithm has an expected regret  $\mathbb{E}_{\pi}[R(K)] \leq \frac{HK}{24}$  for all MDP instances, there exists an MDP instance in which the algorithm suffers expected violation  $\mathbb{E}_{\pi}[C(K)] = \Omega(\sqrt{ST})$ .
Now we validate the optimality in terms of regret. Note that if we do not consider safety constraints, the lower bound for classic RL (Osband & Van Roy, 2016) can be applied to our setting. Thus, we also have an  $\Omega(\sqrt{T})$  regret lower bound. To understand the essential hardness brought by safety constraints, we further investigate whether safety constraints will lead to an  $\Omega(\sqrt{T})$  regret, given that we can achieve an  $o(\sqrt{T})$  regret on some good instances without the safety constraints.
Theorem 5.2. For any  $\alpha \in (0,1)$ , there exists a parameter  $n$  and  $n$  MDPs  $M_1,\ldots ,M_n$  satisfying that:
1. If we do not consider any constraint, there is an algorithm that achieves an  $\tilde{\mathcal{O}}(T^{(1 - \alpha) / 2})$  regret compared to the unconstrained optimal policy on all MDPs.
2. If we consider the safety constraint, any algorithm with a  $O(T^{1 - \alpha})$  expected violation will achieve an  $\Omega(\sqrt{HST})$  regret compared to the constrained optimal policy on one of MDPs.
Intuitively, Theorem 5.2 shows that if one achieves sublinear violation, she must suffer at least an  $\Omega(\sqrt{T})$  regret even if she can achieve an  $o(\sqrt{T})$  regret without considering constraints. This theorem demonstrates the hardness particularly brought by the step-wise constraint, and corroborates the optimality of our results. Combining with Theorem 5.1, the two lower bounds show an essential trade-off between the violation and performance.
6 Safe Reward-Free Exploration with Step-wise Violation Constraints
6.1 Formulation of Safe-RFE-SW
In this section, we consider Safe RL in the reward-free exploration (RFE) setting (Jin et al., 2020; Kaufmann et al., 2021; Ménard et al., 2021) called Safe-RFE-SW, to show the generality of our proposed framework. In the RFE setting, the agent does not have access to reward signals and only receives random safety cost feedback  $z(s) = c(s) + \zeta$ . To impose safety requirements, Safe-RFE-SW requests the agent to keep small safety violations during exploration, and outputs a near-optimal safe policy after receiving the reward function.
Definition 6.1  $((\varepsilon, \delta)$ -optimal safe algorithm for Safe-RFE-SW). An algorithm is  $(\varepsilon, \delta)$ -optimal safe for Safe-RFE-SW if it outputs the triple  $(\hat{\mathbb{P}}, \hat{\Delta}(s, a), \hat{\mathcal{U}}_H)$  such that for any reward function  $r$ , with probability at least  $1 - \delta$ ,
$$
V _ {1} ^ {*} (s _ {1}; r) - V _ {1} ^ {\hat {\pi} ^ {*}} (s _ {1}; r) \leq \varepsilon , \quad \mathbb {E} _ {\pi} \left[ \sum_ {h = 1} ^ {H} \left(c \left(s _ {h}\right) - \tau\right) _ {+} \right] \leq \varepsilon , \forall \pi \in \Pi \tag {4}
$$
where  $\hat{\pi}^*$  is the optimal feasible policy with respect to  $(\hat{\mathbb{P}},\hat{\Delta} (s,a),\hat{\mathcal{U}}_H,r)$ ,  $\Pi$  is the set of feasible policies with respect to  $(\Delta (s,a),\mathcal{U}_H)$ . and  $V(s;r)$  is the value function under reward function  $r$ . We say that a policy is  $\varepsilon$ -optimal if it satisfies the left inequality in Eq. (4) and  $\varepsilon$ -safe if it satisfies the right inequality in Eq. (4). We measure the performance by the number of episodes used before the algorithm terminates, i.e., sample complexity. Moreover, the cumulative step-wise violation till episode  $K$  is defined as  $C(K) = \sum_{k = 1}^{K}\sum_{h = 1}^{H}(c(s_h^k) - \tau)_+$ . In Safe-RFE-SW, our goal is to design an  $(\varepsilon ,\delta)$ -optimal safe algorithm, and minimize both sample complexity and violation.
6.2 Algorithm SRF-UCRL
The Safe-RFE-SW problem requires us to consider extra safety constraints for both the exploration phase and final output policy, which needs new algorithm design and techniques compared to previous RFE algorithms. Also, the techniques for Safe-RL-SW in Section 4.2 are not sufficient for guaranteeing the safety of output policy, because SUCBVI only guarantees a step-wise violation during exploration.
We design an efficient algorithm Safe RF-UCRL (SRF-UCRL), which builds upon previous RFE algorithm RF-UCRL (Kaufmann et al., 2021). SRF-UCRL distinguishes potentially unsafe states and safe actions by backward iteration, and establishes a new uncertainty function to guarantee the safety of output policy. Algorithm 2 illustrates the procedure of SRF-UCRL. Specifically, in each episode  $k$ , we first execute a policy  $\pi_h^k$  computed from previous episodes, and then update the estimated next state set  $\{\Delta_h^k(s,a)\}_{h\in [H]}$  and unsafe state set  $\mathcal{U}_H^k$  by optimistic estimation. Then, we use Eq. (2) to calculate the unsafe state set  $\mathcal{U}_h$  for all steps  $h\in [H]$ . After that, we update the uncertainty function  $\overline{W}^k$  defined in Eq. (5) below and compute the policy  $\pi^{k+1}$  that maximizes the uncertainty to encourage more exploration in the next episode.
Now we provide the definition of the uncertainty function, which measures the estimation error between the empirical MDP and true MDP. For any safe state-action pair  $s \notin \mathcal{U}_h$ ,  $a \in A_h^{k,safe}(s)$ , we define
$$
\overline {{W}} _ {h} ^ {k} (s, a) = \min  \left\{H, M \left(N _ {h} ^ {k} (s, a), \delta\right) + \sum_ {s ^ {\prime}} \hat {\mathbb {P}} _ {h} ^ {k} \left(s ^ {\prime} \mid s, a\right) \max  _ {b \in A _ {h + 1} ^ {k, s a f e} \left(s ^ {\prime}\right)} \overline {{W}} _ {h + 1} ^ {k} \left(s ^ {\prime}, b\right) \right\}. \tag {5}
$$
where  $M(N_h^k (s,a),\delta) = 2H\sqrt{\frac{2\gamma(N_h^k(s,a),\delta)}{N_h^k(s,a)} + \frac{SH\gamma(N_h^k(s,a),\delta)}{N_h^k(s,a)}}$ , and  $\gamma (n,\delta)$  is a logarithmic term that is formally defined in Theorem 6.2. For other state-action pairs  $(s,a)$ , we relax the restriction of  $b\in \mathcal{A}_{h + 1}^{k,safe}(s')$  to  $b\in \mathcal{A}$  in the max function. Our algorithm stops when  $\overline{W}_1^K (s_1,\pi^K (s_1))$  shrinks to within  $\varepsilon /2$ . Compared to previous RFE works (Kaufmann et al. (2021); Menard et al. (2021)), our uncertainty function has two distinctions. First, for any safe state-action pair  $(s,a)\in (\mathcal{S}\setminus \mathcal{U}_h,A_h^{k,safe}(s))$ , Eq. (5) considers only safe actions  $A_{h + 1}^{k,safe}(s)$ , which guarantees that the agent focuses on safe policies. Second, Eq. (5) incorporates another term  $(SH\gamma (N_h^k (s,a),\delta) / N_h^k (s,a))$  to control the expected violation for feasible policies. Now we present our result for Safe-RFE-SW.
Algorithm 2 SRF-UCRL
1: Initialize:  $k = 1$ $\overline{W}_h^0 (s,a) = H$ $\pi_h^1 (s) = a_1$  for all  $s\in S,a\in \mathcal{A}$  and  $h\in [H]$    
2: while  $\overline{W}_h^{k - 1}(s_1,\pi_1^k (s_1))\leq \varepsilon /2$  do   
3: for  $h = 1,\dots ,H$  do   
4: Observe state  $s_h^k$  . Take action  $a_{h}^{k} = \pi_{h}^{k}(s_{h}^{k})$  and observe  $s_{h + 1}^k$  Update the optimistic estimate of cost.   
5: Calculate empirical cost  $\hat{c} (s)$  and  $\bar{c} (s) = \hat{c} (s) - \beta (N^{k}(s),\delta)$  for all  $s\in S$  Update the estimates of  $\Delta_h(s,a)$  and  $\mathcal{U}_h$    
6: Update  $\Delta_h^k (s_h^k,a_h^k) = \Delta_h^{k - 1}(s_h^k,a_h^k)\cup \{s_{h + 1}^k\}$  and  $\{\mathcal{U}_h^k\}_{h\in [H]}$  by Eq. (2).   
7: Calculate  $A_{h}^{k,safe}(s)$  for all  $s\in S$  by Eq. (3).   
8: end for   
9: Update  $N_{h}^{k}(s,a,s^{\prime}),N_{h}^{k}(s,a)$  and  $\hat{\mathbb{P}}_h^k$  for all  $s\in S,a\in A,s'\in S$  and  $h\in [H]$    
10: Compute  $\overline{W}^{k}$  according to Eq. (5). Calculate the greedy policy.   
11: for  $h\in [H],s\in S$  do   
12: if  $A_{h}^{k,safe}(s)\neq \emptyset$  then   
13:  $\pi_h^{k + 1}(s) = \arg \max_{a\in A_h^{k,safe}(s)}\overline{W}_h^k (s,a)$    
14: else   
15:  $\pi_h^{k + 1}(s) = \arg \max_{a\in A}\overline{W}_h^k (s,a)$    
16: end if   
17: end for   
18: Set  $k = k + 1$    
19: end while   
20: return the tuple  $(\hat{\mathbb{P}}_h^{k - 1},\Delta_h^{k - 1}(s,a),\mathcal{U}_H^{k - 1})$
Theorem 6.2. Let  $\gamma (n,\delta) = 2(\log (2SAH / \delta) + (S - 1)\log (e(1 + n / (S - 1)))$ , Algorithm 2 is a  $(\varepsilon ,\delta)$ -PAC algorithm with sample complexity at most<sup>2</sup>
$$
K = \widetilde {\mathcal {O}} \left(\left(\frac {S ^ {2} A H ^ {2}}{\varepsilon} + \frac {H ^ {4} S A}{\varepsilon^ {2}}\right) \left(\log \left(\frac {1}{\delta}\right) + S\right)\right),
$$
The step-wise violation of Algorithm 2 during exploration is  $C(K) = \widetilde{\mathcal{O}}(S^2 AH^2 + \sqrt{ST})$ .
Compared to previous work (Kaufmann et al., 2021) with an  $\widetilde{\mathcal{O}} ((H^4 SA / \varepsilon^2)(\log (1 / \delta) + S))$  sample complexity, our result has an additional term  $\widetilde{\mathcal{O}} ((S^2 AH^2 /\varepsilon)(\log (1 / \delta) + S))$ . This term is due to the additional safety requirement for the final output policy, which was not considered in previous RFE algorithms (Kaufmann et al., 2021; Menard et al., 2021). When  $\varepsilon$  and  $\delta$  are sufficiently small, the leading term is  $\widetilde{\mathcal{O}} ((H^4 SA / \varepsilon^2)\log (1 / \delta))$ , which implies that our algorithm satisfies the safety constraint without suffering additional regret.
6.3 Analysis for Algorithm SRF-UCRL
For the analysis of step-wise violation (Eq. (1)), similar to algorithm SUCBVI, algorithm SRF-UCRL estimates the next state set  $\Delta_h(s,a)$  and potentially unsafe state set  $\mathcal{U}_h$ , which guarantees a  $\widetilde{\mathcal{O}} (\sqrt{ST})$  step-wise violation. Now we give a proof sketch for the  $\varepsilon$ -safe property of output policy (Eq. (4)).
First, if  $\pi$  is a feasible policy for  $(\hat{\mathbb{P}}^k,\Delta^k (s,a),\{\mathcal{U}_h^k\}_{h\in [H]})$  and  $s_{h + 1}\in \Delta_h^K (s_h,a_h)$  for all  $h\in [H]$ , the agent who follows policy  $\pi$  will only visit the estimated safe states. Since each estimated safe state
Here  $\tilde{\mathcal{O}} (\cdot)$  ignores all log  $S$ ,  $\log A$ ,  $\log H$ ,  $\log (1 / \varepsilon)$  and  $\log (\log (1 / \delta))$  terms.
3Ménard et al. (2021) improve the result by a factor  $H$  and replace  $\log(1/\delta) + S$  by  $\log(1/\delta)$  via the Bernstein-type inequality. While they do not consider the safety constraints, we believe that similar improvement can also be applied to our framework, without significant changes in our analysis. However, since our paper mainly focuses on tackling the safety constraint for reward-free exploration, we use the Hoeffding-type inequality to keep the succinctness of our statements.
Figure 1: Experimental results for Safe-RL-SW and Safe-RFE-SW. The left two figures show the average rewards and step-wise violations of algorithms SUCBVI, UCBVI (Azar et al., 2017), OptCMDP-bonus (Efroni et al., 2020b), Triple-Q (Wei et al., 2022) and Optpess (Liu et al., 2021a). The right two figures show the reward and expected violation of the policies outputted by algorithms SRF-UCRL and RF-UCRL (Kaufmann et al., 2021).
only suffers a  $\mathcal{O}(1 / \sqrt{N_t(s_h,\pi_h(s_h))})$  violation, the violation led by this situation is bounded by  $\overline{W}_1^k (s,\pi_1(s_1))$ . Next, we bound the probability that  $s_{h + 1}\notin \Delta_h^K (s_h,a_h)$  for some step  $h\in [H]$ . For any state-action pair  $(s,a)$ , if there is a probability  $\mathbb{P}(s' | s,a)\geq \widetilde{\mathcal{O}} (\log (S / \delta) / N_h^K (s,a))$  that the agent transitions to next state  $s^\prime$  from  $(s,a)$  at step  $h$ , the state  $s^\prime$  is put into  $\Delta_h^K (s,a)$  with probability at least  $1 - \delta /S$ . Then, we can expect that all such states are put into our estimated next state set  $\Delta_h^K (s,a)$ . Thus, the probability that  $s_{h + 1}\notin \Delta_h^K (s_h,a_h)$  is no larger than  $\widetilde{\mathcal{O}} (S\log (S / \delta) / N_h^K (s,a))$  by a union bound over all possible next states  $s^\prime$ . Based on this argument,  $\overline{W}_h^k (s_h,a_h)$  is an upper bound for the total probability that  $s_{h + 1}\notin \Delta_h^K (s_h,a_h)$  for some step  $h$ . This will lead to additional  $\overline{W}_1^K (s_1,\pi_1(s_1))$  expected violation. Hence the expected violation of output policy is upper bounded by  $2\overline{W}_1^K (s_1,\pi_1^K (s_1))\leq \varepsilon$ . The complete proof is provided in Appendix A.
In this section, we provide experiments for Safe-RL-SW and Safe-RFE-SW to validate our theoretical results. For Safe-RL-SW, we compare our algorithm SUCBVI with a classical RL algorithm UCBVI (Azar et al., 2017) and three state-of-the-art CMDP algorithms OptCMDP-bonus (Efroni et al., 2020b), Optpess (Liu et al., 2021a) and Triple-Q (Wei et al., 2022). For Safe-RFE-SW, we report the average reward in each episode and cumulative step-wise violation. For Safe-RFE-SW, we compare our algorithm SRF-UCRL with a state-of-the-art RFE algorithm RF-UCRL (Kaufmann et al., 2021). We do not plot the regret because unconstrained MDP or CMDP algorithms do not guarantee step-wise violation. Applying them to the step-wise constrained setting can lead to negative or large regret and large violations, making the results meaningless. Detailed experiment setup is in Appendix E.
As shown in Figure 1, the rewards of SUCBVI and SRF-UCRL converge to the optimal rewards under safety constraints (denoted by "safe optimal reward"). In contrast, the rewards of UCBVI and UCRL converge to the optimal rewards without safety constraints (denoted by "unconstrained optimal reward"), and those of CMDP algorithms converge to a policy with low expected violation (denoted by "constrained optimal reward"). For Safe-RFE-SW, the expected violation of output policy of SRF-UCRL converges to zero while that of RF-UCRL does not. This corroborates the ability of SRF-UCRL in finding policies that simultaneously achieve safety and high rewards.
In this paper, we investigate a novel safe reinforcement learning problem with step-wise safety constraints. We first provide an algorithmic framework SUCBVI to achieve both an  $\widetilde{\mathcal{O}} (\sqrt{H^3SAT})$  regret and an  $\widetilde{\mathcal{O}} (\sqrt{ST})$  step-wise or an  $\widetilde{\mathcal{O}} (S / \mathcal{C}_{\mathrm{gap}} + S^2 AH^2)$  gap-dependent bounded violation that is independent of  $T$ . Then, we provide two lower bounds to validate the optimality of SUCBVI in both violation and regret in terms of  $S$  and  $T$ . Further, we extend our framework to the safe RFE with a step-wise violation and provide an algorithm SRF-UCRL that identifies a near-optimal safe policy given any reward function  $r$  and guarantees an  $\widetilde{\mathcal{O}} (\sqrt{ST})$  violation during exploration.
This work is supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2020AAA0108400 and 2020AAA0108403 and the Tsinghua Precision Medicine Foundation 10001020109.