SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{c}_i^\dagger = \max \left\{\epsilon, c_i - \sum_{j=1}^i q_{j,i} d_j\right\}$ , ( $\forall i \in [H]$ ), is the lower bound of  $c_i^\dagger$ , and  $d_j = \|a_j - \pi^\dagger(x_j)\|$ ,  $\forall j \in [H]$  is the action difference at round  $j$ .
At each round  $h \in [H]$ , Proposition 4.1 provides a sufficient condition for satisfying all the anytime competitive constraints from round  $h$  to round  $H$  given in (1). The meanings of the parameters in Proposition 4.1 are explained as follows. The weight  $q_{j,i}$  measures the impact of action deviation at round  $j$  on the cost difference  $|c_i - c_i^\dagger|$  at round  $i \geq j$ , and the weight  $\Gamma_{j,n}$  indicates the total impact of the action deviation at round  $j$  on the sum of the cost differences from rounds  $n$  to round  $H$ . Based on the definition of  $q_{j,i}$ , we get  $\hat{c}_i^\dagger$  as a lower bound of the prior cost  $c_i^\dagger$ . With these bounds, we can calculate the maximum allowed total action deviation compared with the prior actions  $\pi^\dagger(x_j)$  from round  $j = h$  to  $h'$  as  $G_{h,h'}$
By applying Proposition 4.1 at initialization, we can guarantee the anytime competitive constraints for all rounds  $h' \in [H]$  if we ensure that for all rounds  $h' \in [H]$ ,  $\sum_{j=1}^{h'} \Gamma_{j,j} \| a_j - \pi^\dagger(x_j) \| \leq G_{1,h'} = h'(\lambda \epsilon + b)$ . This sufficient condition is a long-term constraint relying on the relaxation parameters  $\lambda$  and  $b$ . Although we can guarantee the anytime competitive constraints by the sufficient condition obtained at initialization, we apply Proposition 4.1 at all the subsequent rounds with the cost feedback information to get larger action sets and more flexibility to optimize the average reward. In this way, we can update the allowed deviation according to the next corollary.
Corollary 4.2. At round 1, we initialize the allowed deviation as  $D_{1} = \lambda \epsilon + b$ . At round  $h$ ,  $h > 1$ , the allowed deviation is updated as
$$
D _ {h} = \max  \left\{D _ {h - 1} + \lambda \epsilon + b - \Gamma_ {h - 1, h - 1} d _ {h - 1}, R _ {h - 1} + \lambda \epsilon + b \right\} \tag {6}
$$
where  $R_{h-1} = \sum_{i=1}^{h-1}\left((1+\lambda)\hat{c}_i^\dagger - c_i - \Gamma_{i,h}d_i\right)$  with notations defined in Proposition 4.1. The  $(\lambda, b)$ -anytime competitive constraints in Definition 3.1 are satisfied if it holds at each round  $h$  that  $\Gamma_{h,h}\|a_h - \pi^\dagger(x_h)\| \leq D_h$ .
Corollary 4.2 gives a direct way to calculate the allowed action deviation at each round. In the update rule (6) of the allowed deviation, the first term of the maximum operation is based on the deviation calculation at round  $h - 1$  while the second term is obtained by applying Proposition 4.1 for round  $h$ .
We can find that the conditions to satisfy the anytime competitive constraints can be controlled by parameters  $\lambda$  and  $b$ . With larger  $\lambda$  and  $b$ , the anytime competitive constraints are relaxed and the conditions in Corollary 4.2 get less stringent. Also, the conditions in Corollary 4.2 rely on the minimum cost value  $\epsilon$  and other system parameters including Lipschitz constants  $L_{c}, L_{f}, L_{\pi^{\dagger}}$  and telescoping parameters  $p$  through  $\Gamma_{h,h}$ . Since  $\Gamma_{h,h}$  increases with the Lipschitz and telescoping parameters, even if the estimated Lipschitz constants and the telescoping parameters are higher than the actual values or the estimated minimum cost is lower than the actual value, the obtained condition by Corollary 4.2 is sufficient to guarantee the anytime competitive constraints, although it is more stringent than the condition calculated by the actual parameters.
By Corollary 4.2, we can define the safe action set at each round  $h$  as
$$
\mathcal {A} _ {h} \left(D _ {h}\right) = \left\{a \mid \Gamma_ {h, h} \| a - \pi^ {\dagger} \left(x _ {h}\right) \| \leq D _ {h} \right\}. \tag {7}
$$
Algorithm 1 Anytime-Competitive Decision-making (ACD)
Initialization: Initialize an allowed deviation:  $D_{1} = \lambda \epsilon +b$    
for  $h = 1,\dots ,H$  do Obtain the output of the ML policy  $\tilde{\pi}$  as  $\tilde{a}_h$  . Select the action  $a_{t}$  by projecting  $\tilde{a}_h$  into the safe action set  $\mathcal{A}_h(D_h)$  in (7). Update the allowed deviation  $D_{h + 1}$  by (6).   
end for
With the safe action set design in (7), we propose a projection-based algorithm called ACD in Algorithm [1]. We first initialize an allowed deviation as  $D_{1} = \lambda \epsilon + b$ . When the output  $\tilde{a}_{h}$  of the ML model is obtained at each round  $h$ , it is projected into a safe action set  $\mathcal{A}_{h}(D_{h})$  depending on the allowed deviation  $D_{h}$ , i.e.  $a_{h} = P_{\mathcal{A}_{h}(D_{h})}(\tilde{a}_{h}) = \arg \min_{a \in \mathcal{A}_{h}(D_{h})} \| a - \tilde{a}_{h} \|$ . The projection can be efficiently solved by many existing methods on constrained policy learning [67, 11, 4, 41, 24]. The allowed deviation is then updated based on Corollary 4.2. Intuitively, if the actions are closer to the prior actions before  $h$ , i.e. the action deviations  $\{d_{i}\}_{i=1}^{h-1}$  get smaller, then  $R_{h-1}$  becomes larger and  $D_{h}$  becomes larger, leaving more flexibility to deviate from  $a_{i}^{\dagger}$ ,  $i \geq h$  in subsequent rounds.
4.2 Anytime-Competitive RL
The anytime competitive constraints have been satisfied by Algorithm [1] but it remains to design an RL algorithm to optimize the average reward under the anytime competitive cost constraints, which is given in this section.
The anytime-competitive decision-making algorithm in Algorithm 1 defines a new MDP, with an additional set of allowed deviations  $\mathcal{D}$  to the A-MDP defined in Section 3.1 denoted as  $\tilde{\mathcal{M}}(\mathcal{X},\mathcal{D},\mathcal{A},\mathcal{F},g,H,r,c,\tilde{\pi},\pi^{\dagger})$ . In the new MDP, we define an augmented state  $s_h$  which include the original state  $x_h$ , the allowed deviation  $D_h\in \mathcal{D}$ , and history information  $\{c_i\}_{i = 1}^{h - 1}$  and  $\{d_i\}_{i = 1}^{h - 1}$ . The transition of  $x_{h}$  is defined by  $f_{h}$  in Section 3.1 and needs to be learned while the transition of  $D_h$  is defined in (6) and is known to the agent. The ML policy  $\tilde{\pi}$  gives an output  $\tilde{a}_h$  and the selected action is the projected action  $a_{h} = P_{A_{h}(D_{h})}(\tilde{a}_{h})$ . Then the environment generates a reward  $r_h(x_h,P_{A_h(D_h)}(\tilde{a}_h))$  and a cost  $c_{h}(x_{h},P_{A_{h}(D_{h})}(\tilde{a}_{h}))$ . Thus, the value function corresponding to the ML policy  $\tilde{\pi}$  can be expressed as  $\tilde{V}_h^{\tilde{\pi}}(s_h) = \mathbb{E}\left[\sum_{i = h}^{H}r_i(x_i,P_{A_h(D_h)}(\tilde{a}_h))\right]$  with  $\tilde{a}_h$  being the output of the ML policy  $\tilde{\pi}$ . For notation convenience, we sometimes write the actions of  $\pi^{*}$  and  $\pi^{\dagger}$  as  $\pi^{*}(s)$  and  $\pi^{\dagger}(s)$  even though they only reply on the original state  $x$  in  $s$ .
To solve the MDP, we propose a model-based RL algorithm called ACRL in Algorithm 2. Different from the existing model-based RL algorithms [48, 6, 70], ACRL utilizes the dynamic model of A-CMDP and ACD (Algorithm 1) to optimize the average reward. Given a transition distribution  $g$  at episode  $k$ , we perform value iteration to update  $\tilde{Q}$  functions for  $h = 1, \dots, H$ .
$$
\tilde {Q} _ {h} ^ {k} (s _ {h}, \tilde {a} _ {h}) = r _ {h} (x _ {h}, a _ {h}) + \mathbb {E} _ {g} \left[ \tilde {V} _ {h + 1} ^ {k} (s _ {h + 1}) \mid s _ {h}, a _ {h} \right], \tilde {V} _ {h} ^ {k} (s _ {h}) = \max _ {a \in \mathcal {A}} \tilde {Q} _ {h} ^ {k} (s _ {h}, a),
$$
$$
\mathbb {E} _ {g} \left[ \tilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1}\right) \mid s _ {h}, a _ {h} \right] = \sum_ {f \in \mathcal {F}} \tilde {V} _ {h + 1} ^ {k} \left(s _ {h + 1}\right) g (f), \tag {8}
$$
where  $a_{h} = P_{\mathcal{A}_{h}(D_{h})}(\tilde{a}_{h}), \tilde{Q}_{H + 1,k}(s,a) = 0, \tilde{V}_{H + 1,k}(s) = 0$ . The transition model  $g$  is estimated as
$$
\hat {g} ^ {k} = \arg \min  _ {g \in \mathcal {G}} \sum_ {i = 1} ^ {k - 1} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {g} \left[ \tilde {V} _ {h + 1} ^ {i} \left(s _ {h + 1}\right) \mid s _ {h}, a _ {h} \right] - \tilde {V} _ {h + 1} ^ {i} \left(s _ {h + 1}\right)\right) ^ {2}. \tag {9}
$$
Based on the transition estimation, we can calculate the confidence set of the transition model as
$$
\mathcal {G} _ {k} = \left\{ \right.g \in \mathcal {G} \left| \right. \sum_ {i = 1} ^ {k - 1} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {g} \left[ \tilde {V} _ {h + 1} ^ {i} \left(s _ {h + 1}\right) \mid s _ {h}, a _ {h} \right] - \mathbb {E} _ {\hat {g} ^ {k}} \left[ \tilde {V} _ {h + 1} ^ {i} \left(s _ {h + 1}\right) \mid s _ {h}, a _ {h} \right]\right) ^ {2} \leq \beta_ {k} \left. \right\}, \tag {10}
$$
where  $\beta_{k} > 0$  is a confidence parameter.
Algorithm 2 Anytime-Competitive Reinforcement Learning (ACRL)
1: Initialization: Transition model set  $\mathcal{G}_1 = \{\hat{g}^1\}$   
2: for each episode  $k = 1, \dots, K$  do  
3: Observe the initial state  $s_1^k$ .  
4: Select  $g^k = \arg \max_{g \in \mathcal{G}^k} \mathbb{E}_g[V_1(s_1^k)]$ .  
5: Perform value iteration with  $g^k$  in Eqn. (8) and update  $\tilde{Q}$  functions  $\tilde{Q}_1^k, \dots, \tilde{Q}_H^k$ .  
6: for each round  $h = 1, \dots, H$  do  
7: Run ACD (Algorithm 1) by ML policy  $\tilde{\pi}^k(s_h) = \arg \max_{a \in \mathcal{A}} \tilde{Q}_h^k(s_h, a)$   
8: Observe state  $s_{h+1}^k$  and store values  $\tilde{V}_{h+1}^k(s_{h+1}^k)$ .  
9: end for  
10: Update transition model  $\hat{g}^{k+1}$  using (9) and calculate confidence set  $\mathcal{G}_{k+1}$ .  
11: end for
With a learned ML policy  $\tilde{\pi}^k$  at each episode  $k$ , the policy used for action selection is the ACD policy  $\pi^k$ . Given the optimal ML policy  $\tilde{\pi}^* = \arg \max_{\tilde{\pi}\in \tilde{\Pi}}\tilde{V}_1^{\tilde{\pi}}(s_1)$  with  $\tilde{\Pi}$  being the ML policy space, the optimal ACD policy is denoted as  $\pi^0$ . For state  $s_h$  at round  $h$ ,  $\pi^k$  and  $\pi^0$  select actions as
$$
\pi^ {k} \left(s _ {h}\right) = P _ {\mathcal {A} _ {h} \left(D _ {h}\right)} \left(\tilde {\pi} ^ {k} \left(s _ {h}\right)\right), \quad \pi^ {\circ} \left(s _ {h}\right) = P _ {\mathcal {A} _ {h} \left(D _ {h}\right)} \left(\tilde {\pi} ^ {*} \left(s _ {h}\right)\right). \tag {11}
$$
In the definition of A-CMDP, the dimension of the augmented state  $s_h$  increases with the length of the horizon  $H$ , which cloud cause a scalability issue for implementation. The scalability issues also exist in other RL works with history-dependent states [58, 14]. In practice, tractable methods can be designed through feature aggregation [58] or PODMP [66].
5 Performance Analysis
In this section, we analyze the reward regret of ACRL to show the impacts of anytime cost constraints on the average reward.
5.1 Regret due to Constraint Guarantee
Intuitively, due to the anytime competitive constraints in Eqn. (1), there always exists an unavoidable reward gap between an ACD policy and the optimal-unconstrained policy  $\pi^{*}$ . In this section, to quantify this unavoidable gap, we bound the regret of the optimal ACD policy  $\pi^{\circ}$ , highlighting the impact of anytime competitive cost constraints on the average reward performance.
Theorem 5.1. Assume that the optimal-unconstrained policy  $\pi^{*}$  has a value function  $Q_{h}^{\pi^{*}}(x,a)$  which is  $L_{Q,h}$ -Lipschitz continuous with respect to the action  $a$  for all  $x$ . The regret between the optimal ACD policy  $\pi^{\circ}$  that satisfies  $(\lambda ,b)$ -anytime competitiveness and the optimal-unconstrained policy  $\pi^{*}$  is bounded as
$$
\mathbb {E} _ {x _ {1}} \left[ V _ {1} ^ {\pi^ {*}} \left(x _ {1}\right) - V _ {1} ^ {\pi^ {\circ}} \left(x _ {1}\right) \right] \leq \mathbb {E} _ {y _ {1: H}} \left\{\sum_ {h = 1} ^ {H} L _ {Q, h} \left[ \eta - \frac {1}{\Gamma_ {h , h}} \left(\lambda \epsilon + b + \Delta G _ {h}\right) \right] ^ {+} \right\}, \tag {12}
$$
where  $\eta = \sup_{x\in \mathcal{X}}\| \pi^{*}(x) - \pi^{\dagger}(x)\|$  is the maximum action discrepancy between the policy prior  $\pi^{\dagger}$  and optimal-unconstrained policy  $\pi^{*}$ ;  $\Gamma_{h,h}$  is defined in Proposition 4.1;  $\Delta G_{h} = [R_{h - 1}]^{+}$  is the gain of the allowed deviation by applying Proposition 4.1 at round  $h$ .
The regret bound stated in Theorem 5.1 is intrinsic and inevitable, due to the committed assurance of satisfying the anytime competitive constraints. Such a bound cannot be improved via policy learning, i.e., converge to 0 when the number of episodes  $K \to \infty$ . This is because to satisfy the  $(\lambda, b)$ -anytime competitiveness, the feasible policy set  $\Pi_{\lambda, b}$  defined under (II) is a subset of the original policy set  $\Pi$ , and the derived regret is an upper bound of  $\max_{\pi \in \Pi} \mathbb{E}_{x_1} [V_1^\pi(x_1)] - \max_{\pi \in \Pi_{\lambda, b}} \mathbb{E}_{x_1} [V_1^\pi(x_1)]$ . Moreover, the regret bound relies on the action discrepancy  $\eta$ . This is because if the optimal-unconstrained policy  $\pi^*$  is more different from the prior  $\pi^\dagger$ , its actions are altered to a larger extent to guarantee the constraints, resulting in a larger degradation of the reward performance. More importantly, the regret bound indicates the trade-off
between the reward optimization and anytime competitive constraint satisfaction governed by the parameters  $\lambda$  and  $b$ . When  $\lambda$  or  $b$  becomes larger, we can get a smaller regret because the anytime competitive constraints in (1) are relaxed to have more flexibility to optimize the average reward. In the extreme cases when  $\lambda$  or  $b$  is large enough, all the policies in  $\Pi$  can satisfy the anytime competitive constraints, so we can get zero regret.
Moreover, the regret bound shows that the update of allowed deviation by applying Proposition 4.1 based on the cost feedback at each round will benefit the reward optimization. By the definition of  $R_{h-1}$  in Corollary 4.2 if the real actions deviate more from the prior actions before  $h$ , the gain  $\Delta G_i$  for  $i \geq h$  can be smaller, so the actions must be closer to the prior actions in the subsequent rounds, potentially causing a larger regret. Thus, it is important to have a good planing of the action differences  $\{d_i\}_{i=1}^H$  to get larger allowed action deviations for reward optimization. Exploiting the representation power of machine learning, ACRL can learn a good planning of the action differences, and the ACD policy  $\pi^\circ$  corresponding to the optimal ML policy  $\tilde{\pi}^*$  can achieve the optimal planing of the action differences.
Last but not least, Theorem 5.1 shows the effects of the systems parameters in Assumption 3.2 and Assumption 3.4 on the regret through  $\Gamma_{h,h}$  defined in Proposition 4.1 and the minimum cost  $\epsilon$ . Observing that  $\Gamma_{h,h}$  increases with the systems parameters including the Lipschitz parameters  $L_{f}, L_{c}, L_{\pi^{\dagger}}$  and telescoping parameters  $p$ , a higher estimation of the Lipschitz parameters and telescoping parameters can cause a higher regret. Also, a lower estimation of the minimum cost value can cause a higher regret. Therefore, although knowing the upper bound of the Lipschitz parameters and telescoping parameters and the lower bound of the minimum cost value is enough to guarantee the anytime competitive cost constraints by Proposition 4.1 a lower reward regret can be obtained with a more accurate estimation of these system parameters.
5.2 Regret of ACRL
To quantify the regret defined in Eqn. (2), it remains to bound the reward gap between the ACD policy  $\pi^k$  and the optimal ACD policy  $\pi^\circ$ . In this section, we show that  $\pi^k$  by ACRL approaches the optimal one  $\pi^\circ$  as episode  $K \to \infty$  by bounding the pseudo regret
$$
\operatorname {P R e g} (K) = \mathbb {E} _ {x _ {1}} \left[ \sum_ {k = 1} ^ {K} \left(V _ {1} ^ {\pi^ {\circ}} \left(s _ {1}\right) - V _ {1} ^ {\pi^ {k}} \left(s _ {1}\right)\right) \right]. \tag {13}
$$
Theorem 5.2. Assume that the value function is bounded by  $\bar{V}$ . Denote a set of function as
$$
\mathcal {Q} = \left\{q \mid \exists g \in \mathcal {G}, \forall (s, a, v) \in \mathcal {S} \times \mathcal {A} \times \mathcal {V}, q (s, a, v) = \mathbb {E} _ {f \sim g} \left[ v \left(s ^ {\prime}\right) \mid s, a \right] \right\}. \tag {14}
$$
If  $\beta_{k} = 2(\bar{V} H)^{2}\log \left(\frac{2\mathcal{N}(\mathcal{Q},\alpha,\| \cdot \|_{\infty})}{\delta}\right) + C\bar{V} H$  with  $\alpha = 1 / (KH\log (KH / \delta))$ ,  $C$  being a constant, and  $\mathcal{N}(\mathcal{Q},\alpha ,\| \cdot \|_{\infty})$  being the covering number of  $\mathcal{Q}$ , with probability at least  $1 - \delta$ , the pseudo regret of Algorithm 2 is bounded as
$$
\operatorname {P R e g} (K) \leq 1 + d _ {\mathcal {Q}} H \bar {V} + 4 \sqrt {d _ {\mathcal {Q}} \beta_ {K} K H} + H \sqrt {2 K H \log (1 / \delta)}, \tag {15}
$$
where  $d_{\mathcal{Q}} = \dim_E(\mathcal{Q},\frac{1}{KH})$  is the Eluder dimension of  $\mathcal{Q}$  defined in [50].
Theorem 5.2 bounds the pseudo regret for each episode  $k$ . The confidence parameter  $\beta_{k}$  to balance the exploration and exploitation is chosen to get the pseudo regret bound as shown in Theorem 5.2. A higher  $\beta_{k}$  is chosen to encourage the exploration if the covering number of the function space  $\mathcal{Q}$ , the episode length, or the maximum value becomes larger. Also, the pseudo regret relies on the size of the function space  $\mathcal{Q}$  through  $d_{\mathcal{Q}}$  and  $\beta_{K}$ . With smaller  $\lambda$  or  $b$ , less actions satisfy Corollary 4.2 given a state, and so a smaller state-action space  $S \times \mathcal{A}$  is obtained, which results in a smaller size of the function space  $\mathcal{Q}$  and thus a smaller regret.
To get more insights, we also present the overall regret bound when the transition model  $g$  can be represented by a linear kernel as in [6, 70], i.e.  $g(f) = \langle \phi(f), \theta \rangle$  with dimension of  $\theta$  as  $d_{\theta}$ , the reward regret in Eqn2 is bounded as
$$
\operatorname {R e g r e t} (K) \leq K \mathbb {E} _ {y _ {1: H}} \left\{\sum_ {h = 1} ^ {H} L _ {Q, h} \left[ \eta - \frac {1}{\Gamma_ {h , h}} \left(\lambda \epsilon + b + \Delta G _ {h}\right) \right] ^ {+} \right\} + \tilde {O} \left(\sqrt {H ^ {3} \bar {V} ^ {2} K \log (1 / \delta)}\right), \tag {16}
$$
(a) Regret per episode
(b) Regret w.r.t.  $\lambda$
Figure 1: Regret and cost violation rate of different algorithms. Shadows in Figure 2(b) show the range of the regret.
(c) Violation rate w.r.t.  $\lambda$
where  $L_{Q,h}, \eta, \Gamma_{h,h}$ , and  $\Delta G_h$  are all defined in Theorem 5.1. The overall regret bound is obtained because under the assumption of linear transition kernel, we have  $\beta_K = O((\bar{V}H)^2 \log(\frac{1}{\delta} \mathcal{N}(\mathcal{Q}, \alpha, \| \cdot \|_{\infty}))) = \tilde{O}((\bar{V}H)^2 (d_\theta + \log(1/\delta)))$  [6], and the Eluder dimension is  $d_{\mathcal{Q}} = \tilde{O}(d_\theta) \ll 50$ . Thus the pseudo regret is  $\mathrm{PReg}(K) = \tilde{O}(\sqrt{H^3 \bar{V}^2 K \log(1/\delta)})$  which is sublinear in terms of  $K$ . With the sublinear pseudo regret  $\mathrm{PReg}(K)$ , the ACD policy  $\pi^k$  performs as asymptotically well as the optimal ACD policy  $\pi^\circ$  when  $K \to \infty$ . Combining with the regret of the optimal ACD policy in Theorem 5.1, we can bound the overall regret of ACRL. Since in the definition of regret, ACD policy is compared with the optimal-unconstrained policy  $\pi^*$ , the regret bound also includes an unavoidable linear term due to the commitment to satisfy the anytime competitive constraints. The linear term indicates the trade-off between the reward optimization and the anytime competitive constraint satisfaction.
6 Empirical Results
We experiment with the application of resource management for carbon-aware computing [49] to empirically show the benefits of ACRL. The aim of the problem is to jointly optimize carbon efficiency and revenue while guaranteeing the constraints on the quality-of-service (QoS). In this problem, there exists a policy prior  $\pi^{\dagger}$  which directly optimizes QoS based on estimated models. In our experiment, we apply ACRL to optimize the expected reward and guarantee that the real QoS cost is no worse than that of the policy prior. The concrete settings can be found in Appendix A.
Figure 1(a) gives the regret changing the in first 500 episodes. Figure 1(b) shows the regret with different  $\lambda$  and  $b$ , demonstrating the trade-off between reward optimization and the satisfaction of anytime competitive constraints. Figure 1(c) shows the probability of the violation of the anytime competitive constraints by RL and constrained RL. ACRL and ML models with ACD have no violation of anytime competitive constraints. More analysis about the results are provided in Appendix A due to space limitations.
7 Concluding Remarks
This paper considers a novel MDP setting called A-CMDP where the goal is to optimize the average reward while guaranteeing the anytime competitive constraints which require the cost of a learned policy never exceed that of a policy prior  $\pi^{\dagger}$  for any round  $h$  in any episode. To guarantee the anytime competitive constraints, we design ACD, which projects the output of an ML policy into a safe action set at each round. Then, we formulate the decision process of ACD as a new MDP and propose a model-based RL algorithm ACRL to optimize the average reward under the anytime competitive constraints. Our performance analysis shows the tradeoff between the reward optimization and the satisfaction of the anytime competitive constraints.
Future directions. Our results are based on the assumptions on the Lipschitz continuity of the cost, dynamic functions, and the policy prior, as well as the telescoping properties of the policy prior, which are also supposed or verified in other literature [5, 28, 60, 40]. In addition, to guarantee the anytime competitive constraints, the agent is assumed to have access to the Lipschitz constants, the minimum cost value, and the perturbation function. However, since the anytime competitive constraints are much stricter than the expected constraints or the constraints with a high probability, there is no way to guarantee them without any knowledge of the key properties of a mission-critical system. Our work presents the first policy design to solve A-CMDP, but it would be interesting to design anytime-competitive policies with milder assumptions in the future.
We would like to thank the anonymous reviewers for their helpful comments. Jianyi Yang, Pengfei Li and Shaolei Ren were supported in part by the U.S. NSF under the grant CNS-1910208. Tongxin Li was partially supported by the NSFC grant No. 72301234, the Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence, and the start-up funding UDF01002773 of CUHK-Shenzhen. Adam Wierman was supported in part by the U.S. NSF under grants CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648.