SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianyi Yang
UC Riverside Riverside, CA, USA jyang239@ucr.edu
Pengfei Li  
UC Riverside  
Riverside, CA, USA  
pli081@ucr.edu
Tongxin Li  
CUHK Shenzhen  
Shenzhen, Guangdong, China  
litongxin@cuhk.edu.cn
Adam Wierman
Caltech Pasadena, CA, USA adamw@caltech.edu
Shaolei Ren
UC Riverside Riverside, CA, USA shaolei@ucr.edu
This paper studies the problem of Anytime-Competitive Markov Decision Process (A-CMDP). Existing works on Constrained Markov Decision Processes (CMDPs) aim to optimize the expected reward while constraining the expected cost over random dynamics, but the cost in a specific episode can still be unsatisfactorily high. In contrast, the goal of A-CMDP is to optimize the expected reward while guaranteeing a bounded cost in each round of any episode against a policy prior. We propose a new algorithm, called Anytime-Competitive Reinforcement Learning (ACRL), which provably guarantees the anytime cost constraints. The regret analysis shows the policy asymptotically matches the optimal reward achievable under the anytime competitive constraints. Experiments on the application of carbon-intelligent computing verify the reward performance and cost constraint guarantee of ACRL.
In mission-critical online decision-making problems such as cloud workload scheduling [49, 20], cooling control in datacenters [63, 16, 46], battery management for Electrical Vehicle (EV) charging [57, 36], and voltage control in smart grids [54, 73], there is always a need to improve the reward performances while meeting the requirements for some important cost metrics. In these mission-critical systems, there always exist some policy priors that meet the critical cost requirements, but they may not perform well in terms of the rewards. In the application of cooling control, for example, some rule-based heuristics [19, 46] have been programmed into the real systems for a long time and have verified performance in maintaining a safe temperature range, but they may not achieve a high energy efficiency. In this paper, we design a Reinforcement Learning (RL) algorithm with the goal of optimizing the reward performance under the guarantee of cost constraints against a policy prior for any round in any episode.
Constrained RL algorithms have been designed to solve various Constrained MDPs (CMDPs) with reward objective and cost constraints. Among them, some are designed to guarantee the expected cost constraints [61, 62], some can guarantee the cost constraints with a high probability (w.h.p.) [13], and others guarantee a bounded violation of the cost constraints [26, 29, 21, 22, 1]. In addition, conservative RL algorithms [27, 68, 31, 65] consider constraints that require the performance of a learning agent is no worse than a known baseline policy in expectation or with a high probability. [27] points out that it is impossible to guarantee the constraints for any round with probability
Methods	Unknown dynamic	Expected constraints or constraints w.h.p.	Any-episode constraints	Anytime-competitive constraints
With violation	No violation
Learning-augmentation	X	N/A	N/A	38	X
Constrained RL	✓	[29, 21, 22, 11, 67, 26]	[62, 61]	X	X
Conservative RL	✓	N/A	[68, 27, 31, 65]	X	X
ACRL (this work)	✓	N/A	✓	✓	✓
Table 1: Comparison between ACRL and most related works.
one while such guarantees are desirable. In real mission-critical applications, the cost constraints are often required to be satisfied at each round in any episode even in the worst case, but such anytime constraints are not guaranteed by the existing constrained/conservative RL policies. Recently, learning-augmented online control algorithms [37, 38, 69, 51, 17, 35] have been developed to exploit machine learning predictions with the worst-case control performance guarantee. Nonetheless, the learning-augmented control algorithms require the full knowledge of the dynamic models, which limits their applications in many systems with unknown random dynamic models. A summary of most relevant works is given in Table 1.
To fill in this technical blank, we model the mission-critical decision-making problem as a new Markov Decision Process (MDP) which is called the Anytime-Competitive MDP (A-CMDP). In A-CMDP, the environment feeds back a reward and a cost corresponding to the selected action at each round. The next state is updated based on a random dynamic model which is a function of the current action and state and is not known to the agent. The distribution of the dynamic model is also unknown to the agent and needs to be learned. Importantly, at each round  $h$  in any episode, the policy of A-CMDP must guarantee that the cumulative cost  $J_{h}$  is upper bounded by a scaled cumulative cost of the policy prior  $\pi^{\dagger}$  plus a relaxation, i.e.  $J_{h} \leq (1 + \lambda)J_{h}^{\dagger} + hb$  with  $\lambda, b > 0$ , which is called an anytime competitive constraint or anytime competitiveness. The term "competitive" or "competitiveness" is used to signify the performance comparison with a policy prior. Under the anytime cost competitiveness for all rounds, the RL agent explores the policy to optimize the expected reward.
The anytime competitiveness guarantee is more strict than the constraints in typical constrained or conservative MDPs, which presents new challenges for the RL algorithm design. First of all, the anytime competitive constraints are required to be satisfied for any episode, even for the early episodes when the collected sequence samples are not enough. Also, to guarantee the constraints for each round, we need to design a safe action set for each round to ensure that feasible actions exist to meet the constraints in subsequent rounds. Last but not least, without knowing the full transition model, the agent has no access to the action sets defined by the anytime competitive constraints. Thus, in comparison to the control settings with known transition models [38], ensuring the anytime competitiveness for MDPs is more challenging.
Contributions. In this paper, we design algorithms to solve the novel problem of A-CMDP. The contributions are summarized as follows. First, we propose an Anytime-Competitive Decision-making (ACD) algorithm to provably guarantee the anytime competitive constraints for each episode. The key design in ACD is a projection to a safe action set in each round. The safe action set is updated at each round according to a designed rule to gain as much flexibility as possible to optimize the reward. Then, we develop a new model-based RL algorithm (ACRL) to learn the optimal ML model used in ACD. The proposed model-based RL can effectively improve the reward performance based on the new dynamic defined by ACD. Last but not least, we give rigorous analysis on the reward regret of ACRL compared with the optimal-unconstrained policy. The analysis shows that the learned policy performs as well as the optimal ACD policy and there exists a fundamental trade-off between the optimization of the average reward and the satisfaction of the anytime competitive constraints.
Constrained RL. Compared with the existing literature on constrained RL [68, 1, 67, 10, 13, 2, 21, 29, 26, 22, 61], our study has important differences. Concretely, the existing constrained RL works consider an average constraint with or without constraint violation. In addition, existing conservative RL works [68] consider an average constraint compared with a policy prior. However, the constraints can be violated especially at early exploration episodes. In sharp contrast, our anytime competitive
constraint ensures a strict constraint for any round in each episode, which has not been studied in the existing literature as shown in Table I. In fact, with the same policy prior, our anytime competitive policy can also meet the average constraint without violation in conservative/constrained RL [68, 25]. [56] considers MDPs that satisfy safety constraint with probability one and proposes an approach with a high empirical performance. However, there is no theoretical guarantee for constraint satisfaction. Comparably, our method satisfies the constraint with theoretical guarantee, which is essential to deploy AI for mission-critical applications.
Our study is also relevant to safe RL. Some studies on safe RL [10,2,13] focus on constraining that the system state or action at each time  $h$  cannot fall into certain pre-determined restricted regions (often with a high probability), which is orthogonal to our anytime competitiveness requirement that constrains the cumulative cost at each round of an episode. Our study is related to RL with safety constraints [10,39], but is highlighted by the strict constraint guarantee for each round in each episode. In a study on safe RL [10], the number of safety violation events is constrained almost surely by a budget given in advance, but the safety violation value can still be unbounded. By contrast, our work strictly guarantees the anytime competitive constraints by designing the safety action sets. In a recent study [3], the safety requirement is formulated as the single-round cost constraints. Differently, we consider cumulative cost constraints which have direct motivation from mission-critical applications.
Learning-augmented online decision-making. Learning-based policies can usually achieve good average performance but suffer from unbounded worst-case performance. To meet the requirements for the worst-case performance of learning-based policies, learning-augmented algorithms are developed for online control/optimization problems [38, 51, 17, 35, 34]. To guarantee the performance for each problem instance, learning-augmented algorithm can perform an online switch between ML policy and prior [51], combine the ML policy and prior with an adaptive parameter [38], or project the ML actions into safe action sets relying on the prior actions [35]. Compared with learning-augmented algorithms, we consider more general online settings without knowing the exact dynamic model. Also, our problem can guarantee the cost performance for each round in any episode compared with a policy prior, which has not been studied by existing learning-augmented algorithms.
3 Problem Formulation
3.1 Anytime-Competitive MDP
In this section, we introduce the setting of a novel MDP problem called Anytime-Competitive Markov Decision Process (A-CMDP), denoted as  $\mathcal{M}(\mathcal{X},\mathcal{A},\mathcal{F},g,H,r,c,\pi ,\pi^{\dagger})$ . In A-CMDP, each episode has  $H$  rounds. The state at each round is denoted as  $x_{h}\in \mathcal{X},h\in [H]$ . At each round of an episode, the agent selects an action  $a_{h}$  from an action set  $\mathcal{A}$ . The environment generates a reward  $r_h(x_h,a_h)$  and a cost  $c_{h}(x_{h},a_{h})$  with  $r_h\in \mathcal{R}$  and  $c_{h}\in \mathcal{C}$ . We model the dynamics as  $x_{h + 1} = f_h(x_h,a_h)$  where  $f_{h}\in \mathcal{F}$  is a random transition function drawn from an unknown distribution  $g(f_{h})$  with the density  $g\in \mathcal{G}$ . The agent has no access to the random function  $f_{h}$  but can observe the state  $x_{h}$  at each round  $h$ . Note that we model the dynamics in a function style for ease of presentation, and this dynamic model can be translated into the transition probability in standard MDP models [59, 5] as  $\mathbb{P}(x_{h + 1}\mid x_h,a_h) = \sum_{f_h}\mathbb{1}(f_h(x_h,a_h) = x_{h + 1})g(f_h)$ . A policy  $\pi$  is a function which gives the action  $a_{h}$  for each round  $h\in [H]$ . Let  $V_{h}^{\pi}(x) = \mathbb{E}\left[\sum_{i = h}^{H}r_{i}(x_{i},a_{i})\right]\mid x_{h} = x$ . denote the expected value of the total reward from round  $h$  by policy  $\pi$ . One objective of A-CMDP is to maximize the expected total reward starting from the first round which is denoted as  $\mathbb{E}_{x_1}[V_1^\pi (x_1)] = \mathbb{E}\left[\sum_{h = 1}^{H}r_h(x_h,a_h)\right]$ .
Besides optimizing the expected total reward as in existing MDPs, A-CMDP also guarantees the anytime competitive cost constraints compared with a policy prior  $\pi^{\dagger}$ . The policy prior can be a policy that has verified cost performance in real systems or a heuristic policy with strong empirically-guaranteed cost performance, for which concrete examples will be given in the next section. Denote  $y_{h} = (f_{h},c_{h},r_{h})$ , and  $y_{1:H} = \{y_h\}_{h=1}^H \in \mathcal{V} = \mathcal{F} \times \mathcal{R} \times \mathcal{C}$  is a sampled sequence of the models in an A-CMDP. Let  $J_{h}^{\pi}(y_{1:H}) = \sum_{i=1}^{h} c_{i}(x_{i},a_{i})$  be the cost up to round  $h \in [H]$  with states  $x_{i}, i \in [h]$  and actions  $a_{i}, i \in [h]$  of a policy  $\pi$ . Also, let  $J_{h}^{\pi^{\dagger}}(y_{1:H}) = \sum_{i=1}^{h} c_{i}(x_{i}^{\dagger},a_{i}^{\dagger})$  be the cost of the prior with states  $x_{i}^{\dagger}, i \in [h]$  and actions  $a_{i}^{\dagger}, i \in [h]$  of the prior  $\pi^{\dagger}$ . The anytime competitive constraints are defined as below.
Definition 3.1 (Anytime competitive constraints). If a policy  $\pi$  satisfies  $(\lambda, b)$  --anytime competitiveness, the cost of  $\pi$  never exceeds the cost of the policy prior  $\pi^{\dagger}$  relaxed by parameters  $\lambda \geq 0$  and  $b \geq 0$ , i.e. for any round  $h$  in any model sequence  $y_{1:H} \in \mathcal{Y}$ , it holds that  $J_h^\pi(y_{1:H}) \leq (1 + \lambda) J_h^{\pi^{\dagger}}(y_{1:H}) + hb$ .
Now, we can formally express the objective of A-CMDP with  $\Pi$  being the policy space as
$$
\max  _ {\pi \in \Pi} \mathbb {E} _ {x _ {1}} \left[ V _ {1} ^ {\pi} \left(x _ {1}\right) \right], \quad s. t. J _ {h} ^ {\pi} \left(y _ {1: H}\right) \leq (1 + \lambda) J _ {h} ^ {\pi^ {\dagger}} \left(y _ {1: H}\right) + h b, \forall h \in [ H ], \forall y _ {1: H} \in \mathcal {Y}. \tag {1}
$$
Let  $\Pi_{\lambda,b}$  be the collection of policies that satisfy the anytime competitive constraints in (I). We design an anytime-competitive RL algorithm that explores the policy space  $\Pi_{\lambda,b}$  in  $K$  episodes to optimize the expected reward  $\mathbb{E}_{x_1}[V_1^\pi(x_1)]$ . Note that different from constrained/conservative MDPs [26, 29, 61, 68, 11, 67], the anytime competitive constraints in (I) must be satisfied for any round in any sampled episode  $y_{1:H} \in \mathcal{V}$  given relaxed parameters  $\lambda, b \geq 0$ . To evaluate the performance of the learned policy  $\pi^k \in \Pi_{\lambda,b}, k \in [K]$  and the impact of the anytime competitive constraints, we consider the regret performance metric defined as
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \mathbb {E} _ {x _ {1}} \left[ V _ {1} ^ {\pi^ {*}} \left(x _ {1}\right) - V _ {1} ^ {\pi^ {k}} \left(x _ {1}\right) \right], \text {w i t h} \pi^ {k} \in \Pi_ {\lambda , b} \tag {2}
$$
where  $\pi^{*} = \arg \max_{\pi \in \Pi}\mathbb{E}_{x_1}[V_1^{\pi}(x_1)]$  is an optimal policy without considering the anytime competitive constraints. When  $\lambda$  or  $b$  becomes larger, the constraints get less strict and the algorithm has more flexibility to minimize the regret in (2). Thus, the regret analysis will show the trade-off between optimizing the expected reward and satisfying the anytime cost competitiveness.
In this paper, we make additional assumptions on the cost functions, transition functions, and the prior policy which are important for the anytime-competitive algorithm design and analysis.
Assumption 3.2. All the cost functions in the space  $\mathcal{C}$  have a minimum value  $\epsilon \geq 0$ , i.e.  $\forall (x,a),\forall h\in [H],c_{h}(x,a)\geq \epsilon \geq 0$ , and are  $L_{c}$ -Lipschitz continuous with respect to action  $a_{h}$  and the state  $x_{h}$ . All the transition functions in the space  $\mathcal{F}$  are  $L_{f}$ -Lipschitz continuous with respect to action  $a_{h}$  and the state  $x_{h}$ . The parameters  $\epsilon, L_{c}$  and  $L_{f}$  are known to the agent.
The Lipschitz continuity of cost and transition functions can also be found in other works on model-based MDP [45, 5, 28]. The Lipschitz assumptions actually apply to many continuous mission-critical systems like cooling systems [16], power systems [54, 15] and carbon-aware datacenters [49]. In these systems, the agents have no access to concrete cost and transition functions, but they can evaluate the Lipschitz constants of cost and dynamic functions based on the prior knowledge of the systems. The minimum cost value can be as low as zero, but the knowledge of a positive minimum cost  $\epsilon$  can be utilized to improve the reward performance which will be discussed in Section 5.1
Definition 3.3 (Telescoping policy). A policy  $\pi$  satisfies the telescoping property if the policy is applied from round  $h_1$  to  $h_2$  with initialized states  $x_{h_1}$  and  $x_{h_1}^\prime$ , it holds for the corresponding states  $x_{h_2}$  and  $x_{h_2}^\prime$  at round  $h_2$  that
$$
\left\| x _ {h _ {2}} - x _ {h _ {2}} ^ {\prime} \right\| \leq p \left(h _ {2} - h _ {1}\right) \left\| x _ {h _ {1}} - x _ {h _ {1}} ^ {\prime} \right\|, \tag {3}
$$
where  $p(h)$  is called a perturbation function with  $h$  and  $p(0) = 1$ .
Assumption 3.4. The prior policy  $\pi^{\dagger}$  satisfies the telescoping property with some perturbation function  $p$ . Furthermore,  $\pi^{\dagger}$  is Lipschitz continuous.
The telescoping property in Definition 3.3 indicates that with an initial state perturbation at a fixed round, the maximum divergence of the states afterwards is bounded. Thus, the perturbation function  $p$  measures the sensitivity of the state perturbation with respect to a policy prior. The telescoping property is satisfied for many policy priors [60, 43]. It is also assumed for perturbation analysis in model predictive control [42].
Note that in A-CMDP, the constraints are required to be satisfied for any round in any sequence, which is much more stringent than constraint satisfaction in expectation or with a high probability. In fact, the any-time constraints cannot be theoretically guaranteed without further knowledge on the system [8, 56]. This paper firstly shows that Assumption 3.2 and Assumption 3.4, which are reasonable for many mission-critical applications [16, 54, 15, 49], are enough to guarantee the anytime competitive constraints, thus advancing the deployment of RL in mission-critical applications.
3.2 Motivating Examples
The anytime competitiveness has direct motivations from many mission-critical control systems. We present two examples in this section and defer other examples to the appendix.
Safe cooling control in data centers. In mission-critical infrastructures like data centers, the agent needs to make decisions on cooling equipment management to maintain a temperature range and achieve a high energy efficiency. Over many years, rule-based policies have been used in cooling systems and have verified cooling performance in maintaining a suitable temperature for computing [46]. Recently, RL algorithms are developed for cooling control in data centers to optimize the energy efficiency [63, 16, 46]. The safety concerns of RL policies, however, hinder their deployment in real systems. In data centers, an unreliable cooling policy can overheat devices and denial critical services, causing a huge loss [19, 46]. The safety risk is especially high at the early exploration stage of RL in the real environment. Therefore, it is crucial to guarantee the constraints on cooling performance at anytime in any episode for safety. With the reliable rule-based policies as control priors, A-CMDP can accurately model the critical parts of the cooling control problem, opening a path towards learning reliable cooling policies for data centers.
Workload scheduling in carbon-intelligent computing. The world is witnessing a growing demand for computing power due to new computing applications. The large carbon footprint of computing has become a problem that cannot be ignored [49, 64, 52, 23]. Studies find that the amount of carbon emission per kilowatt-hour on electricity grid varies with time and locations due to the various types of electricity generation [33, 32, 9]. Exploiting the time-varying property of carbon efficiency, recent studies are developing workload scheduling policies (e.g. delay some temporally flexible workloads) to optimize the total carbon efficiency [49]. However, an unreliable workload scheduling policy in data centers can cause a large computing latency, resulting in an unsatisfactory Quality of Service (QoS). Thus, to achieve a high carbon efficiency while guaranteeing a low computing latency, we need to solve an A-CMDP which leverages RL to improve the carbon efficiency while guaranteeing the QoS constraints compared with a policy prior targeting at computing latency [20, 30, 12, 72, 71]. This also resembles the practice of carbon-intelligent computing adopted by Google [49].
In this section, we first propose an algorithm to guarantee the anytime competitive constraints for any episode, and then give an RL algorithm to achieve a high expected reward under the guarantee of the anytime competitive constraints.
4.1 Guarantee the Anytime Constraints
It is challenging to guarantee the anytime competitive constraints in (1) for an RL policy in any episode due to the following. First of all, in MDPs, the agent can only observe the real states  $\{x_h\}_{h=1}^H$  corresponding to the truly-selected actions  $\{a_h\}_{h=1}^H$ . The agent does not select the actions  $a_h^\dagger$  of the prior, so the states of the prior  $x_h^\dagger$  are virtual states that are not observed. Thus, the agent cannot evaluate the prior cost  $J_h^{\pi^{\dagger}}$  which is in the anytime competitive constraint at each round  $h$ . Also, the action at each round  $h$  has an impact on the costs in the future rounds  $i, i > h$  based on the random transition models  $f_i, i \geq h$ . Thus, besides satisfying the constraints in the current round, we need to have a good planning for the future rounds to avoid any possible constraint violations even without the exact knowledge of transition and/or cost models. Additionally, the RL policy may be arbitrarily bad in the environment and can give high costs (especially when very limited training data is available), making constraint satisfaction even harder.
Despite the challenges, we design safe action sets  $\{\mathcal{A}_h, h \in [H]\}$  to guarantee the anytime competitive constraints: if action  $a_h$  is strictly selected from  $\mathcal{A}_h$  for each round  $h$ , the anytime competitive constraints for all rounds are guaranteed. As discussed above, the anytime competitive constraints cannot be evaluated at any time since the policy prior's state and cost information is not available. Thus, we propose to convert the original anytime competitive constraints into constraints that only depend on the known parameters and the action differences between the real policy and the policy prior. We give the design of the safe action sets based on the next proposition. For the ease of presentation, we denote  $c_i = c_i(x_i, a_i)$  as the real cost and  $c_i^\dagger = c_i(x_i^\dagger, \pi(x_i^\dagger))$  as the cost of the policy prior at round  $i$ .
Proposition 4.1. Suppose that Assumption  $\boxed{3.2}$  and  $\boxed{3.4}$  are satisfied. At round  $h$  with costs  $\{c_i\}_{i=1}^{h-1}$  observed, the anytime competitive constraints  $J_{h'}^{\pi} \leq (1 + \lambda) J_{h'}^{\pi\dagger} + h' b$  for rounds  $h' = h, \dots, H$  are satisfied if for all subsequent rounds  $h' = h, \dots, H$ ,
$$
\sum_ {j = h} ^ {h ^ {\prime}} \Gamma_ {j, j} \| a _ {j} - \pi^ {\dagger} \left(x _ {j}\right) \| \leq G _ {h, h ^ {\prime}}, \forall h ^ {\prime} = h, \dots , H, \tag {4}
$$
where  $\Gamma_{j,n} = \sum_{i=n}^{H} q_{j,i}$ ,  $(j \in [H], \forall n \geq j)$ , with  $q_{j,i} = L_c \mathbb{1}(j = i) + L_c (1 + L_{\pi^\dagger}) L_f p(i - 1 - j) \mathbb{1}(j < i)$ ,  $(\forall j \in [H], i \geq j)$ , relying on known parameters, and  $G_{h,h'}$  is called the allowed deviation which is expressed as
$$
G _ {h, h ^ {\prime}} = \sum_ {i = 1} ^ {h - 1} \left((1 + \lambda) \hat {c} _ {i} ^ {\dagger} - c _ {i} - \Gamma_ {i, h} d _ {i}\right) + (h ^ {\prime} - h + 1) (\lambda \epsilon + b), \tag {5}
$$