Title: Anytime-Competitive Reinforcement Learning with Policy Prior
Venue: NeurIPS 2023
Accepted: yes
Keywords: Markov Decision Process

Constrained Reinforcement Learning

Anytime Competitive Constraints
Abstract: This paper studies the problem of Anytime-Competitive Markov Decision Process (A-CMDP). Existing works on Constrained Markov Decision Processes (CMDPs) aim to optimize the expected reward while constraining the expected cost over random dynamics, but the cost in a specific episode can still be unsatisfactorily high. In contrast, the goal of A-CMDP is to optimize the expected reward while guaranteeing a bounded cost in each round of any episode against a policy prior. We propose a new algorithm, called Anytime-Competitive Reinforcement Learning (ACRL), which provably guarantees the anytime cost constraints. The regret analysis shows the policy asymptotically matches the optimal reward achievable under the anytime competitive constraints. Experiments on the application of carbon-intelligent computing verify the reward performance and cost constraint guarantee of ACRL.

Anytime-Competitive Reinforcement Learning with Policy Prior

Abstract

This paper studies the problem of Anytime-Competitive Markov Decision Process (A-CMDP). Existing works on Constrained Markov Decision Processes (CMDPs) aim to optimize the expected reward while constraining the expected cost over random dynamics, but the cost in a specific episode can still be unsatisfactorily high. In contrast, the goal of A-CMDP is to optimize the expected reward while guaranteeing a bounded cost in each round of any episode against a policy prior. We propose a new algorithm, called Anytime-Competitive Reinforcement Learning ( ACRL ), which provably guarantees the anytime cost constraints. The regret analysis shows the policy asymptotically matches the optimal reward achievable under the anytime competitive constraints. Experiments on the application of carbonintelligent computing verify the reward performance and cost constraint guarantee of ACRL .

1 Introduction

In mission-critical online decision-making problems such as cloud workload scheduling [49, 20], cooling control in datacenters [63, 16, 46], battery management for Electrical Vehicle (EV) charging [57, 36], and voltage control in smart grids [54, 73], there is always a need to improve the reward performances while meeting the requirements for some important cost metrics. In these missioncritical systems, there always exist some policy priors that meet the critical cost requirements, but they may not perform well in terms of the rewards. In the application of cooling control, for example, some rule-based heuristics [19, 46] have been programmed into the real systems for a long time and have verified performance in maintaining a safe temperature range, but they may not achieve a high energy efficiency. In this paper, we design a Reinforcement Learning (RL) algorithm with the goal of optimizing the reward performance under the guarantee of cost constraints against a policy prior for any round in any episode.

Constrained RL algorithms have been designed to solve various Constrained MDPs (CMDPs) with reward objective and cost constraints. Among them, some are designed to guarantee the expected cost constraints [61, 62], some can guarantee the cost constraints with a high probability (w.h.p.) [13], and others guarantee a bounded violation of the cost constraints [26, 29, 21, 22, 1]. In addition, conservative RL algorithms [27, 68, 31, 65] consider constraints that require the performance of a learning agent is no worse than a known baseline policy in expectation or with a high probability. [27] points out that it is impossible to guarantee the constraints for any round with probability

one while such guarantees are desirable. In real mission-critical applications, the cost constraints are often required to be satisfied at each round in any episode even in the worst case, but such anytime constraints are not guaranteed by the existing constrained/conservative RL policies. Recently, learning-augmented online control algorithms [37, 38, 69, 51, 17, 35] have been developed to exploit machine learning predictions with the worst-case control performance guarantee. Nonetheless, the learning-augmented control algorithms require the full knowledge of the dynamic models, which limits their applications in many systems with unknown random dynamic models. A summary of most relevant works is given in Table 1.

To fill in this technical blank, we model the mission-critical decision-making problem as a new Markov Decision Process (MDP) which is called the Anytime-Competitive MDP (A-CMDP). In A-CMDP, the environment feeds back a reward and a cost corresponding to the selected action at each round. The next state is updated based on a random dynamic model which is a function of the current action and state and is not known to the agent. The distribution of the dynamic model is also unknown to the agent and needs to be learned. Importantly, at each round h in any episode, the policy of A-CMDP must guarantee that the cumulative cost J h is upper bounded by a scaled cumulative cost of the policy prior π † plus a relaxation, i.e. J h ≤ (1 + λ ) J † h + hb with λ , b > 0 , which is called an anytime competitive constraint or anytime competitiveness. The term "competitive" or "competitiveness" is used to signify the performance comparison with a policy prior. Under the anytime cost competitiveness for all rounds, the RL agent explores the policy to optimize the expected reward.

The anytime competitiveness guarantee is more strict than the constraints in typical constrained or conservative MDPs, which presents new challenges for the RL algorithm design. First of all, the anytime competitive constraints are required to be satisfied for any episode, even for the early episodes when the collected sequence samples are not enough. Also, to guarantee the constraints for each round, we need to design a safe action set for each round to ensure that feasible actions exist to meet the constraints in subsequent rounds. Last but not least, without knowing the full transition model, the agent has no access to the action sets defined by the anytime competitive constraints. Thus, in comparison to the control settings with known transition models [38], ensuring the anytime competitiveness for MDPs is more challenging.

Contributions . In this paper, we design algorithms to solve the novel problem of A-CMDP. The contributions are summarized as follows. First, we propose an Anytime-Competitive Decision-making ( ACD ) algorithm to provably guarantee the anytime competitive constraints for each episode. The key design in ACD is a projection to a safe action set in each round. The safe action set is updated at each round according to a designed rule to gain as much flexibility as possible to optimize the reward. Then, we develop a new model-based RL algorithm ( ACRL ) to learn the optimal ML model used in ACD . The proposed model-based RL can effectively improve the reward performance based on the new dynamic defined by ACD . Last but not least, we give rigorous analysis on the reward regret of ACRL compared with the optimal-unconstrained policy. The analysis shows that the learned policy performs as well as the optimal ACD policy and there exists a fundamental trade-off between the optimization of the average reward and the satisfaction of the anytime competitive constraints.

2 Related Work

Constrained RL . Compared with the existing literature on constrained RL [68, 1, 67, 10, 13, 2, 21, 29, 26, 22, 61], our study has important differences. Concretely, the existing constrained RL works consider an average constraint with or without constraint violation. In addition, existing conservative RL works [68] consider an average constraint compared with a policy prior. However, the constraints can be violated especially at early exploration episodes. In sharp contrast, our anytime competitive

constraint ensures a strict constraint for any round in each episode, which has not been studied in the existing literature as shown in Table 1. In fact, with the same policy prior, our anytime competitive policy can also meet the average constraint without violation in conservative/constrained RL [68, 25]. [56] considers MDPs that satisfy safety constraint with probability one and proposes an approach with a high empirical performance. However, there is no theoretical guarantee for constraint satisfaction. Comparably, our method satisfies the constraint with theoretical guarantee, which is essential to deploy AI for mission-critical applications.

Our study is also relevant to safe RL. Some studies on safe RL [10, 2, 13] focus on constraining that the system state or action at each time h cannot fall into certain pre-determined restricted regions (often with a high probability), which is orthogonal to our anytime competitiveness requirement that constrains the cumulative cost at each round of an episode. Our study is related to RL with safety constraints [10, 39], but is highlighted by the strict constraint guarantee for each round in each episode. In a study on safe RL [10], the number of safety violation events is constrained almost surely by a budget given in advance, but the safety violation value can still be unbounded. By contrast, our work strictly guarantees the anytime competitive constraints by designing the safety action sets. In a recent study [3], the safety requirement is formulated as the single-round cost constraints. Differently, we consider cumulative cost constraints which have direct motivation from mission-critical applications.

Learning-augmented online decision-making . Learning-based policies can usually achieve good average performance but suffer from unbounded worst-case performance. To meet the requirements for the worst-case performance of learning-based policies, learning-augmented algorithms are developed for online control/optimization problems [38, 51, 17, 35, 34]. To guarantee the performance for each problem instance, learning-augmented algorithm can perform an online switch between ML policy and prior [51], combine the ML policy and prior with an adaptive parameter [38], or project the MLactions into safe action sets relying on the prior actions [35]. Compared with learning-augmented algorithms, we consider more general online settings without knowing the exact dynamic model. Also, our problem can guarantee the cost performance for each round in any episode compared with a policy prior, which has not been studied by existing learning-augmented algorithms.

3 Problem Formulation

3.1 Anytime-Competitive MDP

In this section, we introduce the setting of a novel MDP problem called Anytime-Competitive Markov Decision Process (A-CMDP), denoted as M ( X , A , F , g, H, r, c, π , π † ) . In A-CMDP, each episode has H rounds. The state at each round is denoted as x h ∈ X , h ∈ [ H ] . At each round of an episode, the agent selects an action a h from an action set A . The environment generates a reward r h ( x h , a h ) and a cost c h ( x h , a h ) with r h ∈ R and c h ∈ C . We model the dynamics as x h +1 = f h ( x h , a h ) where f h ∈ F is a random transition function drawn from an unknown distribution g ( f h ) with the density g ∈ G . The agent has no access to the random function f h but can observe the state x h at each round h . Note that we model the dynamics in a function style for ease of presentation, and this dynamic model can be translated into the transition probability in standard MDP models [59, 5] as P ( x h +1 | x h , a h ) = ∑ f h 1 ( f h ( x h , a h ) = x h +1 ) g ( f h ) . A policy π is a function which gives the action a h for each round h ∈ [ H ] . Let V π h ( x ) = E [ ∑ H i = h r i ( x i , a i )) | x h = x ] denote the expected value of the total reward from round h by policy π . One objective of ACMDP is to maximize the expected total reward starting from the first round which is denoted as E x 1 [ V π 1 ( x 1 )] = E [ ∑ H h =1 r h ( x h , a h )) ] .

Besides optimizing the expected total reward as in existing MDPs, A-CMDP also guarantees the anytime competitive cost constraints compared with a policy prior π † . The policy prior can be a policy that has verified cost performance in real systems or a heuristic policy with strong empiricallyguaranteed cost performance, for which concrete examples will be given in the next section. Denote y h = ( f h , c h , r h ) , and y 1: H = { y h } H h =1 ∈ Y = F × R × C is a sampled sequence of the models in an A-CMDP. Let J π h ( y 1: H ) = ∑ h i =1 c i ( x i , a i ) be the cost up to round h ∈ [ H ] with states x i , i ∈ [ h ] and actions a i , i ∈ [ h ] of a policy π . Also, let J π † h ( y 1: H ) = ∑ h i =1 c i ( x † i , a † i ) be the cost of the prior with states x † i , i ∈ [ h ] and actions a † i , i ∈ [ h ] of the prior π † . The anytime competitive constraints are defined as below.

Definition 3.1 (Anytime competitive constraints) . If a policy π satisfies ( λ , b ) -anytime competitiveness, the cost of π never exceeds the cost of the policy prior π † relaxed by parameters λ ≥ 0 and b ≥ 0 , i.e. for any round h in any model sequence y 1: H ∈ Y , it holds that J π h ( y 1: H ) ≤ (1+ λ ) J π † h ( y 1: H )+ hb.

Now, we can formally express the objective of A-CMDP with Π being the policy space as

Let Π λ ,b be the collection of policies that satisfy the anytime competitive constraints in (1). We design an anytime-competitive RL algorithm that explores the policy space Π λ ,b in K episodes to optimize the expected reward E x 1 [ V π 1 ( x 1 )] . Note that different from constrained/conservative MDPs [26, 29, 61, 68, 1, 67], the anytime competitive constraints in (1) must be satisfied for any round in any sampled episode y 1: H ∈ Y given relaxed parameters λ , b ≥ 0 . To evaluate the performance of the learned policy π k ∈ Π λ ,b , k ∈ [ K ] and the impact of the anytime competitive constraints, we consider the regret performance metric defined as

where π ∗ = arg max π ∈ Π E x 1 [ V π 1 ( x 1 )] is an optimal policy without considering the anytime competitive constraints. When λ or b becomes larger, the constraints get less strict and the algorithm has more flexibility to minimize the regret in (2). Thus, the regret analysis will show the trade-off between optimizing the expected reward and satisfying the anytime cost competitiveness.

In this paper, we make additional assumptions on the cost functions, transition functions, and the prior policy which are important for the anytime-competitive algorithm design and analysis.

Assumption 3.2. All the cost functions in the space C have a minimum value /epsilon1 ≥ 0 , i.e. ∀ ( x, a ) , ∀ h ∈ [ H ] , c h ( x, a ) ≥ /epsilon1 ≥ 0 , and are L c -Lipschitz continuous with respect to action a h and the state x h . All the transition functions in the space F are L f -Lipschitz continuous with respect to action a h and the state x h . The parameters /epsilon1 , L c and L f are known to the agent.

The Lipschitz continuity of cost and transition functions can also be found in other works on modelbased MDP [45, 5, 28]. The Lispchitz assumptions actually apply to many continuous mission-critical systems like cooling systems [16], power systems [54, 15] and carbon-aware datacenters [49]. In these systems, the agents have no access to concrete cost and transition functions, but they can evaluate the Lipschitz constants of cost and dynamic functions based on the prior knowledge of the systems. The minimum cost value can be as low as zero, but the knowledge of a positive minimum cost /epsilon1 can be utilized to improve the reward performance which will be discussed in Section 5.1.

Definition 3.3 (Telescoping policy) . A policy π satisfies the telescoping property if the policy is applied from round h 1 to h 2 with initialized states x h 1 and x ′ h 1 , it holds for the corresponding states x h 2 and x ′ h 2 at round h 2 that

where p ( h ) is called a perturbation function with h and p (0) = 1 .

Assumption 3.4. The prior policy π † satisfies the telescoping property with some perturbation function p . Furthermore, π † is Lipschitz continuous.

The telescoping property in Definition 3.3 indicates that with an initial state perturbation at a fixed round, the maximum divergence of the states afterwards is bounded. Thus, the perturbation function p measures the sensitivity of the state perturbation with respect to a policy prior. The telescoping property is satisfied for many policy priors [60, 43]. It is also assumed for perturbation analysis in model predictive control [42].

Note that in A-CMDP, the constraints are required to be satisfied for any round in any sequence, which is much more stringent than constraint satisfaction in expectation or with a high probability. In fact, the any-time constraints cannot be theoretically guaranteed without further knowledge on the system [8, 56]. This paper firstly shows that Assumption 3.2 and Assumption 3.4, which are reasonable for many mission-critical applications [16, 54, 15, 49], are enough to guarantee the anytime competitive constraints, thus advancing the deployment of RL in mission-critical applications.

3.2 Motivating Examples

The anytime competitiveness has direct motivations from many mission-critical control systems. We present two examples in this section and defer other examples to the appendix.

Safe cooling control in data centers. In mission-critical infrastructures like data centers, the agent needs to make decisions on cooling equipment management to maintain a temperature range and achieve a high energy efficiency. Over many years, rule-based policies have been used in cooling systems and have verified cooling performance in maintaining a suitable temperature for computing [46]. Recently, RL algorithms are developed for cooling control in data centers to optimize the energy efficiency [63, 16, 46]. The safety concerns of RL policies, however, hinder their deployment in real systems. In data centers, an unreliable cooling policy can overheat devices and denial critical services, causing a huge loss [19, 46]. The safety risk is especially high at the early exploration stage of RL in the real environment. Therefore, it is crucial to guarantee the constraints on cooling performance at anytime in any episode for safety. With the reliable rule-based policies as control priors, A-CMDP can accurately model the critical parts of the cooling control problem, opening a path towards learning reliable cooling policies for data centers.

Workload scheduling in carbon-intelligent computing. The world is witnessing a growing demand for computing power due to new computing applications. The large carbon footprint of computing has become a problem that cannot be ignored [49, 64, 52, 23]. Studies find that the amount of carbon emission per kilowatt-hour on electricity grid varies with time and locations due to the various types of electricity generation [33, 32, 9]. Exploiting the time-varying property of carbon efficiency, recent studies are developing workload scheduling policies (e.g. delay some temporally flexible workloads) to optimize the total carbon efficiency [49]. However, an unreliable workload scheduling policy in data centers can cause a large computing latency, resulting in an unsatisfactory Quality of Service (QoS). Thus, to achieve a high carbon efficiency while guaranteeing a low computing latency, we need to solve an A-CMDP which leverages RL to improve the carbon efficiency while guaranteeing the QoS constraints compared with a policy prior targeting at computing latency [20, 30, 12, 72, 71]. This also resembles the practice of carbon-intelligent computing adopted by Google [49].

4 Methods

In this section, we first propose an algorithm to guarantee the anytime competitive constraints for any episode, and then give an RL algorithm to achieve a high expected reward under the guarantee of the anytime competitive constraints.

4.1 Guarantee the Anytime Constraints

It is challenging to guarantee the anytime competitive constraints in (1) for an RL policy in any episode due to the following. First of all, in MDPs, the agent can only observe the real states { x h } H h =1 corresponding to the truly-selected actions { a h } H h =1 . The agent does not select the actions a † h of the prior, so the states of the prior x † h are virtual states that are not observed. Thus, the agent cannot evaluate the prior cost J π † h which is in the anytime competitive constraint at each round h . Also, the action at each round h has an impact on the costs in the future rounds i, i > h based on the random transition models f i , i ≥ h . Thus, besides satisfying the constraints in the current round, we need to have a good planning for the future rounds to avoid any possible constraint violations even without the exact knowledge of transition and/or cost models. Additionally, the RL policy may be arbitrarily bad in the environment and can give high costs (especially when very limited training data is available), making constraint satisfaction even harder.

Despite the challenges, we design safe action sets {A h , h ∈ [ H ] } to guarantee the anytime competitive constraints: if action a h is strictly selected from A h for each round h , the anytime competitive constraints for all rounds are guaranteed. As discussed above, the anytime competitive constraints cannot be evaluated at any time since the policy prior's state and cost information is not available. Thus, we propose to convert the original anytime competitive constraints into constraints that only depend on the known parameters and the action differences between the real policy and the policy prior. We give the design of the safe action sets based on the next proposition. For the ease of presentation, we denote c i = c i ( x i , a i ) as the real cost and c † i = c i ( x † i , π ( x † i )) as the cost of the policy prior at round i .

Proposition 4.1. Suppose that Assumption 3.2 and 3.4 are satisfied. At round h with costs { c i } h -1 i =1 observed, the anytime competitive constraints J π h ′ ≤ (1 + λ ) J π † h ′ + h ′ b for rounds h ′ = h, · · · , H are satisfied if for all subsequent rounds h ′ = h, · · · , H ,

where Γ j,n = ∑ H i = n q j,i , ( j ∈ [ H ] , ∀ n ≥ j ) , with q j,i = L c 1 ( j = i ) + L c (1 + L π † ) L f p ( i -1 -j ) 1 ( j < i ) , ( ∀ j ∈ [ H ] , i ≥ j ) , relying on known parameters, and G h,h ′ is called the allowed deviation which is expressed as

where ˆ c † i = max { /epsilon1 , c i -∑ i j =1 q j,i d j } , ( ∀ i ∈ [ H ]) , is the lower bound of of c † i , and d j = ‖ a j -π † ( x j ) ‖ , ∀ j ∈ [ H ] is the action difference at round j .

At each round h ∈ [ H ] , Proposition 4.1 provides a sufficient condition for satisfying all the anytime competitive constraints from round h to round H given in (1). The meanings of the parameters in Proposition 4.1 are explained as follows. The weight q j,i measures the impact of action deviation at round j on the cost difference | c i -c † i | at round i ≥ j , and the weight Γ j,n indicates the total impact of the action deviation at round j on the sum of the cost differences from rounds n to round H . Based on the definition of q j,i , we get ˆ c † i as a lower bound of the prior cost c † i . With these bounds, we can calculate the maximum allowed total action deviation compared with the prior actions π † ( x j ) from round j = h to h ′ as G h,h ′

By applying Proposition 4.1 at initialization, we can guarantee the anytime competitive constraints for all rounds h ′ ∈ [ H ] if we ensure that for all rounds h ′ ∈ [ H ] , ∑ h ′ j =1 Γ j,j ‖ a j -π † ( x j ) ‖ ≤ G 1 ,h ′ = h ′ ( λ/epsilon1 + b ) . This sufficient condition is a long-term constraint relying on the relaxation parameters λ and b . Although we can guarantee the anytime competitive constraints by the sufficient condition obtained at initialization, we apply Proposition 4.1 at all the subsequent rounds with the cost feedback information to get larger action sets and more flexibility to optimize the average reward. In this way, we can update the allowed deviation according to the next corollary.

Corollary 4.2. At round 1, we initialize the allowed deviation as D 1 = λ/epsilon1 + b . At round h, h > 1 , the allowed deviation is updated as

where R h -1 = ∑ h -1 i =1 ( (1 + λ )ˆ c † i -c i -Γ i,h d i ) with notations defined in Proposition 4.1. The ( λ , b ) -anytime competitive constraints in Definition 3.1 are satisfied if it holds at each round h that Γ h,h ‖ a h -π † ( x h ) ‖ ≤ D h .

Corollary 4.2 gives a direct way to calculate the allowed action deviation at each round. In the update rule (6) of the allowed deviation, the first term of the maximum operation is based on the deviation calculation at round h -1 while the second term is obtained by applying Proposition 4.1 for round h .

We can find that the conditions to satisfy the anytime competitive constraints can be controlled by parameters λ and b . With larger λ and b , the anytime competitive constraints are relaxed and the conditions in Corollary 4.2 get less stringent. Also, the conditions in Corollary 4.2 rely on the minimum cost value /epsilon1 and other system parameters including Lipschitz constants L c , L f , L π † and telescoping parameters p through Γ h,h . Since Γ h,h increases with the Lipschitz and telescoping parameters, even if the estimated Lipschitz constants and the telescoping parameters are higher than the actual values or the estimated minimum cost is lower than the actual value, the obtained condition by Corollary 4.2 is sufficient to guarantee the anytime competitive constraints, although it is more stringent than the condition calculated by the actual parameters.

By Corollary 4.2, we can define the safe action set at each round h as

With the safe action set design in (7), we propose a projection-based algorithm called ACD in Algorithm 1. We first initialize an allowed deviation as D 1 = λ/epsilon1 + b . When the output ˜ a h of the ML model is obtained at each round h , it is projected into a safe action set A h ( D h ) depending on the allowed deviation D h , i.e. a h = P A h ( D h ) (˜ a h ) = arg min a ∈ A h ( D h ) ‖ a -˜ a h ‖ . The projection can be efficiently solved by many existing methods on constrained policy learning [67, 11, 4, 41, 24]. The allowed deviation is then updated based on Corollary 4.2. Intuitively, if the actions are closer to the prior actions before h , i.e. the action deviations { d i } h -1 i =1 get smaller, then R h -1 becomes larger and D h becomes larger, leaving more flexibility to deviate from a † i , i ≥ h in subsequent rounds.

4.2 Anytime-Competitive RL

The anytime competitive constraints have been satisfied by Algorithm 1, but it remains to design an RL algorithm to optimize the average reward under the anytime competitive cost constraints, which is given in this section.

The anytime-competitive decision-making algorithm in Algorithm 1 defines a new MDP, with an additional set of allowed deviations D to the A-MDP defined in Section 3.1, denoted as ˜ M ( X , D , A , F , g, H, r, c, ˜ π , π † ) . In the new MDP, we define an augmented state s h which include the original state x h , the allowed deviation D h ∈ D , and history information { c i } h -1 i =1 and { d i } h -1 i =1 . The transition of x h is defined by f h in Section 3.1 and needs to be learned while the transition of D h is defined in (6) and is known to the agent. The ML policy ˜ π gives an output ˜ a h and the selected action is the projected action a h = P A h ( D h ) (˜ a h ) . Then the environment generates a reward r h ( x h , P A h ( D h ) (˜ a h )) and a cost c h ( x h , P A h ( D h ) (˜ a h )) . Thus, the value function corresponding to the ML policy ˜ π can be expressed as ˜ V ˜ π h ( s h ) = E [ ∑ H i = h r i ( x i , P A h ( D h ) (˜ a h )) ] with ˜ a h being the output of the ML policy ˜ π . For notation convenience, we sometimes write the actions of π ∗ and π † as π ∗ ( s ) and π † ( s ) even though they only reply on the original state x in s .

To solve the MDP, we propose a model-based RL algorithm called ACRL in Algorithm 2. Different from the existing model-based RL algorithms [48, 6, 70], ACRL utilizes the dynamic model of ACMDP and ACD (Algorithm 1) to optimize the average reward. Given a transition distribution g at episode k , we perform value iteration to update ˜ Q functions for h = 1 , · · · , H .

where a h = P A h ( D h ) (˜ a h ) , ˜ Q H +1 ,k ( s, a ) = 0 , ˜ V H +1 ,k ( s ) = 0 . The transition model g is estimated as

Based on the transition estimation, we can calculate the confidence set of the transition model as

Algorithm 2 Anytime-Competitive Reinforcement Learning ( ACRL ) 1: Initialization: Transition model set G 1 = { ˆ g 1 } . 2: for each episode k = 1 , · · · , K do 3: Observe the initial state s k 1 . 4: Select g k = arg max g ∈ G k E g [ V 1 ( s k 1 ) ] . 5: Perform value iteration with g k in Eqn. (8) and update ˜ Q functions ˜ Q k 1 · · · , ˜ Q k H . 6: for each round h = 1 , · · · , H do 7: Run ACD (Algorithm 1) by ML policy ˜ π k ( s h ) = arg max a ∈ A ˜ Q k h ( s h , a ) 8: Observe state s k h +1 and store values ˜ V k h +1 ( s k h +1 ) . 9: end for 10: Update transition model ˆ g k +1 using (9) and calculate confidence set G k +1 . 11: end for

With a learned ML policy ˜ π k at each episode k , the policy used for action selection is the ACD policy π k . Given the optimal ML policy ˜ π ∗ = arg max ˜ π ∈ ˜ Π ˜ V ˜ π 1 ( s 1 ) with ˜ Π being the ML policy space, the optimal ACD policy is denoted as π ◦ . For state s h at round h , π k and π ◦ select actions as

In the definition of A-CMDP, the dimension of the augmented state s h increases with the length of the horizon H , which cloud cause a scalability issue for implementation. The scalability issues also exit in other RL works with history-dependent states [58, 14]. In practice, tractable methods can be designed through feature aggregation [58] or PODMP [66].

5 Performance Analysis

In this section, we analyze the reward regret of ACRL to show the impacts of anytime cost constraints on the average reward.

5.1 Regret due to Constraint Guarantee

Intuitively, due to the anytime competitive constraints in Eqn. (1), there always exists an unavoidable reward gap between an ACD policy and the optimal-unconstrained policy π ∗ . In this section, to quantify this unavoidable gap, we bound the regret of the optimal ACD policy π ◦ , highlighting the impact of anytime competitive cost constraints on the average reward performance.

Theorem 5.1. Assume that the optimal-unconstrained policy π ∗ has a value function Q π ∗ h ( x, a ) which is L Q,h -Lipschitz continuous with respect to the action a for all x . The regret between the optimal ACD policy π ◦ that satisfies ( λ , b ) -anytime competitiveness and the optimal-unconstrained policy π ∗ is bounded as

where η = sup x ∈ X ‖ π ∗ ( x ) -π † ( x )) ‖ is the maximum action discrepancy between the policy prior π † and optimal-unconstrained policy π ∗ ; Γ h,h is defined in Proposition 4.1; ∆ G h = [ R h -1 ] + is the gain of the allowed deviation by applying Proposition 4.1 at round h .

The regret bound stated in Theorem 5.1 is intrinsic and inevitable, due to the committed assurance of satisfying the anytime competitive constraints. Such a bound cannot be improved via policy learning, i.e., converge to 0 when the number of episodes K → ∞ . This is because to satisfy the ( λ , b ) -anytime competitiveness, the feasible policy set Π λ ,b defined under (1) is a subset of the original policy set Π , and the derived regret is an upper bound of max π ∈ Π E x 1 [ V π 1 ( x 1 )] -max π ∈ Π λ ,b E x 1 [ V π 1 ( x 1 )] . Moreover, the regret bound relies on the action discrepancy η . This is because if the optimal-unconstrained policy π ∗ is more different from the prior π † , its actions are altered to a larger extent to guarantee the constraints, resulting in a larger degradation of the reward performance. More importantly, the regret bound indicates the trade-off

between the reward optimization and anytime competitive constraint satisfaction governed by the parameters λ and b . When λ or b becomes larger, we can get a smaller regret because the anytime competitive constraints in (1) are relaxed to have more flexibility to optimize the average reward. In the extreme cases when λ or b is large enough, all the policies in Π can satisfy the anytime competitive constraints, so we can get zero regret.

Moreover, the regret bound shows that the update of allowed deviation by applying Proposition 4.1 based on the cost feedback at each round will benefit the reward optimization. By the definition of R h -1 in Corollary 4.2, if the real actions deviate more from the prior actions before h , the gain ∆ G i for i ≥ h can be smaller, so the actions must be closer to the prior actions in the subsequent rounds, potentially causing a larger regret. Thus, it is important to have a good planing of the action differences { d i } H i =1 to get larger allowed action deviations for reward optimization. Exploiting the representation power of machine learning, ACRL can learn a good planning of the action differences, and the ACD policy π ◦ corresponding to the optimal ML policy ˜ π ∗ can achieve the optimal planing of the action differences.

Last but not least, Theorem 5.1 shows the effects of the systems parameters in Assumption 3.2 and Assumption 3.4 on the regret through Γ h,h defined in Proposition 4.1 and the minimum cost /epsilon1 . Observing that Γ h,h increases with the systems parameters including the Lipschitz parameters L f , L c , L π † and telescoping parameters p , a higher estimation of the Lipschitz parameters and telescoping parameters can cause a higher regret. Also, a lower estimation of the minimum cost value can cause a higher regret. Therefore, although knowing the upper bound of the Lipschitz parameters and telescoping parameters and the lower bound of the minimum cost value is enough to guarantee the anytime competitive cost constraints by Proposition 4.1, a lower reward regret can be obtained with a more accurate estimation of these system parameters.

5.2 Regret of ACRL

To quantify the regret defined in Eqn. (2), it remains to bound the reward gap between the ACD policy π k and the optimal ACD policy π ◦ . In this section, we show that π k by ACRL approaches the optimal one π ◦ as episode K →∞ by bounding the pseudo regret

Theorem 5.2. Assume that the value function is bounded by ¯ V . Denote a set of function as

If β k = 2( ¯ V H ) 2 log ( 2 N ( Q , α , ‖ · ‖ ∞ ) δ ) + C ¯ V H with α = 1 / ( KH log( KH/ δ )) , C being a constant, and N ( Q , α , ‖ · ‖ ∞ ) being the covering number of Q , with probability at least 1 -δ , the pseudo regret of Algorithm 2 is bounded as

where d Q = dim E ( Q , 1 KH ) is the Eluder dimension of Q defined in [50].

Theorem 5.2 bounds the pseudo regret for each episode k . The confidence parameter β k to balance the exploration and exploitation is chosen to get the pseudo regret bound as shown in Theorem 5.2. A higher β k is chosen to encourage the exploration if the covering number of the function space Q , the episode length, or the maximum value becomes larger. Also, the pseudo regret relies on the size of the function space Q through d Q and β K . With smaller λ or b , less actions satisfy Corollary 4.2 given a state, and so a smaller state-action space S × A is obtained, which results in a smaller size of the function space Q and thus a smaller regret.

To get more insights, we also present the overall regret bound when the transition model g can be represented by a linear kernel as in [6, 70], i.e. g ( f ) = 〈 φ ( f ) , θ 〉 with dimension of θ as d θ , the reward regret in Eqn.2 is bounded as

(a) Regret per episode

(b) Regret w.r.t.

λ

(c) Violation rate w.r.t.

λ

where L Q,h , η , Γ h,h , and ∆ G h are all defined in Theorem 5.1. The overall regret bound is obtained because under the assumption of linear transition kernel, we have β K = O (( ¯ V H ) 2 log( 1 δ N ( Q , α , ‖ · ‖ ∞ ))) = ˜ O (( ¯ V H ) 2 ( d θ +log(1 / δ ))) [6], and the Eluder dimension is d Q = ˜ O ( d θ ) [50]. Thus the pseudo regret is PReg( K ) = ˜ O ( √ H 3 ¯ V 2 K log(1 / δ )) which is sublinear in terms of K . With the sublinear pseudo regret PReg( K ) , the ACD policy π k performs as asymptotically well as the optimal ACD policy π ◦ when K →∞ . Combining with the regret of the optimal ACD policy in Theorem 5.1, we can bound the overall regret of ACRL . Since in the definition of regret, ACD policy is compared with the optimal-unconstrained policy π ∗ , the regret bound also includes an unavoidable linear term due to the commitment to satisfy the anytime competitive constraints. The linear term indicates the trade-off between the reward optimization and the anytime competitive constraint satisfaction.

6 Empirical Results

We experiment with the application of resource management for carbon-aware computing [49] to empirically show the benefits of ACRL . The aim of the problem is to jointly optimize carbon efficiency and revenue while guaranteeing the constraints on the quality-of-service (QoS). In this problem, there exists a policy prior π † which directly optimizes QoS based on estimated models. In our experiment, we apply ACRL to optimize the expected reward and guarantee that the real QoS cost is no worse than that of the policy prior. The concrete settings can be found in Appendix A.

Figure 1(a) gives the regret changing the in first 500 episodes. Figure 1(b) shows the regret with different λ and b , demonstrating the trade-off between reward optimization and the satisfaction of anytime competitive constraints. Figure 1(c) shows the probability of the violation of the anytime competitive constraints by RL and constrained RL. ACRL and ML models with ACD have no violation of anytime competitive constraints. More analysis about the results are provided in Appendix A due to space limitations.

7 Concluding Remarks

This paper considers a novel MDP setting called A-CMDP where the goal is to optimize the average reward while guaranteeing the anytime competitive constraints which require the cost of a learned policy never exceed that of a policy prior π † for any round h in any episode. To guarantee the anytime competitive constraints, we design ACD , which projects the output of an ML policy into a safe action set at each round. Then, we formulate the decision process of ACD as a new MDP and propose a model-based RL algorithm ACRL to optimize the average reward under the anytime competitive constraints. Our performance analysis shows the tradeoff between the reward optimization and the satisfaction of the anytime competitive constraints.

Future directions. Our results are based on the assumptions on the Lipschitz continuity of the cost, dynamic functions, and the policy prior, as well as the telescoping properties of the policy prior, which are also supposed or verified in other literature [5, 28, 60, 40]. In addition, to guarantee the anytime competitive constraints, the agent is assumed to have access to the Lipschitz constants, the minimum cost value, and the perturbation function. However, since the anytime competitive constraints are much stricter than the expected constraints or the constraints with a high probability, there is no way to guarantee them without any knowledge of the key properties of a mission-critical system. Our work presents the first policy design to solve A-CMDP, but it would be interesting to design anytime-competitive policies with milder assumptions in the future.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Sanae Amani, Christos Thrampoulidis, and Lin Yang. Safe reinforcement learning with linear function approximation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 243-253. PMLR, 18-24 Jul 2021.

Sanae Amani, Christos Thrampoulidis, and Lin Yang. Safe reinforcement learning with linear function approximation. In International Conference on Machine Learning , pages 243-253. PMLR, 2021.

Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145. PMLR, 2017.

Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning , pages 264-273. PMLR, 2018.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020.

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5:411-444, 2022.

Duncan S Callaway, Meredith Fowlie, and Gavin McCormick. Location, location, location: The variable value of renewable energy and demand-side efficiency resources. Journal of the Association of Environmental and Resource Economists , 5(1):39-75, 2018.

Agustin Castellano, Hancheng Min, Juan Bazerque, and Enrique Mallada. Reinforcement learning with almost sure constraints. In Learning for Dynamics and Control , 2022.

Bingqing Chen, Priya L Donti, Kyri Baker, J Zico Kolter, and Mario Bergés. Enforcing policy feasibility constraints through differentiable projection for energy optimization. In Proceedings of the Twelfth ACM International Conference on Future Energy Systems , pages 199-210, 2021.

Shuang Chen, Christina Delimitrou, and José F Martínez. Parties: Qos-aware resource partitioning for multiple interactive services. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems , pages 107-120, 2019.

Weiqin Chen, Dharmashankar Subramanian, and Santiago Paternain. Policy gradients for probabilistic constrained reinforcement learning, 2022.

Xiaoyu Chen, Xiangming Zhu, Yufeng Zheng, Pushi Zhang, Li Zhao, Wenxue Cheng, Peng Cheng, Yongqiang Xiong, Tao Qin, Jianyu Chen, et al. An adaptive deep rl method for non-stationary environments with piecewise stable context. Advances in Neural Information Processing Systems , 35:35449-35461, 2022.

Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li. Reinforcement learning for decision-making and control in power systems: Tutorial, review, and vision. arXiv , 2021.

Yuri Chervonyi, Praneet Dutta, Piotr Trochim, Octavian Voicu, Cosmin Paduraru, Crystal Qian, Emre Karagozler, Jared Quincy Davis, Richard Chippendale, Gautam Bajaj, et al. Semi-analytical industrial cooling system model for reinforcement learning. arXiv preprint arXiv:2207.13131 , 2022.

Nicolas Christianson, Junxuan Shen, and Adam Wierman. Optimal robustness-consistency tradeoffs for learning-augmented metrical task systems. In AI STATS , 2023.

Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bianchini. Resource central: Understanding and predicting workloads for improved resource management in large cloud platforms. In Proceedings of the 26th Symposium on Operating Systems Principles , pages 153-167, 2017.

Google DeepMind. Safety-first ai for autonomous data centre cooling and industrial control. https://www.deepmind.com/blog/ safety-first-ai-for-autonomous-data-centre-cooling-and-industrial-control , 2018.

Christina Delimitrou and Christos Kozyrakis. Paragon: Qos-aware scheduling for heterogeneous datacenters. ACM SIGPLAN Notices , 48(4):77-88, 2013.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pages 3304-3312. PMLR, 2021.

Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A Smith, Nicole DeCario, and Will Buchanan. Measuring the carbon intensity of ai in cloud instances. In 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1877-1894, 2022.

Priya L Donti, David Rolnick, and J Zico Kolter. Dc3: A learning method for optimization with hard constraints. arXiv preprint arXiv:2104.12225 , 2021.

Yihan Du, Siwei Wang, and Longbo Huang. A one-size-fits-all solution to conservative bandit problems. In AAAI , 2021.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps. arXiv preprint arXiv:2003.02189 , 2020.

Evrard Garcelon, Mohammad Ghavamzadeh, Alessandro Lazaric, and Matteo Pirotta. Conservative exploration in reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 1431-1441. PMLR, 2020.

Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning , pages 2170-2179. PMLR, 2019.

Arnob Ghosh, Xingyu Zhou, and Ness Shroff. Provably efficient model-free constrained rl with linear function approximation. arXiv preprint arXiv:2206.11889 , 2022.

Íñigo Goiri, Kien Le, Md E Haque, Ryan Beauchea, Thu D Nguyen, Jordi Guitart, Jordi Torres, and Ricardo Bianchini. Greenslot: scheduling energy consumption in green datacenters. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-11, 2011.

Sumeet Katariya, Branislav Kveton, Zheng Wen, and Vamsi K Potluru. Conservative exploration using interleaving. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 954-963. PMLR, 2019.

Imran Khan. Temporal carbon intensity analysis: renewable versus fossil fuel dominated electricity systems. Energy Sources, Part A: Recovery, Utilization, and Environmental Effects , 41(3):309-323, 2019.

Mo Li, Timothy M Smith, Yi Yang, and Elizabeth J Wilson. Marginal emission factors considering renewables: A case study of the us midcontinent independent system operator (miso) system. Environmental science & technology , 51(19):11215-11223, 2017.

Pengfei Li, Jianyi Yang, and Shaolei Ren. Expert-calibrated learning for online optimization with switching costs. Proc. ACM Meas. Anal. Comput. Syst. , 6(2), Jun 2022.

Pengfei Li, Jianyi Yang, and Shaolei Ren. Robustified learning for online optimization with memory costs. INDOCOM , 2023.

Tongxin Li, Yue Chen, Bo Sun, Adam Wierman, and Steven H Low. Information aggregation for constrained online control. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 5(2):1-35, 2021.

Tongxin Li, Ruixiao Yang, Guannan Qu, Yiheng Lin, Adam Wierman, and Steven H Low. Certifying black-box policies with stability for nonlinear control. IEEE Open Journal of Control Systems , 2:49-62, 2023.

Tongxin Li, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low. Robustness and consistency in linear quadratic control with untrusted predictions. Proc. ACM Meas. Anal. Comput. Syst. , 6(1), feb 2022.

Yingying Li, Subhro Das, Jeff Shamma, and Na Li. Safe adaptive learning-based control for constrained linear quadratic regulators with regret guarantees. arXiv preprint arXiv:2111.00411 , 2021.

Yingying Li, James A Preiss, Na Li, Yiheng Lin, Adam Wierman, and Jeff Shamma. Online switching control with stability and regret guarantees. arXiv preprint arXiv:2301.08445 , 2023.

Enming Liang, Minghua Chen, and Steven H. Low. Low complexity homeomorphic projection to ensure neural-network solution feasibility for optimization over (non-)convex set. In ICML , 2023.

Yiheng Lin, Yang Hu, Guannan Qu, Tongxin Li, and Adam Wierman. Bounded-regret mpc via perturbation analysis: Prediction error, constraints, and nonlinearity. arXiv preprint arXiv:2210.12312 , 2022.

Yiheng Lin, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman. Perturbation-based regret analysis of predictive control in linear time varying systems. Advances in Neural Information Processing Systems , 34:5174-5185, 2021.

Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language model. arXiv preprint arXiv:2211.02001 , 2022.

Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-based reinforcement learning. arXiv preprint arXiv:2206.09328 , 2022.

Jerry Luo, Cosmin Paduraru, Octavian Voicu, Yuri Chervonyi, Scott Munns, Jerry Li, Crystal Qian, Praneet Dutta, Jared Quincy Davis, Ningjia Wu, et al. Controlling commercial cooling systems using reinforcement learning. arXiv preprint arXiv:2211.07357 , 2022.

California Independent System Operator. Calfornia renewable datasets. https://www.caiso. com/Pages/default.aspx , 2023.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

Ana Radovanovi´ c, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, et al. Carbon-aware computing for datacenters. IEEE Transactions on Power Systems , 38(2):1270-1280, 2022.

Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

Daan Rutten, Nico Christianson, Debankur Mukherjee, and Adam Wierman. Online optimization with untrusted predictions. arXiv preprint arXiv:2202.03519 , 2022.

Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM , 63(12):54-63, 2020.

Mohammad Shahrad, Cristian Klein, Liang Zheng, Mung Chiang, Erik Elmroth, and David Wentzlaff. Incentivizing self-capping to increase cloud utilization. In Proceedings of the 2017 Symposium on Cloud Computing , pages 52-65, 2017.

Yuanyuan Shi, Guannan Qu, Steven Low, Anima Anandkumar, and Adam Wierman. Stability constrained reinforcement learning for real-time voltage control. In 2022 American Control Conference (ACC) , pages 2715-2721. IEEE, 2022.

Youngbin Song, Minjun Park, Minhwan Seo, and Sang Woo Kim. Improved soc estimation of lithium-ion batteries with novel soc-ocv curve estimation method using equivalent circuit model. In 2019 4th International Conference on Smart and Sustainable Technologies (SpliTech) , pages 1-6. IEEE, 2019.

Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning , pages 20423-20443. PMLR, 2022.

Bo Sun, Ali Zeynali, Tongxin Li, Mohammad Hajiesmaili, Adam Wierman, and Danny HK Tsang. Competitive algorithms for the online multiple knapsack problem with application to electric vehicle charging. ACMonMeasurement and Analysis of Computing Systems (POMACS) , 4(3), 2021.

Guy Tennenholtz, Nadav Merlis, Lior Shani, Martin Mladenov, and Craig Boutilier. Reinforcement learning with history-dependent dynamic contexts. ICML , 2023.

Garrett Thomas. Markov decision processes. 2007.

Hiroyasu Tsukamoto, Soon-Jo Chung, and Jean-Jaques E Slotine. Contraction theory for nonlinear stability analysis and learning-based control: A tutorial overview. Annual Reviews in Control , 52:135-169, 2021.

Sharan Vaswani, Lin Yang, and Csaba Szepesvári. Near-optimal sample complexity bounds for constrained mdps. Advances in Neural Information Processing Systems , 35:3110-3122, 2022.

Honghao Wei, Xin Liu, and Lei Ying. Triple-q: A model-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation. In International Conference on Artificial Intelligence and Statistics , pages 3274-3307. PMLR, 2022.

William Wong, Praneet Dutta, Octavian Voicu, Yuri Chervonyi, Cosmin Paduraru, and Jerry Luo. Optimizing industrial hvac systems with hierarchical reinforcement learning. arXiv preprint arXiv:2209.08112 , 2022.

Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems , 4:795-813, 2022.

Yifan Wu, Roshan Shariff, Tor Lattimore, and Csaba Szepesvári. Conservative bandits. In International Conference on Machine Learning , pages 1254-1262. PMLR, 2016.

Yi Xiong, Ningyuan Chen, Xuefeng Gao, and Xiang Zhou. Sublinear regret for learning pomdps. Production and Operations Management , 31(9):3491-3504, 2022.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-based constrained policy optimization. In International Conference on Learning Representations , 2020.

Yunchang Yang, Tianhao Wu, Han Zhong, Evrard Garcelon, Matteo Pirotta, Alessandro Lazaric, Liwei Wang, and Simon Shaolei Du. A reduction-based framework for conservative bandits and reinforcement learning. In International Conference on Learning Representations , 2022.

Runyu Zhang, Yingying Li, and Na Li. On the regret analysis of online lqr control with predictions. In 2021 American Control Conference (ACC) , pages 697-703. IEEE, 2021.

Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning , pages 12793-12802. PMLR, 2021.

Xingyu Zhou, Ness Shroff, and Adam Wierman. Asymptotically optimal load balancing in large-scale heterogeneous systems with multiple dispatchers. ACM SIGMETRICS Performance Evaluation Review , 48(3):57-58, 2021.

Xingyu Zhou, Jian Tan, and Ness Shroff. Flexible load balancing with multi-dimensional state-space collapse: Throughput and heavy-traffic delay optimality. ACM SIGMETRICS Performance Evaluation Review , 46(3):10-11, 2019.

Xinyang Zhou, Masoud Farivar, Zhiyuan Liu, Lijun Chen, and Steven H Low. Reverse and forward engineering of local voltage control in distribution networks. IEEE Transactions on Automatic Control , 66(3):1116-1128, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of anytime constrained RL. Contrary to traditional constrained RL where the constraints are expressed as expected costs, this work looks at a stronger criteria that the cost should be bounded for any episode. The cost is defined relative to the performance of an existing safe policy deployed in the environment of interest.\n\nAn algorithm (ACD) is derived which consists in deriving a safe action set by computing the maximum allowed deviation from the prior policy. The computation relies on the knowledge of Lipschitz constants characterizing the transition model, the cost variation, and the policy prior. This algorithm allows to take safe actions but does not address optimizing the return. \n\nACD is extended to ACRL which adds the learning and optimization parts to maximize the return while taking safe actions. ACRL is a model based algorithm. An analysis of the regret of the algorithm is provided to analyze the regret from the anytime constraint and from the learning part.\n\nSTRENGTHS:\nThe introduced framework of A-CMDP is very sound and fits many practical applications. The main assumption of the work that there exist a policy prior is a strong one but I believe it applies to a lot of problems and is quite relevant. The associated definition of regret makes sense, although the formalism of the A-CMDP is sometimes unclear.\n\nThe article is fairly well written. All propositions and assumptions are followed by an intuitive explanation and they all seem sound. \n\nThe discussion of the related work and their relation to what is being presented is clear and seems to cover most of the relevant topics.\n\nThe structure of deriving first ACD and ACRL is useful to follow the work and the algorithms make sense given the proposed assumptions.\n\nWEAKNESSES:\nThe authors did not include the experiments in the main body of the paper. The experiments show that the algorithm is better in terms of worst case cost, but does not provide the best regret per episode (expected since stronger constraints). The algorithm is evaluated on only one problem, it would have been useful to test it on at least one other problem to show its value, maybe some classical benchmark from the safe RL literature. An ablation study of showing the performance of the model-based approach without the constraints would have been useful as well. Adding the experiments to the main body seems necessary to me.\n\nAlthough the intuitive explanation of the A-CMDP is clear, there are some unclarities in the mathematical definition of the A-CMDP (see questions).\n\nThe main assumptions of the article require the cost functions, transition functions, and prior policy to have some sort of Lipschitz continuity property with known constants. At the same time, a main advantage of the proposed algorithm is that it does not require to know much of the dynamics, it seems a bit contradictory. The author provide some arguments with support from related work but I am still curious whether a precise knowledge of this constant is important or not.\n\nIn ACRL, the state augmentation part increasing significantly the size of state by adding a dependency on the history. This approach raises scalability issues which are not discussed in the paper.\n\nQUESTIONS:\n- I am not sure I understand why the notion of model (y) is needed in the definition the A-CMDP, as opposed to the notion of trajectories more common in the RL literature?\n\n- Why is the value definition an expectation over the first state $x_1$? Shouldn’t it be an expectation over trajectories like in regular MDPs?\n\n- How does one find the necessary Lipschitz constants in a problem where the dynamics is unknown? Is it important to have accurate values?\n\n- In proposition 4.1, how can the cost of the policy prior be known if it is not executed?\n\nLIMITATIONS:\nThe authors are explicitly mentioning the limitations at the end of the paper. However, I find the discussion somewhat contradicting. One of the main related works rely on knowledge of the dynamics and this work relies on knowledge of the Lipschitz constants and some information about the policy prior. Is it really much simpler to get this information compared to an approximate dynamics model?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting work on Constrained Markov Decision Processes (CMDPs) typically formulate a problem where the expected cumulative reward \nis maximized under a constraint where the expected cost over random dynamics is less than some threshold. Such existing work has a drawback that the safety constraint is violated in a specific episode. Hence, the authors study a problem named Anytime-CMDP (A-CMDP). The objective of the authors is to optimize the reward value function while guaranteeing the satisfaction of the safety constraint in each round of any episode. Under several assumptions (e.g., telescoping policy prior, Lipschitz continuity), the authors propose a new algorithm, called Anytime-Constrained Reinforcement Learning (ACRL) and provided theoretical results on constraint satisfaction and regret. The authors conduct experiments on the application of carbon-intelligent computing.\n\nSTRENGTHS:\n- Important problem settings. The drawbacks of the typical CMDPs have been discussed in recent years, so I think this paper will attract much attention within safe RL research community.\n\n- Good theoretical results. Though there are many assumptions, the obtained theoretical results are indeed good. Given the difficulty of the problem (and the theoretical results the authors try to provide), the number and strength of the assumptions are somewhat reasonable.\n\nWEAKNESSES:\n- There are several important missing citations and comparisons. Especially, Ref-1 is very relevant to this paper, which try to guarantee the satisfaction of the safety constraint with probability of 1. This paper is exactly the same motivation to Ref-1 though the approaches are quite different. Though I understand there are pros and cons as follows, it is extremely important to discuss the differences between Ref-1 and this paper. \n\n    ```\n    [Ref-1]\n        + High performance in complicated environments (e.g., Safety-Gym)\n        + High affinity with deep RL\n        + Assumptions are weak\n        - No theoretical guarantee\n    [This paper]\n        + Theoretical guarantee\n        - High affinity with deep RL\n        - Assumptions are strong\n    ```\n\n- Also, anytime constraints have been also studied in several literature. For example, [Ref-2] and [Ref-3] formulate their safety constraint as an instantaneous one $c(s_t, a_t) \\le d$ and propose algorithms to guarantee its satisfaction with high probability (though it is not probably of 1). It seems quite miss-leading that Table 1 show that ACRL is the first and only work to deal with any-time constraint.\n\n- Regarding experiments, the benchmark problem used in this paper is not popular and it would be better to evaluate their method in standard ones (e.g., Safety-Gym). Especially, if I understand correctly, the SOTA algorithm in the problem settings the authors address is Saute-RL in [Ref-1] and the I consider that there should be some comparison between ACRL and Saute-RL.\n\n- There are so many strong assumptions, which makes us to follow this paper. I guess it is partly because the authors try to derive regret bound as well as guarantee safety. I wonder what is the minimal set of assumptions for guaranteeing safety. It may be better to focus on guaranteeing safety with a small number of assumptions, which make this paper easy to follow and the contribution clearer.\n\n- References\n\n    - [Ref-1] Sootla, Aivar, et al. \"Sauté RL: Almost surely safe reinforcement learning using state augmentation.\" International Conference on Machine Learning. PMLR, 2022.\n\n    - [Ref-2] Wachi, Akifumi, and Yanan Sui. \"Safe reinforcement learning in constrained Markov decision processes.\" International Conference on Machine Learning. PMLR, 2020.\n\n    - [Ref-3] Amani, Sanae, Christos Thrampoulidis, and Lin Yang. \"Safe reinforcement learning with linear function approximation.\" International Conference on Machine Learning. PMLR, 2021.\n\nQUESTIONS:\n[Q1] Is there any reason why standard safe RL benchmark problems (e.g., Safety Gym) were not used?\n\n[Q2] What is the minimal set of assumptions that are needed for guaranteeing safety?\n\nLIMITATIONS:\nLimitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a novel problem of Anytime-Constrained Markov Decision Process (A-CMDP), where the goal is to optimize the expected reward while guaranteeing that the cumulative cost in each round of any episode does not exceed a relaxed version of the cost of a policy prior. The paper proposes a projection-based algorithm, called Anytime-Constrained Decision-making (ACD), which ensures the anytime cost constraints by projecting the output of a machine learning policy into safe action sets at each round. The paper also develops an efficient model-based reinforcement learning algorithm, called Anytime-Constrained Reinforcement Learning (ACRL), which learns the optimal ML policy for ACD via value iteration and model estimation. The paper provides theoretical analysis on the regret performance of ACRL, showing the trade-off between reward optimization and anytime constraint satisfaction, as well as the effectiveness of policy learning.\n\nSTRENGTHS:\n- The paper addresses an important and novel problem of A-CMDP, which has direct motivations from many mission-critical applications that require strict cost constraints for safety and reliability.\n- The paper introduces a novel algorithm, ACD, which can guarantee the anytime constraints given any ML policy by projecting it into safe action sets. The paper also develops a model-based RL algorithm, ACRL, which can learn an optimal ML policy to be used in ACD. \n- The paper provides rigorous theoretical analysis on the regret performance of ACRL and shows how it depends on the relaxation parameters and the discrepancy between the policy prior and the optimal-unconstrained policy.\n- The paper presents empirical results on an application of resource management for carbon-aware computing, although all of this part is in appendix.\n\nWEAKNESSES:\n- The paper makes some strong assumptions on the Lipschitz continuity of the cost and transition functions, and the telescoping property of the policy prior, which may limit its applicability to more general environments. Specifically, the telescoping property of the policy prior is a strong assumption for both the policy and the dynamics. It would be interesting to explore how to relax these assumptions or how to estimate them online without compromising safety.\n- The paper also relies on the assumption that the safe policy prior can be queried on any state, even if the state is not explored on the policy prior trajectory. This assumption may not be feasible in real applications.\n- The paper does not provide much intuition or discussion on how the relaxation parameters or other parameters would affect the exploration-exploitation trade-off and the robustness of ACRL.\n\nQUESTIONS:\n- How does ACRL scale with large state-action spaces or long horizons? What would be the challenges there and how would the scale affect the convergence?\n- What are the computational challenges or bottlenecks in implementing ACRL?\n- How sensitive is ACRL to errors or uncertainties in estimating the Lipschitz constants, minimum cost value, or perturbation function?\n- In some applications there might be several constraints. How can ACRL be extended to handle multiple costs or multiple constraints?\n\nLIMITATIONS:\nThe authors have adequately listed most limitations in their work, such as assuming Lipschitz continuity, telescoping property, and known parameters. Those are reasonable to some extent since as stated by the authors, there is no way to guarantee any time safety constraints without any knowledge of the properties of the dynamics. However, they could also discuss some possible extensions or future directions based on their limitations, such as relaxing some assumptions, incorporating online estimation, exploring more applications, or improving scalability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposed a novel problem called Anytime-constrained Markov Decision Process. Unlike CMDP, at each step, a constraint should be satisfied. A projection-based algorithm is proposed to solve the Anytime-constrained MDP. The authors gave a perturbation-based regret analysis for the algorithm.

STRENGTHS:
The paper has great novelty. The anytime constrained MDP is a new problem formulation and has good value for real-world safety-critical problems. The perturbation-based analysis is solid to me.

WEAKNESSES:
Although the problem formulation is novel, the analysis might be weak. The assumptions of known Lipschitz constant, a prior policy, might be too strict. Thus, the regret analysis might have limited guidance for the real-world problem. I have seen that the authors already mentioned it in the limitations.

The writing needs to be improved. Some words in the important part of the paper are confusing. For example, 

1. in Definition 3.3, the initialized and corresponding look quite confusing to me. Initial distribution usually refers to the states at step, or round, 0. The correspondence is also not clear. You might need to add more explanation to lines 153-159.

2. in line 328, the linear kernel assumption is mentioned inline, which is critical to the major contribution of this paper. The assumptions should be highlighted and discussed more.

QUESTIONS:
1. From intuitions of the constrained optimization theory, the projection-based algorithm might not be guaranteed to be optimal with non-convex objectives/constraints. Would you mind giving me some intuitions about why the projection-based algorithm will result in an asymptotically optimal ACD policy?

LIMITATIONS:
The authors addressed the limitations well.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06