Imitating Cost Constrained Behaviors in Reinforcement Learning

Venue: icaps-conference_org_ICAPS_2024_Conference
Published: 2023-12-07 10:25:08

Abstract: Complex planning and scheduling problems have long been solved using various optimization or heuristic approaches. In recent year, imitation learning that aims to learn from expert demonstrations has been proposed as a viable alternative in solving these problems. Generally speaking, imitation learning is designed to learn either the reward (or preference) model or directly the behavioral policy by observing the behavior of an expert. Existing work in imitation learning and inverse reinforcement learning has focused on imitation primarily in unconstrained settings (e.g., no limit on fuel consumed by the vehicle). However, in many real-world domains, the behavior of an expert is governed not only by reward (or preference) but also by constraints. For instance, decisions on self-driving delivery vehicles are dependent not only on the route preferences/rewards (depending on past demand data) but also on the fuel in the vehicle and the time available. In such problems, imitation learning is challenging as decisions are not only dictated by the reward model but are also dependent on a cost constraint model. In this paper, we provide multiple methods that match expert distributions in the presence of trajectory cost constraints through: (a) Lagrangian-based method; (b) Meta-gradients to find a good trade-off between expected return and minimizing constraint violation; and (c) Cost-violation-based alternating gradient. We empirically show that leading imitation learning approaches imitate cost-constrained behaviors poorly and show that our meta-gradient-based approach achieves the best performance.

Main paper: 
 : 
 learning is designed to learn either the reward (or preference) model or directly the behavioral policy by observing the behavior of an expert. Existing work in imitation learning and inverse reinforcement learning has focused on imitation primarily in unconstrained settings (e.g., no limit on fuel consumed by the vehicle). However, in many real-world domains, the behavior of an expert is governed not only by reward (or preference) but also by constraints. For instance, decisions on self-driving delivery vehicles are dependent not only on the route preferences/rewards (depending on past demand data) but also on the fuel in the vehicle and the time available. In such problems, imitation learning is challenging as decisions are not only dictated by the reward model but are also dependent on a cost constraint model. In this paper, we provide multiple methods that match expert distribu-Complex planning and scheduling problems have long been studied in the literature using a wide variety of optimization algorithms and heuristics. One generic way to think about the process of solving these complex problems is to first define a set of states, where a state represents the minimal amount of information required to describe a planning or scheduling instance; then for each state, compute the recommended action by optimizing certain well-defined performance measure. A comprehensive list of (state, action recommendation) tuples is then called the policy for the solved problem, and can be used in practice easily by looking up the encountered states.An alternative paradigm to the above classical approach is to directly "learn" from an expert. More specifically, with a few expert-demonstrated traces of actions, try to generalize and derive actionable policies. Such an approach is generally called "imitation learning", which aims to replicate expert behaviors by directly observing human demonstrations, eliminating the need for designing explicit reward signals as in reinforcement learning (RL) (Abbeel and Ng 2004) . This approach has been successfully applied in a variety of domains such as robotics (Fang et al. 2019) , autonomous vehicles (Kuefler et al. 2017) , game AI (Hussein et al. 2017) , and more recently scheduling (Ingimundardottir and Runarsson 2018) . Typically, this is achieved through techniques such as behavioral cloning (Bain and Sammut 1995) , inverse reinforcement learning (Ng, Russell et al. 2000) , and generative inverse reinforcement learning (GAIL) (Ho and Ermon 2016) .The previous research in the fields of imitation learning and inverse reinforcement learning has primarily concentrated on mimicking human behaviors in unconstrained environments, such as mimicking driving a vehicle without any limitations on fuel consumption by the vehicle. However, in many practical planning and scheduling scenarios, experts consider not only rewards or preferences but also limitations or constraints. For example, the decisions made by a selfdriving delivery vehicle are not only based on route preferences or rewards, which are derived from past demand data but also on the amount of fuel/power available in the vehicle. As another example, when an agent is being trained to drive a car on a race track, the expert demonstrations that the agent is mimicking involve high-speed driving and precision maneuvering, which are critical for success in a race. However, it is also essential for the agent to adhere to safety constraints, such as staying within the track's boundaries and avoiding collisions with other vehicles. These safety constraints differ from the reward function, which may focus on achieving a fast lap time or winning the race. Therefore, the agent must strike a balance between the goal of imitating the expert demonstrations and the need to adhere to the safety constraints in order to successfully complete the task.In scenarios where the decision-making process is influenced by both a reward model and a cost constraint model, the implementation of imitation learning becomes significantly more complex. This is because the decisions made are not solely based on the reward model, but also take into consideration the limitations imposed by the cost constraint model. To that end, we provide a new imitation learning problem in cost-constrained environments.Our work is closely related to Malik et al. (2021) and Cheng et al. (2023) . In Malik et al. (2021) , cost constraints have to be learned from expert trajectories in scenarios where the reward function is already specified. On the other hand, in Cheng et al. (2023) , both safety constraint (maximum cost limit) and cost signals are known and provided, while the reward signal remains undisclosed. The major differences of our proposed work to the above two past studies are that we assume cost signals (from sensors) from the environment are known and provided, but the safety constraint (maximum cost limit) is not given. In other words, we assume that we have necessary sensors to monitor cost signals (e.g., battery level and temperature of an EV), yet we do not know the constraints on these sensor values (e.g., safety ranges on battery level and temperature).In terms of the solution approach, our work is similar to Cheng et al. (2023) in that we also utilize the combination of the Lagrangian-base method and the GAIL framework (Ho and Ermon 2016). However, since we do not assume knowledge of the safety constraint, we cannot include the maximum cost limit in our objective function during the training process.The major challenges we plan to address with this paper are approaches that could execute imitation learning while ensuring cost threshold constraint, which is revealed only via the observations of expert trajectories. Our key contributions are summarized as follows:• First, we formulate the cost-constrained imitation learning problem which represents the challenge of imitation learning in cost-constrained environments (where cost signals are known).• We propose three methods to address the costconstrained imitation learning problems. First, we design a Lagrangian-based method utilizing a three-way gradient update to solve the cost-constrained imitation learning problem. We then provide a meta-gradient approach that is able to tune the Lagrangian penalties of the first approach to significantly improve the performance. Finally, we use a cost-violation-based alternating gradient approach which has a different gradient update depending on the current solution feasibility.• To further validate the effectiveness of our proposed method, we conducted extensive evaluations in Safety Gym environments (Ray, Achiam, and Amodei 2019) and MuJoCo environments (Todorov, Erez, and Tassa 2012) . Our numerical evaluations show that the ensemble of our three proposed approaches can effectively imitate expert behavior while satisfying cost constraints.In this section, we describe the two problem models of relevance in this paper, namely Constrained MDPs and ImitationLearning. We also briefly review related work.Reinforcement Learning problems are characterized by an underlying Markov Decision Process (MDP), which is defined by the tuple (S, A, R, P). Where S represents the set 145 of states, A represents the set of actions. The reward function, R : S × A → R, provides a quantitative measure of how well the system is performing based on the current state and action. The transition function, P : S × A × S → [0, 1], defines the probability of transitioning from one state to an-150 other, given the current state and action taken. Specifically, the probability of transitioning from state s to s ′ , given that action a is taken, is represented by P(s ′ |s, a). A feasible set of policies, denoted as Π, contains all possible policies that can be implemented in the system. The objective of MDP is 155 to find an optimal policy, π ∈ Π, by maximizing the rewardbased objective function, which is defined as follows:EQUATIONIn this work, we examine the scenario in which agents aim to optimize their rewards while adhering to policy-based cost constraints. This leads to an extension of the traditional 160 MDP framework referred to as the Constrained Markov Decision Process (CMDP) (Altman 1999) . The objective in a CMDP problem is succinctly formulated as:max π∈Π E π [ ∞ t=0 γ t r(s t , a t )], s.t. E π [ ∞ t=0 γ t d(s t , a t )] ≤ d 0 . (2)Where d(s, a) is the cost associated with taking action a in state s and is independent of the reward function, r(s, a). 165 d 0 is the expected cost threshold for any selected policy. There have been numerous approaches proposed for solving Constrained MDPs (Satija, Amortila, and Pineau 2020; Pathmanathan and Varakantham 2023) , when the reward and transition models are not known a priori.Methods of Reinforcement Learning require clearly defined and observable reward signals, which provide the agent with feedback on their performance. However, in many realworld scenarios, defining these rewards can be very chal-175 lenging. Imitation learning, on the other hand, offers a more realistic approach by allowing agents to learn behavior in an environment through observing expert demonstrations, without the need for accessing a defined reward signal.An effective method for addressing imitation learning 180 challenges is Behavior Cloning (BC) (Bain and Sammut 1995) . This approach utilizes the states and actions of a demonstrator as training data, allowing the agent to replicate the expert's policy (Pomerleau 1991) . One of the advantages of this method is that it does not require the agent to actively 185 interact with the environment, instead, it operates as a form of supervised learning, similar to classification or regression. Despite its simplicity, BC is known to suffer from a significant drawback: the compounding error caused by covariate shift (Ross, Gordon, and Bagnell 2011) . This occurs when 190 minor errors accumulate over time, ultimately resulting in a significantly different state distribution.Another approach, Inverse Reinforcement Learning (IRL) (Ng, Russell et al. 2000) aims to identify the underlying reward function that explains the observed behavior of an expert. Once the reward function is determined, a standard Reinforcement Learning algorithm can be used to obtain the optimal policy. The reward function is typically defined as a linear (Ng, Russell et al. 2000; Abbeel and Ng 2004) or convex (Syed, Bowling, and Schapire 2008) combination of 200 the state features, and the learned policy is assumed to have the maximum entropy (Ziebart et al. 2008) or maximum causal entropy (Ziebart, Bagnell, and Dey 2010) . However, many IRL methods are computationally expensive and may produce multiple possible formulations for the true reward 205 function. To address these challenges, Generative Adversarial Imitation Learning (GAIL) (Ho and Ermon 2016) was proposed. GAIL directly learns a policy by using a discriminator to distinguish between expert and learned actions, with the output of the discriminator serving as the reward signal.A more recent method, known as Inverse soft-Q Learning (IQ-Learn) (Garg et al. 2021) , takes a different approach by learning a single Q-function that implicitly represents both the reward and policy, thereby avoiding the need for adversarial training. With their state-of-the-art performance in various applications, we designate GAIL and IQ-learn as baselines for our algorithms.We now describe the imitation learning problem and GAIL approach here as it serves as the basis for our method. The learner's goal is to find a policy, denoted as π, that performs at least as well as an expert policy, denoted as π E , with respect to an unknown reward function, denoted as r(s, a). For a given policy π ∈ Π, we define its occupancy measure, denoted as ρ π ∈ Γ, as (Puterman 2014 )ρ π (s, a) = π(a|s) ∞ t=0 γ t P (s t = s|π)The occupancy measure represents the distribution of stateaction pairs that an agent encounters when navigating the environment with the specified policy π. It is important to note that there is a one-to-one correspondence between the set of policies, Π, and the set of occupancy measures, Γ. Therefore, an imitation learning problem can be equivalently formulated as a matching learning problem between the occupancy measure of the learner's policy, ρ π , and the occupancy measure of the expert's policy, ρ π E . In general, the objective can be succinctly represented as the task of finding a policy that closely matches the occupancy measure of the expert's policy, which is represented as:EQUATIONwhereH(π) ≜ E π [-log π(a|s)]is the causal entropy of the policy π, which is defined as the expected value of the negative logarithm of the probability of choosing an action a given a state s, under the distribution of the policy π.Additionally, the distance measure between the state-action distribution of the policy π, represented by ρ π , and the expert's state-action distribution, represented by ρ π E , is represented by the symbol ψ * . Specifically, the distance measure (Jensen-Shannon divergence) employed by the GAIL framework is defined as follows:ψ * (ρ π -ρ π E ) = max D E π [log D(s, a)]+E π E [log(1-D(s, a))](4) The GAIL method utilizes a combination of imitation learning and generative adversarial networks, where D ∈ (0, 1) S×A acts as the discriminator. Through this formalism, the method trains a generator, represented by π θ , to generate state-action pairs that the discriminator attempts to distinguish from expert demonstrations. The optimal policy is achieved when the discriminator is unable to distinguish between the data generated by the generator and the expert data.In our problem, as we aim to address the imitation learning problem within the constraints of an MDP, we have employed a unique distance measure that diverges from the traditional GAIL framework. This approach allows us to more effectively navigate the complexities of the constrained MDP setting and achieve our desired outcome.In this section, we first describe the problem of costconstrained imitation learning and outline our approach to compute the policy that mimics expert behaviors while satisfying the cost constraints.We work in the γ-discounted infinite horizon setting, and we denote the expected reward and cost in association with policy π ∈ Π as:E π [r(s, a)] ≜ E π [ ∞ t=0 γ t r(s t , a t )] and E π [d(s, a)] ≜ E π [ ∞ t=0 γ t d(s t , a t )], where s 0 ∼ p 0 , a t ∼ π(•|s t ), and s t+1 ∼ P(•|s t , a t ) for t ≥ 0. Formally, the Cost-Constrained Imitation Learning problem is a combination of the CMDP problem (2) and the Imitation Learning problem (3), and can be characterized as:EQUATIONThe imitation learning objective has two terms. The first term maximizes the entropy of the policy (to ensure no feasible alternative is excluded), while the second term minimizes the difference in occupation measures obtained using the policy and the expert trajectories. The constraint ensures that the expected cost computed using the policy is lower than the expected cost of the expert trajectories. In problems of interest, the cost function d(s, a) is known but the reward is unknown.Our generative approach to computing the policy that mimics the behavior of the expert within cost constraints relies on computing a solution to the unconstrained objective function of Equation ( 6) below. The theoretical justification for choosing this objective function is provided in the Appendix, and our theoretical analysis is based on the GAIL framework. Equation ( 6) is composed of three optimizations:• Minimize the distance between state, action distributions of the new policy, π θ , and expert policy, π E . This is transformed into the loss associated with a discriminator, D ω , which discriminates between (state, action) pairs from experts, and (state, action) pairs generated by the new policy π θ .• Maximize the entropy of the new policy, π θ , to ensure none of the correct policies are ignored. • Minimize the cost constraint violations corresponding to the new policy, π θ .EQUATIONwhere θ represents the parameters of the policy, β is the parameter corresponding to the causal entropy (since we maximize entropy similar to imitation learning) and finally, λ is the Lagrangian multiplier corresponding to the cost constraint.H(π θ ) ≜ E π θ [-log π θ (a|s)]is the casual entropy of policy π θ .The given expert policy is represented by π E , and a known cost function, represented by d, is also incorporated into the objective function.Given the three optimization components, we do not choose one of the three but instead compute a saddle point (θ, ω, λ) for (6). To accomplish this, we employ a parameterized policy, represented by π θ , with adjustable weights θ, as well as a discriminator network, represented by D ω , which maps states and actions to a value between 0 and 1, and has its own set of adjustable weights ω. The Lagrangian multiplier, denoted by λ, is for penalizing the number of cost constraint violations.To obtain the saddle point, we update the parameters of policy, discriminator, and Lagrangian multiplier sequentially:Updating ω: The gradient of (6) with respect to ω is calcu-315 lated as:EQUATIONWe utilize the Adam gradient step method (Kingma and Ba 2014) on the variable ω, targeting the maximization of (6) in relation to D. Updating θ: To update policy parameters, we adopt the Trust Region Policy Optimization (TRPO) method (Schulman et al. 2015a). The theoretical foundation of the TRPO update process involves utilizing a specific algorithm to improve the overall performance of the policy by optimizing the parameters within a defined trust region:EQUATION)The key challenge in applying the TRPO update process is the computation of the surrogate advantage, denoted by L(θ k , θ). It is a metric that quantifies the relative performance of a new policy π θ in comparison to an existing policy π θ k , based on data collected from the previous policy:EQUATIONWe do not have a reward function to compute the advantage and hence we utilize the output of the discriminator, represented by -log D ω (s, a), as the reward signal. Subsequently, we employ the Generalized Advantage Estimation (GAE) method outlined in Schulman et al. (2015b) to calculate the advantage of the reward, A π θ k r (s, a). Additionally, we also calculate the advantage pertaining to cost, denoted as A π θ k d (s, a), by utilizing the GAE method, as we know the cost function.The average KL-divergence, represented by DKL (θ||θ k ), between policies across states visited by the previous policy, can be computed as:EQUATIONUpdating ϕ r , ϕ d : We do an Adam gradient step on ϕ r (reward value network parameters),ϕ d (cost value network parameters)to minimize the mean-squared error of reward value and cost value, as we minimize these two loss functions:EQUATIONHere Rr t , Rd t are the reward to go and cost to go, which are calculated by the GAE method. Updating λ: We do an Adam gradient step on λ to increase (6), the gradient of ( 6) with respect to λ is calculated as:EQUATIONAlgorithm 1 shows the pseudocode for our approach, CCIL (Cost ConstraIned Lagrangian).In this section, we introduce a meta-gradient approach to improve the Lagrangian method introduced in the previous section by applying cross-validation to optimize the Lagrangian multipliers. We call this approach MALM.Meta-gradient is a strategy designed for the optimization of hyperparameters, such as the discount factor and learning rates in Reinforcement Learning problems. This approach involves the simultaneous execution of online crossvalidation while pursuing the optimization objective of reinforcement learning, such as the maximization of expected return (Xu, van Hasselt, and Silver 2018; Calian et al. 2020 ). The goal is to optimize both inner and outer losses. The update of parameters on the inner loss is to update the parameters of the policy. The outer loss can be based on measures such as policy gradient loss and temporal difference loss, and is optimized by updating hyperparameters (Calian et al. 2020) Instead of optimizing the hyperparameters, the key idea of MALM is to update the Lagrangian multiplier such that there is a better balance between reward maximization and Algorithm 1: Cost ConstraIned Lagrangian (CCIL) Input: initial parameters of policy θ, reward value network ϕ r , cost value network ϕ d , discriminator network ω, batch size K, a set of expert trajectories Φ E = {τ E ∼ π E }, initial Lagrangian multipliers λ, entropy parameter β, learning rates α r , α d , α λ , α ω . Output: Optimal policy π θ 1: Compute the average cost of expert trajectories:J E = 1 |Φ E | τ ∈Φ E T t=1 d t 2: for k = 1, 2, ... do 3:Collect set of learner's trajectories Φ k = {τ i } by running policy π θ k for K time steps.Collect the reward r t of K time steps by using the discriminator output:r t = -log(D ω (s t , a t )) Compute the average episode cost of learner's trajectories:J k = 1 |Φ k | τ ∈Φ k T t=1 d t 8:Update policy by using TRPO rule:θ ′ = arg max θ K t=1 πθ(at|st) πθ k (at|st) (A r (s t , a t ) -λA d (s t , a t )) + βH(π θk ) 9:Update reward value network:ϕ ′ r ← ϕ r -1 K K t=1 α r ▽ ϕr (V r ϕr (s t ) -Rr t ) 2 10:Update cost value network:ϕ ′ d ← ϕ d -1 K K t=1 α d ▽ ϕ d (V d ϕ d (s t ) -Rd t ) 2 11:Update discriminator network:ω ′ ← ω + 1 K K t=1 α ω (▽ ω [log(D ω (s t , a t ))]+ ▽ ω [log(1 -D ω (s t , a t ))]) 12:Update Lagrangian multipliers:λ ′ ← λ + α λ (J k -J E ) 13: θ ← θ ′ , ϕ r ← ϕ ′ r , ϕ d ← ϕ ′ d , ω ← ω ′ , λ ← λ ′. 14: end for cost constraint enforcement. Specifically, we use the outer loss that is defined as follows:L outer (λ) = E π θ (A r (s, a) -λd(s, a)) 2 (13)Every batch is divided into training and validation data sets. The parameter update equations for the training data set are the same as described in the Lagrangian-based method (Equations ( 8) -( 12)), and for the validation data set, we update the Lagrangian multiplier by minimizing the above outer loss function. MALM is similar to Algorithm 1 except that from lines 7 to 12, we update policy parameters, discriminator parameters, and Lagrangian multipliers based on the training data set. We also have an additional procedure to update the Lagrangian multipliers by minimizing the outer loss function based on the validation data set (see the Appendix for detailed pseudocode). grangian multipliers. Like previous methods, this method also maintains a policy network, θ, reward value network, ϕ r , and cost value network, ϕ d . The key novelty of this approach is in doing a feasibility check-based gradient update that is fairly intuitive. If the average episode cost of the learner does not exceed the average episode cost of experts (cost constraint), then we update the policy parameters towards the direction of maximizing the return, which would be the following equation in TRPO:EQUATIONOtherwise, we update the policy towards the direction of minimizing the costs.EQUATIONThe detailed pseudocode is provided in the Appendix.Experiments In this section, we compare our approaches against leading approaches for imitation learning (IL), including GAIL (Ho and Ermon 2016), IQ-Learn (Garg et al. 2021) , Behavioral Cloning (BC) (Bain and Sammut 1995) , and LGAIL (Cheng et al. 2023 ) in cost-constrained environments. This is to illustrate that a new approach is needed to mimic expert behaviors when there are unknown cost constraints. As we will soon demonstrate, all baselines suffer from either extensive cost constraint violations or low reward.Environments and Cost Definition We selected Safety Gym (Ray, Achiam, and Amodei 2019) and MuJoCo (Todorov, Erez, and Tassa 2012) , two well-known environments from the literature for our evaluation purpose.The Safety Gym environment is a standard platform for evaluating constrained RL algorithms. Each instance of this environment features a robot tasked with navigating a cluttered space to achieve a goal, all while adhering to specific constraints governing its interactions with objects and surrounding areas. In our experiments, there are three robotic agents (Point, Car, and Doggo) and two task types (Goal and Button), resulting in a total of six unique scenarios. The difficulty level for these environments is standardized to 1. Throughout each timestep, the environment provides distinct cost signals for various unsafe elements, each linked Table 2 : Overall performance of different environments. Normalized penalized return R pen captures the trade-off between achieving higher rewards and making the episode cost go below the expert's episode cost, higher is better. Recovered return R rec evaluates the extent that the agent imitates the expert's behavior, higher R rec means the agent imitated the experts better. Cost-Vio captures the extent that agent's episode cost goes beyond the expert's episode cost, lower is better. to an associated constraint. Additionally, an aggregate cost signal is provided, encapsulating the overall impact of the agent's interactions with unsafe elements. The cost functions are straightforward indicators, evaluating whether an unsafe interaction has occurred (c t = 1 if the agent engages in an unsafe action, otherwise c t = 0). In order to illustrate the robustness of our methods, we also adopt MuJoCo for its wide array of continuous control tasks, such as Walker2d and Swimmer. These tasks are frequently employed to assess RL and IL algorithms. For reward, we utilized the default MuJoCo environment settings, however, as the MuJoCo environment does not have built-in cost constraints like in Safety Gym, we introduce constraints on features of the state space and action space:• Control Cost. The agent faces penalties for taking excessively large actions in HalfCheetah-v3 and Hopper-v3. • Control Cost plus Contact Cost. This is utilized in Ant-450 v3 and Humanoid-v3. The contact cost of an agent is generated if the external contact force is too large, the cost indicator is the sum of control cost and contact cost. • Speed Limit: This is utilized in Swimmer-v3 and Walker2d-v3. An agent is penalized when it moves at 455 a much higher speed.Safety coefficient is the cost threshold for the cost indicator. At each time step, if the cost indicator is larger than the safety coefficient, then the agent will get a cost of 1, otherwise, the cost is denoted as 0. The expert trajectories 460 were generated by solving a forward-constrained RL problem, and the statistics of these trajectories and Safety Coefficient parameters are summarized in Table 1 .Baselines and Code. In order to evaluate the performance of our algorithms, we performed a comparison with three popular IL methods without explicit cost constraints consideration (GAIL, IQ-Learn, and BC). For the constraint-aware IL method, we use LGAIL as the baseline. As noted earlier, LGAIL assumes the knowledge of cost limit d ′ , which is used as a parameter in their objective function:EQUATIONAs they defined d ′ to be less than the minimal episode cost of expert trajectories, so we use the 90% of minimal episode cost of expert trajectories as d ′ . In the case of BC, the expert trajectories dataset, which consists of state-action pairs, was divided into a 70% training data set and a 30% validation data set. The policy was then trained using supervised learning techniques. Then in the GAIL method, the policy network, reward value network, cost value network, and discriminator network all employ the same architectures, comprising two hidden layers of 100 units each, with tanh non-480 linearities being utilized in the layers. Lastly, for the IQ-Learn method, we use the same setting as illustrated in (Garg et al. 2021 ), that we use critic and actor networks with an MLP architecture with 2 hidden layers and 256 hidden units.Implementation. We employ a neural network architecture consistent with the one utilized in the GAIL method. In addition, our approaches add a cost value network and a Lagrangian penalty term λ (in CCIL and MALM), which distinguishes our method from GAIL. The policy, value, and cost value network are optimized through gradient descent with the Adam optimizer (Kingma and Ba 2014) . The initial value of λ is set to 0.01 and optimized using the Adam optimizer. In MALM, the state-action pairs of each batch size are divided into a 70% training data set and a 30% validation data set. We run each algorithm for 5 different random seeds. The algorithms ran for 2000 time steps (batch size) during each iteration, and the episode's total true reward and cost are recorded. The details of the hyper-parameters used in the experiments can be found in the Appendix. The implementation of all code is based on the OpenAI Baselines library (Dhariwal et al. 2017) .Performance Metrics. We use different performance metrics to compare overall performance. Firstly, followed by (Ray, Achiam, and Amodei 2019), we record the average episode's true reward, the average episode's cost, and the cost rate over the entirety of training (the sum of all costs divided by the total number of environment interaction steps) throughout the training. We also incorporate the normalized penalized return introduced by (Calian et al. 2020) . This metric effectively captures the delicate balance between maximizing rewards and ensuring that the episode cost remains below the expert's episode cost. The formulation is represented asR pen = R/R E -K max(J π d /J π E d -1, 0), where R and J π d denote the average episode true reward/cost for the algorithm upon convergence (computed as an average over the last 100 iterations), and R E , J π E d represent the average episode reward/cost of the expert. The second term in the equation introduces a penalty if the episode cost exceeds the expert cost; otherwise, this term remains zero. The constant K serves as a fixed parameter determining the weight assigned to the constraint violation penalty. To effectively penalize algorithms consistently breaching cost constraints during evaluation, we set K = 1.2. The recovered return metric captures the degree to which the agent replicates the expert's behavior, denoted as R rec = R/R E ×100.A value of R rec ≥ 100 signifies proficient imitation of the expert, while an approach to 0 indicates the agent's inability to replicate the expert's behavior (with R rec being 0 when returns are negative).Finally, cost violation is defined asϕ d = max(0, J π d -J π E d ).In cases where the agent's episode cost is less than the expert's episode cost, denoting no cost violation, ϕ d is set to 0.In order to assess the effectiveness of each algorithm, we used average episode true reward, average episode cost, and cost rate at each iteration as an evaluation. Here are key observations from Figures 1 (Training curves for these three evaluation metrics in Mujoco and DoggoButton tasks can be found in the Appendix):• In the BC method, which doesn't consider cost constraints, three key observations stand out. First, in tasks like Hopper, both reward and cost remain consistently low. Second, while the reward tends to approach expert levels, the cost often surpasses the constraint or the expert's cost during training. Lastly, there's a pattern where 545 the average return initially rises but later drops in tasks like CarGoal and PointButton. This might be due to BC being prone to overfitting. Initially, the model fits the training data well, boosting performance. However, as training continues, the model can become too special-550 ized, struggling to handle new situations. Additionally, the cost rate in this method is relatively high compared to other approaches. • The GAIL method, which also overlooks cost constraints, exhibited a pattern wherein the reward approached the expert reward during training. However, the cost consistently exceeded the expert's cost throughout the training stage, and the cost rate was notably the highest across most environments. • For the IQ-Learn method, the cost of the policy obtained was generally lower than that of the expert cost in most tasks. However, the reward obtained was well below the expert reward.• The LGAIL method can achieve performance close to the expert's reward in certain tasks, demonstrating com-565 petitive efficacy compared to CCIL. However, it tends to incur higher costs and cost rates throughout the training process. • Our approach, MALM, demonstrated superior performance compared to all other methods, consistently offering the optimal balance of reward and adherence to cost constraints. Notably, in the HalfCheetah task, MALM excelled in approximating expert behavior, outperforming other methods that exhibited a trend of decreasing rewards or surpassing the expert's cost. Following MALM, CVAG, and CCIL secured the second-best performance, surpassing GAIL, BC, and IQ-Learn in terms of both reward and cost. Table 2 presents a performance comparison of various methods based on penalized return, recovered return, and cost violation. In terms of these metrics, we observed that while the GAIL method can achieve the highest recovery of experts' return in most environments, it often violates cost constraints. Our methods consistently attained the highest penalized return and exhibited no cost violation in the majority of tasks. LGAIL also violated cost constraints in certain tasks. Although BC can recover a high percentage of experts' returns in some tasks, it tends to violate cost constraints. IQ-Learn falls short in competitive performance in recovering expert's returns and avoiding exceeding experts' episode costs. In comparison, our methods demonstrate superior optimization in both cost and reward aspects.In this study, we address a novel challenge of solving the imitation learning problem within cost constraints. To tackle this issue, we provide three methods that are both scalable and effective. First, we provide a Lagrangian method to solve the cost-constrained imitation learning problem. We then provide a meta-gradient approach that is able to tune the Lagrangian penalties of the first approach to significantly improve the performance. Finally, we provide a cost violation-based alternating gradient approach that has a different gradient update depending on the current solution's feasibility. Experiments demonstrate that our methods can effectively imitate expert behavior while satisfying cost constraints, compared to other imitation learning methods that do not consider cost constraints. Among our approaches, the meta-gradient approach achieved the best trade-off between achieving high expected rewards while satisfying the cost constraints. 
 
 Appendix:  


Decision: The paper presents three new methods for imitation learning with cost constraints, namely a Lagrangian-based method, a meta-gradient method, and cost-violoation-based Alternating Gradient. The experimental evaluation uses Safety Gym and MuJoCo scenarios.

Reviewers agreed that the presented approaches are novel and the approached problem is interesting.
We recommend to include the rebuttal information in the camera-ready paper.

Reviews: review: strength_weakness: strength_weakness: Although three different approaches are presented, they are not a break through and follows direct from well-known paper from literature.

Although a good set of experiments are presented, there is no discussion or hint on which approach to use and when.
questions: questions: The problem addressed by LGAIL is very similar to the problem defined in the article. In such a way that LGAIL can be easily adapted to the problem in question. What is the difference between LGAIL algorithm and CCIL algorithm?
strength_weakness: strength_weakness: Absence of publicly available code impacting reproducibility.
questions: questions: 1. How can you check for scalability of your proposal?

2. What kind of real world application the authors can suggest for the proposed method?
strength_weakness: strength_weakness: - Lack of methodological focus (see comment 1) 
- Lack of related work on cost-constrained reinforcement learning (see comment 2)
- Missing clarity of the experimental results (see comments 4 and 5)
questions: questions: 1) Lack of focus: The authors present three approaches, some of which build on each other. However, there is no detailed discussion of the approaches (e.g., which of the approaches is suitable for which scenarios?) or an experimental comparison with derived results (e.g., in which experiments did which approach work well and why?). As a result, the contribution is not clear and the transferability of the results is not clear to the reader. I recommend that the authors focus on one of the approaches (e.g. MALM) and discuss it in the paper in more detail. 
2) State of the art and related work in cost-constrained reinforcement learning. The authors mention some approaches in imitation learning, especially those on which their approaches are based (e.g. LGAIL). However, I would be interested to know what is the state of research on cost-constrained RL? Why weren't some of the studies mentioned in the paper or used as a baseline? How do the authors distinguish their work from these approaches?
3) The authors state in section 1 that the approach of Cheng et al. (2023) is very similar to their approach, but that the authors "do not assume knowledge of the security constraints". This aspect seems to be an important difference between the approaches. However, the authors did not explicitly prove this later in the paper, did they? Could the authors explain how they addressed this in their experiments and whether they took this into account? 
4) In the results section, the authors state that the MALM approach "demonstrated superior performance compared to all other methods, consistently offering the optimal balance of reward and adherence to cost constraints." I have issues with the statement. 1) What does "optimal balance" mean, could the authors clarify what exactly they mean by this statement? 2) I don't see the overperformance of MALM in the plots shown. There are many environments where MALM does not perform best according to the plots, e.g. CarGoal, PointGoal, DoggoGoal.  I recommend a more detailed discussion of the results here. 
5) The authors barely address the results in Table 2, in particular I wonder how the results relate to the results in Figure 1. Overall, I miss a discussion of the results as a whole, especially since many approaches are compared. Three of them come from the authors, with a different approach always performing best. This leaves me wondering what to take away from the results. 

Overall, the paper presents relevant and good results. If the authors address the weaknesses and comments, I belive that the quality of the paper can be improved.