IPO

In this paper, we study reinforcement learning (RL) algorithms to solve real-world decision problems with the objective of maximizing the long-term reward as well as satisfying cumulative constraints. We propose a novel first-order policy optimization method, Interior-point Policy Optimization (IPO), which augments the objective with logarithmic barrier functions, inspired by the interior-point method. Our proposed method is easy to implement with performance guarantees and can handle general types of cumulative multiconstraint settings. We conduct extensive evaluations to compare our approach with state-of-the-art baselines. Our algorithm outperforms the baseline algorithms, in terms of reward maximization and constraint satisfaction. Introduction Recent advances have demonstrated significant potentials of deep reinforcement learning (RL) in solving complex sequential decision and control problems, e.g., the Atari game (Mnih et al. 2015), robotics (Andrychowicz et al. 2018), Go (Silver et al. 2016), etc. In such RL problems, the objective is to maximize the discounted cumulative reward. In many other problems, in addition to maximizing the reward, a policy needs to satisfy certain constraints. For example, in a cellular network, a common objective for the network operator is to maximize the throughput or cumulative data transmitted to users. At the same time, users may have different requirements for the quality of service, such as the requirements on average latency, cumulative throughput, or the average package loss rate, which are constraints on the optimization problem (Julian et al. 2002). Consider another example of robot manipulation and control. In the task of placing an object (Pham, De Magistris, and Tachibana 2018), the reward is the measurement of how well the object is placed, while there are constraints on the motion of the robot arm, such as how much the arm can twist. RL with constraints is usually modeled as a Constrained Markov Decision Process (CMDP)(Altman 1999), where the agent must act with respect to constraints, in addition to reward maximization. There are two types of constraints: Copyright c © 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. instantaneous constraints (e.g. robot arm twist) and cumulative constraints (e.g. average latency). An instantaneous constraint is a constraint that the chosen action needs to satisfy in each step. A cumulative constraint requires that the sum of one constraint variable from the beginning to the current time step is within a certain limit. In this work, we focus on cumulative constraints, including both discounted cumulative constraints and mean valued constraints. A common approach to solve CMDPs is the Lagrangian relaxation method (Chow et al. 2017; Tessler, Mankowitz, and Mannor 2018). The constrained optimization problem is reduced to an unconstrained one by augmenting the objective function with a sum of the constraint functions weighted by their corresponding Lagrange multipliers. Then, the Lagrange multipliers are updated in the dual problem to satisfy the constraints. Although constraints are satisfied when the policy converges, this approach is sensitive to the initialization of the Lagrange multipliers and the learning rate, and the policy obtained during training does not consistently satisfy the constraints, as discussed in (Achiam et al. 2017; Chow et al. 2019). Constrained policy optimization (CPO) (Achiam et al. 2017) is proposed to solve CMDPs. It extends the trustregion policy optimization (TRPO) algorithm (Schulman et al. 2015) to handle the constraints. CPO monotonically improves the policy during training, demonstrating promising empirical performance, and it guarantees constraint satisfaction during the training process once the constraints are satisfied (Chow et al. 2019). However, CPO needs to calculate the second-order derivatives and thus is complicated to compute and implement. In addition, CPO does not handle mean valued constraints (Tessler, Mankowitz, and Mannor 2018), and it is difficult to employ CPO when there are multiple constraints. In this paper, we propose a first-order optimization method, Interior-point Policy Optimization (IPO), to solve CMDPs with different types of cumulative constraints. Specifically, inspired by the interior-point method (Boyd and Vandenberghe 2004), we augment the objective function of IPO with logarithmic barrier functions as penalty functions to accommodate the constraints. Intuitively, we would like to construct functions such that 1) if a constraint is satisfied, the penalty added to the reward function is zero, and 2) if the constraint is violated, the penalty goes to negative infinity. The logarithmic barrier functions satisfy these requirements, are easy to implement, and also provide nice analytical properties. For policy optimization, we leverage PPO (Schulman et al. 2017), and thus inherit its trust region property. We note that other policy optimization algorithms can be integrated when needed, which increases the flexibility of the proposed methodology. Our algorithm is easy to implement and the hyperparameters are convenient to tune. In summary, our contributions are as follows: • We propose IPO, a first-order optimization RL algorithm under cumulative constraints. The algorithm is easy-toimplement, can handle different types and multiple constraints, with easy-to-tune hyperparameters. • We provide the performance bound of the IPO in terms of reward functions using primal-dual analysis. • We conduct extensive experiments to compare IPO with the Lagrangian relaxation method and CPO on continuous control tasks, such as MuJoco and grid-world in the robotics setting. IPO outperforms the state-of-art methods with higher long-term reward and lower cumulative constraint values.