Multi-Objective Value Iteration with Parameterized Threshold-Based Safety Constraints

1 INTRODUCTION . In reinforcement learning ( RL ) , we often face scenarios where one objective conflicts with safety constraints . For example , Pineau et al . ( 2007 ) ; Rush et al . ( 2004 ) ; Zhao et al . ( 2009 ) have used RL in controlled trial-based treatment analysis . In each treatment stage a medication is administered after which tests are performed to measure symptoms and side effects , and the objective of the doctor may be to optimize treatment for the symptoms while keeping the side effects below some threshold . Different patients may have different preferred thresholds , and hence , it is necessary to generate a family of treatment plans that covers all preferences . One can think of these preferences as values of parameters for threshold-based safety constraints . We present a value iteration algorithm to compute optimal policies that satisfy safety constraints . We consider a scenario where the safety constraints are parameterized , and the computed family of policies has to cover all possible parameter values . Specifically , we consider an environment that provides the agent with a reward vector in Rd+1 corresponding to d+1 reward functions after doing an action a at a state s and time t. We consider a discrete and finite action space A and time horizon T . The state space S can be finite or not . Both cases are discussed separately . The aim of the agent is to maximize the sum over time of the ( d + 1 ) st coordinate of the reward vector while making sure the other d coordinates pass their thresholds at each time step . The thresholds are encoded in a vector δ ∈ ∆ : = Rd . To do that , at each time step , we map the reward vector to a scalar equal to its ( d + 1 ) st coordinate when the thresholds are met and to negative infinity otherwise . Hence , the scalarized reward is parameterized by δ . We present two data structures DecRect and ContDecRect to store the Q functions of the Bellman equation for discrete and continuous state spaces , respectively . They utilize the structure of the scalarized reward function for any state and action as a constant over an axis-parallel hyperrectangle in ∆ with bottom left corner at [ −∞ ] d and −∞ elsewhere . We are abusing notation here since [ −∞ ] d is not a point . This geometrical perspective to the problem inspires intuitive and efficient algorithms to find the pointwise maximum , sum and non-negative scalar multiplication of such functions . These three operations are the only ones needed in the computations of the Bellman equation . We show that they can be done by computing intersections of hyperrectangles in ∆ . Hence , in discrete state spaces , we represent anyQ function for a given state and action by a DecRect which is a set of pairs of hyperrectangles in ∆ and values in R. In continuous state spaces , we assume linear dependency of the reward functions on a feature space of the states of dimension c. Then , we represent the Q function for a fixed action as sets of d + 1 vectors that when multiplied by the feature vector of the state , provide a set of hyperrectangles and values from which the value of the function can be inferred . In both discrete and continuous state space cases , the Q function will be non-increasing in each dimension and hence the names DecRect and ContDecRect for decreasing , rectangles and continuous . Using DecRect , for finite state spaces , we present an algorithm that uses polynomial time and space in |S| and |A| , linear in T and exponential in d , to compute a family of policies that cover all possible values of δ . Moreover , we present an efficient method to identify dominated actions using Pareto front computations . For continuous state spaces , we provide an algorithm that uses ContDecRect and with time and space complexity that is exponential in T and polynomial in |A| , d and c , to compute a similar family of policies . The paper is inspired by the work : “ Efficient Reinforcement Learning with Multiple Reward Functions for Randomized Controlled Trial Analysis ” by Lizotte et al . ( 2010 ) and its extended version Lizotte et al . ( 2012 ) . That work aims to maximize the sum over time of a weighted average of the reward vector components instead of a single component with constraints on the others . They assume finite time horizon T and unknown weights , too . They provided an efficient algorithm , yet exponential in T , to synthesize a set of optimal policies that cover all possible weights for cases with up to three reward functions . For more than three , their time complexity bound becomes doubly exponential . There is rich literature focused on multi-objective RL with constraints . For example , Gábor et al . ( 1998 ) present a lexicographical order on value functions of policies represented as pairs of reals while having a constraint on the first component . They provide an algorithm to learn an optimal policy based on that order . There are multiple differences with our formulation : their constraint on the first reward is on the total expected reward of the policy while ours is on multiple immediate ( instantaneous ) rewards . Second , they synthesize a single policy while we generate a family of policies that covers all possible thresholds ∆ . Finally , they consider an infinite time discounted reward scenario , while we consider a bounded time scenario . A nice overview of existing algorithms for multi-objective reinforcement learning can be found in Roijers et al . ( 2013 ) . Our contributions can be summarized as follows : • A threshold-based scalarization function of the reward vector that encodes the goal achievement objective and the safety constraints . It inspires geometric perspective that allows efficient computation and representation of policies . • Efficient data structures to store the Q and V functions of the Bellman equation for both continuous and discrete state spaces . • Efficient algorithms that construct a family of policies for discrete and continuous state spaces with arbitrary number of features that cover a set of safety preferences encoded as threshold vectors . We start by defining some notations in Section 2 . Then , we formally describe our setup and problem statement in Section 3 . Section 4 presents the data structure DecRect and a value iteration algorithm to compute a family of policies for all possible thresholds in the case of discrete state space . After that , Section 5 present similar results for the case of continuous state spaces using the data structure ContDecRect . Finally , we conclude and suggest future work in Section 6 . 2 PRELIMINARIES . For any positive integer d , we denote the set { 1 , . . . , d } by [ d ] . The natural partial ordering on vectors in Rd is defined as follows : ∀x , x′ ∈ Rd , x ≤ x′ if and only if x [ i ] ≤ x′ [ i ] for all i ≤ d. Given any two matrices X1 ∈ Rd1×d2 and X2 ∈ Rd ′ 1×d2 , we denote by [ X1 , X2 ] ∈ R ( d1+d ′ 1 ) ×d2 the matrix that results from concatenating X1 and X2 . Similarly , given any two matrices X1 ∈ Rd1×d2×d3 and X2 ∈ Rd ′ 1×d2×d3 , we denote by [ X1 , X2 ] ∈ R ( d1+d ′ 1 ) ×d2×d3 the matrix that results from concatenating X1 and X2 . Given a finite set S , we denote the cardinality of S by |S| . We define a special class of functions and some of its properties . Definition 1 . A function f : Rd → R said to be piecewise constant function ( PWC ) if there exists finite number of disjoint axis parallel hyperrectangles , possibly with corners at infinity , over which the value of f is constant . If f is PWC and non-increasing along each dimension , it is said to be non-increasing PWC ( NIPWC ) . The classes of PWC and NIPWC functions are closed under addition , non-negative scalar multiplication , and pointwise maximization . That is , given PWC functions f1 and f2 : Rd → R , the functions fmax ( δ ) : = max { f1 ( δ ) , f2 ( δ ) } and fsum ( δ ) : = f1 ( δ ) + f2 ( δ ) , for δ ∈ Rd are also PWC . Moreover , for any α ∈ R , αf1 is PWC . If f1 and f2 are NIPWC and α ≥ 0 , then fmax , fsum , and αf1 are NIPWC . 3 SETUP . We consider an agent operating in an environment over a finite time horizon T . The state and action spaces are S and A . If the agent takes action at ∈ A at state st ∈ S and time t ≤ T , then the environment returns a ( d + 1 ) -dimensional reward vector rt ( st , at ) ∈ Rd+1 . The first d components of the reward are used to specify safety constraints and the ( d + 1 ) st component corresponds to achievement of a goal . For a parameter δ ∈ ∆ , our aim is to maximize the sum of the last component of the reward over the time horizon while keeping the other d components above their correspondent thresholds specified by δ . Our objective is to design a parameterized policy πδ that solves the problem for any constant δ ∈ ∆ . Formally , we consider a Markov Decision Process ( MDP ) Mδ parameterized by δ ∈ ∆ with state space S , finite action space A , state transition matrix P and a reward function rt : S × A→ Rd+1 . Using rt we define the scalarized reward Rt at time t defined as follows : for any st ∈ S and at ∈ A , Rt ( st , at , δ ) : = { rt ( st , at ) [ d+ 1 ] , if ∀ i ≤ d , δ [ i ] ≤ rt ( st , at ) [ i ] , and , −∞ , otherwise . ( 1 ) A policy π for Mδ maps a time point and a state to an action , that is , π : [ T ] × S → A . Problem Statement The aim is to design a family of policies Π : = { π : [ T ] × S → A } such that for any δ ∈ ∆ , we can find a corresponding optimal policy πδ ∈ Π that maximizes E [ ∑T t=1Rt ( st , at , δ ) ] . For any t ≤ T , we let Qt : S × A ×∆ → R be the optimal Q-function and let Vt : S ×∆ → R be the value function at the tth time step . For a given value of the parameter δ ∈ ∆ , Qt ( st , at , δ ) is the total reward if action at ∈ A is taken at state st ∈ S and the optimal policy is followed till time T . Similarly , Vt ( st , δ ) is the total reward if the optimal policy was followed from the time step t till time T starting from state st ∈ S. For a terminal state sT ∈ S and action aT ∈ A , QT ( sT , aT , δ ) : = RT ( sT , aT , δ ) ( 2 ) is the terminal reward . Similar to Lizotte et al . ( 2010 ) , we will present a value iteration algorithm which computes the Qt and Vt functions for all possible inputs recursively starting from T and backwards using the Bellman equation : for any δ ∈ ∆ , t ≤ T , st ∈ S , and at ∈ A : Qt ( st , at , δ ) = Rt ( st , at , δ ) + Est+1|st , at [ Vt+1 ( st+1 , δ ) ] , ( 3 ) where Vt ( st , δ ) : = maxa∈A ( Qt ( st , a , δ ) ) . As usual , once we evaluate the Q-functions , we can compute the optimal policy for a given δ ∈ ∆ by applying the action arg maxa∈AQt ( st , a , δ ) at state st ∈ S and time t ≤ T .

Summary: The authors consider RL with safety constraints, which is framed as a multi-reward problem. At a high-level, this involves finding the Pareto front, which optimally trades off objectives. The paper primarily introduces and discusses a discretization scheme and methods to model the Q-value function as a NIPWC (non-increasing piecewise constant function). NIPWC are stored as values over discrete partitions of state-action spaces. To do so, the authors introduce two data structures DecRect and ContDecRect to store Q function values over geometric combinations of subsets of state-action space.