Safe Policy Optimization with Local Generalized Linear Function Approximations

1 Introduction . Applying reinforcement learning ( RL ) to applications with unknown safety constraints is challenging as RL is inherently an exploratory process and requires agents to learn the whole environment first . Though there has been a surge of attempts to incorporate safety in RL [ 3 , 6 , 11 , 16 ] , it is essentially difficult for most of the algorithms to guarantee safety , especially in the exploration phase . If a single mistake could lead to catastrophic failures , conventional algorithms can not be directly applied . To guarantee safety while learning an environment , safe exploration problems have been actively studied as a sub-field of safe RL [ 17 ] . A mainstream safe-exploration method is to utilize a Gaussian process ( GP , [ 24 ] ) under problem settings in which the agent can observe only the safety function value of its current state . Since the agent can not get any information on the neighboring states , it is necessary to assume desirable properties for the function . Specifically , Sui et al . [ 31 ] assumed that a safety function is Lipschitz continuous and has regularity that can be captured by appropriate kernel functions . However , these assumptions do not hold in many environments . For example , in the context of autonomous driving , the degree of safety varies steeply depending on the presence or absence of other vehicles and pedestrians . Also , previous work on GP-based safe exploration suffers from inconsistency between their theoretical guarantees and computational cost . Previous work has proved the completeness of the predicted safe region [ 34 ] and the optimality of the acquired policy [ 36 ] , after a large number of samples . However , the computational cost of GP is known to be large . Hence , it is hard to achieve theoretically guaranteed performance in a large environment . Based on the fact that robots are equipped with sensors , it is more reasonable to formulate the problem assuming that robots can obtain “ feature vectors ” for inferring the degree of safety of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( observable ) states of their neighbors . This problem formulation is realistic for real-world robots , and it also allows us to deal with safety that changes steeply , which previous research could not handle ( see Figure 1 ) . Under this new problem setting , we aim to propose a theoretically guaranteed and empirically efficient safe RL algorithm that is applicable to real , large-scale problems . Related Work RL with ( generalized ) linear function approximation . Function approximation , especially that based on deep neural networks ( DNNs ) , has been one of the main reasons for the success of RL in recent years , and it has attracted more attention in both theory and practice [ 10 , 32 ] . Given the difficulty of theoretical analysis of DNNs , linear MDPs [ 19 , 22 ] characterized by linear transition kernels and reward are frequently used due to their tractability . The main interest of the theoretical analysis in existing studies has been the efficiency of RL algorithms . In fact , under the assumption of linear MDPs , recent studies ( e.g. , Jin et al . [ 18 ] , Yang and Wang [ 39 ] ) have proposed algorithms with a theoretical guarantee regarding sample efficiency . Wang et al . [ 38 ] then extended Jin et al . [ 18 ] to generalized linear model ( GLM ) settings and presented a practical and provably efficient algorithm under strictly weaker assumptions . Safe RL . A constrained MDP ( CMDP , [ 4 ] ) is a popular model for formulating safe RL problems . In the CMDP formulation , an agent needs to maximize the expected cumulative reward while ensuring that the expected cumulative cost is less than a threshold . There has been a number of attempts to solve CMDPs [ 26 ] , as represented by constrained policy optimization ( CPO [ 2 ] ) , reward constrained policy optimization [ 33 ] , Lagrangian-based actor-critic [ 7 , 9 , 13 ] , primal-dual policy optimization [ 23 , 39 ] , and interior-point policy optimization [ 21 ] . As a recent , closely related work , Ding et al . [ 14 ] proposed an algorithm with theoretical guarantees in regard to regret and constraint violation for CMDPs with linear structures of transition , reward , and safety . In the previous papers mentioned above , however , a safety constraint is defined over a horizon ; hence , they deal with less strict safety constraints than our paper that requires the agent to satisfy safety constraints at every time step . There has been research aimed at guaranteeing safety at every time step in both state-less and stateful settings . For state-less settings , SAFEOPT [ 31 ] and SAFE-LUCB [ 5 ] were respectively proposed for Bayesian optimization problems and linear stochastic bandit problems . For stateful settings , Turchetta et al . [ 34 ] proposed a GP-based SAFEMDP algorithm that extends SAFEOPT to stateful settings . Under the assumptions of the deterministic transition and regularity of safety , while Turchetta et al . [ 34 ] aimed to fully explore the safe region , Wachi et al . [ 37 ] proposed a GP-based algorithm for maximizing the cumulative reward under safety constraint . Our Contributions We model a new class of safe exploration problems where features are locally available within a limited sensor range . This is a realistic formulation for some real-world robots with sensors , and it allows us to get rid of the regularity and Lipschitz assumptions that previous safe exploration algorithms relied on . We develop safe policy optimization with a local feature ( SPO-LF ) , an RL algorithm that maximizes the cumulative reward under safety constraints while predicting reward and safety function values on the basis of local features via GLMs . We analyze the theoretical properties of SPO-LF and prove that an agent will achieve a near-optimal policy with high probability while satisfying safety constraints at every time step . Finally , we evaluate our algorithm in two experiments . In the first experiment using Gym-MiniGrid [ 12 ] , we show that our algorithm is 1 ) more efficient in terms of sample complexity and computational cost and 2 ) more applicable to large-scale problems than previous , highly-theoretical GP-based methods . In the second experiment using Safety-Gym [ 25 ] , we show that our algorithm achieves comparable performance in terms of the reward to several notable safe RL baselines ( e.g. , CPO [ 2 ] ) without executing even a single unsafe action . 2 Problem Formulation . We consider a safety-constrained MDP , defined as a tuple M = 〈S , A , f , H , r , g , φ , ψ〉 , . where S is a finite set of states { s } , A is a finite set of actions { a } , f : S×A → S is the deterministic state transition , and H ∈ N is the finite horizon . Also , r : S → [ 0 , 1 ] and g : S → [ 0 , 1 ] are bounded reward and safety functions . We assume that both r and g are not known a priori.1 At every time step t ∈ N , the agent must be in a safe state . Specifically , for a state st , the safety function value g ( st ) must be above a threshold h ∈ ( 0 , 1 ] ; that is , the safety constraint is represented as g ( st ) ≥ h. In real applications , a robot is equipped with sensors and can infer the values of reward or safety functions by using such measurements as “ hints. ” To formulate this mechanism , M contains two additional functions . First , φ : S → Bd is an a priori unknown function for mapping a state s to a d-dimensional feature vector , where Bd : = { x ∈ Rd | ‖x‖ ≤ 1 } . For convenience , let φs : = φ ( s ) denote the feature vector for a state s. Second , we define a function ψ : S → S for returning a set of states within the sensor range from a certain state . Note that , while conventional methods for solving MDPs with linear function approximation assume the prior availability of the feature map for a whole state space , we consider the problem where features are locally obtained upon observation . Near-sighted and far-sighted observations . On the basis of the two functions φ and ψ , we model two types of observation : near-sighted observation and far-sighted observation . Near-sighted observation is associated with direct observations of reward and safety , which is commonly used in typical RL settings including previous GP-based safe exploration studies . Using near-sighted observations , the agent obtains a set of environmental information that contains noisy observations of the reward and safety function values along with the feature vector for the current state . That is , the agent obtains ( st , φst , yr , yg ) , where yr and yg are noisy observations of reward and safety , which are represented as yr = r ( s ) + nrt and yg = g ( s ) + n g t , with independent zero-mean noises nrt and n g t . Far-sighted observation is more related to feature vectors . In this observation scheme , an agent obtains features for ψ ( s ) . For convenience , let Ψt ⊆ S denote the set of states that have been observed at least once until time t ; that is , Ψt : = ψ ( s0 ) ∪ψ ( s1 ) ∪ . . .∪ψ ( st ) . Intuitively , the agent needs to predict the reward and safety function values by improving the internal GLMs on the basis of near-sighted observations and applying them to the features obtained by far-sighted observations . Goal . The quality of a policy π : S → A is evaluated in terms of the cumulative reward achieved by the agent under the safety constraint . The problem is formally written as maximize : V πM ( st ) = E [ H∑ τ=0 γτr ( st+τ ) ∣∣∣∣∣π ] subject to : g ( st ) ≥ h. There are a number of previous studies on solutions to CMDPs , but most of them violate safety constraints during training ( e.g. , [ 2 ] ) . In contrast , we require the agent to learn a policy in M without even a single constraint violation . Without assumptions , it is intractable to solve the problem mentioned above . In the following , we list assumptions in this paper . Generalized linear function approximation . The first assumption is on the structure of the reward and safety functions . In this paper , we are concerned with generalized linear models ( GLMs ) regarding the reward and safety functions that are characterized by a feature vector . 1For simplicity , we assume that the state transition function , f is known a priori . Note that it is not difficult to extend our setting to a priori unknown f by referring to Biyik et al . [ 8 ] . Assumption 1 . The reward and safety functions are respectively represented using GLMs that are characterized by unknown coefficient vectors θ∗r , θ ∗ g ∈ Rd and fixed , strictly increasing link functions µr , µg : R→ [ 0 , 1 ] such that r ( s ) = µr ( φ > s θ ∗ r ) and g ( s ) = µg ( φ > s θ ∗ g ) . Note that linear and logistic models are the special cases of GLM with µ ( x ) = x and µ ( x ) = 11+e−x , respectively . To predict the coefficients , we obtain maximum likelihood estimators ( MLEs ) for reward and safety functions as in Filippi et al . [ 15 ] . Let θ̃r and θ̃g be the MLEs for reward and safety , respectively . Here , θ̃r and θ̃g are the unique solutions of the following equations . t∑ τ=0 ( r ( sτ ) − µr ( φ > sτ θr ) ) φsτ = 0 and t∑ τ=0 ( g ( sτ ) − µg ( φ > sτ θg ) ) φsτ = 0 . The optimal solutions of the above equations can be found efficiently using such standard algorithms as Newton ’ s method . We additionally make an assumption on the properties of the link functions . Assumption 2 . Let denote either r or g. The link function µ is twice differentiable , and the first and second order derivatives are respectively bounded by L and M . Also , the link function µ satisfies ξ = inf‖θ −θ∗ ‖≤1 , ‖φs‖≤1 µ̇ ( φ > s θ ) > 0 . Prior knowledge on safety . Without prior information on the states known to be safe , an agent can not guarantee safety at the first step . Hence , we make the following assumption . Assumption 3 . The agent is initially in a set of states , S0 ⊆ S , that is known to be safe a priori . Sub-Gaussian noise . The final assumption is on the properties of the observation noise , which has been commonly made in previous work ( e.g. , Abbasi-yadkori et al . [ 1 ] ) . Assumption 4 . Let be either r or g. The noise n t is sub-Gaussian with fixed ( positive ) parameters σ ; that is , for all t , we have E [ eω n t | F t−1 ] ≤ eω2 σ2 /2 , where { F n } is increasing sequences of sigma fields such that n t is F t -measurable with E [ n t | F t−1 ] = 0 .

Summary: This paper studies the problem of safe exploration in reinforcement learning contexts. It develops a new algorithm which, by making several assumptions, can stay safe while exploring and achieves good sample efficiency. The proposed approach incorporates sensor measurements ("far-sighted observations") and estimates of safety based on those observations, as well as the idea of an optimistic and pessimistic safe set, to stay safe during exploration. It also introduces a novel way to ensure that the algorithm does not get stuck in a safe area by detecting this case and more aggressively expanding the set of considered exploration actions.