On the Value of Interaction and Function Approximation in Imitation Learning

1 Introduction . In many practical sequential decision making problems it is difficult to manually design reward functions that capture the essence of carrying out the task “ nicely ” . Furthermore , many modern-day 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reinforcement learning tasks operate in very large state and action spaces - with sparse reward feedback it is difficult to train good agents without additional feedback or supervision . This motivates the setting of Imitation Learning ( IL ) where the learner operates in a setting of unknown or unreliable rewards , but with an expert that provides demonstrations as to how to carry out the task in the desirable way . The work of [ 21 ] first showed that using expert demonstrations can significantly improve performance in autonomous driving applications . Imitation Learning approaches have found remarkable success in practice over the last decade since expert demonstrations are often available abundantly such as in game AI [ 13 , 1 ] , as well as more recently in autonomous-driving applications such as [ 5 , 20 ] . In this paper , we study IL in the episodic Markov Decision Process ( MDP ) formalism . Notation : An MDPM = ( S , A , P , ρ , r ) , where S is the state space , A is the action space P is the MDP transition , ρ is initial state distribution and r is the reward function . The value Jr ( π ) of a policy π is defined as the expected cumulative reward accumulated over an episode of length H , Jr ( π ) = Eπ [ ∑H t=1 rt ( st , at ) ] , where the notation Eπ [ · ] denotes expectation with respect to a random trajectory { ( s1 , a1 ) , · · · , ( sH , aH ) } obtained by rolling out the policy π = ( π1 , · · · , πH ) , where the initial state s1 is sampled independently from an initial state distribution ρ ( · ) . Here πt denotes the policy at time step t. In the IL setting , we assume that the underlying reward function is unknown and unobserved . The reward function r = { r1 , · · · , rH } is assumed to be time-variant and pointwise bounded in [ 0 , 1 ] , and the transition function P = ( P1 , · · · , PH−1 ) of the MDP is also assumed to be time-variant . The simplest IL setting is the no-interaction setting [ 22 ] . Definition 1 ( IL in the no-interaction setting ) . The learner is provided an offline dataset D of N trajectories ( without rewards ) drawn by independently rolling out an ( unknown ) expert policy π∗ through the MDP . The learner is not allowed to interact with the MDP . The learner ’ s objective in IL is to construct a policy π̂ with small suboptimality , defined as the difference in the expert ’ s and learner ’ s values : Jr ( π∗ ) − Jr ( π̂ ) . In this paper we restrict the expert policy to be deterministic and define π∗t ( s ) as the action played by the expert at time t at state s. An IL instance refers to the tuple ( M , π∗ ) .The Q-function of a policy π is defined as the expected rewardto-go , Qπt ( s , a ) = Eπ [ ∑H t′=t rt′ ( st′ , at′ ) |st = s , at = a ] , and fπt ( s ) is defined as the distribution over states induced at time t , by rolling out the policy π . Since the expert policy is a collection of actions played at different states visited by the expert , a natural approach to IL is to use any classification algorithm to learn a mapping from states to actions as the learner ’ s policy . This supervised learning approach has proved to be quite popular in practice and is known as behavior cloning ( BC ) . [ 25 ] study BC from a theoretical point of view , and bound the suboptimality of a policy in terms of the 0-1 loss of the resulting classifier . More recently , in the tabular setting , [ 22 ] show that BC is statistically optimal in the no-interaction setting , achieving expected suboptimality . |S|H 2 N . This H 2 dependence is known as error-compounding and is shown to be necessary even if expert is optimal or the learner can actively query the expert . Definition 2 ( IL in the active setting ) . In this setting , the learner is not provided a dataset of expert demonstrations up front . The learner can instead interact with the MDP for N episodes . While interacting , the learner can query an oracle to return the expert ’ s action π∗t ( s ) at the current state s. It begs the question as to why approaches such as DAGGER ( [ 25 ] ) and AGGREVATE ( [ 26 ] ) which actively query the expert often perform better than BC in practice and to explain this gap , additional assumptions must be imposed . To this end , we look at the minimax lower bound of [ 22 ] in the no-interaction setting . The key idea of the lower bound is to include an absorbing “ bad ” state in the MDP which is never visited in the expert dataset and offers no reward . Any policy which visits this state is doomed to incur a large suboptimality - in the absence of full information , the learner is forced to visit this often . The lower bound instance is pathological in the sense that even if the expert itself visits the bad state , it is never able to “ recover ” and return to the remaining states . Indeed in practical situations such as driving a car , experts often can recover and collect a high reward even if a mistake is made locally . [ 25 ] introduce an assumption to this effect , which we refer to as µ-recoverability . Definition 3 ( µ-recoverability ) . An IL instance is said to satisfy µ-recoverability if for each t ∈ [ H ] and s ∈ S , Ea∼π∗t ( ·|s ) [ Qπ ∗ t ( s , a ) ] −Qπ∗t ( s , a ) ≤ µ for all actions a ∈ A . Informally , if the expert plays an “ incorrect ” action at any state s at a single time t and goes back to choosing the correct actions afterwards , the expected reward collected is less by at most µ . Under the µ-recoverability assumption , [ 25 ] show that a learner which minimizes the 0-1 loss under the learner ’ s own state distribution to admits a suboptimality upper bound of µH . However , it is a-priori unclear how small can be made as a function of the number of the size of expert dataset / number of MDP interactions , N in the no-interaction / active settings . This is a drawback of the reduction approach followed by [ 25 , 27 ] since it can not distinguish the power of learners in different interaction models . In this paper , we propose an policy in the active setting with expected 0-1 loss under the learner ’ s own state distribution bounded by |S|/N . This “ completes ” the reduction in a sense , and establishes suboptimality bounds for the active setting as an explicit function of the number of states |S| , interactions N and horizon H . Informal Theorem 1 ( Formal version : Theorem 1 and 2 ) . In the active setting , under the µ-recoverability assumption , there exists a learner π̂ which incurs expected suboptimality E [ J ( π∗ ) − J ( π̂ ) ] . µH|S|/N . Furthermore , if N ≥ |S|H , for any learner π̂ in the active setting , there exists an MDP such that the expected suboptimality E [ J ( π∗ ) − J ( π̂ ) ] & µH|S|/N . The key challenge for a learner to minimize the 0-1 under its own state distribution is that the learner ’ s policy changes over the course of optimization . Note however that it is possible to compute an unbiased estimate of the 0-1 loss under the learner ’ s own state distribution by rolling out just a single trajectory . Thus the active sampling model plays a crucial role in this regard . idea is crucial towards constructing the learner policy discussed in Informal Theorem 1 . Under the same µ-recoverability assumption , we next consider learners in the no-interaction setting . In contrast to the active setting , we show that error compounding is unavoidable for no-interaction learners . Informal Theorem 2 ( Formal version : Theorem 3 ) . In the no-interaction setting , for any learner π̂ there exists an IL instance which satisfies µ-recoverability for µ ≥ 1 such that the expected suboptimality is E [ J ( π∗ ) − J ( π̂ ) ] & H2|S|/N . This is the first result to establish a clear separation in the statistical minimax rate of the suboptimality incurred by learners in the no-interaction setting such as BC , and learners which can interact with the MDP , such as DAGGER [ 27 ] and AGGRAVATE [ 26 ] . A common theme of the previous bounds in the tabular setting is that the suboptimality necessarily scales linearly in the number of states . In practical RL settings , state and actions spaces are often continuous , and thus additional assumptions are required to carry out efficient learning . In this paper , we study IL with function approximation , in particular in the linear-expert setting . Definition 4 ( Linear-expert setting ) . In this setting , for each ( s , a , t ) tuple , the learner is provided a feature representation φt ( s , a ) ∈ Rd . For each t ∈ [ H ] there exists an unknown vector θ∗t ∈ Rd such that ∀s ∈ S , π∗t ( s ) = arg maxa∈A〈θ∗ , φt ( s , a ) 〉 . As we discuss in Remark 1 , the linear-expert setting generalizes several known settings such as when the expert is an optimal policy under the linear-Q∗ assumption as well as the tabular setting with an optimal expert . We first establish a bound on the expected suboptimality incurred by BC . Informal Theorem 3 ( Formal version : Theorem 4 ) . Under the linear-expert setting , the policy π̂ returned by BC incurs suboptimality J ( π∗ ) − J ( π̂ ) . ( d+log ( 1/δ ) ) H 2 log ( N ) N with probability ≥ 1− δ . The presence of this error-compounding is not so surprising because the tabular setting with an optimal expert is a special case of the linear-expert setting where [ 22 ] show that error compounding is unavoidable for no-interaction learners . In order to break this H2-dependence , we introduce a natural variant of the linear-expert setting known as linear-expert setting with parameter sharing . Definition 5 ( Linear-expert with parameter sharing ) . This setting is the same as the linear-expert setting ( Definition 4 ) , with the added constraint that for all t , θ∗t = θ ∗ is shared across time . Our main contribution is to show that in the linear-expert setting with parameter sharing , IL can be reduced to sequence multi-class linear classification where we learn linear classifiers from SH → AH . The supervised learning reduction of [ 25 ] posits to learn separate classifiers from S → A or S × [ H ] → A : this fails to account for the shared parameter θ∗ across time . While in both cases the resulting policy is an ERM classifier , the suboptimality grows quadratically in H using the supervised learning reduction . In contrast , using the multi-class classification algorithm of [ 8 ] , we also provide an algorithm π̂ with suboptimality growing linearly in H . Informal Theorem 4 ( Formal version : Theorem 5 ) . Under the linear-expert setting with parameter sharing , there exists a learner π̂ with suboptimality J ( π∗ ) − J ( π̂ ) . ( d+log ( 1/δ ) ) H log ( N ) N with probability ≥ 1− δ . With the additional linearity assumption on the expert , the learner can potentially infer the expert ’ s action on states that are not observed in the dataset . However in the absence of transition information or the parameter sharing assumption , a learner can not even distinguish between different actions at the remaining states , which is what leads to catastrophic error compounding . To remedy this issue , we borrow from the work of [ 22 , 23 ] who study IL in the known-transition setting in tabular MDPs where the learner exactly knows the Markov transition kernel and the initial state distribution of the MDP . The motivation for this setting stems from autonomous driving applications where policies are often learned in a simulated environment prior to fine-tuning in the real world [ 9 , 35 ] and in the simulator the rewards functions are still difficult to specify . In such settings , the state and action spaces are indeed unbounded , which makes it ideal to study through the frame of function approximation . Definition 6 ( IL in the known-transition setting ) . The learner is provided an dataset D of N trajectories ( without rewards ) drawn by independently rolling out the expert policy π∗ through the MDP . The learner also knows the MDP transition P and initial state distribution ρ exactly . In the tabular setting , the known transition setting has an interesting landscape : it is known from [ 22 ] that the quadratic-H barrier can be broken - the authors propose the MIMIC-MD algorithm which achieves an expected suboptimality upper bound of |S|H3/2/N and this dependence on the horizon is optimal [ 23 ] . The key idea is that with access to the MDP transition structure , as long as the visited states are conditioned to be observed in the dataset ( so the expert ’ s action is known ) , the learner can simulate artificial trajectories according to the expert ’ s policy to generate more training data . While the approach of simulating artificial trajectories in MIMIC-MD achieves the minimax optimal bounds here , a natural question to ask is whether the approach is tailored to work only in the tabular setting . Indeed in the presence of continuous state spaces , the learner may observe but a measure-0 subset of the state-space in the expert dataset . In spite of this , to apply the approach of simulating artificial trajectories , the learner must be able to infer the expert ’ s actions on a large fraction of the state-space . To this end , in the known-transition setting , we propose a problem known as confidence set linear classification which extends multi-class linear classification and we prove that algorithms with small expected loss for confidence set linear classification can be used to construct policies with small suboptimality , using the approach of simulating artificial trajectories . At a high level , the objective of the learner is to not only otput a classifier , but also a set of inputs ( confidence set ) where the classifier certifiably outputs the correct label . Definition 7 ( Confidence set linear classification ) . Consider a classification problem on X with input distribution ρX , output space Y , with features φ : X × Y → Rd and a dataset D of N examples drawn i.i.d . as xi ∼ ρX and yi ∼ h∗ ( xi ) , where h∗ is an unknown linear multi-class classifier mapping x 7→ arg maxy∈Y 〈θ∗ , φ ( x , y ) 〉 . Given the dataset D , a confidence set linear classifier returns a tuple ( ĥ , X ) where ĥ is any classifier from X → Y and X ⊆ X is a measurable set of inputs ( known as the confidence set ) such that ∀x ∈ X , ĥ ( x ) = h∗ ( x ) . The learner ’ s objective is to minimize the expected loss E [ 1− ρX ( X ) ] . Sample-efficient confidence set linear classification algorithms can be used to construct learners with small suboptimality . We prove such a reduction in the linear-expert setting with linear rewards : here , in addition to the linear-expert setting , for each t ∈ [ H ] , the reward function rt ( · , · ) is also constrained to be a linear function of the feature representations φt ( · , · ) . The linear reward setting was considered previously in the known transition setting in [ 3 ] . We formally define it in Definition 8 . Informal Theorem 5 ( Formal version : Theorem 6 ) . Consider any algorithm Alg for confidence set linear classification and define RN , d ( ρX , Y , ψ ) as the expected loss ( Definition 7 ) incurred by Alg when ( i ) the input distribution is ρX , ( ii ) features are ψ : X × Y → Rd and ( iii ) the learner is provided a dataset of N samples ( with labels from an unknown multi-class linear classifier ) . In the linear-expert setting with linear rewards , there exists a learner policy π̂ with expected suboptimality : E [ J ( π∗ ) − J ( π̂ ) ] . H3/2 √ d N ∑H t=1 RN , d ( f π∗ t , A , φt ) H ( 1 ) Informal Theorem 5 shows that it suffices to find good algorithms for confidence set linear classification and bound Rd , N ( ρX , Y , φ ) to carry out sample efficient IL . However , even in the case of binary output space Y = { 0 , 1 } and uniformly distributed features , the answer to this question is quite challenging and admits a non-standard rate . Informal Theorem 6 ( Formal version : Theorem 7 ) . Consider an instance of confidence set linear classification where Y = { 0 , 1 } , ρX = Unif ( Sd−1 ) and φ ( x , 0 ) = −φ ( x , 1 ) = x/2 ∈ Rd . Then , for sufficiently large N , ( i ) For any algorithm d 3/2 N √ log ( d ) . Rd , N , A ( ρX , Y , φ ) . ( ii ) There exists an algorithm such that Rd , N , A ( ρX , Y , φ ) . d3/2 log ( d ) N . This result shows that the minimax risk for confidence set linear classification necessarily grows as & d3/2/N . This rate establishes a fundamental difference between function approximation and tabular settings . In the tabular setting , where the features for each state-action pair are orthogonal , the learner can not conclude the labels at unobserved states . Thus , the minimax risk of confidence set linear classification corresponds to the expected probability mass on unobserved states , which is also known as the missing mass [ 18 ] . It is known from [ 22 , Lemma A.20 ] that the expected missing mass is . |S|/N ≡ d/N under any distribution over states and binary action space . Informal Theorem 6 establishes a fundamental difference between the tabular setting and the linear-expert setting with linear rewards , in terms of the suboptimality guarantees achieved by the approach of simulating artificial trajectories . While the upper bound in Informal Theorem 6 ( ii ) only applies in the special case of binary classification with uniformly distributed features , we conjecture that the minimax expected loss for confidence set linear classification is d 3/2 N . If this conjecture is true , then Informal Theorem 6 shows that there exists a learner π̂ in the known-transition setting with linear-expert and linear rewards such that the expected suboptimality is . H3/2d5/4/N . For sufficiently large H , this improves the Θ̃ ( dH2/N ) of BC and achieves the optimal dependence on the horizon [ 23 ] .

Summary: They study imitation learning in several variations, in particular with interaction (i.e., the expert can be queried during training) and without. And under some interesting additional assumptions, such as mu-recoverability (i.e., if the imitator visits suboptimal states, this has small impact of the overall value, by being able to go back to optimal easily).  They provide several theoretical results,  - negative ones -- i.e., lower bounds on the suboptimality that can be achieved in terms of difference of imitator versus optimal expert value incurred -- as well as  - positive ones -- algorithms that achieve linear instead of quadratic suboptimality, in particular their main contribution thm4. This works by reduction to supervised learning paradigm sample bounds.  As a means to this, they also have a little taxonomy on versions of IL.  One thing that's hard to judge for me is novelty. I feel this is novel, but I'm not sure how much overlap there is with DAgger and all the subsequent work in this direction, not an expert on this.