Title: Regularized Behavior Cloning for Blocking the Leakage of Past Action Information
Venue: NeurIPS 2023
Accepted: yes
Keywords: Imitation learning

Information leakage

Causal Confusion
Abstract: For partially observable environments, imitation learning with observation histories (ILOH) assumes that control-relevant information is sufficiently captured in the observation histories for imitating the expert actions. In the offline setting wherethe agent is required to learn to imitate without interaction with the environment, behavior cloning (BC) has been shown to be a simple yet effective method for imitation learning. However, when the information about the actions executed in the past timesteps leaks into the observation histories, ILOH via BC often ends up imitating its own past actions. In this paper, we address this catastrophic failure by proposing a principled regularization for BC, which we name Past Action Leakage Regularization (PALR). The main idea behind our approach is to leverage the classical notion of conditional independence to mitigate the leakage. We compare different instances of our framework with natural choices of conditional independence metric and its estimator. The result of our comparison advocates the use of a particular kernel-based estimator for the conditional independence metric. We conduct an extensive set of experiments on benchmark datasets in order to assess the effectiveness of our regularization method. The experimental results show that our method significantly outperforms prior related approaches, highlighting its potential to successfully imitate expert actions when the past action information leaks into the observation histories.

Regularized Behavior Cloning for Blocking the Leakage of Past Action Information

Abstract

For partially observable environments, imitation learning with observation histories (ILOH) assumes that control-relevant information is sufficiently captured in the observation histories for imitating the expert actions. In the offline setting where the agent is required to learn to imitate without interaction with the environment, behavior cloning (BC) has been shown to be a simple yet effective method for imitation learning. However, when the information about the actions executed in the past timesteps leaks into the observation histories, ILOH via BC often ends up imitating its own past actions. In this paper, we address this catastrophic failure by proposing a principled regularization for BC, which we name Past Action Leakage Regularization (PALR). The main idea behind our approach is to leverage the classical notion of conditional independence to mitigate the leakage. We compare different instances of our framework with natural choices of conditional independence metric and its estimator. The result of our comparison advocates the use of a particular kernel-based estimator for the conditional independence metric. We conduct an extensive set of experiments on benchmark datasets in order to assess the effectiveness of our regularization method. The experimental results show that our method significantly outperforms prior related approaches, highlighting its potential to successfully imitate expert actions when the past action information leaks into the observation histories.

1 Introduction

Imitation learning (IL) aims at learning a policy that recovers an expert's behavior from a demonstration dataset. Leveraging the information about state and expert action available in the dataset, IL has been successful in many real-world applications [4, 12, 26, 27, 32, 38, 43]. Although IL problems can be addressed using either online [17, 36] or offline algorithms [20, 33, 48], real-world tasks often impose restrictions on interacting with the environment due to safety, cost, and ethical concerns. Consequently, the practical necessity lies in the development of effective offline IL algorithms. Behavior cloning (BC) [33] is one of the most prominent offline IL algorithms, which learns to predict an expert's action for each given state using an offline expert dataset via supervised learning. BC has gained widespread recognition [7, 42] for providing a straightforward and effective solution, especially when full access to state information is available and the dataset is sufficiently extensive. However, when the state information is only partially available in an observation at each timestep, which is closer to the realistic scenario (e.g. an autonomous vehicle with a limited number

of sensors), training with any IL algorithm can be complicated. To enhance the ability of agent to infer missing control-relevant information from observations, incorporating the history of observations from adjacent past timesteps as input can be beneficial [2, 24].

When utilizing observation history for behavior cloning, a notable challenge emerges: the unnecessary dependence of the imitator's actions on the preceding actions, resulting in a suboptimal behavior in test time. In extreme cases, this dependency can lead to undesirable behaviors, such as the imitation policy that merely replicates its own actions from previous time steps 1 , which leads to a catastrophic result particularly in safety-critical tasks like autonomous driving. This is often rooted at the inability to differentiate between actual and spurious causal relationship between observation features and expert actions within the collected data [8]. Past action information is a representative instance of such nuisance features, since it is a strongly correlated to the target expert action. This correlation can mislead the IL algorithm into recognizing past action information, which is not causally related to the target expert action, as a crucial feature for prediction. In this work, we refer to this misleading phenomenon as past action information leakage.

A natural way to mitigate this phenomenon is to adopt a mechanism that blocks the leakage of past action information irrelevant to the target expert action. In fact, Wen et al. [44] proposed an adversarial training approach to discard such redundant information in the representation of the observation history. This is achieved by maximizing the conditional entropy of the expert's previous action given the joint of the representation and target expert action. However, due to the intractability of direct computation of the conditional entropy, this approach relies on unstable adversarial learning. More recently, Chuang et al. [6] focuses on image-based observations and attempts to mitigate the past action dependency by splitting the policy representation into two parts: a representation of the observation history and a representation of the current observation. This method assumes a particular policy structure and does not provide a systematic general approach to block the leakage of past action information into the representation.

In this paper, we present a regularized behavior cloning framework that effectively mitigates the leakage of past action information. We formally define the problem of the leakage of past action information and establish a metric to quantify the magnitude of the leakage, employing the kernelbased method called HSCIC (Hilbert-Schmidt Conditional Independence Criterion) [30]. Building upon the metric, we devise an objective function that addresses the aforementioned problem, allowing for a more comprehensive understanding and interpretation of existing work [44]. Moreover, we propose a stable and efficient kernel-based regularization method that circumvents challenges such as adversarial learning, nested optimization and reliance on a neural estimator. Lastly, we conduct an extensive set of experiments which empirically show that our regularization method effectively blocks the leakage of past action information across a variety of control task benchmarks.

Our contributions are summarized as follows:

We formally define the problem of the leakage of past action information based on the concept of conditional independence by quantifying the amount of leaked past action information.

We introduce a principled framework for a behavior cloning with Past Action Leakage Regularization (PALR), which prevents the imitator from overfitting to leaked past information.

We provide experimental results on established benchmarks, demonstrating the effectiveness of our method.

2 Related Work

Invariant representation learning Learning representation invariant to any unwanted factors has been widely studied in various domains such as fair classification [22, 25, 37, 47], domain adaptation [11, 13, 18, 49], and imitation learning [6, 44]. One of the dominant approaches in invariant representation learning is adversarial learning [11, 13, 44, 47]. Adversarial learning algorithms

commonly train additional networks that predict the unwanted factors from the representations while enforcing representations to make those prediction models fail. Consequently, they require alternating optimization between the main and the additional models [13, 44, 47] or show numerical instability [11]. To bypass those shortcomings of adversarial learning, several information-theoretic approaches [18, 25] have been proposed. Based on Variational Auto-Encoders (VAE) [21], these methods proposed end-to-end learning algorithms that jointly optimize all of their components with numerical stability. However, these methods assume that the distribution of the representation is Gaussian, which restricts the flexibility of the representation. Wen et al. [46] concentrated on eliminating shortcuts in supervised learning by incorporating supplementary key information, demonstrating its enhancement of behavior cloning with observation histories. Recent methods have proposed to learn counterfactually or conditionally invariant representation, leveraging on kernelbased conditional independence metric [31, 35]. These methods demonstrated promising empirical results in synthetic domains, effectively mitigating the impact of nuisance correlations.

Information leakage in imitation learning There is a growing understanding of the importance of addressing the correlation between expert actions and nuisance features, which are not essential for control and may even hinder performance. [8, 29, 39] Especially, termed as causal confusion [8], imitation learning exhibits a paradoxical phenomenon: having more information can lead to worse performance. Aligning with such observation, recent works have also demonstrated that accessing more information from the observation history leaks past action information so that the imitator may learn an undesirable policy that simply repeats the same action in the past [6, 7, 40, 44, 45]. To avoid learning degenerate solutions from the past action information, Wen et al. [44] proposed the regularization method based on the conditional entropy of the previous action given the representation of the history and the current action. However, their method involves a nested minimax optimization along with an additional neural network, which complicates the training process. We closely investigate their formulation in Section 4.2.1 to show their limitations as well as their connection to our method. Wen et al. [44] also introduced the action predictability metric to quantify the dependence between past action histories and imitator actions, relative to expert actions. However, this metric is not based on conditional independence, which is central to our argument concerning the past action leakage. Wen et al. [45] addressed this problem using a weighted behavior cloning method that upweights 'keyframe' samples, which are more likely to be predicted from the action histories. Swamy et al. [40] showed that online interaction is both necessary and sufficient to resolve repeating behavior. However, in many real-world applications [4, 12, 26, 27, 32, 38, 43], online interaction is often infeasible due to safety, cost, and ethical considerations. In this work, we develop an offline algorithm that can robustly handle the leakage of past action information.

3 Preliminaries

3.1 Conditional independence

For random variables X,Y which taking a value in X , Y respectively, we write P X , P XY , and P X | Y for the marginal distribution of X , the joint distribution of X and Y , and the conditional distribution of X given Y , respectively. We say that random variables X and Y are conditionally independent given a random variable Z or simply Z -conditionally independent if P XY | Z ( x, y ) = P X | Z ( x ) P Y | Z ( y ) for all x, y ∈ X × Y . Also, we say that X and Y are independent if P XY ( x, y ) = P X ( x ) P Y ( y ) for all x, y . We denote Z -conditional independence by X ⊥ ⊥ Y | Z , and independence by X ⊥ ⊥ Y .

Conditional Mutual Information (CMI) Mutual information (MI) is an information-theoretic quantity to measure the dependency between two random variables. MI between random variables X and Y is defined as I ( X ; Y ) = D KL ( P XY , P X P Y ) = E x,y ∼ P XY [log P XY ( x, y ) -log P X ( x ) P Y ( y )] . MI is always non-negative (i.e., I ( X ; Y ) ≥ 0 ), and it becomes zero if and only if X ⊥ ⊥ Y . Conditional mutual information (CMI) is defined similarly, but with conditional distributions. CMI between random variables X and Y given a random variable Z is defined as I ( X ; Y | Z ) = E P Z [ D KL ( P XY | Z , P X | Z P Y | Z )] . As in the case of MI, I ( X ; Y | Z ) ≥ 0 , and also I ( X ; Y | Z ) = 0 if and only if X ⊥ ⊥ Y | Z .

Hilbert-Schmidt Conditional Independence Criterion (HSCIC) Hilbert-Schmidt Independence Criterion (HSIC) [14] is a kernel-based measure that quantifies the dependency between two random variables. Let H X and H Y be reproducing kernel Hilbert spaces (RKHS) on X and Y with the corresponding reproducing kernels k X and k Y , respectively. The tensor product k X ⊗ k Y of the kernels is a binary function on X × Y defined by ( k X ⊗ k Y )(( x 1 , y 1 ) , ( x 2 , y 2 )) := k X ( x 1 , x 2 ) k Y ( y 1 , y 2 ) . The RKHS on X ×Y with the kernel k X ⊗ k Y is called the tensor product RKHS of H X and H Y , and it is denoted by H X ⊗H Y . HSIC between X and Y is defined to be the maximum mean discrepancy (MMD) between P XY and P X P Y under H X ⊗H Y , i.e, the distance in the Hilbert space H X ⊗H Y between the so-called kernel mean embeddings of P XY and P X P Y to that space:

where µ P XY is the kernel mean embedding of the distribution P XY to H X ⊗H Y , i.e. µ P XY ( x, y ) := E P XY [( k X ⊗ k Y )(( X,Y ) , ( x, y ))] , and µ P X , µ P Y are defined similarly but using H X , H Y instead. The ( µ P X ⊗ µ P Y ) is simply the function in H X ⊗H Y that maps ( x, y ) to µ P X ( x ) µ P Y ( y ) . When k X ⊗ k Y satisfies some condition (i.e., characteristic), HSIC( X,Y ) = 0 if and only if X ⊥ ⊥ Y [14].

HSCIC (Hilbert-Schmidt Conditional Independence Criterion) is an extension of HSIC that quantifies the amount of conditional dependency between two random variables given another random variable. In this paper, we follow the definition of HSCIC based on conditional mean embedding [30], and use an estimator of HSCIC that draws samples from the joint distribution P XYZ and performs vector-valued RKHS regression. The conditional mean embedding of X given Z is a function in the RKHS H X that is parameterized by the value of Z . Given a value z of Z , it maps x ∈ X to µ P X | Z = z ( x ) := E P X | Z = z [ k X ( X,x )] . Then, HSCIC is a mapping from a value of Z to a non-negative real defined as follows: for all z ∈ Z ,

When k X ⊗ k Y satisfies the condition mentioned from above, X ⊥ ⊥ Y | Z if and only if E P Z [HSCIC( X,Y | Z )] = 0 [30]. We use an empirical estimator of the expectation here, denoted by ̂ HSCIC( X,Y | Z ) . See Section A in the supplementary material for more details for definition of HSCIC and its empirical estimator.

3.2 Imitation learning from observation histories

We consider a Partially Observable Markov Decision Process (POMDP) [19] without reward, which is defined as a tuple of ⟨S , Z , A , O, P, ρ 0 ⟩ . Here S , Z and A are an (underlying) state space, an observation space and an action space, respectively. The next O : S → ∆ Z specifies the conditional probability O ( z | s ) of an observation z ∈ Z given a state s ∈ S . Finally, ρ 0 ∈ ∆ S defines the probability ρ 0 ( s 0 ) that the process starts from s 0 ∈ S , and P : S × A → ∆ S defines the probability P ( s ′ | s, a ) of transitioning to state s ′ when action a ∈ A is performed in state s ∈ S .

Assume a (stochastic) expert policy π E : S → ∆ A that is defined as a conditional probability π E ( a | s ) of the expert's performing an action a ∈ A given a state s ∈ S . Also, assume that we have a dataset D = { τ (1) , . . . , τ ( N ) } of the expert trajectories where each τ = { ( z t , a t ) } T t =0 is sampled by

That is, τ in the dataset is drawn from the following joint distribution of all observations and actions:

In our study, we consider POMDP scenarios where an ideal imitator policy is able to match the expert policy's performance, even when the imitator's actions are solely determined by observation histories. Specifically, we focus on situations where the observation histories z t -w -1: t (for some fixed 1 ≤ w ≤ T -1 ) encompass all information about the true states s t utilized by the expert policy.

In this setting, our goal is to learn an imitator policy π I : Z w → ∆ A that acts as closely to π E as possible on the given dataset D . To achieve this goal, we consider the following joint distribution of actions of both π I and π E :

4 Behavior Cloning with Past Action Leakage Regularization

In this section, we propose a framework that effectively mitigates the past action leakage problem in IL. We first define the problem by formalizing the absence of leaked past-action information via conditional independence (Section 4.1). Then, we compare several regularization-based approaches that attempt to achieve the absence of such information in the context of offline IL (Section 4.2). These approaches performs regularized BC where the regularizer is derived from a metric for measuring the amount of conditional dependence among random variables. The choice of the metric differentiates these approaches, and our comparison advocates the use of the HSCIC-based approach.

4.1 Past action leakage problem in imitation learning

When a policy takes an observation history as an input, the input history may include the information about past actions unexpectedly. The inclusion of such information can have detrimental effects in the context of offline imitation learning by confusing the imitator and making it fail to predict expert actions accurately. Intuitively, the past action leakage problem in imitation learning refers to this failure of the imitator due to the leakage of such harmful past action information.

To express this intuition formally, for each timestep t , let A E t and A I t denote random variables of expert action and imitator action at timestep t . Note that for each 0 < t ≤ T , the joint distribution of the three random variables A E t -1 , A E t and A I t is

where p ( a I 0: T , a E 0: T ) is the distribution in Eq. (1).

We formalize the absence of the leakage of harmful past-action information by conditional independence between the imitator's current actions and the expert's previous actions:

This conditional independence says that the imitator's current action never depends on some information that is only about the expert's past action but not about the expert's current action. This past-specific information corresponds to harmful information in our intuitive explanation from above.

Ideally we would like to achieve conditional independence in Eq. (2), which ensures the absence of leaked past-action information that was harmful to the imitator. However, in practice, we can achieve it only approximately, so that we need a quantitative measure for conditional independence or the lack of conditional independence between A E t -1 and A E t given A I t . Such a measure is also needed to design an offline IL algorithm that does not suffer from such leaked harmful past-action information. In the following subsection, we consider two quantitative measures for the lack of conditional independence, namely, (1) conditional mutual information (CMI) and (2) Hilbert-Schmidt Conditional Independence Criterion (HSCIC).

4.2 Behavior cloning with past action leakage regularization

We aim to learn the representation φ t of observation history z t -w +1: t that removes any unnecessary information on the past action. To simplify the notation, let t w denote t -w +1 . Our method is based on the following observation.

Theorem 1. Let A I t be the action from the imitator policy π I ( a t | φ t ) based on the representation φ t of observation history. Then, φ t ⊥ ⊥ A E t -1 | A E t = ⇒ A I t ⊥ ⊥ A E t -1 | A E t .

Proof. See Section B in the supplementary material.

To this end, we formulate our objective of regularized BC framework as follows:

where L bc is an BC objective such that L bc ( π, φ ; D ) := E ( z t w : t ,a E t ) ∼D ,φ t ∼ φ ( z t w : t ) [ -log π ( a E t | φ t ) ] and L reg is a past action leakage regularization objective. For notational simplicity, we abbreviate the expectation with respect to ( a E t -1 , z t w : t , a E t ) ∼ D , φ t ∼ φ ( z t w : t ) to E φ t . In the following subsections, we discuss candidates for L reg .

4.2.1 Information-theoretic regularization

A straightforward way to address the conditional independence we discussed in Section 4.1 is minimizing the CMI, which can be decomposed by its definition into two conditional entropy terms:

It is important to note that H ( a E t -1 | a E t ) is determined by the data distribution D and thus constant. As a result, we can simply consider the minimization of -H ( a E t -1 | φ t , a E t ) .

Interestingly, Eq. (4) coincides with the negative conditional entropy maximization objective in FCA [44]. Since the direct computation of Eq. (4) requires to know the intractable distribution p ( a E t -1 | φ t , a E t ) , FCA trains an additional prediction model ˆ p (ˆ a t -1 | φ t , a t ) to estimate the negative entropy with -ˆ H ( a E t -1 | φ t , a E t ) := E φ t [log ˆ p ( a E t -1 | φ t , a E t )] . However, this approach faces the following challenges:

The estimated negative entropy -ˆ H lower bounds Eq. (4), while an upper bound would be desirable for minimizing the objective. Consequently, it imposes nested (minimax) optimization; minimizing with respect to φ t after maximizing with respect to ˆ p .

It is required to train an additional neural network to model ˆ p , which consumes additional computational cost.

Assuming that the family of variational distribution ˆ p to be Gaussian, FCA tries to tighten the lower bound estimation -ˆ H with respect to ˆ p by minimizing reconstruction error of a E t -1 . The assumption restricts the flexibility of ˆ p and thus the estimation can be inaccurate.

To avoid inaccurate estimation of the entropy, we can also decompose CMI into two MI terms by the chain rule of MI as an alternative:

Similar to H ( a E t -1 | a E t ) , I ( a E t -1 ; a E t ) can be safely ignored in the regularization. As a result, the regularization is about simply minimizing I ( a E t -1 ; φ t , a E t ) while ignoring the constant MI term.

Since the direct computation of Eq. (5) requires to know densities of a E t -1 , φ t , a E t , one needs to train sample-based MI estimators [3, 16, 28, 34]. Thanks to those estimators, the estimated MI can be minimized without confining any distribution (to be Gaussian). However, this approach still has issues similar to FCA such that (1) it lower bounds MI and consequently imposes (nested) minimax optimization and (2) it introduces an additional neural network particularly sensitive to hyperparameters.

4.2.2 HSCIC regularization

To bypass those shortcomings in information-theoretic regularization, we consider HSCIC [30], a kernel-based conditional independence metric for the past action leakage regularization.

Let H A , H Φ be RKHSs over A , Φ and k A , k Φ be their associated kernels, where Φ is the representation space induced by the encoder φ . Given n samples { ( φ t ( i ) , a E t -1 ( i ) , a E t ( i )) } n i =1 from D and φ , let K φ t , K a E t -1 , K a E t be the n × n kernel matrices where [ K φ t ] i,j = k Φ ( φ t ( i ) , φ t ( j )) , [ K a E t -1 ] i,j = k A ( a E t -1 ( i ) , a E t -1 ( j )) , [ K a E t ] i,j = k A ( a E t ( i ) , a E t ( j )) . Then, HSCIC estimator for past action leakage regularization can be defined as follows:

where W = ( K a E t + nλ I ) -1 , λ > 0 is a ridge regression coefficient, ⊙ is the element-wise matrix multiplication.

Since Eq. (7) can be estimated using the samples from the joint distribution p ( φ t , a E t -1 , a E t ) , we can directly plug-in HSCIC estimates into the regularization objective. By leveraging this estimator, HSCIC regularization offers several advantages compared to the conditional entropy regularization and MI regularization objectives discussed earlier:

Direct computation of the closed-form solution in Eq. (7) allows HSCIC regularization to bypass any nested optimization.

HSCIC regularization does not employ any additional deep neural networks that require careful hyperparameters.

Since HSCIC is a non-parametric measure, it does not impose any parametric assumption on the data distribution and does not require any density estimation.

These advantages strongly imply that promoting conditional independence via the HSCIC estimator will be more desirable compared to other estimators, thereby improving the overall effectiveness of the regularization. Hence, we propose HSCIC regularization to address the past action leakage problem, which we call PALR.

5 Experiment

In this section, we present the experimental results of our approach. Initially, we investigate the correlation between the extent of past action information leakage and BC's performance. Subsequently, we compare our approach with several offline ILOH baseline methods across four continuous control tasks from the MuJoCo simulator [41]: hopper , walker2d , halfcheetah , and ant , as well as one pixel-based autonomous driving task from the CARLA simulator [9]: carla-lane 2 .

Dataset For tasks from the MuJoCo simulator, we transformed them into POMDP scenarios for ILOH by excluding specific state variables (such as velocity information). The remaining state variables, like positional information and joint angles, were treated as observation variables at individual timesteps 3 . We organized these observations into fixed-size stacks to configure each

problem setting, denoted as [envname]-W[stacksize] with stack sizes w ∈ { 2 , 4 } . For a task from the CARLA simulator, we used pixel images as observations for ILOH. To extract features from these observations, we employed a pretrained ResNet [15], keeping its parameters fixed during training. In carla-lane task, we stacked the extracted features with a fixed stack size of w = 3 and used it as input for the policy. All our experiments utilized expert demonstrations from the D4RL benchmark dataset [10] to ensure the validity and reliability of our results.

Evaluation Metric For the performance evaluation of the learned policy, we measure the normalized score that ranges from 0 to 100. To evaluate how much the past action information is leaked into a policy π , we measure ̂ HSCIC( a I t ; a E t -1 | a E t ) using the estimator (7) in the held-out dataset. To estimate HSCIC estimator, we fix a ridge regression coefficient λ = 10 -5 and all kernels are chosen as Gaussian kernels with the bandwidth σ 2 = 1 .

5.1 Relationship between past action information leakage and performance

To see if the problem of the past action information leakage occurs, we conduct an empirical study using BC from observation histories with stack size w = 2 . The objective of this experiment is to confirm the correlation between the performance and the degree of past action information leakage, similar to FCA [44].

Experimental setup To achieve multiple policies at different levels, we train BC policies for 500K steps with 5 different seeds and 7 different training dataset size N ∈ { 100 K , 50 K , 30 K , 10 K , 5 K , 3 K , 1 K } (the number of observation history-action pairs). Using a held-out dataset composed of 2,000 samples, the degree of past action information leakage is measured by ̂ HSCIC( a I t ; a E t -1 | a E t ) of each policy π I .

0

20

40

60

80

Normalized

HSCIC

(

a I

t

;

a E

t

1

|

a E

t

)

20

0

20

40

60

80

100

Normalized Score

hopper

walker2d

halfcheetah

ant

Results To compare ̂ HSCIC of fully trained BC in all tasks, we normalize ̂ HSCIC of each policy using the maximum and minimum values in each task. Grouping trained policies by the dataset size, we report the mean value of both the normalized score and the normalized HSCIC estimates in Figure 1. It shows that there are clear negative correlations between the normalized HSCIC and the normalized score in all environments. The result implies that BC from observation histories tends to train a policy in which there is a negative correlation between the conditional dependence related to its action and the performance. This insight suggests a potential opportunity for employing regularization methods that enforce conditional independence.

5.2 Performance Evaluation

Experimental setup We evaluate the performance of our method and offline baseline methods across 5 environments see effectiveness of our method. We train policies using 5 different algorithms, including our method, for 500K, 1M training steps MuJoCo and CARLA tasks respectively with 5 different seeds.

Baseline methods We compare our method with 6 offline ILOH baseline methods: BC, KF [45], PrimeNet [46] , RAP [6] 4 , FCA [44], MINE [3]. While all of these are commonly based on BC, we specify their differences as follows:

BC : the standard BC algorithm from observation history, which optimizes an objective without any regularization term ( α = 0 ).

KF [45] : the weighted BC algorithm that assigns higher weights to keyframes, which contain actions that are highly predictable from their corresponding action histories.

PrimeNet [46] : a supervised learning method designed to prevent undesirable shortcuts by leveraging additional key inputs.

RAP [6] : it employs a dual-stream of policy representation that learns from both observation history and individual observations, maximizing a lower bound that enforces conditional dependence between the representation and expert action, given the past action.

FCA [44] : it maximizes the conditional entropy that corresponds to Eq. (4) with adversarial training, which is an instance of our regularized BC framework.

MINE [3] : it minimizes the MI estimate corresponding to Eq. (5) using MINE estimator [3], which is one of the representative sample-based neural estimators for MI. It is also an instance of our regularized BC framework.

Our method regularizes HSCIC with its estimator defined as Eq. (7), where Gaussian kernel with fixed bandwidth σ 2 = 1 are used for all kernels of φ t , a E t -1 , a E t and a ridge regression coefficient λ = 10 -5 . Across all regularization methods, we searched for the optimal α according to the best mean normalized score. Further implementation details can be found in Section C.

Results Table 1 summarizes the results of performance evaluation for each problem settings. Our method significantly outperforms other baselines in 7 settings out of 9 and shows competitive performance in the rest 2 settings. In particular, our method shows strong performance in carla-lane , highlighting its effectiveness in enhancing performance within high-dimensional offline ILOH scenarios. We observe that RAP shows the least competitive performance across all tasks. This is because they do not have any penalization of the dependence between the imitator action and expert action in their objective function. However, we also observe that FCA and MINE fail to show consistent improvement over BC in most tasks except walker2d . This is because the lower bound estimators of their regularization objectives are not sufficiently accurate even at the cost of their inefficient alternating optimization. On the other hand, our method consistently outperforms BC, which clearly indicates the effect of promoting the conditional independence as we discussed in Section 4.1.

To better understand the degenerate performance of FCA and MINE, we evaluate ̂ HSCIC( a I t ; a E t -1 | a E t ) of each regularization method during training using the held-out dataset in Figure 2a. In contrast to our method, the result clearly demonstrates that FCA and MINE commonly

0

100K

200K

300K

400K

500K

Iterations

0.016

0.017

0.018

0.019

0.020

HSCIC(

a I

t

;

a E

t

1

|

a E

t

)

BC

FCA

MINE

PALR (Ours)

0

10 0

10 1

10 2

10 3

10 4

10 5

10 6

0.4

0.5

0.6

0.7

0.8

0.9

Normalized score

0.00

0.02

0.04

0.06

0.08

HSCIC(

t

;

a E

t

1

|

a E

t

)

(a) HSCIC estimate during training on hopper-W4 . (b) Varying regularization coefficient in walker2d-W2 .

fail to promote conditional independence throughout the learning process, elucidating their underperformance. To further investigate, we measure the HSCIC and conditional MI estimates of the finalized imitator policies across 8 MuJoCo settings (refer to Section D.5). In essence, our method adeptly mitigates the leakage of past action information, surpassing the efficacy of alternative approaches.

Regularization coefficient In this experiment, we aim to ascertain how the selection of coefficient α influences the performance of our method. We evaluate the asymptotic performance of our method with varying values of α ∈ { 0 , 10 0 , 10 1 , 10 2 , 10 3 , 10 4 , 10 5 , 10 6 } on the walker2d-W2 problem setting, which shows the largest performance gap between PALR and BC among our problem settings. As depicted in Figure 2b, the converged regularization loss L reg -HSCIC progressively decreases as α increases. Notably, the α that minimizes the regularization loss is not necessarily equal to the optimal hyperparameter that maximizes performance. We observe that the selection of α is important, as it adjusts the trade-off between robustness to the leakage and alignment with expert data.

6 Conclusion and Future Work

Grounded in the classical notion of conditional independence, we proposed a principled regularization framework for BC that mitigates past action information leakage problem. Within our framework, we have explored multiple choices of conditional independence metric and compared their estimators. Finally, we identified that our method with HSCIC estimator is the most favorable regularization of BC over other choices in terms of robustness to the leaked information of past action. In an extensive set of experiments on D4RL datasets, we empirically showed that our method significantly outperforms baseline methods. We also observed in our experiments that all the comparing methods including ours were sensitive to the choice of α . Without assuming any interaction with the environment, it is challenging to find the optimal α only with the given offline dataset. We believe that the discovery of optimal α in offline manner would be an interesting research topic, which we leave as future work.

Acknowledgments and Disclosure of Funding

This work was partly supported by IITP grant funded by MSIT (No.2020-0-00940, Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, AI Graduate School Program (KAIST); No.20210-02068, AI Innovation Hub), NRF of Korea (NRF2019R1A2C1087634; RS-2023-00279680; NRF2018R1A5A1059921), Field-oriented Technology Development Project for Customs Administration through NRF of Korea funded by the MSIT and Korea Customs Service (NRF2021M3I1A1097938), ETRI grant (22ZS1100, Core Technology Research for Self-Improving Integrated AI System), KAIST-NAVER Hypercreative AI Center.

References

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations , 2016.

Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079 , 2018.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning , 2018.

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Chia-Chi Chuang, Donglin Yang, Chuan Wen, and Yang Gao. Resolving copycat problems in visual imitation learning via residual action prediction. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX , pages 392-409. Springer, 2022.

Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9329-9338, 2019.

Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems , 32, 2019.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning , pages 1-16. PMLR, 2017.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning , pages 1180-1189. PMLR, 2015.

Alessandro Giusti, Jérôme Guzzi, Dan C Cire¸ san, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters , 1 (2):661-667, 2015.

Abel Gonzalez-Garcia, Joost van de Weijer, and Yoshua Bengio. Image-to-image translation for crossdomain disentanglement. 2018.

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic Learning Theory: 16th International Conference, ALT 2005, Singapore, October 8-11, 2005. Proceedings 16 , pages 63-77. Springer, 2005.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Variational interaction information maximization for cross-domain disentanglement. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 22479-22491. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/ 2020/file/fe663a72b27bdc613873fbbb512f6f67-Paper.pdf .

Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence , 1998.

Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations , 2022.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard S. Zemel. The variational fair autoencoder. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. URL http://arxiv.org/abs/1511.00830 .

Charles A Micchelli and Massimiliano Pontil. On learning vector-valued functions. Neural computation , 17(1):177-204, 2005.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invariant representations without adversarial training. Advances in Neural Information Processing Systems , 31, 2018.

Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann Cun. Off-road obstacle avoidance through end-to-end learning. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems , volume 18. MIT Press, 2005. URL https://proceedings.neurips.cc/paper_ files/paper/2005/file/fdf1bc5669e8ff5ba45d02fded729feb-Paper.pdf .

Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research , 32(3):263-279, 2013. doi: 10.1177/0278364912472380.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, et al. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint arXiv:2110.10819 , 2021.

Junhyung Park and Krikamol Muandet. A measure-theoretic approach to kernel conditional mean embeddings. Advances in neural information processing systems , 33:21247-21259, 2020.

Roman Pogodin, Namrata Deka, Yazhe Li, Danica J. Sutherland, Victor Veitch, and Arthur Gretton. Efficient conditionally invariant representation learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=dJruFeSRym1 .

Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In NIPS , 1988.

Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988.

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning , 2019.

Francesco Quinzan, Cecilia Casolo, Krikamol Muandet, Niki Kilbertus, and Yucen Luo. Learning counterfactually invariant predictors. arXiv preprint arXiv:2207.09768 , 2022.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Proteek Chandan Roy and Vishnu Naresh Boddeti. Mitigating information leakage in image representations: A maximum entropy approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2586-2594, 2019.

Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences , 3: 233-242, 1999.

Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872 , 2021.

Gokul Swamy, Sanjiban Choudhury, J Bagnell, and Steven Z Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems , 35:17665-17676, 2022.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Dequan Wang, Coline Devin, Qi-Zhi Cai, Philipp Krähenbühl, and Trevor Darrell. Monocular plan view networks for autonomous driving. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2876-2883. IEEE, 2019.

Sean Welleck, Kianté Brantley, Hal Daumé Iii, and Kyunghyun Cho. Non-monotonic sequential text generation. In International Conference on Machine Learning , pages 6716-6726. PMLR, 2019.

Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 2564-2575. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/ file/1b113258af3968aaf3969ca67e744ff8-Paper.pdf .

Chuan Wen, Jierui Lin, Jianing Qian, Yang Gao, and Dinesh Jayaraman. Keyframe-focused visual imitation learning. In International Conference on Machine Learning , pages 11123-11133. PMLR, 2021.

Chuan Wen, Jianing Qian, Jierui Lin, Jiaye Teng, Dinesh Jayaraman, and Yang Gao. Fighting fire with fire: Avoiding dnn shortcuts through priming. In International Conference on Machine Learning , pages 23723-23750. PMLR, 2022.

Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. Advances in neural information processing systems , 30, 2017.

Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning , pages 24725-24742. PMLR, 2022.

Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International conference on machine learning , pages 7523-7532. PMLR, 2019.

Supplementary Material

Contents

A Details of HSCIC and Its Empirical Estimation

A.1 Definitions

Definition 1. (Reproducing kernel Hilbert space)

Let H X be a Hilbert space of functions f : X → R functions which the metric is defined by inner product ⟨· , ·⟩ H X . A symmetric function k X : X × X → R , which is called a reproducing kernel of H X , that satisfies

A space H X is called a reproducing kernel Hilbert spaces (RKHS) corresponding a reproducing kernel k X .

Definition 2. (Kernel Mean Embedding)

Given a distribution P X on X , assume ∫ X √ k X ( x, x ) dP X ( x ) < ∞ . We define the kernel mean embedding µ P X of P X as µ P X ( · ) = ∫ X k X ( x, · ) dP X ( x ) .

Definition 3. (Maximum Mean Discrepancy)

Given distributions P, Q on X and suppose kernel mean embeddings of P, Q exist on H , denote µ P , µ Q . Then, the maximum mean discrepancy (MMD ) between P X , Q Y is defined as follows:

Let F , G be Hilbert spaces. Define ( f i ) i ∈ I , ( g j ) j ∈ J to be orthonormal basis for F , G respectively. Define two linear operators L : G → F , M : G → F .

The Hilbert-Schmidt norm of the operator L is defined as:

The operator L is called Hilbert-Schmidt Operator when ∥ L ∥ 2 HS is finite.

Definition 5. (Cross-covariance Operator)

Given RKHS F , G with kernels k, l , feature maps ϕ, ψ respectively. Define the uncentered covariance operator ˜ C XY : G → F and the cross-covariance operator C XY : G → F as follows:

Definition 6. (Hilbert-Schmidt Independence Criterion)

The Hilbert-Schmidt Independence Criterion (HSIC) is defined as Hilbert-Schmidt norm of the cross-covariance operator C XY , i.e., HSIC( P XY ; F , G ) := ∥ C XY ∥ 2 HS . Equivalently, we can define HSIC as MMD between P XY and P X P Y with the product kernel, i.e.

where κ (( x, y ) , ( x ′ , y ′ )) = k ( x, x ′ ) l ( y, y ′ ) .

Definition 7. (Conditional Mean Embedding) Assuming a random variable X satisfies ∫ X √ k ( x, x ) dP < ∞ . Define the conditional mean embedding of X given Z as µ P X | Z ( · ) := E X | Z [ k X ( X, · ) | Z ] .

Note that µ P X | Z is a random variable. In the main text of our paper, we denote the realization of µ P X | Z with Z = z as µ P X | Z = z .

Definition 8. (Hilbert-Schmidt Conditional Independence Criterion)

Define the Hilbert-Schmidt Conditional Independence Criterion (HSCIC) between X and Y given Z to be

A.2 Empirical estimation

We start to explain from decoupling the conditional mean embedding into their deterministic function and conditional random variable.

Lemma 1. (Decomposition of conditional mean embedding, Theorem 4.1 of [30]) There exists a deterministic function F P X | Z : Z → H X , which satisfies µ P X | Z = F P X | Z ◦ Z .

By this lemma, we can decompose conditional mean embeddings and HSCIC into some deterministic function and conditioned random variable Z , i.e., µ P X | Z = F P X | Z ◦ Z, µ P Y | Z = F P Y | Z ◦ Z, µ P XY | Z = F P XY | Z ◦ Z, HSCIC( X,Y | Z ) = F HSCIC X,Y | Z ◦ Z respectively. By using vector-valued RKHS regression with a regularization parameter λ > 0 and the representer theorem [23], we can obtain closed-form estimates of F P X | Z , F P Y | Z , F P XY | Z as follows (see [30] for more details):

where k X ( · ) := ( k X ( x 1 , · ) , ..., k X ( x n , · )) ⊤ , k Y ( · ) := ( k Y ( y 1 , · ) , ..., k Y ( y n , · )) ⊤ , k Z ( · ) := ( k Z ( z 1 , · ) , ..., k Z ( z n , · )) ⊤ , [ K Z ] ij := k Z ( z i , z j ) and ⊙ is the element-wise multiplication operator of matrices. By plugging-in conditional mean embedding estimators, we can obtain a closed-form estimator of F HSCIC X,Y | Z as follows:

where [ K X ] ij := k X ( x i , x j ) , [ K Y ] ij := k Y ( y i , y j ) .

B Proof of Theorem 1

...

...

Proof. By the chain rule of MI, following equalities hold:

Thus, I ( φ t ; a E t -1 | a E t ) ≥ I ( a I t ; a E t -1 | a E t ) if and only if I ( a E t -1 ; φ t , a E t ) ≥ I ( a E t -1 ; a I t , a E t ) .

By the chain rule of MI,

Directly regularizing the imitator policy to enforce conditional independence may cause harmful results of training. This is because any policy that ensures the conditional independence does not necessarily yield optimal behavior. For instance, constant actions or actions sampled from uniform random distribution are independent of other random variables and thus satisfy Eq. (2), although they are not optimal in general. Based on Theorem 1, we adjust the representation of policy to inject the conditional independence. The effectiveness of our approach is compared with direct action regularization through an empirical study. The results are provided in Table 9 of Section D.1.

C Implementation Details

This section provides detailed specifications and implementation settings.

C.1 Observation specification

Based on the observation specification from Open AI Gym documentation 5 , we take a partial observation in each task by excluding certain variables (e.g. velocity information) in the original state variables. We mark the variables we used in our observation as ⃝ and not used are represented as × in the observation column of the corresponding table.

C.1.1 Hopper

C.1.2 Walker2d

C.1.3 HalfCheetah

Open AI Gym documentation does not match with the XML file of halfcheetah 6 , we denote both descriptions on the following tables.

C.2 Dataset preprocessing

MuJoCo Tasks We utilized the expert dataset provided in D4RL benchmark that has a suffix -expert-v2 . For all experiment, we configured partial observations from the original data based on Section C.1. These observations are standardized with statistics from the entire dataset. Each dataset in D4RL benchmark has a total of 1M transitions. Given the size of the training data D , we take the first D samples. Among the last 100K samples in the entire dataset, we also sampled heldout data which are not included in the training data. Given a stacksize w ∈ { 2 , 4 } , we configure each transition as (observation history, action, past action) tuple. For the main performance evaluation experiment (Table 1), we use 30K transitions for training.

CARLA Tasks We extract features from observation images using the pretrained network ResNet34 [15] and consider these features as policy inputs during training. For the performance evaluation (Table 1, 10), we utilize 1K and 100K transitions for carla-lane , carla-town respectively.

C.3 Hyperparameter selection

This section outlines the settings of the hyperparameters for each method. To ensure a fair comparison, we aim to maintain consistent network architecture and policy-related hyperparameters, whenever feasible. For all baselines and our method, we employ a policy architecture consisting of fullyconnected layers with [128, 64, 128] hidden units. The layer with 64 units is treated as a policy representation, denoted as φ .

KF [45] We used a softmax function as a weighting function of the weighted BC in MuJoCo tasks and a step function in CARLA tasks, aligning with the the original paper's experiments. We adopt the action prediction network for the additional network. Additionally, we tuned hyperparameters for the the softmax temperature in MuJoCo tasks, analogous to α . In CARLA tasks, we used the same hyperparameters (threshold=0.1, weight=5) consistent with the original paper [45].

PrimeNet [46] We treat observation histories as raw inputs and individual last observations as key inputs, consistent with the approach used in the original experiments. The architecture of the prime network is implemented as the same with the policy.

RAP [6] RAP incorporates two streams of policy representation: (1) one stream derived from observation history (referred to as the memory extraction stream) and (2) the other originating from a single observation (referred to as the policy stream). For the memory extraction stream, we use an architecture of [128, 64, 300] hidden units that outputs predictions of a residual of expert actions. The layer with 64 units is employed as a representation. For the policy stream, we similarly utilize an architecture of [128, 64(+64), 128] which outputs a policy action. The middle units indicate that the policy utilizes both the single observation representation and the memory-extracted representation (with a stop-gradient layer) as inputs.

A summary of hyperparameters is provided in Table 7. The first 3 rows of the table display the hyperparameters universally applied across all methods. The next 4 rows detail hyperparameters for methods that incorporate additional neural networks. The last 2 rows of the table present hyperparameters specific to our method, encompassing hyperparameters for HSCIC estimates.

Optimal α As discussed in Section 5.2, we train 5 policies incorporating different regularization coefficients α within the set { 0 . 01 , 0 . 1 , 1 , 10 , 100 } for MuJoCo tasks and { 0 . 001 , 0 . 1 , 10 , 1000 } for CARLA task. We then select the best α for each problem setting. Table 8 shows the chosen values of α for each method and each problem setting, as utilized in the experimental results of the paper.

D Additional Experiments

D.1 Direct regularization on imitator action

To investigate the effectiveness of representation regularization, we compare our method to the one that directly regularizes action in MuJoCo settings. The objective function of the direct method is defined as follows:

We optimize the imitator policy π with the objective by using the reparameterization trick [21]. The method is also trained with different regularization coefficients α ∈ { 0 . 01 , 0 . 1 , 1 , 10 , 100 } and we select the best α of each method. Table 9 summarizes the results. Our representation regularization method (PALR) is more successful in 6 out of the 8 problem settings when compared to direct action regularization. In ant-W2 , ant-W4 problem settings, neither method demonstrates significant effectiveness when compared to BC method.

D.2 Experiment on carla-town

D4RL benchmark provides two CARLA tasks: carla-lane , carla-town . As shown in Table 10, all algorithms fail to show meaningful imitation results, as none of them exceed random performance levels. This outcome is likely due to the dataset containing numerous instances of the same observations paired with different actions (59,984 observation-action pairs out of 100K pairs in total). Consequently, we have omitted the experiment results for carla-town from Table 1.

D.3 Comparison with VIB (Variational Information Bottleneck)

To highlight the leakage of past action information is not trivially prevented by general representation regularization methods, we compare our approach with VIB [1], a representative method. We train BC with VIB with two different β values, specifically, β ∈ { 10 -3 , 10 -5 } , 5 seeds for each problem setting. whereas our method demonstrates significant improvements in 6 out of 8 problem settings.

D.4 Effectiveness on DT (Decision Transformer) policy

To assess the efficacy of our method in regularizing policies with complex network architectures, we conducted additional experiments employing the Decision Transformer (DT) [5], one of the prominent offline RL methods. Given that reward information is absent in the offline IL dataset, we inserted a reward input of 0 into DT's structure to retain its original configuration. Our regularization approach, termed DT-PALR, was applied to the last hidden state of DT. We evaluated both the standard DT and DT-PALR across 3 POMDP versions of MuJoCo tasks, consistent with the scenarios outlined in Table 1.

The results are presented in Table 12. With the exception of the halfcheetah task, it becomes evident that DT's performance lags behind that of BC, as demonstrated in Table 1. This divergence might be attributed to DT's utilization of a larger network size, rendering it more susceptible to capturing spurious causal relationships in tasks where access to complete states and rewards is restricted. Encouragingly, our results indicate a substantial enhancement in the performance of DT for the hopper and walker2d tasks when subjected to the DT-PALR method. This observation strongly suggests the adaptability of our approach to intricate architectures.

D.5 Quantitative analysis on past action information leakage

We comprehensively evaluate HSCIC scores across 8 MuJoCo problem settings in Table 13. The results demonstrate that our method consistently reduces the conditional dependence. Notably, our method achieves lowest HSCIC scores in entire problem settings. Moreover, to show a more robust assessment of conditional dependence, we also estimate conditional mutual information (CMI). (see Table 14) As we described in Section 4.2.1, CMI can be decomposed into two MI terms, I ( a E t -1 ; φ t , a E t ) and I ( a E t -1 ; a E t ) , we estimate them using MINE [3] respectively. The table shows that our approach consistently presents lower CMI estimates compared to BC across 6 out of 8 problem settings.

D.6 Details on the leakage-performance correlation

We first clarify the relationship between the number of training data and HSCIC scores in Section 5.1. To clarify, we plot the correlation into Figure 4. The correlation indicates that when the training data is insufficient, the problem of past action information leakage becomes more severe. This phenomenon can be attributed to the higher risk of overfitting in cases with smaller training instances, which can lead to the capture of false causal relationships within the training data. These findings align with similar results reported in [8] (see Figure 4 in [8]).

1K

10K

100K

Number of Training Dataset

0

20

40

60

80

Normalized

HSCIC

(

a I

t

;

a E

t

1

|

a E

t

)

hopper

walker2d

halfcheetah

ant

Furthermore, to provide more insights for understanding Figure 1, we plot all points that indicate the normalized HSCIC and the normalized score of each BC policy in Figure 5. In addition, we calculate the sample Pearson's correlation coefficients using every point of each task (the first 4 columns) and all tasks (the last column) on Table 15.

0

20

40

60

80

100

Normalized

HSCIC

(

a I

t

;

a E

t

1

|

a E

t

)

40

20

0

20

40

60

80

100

Normalized Score

hopper

walker2d

halfcheetah

ant

E Broader Impacts

This work primarily addresses the reduction of the pervasive past action leakage effect in imitation learning with observation histories. Notably, our findings suggest potential vulnerabilities that may be exploited by adversarial actors. For instance, malicious attackers could introduce the conditional dependencies into the system, leading to potentially catastrophic outcomes, such as repeating the same actions. This will produce potentially irreversible consequences on the system, especially for security-critic tasks such as autonomous driving and medical devices, or industrial automation. Therefore, understanding and mitigating this risk is crucial for the safe and efficient operation of such systems.

F Limitations

While our approach is easily applicable to real-world tasks, the range of experiments in this paper is limited to robotics simulation. A broader scope of experiments, including more realistic observation settings like image-based or noise-inclusive observations, could provide a stronger argument for the effectiveness and real-world applicability of our method. Additionally, our method assumes that control-relevant information is sufficiently captured in the observation histories. However, this assumption might not hold in some real-world scenarios where key control information is missing and not retrievable from the observation histories. In such cases, the learning model requires additional information, inductive bias or expert knowledge to function effectively.

G Computation Resources

For our experiments, we used a cluster system with 20 nodes that have the following system specs:

CPU: Intel i7-9700K CPU (3.60GHz)

GPU: TITAN Xp (VRAM 12 GB)

H Licenses

Our code has been developed based on publicly available code repositories released under MIT licenses. The code used to generate our experimental results also follows to the MIT license. For more detailed information, see the README.md and LICENSE files included in our code files.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Past Action Leakage Regularization (PALR) for blocking the leakage of past action information during behavior cloning. Concretely, PALR focuses on the problem when a BC agent simply remembers the past actions rather than learning a generalized behavior, which gives a degenerated policy. Specifically, PALR tackles the problem by learning a representation of the history to remove unnecessary from the observations via conditional independence. PALR adopts the HSCIC metric to measure the conditional independence, which alleviates the shortcomings of the information-theoretic metric. In their experiments, the authors compared different metrics for behavior cloning on a range of continuous control tasks.\n\nSTRENGTHS:\nThis paper is overall well-written and easy to follow. The idea is well-motivated as well.\n\nWEAKNESSES:\nThe paper has the following main issues.\n\nFirstly, the contribution of this paper is relatively incremental. The formulation of the BC regularization as conditional entropy is new. Wen et al. [1] was the first to propose this formulation. As a result, the main contribution of this work is only the HSCIC regularization term.\n\nIn addition, the paper has claimed that the focus of the work is POMDPs, where the past action leakage might come from the history. However, in the experiments, the only domain considered is state-based continuous control, which is a fully observable MDP. In my opinion, such a domain does not support the claim of the work, and is unsuitable to measure the capability of the proposed method. More complex and natural POMDP domains should be considered, e.g., indoor navigation with pixel-based observations, to better support the claims.\n\nReferences:\n\n[1] Wen, C., Lin, J., Darrell, T., Jayaraman, D., & Gao, Y. (2020). Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 33, 2564-2575.\n\nQUESTIONS:\nHow does the method compare with more advanced imitation learning algorithms that require interactions with the environment, e.g., DAC [1] and PWIL [2]? BC is a relatively weak baseline. Can this regularization term be used in these algorithms to further boost the performance?\n\nReferences:\n\n[1] Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., & Tompson, J. (2018). Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925.\n\n[2] Dadashi, R., Hussenot, L., Geist, M., & Pietquin, O. (2020). Primal wasserstein imitation learning. arXiv preprint arXiv:2006.04678.\n\nLIMITATIONS:\nThis paper has relatively limited novelty and insufficient experiment results. Given its current state, I think the paper is not ready for publication yet.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper solves the past action information leakage problem in behavior cloning. The paper first formally defines this problem, and then provides some potential regularization methods. After careful analysis of the pros and cons of each method, the authors decide to use HSCIC and validate its performance on several tasks.

STRENGTHS:
- The flow of this paper is very clear: problem definition -- potential solutions -- analysis -- evaluation
- In the evaluation, the authors not only show the improvement of the final performance of BC but also highlight the correlation between the performance and HSCIC regularization. This strongly proves that regularization is the major contribution to performance improvement.
- Comprehensive ablation study and analysis are provided in the paper.

WEAKNESSES:
The major concern is how to apply this regularization to complex environments with images (or even multi-modality) as input and high-dimensional action space (6 DoF or even more). If the proposed method can be applied to such complex tasks, I'm willing to increase my score.

QUESTIONS:
The BC policy used in this paper is just a feed-forward policy p(a|s).
Is it possible to apply the regulation to other kinds of policy, for example, an energy-based model [1] or even a diffusion model[2]? 

[1] Florence, Pete, et al. "Implicit behavioral cloning." Conference on Robot Learning
[2] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." arXiv preprint arXiv:2303.04137

LIMITATIONS:
no explicit limitation discussion are provided in the paper
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use HSCIC (Hilbert-Schmidt Conditional Independence Criterion) to alleviate the leakage of information from past actions in behavior cloning from observation histories. The advantage of HSCIC compared with information-theoretic regularization is it can compute in closed-form and does not require parametric assumption on the data distribution.\n\nExperimental results show HSCIC is a good indicator of agent performance and by directly minimizing HSCIC, it consistently improves over naive BC and other baselines with different regularizations.\n\nSTRENGTHS:\nExperimental results show significant improvement over baselines across different locomotion tasks.\nGood analysis showing negative correlation between HSCIC and agent performance, which motivates to minimize it directly in training objectives.\nWriting is clear and straightforward. Good comparison of kernel-based regularization and information-theoretic approaches.\n\nWEAKNESSES:\nLack of comparisons with non-regularization based approaches to tackle past action information leakage. For example, Key-frame focused visual imitation learning proposes a sampling/weighting approach and Fighting fire with fire: Avoiding DNN shortcuts through priming proposes to use additional prior information as inductive biases.\nExperiments are all on simple state-based results. Hard to judge the performance on more complex domains, e.g. visual imitation learning, robotic manipulation or autonomous driving.\nKernel-based approaches have time complexity depending on the number of samples. It would be worth comparing the computation tradeoff between different approaches.\n\nQUESTIONS:\nHow well does the proposed methods work on more complex environments, e.g. Atari, Carla, etc.?\nCould you show some quantitative results on how the agents avoid failure due to leakage of previous action?\nIs there any heuristics of turning the hyperparameter alpha?\nCould this regularization be combined with other approaches (e.g. information bottleneck, sampling) to give better results?\n\nLIMITATIONS:\nAuthors state theirhyperparameters are sensitive and hard to tune without interaction with the environment. No broader impact section found in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper aims to solve the problem of negative action leakage from past observations in the context of imitation learning. This is specifcally applicable to imitation learning when considering a history of observations. The paper argues that an information theoretic (entropy or MI based) regularization requires training an additional network and a nested optimization, however, HSCIC avoids both of these inefficiencies. The experiments show that (1) HSCIC is a good indicator of leakage, and (2) the propose approach which regularizes the BC loss with HSCIC outperforms other information theoritic and vanilla baselines quite a bit, in offline imitation settings.

STRENGTHS:
- To my knowledge this is a novel approach 
- The method is very interesting, and explained in a way that is easy to understand, and is also an innovative solution 
- The motivation is well described, and this is an important area to explore 
- The results show strong performance, and the analysis of the utility of HSCIC is insightful as well 
- The paper is well written 

In general, I believe this paper should be presented at the conference.

WEAKNESSES:
I think a couple of things could have been discussed: 

- How sequence modelling architectures can play a role here such as transformer/Decision Transformer (Chen et al., 2021)
- What are the different modes of leakage, and do they affect performance in different ways

QUESTIONS:
See weaknesses

LIMITATIONS:
Limitations are sufficienlty addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the information leakage problem of imitation learning with observation histories. To this end, the paper measures the leakage of past action information based on conditional independence and proposes Past Action Leakage Regularization (PALR) for behavioral cloning (BC). The experiments show that the proposed method outperforms four baselines on four MuJoCo continuous control environments. Ablation studies suggest PALR can improve BC with a proper coefficient. This work defines and studies an essential problem of imitation learning from partially observable environments. However, the experiments can be improved to make the work more concrete.\n\nSTRENGTHS:\n**Clarity**\n- The overall writing is clear. The paper gives clear descriptions in both theoretical and intuitive ways. The notations, formulations, and theorems are well-explained. \n\n**Ablation study**\n- Ablation studies are comprehensive. The provided ablation studies help understand the effectiveness of the regularization coefficient (Sec.5.2) and the target to apply regularization (Sec.D).\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed framework.\n- Given the clear description in the main paper and the details provided in the supplementary materials, I believe reproducing the results is possible.\n\nWEAKNESSES:\n**Method**\n- This work proposes a kernel-based method to regularize BC for imitation learning from partially observable environments. While I am aware of the advantages of kernel-based methods (stable, does not need additional networks or hyperparameters tuning), recent works use neural networks, such as variational model [1] or causal transformer [2], to capture essential information from observation histories and have shown promising results. \nThe effectiveness of the proposed PALR would be more convincing if the authors could demonstrate comparisons to the above methods. \n\n**Experiments**\n- Some parts of the experimental results are not easily interpretable. Figure 1 shows the negative correlation between the number of training data and the HSCIC score. Can the author explain why the information leakage problem is more severe when the training data is insufficient?\n\n- I am not entirely convinced by the explanation of why FCA [3] and MINE [4] get inferior results (Line 340-345 & Figure 2a).\nThe author only evaluates the HSCIC score of each method on hopper-W4. However, the proposed PALR is directly regularized by the HSCIC metric, so it is straightforward that the proposed PALR gets the lowest HSCIC score.\nIt would be better to evaluate the HSCIC score on all environment setups in Table 1 to 1) support the negative correlation between HSCIC estimations and the performance of algorithms and 2) analyze the performance of FCA and MINE, which outperform BC only on walker2d and halfcheetah environments.\n- The paper only evaluates the proposed method on four continuous control environments, which is insufficient. RL tasks such as navigation, robot arm manipulation, or Atari games can also be considered.\n\n**Experiment details**\n- The details of the normalized score (line 280) are missing. What is the maximum score and the minimum score?\n\n**Typo**\nLine 323: ..., which is a one of the ...\n\n[1] Rafailov, R., Yu, T., Rajeswaran, A., & Finn, C. (2021). Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 34, 3016-3028.\n[2] Bonatti, R., Vemprala, S., Ma, S., Frujeri, F., Chen, S., & Kapoor, A. (2022). Pact: Perception-action causal transformer for autoregressive robotics pre-training. arXiv preprint arXiv:2209.11133.\n[3] Wen, C., Lin, J., Darrell, T., Jayaraman, D., & Gao, Y. (2020). Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 33, 2564-2575.\n[4] Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., & Hjelm, D. (2018, July). Mutual information neural estimation. In International conference on machine learning (pp. 531-540). PMLR.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Past Action Leakage Regularization (PALR) to resolve the copycat problem in behavior cloning (BC) methods:\n\n1. mathematically defines the past-information-leakage problem.\n\n2. introduces PALR to formalize the methods.\n\n3. uses Hilbert-Schmidt Conditional Independence Criterion(HSCIC) to measure the conditional independence, and analyses the advantages of HSCIC over CMI as the metric.\n\n4. provides experiments verifying the correlation between HSCIC and performance, and the superiority of the HSCIC-based method over baseline methods.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow in most parts.\n\n2. The authors provide sufficient experiments to verify their claims.\n\n3. The idea of replacing CMI with HSCIC is natural in maths and easy to apply. It does bring a big improvement over the previous methods. \n\n4. The paper comprehensively explains how HSCIC becomes a better metric than CMI.\n\nWEAKNESSES:\n1. Some small typos: $H_{\\mathcal{X}}$ in line 133 should be $\\mathcal{H}_{\\mathcal{X}}$; the full name of MOMDP should use \\emph{observable} other than \\emph{observed} in line 153. I didn't go through every word, so there may be other mistakes. \n\n2. In line 203, $\\phi_t=\\phi(z_{t_\\omega:t})$, but in line 210, 212, $\\phi\\sim\\phi(z_{t_\\omega:t})$. This caused some confusion in reading. Did I miss anything?\n\n3. In line 193, ''Ideally we would like to achieve conditional independence in Eq. (2)\" is not obvious. And actually, it's not reasonable to require eq (2) to be true, for even a perfect imitator can't do this. The structure of MOMDP described in section 3.2 determines that it's not mathematically possible to predict $A_t^I$ given $A_t^E$. A better objective may be minimizing the CMI, as is practically done in the experiment. Furthermore, I know that this has been an unsolved problem from the first time copycat problem being proposed, but you may modify it by coming up with a better lower bound of CMI or HSCIC other than 0 to fix this mathematical flaw.\n\n4. Theorem 1 does not help much in proving the claim that lower loss indicates lower dependence, which only proves the equality between two independences. I believe that this is true and thus PALR is effective. Can you come up with a new theorem to fix this?\n\n5. Some important references are missing:\n\n[1] Ortega P A, Kunesch M, Delétang G, et al. Shaking the foundations: delusions in sequence models for interaction and control[J]. arXiv preprint arXiv:2110.10819, 2021.\n\n[2] Wen C, Lin J, Qian J, et al. Keyframe-Focused Visual Imitation Learning[C]//International Conference on Machine Learning. PMLR, 2021: 11123-11133.\n\n[3] Wen C, Qian J, Lin J, et al. Fighting fire with fire: Avoiding dnn shortcuts through priming[C]//International Conference on Machine Learning. PMLR, 2022: 23723-23750.\n\n[4] Spencer J, Choudhury S, Venkatraman A, et al. Feedback in imitation learning: The three regimes of covariate shift[J]. arXiv preprint arXiv:2102.02872, 2021.\n\nQUESTIONS:\n1. Do you have any explanation for the phenomenon that more historical information damages the performances in HalfCheetah? \n\n2. Why only caring about $A_{t-1}^E$? I think $A_{t-2}^E$ can also cause leakage. Or can you prove that the former independence indicates the latter one?\n\nLIMITATIONS:\nAs mentioned in Weaknesses, the current theories are not rigorous. They can be revised to make the whole logic consistent.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}