Regularized Behavior Cloning for Blocking the Leakage of Past Action Information
Authors: Seokin Seo
HyeongJoo Hwang
Hongseok Yang
Kee-Eung Kim
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Imitation learning
Information leakage
Causal Confusion

Abstract: For partially observable environments, imitation learning with observation histories (ILOH) assumes that control-relevant information is sufficiently captured in the observation histories for imitating the expert actions. In the offline setting wherethe agent is required to learn to imitate without interaction with the environment, behavior cloning (BC) has been shown to be a simple yet effective method for imitation learning. However, when the information about the actions executed in the past timesteps leaks into the observation histories, ILOH via BC often ends up imitating its own past actions. In this paper, we address this catastrophic failure by proposing a principled regularization for BC, which we name Past Action Leakage Regularization (PALR). The main idea behind our approach is to leverage the classical notion of conditional independence to mitigate the leakage. We compare different instances of our framework with natural choices of conditional independence metric and its estimator. The result of our comparison advocates the use of a particular kernel-based estimator for the conditional independence metric. We conduct an extensive set of experiments on benchmark datasets in order to assess the effectiveness of our regularization method. The experimental results show that our method significantly outperforms prior related approaches, highlighting its potential to successfully imitate expert actions when the past action information leaks into the observation histories.

Regularized Behavior Cloning for Blocking the Leakage of Past Action Information
Scokin Sco', HyeongJJoo Hwang', Hongseok Yang* Kee-Eung Kim
Kim Jaechul Graduate School AI, KAIST
School of Computing, KAIST
siseoβai aateolai.kaist.oo kww.t.iitt....... ac.kk, kr, hjhwang@ai kaist. kr, hongseok. yangokaist.n.. ac. ac.kr, kr, kekim@kaist.ac. ac.
Abstract
For partially observable environments, imitation learning with observation histories (ILOH) assumes that control-relevant information sufficiently captured in the observation histories for imitating the expert actions. In the offline setting where the agent is required to learn to imitate without interaction with the environment, behavior cloning (BC) has been shown be simple yet effective method for imitation learning. However, when the information about the actions executed in the past timesteps leaks into the observation histories, ILOH via BC often ends up imitating its own past actions. In this paper, address this catastrophic failure by proposing principled regularization for BC, which we name Past Action Leakage Regularization (PALR). The main idea behind our approach is to leverage the classical notion conditional independence to mitigate the leakage. We compare different instances of our framework with natural choices of conditional independence metric and its estimator. The result of our comparison advocates the use of particular kernel-basee estimator for the conditional independence metric. We conduct extensive set of experiments order to assess the effectiveness of our regularization method. The experimental results show that our method significantly outperforms prior related approaches, highlighting its potential to successfully imitate expert actions when the past action information leaks into the observation histories.
benchmark datasets in
Introduction
Imitation learning (IL) aims at learning policy that recovers expert's behavior from demon- stration dataset. Leveraging the information about state and expert action available in the dataset,
IL has been successful in many real-world applications 4
12
26
27
32
38
43
Although IL
real-world tasks often impose restrictions on interacting with the environment due to safety, cost, and ethical concerns. Consequently, the practical necessity lies in the development of effective offline II al- gorithms. Behavior cloning (BC) 33 is one of the most prominent offline IL algorithms, which leams to predict expert's action for each given state using offline expert dataset via supervised learning. BC has gained widespread recognition 7 42 for providing straightforward and effective solution, especially when full access to state information is available and the dataset is sufficiently extensive. However, when the state information is only partially available an observation at each timestep, which closer to the realistic scenario (e.g. an autonomous vehicle with a limited number
problems can be addressed using either online [17] 36 or offline algorithms
20
33 48
37th Conference Neural Information Processing Systems (NeurIPS 2023).

---

of sensors), training with any IL algorithm can be complicated. To enhance the ability of agent to infer missing control-relevant information from observations, incorporating the history of observations from adjacent past timesteps as input can be beneficial 2 24).
When utilizing observation history for behavior cloning, a notable challenge emerges: the unnecessary dependence of the imitator's actions the preceding actions, resulting in suboptimal behavior in test time. In extreme cases, this dependency can lead to undesirable behaviors, such as the imitation policy that merely replicates its own actions from previous time steps 0 which leads to a catastrophic result particularly in safety-critical tasks like autonomous driving. This is often rooted at the inability to differentiate between actual and spurious causal relationship between observation features and expert actions within the collected data [8] Past action information is a representative instance of such nuisance features, since it is a strongly correlated the target expert action. This correlation can mislead the II. algorithm into recognizing past action information, which is not causally related to the target expert action, crucial feature for prediction. In this work, we refer to this misleading phenomenon as past action information leakage.
natural way to mitigate this phenomenon is to adopt a mechanism that blocks the leakage of past action information irrelevant the target expert action. In fact, Wen et al. 44 an adversarial training approach discard such redundant information in the representation of the observation history. This is achieved by maximizing the conditional entropy of the expert's 's previous action given the joint of the representation and target expert action. However, due to the intractability of direct computation of the conditional entropy. this approach relies on unstable adversarial learning, More recently, Chuang et al. 66] focuses on image-based observations and attempts to mitigate the past action dependency by splitting the policy representation into two parts: a representation of the observation history and representation of the current observation. This method assumes a particular policy structure and does not provide systematic general approach to block the leakage of past. action information into the representation.
proposed
In this paper, we present a regularized behavior cloning framework that effectively mitigates the leakage of past action information. We formally define the problem of the Icakage of past action information and establish a metric to quantify the magnitude of the leakage, employing the kernel- based method called HSCIC (Hilbert-Schmidt Conditional Independence Criterion) B3). Building upon the metric, devise objective function that addresses the aforementioned problem, allowing for a more comprehensive understanding and interpretation of existing work 44]. Moreover, propose a stable and efficient kernel-based regularization method that circumvents challenges such as ad adversarial learning, nested optimization and reliance on neural estimator. Lastly, we conduct an extensive set of experiments which empirically show that our regularization method effectively blocks the leakage of past action information across a variety of control task benchmarks.
Our contributions summarized follows:
We formally define the problem of the Icakage of past action information based on the concept of conditional independence by quantifying the amount of leaked past action information.
We introduce principled framework for behavior cloning with Past Action Leakage Reg- ularization (PALR), which prevents the imitator from overfitting to leaked past information.
We provide experimental results on established benchmarks, demonstrating the effectiveness of our method.
2 Related Work
Invariant representation learning Learning representation invariant to any unwanted factors
has been widely studied in various domains such as fair classification (2 25
37 47. domain
adaptation [Т 49), and imitation learning 66
44
One of the dominant approaches invariant Adversarial learning algorithms
representation learning is adversarial learning
471
'The problem has many names: copycal problem 6 44
inertia problem 7. lutching effect 40 Beyond the falsely
leaded action repeating behavior, we introduce the past action information leakage separate the reason and the result.

---

commonly train additional networks that predict the unwanted factors from the representations while enforcing representations to make those prediction models fail. Consequently, they require alternating optimization between the main and the additional models 13 instability [[]]. To bypass those shortcomings of adversarial learning, several theoreiiion-theoreticc approaches [18 25] have been proposed. Based on Variational Auto-Encoders (VAE) [21), these
or show numerical
methods proposed end-to-end learning algorithms that jointly optimize all of their components with numerical stability. However, these methods assume that the distribution of the representation is Gaussian, which restricts the flexibility of the representation. Wen et al. 46 concentrated on climinating shortcuts in supervised learning by incorporating supplementary key information, demonstrating its enhancement of behavior cloning with observation histories. Recent methods have proposed to learn counterfactually or conditionally invariant representation, leveraging on kernel- based conditional independence metric 31335 These methods demonstrated promising empirical results in synthetic domains, effectively mitigating the impact of nuisance correlations.
Information leakage in imitation learning There a growing understanding of the importance of addressing the correlation between expert actions and nuisance features, which are not essential Especially, termed as causal confusion (8) imitation learning exhibits paradoxical phenomenon: having more information can lead to worse performance. Aligning with such observation, recent works have also demonstrated that accessing more information from the observation history leaks past action information so that the imitator may
for control and may even hinder performance. [8 29 39
learn an undesirable policy that simply repeats the same action in the past 66 avoid learning degenerate solutions from the past action information, Wen al.
44 [45.. To
44 proposed the
regularization method based on the conditional entropy of the previous action given the representation of the history and the current action. However, their method involves nested minimax optimization along with an additional neural network, which complicates the training process. We closely investigate their formulation in Section 4.2.1 to show their limitations as well as their connection to our method. Wen et al. also introduced the action predictability metric to quantify the dependence between past action histories and imitator actions, relative to expert actions. However, this metric not based on conditional independence, which is central to our argument concerning the past action leakage. Wen et al. 45) addressed this problem using a weighted behavior cloning method that upweights "keyframe" samples, which more likely to be predicted from the action histories. Swamy et al. [40] showed that online interaction is both necessary and sufficient to resolve repeating behavior. However, in many real-world applications [4, 12 26 27
32 38 43), online interaction is often infcasible due to safety, cost, and ethical considerations. In this work, we develop an offline algorithm that can robustly handle the leakage of past action information.
3 Preliminaries
3.1
Conditional independence
For random variables X, Y which taking value in X/ respectively, we write Px. Pxy, and Px for the marginal distribution of X, the joint distribution of X and Y, and the conditional distribution of X given Y, respectively. We say that random variables X and Y are conditionally independent given a random variable or simply Z-conditionally independent PxyIz = Pxi (x) Pyiz (v) for all xr y X x V. Also, we say that X and Y are independent if Pxy (z. y) (x) (1) for all x. y. We denote Z-conditional independence by X IL Y Z, and independence by X L Y.
Conditional Mutual Information (CMI) Mutual information (MI) is an information -theoretic quantity to measure the dependency between two random variables. MI between random vari- ables X and Y is defined as I(X; Y) = DRI. ( Px = Ex [log (xy y) log Px (x) Py]]] MI is always non non-negative (i.c., I(X;Y) 2 0), and it becomes zero and only if X IL Y. Conditional mutual information (CMT) is defined similarly, but with conditional distributions. CMI between random variables X and Y given a random variable Z is defined as
I(X;Y| Z) Epz [KKL (PxyIz, Px]zPr]z)]- As in the case of MI, I(X;Y Z) 2 0, and also
Y(X;Y| Z) = 0 if and only if X IL Y Z.
3

---

Hilbert-Schmidt Conditional Independence Criterion (HSCIC) Hilbert-Schmidt Independence Criterion (HSIC) [14] is a kernel-based measure that quantifies the dependency between two random variables. Let Hxx and My be reproducing kernel Hilbert spaces (RKHS) on X' and y with the corresponding reproducing kernels kx and ky. respectively. The tensor product & of the kernels a binary function on X x y defined by (kx @ (z)((21 81), (22. == kx (21, 22 22)ky (y) y2). The RKHS on X' 3/ with the kernel xxx ky is called the tensor product RKHS of H.x and Hy, and it is denoted by Hxx @ Hy. HSIC between X and Y is defined to be the maximum mean discrepancy (MMD) between Pxy and Px Py under H.x & Hyy, i.e, the distance in the Hilbert space H. 6 Hy between the so-called kernel mean embeddings of Pxy and Px Py to that space:
HSIC Y)= == -MDD (Pxy, Px Hx @ Hy)
= ||wpxx MPx & HPy 1l
where jx is the kernel mean embedding of the distribution Pxy Hx @ Hy, i.e. pX (2, v) Exxy [hkx @ ky)((X,)) Y), (I, v))], and Px MPy are defined similarly but using Hx. Hy instead. The (pPx @ (oy) is simply the function in Hx @ Hy that maps (z, y) to MPx (x)pp, (v). When kx @ by satisfies some condition (i.e., characteristic), HSIC(X, = if and only if X 11 Y 14.
HSCIC Htilbert-Schmid Conditional Independence Criterion) is an extension of HSIC that quantifies the amount of conditional dependency between two random variables given another random variable. In this paper, we follow the definition of HSCIC based conditional mean embedding (30), and use an estimator of HSCIC that draws samples from the joint distribution Pxyz and performs vector- RKHS regression. The conditional mean embedding of X given Z is a function in the RKHS H.x that is parameterized by the value of Z. Given a value = of Z, it maps x X to p (x) == Exxzze [bxx (X, 2)). Then, HSCIC is a mapping from a value of to a non-negative real defined as follows: for all 2 E Z.
HSCIC(X, Y = =) 1111| | PX @ 1
When hx @ by satisfies the condition mentioned from above, X
2 if and only Epz [HSCIC(X, = 0 30) We use an empirical estimator of the expectation here, denoted by HSCIC(X, YlZ). See Section A in the supplementary material for more details for definition of HSCIC and its empirical estimator.
3.2 Imitation learning from observation histories
We consider a Partially Observable Markov Decision Process (POMDP) 19 without reward, which is defined as a tuple of (S, 2, A, O, po). Here S, Z and A are an (underlying) state space, an observation space and an action space, respectively. The next O: - AZ specifies the conditional probability O(2[s) of observation z Z given state s E S. Finally, Pu E AS defines the probability (o(5)) that the process starts from 80 S, and P S x A -> AS defines the probability P(s'|s, [ef|ss,o a) of transitioning to state s' when action E A is performed in state S.
Assume a (stochastic) expert policy π F. S + AAA that is defined as a conditional probability n (als) of the expert's performing an action a e A given a state s E S. Also, assumee that we have a dataset (7(1). ...., ((N) of the expert trajectories where each T = {(21, a;)) o is sampled by
80 Po. of E ~ n (-]st), ~ ((-1s),, 81+1 P(-|st, 0.) for E (0, 1,.,,....T T}.
That is, T in the dataset is drawn from the following joint distribution of all observations and actions:
pp(or. ) po(so) IT (
((.4) (afss))((( )dso..41.
our study, we consider POMDP scenarios where an ideal imitator policy is able to match the expert policy's performance, even when the imitator's actions are solely determined by observation histories. Specifically, we focus on situations where the observation histories 21-w-1:t (for some fixed 1 < < T 1) encompass all information about the true states s; utilized by the expert policy.

---

In this setting, our goal is to learn an imitator policy x' 2 - AA that acts as closely to nT as possible on the given dataset DD To achieve this goal, we consider the following joint distribution of actions of both '/ and π
p(aber (
T (dd (st) P(s+11lss Jddo (1)
f poo(s)) t=
4 Behavior Cloning with Past Action Leakage Regularization
In this section, we propose a framework that effectively mitigates the past action leakage problem IL. We first define the problem by formalizing the absence of leaked past-action information via conditional independence (Section 4.1 Then, we compare several regularization- -bseed approaches that attempt achieve the absence of such information in the context of offline II. (Section 4.2). These approaches performs regularized BC where the regularizer is derived from metric for measuring the amount of conditional dependence among random variables. The choice of the metric differentiates these upproaches, and our comparison advocates the use of the HSCIC-based approach.
4.1 Past action leakage problem in imitation learning
When a policy takes an observation history as an input, the input history may include the information about past actions unexpectedly. The inclusion of such information can have detrimental effects in the context of offline imitation learning by confusing the imitator and making it fail to predict expert actions accurately. Intuitively, the past action leakage problem in imitation learning refers this failure of the imitator due the leakage of such harmful past action information.
To express this intuition formally, for each timestep f, let A and A! denote random variables of expert action and imitator action at timestep t. Note that for each < T. the joint distribution of the three random variables A A, and A
plaf-1,0 E at) = plah.r pdc do d. a...oa.a))))
where p(ab.r. af.r) is the distribution in Eq. 1
We formalize the absence of the leakage of harmful past-action information by conditional independence between the imitator's current actions and the expert's previous actions:
A A A, for all <TST.
This conditional independence says that the imitator's current action never depends some infor- mation that is only about the expert's past action but not about the expert's current action. This past-specific information corresponds to harmful information in our intuitive explanation from above.
Ideally we would like to achieve conditional independence in Eq. 2 which ensures the absence of leaked past-action information that was harmful to the imitator. However, practice, we can achieve it only approximately, that need quantitative measure for conditional independence or the lack of conditional independence between AE, and A given AL. Such measure is also needed to design an offline IL algorithm that does not suffer from such leaked harmful past-action information. In the following subsection, we consider two quantitative measures for the lack of conditional independence, namely, (1) conditional mutual information (CMI) and (2) Hilbert-Schmidt Conditional Independence Criterion (HSCIC).
4.2 Behavior cloning with past action leakage regularization
We aim to learn the representation i2 observation history 2t-w+1= that removes any unnecessary information the past action. To simplify the notation, let t. denote 1. Our method is based on the following observation.
5

---

Theorem 1. Let A be the action from the imitator policy x/ (aclp:) based on the representation Pt of observation history. Then, Pp 11. A A - Ag A A..
Proof. See Section B in the supplementary material.
To this end, we formulate our objective of regularized BC framework follows:
(;; (T;
Lreg(p;
where Lbe an BC objective such that (T yp; D) == (s af DDDei and Greg is past action leakage regularization objective. For notational simplicity, we abbreviate the expectation with respect to (a Stg at) D, ((ttau) to Eyp. In the following subsections, we discuss candidates for Lreg.
log (a (a)))
4.2.1 Information- regularization
A straightforward way to address the conditional independence we discussed in Section is minimizing the CMI, which can be decomposed by its definition into two conditional entropy terms:
T(af-1 Pe a. H (( H H( af).
It is important to note that H (af=1 a is determined by the data distribution D and thus constant. 1
As result, simply consider the minimization of - H (a. a.))
af a.) == (a. Pr, a.)
[logp(af_1 a.)].
(4)
Interestingly, Eq. 4 coincides with the negative conditional entropy maximization objective in FCA Since the direct computation of Eq. (4) requires to know the intractable distribution p(af_
Ppt, (ptttaa)) aaaa)) FCA trains an additional prediction model p(ā;-1 (P,, ax) to estimate the negative entropy with -H (a (ee, ap), == Exp, [lg (Ps, a.)] However, this approach faces the following challenges:
1. The estimated negative entropy H lower bounds Eq. 4 while upper bound would be desirable for minimizing the objective. Consequently, imposes nested (minimax) optimization; minimizing with respect to j after maximizing with respect to p.
2. It is required to train an additional neural network model which consumes additional computational cost.
3. Assuming that the family of variational distribution P to be Gaussian, FCA tries tighten the lower bound estimation - ff with respect to ý by minimizing reconstruction error of of_ The assumption restricts the flexibility of ý and thus the estimation can be inaccurate.
To avoid inaccurate estimation of the entropy, we can also decompose CMI into two MI terms by the chain rule of MI as an alternative:
1i.. I 1affii iif
(aff
Similar to Ha. af)), regularization is about simply minimizing [a az) while ignoring the constant MI term.
can be safely ignored in the regularization. As result, the
ii
Since the direct computation of Eq. (5) requires to know densities of 1, a., one needs to
train sample-based MI estimators 3 16 34]. Thanks to those estimators, the estimated MI can be minimized without confining any distribution (to be Gaussian). However, this approach still has issues similar to FCA such that (1) it lower bounds MI and consequently imposes (nested) minimax optimization and (2) it introduces an additional neural network particularly sensitive to hyperparameters.

---

4.2.2 HSCIC regularization
To bypass those shortcomings in information-theoretie regularization, we consider HSCIC kernel-based conditional independence metric for the past action leakage regularization.
30
Creg- Hseg-Hscic(pu: of a.) == af
(6)
Let H... Ho be RKHSs over A, + and A ke be their associated kernels, where O is the representa- tion space induced by the encoder yp Given n samples {((r(1), ( (f)) H= from D and y, let p. o _
o e be the n kernel matrices where pK., he (f) (.()). K,
(a (f), a. (i)); Fk, kalaf (1), (()). Then, HSCIC estimator for past action leakage regularization can be defined as follows:
1
HSCIC af (a) -race K a
) W(K,, p, KG JW K aF
2K. W W K. K. W W
K WK W O WK.5 W K&t)
(7)
where W =K,, + -AI) is ridge regression coefficient, @ is the element-wist matrix multiplication.
Since Eq. 7 can be estimated using the samples from the joint distribution p(Pc. of af"). we can directly plug-in HSCIC estimates into the regularization objective. By leveraging this estimator, HSCIC regularization offers several advantages compared to the conditional entropy regularization and MI regularization objectives discussed earlier:
1. Direct computation of the closed-form solution Eq. 7 allows HSCIC regularization to bypass any nested optimization.
2. HSCIC regularization does not employ any additional deep neural networks that require careful hyperparameters.
3. Since HSCIC is non con-aaamettric measure, it does not impose any parametric assumption on the data distribution and does not require any density estimation.
These advantages strongly imply that promoting conditional independence via the HSCIC estima- tor will be more desirable compared to other estimators, thereby improving the overall effectiveness of the regularization. Hence, we propose HSCIC regularization address the past action leakage problem, which call PALR.
5 Experiment
In this section, present the experimental results of approach. Initially, we investigate the corre- lation between the extent of past action information leakage and BC's performance. Subsequently, we compare our approach with several offline ILOH baseline methods across four continuous control tasks from the MuJoCo simulator [41]: hopper, walker2d, half cheetah, and ant, as well as one pixel-based autonomous driving task from the CARLA simulator
9 carla-lane
For tasks from the MuJoCo simulator, we transformed them into POMDP scenarios for ILOH by excluding specific state variables (such as velocity information). The remaining state variables, like positional information and joint angles,
Dataset
treated as observation variables at We organized these observations into fixed-size stacks to configure each individual timesteps
"The D4RL benchmark also featured another task, carla-town, but none of the algorithms surpassed the random policy's performance, indicating lack of meaningful imitation results. Further details can be found in Section D.2 Details about the composition of observation variables for each task available in Section C.1
7

---

problem setting, denoted as [envname] -W -I[sacksizel with stack sizes E (2,4]. For task from the CARLA simulator, we used pixel images as observations for ILOH. To extract features from these observations, we employed a pretrained ResNet [5]. keeping its parameters fixed during training. carla-lane task, we stacked the extracted features with fixed stack size of w 3 and used it as imput for the policy. All our experiments utilized expert demonstrations from the D4RL benchmark dataset [TO] to ensure the validity and reliability of our results.
Evaluation Metric For the performance evaluation of the learned policy, we measure the normalized score that ranges from to 100. To evaluate how much the past action information is leaked into policy we measure HSSCIC(a aff)laa))) using the estimator 7 in the held-out dataset. To estimate HSCIC estimator, we fix ridge regression coefficient À = 10 5 and all kernels are chosen as Gaussian kernels with the bandwidth x²
5.1 Relationship between past action information leakage and performance
To see if the problem of the past action information
100
leakage occurs, we conduct an empirical study using BC from observation histories with stack size w 2. The objective of this experiment is to confirm the cor- relation between the performance and the degree of
halft hhetlh
Normalized Score
past action information leakage, similar to FCA 44
20
To achieve multiple policies at different levels, we train BC policies for 500K steps with different seeds and different training dataset size N {100K, 50K, 30K, 10K, 5K, 3K, 1K} (the number of observation history-action pairs). Using held-out dataset composed of 2,000 samples, the degree of past action information leakage is measured by HSCIC(af; af_ laf) of each policy x/.
Experimental setup
-20
20
40 60
80
Normalized HSCIC(al; .laf)
Figure 1: HSCIC estimates and performance of BC with 7 different dataset sizes.
Negative correlation between
Results To compare HSCIC of fully trained BC in all tasks, we normalize HSCIC of each policy using the maximum and minimum values in each task. Grouping trained policies by the dataset size, we report the mean value of both the normalized score and the normalized HSCIC estimates in Figure [ shows that there are clear negative correlations between the normalized HSCIC and the normalized score in all environments. The result implies that BC from observation histories tends to train a policy in which there is negative correlation between the conditional dependence related to its action and the performance. This insight suggests potential opportunity for employing regularization methods that enforce conditional independence.
5.2 Performance Evaluation
Experimental setup We evaluate the performance of our method and offline baseline methods across 5 environments see effectiveness our method. We train policies using different algorithms, including our method, for 500K, 1M training steps MuJoCo and CARLA tasks respectively with different seeds.
Baseline methods PrimeNet [46] RAP 6 FCA 44, MINE B While all of these are commonly based on BC, we
We compare our method with 6 offline ILOH baseline methods: BC, KF 45)
specify their differences as follows:
*For the sake of simplicity, here we refer to the baselines 6 45 as KF (Keyframe-Focutt Focused visual imitation learning). RAP (Residual Action Prediction) respectively, following the titles of their respective papers.

---

Task
BC
KF
PrimeNet
RAP
FCA
MINE
PALR (Ours)
hopper
2
32.5 12.5:299
32.0 +
1.9
20.2 1.60
1.4
3..9 25.0
42.0 + 2.4
47.7 45.7
45.3
2.6
36,9 37.
2.8
walker2d
2
53.0 2.3
48.3
1.0
63.1 5.56
2.3
6.3 7.
79.2
2.14
$1.93.. 68.7 6.7
5.4
halfchectah
2
2.31 14
61
6.1
78.2 76.3
86
2.6 55.7
59.0 45
2.69 73.4 2.4
79.1
4.3
ant
2
56.3 + 3.5 54.9
+ 1.7
51.7
44.1 + 1.2
51.1
53.9 1.9
59.6 + 3.0
64.4
48.6
3.8
58.2 +
18.6 2.6
57.7 1.3
56.6
+ 1.8
64.6
2.5
carla-lane
52.5 + 6.2 66.6 +: 2.1 58.2 + 2.2 25.3 + 5.4
57.1 4: 3.1 60.1 + 4.1
72.9 + 2.6
Table I: Performance evaluation of baseline and regularization methods. The normalized scores averaged over the final 50 evaluations during training and we report mean and standard error over different seeds. The rightmost three algorithms are incorporated into our regularization framework. The method with the highest mean score and its competitive methods (within standard error) are highlighted in bold in each problem setting.
BC: : the standard BC algorithm from observation history, which optimizes an objective without any regularization term 0).
the weighted BC algorithm that assigns higher weights to keyframes, which contain actions that are highly predictable from their corresponding action histories.
445
PrimeNet [46]
supervised learning method designed to prevent undesirable shortcuts by leveraging additional key inputs.
RAP 66: it employs dual-stream of policy representation that learns from both observation history and individual observations, maximizing a lower bound that enforces conditional dependence between the representation and expert action, given the past action.
FCA it maximizes the conditional entropy that corresponds to Eq. with adversarial training, which is an instance of our regularized BC framework.
MINE 3
it minimizes the MI estimate corresponding to Eq. (5) using MINE estimator Kl
which is one of the representative sample- sample-based based neural estimators for MI. It is also an instance of our regularized BC framework.
Our method regularizes HSCIC with its estimator defined as Eq. 7 where Gaussian kernel with fixed bandwidth x² are used for all kernels (t,, a at and a ridge regression coefficient 5 Across all regularization methods, we searched for the optimal a according to the best mean normalized score. Further implementation details can be found in Section
Results Table summarizes the results of performance evaluation for each problem settings. Our method significantly outperforms other baselines settings out of and shows competitive perfor- mance in the rest settings. In particular, our method shows strong performance in carl a a-lane, highlighting its effectiveness in enhancing performance within high dibeeeiinnii offline ILOH scenar- ios. We observe that RAP shows the least competitive performance across all tasks. This is because they do not have any penalization of the dependence between the imitator action and expert action in their objective function. However, we also observe that FCA and MINE fail to show consistent improvement over BC most tasks except walker2d. This is because the lower bound estimators of their regularization objectives are not sufficiently accurate even at the cost of their inefficient alternating optimization. On the other hand, our method consistently outperforms BC, which clearly indicates the effect of promoting the conditional independence as discussed in Section 4.1
To better understand the degenerate performance of FCA and MINE, we evaluate HSCIC(a/; ,( of each regularization method during training using the held-out dataset in Figure 2a In contrast to our method, the result clearly demonstrates that FCA and MINE commonly

---

1,90
0.000
1la5 (55)
5.010
Normalized score
0.8
1
HSCIC(a at
0.7
HSCIC
3.377
0.02
0.4
0
100K
200K
300K
400K
500K
Iterations
16a
30-
10'
10-
10a
10
10*
BC
FCA
MINE
PALR (Ours)
(a) HSCIC estimate during training on hopper-W4.
(b) Varying regularization coefficient walker2d-W2.
Figure 2: Investigating past action leakage regularization methods on D4RL dataset.
fail to promote conditional independence throughout the learning process, elucidating their underper- formance. To further investigate, we measure the HSCIC and conditional MI estimates of the finalized imitator policies across 8 MuloCo settings (refer to Section D.5). In essence, method adeptly mitigates the leakage of past action information, surpassing the efficacy of alternative approaches.
Regularization coeflicient In this experiment, we aim ascertain how the selection of coefficient influences the performance of our method. We evaluate the asymptotic performance of our method with varying values of a E {0, 10 10t, 102, 10 10a, 10 10%) on the walker2d-W2 problem setting, which shows the largest performance gap between PALR and BC among our problem settings. As depicted Figure 26] the converged regularization loss Crex- HSCTC progressively decreases as a increases. Notably, the a that minimizes the regularization loss not necessarily equal to the optimal hyperparameter that maximizes performance. We observe that the selection of is important, it adjusts the trade-off between robustness to the leakage and alignment with expert data.
Conclusion and Future Work
Grounded in the classical notion of conditional independence, proposed a principled regularization framework for BC that mitigates past action information leakage problem. Within framework, have explored multiple choices of conditional independence metric and compared their estimators. Finally, we identified that our method with HSCIC estimator is the most favorable regularization of BC over other choices in terms of robustness to the leaked information of past action. In an extensive set of experiments on D4RL datasets, we empirically showed that our method significantly outperforms baseline methods. We also observed in our experiments that all the comparing methods including ours were sensitive to the choice a. Without assuming any interaction with the environment, is challenging find the optimal only with the given offline dataset. We believe that the discovery of optimal in offline manner would be an interesting research topic, which we leave future work.
Acknowledgments and Disclosure of Funding
This work was partly supported by IITP grant funded by MSIT (No.2020-0-00940) Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing: No. No.2022-0-00311
Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Ma- nipulation Everyday Objects; No.2019-0-00075. AI Graduate School Program (KAIST); No.2021- 0-02068, AI Innovation Hub), NRF of Korea NNRF2019RA22II0S1 2018R1 ASA 1059921), Field-oriented Technology Development Project for Customs Administration through NRF Korea funded by the MSIT and Korea Customs Service (NRF202IM3IIA A197993)) ETRI grant (22ZS1 100, Core Technology Research for Self- self-Improvin Integrated AI System), KAIST-NAVER Hypercreative AI Center.
RS-2023-0027968) NRF-
10

---

References
[1] Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.
[2] Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:18/2.03379, 2018.
[3] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, 2018.
[4] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars, arXiv preprint arXiv: 1604.07300 2016.
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement Icarning via sequence modeling. Advances in neural information processing systems, 34:15084-15097 2021.
[6] Chia-Chi Chuang, Donglin Yang, Chuan Wen, and Yang Gao. Resolving copycat problems in visual imitation learning via residual action prediction. In Computer Vision-ECCV 2022: 17th European Conference. Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XXXIX, pages 392-409. Springer, 2022.
[7] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 9329-9329999 9338, 2019.
[8] Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
[9] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[10] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. 2020.
[11] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pages 1180 1189. PMLR, 2015.
[12] Alessandro Giusti, Jérôme Guzzi, Dan C Ciresan, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, al. machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, (2):661-667, 2015.
[13] Abel Gonzalez- Gariii, Joost van de Weijer, and Yoshua Bengio. Image-to-imas translation for cross- domain disentanglement. 2018.
[14] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic Learning Theory: 16th International Conference, ALT 2005, Singapore, October 8-11, 2005. Proceedings 16, pages 63-77, Springer, 2005.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint wwXiv:1808.06670 2018.
Karan Grewal, Phil Bachman, Adam
[17] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances neural informa- tion processing systems, 29, 2016.

---

[18] HyeongJoo Hwang, Geon-Ilyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Variational interaction information maximization for cross-domain disentanglement. In Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12479-22492 Curran Associates, Inc., 2020. URL htttp::/pproceedingg.. wtwpoppeeaaggs.nuuus.....cecee Paper pdf
[19] Leslie Pack Kaclbling, Michael I.. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 1998.
[20] Geon-Hyeong Kim, Scokin Sco, Jongmin Lee, Wonscok Jcon, HycongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference Learning Representations, 2022.
[21] Diederik P. Kingma Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings. 2014.
[22] Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard S. Zemel. The variational fair autoencoder. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations. ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL httptp/////axi org/ wtggaab//5511..000 http//1111.008800
[23] Charles Micchelli and Massimiliano Pontil. On learning vector-valued functions. Neural computation, 17(1):177-204, 2005.
[24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforeement learning. arXiv preprint arXiv:1312.5602. 2013.
[25] Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invariant rep- resentations without adversarial training. Advances Neural Information Processing Systems, 31, 2018.
[26] Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp. and Yann Cun. Off-road obstacle avoidance through end-to-end learning. Y. Weiss, B. Schölkopf. and J. Platt, editors, Advances in Neural Information Processing Systems, volume 18. MIT Press, 2005. htRp/ttttp://prrcceding
wwt.gg.....ri..... cc/paper
f htt.:///enn...t.ooo.oooooooeoee.
.pdf
[27] Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3):263 2013.
doi: 10. 1 000117772783344914472255
[28] Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini urXiv.1807.03744 2018.
[29] Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, et al. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint arXiv:2110. 10819, 2021.
[30] Junhyung Park and Krikamol Muandct. measure- mcasure-thcoretic approach kerel conditional mean embeddings. Advances in neural information prromaationpooosssiinng systems, 33:21247-21259, 2020.
[31] Roman Pogodin, Namrata Deka, Yazhe Li, Danica Sutherland, Victor Veitch, and Arthur Gretton. Efficient conditionally invariant representation Icarning. In The Eleventh International Conference on Learning Representations, 2023. URL [tR[ittts///ppeeeevvee
net:/forumPddddduuess..
[32] Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. NIPS, 1988.
[33] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1988.
[34] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, 2019.
12

---

[35] Francesco Quinzan. Cecilia Casolo, Krikamol Muandet, Niki Kilbertus, and Yucen Luo. Learning counterfactually invariant predictors. arXiv preprint arXiv:2207.09776, 2022.
[36] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proccedings, 201 1.
[37] Proteck Chandan Roy and Vishnu Naresh Boddeti. Mitigating information leakage in image representa- tions: A maximum entropy approach. In Proceedings the IEEE/CVF Conference Computer Vision and Pattem Recognition, pages 1586-25594 2019.
[38]
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 3: 233 233-242, 1999.
[39] Jonathan Spencer, Sanjiban Choudhury, Anun Venkatraman, Brian Ziebart, and Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02877 2021.
[40] Gokul Swamy, Sanjiban Choudhury, J Bagnell, and Steven Z Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems, 11776-176776 2022.
[41] Emanuel Todorow, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In IEEE,
2012 IEEE/RSJ international conference on intelligent robots and systems, pages
2012.
[42] Dequan Wang, Coline Devin, Qi-Zhi Cai, Philipp Krähenbühl, and Trevor Darrell. Monocular plan view networks for autonomous driving. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2876-208 2883. IEEE, 2019.
[43] Sean Welleck. Kianté Brantley. Hal Daumé li,, and Kyunghyun Cho. Non-monotonic sequential text generation. International Conference on Machine Learning, pages 6716-6726 PMLR, 2019.
[44] Chuan Wen, Jicrui Lin, Trevor Darrell, Dinesh Jayaramam, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2564-2575. Curran Associates, Inc., 2020. URL URI hattps::/proceeeiingg awroceedings.neuris..comm mewripp.ccc/ppgee peurrip..cc/ppper/22200
pdf
[45] Chuan Wen, Jicrui Lin, Jianing Qian, Yang Gao, and Dinesh Jayaraman. Keyframe-focused visual imitation learning. In International Conference on Machine Learning, pages 11123-11 1113-11173
111--1133- 133. PMLR, 2021.
[46] Chuan Wen, Jianing Qian, Jierui Lin, Jiaye Teng, Dinesh Jayaraman, and Yang Gao. Fighting fire with fire: Avoiding dnn shortcuts through priming. In International Conference on Machine Learning, pages 23723-23755 PMLR, 2022.
[47] Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. Advances in neural information processing systems, 30, 2017.
[48] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Diseriminator-woighheed offline imitation learning from suboptimal demonstrations. In International Conference Machine Learning, pages 24725-24742. PMLR, 2022.
[49] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant repre- sentations for domain adaptation. In International conference machine learning, pages 7523-7532. PMLR, 2019.
13

---

Supplementary Material
Contents
Details of HSCIC and Its Empirical Estimation
15
Definitions
15
Empirical estimation
16
Proof of Theorem
17
Implementation Details
18
Observation specification
18
C.I.1 Hopper
18
Walker2d
18
HalfCheetah
19
Ant
20
2
Dataset preprocessing
20
Hyperparameter selection
20
Additional Experiments
22
D.I
Direct regularization imitator action
22
D.2
Experiment on carla-town
22
D.3
Comparison with VIB (Variational Information Bottleneck)
................................
23
D.4
Effectiveness on DT (Decision Transformer) policy
23
D.5
Quantitative analysis on past action information leakage
................................
24
D.6
Details the leakage-performance correlation
25
Broader Impacts
26
Limitations
26
Computation Resources
26
Licenses
26
14

---

A Details of HSCIC and Its Empirical Estimation
A.1 Definitions
Definition 1. (Reproducing kernel Hilbert space)
Let Hx be a Hilbert space of functions f X functions which the metric is defined by inner product &, H)x A symmetric function kx X x X I, which is called reproducing kernel of Hx, that satisfies
1. Vx E X
(x((,,) E Hx
2. Ve EXVE Hx
(S,kx(a, Mux f(x) (called reproducing property)
A space Hx is called a reproducing kernel Hilbert spaces (RKHS) corresponding reproducing kernel kx.
Definition 2. (Kernel Mean Embedding)
Given a distribution Px on X, assume Jx Vxx(ex (z, x x))PP(()) < co. We define the kernel mean embedding Px of Px as P (.) Lx(xx (dPx(x)))))))))))))()))))
Definition 3. (Maximum Mean Discrepancy)
Given distributions P. Q on and suppose kernel mean embeddings of P. Q exist on H, denote μP..Q. Then, the maximum mean discrepancy (MMD) between Px: Qy defined as follows:
MMD(P, Q; H.) = 1pp - -00--1l1l MM2l#
Definition 4. Frilbert-Schmid Norm and Operator)
Let F, 9 be Hilbert spaces. Define (fi)ier, (9)))e] be orthonormal basis for F, F,9 respectively. Define two linear operators L : - F,M F.
1. The Hilbert-Schmidt norm of the operator L is defined as:
L = Σ |1,
jEJ
ΣΣ
fL))
JEJ JET
2. The operator called Hilbert-Schmidt Operator when [| finite.
Definition 5. (Cross-covariance Operator)
Given RKHS F, with kernels feature maps o, v respectively. Define the uncentered covariance operator Čxy F and the cross-covariance operator Cxy
follows:
(Cxy, A)ns = (Fr,)((())) 69 wly), A)
Cxy = px & Jy
Definition 6. (Hilbert-Schmidt Independence Criterion) The Hilbert-Schmidt Independence Criterion (HSIC) defined as Hilbert-Schmidt norm of the cross-covariance operator CxY, i.e., HSIC == ||xx llas- Equivalently, we can define HSIC MMD between Pxy and Px Py with the product kernel, i.e.
HSIC F,9) MMD (Pxy, Px Py:
where <((x,,) y), (x' (')) k(x, (ly))(()///)))))))
Definition 7. (Conditional Mean Embedding)
Assuming a random variable X satisfies Jx / x
Define the conditional mean
embedding of X. given Z as (Px(z()) Ex (Xx 1Z7.
15

---

Note that MPx a is random variable. In the main text of our paper, we denote the realization of 1'xx22 with Z = as Pxxx z=x"
Definition 8. (Hilbert-Schmidi Conditional Independence Criterion)
Define the Hilbert-Schmidt Conditional Independence Criterion (HSCIC) between X and Y given Z to be
HSCIC(X, Y12) = ||ıxxvi2 - 1Px12 8 (Pr|z P
A.2 Empirical estimation
We start to explain from decoupling the conditional mean embedding into their deterministic function and conditional random variable.
Lemma 1. (Decomposition of conditional mean embedding, Theorem 4.1 of (301) There exists a deterministic function FxX1z Z - Hxx which satisfies Pxxxz Px1z Z.
By this lemma, terministic function and conditioned random variable Z, i.e., PXxI FPxx Z, MPrrz FpyIz 2,PPxiz = FXXYIz Z, HSCIC(X, YIZ) = FHSCIC o Z respectively. By us- ing vector-valued RKHS regression with a regularization parameter > 0 and the representer theorem [23] we can obtain closed-form estimates of Fxx]z: Fry 2 FPxx as follows (see 30 for more details):
decompose conditional
embeddings and HSCIC into some de-
Wz = :Kz + nAI)
FX (2) (z))
Ppr1z (2) (=)
FXXVIZ (z) (x) (kx o k-))))
where kx(:) = (kx(21, kx(xn, ()T
== ()), by (yn, ,)T, kz(-)
(k2(21,)),,. (2x: Kzly == k2 (2,, 2j) and @ is the element-wise multiplication opera- tor of matrices. By plugging-in conditional mean embedding estimators, we can obtain a closed-form
estimator of FHSCIC as follows:
FIscICx.Y12 (2) = =1I (3)- Frxiz @ (1)
-k](2) (Kxx O Ky (22((2))
Ky W wz(()))
+ (x Kx W Wyk(())))))
HSCIC(X, Y|Z)
Fascicx.vIz (2)
n
trace
K2 O Kr)
<KK OK O
6 WK ( W, k2)))
(8)
where Kxly == (x KK), = ky(0 vs).
16

---

B Proof of Theorem 1
E
O+
11
Stw
81-1
81
T7-1
Et
4tt
Figure 3: Graphical model of behavior cloning with observation histories.
Proof. By the chain rule of MI, following equalities hold:
I(p;; a ial-iiate Pr, a - 1(a a(af:
E a.) Iaa at) [a 1:0
Thus, [(pr; af_ of) 2 laa: of af*) and only if [(a 1iff I(af af")
By the chain rule of MI,
I(af (aa-11 af of) = ((af-ii0)) a ) + I( ia taat
T(af_ii a
E T(a 10 lat,ar) at)
Thus, I(p:a - | 2 I(a a a 20
Directly regularizing the imitator policy to enforce conditional independence may cause harmful results of training. This is because any policy that ensures the conditional independence does not necessarily yield optimal behavior. For instance, constant actions or actions sampled from uniform random distribution are independent of other random variables and thus satisfy Eq. (2). although they are not optimal in general. Based on Theorem [ we adjust the representation of policy to inject the conditional independence. The effectiveness of our approach is compared with direct action regularization through empirical study. The results are provided in Table of Section D.I
17

---

C Implementation Details
This section provides detailed specifications and implementation settings.
C.1
Observation specification
Based on the observation specification from Open AI Gym documentationf) we take a partial observation in each task by excluding certain variables (e.g. velocity information) in the original state variables. We mark the variables we used in our observation as O and not used are represented as x in the observation column of the corresponding table.
C.1.1 Hopper
Index
Description
Unit
Observation
0
z-coordinate of the top angle of the top angle of the thigh joint angle the leg joint angle of the foot joint velocity the x-coordinate the top velocity of the z-coordinate of the top angular velocity of the angle of the top angular velocity of the thigh hinge angular velocity of the leg hinge angular velocity of the foot hinge angular velocity (rad/s)
position (m)
1
angle (rad)
2
angle (rad)
3
angle (rad)
O
4
angle (rad)
O
5
velocity (m/s)
X
6
velocity (m/s)
7
angular velocity (rad/s)
X
8
angular velocity (rad/s)
9
angular velocity (rad/s)
X
10
Table 2: Composition of observation variables (hopper)
C.1.2
Walker2d
Index
Description
Unit
Observation
0
z-coordinate of the top angle of the top angle of the thigh joint angle of the leg joint angle of the foot joint angle of the left thigh joint angle of the left leg joint angle of the left toot joint velocity the x-coordinate the top velocity of the z-coordinate of the top angular velocity of the angle of the top angular velocity of the thigh hinge angular velocity of the leg hinge angular velocity of the foot hinge
position (m)
1
angle (rad)
2
angle (rad)
3
angle (rad)
4
angle (rad)
5
angle (rad)
6
angle (rad)
7
angle (rad)
8
velocity (m/s)
X
9
velocity (m/s)
10
angular velocity (rad/s)
X
11
angular velocity (rad/s)
12
angular velocity (rad/s)
13
angular velocity (rad/s)
14
angular velocity of the thigh hinge angular velocity of the leg hinge angular velocity the foot hinge
angular velocity (rad/s)
15
angular velocity (rad/s)
16
angular velocity (rad/s)
Table 3: Composition of observation variables (walker2d)
httpps:/wwww.ee http://wwwww....r.rr.../.aaaaaaa http///avviroommnttt/mooooooo
18

---

z-coordinate of the root C.1.3 HalfCheetah
Open Al Gym documentation does match with the XML file of halfcheetal denote both descriptions the following tables.
Index
Description
Unit
Observation
z 7-coordinate front tip angle of the front tip angle of the second rotor angle of the second rotor velocity the tip along the x-axis velocity of the tip along the y-axis angular velocity of front tip angular velocity of the second rotor
position (m)
1
angle (rad)
2
angle (rad)
3
angle (rad)
4
velocity (m/s)
X
5
velocity (m/s)
X
6
angular velocity (rad/s)
X
7
angular velocity (rad/s)
X
8
x-coordinate of front tip y-coordinate of the front tip angle
position (m)
9
position (m)
10
front tip angle of the second rotor angle of the second rotor velocity of the tip along the x-axis velocity the tip along the y-axis angular velocity of the front tip angular velocity the second rotor
angle (rad)
11
angle (rad)
12
angle (rad)
13
velocity (m/s)
14
velocity (m/s)
X
X
15
angular velocity (rad/s)
16
angular velocity (rad/s)
X
Table 4: Composition of observation variables chalccheetah on Gym Documentation.
Index
Description
Unit
Observation
0
position (m)
I
y-coordinate of the root angle of the back thigh angle of back shin angle of the back foot angle of the front thigh angle of the front shin angle
position (m)
2
angle (rad)
angle (rad)
4
angle (rad)
X
5
angle (rad)
x
6
angle (rad)
x
front foot velocity of the root along the x-axis velocity of root along z-axis velocity of the root along the y-axis angular velocity the back thigh angular velocity of the back shin angular velocity of the back foot angular velocity of the front thigh angular velocity of the front shin angular velocity of the front foot angular velocity (rad/s)
angle (rad)
x
8
velocity (m/s)
9
velocity (m/s)
10
velocity (m/s)
angular velocity (rad/s)
12
angular velocity (rad/s)
13
angular velocity (rad/s)
X
14
angular velocity (rad/s)
15
angular velocity (rad/s)
X
16
X
Table Composition of observation variables (halfcheetah) on the XML. file.
luttps://github htgpe///owggggggooooooooooaaoaar
19

---

C.1.4 Ant
Num
Description
Unit
Observation
0
z-coordinate of the torso (centre)
position (m)
x-orientation the torso (centre)
angle (rad)
2
y-orientation the torso (centre)
angle (rad)
3
z-orientation of the torso (centre)
angle (rad)
4
w-orientation of the torso (centre) angle between torso and first link front left angle between two links on front left angle between torso first link on front right angle between the two links on the front right angle between torso and first link on back left angle between the two links the back left angle hetween and first link on back right angle between the two links on the back right
angle (rad)
5
angle (rad)
angle (rad)
angle (rad)
8
angle (rad)
9
angle (rad)
10
angle (rad)
11
angle (rad)
angle (rad)
13
x-coordinate velocity of the torso y-coordinate velocity of the torso 7-coordinate velocity of the torso x-coordinate angular velocity of the torso y-coordinate angular velocity of the torso z-coordinate angular velocity of the torso angular velocity of angle between torso and front left link angular velocity of the angle between front left links angular velocity of angle between
velocity (m/s)
14
velncity (m/s)
15
velocity (m/s)
16
angular velocity (rad/s)
17
ungular velocity (rad/s)
18
angular velocity (rad/s)
19
angular velocity (rad/s)
20
angular velocity (rad/s)
21
and front right link
angular velocity (rad/s)
angular velocity of the angle between front right links
angular velocity (rad/s)
23
angular velocity of angle between torso and back left
angular velocity (rad/s)
X
24
angular velocity of the angle between back left links angular velocity of angle between torso and back right link angular velocity of the angle between back right links contact forces applied to center of of each the links
angular velocity (rad/s)
X
25
angular velocity (rad/s)
26
angular velocity (rad/s)
27-110
force torque
Table 6: Composition of observation variables (ant)
C.2 Dataset preprocessing
We utilized the expert dataset provided D4RL benchmark that has a suffix -expertv2. For all experiment, we configured partial observations from the original data based on Section CT These observations are standardized with statistics from the entire dataset. Each dataset in D4RL benchmark has total of 1M transitions. Given the size of the training data D, take the first D samples. Among the last 100K samples in the entire dataset, data which are not included in the training data. Given a stacksize E {2, 4). we configure each transition as (observation history, action, past action) tuple. For the main performance evaluation experiment (Table T. we use 30K transitions for training.
MuJoCo Tasks
also sampled heldout
CARLA Tasks We extract features from observation images using the pretrained network ResNet34 [15] and consider these features as policy inputs during training. For the performance eval- uation (Table T. 10,, we utilize 1K and 100K transitions for carla-lane, carla-tow respectively.
C.3 Hyperparameter selection
This section outlines the settings of the hyperparameters for each method. To ensure a fair comparison, we aim to maintain consistent network architecture and policy-related hyperparameters, whenever feasible. For all baselines and our method, employ a policy architecture consisting of fully- connected layers with [128, 64, 128] hidden units. The layer with 64 units is treated policy representation, denoted sp.
20

---

We used softmax function as weighting function of the weighted BC in MuJoCo tasks and a step function in CARLA tasks, aligning with the the original paper's experiments. We adopt the action prediction network for the additional network. Additionally, we tuned hyperparameters' for the softmax temperature in MuJoCo tasks, analogous to a. CARLA tasks, we used the same hyperparameters (threshold=0.1, weight=5) consistent with the original paper [45].
KF 45
PrimeNet [46] We treat observation histories as raw inputs and individual last observations as key inputs, consistent with the approach used the original experiments. The architecture of the prime network is implemented the same with the policy.
RAP incorporates two streams of policy representation: (1) one stream derived from observation history (referred as the memory extraction stream) and (2) the other originating from a single observation (referred as the policy stream). For the memory extraction stream, use an architecture of [128, 64, 300] hidden units that outputs predictions of residual of expert actions. The layer with 64 units is employed as representation. For the policy stream, we similarly utilize an architecture of [128, 64(+64), 128] which outputs policy action. The middle units indicate that the policy utilizes both the single observation representation and the memory-extractee representation (with stop-gradient layer) as inputs.
RAP 66
A summary of hyperparameters is provided in Table 7 The first 3 rows of the table display the hyperparameters universally applied across all methods. The next 4 rows detail hyperparameters for methods that incorporate additional neural networks. The last 2 rows of the table present hyperparameters specific to our method, encompassing hyperparameters for HSCIC estimates.
Hyperparameter
KF
PrimeNet
RAP
FCA
MINE
PALR (Ours)
Policy distribution
Tanh Normal
Batch size
1024
BC learming rate
3e-4
Additional network
hidden units
[128,128]
[128,64,128]
[128,64,300]
[300]
[100,100]
num of inner updates
learning rate
3e-4
3e-4
3e-4
le-4
le-4
IB coefficient
0.01
Kernel bandwidth o² Ridge coefficient
le-5
Table The summary of hyperparameters.
Optimal a As discussed in Section 5.2 we train 5 policies incorporating different regularization coefficients a within the set {0.01, 0.1, 1, 1, 10, 100} for MuJoCo tasks and {0.001, 0.1, 10, 1000} for CARLA task. We then select the best a for each problem setting. Table & the chosen values of for each method and each problem setting, as utilized in the experimental results of the paper.
Problem Setting
KF
FCA
MINE
Ours
hopper-W2
0.01
0.1
10
hopper-W4
0.01
0.1
0.01
10
walker2d-W2
0.01
0.1
I
100
walker2d-W4
0.1
0.1
1
100
halfcheetah-W2
0.01
0.1
10
halfcheetah-
0.1
0.1
0.1
10
ant-W2
1
0.1
0.01
10
ant-W4
0.01
1
0.01
10
carla- varla-lad -llaaae-e-W- W3
0.001
10
1000
Table 8: The summary of the best regularization coefficient a.
21

---

D Additional Experiments
D.1 Direct regularization on imitator action
To investigate the effectiveness of representation regularization, we compare our method to the one that directly regularizes action in MuJoCo settings. The objective function of the direct method is defined as follows:
(CT D, (T; D) ) Baffrt (Paaa) 1. yaa (ISCCCC(sCCCC
We optimize the imitator policy with the objective by using the reparameterization trick [2 The method is also trained with different regularization coefficients a E {0.01, 0.1, 1,10, 100) and we select the best a of each method. Table summarizzs the results. Our representation regularization method (PALR) is more successful in 6 out of the 8 problem settings when compared direct action regularization. In ant-W2, ant-W4 problem settings, neither method demonstrates significant effectiveness when compared to BC method.
Task
BC
PALR (ours)
PALR-ACT
hopper
2
32.47 + 2.85
42.01 + 2.44
31,60 + 2.81
4
47.65 + 3.43
58.39 + 2.76
46.14 + 0.90
walker2d
2
53.04 + 2.69
79.83 - 2.29
58.50 + 4.25
4
63.15 + 6.28
83.42 + 5,43
77.41 2.37
halfcheetah
2
74.08 + 2.33
86.44 + 1.09
76.32 1.30
4
68.35 + 2.60
79.05 4.28
75.93 2.59
2
56.25 + 3.45
59.57 + 3.03
58.32 + 1.23
4
64.39 + 1.77
64.64 + 2.53
65.00 + 2.46
Table Performance comparison of BC, our method (PALR) and direct action regularization version of PALR (PALR-ACT). The normalized scores averaged over the final 50 evaluations during training and we report mean and standard error over 5 different seeds. The method with the highest mean score and its competitive methods (within standard error) are highlighted in bold in each setting.
D.2 Experiment on carla-tow
D4RL benchmark provides two CARLA tasks: carla-lane, carla-tom. As shown Table 10 all algorithms fail to show meaningful imitation results, as none of them exceed random performance levels. This outcome likely due the dataset containing numerous instances of the same obser- vations paired with different actions (59,984 observation-action pairs out of 100K pairs in total). Consequently, have omitted the experiment results for carla-tow from Table [
Task
BC
KF
PrimeNet
RAP
FCA
MINE
PALR (Ours)
carla-lane
52.5
6.2
1 2.1 58.2 2.2 25.3 5.4
57.1
1.1
72.9
carla-town
3
-3.1
1.1
6.9 + 0.3 8.8 = 0.5 6..1 1 1.0
-7.0 + 2.1
1.0 + 0.8
-15 1 0.8
Table 10: Performance evaluation of baseline and regularization methods CARLA datasets provided by D4RL benchmark.
22

---

D.3 Comparison with VIB (Variational Information Bottleneck)
To highlight the leakage of past action information is not trivially prevented by general representation regularization methods, we compare our approach with VIB M.. a representative method. We train BC with VIB with two different values, specifically, 3 {10-3, 3, 10-6), 5 seeds for each problem setting. whereas method demonstrates significant improvements in 6 out of 8 problem settings.
Task
BC
PALR (Ours)
VIB (3 10
VIB (3 10
hopper
2
32.47 + 2.85
42.01 + 2.44
30.57 + 1.61
27.89 + 1.55
4
47.65 + 3.43
58.39 = 2.76
51.20 + 2.18
46.59 + 2.26
walker2d
2
53.04 + 2.69
79.83 + 2.29
44.28 2.32
35.98 4.69
63.15 + 6.28
83.42 5.43
67.78 2.67
65.99 1.97
halfcheetah
2
74.08 2.33
86.44 1.09
73.76 2.401
75.43 1.77
4
68.35 + 2.60
79.05 + 4.28
67.89 1.52
73.67 2.86
ant
2
56.25 + 3.45
59.57 = 3.03
59.54 + 1.65
55.29 + 4.27
4
64.39 + 1.77
64.64 2.53
61.63 + 2.77
61.25 + 2.88
Table 11: Performance comparison of BC, our method (PALR) and BC with VIB (Variational Information Bottleneck).. The normalized scores averaged over the final 50 evaluations during training and we report mean and standard error over 5 different seeds. The method with the highest mean score and its competitive methods (within standard error) are highlighted in bold in each setting.
D.4 Effectiveness on DT (Decision Transformer) policy
To assess the efficacy of method in regularizing policies with complex network architectures, we conducted additional experiments employing the Decision Transformer (DT) [5], one of the prominent offline RL methods. Given that reward information is absent in the offline IL dataset, we inserted a reward input into DT''s structure to retain its original configuration. Our regularization approach, termed DT-PALR, was applied to the last hidden state of DT. We evaluated both the standard DT and DT-PALR across 3 POMDP versions of MuJoCo tasks, consistent with the scenarios outlined in T
The results are presented in Table 12 evident that DT's performance lags behind that of BC, as demonstrated in Table I This divergence might be attributed to DT's utilization of a larger network size, rendering more susceptible to capturing spurious causal relationships in tasks where access to complete states and rewards is restricted. Encouragingly, our results indicate a substantial enhancement in the performance of DT for the hopper and walker2d tasks when subjected the DT-PALR method. This observation strongly suggests the adaptability of our approach to intricate architectures.
With the exception of the half cheetah task, it becomes
Method
hopper
walker2d
halfcheetah
DT 20.68 + 5.25
23.93 + 4.13
94.10 10 + 3.54
DT-PALR
26.09 + 6.59
32.58 + 5.63
96.05 + 3.60
Table 12: Effectiveness of PALR on Decision Transformer (DT) architecture. We train DT over MuJoCo tasks based on our observation configurations and apply PALR to the last hidden state.
23

---

D.5 Quantitative analysis on past action information leakage
We comprehensively evaluate HSCIC scores across 8 MuJoCo problem settings in Table 13 The results demonstrate that our method consistently reduces the conditional dependence. Notably, our method achieves lowest HSCIC scores in entire problem settings. Moreover, to show more robust assessment of conditional dependence,
also estimate conditional mutual information (CMI). (see Table 14 As described in Section 4.2.1 CMI can be decomposed into two MI terms, I [af_ 13 Pr, or and ia we estimate them using MINE (3] respectively. The table shows that our approach consistently presents lower CMI estimates compared to BC across 6 out of 8 problem settings.
Task
w
BC
RAP
FCA
MINE
PALR (Ours)
hopper
1.783 x x10
1.779 (4)
1.805 x10 (()
1.802 x10 xt)=²(t)
1.742 x10- *())
4
1.634 x 10-
1.617 x10-2(4)
1.629 x10 (1))
1.650 x10-3(1)
1.587 x10-3())
walker2d
2
5,442 x x
5.523 x1)=?(t)
5,415 x10
5,482 x10=2(t)
5,362 (())
5.279 x10 2
5.350 x1)=²(t)
5.248 x10 2(4)
5.298 x10 ((t)
5.182 x10 x10-=(() 2(1)
halfcheetah
4.010 x10-2
4.044 x10-²(t)
4.112 x10-x(1))
4.082 x10=?(t)
3,984 x10-2(U)
4
4.019 x10
4.052 x1)=
4.092 x10
4.069 x10-²(1
3,968 x x10
ant
2
6.014
x10-2
6.164 (t)
6.182 x10
6.178
x10 ((1)
6.008 x10-
4
5.938
x10
6.187 x1)->(t)
6.093 x10-2
6.103 x10 (t)
5.932 x10-3())
Table 13: HSCIC(a!, "e__lla," evaluation results of regularization methods and baselines. HSCIC
scores measured on held-out datasets and averaged over the final 50 evaluations during training and statistics are calculated with 5 seeds. (J), (f) marks whether the averaged HSCIC score of each method is lower than HSCIC score of BC or not.
Task
BC
RAP
FCA
MINE
PALR (Ours)
hopper
0.1825
0.1826 (f)
0.1949 (t)
0.1891 (t)
0.1821 (1)
0.1489
0.1381 (L)
0.1481 (4)
0.1558 (t)
0.1422 (L)
walker2d
2
0.2685
0.2655 (4)
0.2798 (t)
0.2745 (1)
0.2705 (t)
4
0.2491
0.2353 (4)
0.2522 (t)
0.2461 (1)
0.2411 (4)
halfcheetah
0.1562
0.1496 (4)
0.1612 (f)
0.1519 (1)
0.1509 (4)
0.1396
0.1530 (t)
0.1551 (f)
0.1484 (t)
0.1502 (f)
ant
2
0.1579
0.1623 (t)
0.1743 (t)
0.1567 (J)
0.1374 (4)
0.1585
0.1653 (t)
0.1642 (t)
0.1708 (1)
0.1450 (4)
Table 14: I(a;; af laff)
evaluation results of regularization methods and baselines. Since the conditional MI can be decomposed into two MI terms as described in Section 4.1.1, estimate two terms with MINE estimator respectively. We use the policy of the last iteration for each method to generate policy actions. (L), (1) indicates whether the averaged CMI score of each method is lower than the score of BC or not.
24

---

D.6 Details on the trakage-performance correlation
We first clarify the relationship between the number of training data and HSCIC scores in Section 5.1 To clarify, we plot the correlation into Figure The correlation indicates that when the training data is insufficient, the problem of past action information leakage becomes more severe. This phenomenon can be attributed to the higher risk of overfitting in cases with smaller training instances, which can lead to the capture of false causal relationships within the training data. These findings align with similar results reported in 8
(see Figure 4 in (8).
Normalized HaciC(al; a a ilaf))
hopper
walker2d
halffcheeah
ant
20
100K
Number of Training Dataset
Figure Relationship between the number of training data and the normalized HSCIC score.
Furthermore, to provide more insights for understanding Figure I we plot all points that indicate the normalized HSCIC and the normalized score each BC policy Figure [ addition, we calculate the sample Pearson's 's correlation coefficients using every point of each task (the first columns) and all tasks (the last column) on Table 15
100
hopper walker2d halfcheetah ant
Normalized Score
60
20
-20
-40
20
40
60
80
100
Normalized
Figure 5: Scatterplot that shows correlation between HSCIC estimates and performance. Each small point indicates HSCIC and the performance of each policy, and the diamond point (o) indicates the mean point of each group (which consists of policies trained with the same dataset size).
25

---

hopper
walker2d
halfcheetah
ant
Total
-0.8358
-0.8133
-0.8564
-0.9273
-0.8130
Table 15: Sample Pearson's correlation coefficients between the normalized HSCIC and the normal- ized score.
E
Broader Impacts
This work primarily addresses the reduction of the pervasive past action leakage effect in imitation learning with observation histories. Notably, our findings suggest potential vulnerabilities that may be exploited by adversarial actors. For instance, malicious attackers could introduce the conditional dependencies into the system, leading to potentially catastrophic outcomes, such as repeating the same actions. This will produce potentially irreversible consequences on the system, especially for security-critic tasks such autonomous driving and medical devices, or industrial automation. Therefore, understanding and mitigating this risk is crucial for the safe and efficient operation of such systems.
F
Limitations
While our approach is easily applicable to real-world tasks, the range of experiments in this paper is limited robotics simulation. A broader scope of experiments, including more realistic observation settings like image-based noise-inclusive observations, could provide stronger argument for the effectiveness and real-world applicability of our method. Additionally, our method assumes that control-relevan information sufficiently captured in the observation histories. However, this assumption might not hold in some real-world scenarios where key control information missing and retrievable from the observation histories. In such cases, the learning model requires additional information, inductive bias or expert knowledge function effectively.
G Computation Resources
For our experiments, we used a cluster system with 20 nodes that have the following system spees:
CPU: Intel i7-9700K CPU (3.60GHz)
GPU: TITAN Xp (VRAM 12 GB)
H
Licenses
Our code has been developed based on publicly available code repositories released under MIT licenses. The code used to generate our experimental results also follows to the MIT license. For more detailed information, see the README. md and LICENSE files included in our code files.
26


Reviews:
[{"review_id": "PLNj7Cbyvz", "reviewer": "Reviewer_4vtt", "summary": "This paper proposes Past Action Leakage Regularization (PALR) to resolve the copycat problem in behavior cloning (BC) methods:\n\n1. mathematically defines the past-information-leakage problem.\n\n2. introduces PALR to formalize the methods.\n\n3. uses Hilbert-Schmidt Conditional Independence Criterion(HSCIC) to measure the conditional independence, and analyses the advantages of HSCIC over CMI as the metric.\n\n4. provides experiments verifying the correlation between HSCIC and performance, and the superiority of the HSCIC-based method over baseline methods.\n", "questions": "1. Do you have any explanation for the phenomenon that more historical information damages the performances in HalfCheetah? \n\n2. Why only caring about $A_{t-1}^E$? I think $A_{t-2}^E$ can also cause leakage. Or can you prove that the former independence indicates the latter one?", "limitations": "As mentioned in Weaknesses, the current theories are not rigorous. They can be revised to make the whole logic consistent.\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow in most parts.\n\n2. The authors provide sufficient experiments to verify their claims.\n\n3. The idea of replacing CMI with HSCIC is natural in maths and easy to apply. It does bring a big improvement over the previous methods. \n\n4. The paper comprehensively explains how HSCIC becomes a better metric than CMI. \n\n", "weaknesses": "1. Some small typos: $H_{\\mathcal{X}}$ in line 133 should be $\\mathcal{H}_{\\mathcal{X}}$; the full name of MOMDP should use \\emph{observable} other than \\emph{observed} in line 153. I didn't go through every word, so there may be other mistakes. \n\n2. In line 203, $\\phi_t=\\phi(z_{t_\\omega:t})$, but in line 210, 212, $\\phi\\sim\\phi(z_{t_\\omega:t})$. This caused some confusion in reading. Did I miss anything?\n\n3. In line 193, ''Ideally we would like to achieve conditional independence in Eq. (2)\" is not obvious. And actually, it's not reasonable to require eq (2) to be true, for even a perfect imitator can't do this. The structure of MOMDP described in section 3.2 determines that it's not mathematically possible to predict $A_t^I$ given $A_t^E$. A better objective may be minimizing the CMI, as is practically done in the experiment. Furthermore, I know that this has been an unsolved problem from the first time copycat problem being proposed, but you may modify it by coming up with a better lower bound of CMI or HSCIC other than 0 to fix this mathematical flaw.\n\n4. Theorem 1 does not help much in proving the claim that lower loss indicates lower dependence, which only proves the equality between two independences. I believe that this is true and thus PALR is effective. Can you come up with a new theorem to fix this?\n\n5. Some important references are missing:\n\n[1] Ortega P A, Kunesch M, Delétang G, et al. Shaking the foundations: delusions in sequence models for interaction and control[J]. arXiv preprint arXiv:2110.10819, 2021.\n\n[2] Wen C, Lin J, Qian J, et al. Keyframe-Focused Visual Imitation Learning[C]//International Conference on Machine Learning. PMLR, 2021: 11123-11133.\n\n[3] Wen C, Qian J, Lin J, et al. Fighting fire with fire: Avoiding dnn shortcuts through priming[C]//International Conference on Machine Learning. PMLR, 2022: 23723-23750.\n\n[4] Spencer J, Choudhury S, Venkatraman A, et al. Feedback in imitation learning: The three regimes of covariate shift[J]. arXiv preprint arXiv:2102.02872, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive and insightful feedback.\n\n**1. Assumption on ideal imitator**\n\nThank you for bringing up this important point. To clarify, in our problem setting of imitation learning from observation histories, we make the assumption that the control-relevant (state) information can be fully recovered from observation histories as we described in Abstract and Section 3.2. This assumption means that we consider only those POMDPs where the imitator policy can perform equally well as the expert policy, even when the imitator's action is determined solely by the observation history. Thus, under this assumption, the (marginal) action of an ideal imitator, denoted as $A^I_t$, would be exactly equal to $A^E_t$, satisfying Eq. (2). In such an ideal situation, the true value of CMI or HSCIC of our interest would indeed be 0.\nWe acknowledge the importance of clarifying this assumption in detail to prevent any misunderstanding regarding our problem setting. In our next revision, we will provide a more explicit explanation of the assumption.\n\n**2. Relationship between loss and conditional dependence**\n\nWhile Theorem 1 serves the purpose of justifying the regularization of representations rather than the direct action output within our proposed PALR method, we understand the need for a more direct link between loss reduction and dependence minimization.\nRegarding the proposal for a new theorem, we recognize the complexity of establishing a strict relationship between lower loss and reduced dependence. The challenge arises from the fact that methods like FCA, MINE, and ours rely on individual estimations of dependence measures, and ensuring that lower loss strictly implies lower dependence necessitates accurate estimations across the board. However, existing literature demonstrates that minimizing estimated dependence measures, as observed in [A,B,C,D], has proven effective in practical scenarios.\nConsequently, while we acknowledge the theoretical difficulty in universally proving the connection, our approach aligns with established practices that demonstrate the practical utility of minimizing estimated measures of dependence. We have taken this approach to design our method and validated its effectiveness through empirical evaluations. We hope this clarification addresses the concern and illustrates the rationale behind our choice.\n\n**3. Performance degradation in `halfcheetah`**\n\nIn our observation configuration, we’ve discovered that BC in `halfcheetah` task demonstrates competitive performance when provided with only a single observation. This suggests that the single observation contains sufficient information for effective control, potentially rendering additional historical context redundant in the scenario. In this situation, the remaining part of historical data may not significantly contribute to information for decision making.\n\n**4. Reason for caring only $A^E_{t-1}$**\n\nTo maintain simplicity and unify the existing work into our framework, we focus on the one-step past action in this work. While we recognize the potential influence of $A_{t-2}^E$ or action histories, our primary objective is to show the efficacy of one-step past action regularization within IL-OH problems. The exploration of multi-step past action leakage naturally extends from our current work and we consider it a promising avenue for future research.\n\nWe really appreciate that let us inform typos and important references and we update them in the revised version of our manuscript. We acknowledge the confusion caused by our use of the notation $\\varphi_t$ for both random variables and values of representation. To enhance clarity in our presentation, we will introduce a distinct notation.\n\n**References**\n\n[A] Belghazi et al., “Mutual information neural estimation.”, ICML 2018.\n\n[B] Pogodin et al., “Efficient conditionally invariant representation learning.”, ICLR 2023.\n\n[C] Poole et al., “On variational bounds of mutual information.”, ICML 2019.\n\n[D] Quinzan et al., “Learning counterfactually invariant predictors.”, arxiv 2022.\n"}, {"review_id": "sPbVmwYSdk", "reviewer": "Reviewer_jVBb", "summary": "This paper addresses the information leakage problem of imitation learning with observation histories. To this end, the paper measures the leakage of past action information based on conditional independence and proposes Past Action Leakage Regularization (PALR) for behavioral cloning (BC). The experiments show that the proposed method outperforms four baselines on four MuJoCo continuous control environments. Ablation studies suggest PALR can improve BC with a proper coefficient. This work defines and studies an essential problem of imitation learning from partially observable environments. However, the experiments can be improved to make the work more concrete.", "questions": "See above", "limitations": "See above", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clarity**\n- The overall writing is clear. The paper gives clear descriptions in both theoretical and intuitive ways. The notations, formulations, and theorems are well-explained. \n\n**Ablation study**\n- Ablation studies are comprehensive. The provided ablation studies help understand the effectiveness of the regularization coefficient (Sec.5.2) and the target to apply regularization (Sec.D).\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed framework.\n- Given the clear description in the main paper and the details provided in the supplementary materials, I believe reproducing the results is possible.", "weaknesses": "**Method**\n- This work proposes a kernel-based method to regularize BC for imitation learning from partially observable environments. While I am aware of the advantages of kernel-based methods (stable, does not need additional networks or hyperparameters tuning), recent works use neural networks, such as variational model [1] or causal transformer [2], to capture essential information from observation histories and have shown promising results. \nThe effectiveness of the proposed PALR would be more convincing if the authors could demonstrate comparisons to the above methods. \n\n**Experiments**\n- Some parts of the experimental results are not easily interpretable. Figure 1 shows the negative correlation between the number of training data and the HSCIC score. Can the author explain why the information leakage problem is more severe when the training data is insufficient?\n\n- I am not entirely convinced by the explanation of why FCA [3] and MINE [4] get inferior results (Line 340-345 & Figure 2a).\nThe author only evaluates the HSCIC score of each method on hopper-W4. However, the proposed PALR is directly regularized by the HSCIC metric, so it is straightforward that the proposed PALR gets the lowest HSCIC score.\nIt would be better to evaluate the HSCIC score on all environment setups in Table 1 to 1) support the negative correlation between HSCIC estimations and the performance of algorithms and 2) analyze the performance of FCA and MINE, which outperform BC only on walker2d and halfcheetah environments.\n- The paper only evaluates the proposed method on four continuous control environments, which is insufficient. RL tasks such as navigation, robot arm manipulation, or Atari games can also be considered.\n\n**Experiment details**\n- The details of the normalized score (line 280) are missing. What is the maximum score and the minimum score?\n\n**Typo**\nLine 323: ..., which is a one of the ...\n\n[1] Rafailov, R., Yu, T., Rajeswaran, A., & Finn, C. (2021). Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 34, 3016-3028.\n[2] Bonatti, R., Vemprala, S., Ma, S., Frujeri, F., Chen, S., & Kapoor, A. (2022). Pact: Perception-action causal transformer for autoregressive robotics pre-training. arXiv preprint arXiv:2209.11133.\n[3] Wen, C., Lin, J., Darrell, T., Jayaraman, D., & Gao, Y. (2020). Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 33, 2564-2575.\n[4] Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., & Hjelm, D. (2018, July). Mutual information neural estimation. In International conference on machine learning (pp. 531-540). PMLR. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your constructive and enlightening comments.\n\n**1. Comparison to other methods**\n\nWe appreciate your comments on alternative methods and their potential effectiveness. However, we would like to clarify some distinctions between the alternative methods in [1] and [2] and our proposed PALR.\nFirstly, the variational model [1] is designed for online IL scenarios, where the agent can interact with the environment during the learning process. In contrast, we focus on offline IL, where the policy is trained solely on a pre-collected dataset without any interaction with the environment. Consequently, the direct application of the variational model to our offline setting is not straightforward.\nSimilarly, the causal transformer [2] involves a domain-specific fine-tuning step in each robotics downstream task, making it challenging to perform a fair comparison with our PALR, which is designed to be domain-independent. We specifically designed PALR to be applicable to a broader range of offline IL from observation history, with a focus on addressing past action leakage.\nWhile we recognize the potential effectiveness of the above two methods, we highlight that PALR is a generic approach for offline IL that does not rely on any domain-specific information. Our regularization method aims to improve imitation performance in scenarios where past action leakage remains a challenge, making it applicable across various domains without requiring domain-specific considerations.\n\n**2. Correlation between dataset size and HSCIC**\n\nThank you for pointing out the negative correlation between the number of dataset and HSCIC score in Figure 1 in our manuscript. The correlation indicates that when the training data is insufficient, the problem of past action information leakage becomes more severe. To clarify, we plot the correlation into Figure A in the PDF file.\nThis phenomenon can be attributed to the higher risk of overfitting in cases with smaller training instances, which can lead to the capture of false causal relationships within the training data. These findings align with similar results reported in [A] (see Figure 4 of the paper).\n\n**3. Complete evaluation on conditional independence**\n\nFirst, we clarify that $\\mathrm{HSCIC}(\\varphi_t, a^E_{t-1} | a^E_{t})$ and $\\mathrm{HSCIC}(a^I_t, a^E_{t-1} | a^E_{t})$ (the score recorded in Figure 2a) are different. PALR regularizes HSCIC with respect to representation, not HSCIC with respect to imitator action directly. As shown in Theorem 1, PALR regularization enforces the conditional independence between actions.\nTo provide a more trustful interpretation of our experiment, we comprehensively expand our quantitative analysis: (1) We evaluate the HSCIC score in all 8 problem settings (see Table C of our PDF file.) (2) We evaluate conditional MI as another conditional independence metric to provide robust verification of our argument. (see Table D of PDF) Please refer to our general response.\n\n**4. Evaluation on complex task**\n\nWe perform an evaluation of our approach and the baseline methods in the CARLA environment using the D4RL dataset. Please see our general response and Table A of PDF.\n\n**5. Normalized score**\n\nWe follow the D4RL dataset evaluation protocol, which designates the expert score as the upper limit and the score of a random agent as the lower limit for normalization. We will clarify this in the revision.\n\n**Reference**\n\n[A] Haan et al., “Causal Confusion in Imitation Learning.”, NeurIPS 2019.\n"}, {"review_id": "vyiQL6kXpN", "reviewer": "Reviewer_TkkQ", "summary": "This paper aims to solve the problem of negative action leakage from past observations in the context of imitation learning. This is specifcally applicable to imitation learning when considering a history of observations. The paper argues that an information theoretic (entropy or MI based) regularization requires training an additional network and a nested optimization, however, HSCIC avoids both of these inefficiencies. The experiments show that (1) HSCIC is a good indicator of leakage, and (2) the propose approach which regularizes the BC loss with HSCIC outperforms other information theoritic and vanilla baselines quite a bit, in offline imitation settings. ", "questions": "See weaknesses", "limitations": "Limitations are sufficienlty addressed", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- To my knowledge this is a novel approach \n- The method is very interesting, and explained in a way that is easy to understand, and is also an innovative solution \n- The motivation is well described, and this is an important area to explore \n- The results show strong performance, and the analysis of the utility of HSCIC is insightful as well \n- The paper is well written \n\nIn general, I believe this paper should be presented at the conference. ", "weaknesses": "I think a couple of things could have been discussed: \n\n- How sequence modelling architectures can play a role here such as transformer/Decision Transformer (Chen et al., 2021)\n- What are the different modes of leakage, and do they affect performance in different ways\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback.\n\n**1. Applicability to sequence modeling architecture**\n\nWe acknowledge the potential effectiveness of our approach in sequence modeling architecture. To explore this further, we have applied our regularization to the Decision Transformer architecture you suggested. Please refer to our general response and Table B in the provided PDF.\n\n**2. Different modes of leakage on imitation learning**\n\nThe exploration of information leakage in imitation learning remains a promising direction of research, accompanied by several open questions. One example can be drawn to the concept of target leakage [A] in the context of data mining, where information produced from targets is contained in input data. Similarly, in imitation learning, a related phenomenon arises where information generated by the current expert action can become embedded within the observation. In this situation, the imitator easily captures that spurious information to predict expert action during training, but the imitator will fail in test time. This discrepancy between training and inference will lead to harmful effects on performance.\n\n**Reference**\n\n[A] Kaufman et al., \"Leakage in Data Mining: Formulation, Detection, and Avoidance.\", KDD 2011.\n\n\n"}, {"review_id": "4HqbbkXcE5", "reviewer": "Reviewer_UXMA", "summary": "This paper proposes to use HSCIC (Hilbert-Schmidt Conditional Independence Criterion) to alleviate the leakage of information from past actions in behavior cloning from observation histories. The advantage of HSCIC compared with information-theoretic regularization is it can compute in closed-form and does not require parametric assumption on the data distribution.\n\nExperimental results show HSCIC is a good indicator of agent performance and by directly minimizing HSCIC, it consistently improves over naive BC and other baselines with different regularizations.", "questions": "How well does the proposed methods work on more complex environments, e.g. Atari, Carla, etc.?\nCould you show some quantitative results on how the agents avoid failure due to leakage of previous action?\nIs there any heuristics of turning the hyperparameter alpha?\nCould this regularization be combined with other approaches (e.g. information bottleneck, sampling) to give better results?", "limitations": "Authors state theirhyperparameters are sensitive and hard to tune without interaction with the environment. No broader impact section found in the main paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Experimental results show significant improvement over baselines across different locomotion tasks.\nGood analysis showing negative correlation between HSCIC and agent performance, which motivates to minimize it directly in training objectives.\nWriting is clear and straightforward. Good comparison of kernel-based regularization and information-theoretic approaches.", "weaknesses": "Lack of comparisons with non-regularization based approaches to tackle past action information leakage. For example, Key-frame focused visual imitation learning proposes a sampling/weighting approach and Fighting fire with fire: Avoiding DNN shortcuts through priming proposes to use additional prior information as inductive biases.\nExperiments are all on simple state-based results. Hard to judge the performance on more complex domains, e.g. visual imitation learning, robotic manipulation or autonomous driving.\nKernel-based approaches have time complexity depending on the number of samples. It would be worth comparing the computation tradeoff between different approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value your considerate insights and invaluable suggestions concerning our work.\n\n**1. Comparisons with other non-regularization approaches**\n\nThank you for the pointers to the missing important related work. We recognize that both of your suggestions are valid to compare with our approach. During the remaining rebuttal period, we do our best to evaluate these two approaches and update our experiment.\n\n**2. Experiments in complex domain**\n\nWe additionally conduct evaluation of our method and baselines in CARLA environment provided in D4RL dataset. Please see our general response and Table A of PDF file.\n\n**3. Time complexity comparison**\n\nIt is worth noting that the computation time of our PALR implementation increases with mini-batch sizes (denote $M$), but it does not depend on the number of total data samples; the kernel regression is applied to each mini-batch, not to the entire training set. Furthermore, there are techniques for computing the HSCIC estimator efficiently [A].\n\n In contrast, current information-theoretic approaches often require multiple iterations of inner optimization (denote $k$ as the number of inner iterations) to obtain reliable estimations of quantities of interest (e.g. mutual information or entropy). In practice, the elapsed time of these approaches can be significantly higher than PALR. To compare computation costs, we measure the wall-clock time of one policy update over our method and baselines. ($M$= 1024, $k$=5, 1000 repetitions)\n\n|  | BC | RAP | FCA | MINE | PALR |\n| --- | --- | --- | --- | --- | --- |\n| Big-O | $O(M)$ | $O(M)$ | $O(kM)$ | $O(kM)$ | $O(M^3)$ |\n| Average Elapsed time (ms) | 3.038 | 5.295 | 12.203 | 19.854 | 15.64 |\n\n[Machine specification] CPU : Intel(R) Core(TM) i7-4770 @ 3.40GHz (4 cores), GPU : Titan X,  Memory : 32Gb\n\n Our approach demonstrates scalable time complexity, particularly with moderate batch sizes, for example 512 to 1024. This ensures that PALR remains scalable and effective in various practical scenarios. We will include a detailed discussion of the computation tradeoff comparison in the appendix of our revised manuscript.\n\n\n**4. Quantitative analysis**\n\nTo enhance the credibility of our experimental interpretation, we broaden our quantitative analysis. Firstly, we assess the HSCIC score across all 8 problem settings (refer to Table C in our provided PDF). Secondly, we examine conditional MI as an additional measure of conditional independence to robustly validate our argument (refer to Table D in the PDF). Please see our general response for further details. If you have any further question in additional quantitative results, please feel free to inform us.\n\n\n**5. Heuristics of tuning alpha**\n\nIn the context of offline model selection for sequential decision-making, it is well-acknowledged that identifying optimal hyperparameters from offline dataset is a non-trivial task [B,C,D,E]. Systematic hyperparameter tuning in an offline setting remains an open challenge. In our empirical observations from Mujoco experiments, we note that effective coefficients typically fall within the range of 10 to 1000. Notably, this range tends to align the scale of the HSCIC regularization term with that of the BC loss term.\n\n**6. Combination with other approaches**\n\nPALR could potentially synergize with orthogonal regularization approaches, which we consider for future exploration.\n\n\n**References**\n\n[A] Quinzan et al., “Learning Counterfactually Invariant Predictors”, arxiv 2022.\n\n[B] Hussenot et al., “Hyperparameter Selection for Imitation Learning.”, ICML 2021.\n\n[C] Zhang et al. \"Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning.\" NeurIPS 2021.\n\n[D] Paine et al. \"Hyperparameter Selection for Offline Reinforcement Learning.\" arXiv 2020.\n\n[E] Lee et al., “Batch Reinforcement Learning with Hyperparameter Gradients.”, ICML 2020.\n\n"}, {"review_id": "FodTJUyRmH", "reviewer": "Reviewer_xrec", "summary": "This paper solves the past action information leakage problem in behavior cloning. The paper first formally defines this problem, and then provides some potential regularization methods. After careful analysis of the pros and cons of each method, the authors decide to use HSCIC and validate its performance on several tasks.", "questions": "The BC policy used in this paper is just a feed-forward policy p(a|s).\nIs it possible to apply the regulation to other kinds of policy, for example, an energy-based model [1] or even a diffusion model[2]? \n\n[1] Florence, Pete, et al. \"Implicit behavioral cloning.\" Conference on Robot Learning\n[2] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" arXiv preprint arXiv:2303.04137", "limitations": "no explicit limitation discussion are provided in the paper", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The flow of this paper is very clear: problem definition -- potential solutions -- analysis -- evaluation\n- In the evaluation, the authors not only show the improvement of the final performance of BC but also highlight the correlation between the performance and HSCIC regularization. This strongly proves that regularization is the major contribution to performance improvement.\n- Comprehensive ablation study and analysis are provided in the paper.", "weaknesses": "The major concern is how to apply this regularization to complex environments with images (or even multi-modality) as input and high-dimensional action space (6 DoF or even more). If the proposed method can be applied to such complex tasks, I'm willing to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We hold your thoughtful viewpoints on our work in high regard.\n\n**1. Effectiveness in complex task**\n\nTo validate past action leakage regularization is effective on complex tasks, we conduct an experiment on CARLA environment. Please see our general response and Table A of PDF.\n\n**2. Applicability to complex policy architecture**\n\nWe appreciate your consideration of different policy models. While we chose a feed-forward policy due to task generality, we acknowledge the potential to extend our approach to more specialized policies. Methods like energy-based [1] and diffusion models [2] designed for tasks with visual inputs and complex action spaces could enhance the applicability of our approach. Transitioning to these models is a direction we foresee exploring in adapting PALR for visuomotor control tasks. Furthermore, we've extended our investigations to the Decision Transformer (DT) [A] architecture to assess PALR's effectiveness on complex policies, as detailed in our general response and Table B of PDF.\n\n**3. Limitation Section**\n\nLimitations section can be found in Section F provided in supplementary material.\n\n**Reference**\n\n[A] Chen et al., “Decision Transformer: Reinforcement learning via Sequence Modeling.”, NeurIPS 2021."}, {"review_id": "0vtXJsrIbp", "reviewer": "Reviewer_Xa8t", "summary": "This paper introduces Past Action Leakage Regularization (PALR) for blocking the leakage of past action information during behavior cloning. Concretely, PALR focuses on the problem when a BC agent simply remembers the past actions rather than learning a generalized behavior, which gives a degenerated policy. Specifically, PALR tackles the problem by learning a representation of the history to remove unnecessary from the observations via conditional independence. PALR adopts the HSCIC metric to measure the conditional independence, which alleviates the shortcomings of the information-theoretic metric. In their experiments, the authors compared different metrics for behavior cloning on a range of continuous control tasks.", "questions": "How does the method compare with more advanced imitation learning algorithms that require interactions with the environment, e.g., DAC [1] and PWIL [2]? BC is a relatively weak baseline. Can this regularization term be used in these algorithms to further boost the performance?\n\nReferences:\n\n[1] Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., & Tompson, J. (2018). Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925.\n\n[2] Dadashi, R., Hussenot, L., Geist, M., & Pietquin, O. (2020). Primal wasserstein imitation learning. arXiv preprint arXiv:2006.04678.", "limitations": "This paper has relatively limited novelty and insufficient experiment results. Given its current state, I think the paper is not ready for publication yet.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is overall well-written and easy to follow. The idea is well-motivated as well.", "weaknesses": "The paper has the following main issues.\n\nFirstly, the contribution of this paper is relatively incremental. The formulation of the BC regularization as conditional entropy is new. Wen et al. [1] was the first to propose this formulation. As a result, the main contribution of this work is only the HSCIC regularization term.\n\nIn addition, the paper has claimed that the focus of the work is POMDPs, where the past action leakage might come from the history. However, in the experiments, the only domain considered is state-based continuous control, which is a fully observable MDP. In my opinion, such a domain does not support the claim of the work, and is unsuitable to measure the capability of the proposed method. More complex and natural POMDP domains should be considered, e.g., indoor navigation with pixel-based observations, to better support the claims.\n\nReferences:\n\n[1] Wen, C., Lin, J., Darrell, T., Jayaraman, D., & Gao, Y. (2020). Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 33, 2564-2575.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your constructive and insightful comments.\n\n**1. Novelty in our work**\n\nWe think that although the HSCIC regularization indeed holds significance within our work, our contribution extends beyond the introduction of the HSCIC regularization term: in the paper, we introduced a novel perspective on the issue of past action leakage through the lens of conditional independence. Our approach, which is derived naturally from this perspective and is implemented as a regularizer for conditional independence, covers FCA [A], HSCIC, and all the other measures for conditional independence. Note that an analogous approach based on conditional entropy, not conditional independence, is less general; for instance, it does not cover HSCIC. Identifying the important role of conditional independence in our problem setting is, we believe, an important conceptual contribution of our work.\n\n\n**2. Mujoco experiments in POMDP settings**\n\nWe want to clarify that we have reconfigured the observations of Mujoco tasks to retain only positional information while excluding velocity information from observations, hence we conduct experiments on POMDP versions of Mujoco tasks. For further details, please refer to Section C.1 of the supplementary material.\n\n\n**3. Additional experiment on a complex domain**\n\n To see if our approach is effective on a complex domain, we perform an evaluation of our approach and the baseline methods within the CARLA environment. Please see our general response and Table A in the attached PDF file for more details.\n\n**4. BC as a fundamental baseline**\n\nOur focus is on offline IL-OH, excluding online interaction with the environment. While BC is simple, it's not weak in this context. In our experiments, BC's performance rivaled that of other baselines like FCA and MINE. Additionally, BC's role in simplifying training, away from intricate policy optimization, helps dissect the impact of different regularization methods.\n\n**5. Comparison with online imitation learning algorithms**\n\n We appreciate your comments and would like to provide clarification regarding our research focus. Our research centers on offline IL from observation histories (IL-OH), and while we acknowledge methods like DAC and PWIL focus on online IL with fully-observable MDPs, which is beyond the scope of our present research. Please also note that our study pertains to addressing the challenge from the problematic phenomenon of past action information leakage, which is present in offline IL-OH problems. If such a phenomenon occurs in online IL, we believe that our regularization could effectively mitigate the spurious correlation between past expert actions and imitator actions. We leave this extension as our future research.\n\n\n**References**\n\n[A] Wen et al., \"Fighting Copycat Agents in Behavioral Cloning from Observation Histories.\", NeurIPS 2020.\n"}]

Author rebuttal:
# **General Response**

We sincerely thank the reviewers for their insightful and detailed comments. Below, we address key questions and feedback that have been consistently raised by the reviewers. If there are any aspects that still need clarification or elaboration, we are more than happy to address them during the author-reviewer discussion period.

## **1. Experiment in CARLA Environment (Table A)**

To demonstrate the effectiveness of our method on high-dimensional observation scenarios, we extended our evaluation to the CARLA environment, an image-based autonomous driving task. We only consider pixel images as observations, excluding other information like velocity or sensor data. Leveraging `carla-lane-v0` dataset provided in D4RL dataset [A], we implement the imitator policy built upon ResNet34 architecture, and the parameters of ResNet34 are fixed and only used for feature extraction from images. Detailed results are presented in Table A of the provided PDF file. Notably, PALR outperforms the baseline methods by a significant margin. This observation supports our conclusion that PALR can successfully enhance imitation performance in complex offline IL-OH problems.

## **2. Effectiveness on Decision Transformer Policy (Table B)**

To assess the efficacy of our method in regularizing policies with complex network architectures, we conducted additional experiments employing the Decision Transformer (DT) [B], one of the prominent offline RL methods. Given that reward information is absent in the offline IL dataset, we inserted a reward input of 0 into DT's structure to retain its original configuration. Our regularization approach, termed DT-PALR, was applied to the last hidden state of DT. We evaluated both the standard DT and DT-PALR across three POMDP versions of Mujoco tasks, consistent with the scenarios outlined in Table 1 of the paper.

The results are presented in Table B of the PDF file. With the exception of the `halfcheetah` task, it becomes evident that DT's performance lags behind that of BC, as demonstrated in Table 1 of the manuscript. This divergence might be attributed to DT's utilization of a larger network size, rendering it more susceptible to capturing spurious causal relationships in environments where access to complete states and rewards is restricted. Encouragingly, our results indicate a substantial enhancement in the performance of DT for the `hopper` and `walker2d` tasks when subjected to the DT-PALR method. This observation strongly suggests the adaptability of our approach to intricate models.

## **3. Comprehensive Quantitative Analysis (Table C, D)**

To enhance the reliability of the quantitative analysis provided in our manuscript, we comprehensively evaluate HSCIC scores ($\widehat{\mathrm{HSCIC}}^2(a^I_t, a^E_{t-1} | a^E_{t-1})$) across all 8 problem settings (see Table C in PDF file). The results demonstrate that our method consistently reduces the conditional dependence between $a^I_t$ and $a^E_{t-1}$ given $a^I_{t}$. Notably, PALR achieves lowest HSCIC scores in entire problem settings.

Moreover, to show a more robust assessment of conditional independence, we also estimate conditional mutual information (CMI) $\hat{I}(a^I_t; a^E_{t-1} | a^E_{t-1})$ that also measures conditional dependence of our interest. (see Table D in PDF file) As we described in Section 4.2.1, CMI can be decompose into two MI terms, $I(a^E_{t-1}; \varphi_t, a^E_t)$ and $I(a^E_{t-1}; a^E_t)$, we estimate them using MINE [C] respectively. The table shows that PALR consistently presents lower CMI estimates compared to BC across 6 out of 8 problem settings.

We will include these responses into the appendix of our revised paper. Once again, we extend our gratitude for the valuable feedback from the reviewers.

## **References**

[A] Fu et al., “D4rl: Datasets for deep data-driven reinforcement learning.”, arxiv 2020.

[B] Chen et al., “Decision Transformer: Reinforcement learning via Sequence Modeling.”, NeurIPS 2021.

[C] Belghazi et al., “Mutual Information Neural Estimation.”, ICML 2018.

Meta-review: This is high-quality work that provides a novel method designed to address information leakage in imitation learning, which a problem of interest to the NeurIPS community. The authors should carefully revise the submission to include the additional results that they provided during the discussion period and also address several important points raised by the reviewers, including making more clear the assumptions made, revising notation for clarity, and explicitly acknowledging/explaining the limited experimental setting.
Decision: Accept (spotlight)