SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seokin Seo $^{1}$ , HyeongJoo Hwang $^{1}$ , Hongseok Yang $^{1,2}$ , Kee-Eung Kim $^{1,2}$
<sup>1</sup>Kim Jaechul Graduate School of AI, KAIST
$^{2}$ School of Computing, KAIST
siseo@ai.kaist.ac.kr,hjhwang@ai.kaist.ac.kr,
hongseok.yang@kaist.ac.kr, kekim@kaist.ac.kr
For partially observable environments, imitation learning with observation histories (ILOH) assumes that control-relevant information is sufficiently captured in the observation histories for imitating the expert actions. In the offline setting where the agent is required to learn to imitate without interaction with the environment, behavior cloning (BC) has been shown to be a simple yet effective method for imitation learning. However, when the information about the actions executed in the past timesteps leaks into the observation histories, ILOH via BC often ends up imitating its own past actions. In this paper, we address this catastrophic failure by proposing a principled regularization for BC, which we name Past Action Leakage Regularization (PALR). The main idea behind our approach is to leverage the classical notion of conditional independence to mitigate the leakage. We compare different instances of our framework with natural choices of conditional independence metric and its estimator. The result of our comparison advocates the use of a particular kernel-based estimator for the conditional independence metric. We conduct an extensive set of experiments on benchmark datasets in order to assess the effectiveness of our regularization method. The experimental results show that our method significantly outperforms prior related approaches, highlighting its potential to successfully imitate expert actions when the past action information leaks into the observation histories.
Imitation learning (IL) aims at learning a policy that recovers an expert's behavior from a demonstration dataset. Leveraging the information about state and expert action available in the dataset, IL has been successful in many real-world applications [4, 12, 26, 27, 32, 38, 43]. Although IL problems can be addressed using either online [17, 36] or offline algorithms [20, 33, 48], real-world tasks often impose restrictions on interacting with the environment due to safety, cost, and ethical concerns. Consequently, the practical necessity lies in the development of effective offline IL algorithms. Behavior cloning (BC) [33] is one of the most prominent offline IL algorithms, which learns to predict an expert's action for each given state using an offline expert dataset via supervised learning. BC has gained widespread recognition [7, 42] for providing a straightforward and effective solution, especially when full access to state information is available and the dataset is sufficiently extensive. However, when the state information is only partially available in an observation at each timestep, which is closer to the realistic scenario (e.g. an autonomous vehicle with a limited number
of sensors), training with any IL algorithm can be complicated. To enhance the ability of agent to infer missing control-relevant information from observations, incorporating the history of observations from adjacent past timesteps as input can be beneficial [2, 24].
When utilizing observation history for behavior cloning, a notable challenge emerges: the unnecessary dependence of the imitator's actions on the preceding actions, resulting in a suboptimal behavior in test time. In extreme cases, this dependency can lead to undesirable behaviors, such as the imitation policy that merely replicates its own actions from previous time steps  $^{1}$ , which leads to a catastrophic result particularly in safety-critical tasks like autonomous driving. This is often rooted at the inability to differentiate between actual and spurious causal relationship between observation features and expert actions within the collected data [8]. Past action information is a representative instance of such nuisance features, since it is a strongly correlated to the target expert action. This correlation can mislead the IL algorithm into recognizing past action information, which is not causally related to the target expert action, as a crucial feature for prediction. In this work, we refer to this misleading phenomenon as past action information leakage.
A natural way to mitigate this phenomenon is to adopt a mechanism that blocks the leakage of past action information irrelevant to the target expert action. In fact, Wen et al. [44] proposed an adversarial training approach to discard such redundant information in the representation of the observation history. This is achieved by maximizing the conditional entropy of the expert's previous action given the joint of the representation and target expert action. However, due to the intractability of direct computation of the conditional entropy, this approach relies on unstable adversarial learning. More recently, Chuang et al. [6] focuses on image-based observations and attempts to mitigate the past action dependency by splitting the policy representation into two parts: a representation of the observation history and a representation of the current observation. This method assumes a particular policy structure and does not provide a systematic general approach to block the leakage of past action information into the representation.
In this paper, we present a regularized behavior cloning framework that effectively mitigates the leakage of past action information. We formally define the problem of the leakage of past action information and establish a metric to quantify the magnitude of the leakage, employing the kernel-based method called HSCIC (Hilbert-Schmidt Conditional Independence Criterion) [30]. Building upon the metric, we devise an objective function that addresses the aforementioned problem, allowing for a more comprehensive understanding and interpretation of existing work [44]. Moreover, we propose a stable and efficient kernel-based regularization method that circumvents challenges such as adversarial learning, nested optimization and reliance on a neural estimator. Lastly, we conduct an extensive set of experiments which empirically show that our regularization method effectively blocks the leakage of past action information across a variety of control task benchmarks.
- We formally define the problem of the leakage of past action information based on the concept of conditional independence by quantifying the amount of leaked past action information.
- We introduce a principled framework for a behavior cloning with Past Action Leakage Regularization (PALR), which prevents the imitator from overfitting to leaked past information.
- We provide experimental results on established benchmarks, demonstrating the effectiveness of our method.
Invariant representation learning Learning representation invariant to any unwanted factors has been widely studied in various domains such as fair classification [22, 25, 37, 47], domain adaptation [11, 13, 18, 49], and imitation learning [6, 44]. One of the dominant approaches in invariant representation learning is adversarial learning [11, 13, 44, 47]. Adversarial learning algorithms
<sup>1</sup>The problem has many names: copycat problem [6, 44], inertia problem [7], latching effect [40]. Beyond the falsely leaded action repeating behavior, we introduce the past action information leakage to separate the reason and the result.
commonly train additional networks that predict the unwanted factors from the representations while enforcing representations to make those prediction models fail. Consequently, they require alternating optimization between the main and the additional models [13, 44, 47] or show numerical instability [11]. To bypass those shortcomings of adversarial learning, several information-theoretic approaches [18, 25] have been proposed. Based on Variational Auto-Encoders (VAE) [21], these methods proposed end-to-end learning algorithms that jointly optimize all of their components with numerical stability. However, these methods assume that the distribution of the representation is Gaussian, which restricts the flexibility of the representation. Wen et al. [46] concentrated on eliminating shortcuts in supervised learning by incorporating supplementary key information, demonstrating its enhancement of behavior cloning with observation histories. Recent methods have proposed to learn counterfactually or conditionally invariant representation, leveraging on kernel-based conditional independence metric [31, 35]. These methods demonstrated promising empirical results in synthetic domains, effectively mitigating the impact of nuisance correlations.
Information leakage in imitation learning There is a growing understanding of the importance of addressing the correlation between expert actions and nuisance features, which are not essential for control and may even hinder performance. [8, 29, 39] Especially, termed as causal confusion [8], imitation learning exhibits a paradoxical phenomenon: having more information can lead to worse performance. Aligning with such observation, recent works have also demonstrated that accessing more information from the observation history leaks past action information so that the imitator may learn an undesirable policy that simply repeats the same action in the past [6, 7, 40, 44, 45]. To avoid learning degenerate solutions from the past action information, Wen et al. [44] proposed the regularization method based on the conditional entropy of the previous action given the representation of the history and the current action. However, their method involves a nested minimax optimization along with an additional neural network, which complicates the training process. We closely investigate their formulation in Section 4.2.1 to show their limitations as well as their connection to our method. Wen et al. [44] also introduced the action predictability metric to quantify the dependence between past action histories and imitator actions, relative to expert actions. However, this metric is not based on conditional independence, which is central to our argument concerning the past action leakage. Wen et al. [45] addressed this problem using a weighted behavior cloning method that upweights "keyframe" samples, which are more likely to be predicted from the action histories. Swamy et al. [40] showed that online interaction is both necessary and sufficient to resolve repeating behavior. However, in many real-world applications [4, 12, 26, 27, 32, 38, 43], online interaction is often infeasible due to safety, cost, and ethical considerations. In this work, we develop an offline algorithm that can robustly handle the leakage of past action information.
3.1 Conditional independence
For random variables  $X, Y$  which taking a value in  $\mathcal{X}, \mathcal{Y}$  respectively, we write  $P_X$ ,  $P_{XY}$ , and  $P_{X|Y}$  for the marginal distribution of  $X$ , the joint distribution of  $X$  and  $Y$ , and the conditional distribution of  $X$  given  $Y$ , respectively. We say that random variables  $X$  and  $Y$  are conditionally independent given a random variable  $Z$  or simply  $Z$ -conditionally independent if  $P_{XY|Z}(x,y) = P_{X|Z}(x)P_{Y|Z}(y)$  for all  $x,y \in \mathcal{X} \times \mathcal{Y}$ . Also, we say that  $X$  and  $Y$  are independent if  $P_{XY}(x,y) = P_X(x)P_Y(y)$  for all  $x,y$ . We denote  $Z$ -conditional independence by  $X \perp Y \mid Z$ , and independence by  $X \perp Y$ .
Conditional Mutual Information (CMI) Mutual information (MI) is an information-theoretic quantity to measure the dependency between two random variables. MI between random variables  $X$  and  $Y$  is defined as  $I(X;Y) = D_{\mathrm{KL}}(P_{XY},P_XP_Y) = \mathbb{E}_{x,y\sim P_{XY}}[\log P_{XY}(x,y) - \log P_X(x)P_Y(y)]$ . MI is always non-negative (i.e.,  $I(X;Y)\geq 0$ ), and it becomes zero if and only if  $X\perp Y$ . Conditional mutual information (CMI) is defined similarly, but with conditional distributions. CMI between random variables  $X$  and  $Y$  given a random variable  $Z$  is defined as  $I(X;Y\mid Z) = \mathbb{E}_{P_Z}[D_{\mathrm{KL}}(P_{XY|Z},P_{X|Z}P_{Y|Z})]$ . As in the case of MI,  $I(X;Y\mid Z)\geq 0$ , and also  $I(X;Y\mid Z) = 0$  if and only if  $X\perp Y\mid Z$ .
Hilbert-Schmidt Conditional Independence Criterion (HSCIC) Hilbert-Schmidt Independence Criterion (HSIC) [14] is a kernel-based measure that quantifies the dependency between two random variables. Let  $\mathcal{H}_{\mathcal{X}}$  and  $\mathcal{H}_{\mathcal{Y}}$  be reproducing kernel Hilbert spaces (RKHS) on  $\mathcal{X}$  and  $\mathcal{Y}$  with the corresponding reproducing kernels  $k_{\mathcal{X}}$  and  $k_{\mathcal{Y}}$ , respectively. The tensor product  $k_{\mathcal{X}} \otimes k_{\mathcal{Y}}$  of the kernels is a binary function on  $\mathcal{X} \times \mathcal{Y}$  defined by  $(k_{\mathcal{X}} \otimes k_{\mathcal{Y}})((x_1, y_1), (x_2, y_2)) := k_{\mathcal{X}}(x_1, x_2)k_{\mathcal{Y}}(y_1, y_2)$ . The RKHS on  $\mathcal{X} \times \mathcal{Y}$  with the kernel  $k_{\mathcal{X}} \otimes k_{\mathcal{Y}}$  is called the tensor product RKHS of  $\mathcal{H}_{\mathcal{X}}$  and  $\mathcal{H}_{\mathcal{Y}}$ , and it is denoted by  $\mathcal{H}_{\mathcal{X}} \otimes \mathcal{H}_{\mathcal{Y}}$ . HSIC between  $X$  and  $Y$  is defined to be the maximum mean discrepancy (MMD) between  $P_{XY}$  and  $P_{X}P_{Y}$  under  $\mathcal{H}_{\mathcal{X}} \otimes \mathcal{H}_{\mathcal{Y}}$ , i.e., the distance in the Hilbert space  $\mathcal{H}_{\mathcal{X}} \otimes \mathcal{H}_{\mathcal{Y}}$  between the so-called kernel mean embeddings of  $P_{XY}$  and  $P_{X}P_{Y}$  to that space:
$$
\begin{array}{l} \operatorname {H S I C} (X, Y) := \operatorname {M M D} ^ {2} \left(P _ {X Y}, P _ {X} P _ {Y}; \mathcal {H} _ {\mathcal {X}} \otimes \mathcal {H} _ {\mathcal {Y}}\right) \\ = \left\| \mu_ {P _ {X Y}} - \mu_ {P _ {X}} \otimes \mu_ {P _ {Y}} \right\| _ {\mathcal {H} _ {X} \otimes \mathcal {H} _ {Y}} ^ {2} \\ \end{array}
$$
where  $\mu_{P_{XY}}$  is the kernel mean embedding of the distribution  $P_{XY}$  to  $\mathcal{H}_{\mathcal{X}}\otimes \mathcal{H}_{\mathcal{Y}}$ , i.e.  $\mu_{P_{XY}}(x,y)\coloneqq \mathbb{E}_{P_{XY}}[(k_{\mathcal{X}}\otimes k_{\mathcal{Y}})((X,Y),(x,y))]$ , and  $\mu_{P_X},\mu_{P_Y}$  are defined similarly but using  $\mathcal{H}_X,\mathcal{H}_Y$  instead. The  $(\mu_{P_X}\otimes \mu_{P_Y})$  is simply the function in  $\mathcal{H}_{\mathcal{X}}\otimes \mathcal{H}_{\mathcal{Y}}$  that maps  $(x,y)$  to  $\mu_{P_X}(x)\mu_{P_Y}(y)$ . When  $k_{\mathcal{X}}\otimes k_{\mathcal{Y}}$  satisfies some condition (i.e., characteristic), HSIC  $(X,Y) = 0$  if and only if  $X\perp Y$  [14].
HSCIC (Hilbert-Schmidt Conditional Independence Criterion) is an extension of HSIC that quantifies the amount of conditional dependency between two random variables given another random variable. In this paper, we follow the definition of HSCIC based on conditional mean embedding [30], and use an estimator of HSCIC that draws samples from the joint distribution  $P_{XYZ}$  and performs vector-valued RKHS regression. The conditional mean embedding of  $X$  given  $Z$  is a function in the RKHS  $\mathcal{H}_{\mathcal{X}}$  that is parameterized by the value of  $Z$ . Given a value  $z$  of  $Z$ , it maps  $x \in \mathcal{X}$  to  $\mu_{P_{X|Z = z}}(x) \coloneqq \mathbb{E}_{P_{X|Z = z}}[k_{\mathcal{X}}(X,x)]$ . Then, HSCIC is a mapping from a value of  $Z$  to a non-negative real defined as follows: for all  $z \in \mathcal{Z}$ ,
$$
\operatorname {H S C I C} (X, Y | Z = z) := \left\| \mu_ {P _ {X Y | Z = z}} - \mu_ {P _ {X | Z = z}} \otimes \mu_ {P _ {Y | Z = z}} \right\| _ {\mathcal {H} _ {x} \otimes \mathcal {H} _ {y}} ^ {2}
$$
When  $k_{\mathcal{X}} \otimes k_{\mathcal{Y}}$  satisfies the condition mentioned from above,  $X \perp Y \mid Z$  if and only if  $\mathbb{E}_{P_Z}[\widehat{\mathrm{HSCIC}}(X,Y|Z)] = 0$  [30]. We use an empirical estimator of the expectation here, denoted by  $\widehat{\mathrm{HSCIC}}(X,Y|Z)$ . See Section A in the supplementary material for more details for definition of HSCIC and its empirical estimator.
3.2 Imitation learning from observation histories
We consider a Partially Observable Markov Decision Process (POMDP) [19] without reward, which is defined as a tuple of  $\langle S, \mathcal{Z}, \mathcal{A}, O, P, \rho_0 \rangle$ . Here  $\mathcal{S}$ ,  $\mathcal{Z}$  and  $\mathcal{A}$  are an (underlying) state space, an observation space and an action space, respectively. The next  $O: \mathcal{S} \to \Delta \mathcal{Z}$  specifies the conditional probability  $O(z|s)$  of an observation  $z \in \mathcal{Z}$  given a state  $s \in \mathcal{S}$ . Finally,  $\rho_0 \in \Delta S$  defines the probability  $\rho_0(s_0)$  that the process starts from  $s_0 \in \mathcal{S}$ , and  $P: \mathcal{S} \times \mathcal{A} \to \Delta S$  defines the probability  $P(s'|s, a)$  of transitioning to state  $s'$  when action  $a \in \mathcal{A}$  is performed in state  $s \in \mathcal{S}$ .
Assume a (stochastic) expert policy  $\pi^E: S \to \Delta \mathcal{A}$  that is defined as a conditional probability  $\pi^E(a|s)$  of the expert's performing an action  $a \in \mathcal{A}$  given a state  $s \in S$ . Also, assume that we have a dataset  $\mathcal{D} = \{\tau^{(1)}, \ldots, \tau^{(N)}\}$  of the expert trajectories where each  $\tau = \{(z_t, a_t)\}_{t=0}^T$  is sampled by
$$
s _ {0} \sim \rho_ {0}, a _ {t} ^ {E} \sim \pi^ {E} (\cdot | s _ {t}), z _ {t} \sim O (\cdot | s _ {t}), s _ {t + 1} \sim P (\cdot | s _ {t}, a _ {t}) \text {f o r} t \in \{0, 1, \dots , T \}.
$$
That is,  $\tau$  in the dataset is drawn from the following joint distribution of all observations and actions:
$$
\tau \sim p _ {D} (z _ {0: T}, a _ {0: T} ^ {E}) = \int \rho_ {0} (s _ {0}) \prod_ {t = 0} ^ {T} O (z _ {t} | s _ {t}) \pi^ {E} (a _ {t} ^ {E} | s _ {t}) P (s _ {t + 1} | s _ {t}, a _ {t} ^ {E}) d s _ {0: T + 1}.
$$
In our study, we consider POMDP scenarios where an ideal imitator policy is able to match the expert policy's performance, even when the imitator's actions are solely determined by observation histories. Specifically, we focus on situations where the observation histories  $z_{t - w - 1:t}$  (for some fixed  $1 \leq w \leq T - 1$ ) encompass all information about the true states  $s_t$  utilized by the expert policy.
In this setting, our goal is to learn an imitator policy  $\pi^I:\mathcal{Z}^w\to \Delta \mathcal{A}$  that acts as closely to  $\pi^{E}$  as possible on the given dataset  $\mathcal{D}$ . To achieve this goal, we consider the following joint distribution of actions of both  $\pi^I$  and  $\pi^{E}$ :
$$
\begin{array}{l} p \left(a _ {0: T} ^ {I}, a _ {0: T} ^ {E}\right) = \\ \int \rho_ {0} (s _ {0}) \prod_ {t = 0} ^ {T} O (z _ {t} | s _ {t}) \pi^ {I} \left(a _ {t} ^ {I} | z _ {t - w + 1: t}\right) \pi^ {E} \left(a _ {t} ^ {E} | s _ {t}\right) P \left(s _ {t + 1} | s _ {t}, a _ {t} ^ {E}\right) d s _ {0: T + 1} d z _ {0: T + 1}. \tag {1} \\ \end{array}
$$
4 Behavior Cloning with Past Action Leakage Regularization
In this section, we propose a framework that effectively mitigates the past action leakage problem in IL. We first define the problem by formalizing the absence of leaked past-action information via conditional independence (Section 4.1). Then, we compare several regularization-based approaches that attempt to achieve the absence of such information in the context of offline IL (Section 4.2). These approaches performs regularized BC where the regularizer is derived from a metric for measuring the amount of conditional dependence among random variables. The choice of the metric differentiates these approaches, and our comparison advocates the use of the HSCIC-based approach.
4.1 Past action leakage problem in imitation learning
When a policy takes an observation history as an input, the input history may include the information about past actions unexpectedly. The inclusion of such information can have detrimental effects in the context of offline imitation learning by confusing the imitator and making it fail to predict expert actions accurately. Intuitively, the past action leakage problem in imitation learning refers to this failure of the imitator due to the leakage of such harmful past action information.
To express this intuition formally, for each timestep  $t$ , let  $A_{t}^{E}$  and  $A_{t}^{I}$  denote random variables of expert action and imitator action at timestep  $t$ . Note that for each  $0 < t \leq T$ , the joint distribution of the three random variables  $A_{t-1}^{E}, A_{t}^{E}$  and  $A_{t}^{I}$  is
$$
p \left(a _ {t - 1} ^ {E}, a _ {t} ^ {E}, a _ {t} ^ {I}\right) = \int p \left(a _ {0: T} ^ {I}, a _ {0: T} ^ {E}\right) d a _ {0: t - 2} ^ {E} d a _ {0: t - 1} ^ {I} d a _ {t + 1: T} ^ {E} d a _ {t + 1: T} ^ {I}
$$
where  $p(a_{0:T}^{I},a_{0:T}^{E})$  is the distribution in Eq. (1).
We formalize the absence of the leakage of harmful past-action information by conditional independence between the imitator's current actions and the expert's previous actions:
$$
A _ {t} ^ {I} \perp A _ {t - 1} ^ {E} \mid A _ {t} ^ {E} \quad \text {f o r a l l} 0 <   t \leq T. \tag {2}
$$
This conditional independence says that the imitator's current action never depends on some information that is only about the expert's past action but not about the expert's current action. This past-specific information corresponds to harmful information in our intuitive explanation from above.
Ideally we would like to achieve conditional independence in Eq. (2), which ensures the absence of leaked past-action information that was harmful to the imitator. However, in practice, we can achieve it only approximately, so that we need a quantitative measure for conditional independence or the lack of conditional independence between  $A_{t-1}^{E}$  and  $A_{t}^{E}$  given  $A_{t}^{I}$ . Such a measure is also needed to design an offline IL algorithm that does not suffer from such leaked harmful past-action information. In the following subsection, we consider two quantitative measures for the lack of conditional independence, namely, (1) conditional mutual information (CMI) and (2) Hilbert-Schmidt Conditional Independence Criterion (HSCIC).
4.2 Behavior cloning with past action leakage regularization
We aim to learn the representation  $\varphi_t$  of observation history  $z_{t - w + 1:t}$  that removes any unnecessary information on the past action. To simplify the notation, let  $t_w$  denote  $t - w + 1$ . Our method is based on the following observation.
Theorem 1. Let  $A_{t}^{I}$  be the action from the imitator policy  $\pi^{I}(a_{t}|\varphi_{t})$  based on the representation  $\varphi_{t}$  of observation history. Then,  $\varphi_{t} \perp A_{t - 1}^{E} \mid A_{t}^{E} \Rightarrow A_{t}^{I} \perp A_{t - 1}^{E} \mid A_{t}^{E}$ .
Proof. See Section B in the supplementary material.
To this end, we formulate our objective of regularized BC framework as follows:
$$
\mathcal {L} (\pi , \varphi ; \mathcal {D}, \alpha) := \mathcal {L} _ {\mathrm {b c}} (\pi , \varphi ; \mathcal {D}) + \alpha \cdot \mathcal {L} _ {\mathrm {r e g}} (\varphi ; \mathcal {D}), \tag {3}
$$
where  $\mathcal{L}_{\mathrm{bc}}$  is an BC objective such that  $\mathcal{L}_{\mathrm{bc}}(\pi, \varphi; \mathcal{D}) \coloneqq \mathbb{E}_{(z_{t_w:t}, a_t^E) \sim \mathcal{D}, \varphi_t \sim \varphi(z_{t_w:t})} \left[ -\log \pi(a_t^E | \varphi_t) \right]$  and  $\mathcal{L}_{\mathrm{reg}}$  is a past action leakage regularization objective. For notational simplicity, we abbreviate the expectation with respect to  $(a_{t-1}^E, z_{t_w:t}, a_t^E) \sim \mathcal{D}, \varphi_t \sim \varphi(z_{t_w:t})$  to  $\mathbb{E}_{\varphi_t}$ . In the following subsections, we discuss candidates for  $\mathcal{L}_{\mathrm{reg}}$ .
4.2.1 Information-theoretic regularization
A straightforward way to address the conditional independence we discussed in Section 4.1 is minimizing the CMI, which can be decomposed by its definition into two conditional entropy terms:
$$
I (a _ {t - 1} ^ {E}; \varphi_ {t} \mid a _ {t} ^ {E}) = H (a _ {t - 1} ^ {E} \mid a _ {t} ^ {E}) - H (a _ {t - 1} ^ {E} \mid \varphi_ {t}, a _ {t} ^ {E}).
$$