SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It is important to note that  $H(a_{t - 1}^{E}\mid a_{t}^{E})$  is determined by the data distribution  $\mathcal{D}$  and thus constant. As a result, we can simply consider the minimization of  $-H(a_{t - 1}^{E}\mid \varphi_{t},a_{t}^{E})$ .
$$
\mathcal {L} _ {\text {r e g - E n t}} \left(\varphi_ {t}; a _ {t - 1} ^ {E}, a _ {t} ^ {E}\right) := - H \left(a _ {t - 1} ^ {E} \mid \varphi_ {t}, a _ {t} ^ {E}\right) = \mathbb {E} \left[ \log p \left(a _ {t - 1} ^ {E} \mid \varphi_ {t}, a _ {t} ^ {E}\right) \right]. \tag {4}
$$
Interestingly, Eq. (4) coincides with the negative conditional entropy maximization objective in FCA [44]. Since the direct computation of Eq. (4) requires to know the intractable distribution  $p(a_{t - 1}^{E} \mid \varphi_{t}, a_{t}^{E})$ , FCA trains an additional prediction model  $\hat{p}(\hat{a}_{t - 1} \mid \varphi_{t}, a_{t})$  to estimate the negative entropy with  $-\hat{H}(a_{t - 1}^{E} \mid \varphi_{t}, a_{t}^{E}) \coloneqq \mathbb{E}_{\varphi_{t}}[\log \hat{p}(a_{t - 1}^{E} \mid \varphi_{t}, a_{t}^{E})]$ . However, this approach faces the following challenges:
1. The estimated negative entropy  $-\hat{H}$  lower bounds Eq. (4), while an upper bound would be desirable for minimizing the objective. Consequently, it imposes nested (minimax) optimization; minimizing with respect to  $\varphi_t$  after maximizing with respect to  $\hat{p}$ .
2. It is required to train an additional neural network to model  $\hat{p}$ , which consumes additional computational cost.
3. Assuming that the family of variational distribution  $\hat{p}$  to be Gaussian, FCA tries to tighten the lower bound estimation  $-\hat{H}$  with respect to  $\hat{p}$  by minimizing reconstruction error of  $a_{t-1}^{E}$ . The assumption restricts the flexibility of  $\hat{p}$  and thus the estimation can be inaccurate.
To avoid inaccurate estimation of the entropy, we can also decompose CMI into two MI terms by the chain rule of MI as an alternative:
$$
I (a _ {t - 1} ^ {E}; \varphi_ {t} \mid a _ {t} ^ {E}) = I (a _ {t - 1} ^ {E}; \varphi_ {t}, a _ {t} ^ {E}) - I (a _ {t - 1} ^ {E}; a _ {t} ^ {E}),
$$
Similar to  $H(a_{t-1}^E \mid a_t^E)$ ,  $I(a_{t-1}^E; a_t^E)$  can be safely ignored in the regularization. As a result, the regularization is about simply minimizing  $I(a_{t-1}^E; \varphi_t, a_t^E)$  while ignoring the constant MI term.
$$
\mathcal {L} _ {\text {r e g - M I}} \left(\varphi_ {t}; a _ {t - 1} ^ {E}, a _ {t} ^ {E}\right) := I \left(a _ {t - 1} ^ {E}; \varphi_ {t}, a _ {t} ^ {E}\right). \tag {5}
$$
Since the direct computation of Eq. (5) requires to know densities of  $a_{t-1}^E, \varphi_t, a_t^E$ , one needs to train sample-based MI estimators [3, 16, 28, 34]. Thanks to those estimators, the estimated MI can be minimized without confining any distribution (to be Gaussian). However, this approach still has issues similar to FCA such that (1) it lower bounds MI and consequently imposes (nested) minimax optimization and (2) it introduces an additional neural network particularly sensitive to hyperparameters.
4.2.2 HSCIC regularization
To bypass those shortcomings in information-theoretic regularization, we consider HSCIC [30], a kernel-based conditional independence metric for the past action leakage regularization.
$$
\mathcal {L} _ {\text {r e g - H S C I C}} \left(\varphi_ {t}; a _ {t - 1} ^ {E}, a _ {t} ^ {E}\right) := \text {H S C I C} \left(\varphi_ {t}, a _ {t - 1} ^ {E} \mid a _ {t} ^ {E}\right). \tag {6}
$$
Let  $\mathcal{H}_{\mathcal{A}}, \mathcal{H}_{\Phi}$  be RKHSs over  $\mathcal{A}, \Phi$  and  $k_{\mathcal{A}}, k_{\Phi}$  be their associated kernels, where  $\Phi$  is the representation space induced by the encoder  $\varphi$ . Given  $n$  samples  $\{(\varphi_t(i), a_{t-1}^E(i), a_t^E(i))\}_{i=1}^n$  from  $\mathcal{D}$  and  $\varphi$ , let  $\mathbf{K}_{\varphi t}, \mathbf{K}_{a_{t-1}^E}, \mathbf{K}_{a_t^E}$  be the  $n \times n$  kernel matrices where  $[\mathbf{K}_{\varphi t}]_{i,j} = k_{\Phi}(\varphi_t(i), \varphi_t(j)), [\mathbf{K}_{a_{t-1}^E}]_{i,j} = k_{\mathcal{A}}(a_{t-1}^E(i), a_{t-1}^E(j)), [\mathbf{K}_{a_t^E}]_{i,j} = k_{\mathcal{A}}(a_t^E(i), a_t^E(j))$ . Then, HSCIC estimator for past action leakage regularization can be defined as follows:
$$
\begin{array}{l} \widehat {\mathrm {H S C I C}} (\varphi_ {t}, a _ {t - 1} ^ {E} | a _ {t} ^ {E}) := \frac {1}{n} \mathrm {t r a c e} \left(\mathbf {K} _ {a _ {t} ^ {E}} ^ {\top} \mathbf {W} (\mathbf {K} _ {\varphi_ {t}} \odot \mathbf {K} _ {a _ {t - 1} ^ {E}}) \mathbf {W} ^ {\top} \mathbf {K} _ {a _ {t} ^ {E}} \right. \\ - 2 \mathbf {K} _ {a _ {t} ^ {E}} ^ {\top} \mathbf {W} \left(\mathbf {K} _ {\varphi_ {t}} \mathbf {W} ^ {\top} \mathbf {K} _ {a _ {t} ^ {E}} \odot \mathbf {K} _ {a _ {t - 1} ^ {E}} \mathbf {W} ^ {\top} \mathbf {K} _ {a _ {t} ^ {E}}\right) \\ \left. + \left(\mathbf {K} _ {a _ {t} ^ {E}} ^ {\top} \mathbf {W K} _ {\varphi_ {t}} \mathbf {W} ^ {\top} \mathbf {K} _ {a _ {t} ^ {E}}\right) \odot \left(\mathbf {K} _ {a _ {t} ^ {E}} ^ {\top} \mathbf {W K} _ {a _ {t - 1} ^ {E}} \mathbf {W} ^ {\top} \mathbf {K} _ {a _ {t} ^ {E}}\right)\right) (7) \\ \end{array}
$$
where  $\mathbf{W} = (\mathbf{K}_{a_t^E} + n\lambda \mathbf{I})^{-1}$ ,  $\lambda > 0$  is a ridge regression coefficient,  $\odot$  is the element-wise matrix multiplication.
Since Eq. (7) can be estimated using the samples from the joint distribution  $p(\varphi_t, a_{t-1}^E, a_t^E)$ , we can directly plug-in HSCIC estimates into the regularization objective. By leveraging this estimator, HSCIC regularization offers several advantages compared to the conditional entropy regularization and MI regularization objectives discussed earlier:
1. Direct computation of the closed-form solution in Eq. (7) allows HSCIC regularization to bypass any nested optimization.
2. HSCIC regularization does not employ any additional deep neural networks that require careful hyperparameters.
3. Since HSCIC is a non-parametric measure, it does not impose any parametric assumption on the data distribution and does not require any density estimation.
These advantages strongly imply that promoting conditional independence via the HSCIC estimator will be more desirable compared to other estimators, thereby improving the overall effectiveness of the regularization. Hence, we propose HSCIC regularization to address the past action leakage problem, which we call PALR.
In this section, we present the experimental results of our approach. Initially, we investigate the correlation between the extent of past action information leakage and BC's performance. Subsequently, we compare our approach with several offline ILOH baseline methods across four continuous control tasks from the MuJoCo simulator [41]: hopper, walker2d, halfcheetah, and ant, as well as one pixel-based autonomous driving task from the CARLA simulator [9]: carla-lane $^{2}$ .
Dataset For tasks from the MuJoCo simulator, we transformed them into POMDP scenarios for ILOH by excluding specific state variables (such as velocity information). The remaining state variables, like positional information and joint angles, were treated as observation variables at individual timesteps<sup>3</sup>. We organized these observations into fixed-size stacks to configure each
2The D4RL benchmark also featured another task, carla-town, but none of the algorithms surpassed the random policy's performance, indicating a lack of meaningful imitation results. Further details can be found in Section D.2. 3Details about the composition of observation variables for each task are available in Section C.1.
problem setting, denoted as [envname] -W[stacksize] with stack sizes  $w \in \{2,4\}$ . For a task from the CARLA simulator, we used pixel images as observations for ILOH. To extract features from these observations, we employed a pretrained ResNet [15], keeping its parameters fixed during training. In carla-lane task, we stacked the extracted features with a fixed stack size of  $w = 3$  and used it as input for the policy. All our experiments utilized expert demonstrations from the D4RL benchmark dataset [10] to ensure the validity and reliability of our results.
Evaluation Metric For the performance evaluation of the learned policy, we measure the normalized score that ranges from 0 to 100. To evaluate how much the past action information is leaked into a policy  $\pi$ , we measure  $\widehat{\mathrm{HSCIC}}(a_t^I; a_{t-1}^E | a_t^E)$  using the estimator (7) in the held-out dataset. To estimate HSCIC estimator, we fix a ridge regression coefficient  $\lambda = 10^{-5}$  and all kernels are chosen as Gaussian kernels with the bandwidth  $\sigma^2 = 1$ .
5.1 Relationship between past action information leakage and performance
To see if the problem of the past action information leakage occurs, we conduct an empirical study using BC from observation histories with stack size  $w = 2$ . The objective of this experiment is to confirm the correlation between the performance and the degree of past action information leakage, similar to FCA [44].
Experimental setup To achieve multiple policies at different levels, we train BC policies for 500K steps with 5 different seeds and 7 different training dataset size  $N \in \{100\mathrm{K}, 50\mathrm{K}, 30\mathrm{K}, 10\mathrm{K}, 5\mathrm{K}, 3\mathrm{K}, 1\mathrm{K}\}$  (the number of observation history-action pairs). Using a held-out dataset composed of 2,000 samples, the degree of past action information leakage is measured by  $\widehat{\mathrm{HSCIC}}(a_t^I; a_{t-1}^E | a_t^E)$  of each policy  $\pi^I$ .
Figure 1: Negative correlation between HSCIC estimates and performance of BC with 7 different dataset sizes.
Results To compare HSCIC of fully trained BC in all tasks, we normalize HSCIC of each policy using the maximum and minimum values in each task. Grouping trained policies by the dataset size, we report the mean value of both the normalized score and the normalized HSCIC estimates in Figure 1. It shows that there are clear negative correlations between the normalized HSCIC and the normalized score in all environments. The result implies that BC from observation histories tends to train a policy in which there is a negative correlation between the conditional dependence related to its action and the performance. This insight suggests a potential opportunity for employing regularization methods that enforce conditional independence.
Experimental setup We evaluate the performance of our method and offline baseline methods across 5 environments see effectiveness of our method. We train policies using 5 different algorithms, including our method, for 500K, 1M training steps MuJoCo and CARLA tasks respectively with 5 different seeds.
Baseline methods We compare our method with 6 offline ILOH baseline methods: BC, KF [45], PrimeNet [46], RAP [6]  $^{4}$ , FCA [44], MINE [3]. While all of these are commonly based on BC, we specify their differences as follows:
For the sake of simplicity, here we refer to the baselines [6, 45] as KF (Keyframe-Focused visual imitation learning), RAP (Residual Action Prediction) respectively, following the titles of their respective papers.
Task	w	BC	KF	PrimeNet	RAP	FCA	MINE	PALR (Ours)
hopper	2	32.5 ± 2.9	32.0 ± 1.9	30.0 ± 1.6	20.2 ± 1.4	31.9 ± 2.5	25.0 ± 1.9	42.0 ± 2.4
4	47.7 ± 3.4	45.7 ± 1.0	45.3 ± 2.8	32.6 ± 2.6	36.9 ± 2.4	37.6 ± 3.1	58.4 ± 2.8
walker2d	2	53.0 ± 2.7	50.0 ± 2.3	48.5 ± 3.3	15.8 ± 2.0	63.1 ± 2.7	58.6 ± 5.5	79.8 ± 2.3
4	63.2 ± 6.3	77.4 ± 2.0	79.2 ± 3.3	25.4 ± 2.1	81.9 ± 3.3	68.7 ± 6.7	83.4 ± 5.4
halfcheetah	2	74.1 ± 2.3	64.3 ± 1.4	61.5 ± 1.9	63.9 ± 2.1	78.2 ± 2.8	76.3 ± 1.9	86.4 ± 1.1
4	68.4 ± 2.6	55.7 ± 4.1	45.5 ± 1.7	59.0 ± 2.7	69.9 ± 2.6	73.4 ± 2.4	79.1 ± 4.3
ant	2	56.3 ± 3.5	54.9 ± 1.7	51.7 ± 2.4	44.1 ± 1.2	51.1 ± 2.2	53.9 ± 1.9	59.6 ± 3.0
4	64.4 ± 1.8	48.6 ± 3.8	58.2 ± 1.9	48.6 ± 2.6	57.7 ± 1.3	56.6 ± 1.8	64.6 ± 2.5
carla-lane	3	52.5 ± 6.2	66.6 ± 2.1	58.2 ± 2.2	25.3 ± 5.4	57.1 ± 3.1	60.1 ± 4.1	72.9 ± 2.6
Table 1: Performance evaluation of baseline and regularization methods. The normalized scores averaged over the final 50 evaluations during training and we report mean and standard error over 5 different seeds. The rightmost three algorithms are incorporated into our regularization framework. The method with the highest mean score and its competitive methods (within standard error) are highlighted in bold in each problem setting.
- BC: the standard BC algorithm from observation history, which optimizes an objective without any regularization term ( $\alpha = 0$ ).
- KF [45]: the weighted BC algorithm that assigns higher weights to keyframes, which contain actions that are highly predictable from their corresponding action histories.
- PrimeNet [46]: a supervised learning method designed to prevent undesirable shortcuts by leveraging additional key inputs.
- RAP [6]: it employs a dual-stream of policy representation that learns from both observation history and individual observations, maximizing a lower bound that enforces conditional dependence between the representation and expert action, given the past action.
- FCA [44]: it maximizes the conditional entropy that corresponds to Eq. (4) with adversarial training, which is an instance of our regularized BC framework.
- MINE [3]: it minimizes the MI estimate corresponding to Eq. (5) using MINE estimator [3], which is one of the representative sample-based neural estimators for MI. It is also an instance of our regularized BC framework.
Our method regularizes HSCIC with its estimator defined as Eq. (7), where Gaussian kernel with fixed bandwidth  $\sigma^2 = 1$  are used for all kernels of  $\varphi_t, a_{t-1}^E, a_t^E$  and a ridge regression coefficient  $\lambda = 10^{-5}$ . Across all regularization methods, we searched for the optimal  $\alpha$  according to the best mean normalized score. Further implementation details can be found in Section C.
Results Table 1 summarizes the results of performance evaluation for each problem settings. Our method significantly outperforms other baselines in 7 settings out of 9 and shows competitive performance in the rest 2 settings. In particular, our method shows strong performance in carla-lane, highlighting its effectiveness in enhancing performance within high-dimensional offline ILOH scenarios. We observe that RAP shows the least competitive performance across all tasks. This is because they do not have any penalization of the dependence between the imitator action and expert action in their objective function. However, we also observe that FCA and MINE fail to show consistent improvement over BC in most tasks except walker2d. This is because the lower bound estimators of their regularization objectives are not sufficiently accurate even at the cost of their inefficient alternating optimization. On the other hand, our method consistently outperforms BC, which clearly indicates the effect of promoting the conditional independence as we discussed in Section 4.1.
To better understand the degenerate performance of FCA and MINE, we evaluate  $\widehat{\mathrm{HSCIC}}(a_t^I; a_{t-1}^E | a_t^E)$  of each regularization method during training using the held-out dataset in Figure 2a. In contrast to our method, the result clearly demonstrates that FCA and MINE commonly
Figure 2: Investigating past action leakage regularization methods on D4RL dataset.
fail to promote conditional independence throughout the learning process, elucidating their underperformance. To further investigate, we measure the HSCIC and conditional MI estimates of the finalized imitator policies across 8 MuJoCo settings (refer to Section D.5). In essence, our method adeptly mitigates the leakage of past action information, surpassing the efficacy of alternative approaches.
Regularization coefficient In this experiment, we aim to ascertain how the selection of coefficient  $\alpha$  influences the performance of our method. We evaluate the asymptotic performance of our method with varying values of  $\alpha \in \{0,10^{0},10^{1},10^{2},10^{3},10^{4},10^{5},10^{6}\}$  on the walker2d-W2 problem setting, which shows the largest performance gap between PALR and BC among our problem settings. As depicted in Figure 2b, the converged regularization loss  $\mathcal{L}_{\mathrm{reg - HSCIC}}$  progressively decreases as  $\alpha$  increases. Notably, the  $\alpha$  that minimizes the regularization loss is not necessarily equal to the optimal hyperparameter that maximizes performance. We observe that the selection of  $\alpha$  is important, as it adjusts the trade-off between robustness to the leakage and alignment with expert data.
Grounded in the classical notion of conditional independence, we proposed a principled regularization framework for BC that mitigates past action information leakage problem. Within our framework, we have explored multiple choices of conditional independence metric and compared their estimators. Finally, we identified that our method with HSCIC estimator is the most favorable regularization of BC over other choices in terms of robustness to the leaked information of past action. In an extensive set of experiments on D4RL datasets, we empirically showed that our method significantly outperforms baseline methods. We also observed in our experiments that all the comparing methods including ours were sensitive to the choice of  $\alpha$ . Without assuming any interaction with the environment, it is challenging to find the optimal  $\alpha$  only with the given offline dataset. We believe that the discovery of optimal  $\alpha$  in offline manner would be an interesting research topic, which we leave as future work.
This work was partly supported by IITP grant funded by MSIT (No.2020-0-00940, Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects; No.2019-0-00075, AI Graduate School Program (KAIST); No.2021-0-02068, AI Innovation Hub), NRF of Korea (NRF2019R1A2C1087634; RS-2023-00279680; NRF2018R1A5A1059921), Field-oriented Technology Development Project for Customs Administration through NRF of Korea funded by the MSIT and Korea Customs Service (NRF2021M3I1A1097938), ETRI grant (22ZS1100, Core Technology Research for Self-Improving Integrated AI System), KAIST-NAVER Hypercreative AI Center.
A Details of HSCIC and Its Empirical Estimation 15
A.1 Definitions 15
A.2 Empirical estimation 16
B Proof of Theorem 1 17
C Implementation Details 18
C.1 Observation specification 18
C.1.1 Hopper 18
C.1.2 Walker2d 18
C.1.3 HalfCheetah 19
C.1.4 Ant 20
C.2 Dataset preprocessing 20
C.3 Hyperparameter selection 20
D Additional Experiments 22
D.1 Direct regularization on imitator action 22
D.2 Experiment on carla-town 22
D.3 Comparison with VIB (Variational Information Bottleneck) 23
D.4 Effectiveness on DT (Decision Transformer) policy 23
D.5 Quantitative analysis on past action information leakage 24
D.6 Details on the leakage-performance correlation 25
E Broader Impacts 26
F Limitations 26
G Computation Resources 26
H Licenses 26
A Details of HSCIC and Its Empirical Estimation
Definition 1. (Reproducing kernel Hilbert space)
Let  $\mathcal{H}_{\mathcal{X}}$  be a Hilbert space of functions  $f:\mathcal{X}\to \mathbb{R}$  functions which the metric is defined by inner product  $\langle \cdot ,\cdot \rangle_{\mathcal{H}_{\mathcal{X}}}$ . A symmetric function  $k_{\mathcal{X}}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ , which is called a reproducing kernel of  $\mathcal{H}_{\mathcal{X}}$ , that satisfies
1.  $\forall x\in \mathcal{X}$ $k_{\mathcal{X}}(x,\cdot)\in \mathcal{H}_{\mathcal{X}}$
2.  $\forall x\in \mathcal{X},\forall f\in \mathcal{H}_{\mathcal{X}}$ $\langle f,k_{\mathcal{X}}(x,\cdot)\rangle_{\mathcal{H}_{\mathcal{X}}} = f(x)$  (called reproducing property)
A space  $\mathcal{H}_{\mathcal{X}}$  is called a reproducing kernel Hilbert spaces (RKHS) corresponding to reproducing kernel  $k_{\mathcal{X}}$ .
Definition 2. (Kernel Mean Embedding)
Given a distribution  $P_{X}$  on  $\mathcal{X}$ , assume  $\int_{\mathcal{X}} \sqrt{k_{\mathcal{X}}(x, x)} dP_{X}(x) < \infty$ . We define the kernel mean embedding  $\mu_{P_X}$  of  $P_{X}$  as  $\mu_{P_X}(\cdot) = \int_{\mathcal{X}} k_{\mathcal{X}}(x, \cdot) dP_X(x)$ .
Definition 3. (Maximum Mean Discrepancy)
Given distributions  $P, Q$  on  $\mathcal{X}$  and suppose kernel mean embeddings of  $P, Q$  exist on  $\mathcal{H}$ , denote  $\mu_P, \mu_Q$ . Then, the maximum mean discrepancy (MMD) between  $P_X, Q_Y$  is defined as follows:
$$
\operatorname {M M D} (P, Q; \mathcal {H}) = \| \mu_ {P} - \mu_ {Q} \| _ {\mathcal {H}}
$$
Definition 4. (Hilbert-Schmidt Norm and Operator)
Let  $\mathcal{F},\mathcal{G}$  be Hilbert spaces. Define  $(f_i)_{i\in I},(g_j)_{j\in J}$  to be orthonormal basis for  $\mathcal{F},\mathcal{G}$  respectively. Define two linear operators  $L:\mathcal{G}\to \mathcal{F},M:\mathcal{G}\to \mathcal{F}$ .
1. The Hilbert-Schmidt norm of the operator  $L$  is defined as:
$$
\begin{array}{l} \| L \| _ {\mathrm {H S}} ^ {2} = \sum_ {j \in J} \| L g _ {j} \| _ {\mathcal {F}} ^ {2} \\ = \sum_ {j \in J} \sum_ {i \in I} | \langle L g _ {j}, f _ {i} \rangle_ {\mathcal {F}} | ^ {2} \\ \end{array}
$$
2. The operator  $L$  is called Hilbert-Schmidt Operator when  $\| L\|_{\mathrm{HS}}^2$  is finite.
Definition 5. (Cross-covariance Operator)
Given RKHS  $\mathcal{F},\mathcal{G}$  with kernels  $k,l$  , feature maps  $\phi ,\psi$  respectively. Define the uncentered covariance operator  $\tilde{C}_{XY}:\mathcal{G}\to \mathcal{F}$  and the cross-covariance operator  $C_{XY}:\mathcal{G}\rightarrow \mathcal{F}$  as follows:
$$
\langle \tilde {C} _ {X Y}, A \rangle_ {H S} = \mathbb {E} _ {x, y} \langle \phi (x) \otimes \psi (y), A \rangle_ {\mathrm {H S}}
$$
$$
C _ {X Y} := \tilde {C} _ {X Y} - \mu_ {X} \otimes \mu_ {Y}
$$
Definition 6. (Hilbert-Schmidt Independence Criterion)
The Hilbert-Schmidt Independence Criterion (HSIC) is defined as Hilbert-Schmidt norm of the cross-covariance operator  $C_{XY}$ , i.e.,  $\mathrm{HSIC}(P_{XY};\mathcal{F},\mathcal{G})\coloneqq \| C_{XY}\|_{\mathrm{HS}}^2$ . Equivalently, we can define HSIC as MMD between  $P_{XY}$  and  $P_{X}P_{Y}$  with the product kernel, i.e.
$$
\operatorname {H S I C} \left(P _ {X Y}; \mathcal {F}, \mathcal {G}\right) = \operatorname {M M D} ^ {2} \left(P _ {X Y}, P _ {X} P _ {Y}; \mathcal {H} _ {\kappa}\right),
$$
where  $\kappa ((x,y),(x^{\prime},y^{\prime})) = k(x,x^{\prime})l(y,y^{\prime}).$
Definition 7. (Conditional Mean Embedding)
Assuming a random variable  $X$  satisfies  $\int_{\mathcal{X}} \sqrt{k(x, x)} dP < \infty$ . Define the conditional mean embedding of  $X$  given  $Z$  as  $\mu_{P_{X|Z}}(\cdot) \coloneqq \mathbb{E}_{X|Z}[k_{\mathcal{X}}(X, \cdot)|Z]$ .
Note that  $\mu_{P_{X|Z}}$  is a random variable. In the main text of our paper, we denote the realization of  $\mu_{P_{X|Z}}$  with  $Z = z$  as  $\mu_{P_{X|Z = z}}$ .
Definition 8. (Hilbert-Schmidt Conditional Independence Criterion)
Define the Hilbert-Schmidt Conditional Independence Criterion (HSCIC) between  $X$  and  $Y$  given  $Z$  to be
$$
\operatorname {H S C I C} (X, Y | Z) = \left\| \mu_ {P _ {X Y | Z}} - \mu_ {P _ {X | Z}} \otimes \mu_ {P _ {Y | Z}} \right\| _ {\mathcal {H} _ {X} \otimes \mathcal {H} _ {Y}} ^ {2}.
$$
A.2 Empirical estimation
We start to explain from decoupling the conditional mean embedding into their deterministic function and conditional random variable.
Lemma 1. (Decomposition of conditional mean embedding, Theorem 4.1 of [30])
There exists a deterministic function  $F_{P_{X|Z}}: \mathcal{Z} \to \mathcal{H}_{\mathcal{X}}$ , which satisfies  $\mu_{P_{X|Z}} = F_{P_{X|Z}} \circ Z$ .
By this lemma, we can decompose conditional mean embeddings and HSCIC into some deterministic function and conditioned random variable  $Z$ , i.e.,  $\mu_{P_{X|Z}} = F_{P_{X|Z}} \circ Z$ ,  $\mu_{P_{Y|Z}} = F_{P_{Y|Z}} \circ Z$ ,  $\mu_{P_{XY|Z}} = F_{P_{XY|Z}} \circ Z$ , HSCIC  $(X,Y|Z) = F_{\mathrm{HSCIC}_{X,Y|Z}} \circ Z$  respectively. By using vector-valued RKHS regression with a regularization parameter  $\lambda > 0$  and the representative theorem [23], we can obtain closed-form estimates of  $F_{P_{X|Z}}, F_{P_{Y|Z}}, F_{P_{XY|Z}}$  as follows (see [30] for more details):