SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {W} _ {Z} := (\mathbf {K} _ {Z} + n \lambda \mathbf {I}) ^ {- 1}
$$
$$
\hat {F} _ {P _ {X \mid Z}} (z) = \mathbf {k} _ {Z} ^ {\top} (z) \mathbf {W} _ {Z} \mathbf {k} _ {X} (\cdot)
$$
$$
\hat {F} _ {P _ {Y \mid Z}} (z) = \mathbf {k} _ {Z} ^ {\top} (z) \mathbf {W} _ {Z} \mathbf {k} _ {Y} (\cdot)
$$
$$
\hat {F} _ {P _ {X Y \mid Z}} (z) = \mathbf {k} _ {Z} ^ {\top} (z) \mathbf {W} _ {Z} (\mathbf {k} _ {X} (\cdot) \odot \mathbf {k} _ {Y} (\cdot))
$$
where  $\mathbf{k}_X(\cdot) := (k_{\mathcal{X}}(x_1,\cdot),\dots,k_{\mathcal{X}}(x_n,\cdot))^{\top},\mathbf{k}_Y(\cdot) := (k_{\mathcal{Y}}(y_1,\cdot),\dots,k_{\mathcal{Y}}(y_n,\cdot))^{\top},\mathbf{k}_Z(\cdot) := (k_{\mathcal{Z}}(z_1,\cdot),\dots,k_{\mathcal{Z}}(z_n,\cdot))^{\top},[\mathbf{K}_{Z}]_{ij} := k_{\mathcal{Z}}(z_i,z_j)$  and  $\odot$  is the element-wise multiplication operator of matrices. By plugging-in conditional mean embedding estimators, we can obtain a closed-form estimator of  $F_{\mathrm{HSCIC}_{X,Y|Z}}$  as follows:
$$
\begin{array}{l} \hat {F} _ {\text {H S C I C} _ {X, Y \mid Z}} (z) := \| \hat {F} _ {P _ {X Y \mid Z}} (z) - \hat {F} _ {P _ {X \mid Z}} (z) \otimes \hat {F} _ {P _ {Y \mid Z}} (z) \| _ {\mathcal {H} _ {\mathcal {X}} \otimes \mathcal {H} _ {\mathcal {Y}}} ^ {2} \\ = \mathbf {k} _ {Z} ^ {\top} (z) \mathbf {W} _ {Z} \left(\mathbf {K} _ {X} \odot \mathbf {K} _ {Y}\right) \mathbf {W} _ {Z} ^ {\top} \mathbf {k} _ {Z} (z) \\ - 2 \mathbf {k} _ {Z} ^ {\top} (z) \mathbf {W} _ {Z} \left(\mathbf {K} _ {X} \mathbf {W} _ {Z} ^ {\top} \mathbf {k} _ {Z} (z) \odot \mathbf {K} _ {Y} \mathbf {W} _ {Z} ^ {\top} \mathbf {k} _ {Z} (z)\right) \\ + \left(\mathbf {k} _ {Z} ^ {\top} (z) \mathbf {W} _ {Z} \mathbf {K} _ {X} \mathbf {W} _ {Z} ^ {\top} \mathbf {k} _ {Z} (z)\right) \left(\mathbf {k} _ {Z} ^ {\top} (z) \mathbf {W} _ {Z} \mathbf {K} _ {Y} \mathbf {W} _ {Z} ^ {\top} \mathbf {k} _ {Z} (z)\right) \\ \end{array}
$$
$$
\begin{array}{l} \widehat {\mathrm {H S C I C}} (X, Y | Z) = \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {F} _ {\mathrm {H S C I C} _ {X, Y | Z}} \left(z _ {i}\right) \\ = \frac {1}{n} \operatorname {t r a c e} \left(\mathbf {K} _ {Z} ^ {\top} \mathbf {W} _ {Z} \left(\mathbf {K} _ {X} \odot \mathbf {K} _ {Y}\right) \mathbf {W} _ {Z} ^ {\top} \mathbf {K} _ {Z} \right. \\ - 2 \mathbf {K} _ {Z} ^ {\top} \mathbf {W} _ {Z} \left(\mathbf {K} _ {X} \mathbf {W} _ {Z} ^ {\top} \mathbf {K} _ {Z} \odot \mathbf {K} _ {Y} \mathbf {W} _ {Z} ^ {\top} \mathbf {K} _ {Z}\right) \\ \left. + \left(\mathbf {K} _ {Z} ^ {\top} \mathbf {W} _ {Z} \mathbf {K} _ {X} \mathbf {W} _ {Z} ^ {\top} \mathbf {K} _ {Z}\right) \odot \left(\mathbf {K} _ {Z} ^ {\top} \mathbf {W} _ {Z} \mathbf {K} _ {Y} \mathbf {W} _ {Z} ^ {\top} \mathbf {K} _ {Z}\right)\right) \tag {8} \\ \end{array}
$$
where  $[\mathbf{K}_X]_{ij} \coloneqq k_{\mathcal{X}}(x_i, x_j), [\mathbf{K}_Y]_{ij} \coloneqq k_{\mathcal{Y}}(y_i, y_j)$ .
Figure 3: Graphical model of behavior cloning with observation histories.
Proof. By the chain rule of MI, following equalities hold:
$$
I \left(\varphi_ {t}; a _ {t - 1} ^ {E} \mid a _ {t} ^ {E}\right) = I \left(a _ {t - 1} ^ {E}; \varphi_ {t}, a _ {t} ^ {E}\right) - I \left(a _ {t - 1} ^ {E}; a _ {t} ^ {E}\right)
$$
$$
I (a _ {t} ^ {I}; a _ {t - 1} ^ {E} \mid a _ {t} ^ {E}) = I (a _ {t - 1} ^ {E}; a _ {t} ^ {I}, a _ {t} ^ {E}) - I (a _ {t - 1} ^ {E}; a _ {t} ^ {E})
$$
Thus,  $I(\varphi_t; a_{t-1}^E \mid a_t^E) \geq I(a_t^I; a_{t-1}^E \mid a_t^E)$  if and only if  $I(a_{t-1}^E; \varphi_t, a_t^E) \geq I(a_{t-1}^E; a_t^I, a_t^E)$ .
By the chain rule of MI,
$$
\begin{array}{l} I \left(a _ {t - 1} ^ {E}; \varphi_ {t}, a _ {t} ^ {I}, a _ {t} ^ {E}\right) = I \left(a _ {t - 1} ^ {E}; \varphi_ {t}, a _ {t} ^ {E}\right) + I \left(a _ {t - 1} ^ {E}; a _ {t} ^ {I} \mid \varphi_ {t}, a _ {t} ^ {E}\right) \\ = I (a _ {t - 1} ^ {E}; a _ {t} ^ {I}, a _ {t} ^ {E}) + \widehat {I} (a _ {t - 1} ^ {E}; \varphi_ {t} \mid a _ {t} ^ {I}, a _ {t} ^ {E}) \\ \ge I (a _ {t - 1} ^ {E}; a _ {t} ^ {I}, a _ {t} ^ {E}) \\ \end{array}
$$
Thus,  $I(\varphi_t; a_{t-1}^E \mid a_t^E) \geq I(a_t^I; a_{t-1}^E \mid a_t^E) \forall t \geq 0$ .
Directly regularizing the imitator policy to enforce conditional independence may cause harmful results of training. This is because any policy that ensures the conditional independence does not necessarily yield optimal behavior. For instance, constant actions or actions sampled from uniform random distribution are independent of other random variables and thus satisfy Eq. (2), although they are not optimal in general. Based on Theorem 1, we adjust the representation of policy to inject the conditional independence. The effectiveness of our approach is compared with direct action regularization through an empirical study. The results are provided in Table 9 of Section D.1.
This section provides detailed specifications and implementation settings.
C.1 Observation specification
Based on the observation specification from Open AI Gym documentation<sup>5</sup>, we take a partial observation in each task by excluding certain variables (e.g. velocity information) in the original state variables. We mark the variables we used in our observation as  $\bigcirc$  and not used are represented as  $\times$  in the observation column of the corresponding table.
C.1.1 Hopper
Index	Description	Unit	Observation
0	z-coordinate of the top	position (m)	○
1	angle of the top	angle (rad)	○
2	angle of the thigh joint	angle (rad)	○
3	angle of the leg joint	angle (rad)	○
4	angle of the foot joint	angle (rad)	○
5	velocity of the x-coordinate of the top	velocity (m/s)	×
6	velocity of the z-coordinate of the top	velocity (m/s)	×
7	angular velocity of the angle of the top	angular velocity (rad/s)	×
8	angular velocity of the thigh hinge	angular velocity (rad/s)	×
9	angular velocity of the leg hinge	angular velocity (rad/s)	×
10	angular velocity of the foot hinge	angular velocity (rad/s)	×
C.1.2 Walker2d
Table 2: Composition of observation variables (hopper)
Index	Description	Unit	Observation
0	z-coordinate of the top	position (m)	○
1	angle of the top	angle (rad)	○
2	angle of the thigh joint	angle (rad)	○
3	angle of the leg joint	angle (rad)	○
4	angle of the foot joint	angle (rad)	○
5	angle of the left thigh joint	angle (rad)	○
6	angle of the left leg joint	angle (rad)	○
7	angle of the left foot joint	angle (rad)	○
8	velocity of the x-coordinate of the top	velocity (m/s)	×
9	velocity of the z-coordinate of the top	velocity (m/s)	×
10	angular velocity of the angle of the top	angular velocity (rad/s)	×
11	angular velocity of the thigh hinge	angular velocity (rad/s)	×
12	angular velocity of the leg hinge	angular velocity (rad/s)	×
13	angular velocity of the foot hinge	angular velocity (rad/s)	×
14	angular velocity of the thigh hinge	angular velocity (rad/s)	×
15	angular velocity of the leg hinge	angular velocity (rad/s)	×
16	angular velocity of the foot hinge	angular velocity (rad/s)	×
Table 3: Composition of observation variables (walker2d)
<sup>5</sup>https://www.gymlibrary.dev/environments/mujoco/
C.1.3 HalfCheetah
Open AI Gym documentation does not match with the XML file of halfcheetah<sup>6</sup>, we denote both descriptions on the following tables.
Index	Description	Unit	Observation
0	z-coordinate of the front tip	position (m)	○
1	angle of the front tip	angle (rad)	○
2	angle of the second rotor	angle (rad)	○
3	angle of the second rotor	angle (rad)	○
4	velocity of the tip along the x-axis	velocity (m/s)	×
5	velocity of the tip along the y-axis	velocity (m/s)	×
6	angular velocity of the front tip	angular velocity (rad/s)	×
7	angular velocity of the second rotor	angular velocity (rad/s)	×
8	x-coordinate of the front tip	position (m)	○
9	y-coordinate of the front tip	position (m)	○
10	angle of the front tip	angle (rad)	○
11	angle of the second rotor	angle (rad)	○
12	angle of the second rotor	angle (rad)	○
13	velocity of the tip along the x-axis	velocity (m/s)	×
14	velocity of the tip along the y-axis	velocity (m/s)	×
15	angular velocity of the front tip	angular velocity (rad/s)	×
16	angular velocity of the second rotor	angular velocity (rad/s)	×
Table 4: Composition of observation variables (halfcheetah) on Gym Documentation.
Index	Description	Unit	Observation
0	z-coordinate of the root	position (m)	○
1	y-coordinate of the root	position (m)	○
2	angle of the back thigh	angle (rad)	○
3	angle of the back shin	angle (rad)	○
4	angle of the back foot	angle (rad)	×
5	angle of the front thigh	angle (rad)	×
6	angle of the front shin	angle (rad)	×
7	angle of the front foot	angle (rad)	×
8	velocity of the root along the x-axis	velocity (m/s)	○
9	velocity of the root along the z-axis	velocity (m/s)	○
10	velocity of the root along the y-axis	velocity (m/s)	○
11	angular velocity of the back thigh	angular velocity (rad/s)	○
12	angular velocity of the back shin	angular velocity (rad/s)	○
13	angular velocity of the back foot	angular velocity (rad/s)	×
14	angular velocity of the front thigh	angular velocity (rad/s)	×
15	angular velocity of the front shin	angular velocity (rad/s)	×
16	angular velocity of the front foot	angular velocity (rad/s)	×
Table 5: Composition of observation variables (halfcheetah) on the XML file.
<sup>6</sup>https://github.com/openai/gym/blob/master/gym/envs/mujoco/assets/half_chetah.xml
C.1.4 Ant
Num	Description	Unit	Observation
0	z-coordinate of the torso (centre)	position (m)	○
1	x-orientation of the torso (centre)	angle (rad)	○
2	y-orientation of the torso (centre)	angle (rad)	○
3	z-orientation of the torso (centre)	angle (rad)	○
4	w-orientation of the torso (centre)	angle (rad)	○
5	angle between torso and first link on front left	angle (rad)	○
6	angle between the two links on the front left	angle (rad)	○
7	angle between torso and first link on front right	angle (rad)	○
8	angle between the two links on the front right	angle (rad)	○
9	angle between torso and first link on back left	angle (rad)	○
10	angle between the two links on the back left	angle (rad)	○
11	angle between torso and first link on back right	angle (rad)	○
12	angle between the two links on the back right	angle (rad)	○
13	x-coordinate velocity of the torso	velocity (m/s)	×
14	y-coordinate velocity of the torso	velocity (m/s)	×
15	z-coordinate velocity of the torso	velocity (m/s)	×
16	x-coordinate angular velocity of the torso	angular velocity (rad/s)	×
17	y-coordinate angular velocity of the torso	angular velocity (rad/s)	×
18	z-coordinate angular velocity of the torso	angular velocity (rad/s)	×
19	angular velocity of angle between torso and front left link	angular velocity (rad/s)	×
20	angular velocity of the angle between front left links	angular velocity (rad/s)	×
21	angular velocity of angle between torso and front right link	angular velocity (rad/s)	×
22	angular velocity of the angle between front right links	angular velocity (rad/s)	×
23	angular velocity of angle between torso and back left link	angular velocity (rad/s)	×
24	angular velocity of the angle between back left links	angular velocity (rad/s)	×
25	angular velocity of angle between torso and back right link	angular velocity (rad/s)	×
26	angular velocity of the angle between back right links	angular velocity (rad/s)	×
27-110	contact forces applied to the center of mass of each of the links	force / torque	×
Table 6: Composition of observation variables (ant)
C.2 Dataset preprocessing
MuJoCo Tasks We utilized the expert dataset provided in D4RL benchmark that has a suffix -expert-v2. For all experiment, we configured partial observations from the original data based on Section C.1. These observations are standardized with statistics from the entire dataset. Each dataset in D4RL benchmark has a total of 1M transitions. Given the size of the training data  $D$ , we take the first  $D$  samples. Among the last 100K samples in the entire dataset, we also sampled heldout data which are not included in the training data. Given a stacksize  $w \in \{2,4\}$ , we configure each transition as (observation history, action, past action) tuple. For the main performance evaluation experiment (Table 1), we use 30K transitions for training.
CARLA Tasks We extract features from observation images using the pretrained network ResNet34 [15] and consider these features as policy inputs during training. For the performance evaluation (Table 1, 10), we utilize 1K and 100K transitions for carla-lane, carla-town respectively.
This section outlines the settings of the hyperparameters for each method. To ensure a fair comparison, we aim to maintain consistent network architecture and policy-related hyperparameters, whenever feasible. For all baselines and our method, we employ a policy architecture consisting of fully-connected layers with [128, 64, 128] hidden units. The layer with 64 units is treated as a policy representation, denoted as  $\varphi$ .
KF [45] We used a softmax function as a weighting function of the weighted BC in MuJoCo tasks and a step function in CARLA tasks, aligning with the original paper's experiments. We adopt the action prediction network for the additional network. Additionally, we tuned hyperparameters for the softmax temperature in MuJoCo tasks, analogous to  $\alpha$ . In CARLA tasks, we used the same hyperparameters (threshold=0.1, weight=5) consistent with the original paper [45].
PrimeNet [46] We treat observation histories as raw inputs and individual last observations as key inputs, consistent with the approach used in the original experiments. The architecture of the prime network is implemented as the same with the policy.
RAP [6] RAP incorporates two streams of policy representation: (1) one stream derived from observation history (referred to as the memory extraction stream) and (2) the other originating from a single observation (referred to as the policy stream). For the memory extraction stream, we use an architecture of [128, 64, 300] hidden units that outputs predictions of a residual of expert actions. The layer with 64 units is employed as a representation. For the policy stream, we similarly utilize an architecture of [128, 64(+64), 128] which outputs a policy action. The middle units indicate that the policy utilizes both the single observation representation and the memory-extracted representation (with a stop-gradient layer) as inputs.
A summary of hyperparameters is provided in Table 7. The first 3 rows of the table display the hyperparameters universally applied across all methods. The next 4 rows detail hyperparameters for methods that incorporate additional neural networks. The last 2 rows of the table present hyperparameters specific to our method, encompassing hyperparameters for HSCIC estimates.
Hyperparameter	KF	PrimeNet	RAP	FCA	MINE	PALR (Ours)
Policy distribution	Tanh Normal
Batch size	1024
BC learning rate	3e-4
Additional network
hidden units	[128,128]	[128,64,128]	[128,64,300]	[300]	[100,100]	-
num of inner updates	1	1	1	5	5	-
learning rate	3e-4	3e-4	3e-4	1e-4	1e-4	-
IB coefficient	-	-	-	0.01	-	-
Kernel bandwidth σ2	-	-	-	-	-	1.
Ridge coefficient λ	-	-	-	-	-	1e-5
Optimal  $\alpha$  As discussed in Section 5.2, we train 5 policies incorporating different regularization coefficients  $\alpha$  within the set  $\{0.01, 0.1, 1, 10, 100\}$  for MuJoCo tasks and  $\{0.001, 0.1, 10, 1000\}$  for CARLA task. We then select the best  $\alpha$  for each problem setting. Table 8 shows the chosen values of  $\alpha$  for each method and each problem setting, as utilized in the experimental results of the paper.
Table 7: The summary of hyperparameters.
Problem Setting	KF	FCA	MINE	Ours
hopper-W2	0.01	0.1	1	10
hopper-W4	0.01	0.1	0.01	10
walker2d-W2	0.01	0.1	1	100
walker2d-W4	0.1	0.1	1	100
halfcheetah-W2	0.01	0.1	1	10
halfcheetah-W4	0.1	0.1	0.1	10
ant-W2	1	0.1	0.01	10
ant-W4	0.01	1	0.01	10
carla-lane-W3	-	0.001	10	1000
Table 8: The summary of the best regularization coefficient  $\alpha$  .
D.1 Direct regularization on imitator action
To investigate the effectiveness of representation regularization, we compare our method to the one that directly regularizes action in MuJoCo settings. The objective function of the direct method is defined as follows:
$$
\mathcal {L} _ {\mathrm {P A L R - A C T}} (\pi ; \mathcal {D}, \alpha) := \mathcal {L} _ {\mathrm {b c}} (\pi ; \mathcal {D}) + \alpha \cdot \mathbb {E} _ {a _ {t} ^ {I} \sim \pi (\cdot | z _ {t _ {w}: t}), (z _ {t _ {w}: t}, a _ {t - 1} ^ {E}, a _ {t} ^ {E}) \sim \mathcal {D}} [ \widehat {\mathrm {H S C I C}} (a _ {t} ^ {I}, a _ {t - 1} ^ {E} | a _ {t} ^ {E}) ].
$$
We optimize the imitator policy  $\pi$  with the objective by using the reparameterization trick [21]. The method is also trained with different regularization coefficients  $\alpha \in \{0.01, 0.1, 1, 10, 100\}$  and we select the best  $\alpha$  of each method. Table 9 summarizes the results. Our representation regularization method (PALR) is more successful in 6 out of the 8 problem settings when compared to direct action regularization. In ant-W2, ant-W4 problem settings, neither method demonstrates significant effectiveness when compared to BC method.
Task	w	BC	PALR (ours)	PALR-ACT
hopper	2	32.47 ± 2.85	42.01 ± 2.44	31.60 ± 2.81
4	47.65 ± 3.43	58.39 ± 2.76	46.14 ± 0.90
walker2d	2	53.04 ± 2.69	79.83 ± 2.29	58.50 ± 4.25
4	63.15 ± 6.28	83.42 ± 5.43	77.41 ± 2.37
halfcheetah	2	74.08 ± 2.33	86.44 ± 1.09	76.32 ± 1.30
4	68.35 ± 2.60	79.05 ± 4.28	75.93 ± 2.59
ant	2	56.25 ± 3.45	59.57 ± 3.03	58.32 ± 1.23
4	64.39 ± 1.77	64.64 ± 2.53	65.00 ± 2.46
Table 9: Performance comparison of BC, our method (PALR) and direct action regularization version of PALR (PALR-ACT). The normalized scores averaged over the final 50 evaluations during training and we report mean and standard error over 5 different seeds. The method with the highest mean score and its competitive methods (within standard error) are highlighted in bold in each setting.
D.2 Experiment on carla-town
D4RL benchmark provides two CARLA tasks: carla-lane, carla-town. As shown in Table 10, all algorithms fail to show meaningful imitation results, as none of them exceed random performance levels. This outcome is likely due to the dataset containing numerous instances of the same observations paired with different actions (59,984 observation-action pairs out of 100K pairs in total). Consequently, we have omitted the experiment results for carla-town from Table 1.
Task	w	BC	KF	PrimeNet	RAP	FCA	MINE	PALR (Ours)
carla-lane	3	52.5 ± 6.2	66.6 ± 2.1	58.2 ± 2.2	25.3 ± 5.4	57.1 ± 3.1	60.1 ± 4.1	72.9 ± 2.6
carla-town	3	-3.1 ± 1.1	-6.9 ± 0.3	-8.8 ± 0.5	-6.1 ± 1.0	-7.0 ± 2.1	1.0 ± 0.8	-1.5 ± 0.8
Table 10: Performance evaluation of baseline and regularization methods on CARLA datasets provided by D4RL benchmark.
D.3 Comparison with VIB (Variational Information Bottleneck)
To highlight the leakage of past action information is not trivially prevented by general representation regularization methods, we compare our approach with VIB [1], a representative method. We train BC with VIB with two different  $\beta$  values, specifically,  $\beta \in \{10^{-3}, 10^{-5}\}$ , 5 seeds for each problem setting. whereas our method demonstrates significant improvements in 6 out of 8 problem settings.
Task	w	BC	PALR (Ours)	VIB (β = 10-3)	VIB (β = 10-5)
hopper	2	32.47 ± 2.85	42.01 ± 2.44	30.57 ± 1.61	27.89 ± 1.55
4	47.65 ± 3.43	58.39 ± 2.76	51.20 ± 2.18	46.59 ± 2.26
walker2d	2	53.04 ± 2.69	79.83 ± 2.29	44.28 ± 2.32	35.98 ± 4.69
4	63.15 ± 6.28	83.42 ± 5.43	67.78 ± 2.67	65.99 ± 1.97
halfcheetah	2	74.08 ± 2.33	86.44 ± 1.09	73.76 ± 2.401	75.43 ± 1.77
4	68.35 ± 2.60	79.05 ± 4.28	67.89 ± 1.52	73.67 ± 2.86
ant	2	56.25 ± 3.45	59.57 ± 3.03	59.54 ± 1.65	55.29 ± 4.27
4	64.39 ± 1.77	64.64 ± 2.53	61.63 ± 2.77	61.25 ± 2.88
D.4 Effectiveness on DT (Decision Transformer) policy
To assess the efficacy of our method in regularizing policies with complex network architectures, we conducted additional experiments employing the Decision Transformer (DT) [5], one of the prominent offline RL methods. Given that reward information is absent in the offline IL dataset, we inserted a reward input of 0 into DT's structure to retain its original configuration. Our regularization approach, termed DT-PALR, was applied to the last hidden state of DT. We evaluated both the standard DT and DT-PALR across 3 POMDP versions of MuJoCo tasks, consistent with the scenarios outlined in Table 1.
The results are presented in Table 12. With the exception of the halfcheetah task, it becomes evident that DT's performance lags behind that of BC, as demonstrated in Table 1. This divergence might be attributed to DT's utilization of a larger network size, rendering it more susceptible to capturing spurious causal relationships in tasks where access to complete states and rewards is restricted. Encouragingly, our results indicate a substantial enhancement in the performance of DT for the hopper and walker2d tasks when subjected to the DT-PALR method. This observation strongly suggests the adaptability of our approach to intricate architectures.
Table 11: Performance comparison of BC, our method (PALR) and BC with VIB (Variational Information Bottleneck). The normalized scores averaged over the final 50 evaluations during training and we report mean and standard error over 5 different seeds. The method with the highest mean score and its competitive methods (within standard error) are highlighted in bold in each setting.
Method	hopper	walker2d	halfcheetah
DT	20.68 ± 5.25	23.93 ± 4.13	94.10 ± 3.54
DT-PALR	26.09 ± 6.59	32.58 ± 5.63	96.05 ± 3.60
Table 12: Effectiveness of PALR on Decision Transformer (DT) architecture. We train DT over 3 MuJoCo tasks based on our observation configurations and apply PALR to the last hidden state.
D.5 Quantitative analysis on past action information leakage
We comprehensively evaluate HSCIC scores across 8 MuJoCo problem settings in Table 13. The results demonstrate that our method consistently reduces the conditional dependence. Notably, our method achieves lowest HSCIC scores in entire problem settings. Moreover, to show a more robust assessment of conditional dependence, we also estimate conditional mutual information (CMI). (see Table 14) As we described in Section 4.2.1, CMI can be decomposed into two MI terms,  $I(a_{t-1}^E; \varphi_t, a_t^E)$  and  $I(a_{t-1}^E; a_t^E)$ , we estimate them using MINE [3] respectively. The table shows that our approach consistently presents lower CMI estimates compared to BC across 6 out of 8 problem settings.
Task	w	BC	RAP	FCA	MINE	PALR (Ours)
hopper	2	1.783 ×10-2	1.779 ×10-2(↓)	1.805 ×10-2(↑)	1.802 ×10-2(↑)	1.742 ×10-2(↓)
4	1.634 ×10-2	1.617 ×10-2(↓)	1.629 ×10-2(↓)	1.650 ×10-2(↑)	1.587 ×10-2(↓)
walker2d	2	5.442 ×10-2	5.523 ×10-2(↑)	5.415 ×10-2(↓)	5.482 ×10-2(↑)	5.362 ×10-2(↓)
4	5.279 ×10-2	5.350 ×10-2(↑)	5.248 ×10-2(↓)	5.298 ×10-2(↑)	5.182 ×10-2(↓)
halfcheetah	2	4.010 ×10-2	4.044 ×10-2(↑)	4.112 ×10-2(↑)	4.082 ×10-2(↑)	3.984 ×10-2(↓)
4	4.019 ×10-2	4.052 ×10-2(↑)	4.092 ×10-2(↑)	4.069 ×10-2(↑)	3.968 ×10-2(↓)
ant	2	6.014 ×10-2	6.164 ×10-2(↑)	6.182 ×10-2(↑)	6.178 ×10-2(↑)	6.008 ×10-2(↓)
4	5.938 ×10-2	6.187 ×10-2(↑)	6.093 ×10-2(↑)	6.103 ×10-2(↑)	5.932 ×10-2(↓)
Table 13:  $\widehat{\mathrm{HSCIC}}(a_t^I, a_{t-1}^E | a_t^E)$  evaluation results of regularization methods and baselines. HSCIC scores are measured on held-out datasets and averaged over the final 50 evaluations during training and statistics are calculated with 5 seeds.  $(\downarrow)$ ,  $(\uparrow)$  marks whether the averaged HSCIC score of each method is lower than HSCIC score of BC or not.
Task	w	BC	RAP	FCA	MINE	PALR (Ours)
hopper	2	0.1825	0.1826 (↑)	0.1949 (↑)	0.1891 (↑)	0.1821 (↓)
4	0.1489	0.1381 (↓)	0.1481 (↓)	0.1558 (↑)	0.1422 (↓)
walker2d	2	0.2685	0.2655 (↓)	0.2798 (↑)	0.2745 (↑)	0.2705 (↑)
4	0.2491	0.2353 (↓)	0.2522 (↑)	0.2461 (↓)	0.2411 (↓)
halfcheetah	2	0.1562	0.1496 (↓)	0.1612 (↑)	0.1519 (↓)	0.1509 (↓)
4	0.1396	0.1530 (↑)	0.1551 (↑)	0.1484 (↑)	0.1502 (↑)
ant	2	0.1579	0.1623 (↑)	0.1743 (↑)	0.1567 (↓)	0.1374 (↓)
4	0.1585	0.1653 (↑)	0.1642 (↑)	0.1708 (↑)	0.1450 (↓)