SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {J _ {j} \left(\pi_ {\left(b , i\right)}\right) - \underset {b ^ {\prime} , i ^ {\prime}} {\min } J _ {j} \left(\pi_ {\left(b ^ {\prime} , i ^ {\prime}\right)}\right)}{\underset {b ^ {\prime} , i ^ {\prime}} {\max } J _ {j} \left(\pi_ {\left(b ^ {\prime} , i ^ {\prime}\right)}\right) - \underset {b ^ {\prime} , i ^ {\prime}} {\min } J _ {j} \left(\pi_ {\left(b ^ {\prime} , i ^ {\prime}\right)}\right)}
$$
Note that we only min and max over baseline methods presented in the table.
For PyBullet tasks, we do the same procedure but normalize by the oracle policy's performance (sees both position and velocity and has no history encoder) and the Markovian policy's performance (sees only position or velocity and has no history encoder). For both of these policies, we use what was reported from Ni et al. [50]. Note the our normalized scores differ slightly from those used in Ni et al. [50] since they normalize based on the best and worst returns of any policy; however, we believe our scheme gives a more intuitive picture of how any given policy is performing.
G.2 MSD and DMSD Results
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Fixed / Fixed	-6.14 ± 0.02	-5.76 ± 0.02	-5.75 ± 0.01	-5.69 ± 0.00	-5.76 ± 0.01	-5.75 ± 0.01	-5.73 ± 0.01	-5.83 ± 0.02
Fixed / Small	-7.51 ± 0.04	-7.56 ± 0.03	-7.29 ± 0.01	-7.37 ± 0.01	-7.33 ± 0.04	-7.37 ± 0.01	-7.32 ± 0.03	-7.39 ± 0.03
Fixed / Large	-11.39 ± 0.09	-12.52 ± 0.11	-10.87 ± 0.05	-11.44 ± 0.03	-11.61 ± 0.07	-11.48 ± 0.05	-12.50 ± 0.19	-11.52 ± 0.10
Small / Fixed	-6.26 ± 0.06	-5.80 ± 0.00	-5.92 ± 0.01	-5.95 ± 0.01	-5.93 ± 0.05	-5.89 ± 0.01	-5.92 ± 0.02	-5.91 ± 0.02
Small / Small	-7.49 ± 0.03	-7.02 ± 0.01	-7.15 ± 0.02	-7.14 ± 0.01	-7.12 ± 0.04	-7.09 ± 0.02	-7.15 ± 0.02	-7.12 ± 0.02
Small / Large	-11.18 ± 0.09	-9.82 ± 0.07	-10.01 ± 0.03	-10.88 ± 0.04	-10.43 ± 0.14	-10.42 ± 0.13	-10.43 ± 0.12	-10.07 ± 0.14
Large / Fixed	-6.78 ± 0.16	-6.08 ± 0.01	-6.28 ± 0.03	-6.27 ± 0.01	-6.27 ± 0.03	-6.23 ± 0.04	-6.25 ± 0.04	-6.28 ± 0.05
Large / Small	-7.78 ± 0.12	-7.25 ± 0.02	-7.44 ± 0.05	-7.43 ± 0.02	-7.45 ± 0.03	-7.44 ± 0.05	-7.44 ± 0.04	-7.48 ± 0.06
Large / Large	-11.12 ± 0.05	-9.44 ± 0.02	-9.67 ± 0.05	-10.37 ± 0.02	-9.66 ± 0.04	-9.68 ± 0.05	-9.70 ± 0.05	-9.69 ± 0.06
Average	-8.41	-7.92	-7.82	-8.06	-7.95	-7.93	-8.05	-7.92
Table 14: Unnormalized MSD Results.
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Fixed / Fixed	58.09 ± 1.66	93.18 ± 1.46	94.04 ± 0.83	100.00 ± 0.27	93.18 ± 1.20	93.77 ± 1.26	96.20 ± 1.22	87.16 ± 1.78
Fixed / Small	36.41 ± 5.36	29.74 ± 3.82	64.96 ± 1.38	54.90 ± 0.73	59.54 ± 5.78	54.35 ± 1.35	60.89 ± 3.48	51.84 ± 3.38
Fixed / Large	36.58 ± 2.86	0.00 ± 3.42	53.70 ± 1.71	34.92 ± 0.93	29.55 ± 2.32	33.62 ± 1.71	0.60 ± 6.25	32.51 ± 3.29
Small / Fixed	46.87 ± 5.88	89.05 ± 0.32	78.21 ± 1.31	75.81 ± 0.79	77.27 ± 4.66	81.41 ± 1.20	78.82 ± 1.44	79.64 ± 1.80
Small / Small	38.25 ± 3.44	100.00 ± 0.98	83.49 ± 3.07	84.88 ± 0.81	87.78 ± 5.31	90.66 ± 2.02	83.97 ± 2.40	87.57 ± 2.65
Small / Large	43.52 ± 2.82	87.63 ± 2.28	81.44 ± 0.82	53.21 ± 1.31	68.03 ± 4.43	68.09 ± 4.10	67.78 ± 3.84	79.57 ± 4.71
Large / Fixed	0.00 ± 15.12	63.36 ± 1.17	45.01 ± 3.18	46.37 ± 1.29	46.68 ± 3.06	49.86 ± 3.84	48.52 ± 3.69	45.03 ± 4.72
Large / Small	0.00 ± 15.75	70.44 ± 3.30	45.45 ± 6.93	45.73 ± 2.47	43.66 ± 4.47	44.71 ± 6.45	45.21 ± 5.42	39.64 ± 7.82
Large / Large	45.60 ± 1.71	100.00 ± 0.61	92.60 ± 1.49	69.88 ± 0.69	93.03 ± 1.27	92.36 ± 1.62	91.67 ± 1.68	91.95 ± 1.80
Average	33.92	70.38	70.99	62.86	66.53	67.65	63.74	66.10
Table 15: Normalized MSD Results.
Figure 10: MSD Performance Curves. Each row corresponds to a training environment, and each column corresponds to a testing environment.
Figure 11: MSD Performance Curve for Ablations. Each row corresponds to a training environment, and each column corresponds to a testing environment.
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Fixed / Fixed	-15.33 ± 0.14	-16.20 ± 0.31	-15.41 ± 0.13	-12.64 ± 0.04	-13.49 ± 0.22	-13.92 ± 0.09	-13.35 ± 0.05	-16.77 ± 0.13
Fixed / Small	-21.29 ± 0.29	-25.21 ± 0.32	-21.37 ± 0.16	-18.58 ± 0.05	-19.77 ± 0.24	-21.31 ± 0.07	-20.09 ± 0.08	-23.29 ± 0.15
Fixed / Large	-27.59 ± 0.44	-37.21 ± 0.35	-28.16 ± 0.17	-25.29 ± 0.18	-27.54 ± 0.33	-31.14 ± 0.13	-28.14 ± 0.11	-31.84 ± 0.71
Small / Fixed	-18.15 ± 0.91	-17.75 ± 0.42	-15.86 ± 0.11	-13.43 ± 0.09	-14.37 ± 0.17	-14.35 ± 0.11	-13.57 ± 0.10	-16.85 ± 0.11
Small / Small	-21.78 ± 0.14	-22.49 ± 0.34	-20.56 ± 0.16	-18.09 ± 0.04	-18.67 ± 0.17	-18.93 ± 0.10	-17.97 ± 0.07	-21.77 ± 0.10
Small / Large	-26.57 ± 0.22	-31.27 ± 0.36	-26.04 ± 0.24	-23.82 ± 0.13	-23.65 ± 0.20	-23.66 ± 0.10	-22.72 ± 0.08	-28.26 ± 0.12
Large / Fixed	-21.96 ± 0.62	-22.41 ± 0.32	-18.37 ± 0.30	-14.83 ± 0.12	-15.75 ± 0.14	-16.79 ± 0.04	-15.23 ± 0.12	-18.89 ± 0.28
Large / Small	-22.30 ± 0.44	-26.63 ± 0.39	-22.00 ± 0.24	-19.46 ± 0.08	-19.99 ± 0.15	-21.14 ± 0.07	-19.71 ± 0.12	-23.19 ± 0.32
Large / Large	-25.29 ± 0.30	-29.34 ± 0.30	-24.43 ± 0.21	-24.06 ± 0.03	-22.08 ± 0.14	-23.06 ± 0.07	-21.81 ± 0.09	-25.32 ± 0.19
Average	-22.25	-25.39	-21.36	-18.91	-19.48	-20.48	-19.18	-22.91
Table 16: Unnormalized DMSD Results.
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Fixed / Fixed	72.45 ± 1.44	63.59 ± 3.16	71.62 ± 1.30	100.00 ± 0.39	91.35 ± 2.28	86.93 ± 0.89	92.74 ± 0.55	57.75 ± 1.31
Fixed / Small	61.66 ± 3.35	16.43 ± 3.75	60.71 ± 1.80	93.01 ± 0.60	79.26 ± 2.81	61.50 ± 0.81	75.51 ± 0.97	38.55 ± 1.77
Fixed / Large	62.47 ± 2.86	0.00 ± 2.24	58.78 ± 1.11	77.38 ± 1.14	62.76 ± 2.13	39.41 ± 0.83	58.92 ± 0.73	34.84 ± 4.61
Small / Fixed	43.59 ± 9.27	47.76 ± 4.25	67.02 ± 1.10	91.92 ± 0.90	82.32 ± 1.72	82.52 ± 1.14	90.46 ± 0.99	56.98 ± 1.16
Small / Small	56.04 ± 1.57	47.82 ± 3.96	70.07 ± 1.88	98.69 ± 0.48	91.94 ± 2.00	88.95 ± 1.18	100.00 ± 0.78	56.17 ± 1.11
Small / Large	69.11 ± 1.42	38.57 ± 2.33	72.51 ± 1.58	86.96 ± 0.82	88.08 ± 1.31	87.99 ± 0.64	94.09 ± 0.51	58.08 ± 0.80
Large / Fixed	4.64 ± 6.34	0.00 ± 3.30	41.37 ± 3.09	77.62 ± 1.24	68.16 ± 1.45	57.60 ± 0.36	73.51 ± 1.24	36.06 ± 2.85
Large / Small	50.02 ± 5.07	0.00 ± 4.56	53.45 ± 2.80	82.77 ± 0.98	76.66 ± 1.75	63.36 ± 0.85	79.93 ± 1.43	39.74 ± 3.65
Large / Large	77.38 ± 1.93	51.09 ± 1.98	82.96 ± 1.38	85.37 ± 0.18	98.23 ± 0.90	91.86 ± 0.44	100.00 ± 0.56	77.22 ± 1.21
Average	55.26	29.47	64.28	88.19	82.08	73.35	85.02	50.60
Table 17: Normalized DMSD Results.
Figure 12: DMSD Performance Curves. Each row corresponds to a training environment, and each column corresponds to a testing environment.
Figure 13: DMSD Performance Curve for Ablations. Each row corresponds to a training environment, and each column corresponds to a testing environment.
G.3 Navigation Results
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Sim / Sim	28.94 ± 3.63	96.76 ± 0.15	99.57 ± 0.12	98.33 ± 0.06	100.00 ± 0.07	99.64 ± 0.06	99.66 ± 0.06	99.81 ± 0.09
Sim / Real	43.12 ± 2.08	0.00 ± 3.94	50.55 ± 0.78	68.34 ± 0.57	62.16 ± 0.89	63.17 ± 0.57	59.21 ± 1.15	52.52 ± 0.50
Real / Sim	0.00 ± 4.09	57.49 ± 1.17	68.03 ± 0.40	59.54 ± 0.85	74.88 ± 0.61	72.84 ± 0.64	74.75 ± 0.68	71.13 ± 0.72
Real / Real	67.28 ± 2.05	97.29 ± 0.20	99.20 ± 0.14	95.94 ± 0.04	100.00 ± 0.21	99.19 ± 0.09	99.11 ± 0.21	99.67 ± 0.17
Average	34.83	62.89	79.34	80.54	84.26	83.71	83.18	80.78
Table 18: Normalized Navigation Results. Note that these results are after 1 million collected samples.
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Sim / Sim	-17.23 ± 0.18	-13.82 ± 0.01	-13.68 ± 0.01	-13.74 ± 0.00	-13.65 ± 0.00	-13.67 ± 0.00	-13.67 ± 0.00	-13.66 ± 0.00
Sim / Real	-23.87 ± 0.29	-29.85 ± 0.55	-22.84 ± 0.11	-20.37 ± 0.08	-21.23 ± 0.12	-21.09 ± 0.08	-21.64 ± 0.16	-22.57 ± 0.07
Real / Sim	-18.69 ± 0.21	-15.79 ± 0.06	-15.26 ± 0.02	-15.69 ± 0.04	-14.92 ± 0.03	-15.02 ± 0.03	-14.93 ± 0.03	-15.11 ± 0.04
Real / Real	-20.52 ± 0.28	-16.36 ± 0.03	-16.09 ± 0.02	-16.55 ± 0.01	-15.98 ± 0.03	-16.09 ± 0.01	-16.11 ± 0.03	-16.03 ± 0.02
Average	-20.08	-18.96	-16.97	-16.59	-16.45	-16.47	-16.59	-16.84
Table 19: Unnormalized Navigation Results. Note that these results are after 1 million collected samples.
Figure 14: Navigation Performance Curves. Each row corresponds to a training environment, and each column corresponds to a testing environment. Note that these runs are only done for one million transitions.
Figure 15: Navigation Performance Curve for Ablations. Each row corresponds to a training environment, and each column corresponds to a testing environment. Note that these runs are only done for one million transitions.
G.4 Tokamak Control Results
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Sim / Sim	90.95 ± 0.05	100.00 ± 0.03	99.63 ± 0.03	84.74 ± 0.16	99.75 ± 0.06	99.91 ± 0.02	99.90 ± 0.02	99.47 ± 0.04
Sim / Real	89.15 ± 0.99	40.96 ± 5.45	40.05 ± 11.91	0.00 ± 21.04	55.21 ± 4.44	61.56 ± 7.40	65.65 ± 5.66	35.66 ± 4.41
Real / Sim	50.62 ± 3.96	36.33 ± 3.61	35.26 ± 2.22	0.00 ± 3.48	48.40 ± 4.04	52.62 ± 1.38	56.30 ± 2.25	16.33 ± 5.98
Real / Real	98.45 ± 0.77	98.24 ± 0.38	98.74 ± 0.29	100.00 ± 0.23	99.30 ± 0.64	98.39 ± 0.33	98.55 ± 0.33	98.27 ± 0.37
Average	82.29	68.88	68.42	46.18	75.67	78.12	80.10	62.43
Table 20: Normalized  ${\beta }_{N}$  Tracking Results.
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Sim / Sim	-8.09 ± 0.00	-7.19 ± 0.00	-7.22 ± 0.00	-8.71 ± 0.02	-7.21 ± 0.01	-7.19 ± 0.00	-7.20 ± 0.00	-7.24 ± 0.00
Sim / Real	-16.41 ± 0.30	-31.21 ± 1.67	-31.49 ± 3.66	-43.78 ± 6.46	-26.83 ± 1.36	-24.88 ± 2.27	-23.63 ± 1.74	-32.83 ± 1.35
Real / Sim	-12.12 ± 0.40	-13.55 ± 0.36	-13.66 ± 0.22	-17.18 ± 0.35	-12.34 ± 0.40	-11.92 ± 0.14	-11.55 ± 0.22	-15.55 ± 0.60
Real / Real	-13.56 ± 0.23	-13.62 ± 0.12	-13.47 ± 0.09	-13.08 ± 0.07	-13.30 ± 0.20	-13.58 ± 0.10	-13.53 ± 0.10	-13.61 ± 0.11
Average	-12.55	-16.39	-16.46	-20.69	-14.92	-14.39	-13.98	-17.31
Table 21: Unnormalized  ${\beta }_{N}$  Tracking Results.
Figure 16:  $\beta_N$  Tracking Performance Curves. Each row corresponds to a training environment, and each column corresponds to a testing environment.
Figure 17:  $\beta_{N}$  Tracking Performance Curve for Ablations. Each row corresponds to a training environment, and each column corresponds to a testing environment.
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Sim / Sim	46.78 ± 0.44	99.50 ± 0.12	97.99 ± 0.50	82.96 ± 0.27	100.00 ± 0.15	99.64 ± 0.19	99.97 ± 0.12	96.18 ± 1.35
Sim / Real	83.48 ± 2.63	39.65 ± 5.83	33.22 ± 0.69	0.00 ± 8.87	50.86 ± 1.92	54.36 ± 2.07	52.56 ± 2.44	42.51 ± 2.97
Real / Sim	0.00 ± 8.79	21.31 ± 2.45	7.23 ± 3.86	22.49 ± 1.84	19.02 ± 3.88	22.70 ± 4.42	5.20 ± 20.06	15.35 ± 8.29
Real / Real	91.76 ± 0.84	98.07 ± 0.52	96.05 ± 0.31	97.94 ± 0.23	99.73 ± 0.46	97.62 ± 0.46	100.00 ± 0.28	96.33 ± 0.47
Average	55.51	64.63	58.62	50.85	67.40	68.58	64.43	62.59
Table 22: Normalized  ${\beta }_{N}$  -Rotation Tracking Results.
PID Controller	GRU	Transformer	PIDE	GPIDE	GPIDE-ES	GPIDE-ESS	GPIDE-Attn
Sim / Sim	-27.56 ± 0.08	-18.53 ± 0.02	-18.79 ± 0.09	-21.36 ± 0.05	-18.45 ± 0.03	-18.51 ± 0.03	-18.45 ± 0.02	-19.10 ± 0.23
Sim / Real	-30.08 ± 0.95	-45.91 ± 2.10	-48.23 ± 0.25	-60.23 ± 3.20	-41.86 ± 0.69	-40.60 ± 0.75	-41.25 ± 0.88	-44.88 ± 1.07
Real / Sim	-35.57 ± 1.50	-31.92 ± 0.42	-34.33 ± 0.66	-31.72 ± 0.32	-32.31 ± 0.66	-31.68 ± 0.76	-34.68 ± 3.43	-32.94 ± 1.42
Real / Real	-27.09 ± 0.30	-24.81 ± 0.19	-25.54 ± 0.11	-24.86 ± 0.08	-24.21 ± 0.16	-24.98 ± 0.17	-24.12 ± 0.10	-25.44 ± 0.17
Average	-30.08	-30.29	-31.72	-34.54	-29.21	-28.94	-29.62	-30.59
Table 23: Unnormalized  ${\beta }_{N}$  -Rotation Tracking Results.
Figure 18:  $\beta_{N}$ -Rotation Tracking Performance Curves. Each row corresponds to a training environment, and each column corresponds to a testing environment.
Figure 19:  $\beta_{N}$ -Rotation Tracking Performance Curve for Ablations. Each row corresponds to a training environment, and each column corresponds to a testing environment.
G.5 PyBullet Results
For these results, SAC encodes observations, actions and rewards. TD3 encodes observations and actions since it is the best performing on average.
PPO-GRU	A2C-GRU	SAC-LSTM	TD3-GRU	VRM	SAC-Transformer	SAC-GPIDE	SAC-GPIDE-ES	SAC-GPIDE-ESS	SAC-GPIDE-Attn
HalfCheetah-P	27.09 ± 7.85	-22.00 ± 5.13	77.06 ± 7.96	85.80 ± 5.15	-107.00 ± 1.39	37.00 ± 9.97	82.63 ± 3.46	85.45 ± 4.83	71.08 ± 6.95	77.63 ± 4.60
Hopper-P	49.00 ± 5.22	-2.29 ± 3.33	64.36 ± 9.94	84.63 ± 8.33	3.53 ± 1.63	59.54 ± 19.64	93.27 ± 13.56	111.48 ± 4.20	113.95 ± 4.67	104.87 ± 8.76
Walker-P	1.67 ± 4.39	-10.48 ± 1.73	40.92 ± 15.56	29.08 ± 9.67	-3.89 ± 1.25	24.89 ± 14.80	96.61 ± 1.60	76.58 ± 5.47	94.58 ± 11.00	71.36 ± 6.57
Ant-P	39.48 ± 3.74	-13.06 ± 6.52	60.97 ± 3.54	-36.36 ± 3.35	-36.39 ± 0.17	-10.57 ± 2.34	66.66 ± 2.94	64.73 ± 3.82	57.78 ± 3.78	63.19 ± 5.32
HalfCheetah-V	19.68 ± 11.71	-50.13 ± 9.50	18.54 ± 33.09	59.03 ± 2.88	-80.49 ± 2.97	-41.31 ± 26.15	20.39 ± 29.60	51.03 ± 13.93	53.14 ± 5.86	-54.70 ± 19.89
Hopper-V	13.86 ± 4.80	-0.60 ± 3.33	16.26 ± 12.44	57.43 ± 8.63	10.08 ± 3.51	0.28 ± 8.49	90.98 ± 4.28	72.63 ± 19.28	90.09 ± 2.50	30.73 ± 1.60
Walker-V	8.12 ± 5.43	-8.02 ± 0.57	-1.57 ± 1.88	-4.63 ± 1.30	-1.80 ± 0.70	-8.21 ± 1.31	36.90 ± 16.59	68.30 ± 4.33	67.54 ± 3.60	14.85 ± 11.26
Ant-V	1.43 ± 3.26	-13.67 ± 1.83	-16.95 ± 1.29	17.03 ± 6.55	-13.41 ± 0.12	0.81 ± 1.31	18.03 ± 5.10	4.56 ± 5.20	12.85 ± 1.67	-1.84 ± 5.76
Average	20.04	-15.03	32.45	36.50	-28.67	7.80	63.18	66.84	70.13	38.26
Table 24: Normalized PyBullet Scores.
PPO-GRU	A2C-GRU	SAC-LSTM	TD3-GRU	VRM	SAC-Transformer	SAC-GPIDE	SAC-GPIDE-ES	SAC-GPIDE-ESS	SAC-GPIDE-Attn
HalfCheetah-P	1445.81 ± 166.79	403.35 ± 108.97	2506.88 ± 168.93	2692.53 ± 109.43	-1401.67 ± 29.62	1656.13 ± 211.75	2625.13 ± 73.49	2684.98 ± 102.57	2379.79 ± 147.67	2519.06 ± 97.72
Hopper-P	1436.43 ± 102.09	433.19 ± 65.09	1736.81 ± 194.51	2133.42 ± 162.93	546.93 ± 31.81	1642.63 ± 384.10	2302.31 ± 265.21	2658.48 ± 82.18	2706.81 ± 91.39	2529.31 ± 171.41
Walker-P	501.06 ± 76.99	288.10 ± 30.39	1189.28 ± 272.77	981.63 ± 169.46	403.60 ± 21.85	908.17 ± 259.52	2165.52 ± 28.10	1814.40 ± 95.91	2129.91 ± 192.87	1722.81 ± 115.22
Ant-P	2025.52 ± 84.58	837.57 ± 147.53	2511.54 ± 80.13	310.72 ± 75.68	310.24 ± 3.83	893.84 ± 52.83	2640.16 ± 66.46	2596.63 ± 86.26	2439.48 ± 85.37	2561.67 ± 120.21
HalfCheetah-V	1005.13 ± 289.84	-723.40 ± 235.29	977.02 ± 819.24	1979.56 ± 71.40	-1475.15 ± 73.42	-505.00 ± 647.43	1022.93 ± 732.93	1781.36 ± 344.95	1833.60 ± 145.14	-836.47 ± 492.45
Hopper-V	534.05 ± 105.85	215.22 ± 73.48	587.10 ± 274.42	1495.11 ± 190.42	450.77 ± 77.35	234.49 ± 187.36	2235.02 ± 94.45	1830.26 ± 425.16	2215.47 ± 55.16	906.05 ± 35.29
Walker-V	377.80 ± 109.11	53.25 ± 11.45	182.97 ± 37.89	121.44 ± 26.14	178.28 ± 14.09	49.32 ± 26.43	956.43 ± 333.46	1587.56 ± 87.15	1572.41 ± 72.46	513.07 ± 226.34
Ant-V	684.36 ± 89.48	269.32 ± 50.35	178.98 ± 35.57	1113.19 ± 179.93	276.33 ± 3.18	667.20 ± 35.98	1140.73 ± 140.22	770.51 ± 143.02	998.35 ± 46.04	594.54 ± 158.48
Average	1001.27	222.08	1233.82	1353.45	-88.84	693.35	1886.03	1965.52	2034.48	1313.76
Table 25: Unnormalized PyBullet Scores.
Figure 20: PyBullet Performance Curves.
Figure 21: PyBullet Performance Curve for Ablations.
Interestingly, we found that GPIDE policies often outperform the oracle policy on Hopper-P. While the oracle performance here was taken from Ni et al. [50], we confirmed this also happens with our own implementation of an oracle policy. We hypothesize that this may be due to the fact the GPIDE policy gets to see actions and rewards and the oracle does not.
G.6 Attention Scheme Visualizations
We generate the attention visualizations (as seen in Figure 4) by doing a handful of rollouts with a GPIDE policy using only attention heads. During this rollout we collect all of the weighting schemes, i.e. softmax  $\left(\frac{q_{1:t}k_{1:t}^T}{\sqrt{D}}\right)$ , generated throughout the rollouts and average them together. Below, we show additional attention visualizations. In all figures, each plot shows one of the different six heads. For each of these, the policies were evaluated on the same version of the environment they were trained on.
Figure 22: MSD-Fixed Attention.
Figure 23: MSD-Small Attention.
Figure 24: MSD-Large Attention.
Figure 25: DMSD-Fixed Attention.
Figure 26: DMSD-Small Attention.
Figure 27: DMSD-Large Attention.
Figure 28: Navigation No Friction Attention.
Figure 29: Navigation Friction Attention.
Figure 30:  $\beta_{N}$  Tracking Sim Attention.
Figure 31:  $\beta_{N}$  Tracking Rotation Attention.
Figure 32:  $\beta_{N}$ -Rotation Tracking Sim Attention.
Figure 33:  $\beta_{N}$ -Rotation Tracking Rotation Attention.
Figure 34: HalfCheetah-P Attention.
Figure 35: HalfCheetah-V Attention.
Figure 36: Hopper-P Attention.
Figure 37: Hopper-V Attention.
Figure 38: Walker-P Attention.
Figure 39: Walker-V Attention.
Figure 40: Ant-P Attention.
Figure 41: Ant-V Attention. Note that total path length is less than 64 here since the agent falls down pretty fast.