SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We have added visualizations of task representations for CSRO, FOCAL [20], and CORRO [37] in the Point-Robot environment, as shown in Figure 9. We can see that FOCAL is worse. In CORRO, similar tasks are closer, and many points of the same task are far apart.
Figure 8: Visualization of contexts and trajectory after using the non-prior context collection strategy in the Point-Robot environment.
Figure 9: t-SNE visualization of the learned task representation space in Point-Robot. Each point represents a context sampled by the non-prior context collection strategy from test tasks. The direction of the goal of the task is from 0 degrees to 180 degrees, from red to purple.
F.5 Additional Exploration Policy
In Section 4.3, we discussed how current online meta-RL attempts to train an exploration policy for collecting context, an approach that isn't suitable for addressing the context shift problem in offline meta-RL. Here, we made an attempt to train an exploration policy in MetaCURE [38] using offline data, and the experimental results are presented in Table 6.
Observably, there is a substantial drop in experimental performance when using MetaCURE's exploration policy. This is because, in an offline setting, the exploration policy cannot explore the environment, and training the exploration policy with offline data confines it to the vicinity of that data. Consequently, during testing, the exploration policy becomes ineffective when encountering previously unobserved states.
Table 6: Training MetaCURE's exploration policy with offline datasets for the purpose of collecting context during testing.
Env	Point-Robot	Half-Cheetah-Vel
CSRO	-6.4±0.8	-48.4±3.9
CSRO+MetaCURE	-13.2±2.3	-87.9±8.9
F.6 Compare with IDAQ
We compare our work with IDAQ [31], which is a recent study addressing the context shift issue. IDAQ uses the same training approach as FOCAL, but it tackles this problem by attempting to collect context during testing that closely matches the distribution seen during training. As depicted in Table 7, the experimental results reveal that the performance of CSRO is comparable to that of IDAQ. CSRO exhibits slightly lower performance on the Point-Robot compared to IDAQ, but it significantly outperforms IDAQ on the Half-Cheetah-Vel and Walker-Rand-Params tasks.
Table 7: CSRO and IDAQ compare performance on three environments.
Env	Point-Robot	Half-Cheetah-Vel	Walker-Rand-Params
CSRO	-6.4±0.8	-48.4±3.9	344.2±38.0
IDAQ	-5.2±0.1	-60.9±6.5	297.0±23.6
Furthermore, we also compared the performance of several methods in sparse settings, as shown in Table 8. It can be observed that, except for IDAQ, all other methods yield poor results. For CSRO, with a non-prior context collection strategy, it becomes challenging to obtain context with rewards and infer the environment, rendering it ineffective.
Table 8: Compared the performance of several methods in Sparse-Point-Robot.
Method	CSRO	FOCAL	OffPearl	IDAQ
Sparse-Point-Robot	0.8±0.1	0.8±0.1	0.6±0.1	6.5±0.3
We conduct a detailed analysis of IDAQ's performance in the Sparse-Point-Robot. First, we introduce the configuration of the Sparse-Point-Robot environment: The task's goals are randomly distributed along a semicircular arc with a radius of 1 from the starting point. We select 30 goals from this distribution for training tasks and 10 goals for testing tasks. The agent only receives a non-zero reward when its distance to the goal is less than 0.2. This implies that, on average, there are four training goals within a radius of 0.2 for each tested goal.
Next, we will analyze IDAQ in Sparse-Point-Robot: IDAQ retains all the  $z$  from the training tasks and maintains a used  $z$ -set during the testing phase. During the prior exploration phase, IDAQ samples a  $z$  from the training tasks which has the maximum distance from the used  $z$ -set (the difference between the direction of movement of sampled  $z$  and used  $z$ -set is the largest, and adds it to the used  $z$ -set) to gather context. If the accumulated reward from the collected context surpasses that of all previously collected contexts, the current context is retained; otherwise, it is discarded. IDAQ samples the  $z$  with the maximum distance a total of 10 times.
This signifies that the agent explores 10 directions of training goals which have large differences in direction, effectively covering a substantial portion of the entire space. Considering that there are 4 training goals near each test goal, this implies that the sampling can capture the vicinity of the test goal, enabling the agent to attain a non-zero reward. By retaining only high-reward contexts, this indicates that after the prior exploration phase, the remaining contexts exhibit a small context shift and offer non-zero rewards (containing environmental information). As a result, during the subsequent posterior sampling, IDAQ can update its belief and make improved inferences about the testing task.
Based on the aforementioned analysis, we can identify the key environmental factors under which IDAQ operates effectively in the Sparse-Point-Robot: presence of training goals within a radius of 0.2 from the testing goal is crucial, without this condition, the prior exploration process may encounter difficulty in collecting rewarding contexts. To validate this hypothesis, we modified the configuration of the Sparse-Point-Robot environment, resulting in three distinct setups: Setting A reduces the radius
for obtaining rewards from 0.2 to 0.05; Setting B removes the four training goals near the testing goal. Setting C introduces out-of-distribution (OOD) tasks, where training goals are located within the arc of  $[0, \frac{3}{4}\pi)$ , while testing goals are situated in the interval  $[\frac{3}{4}\pi, \pi]$ . In these three setups, there are no training goals within the reward radius of the testing goals. We conducted performance testing of IDAQ under these three setups, as shown in Table 9 above. Significant performance degradation can be observed, confirming our conjecture.
Table 9: Comparing IDAQ's performance on the Sparse-Point-Robot under various settings.
Sparse-Point-Robot	Setting A	Setting B	Setting C
IDAQ	6.5±0.3	0.4±0.1	1.1±0.8	1.3±0.2
Hence, we can conclude that for sparse environments, IDAQ is suitable when both the training tasks and testing tasks follow independent and identically distributed (IID). In addition, it is essential for the reward range of testing tasks to encompass certain training tasks.
Overall, both CSRO and IDAQ aim to address the issue of context shift in offline meta RL. CSRO's goal is to acquire a more essential context encoder and reduce the shift during prior exploration. IDAQ, on the other hand, aims to collect a context during the testing phase that shares the same distribution as that of the training phase. Hence, both methods exhibit certain limitations. For instance, CSRO exhibits poor performance in sparse environments; IDAQ needs more exploration steps and has limitations in its applicability to certain sparse environments.