SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 8: Pearson correlation coefficients among variables: a) on raw data, b) on weighted data, and c) on saliency-guided weighted data. The variables  $S_{8}$ ,  $S_{9}$ , and  $S_{10}$  are the ones that have been changed among the environments.
C.1 Additional Correlation Studies on the Weighted Samples
To further evaluate the decorrelation capability of our model, we implement the 'Pushing' task in Causal World and allow the model to access state features that have physical meaning. We record the Pearson correlation coefficients among features under three conditions: a) on raw data, b) on weighted data, and c) on saliency-guided weighted data (SGFD). Among these features, we designate the  $S_{8}$ ,  $S_{9}$ , and  $S_{10}$ , which represent the length, width, and height, as the features of interest for environmental variations. Therefore, when the model is required to push an object with a new size, the correlation between size and other features needs to be carefully considered. As shown in Figure 8, due to limited samples and strong correlation between features, the effectiveness of directly decorrelating all features is greatly reduced. In contrast, SGFD is capable of effectively reducing the correlation between the size features ( $S_{8}$ ,  $S_{9}$ , and  $S_{10}$ ) and other features.
C.2 Additional Ablation Studies
We performed additional ablation experiments to test the effect of RFF and the saliency-guided model. As depicted in Figure 9 (a) and Figure 9 (b), when RFF is removed, the generalization performance drops significantly. In contrast, the global decorrelation can cause over-adjusting of samples weighting and impair the performance, as demonstrated by the reward curves in Figure 9 (c). Due to the need to eliminate correlations between variables using limited samples, the model may assign very low weights to certain training samples. However, this can potentially harm the efficiency of training. Thanks to the saliency-guided model, SGFD balances both efficiency and decorrelation.
C.3 Evaluation on the Classification Model
We recorded the converge curve of the classifier during training. The classifier starts to update when the replay buffer collects 1e5 steps. To avoid overfitting, we stopped to update the classifier when the accuracy exceeded 0.9. As shown in Figure 10, the classifier converged quickly and achieved the accuracy of 0.9, which means that the classifier does not take up many computing resources.
Figure 9: Ablation studies. (a)-(b) The performance difference of ablation studies on the RFF. (c)-(d) The performance difference of ablation studies on the saliency-guided model. The x-axis denotes the steps in the training environments. The y-axis denotes the cumulative reward recorded in the testing environment with background noises (walker-run) or varied robot parameters (MT-walker-walk).
Figure 10: The accuracy of the classifier during training. The x-axis denotes the number of training steps taken in training environments, while the y-axis indicates the accuracy. The classifier starts to update when the replay buffer collects 1e5 steps. The red line corresponds to the 0.9 threshold, which indicates that the classifier stops updating if its accuracy exceeds 0.9.
C.4 Saliency Maps on the Policy Models
An intuitive approach to explaining visually-based models involves identifying pixels that have a significant impact on the final decision [1]. To determine whether the model focuses on the object or the background during decisions, we visualize the gradient of the action with respect to the input pixels. Experiments are conducted in environments with varying backgrounds and robot parameters, as depicted in Figure 11. Saliency maps of the baseline models reveal that various backgrounds draw considerable attention from the policy model while failing to make decisive contributions to our model. Moreover, SGFD can notice the changing aspects of the robot's body that other baseline models overlook. As a result, the saliency maps demonstrate that the decorrelation benefits the model's generalization in both task-relevant and task-irrelevant situations.
C.5 Additional Visualization of the Weighted Samples
In this experiment, we significantly expanded the dataset to include 300 state instances, which is substantially larger than the 20 states mentioned in Section 5.3. The identified states were also sorted into three groups in each environment, each consisting of multiple similar state instances. Moreover, we carefully analyzed the distribution of these states within each group, and the corresponding proportions are visually depicted in Figure 12. As illustrated in Figure 12 (a), the green dotted boxes indicate that  $50\%$  of the states in environment A and  $35.4\%$  of the states in environment B exhibited similarities. After the reweighting process, both environments A and B had approximately  $42\%$  similar states, effectively neutralizing the correlation between background and body posture. A similar outcome was observed in Figure 12 (b), wherein the reweighting process endeavored to balance the number of states in each group across environments with varied body lengths.
SGON
Figure 11: Visualization of saliency maps produced by the SGFD and other baselines when the robot with different backgrounds and robot parameters. The brighter the pixel is, the more contributions it makes to the decision.
C.6 Evaluation on the Generalization of SGFD
Evaluation on Changed Task-Irrelevant Features. We report the training curves of SGFD in environments with different task-irrelevant features, where the end points of the curves correspond to the results in Table 1. As illustrated in Figure 13, our model achieved superior performance compared to other baselines in previously unseen background environments. As the task becomes difficult (Figure 13 (b) and Figure 13 (d)), the advantage of SGFD achieves more significance. The results demonstrate that the decorrelation is suitable for RL tasks with different task-irrelevant features, and SGFD exhibits strong generalization capabilities to new values of task-irrelevant features.
Evaluation on Changed Task-Relevant Features. We also report the training curves of SGFD in environments with different task-relevant features, where the end points of the curves correspond to the results in Table 2. As shown in Figure 14, the reward curve exhibited a consistent positive correlation between the number of training steps and the average reward in the testing environment under both interpolation and extrapolation setups. Generally speaking, the extrapolation setting is more challenging for generalization because it requires the model to fully understand how changed features affect optimal decisions. Benefiting from decorrelation, our model achieved superior performance compared to other baselines, where the gaps were more significant under the extrapolation setting, as illustrated in Figure 14 (f). Our results demonstrate that the decorrelation facilitates our model's understanding of the true associations between changed features and decisions, leading to strong generalization on new values of task-relevant features.
(b) 300 states from environments with different robot parameters
Figure 12: Visualization of weighted samples obtained from different environments with background noises and robot parameters. States that belong to the same dotted box with a specific color are considered semantically similar. SGFD balances the weights of samples to eliminate the correlations between changed features and other features. As shown in (a), the walker agents in Env A and Env B have different backgrounds, entangled with different distributions of the body posture. SGFD reduces the weighting of states with backgrounds and enhances the weighting of states without backgrounds, effectively neutralizing the correlation between background and body posture. A comparable result was observed in (b), where the reweighting process aimed to equalize the number of states in each group across environments with different body lengths.
Figure 13: Generalization to changed task-irrelevant features. The x-axis denotes the number of training steps taken in training environments, while the y-axis indicates the average reward in the testing environment under different backgrounds. The reward curve demonstrates that our SGFD model generalizes well to new values of task-irrelevant features.
Figure 14: Generalization to changed task-relevant features. The x-axis denotes the number of training steps taken in training environments, while the y-axis represents the average reward in the testing environment with different robot parameters. We consider two setups for evaluation: an interpolation setup ((a)-(e)) and an extrapolation setup ((f)-(j)), where the changes in the task-relevant features are interpolations and extrapolations between the changes in the training environments, respectively. The reward curve shows that our SGFD generalizes well to the new values of task-relevant features.