SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Even though the testing results have been averaged over 8 seeds, it is still very volatile. Previous work often use the maximum of the mean success rate in evaluation, which is much higher than the average, since the max operation will bring much larger variances. In our experiments with 8 seeds, there is a big gap between the maximum of the mean success rate and the maximum of the smoothed (smooth factor 0.8) mean success rate, which is about  $5\% \sim 10\%$ , bringing more uncertainty to the evaluation of model performance. Predictably, this uncertainty is much bigger in the case of fewer seeds. [44, 34] tries to reduce uncertainty by evaluating the agent several times at once, but it doesn't really work. Because in the original MT10 and MT50, each task environment has a fixed initial position, and the policy based on SAC is determined during evaluating. In other words, given the parameters of the model, no matter how many times we evaluate the agent, the result is exactly the same as the first time in the testing stage. Thus, compared to the maximum of the mean success rate, we use the maximum of the smoothed mean success rate to measure the performance of models, which is more stable and realistic.
Table 1: Evaluation performance on MT10 and MT50 tasks after training for 2.4 million steps (for each task), using the metric of max success rate and max smoothed success rate, where the prior has smaller variance. Results have been averaged over 8 seeds. Our CMTA outperforms other methods on all test environments, including the Single-SAC which is considered as the upper bound. TE = Task Encoder.
agent	MT10-Fixed	MT10-Mixed	MT50-Fixed	MT50-Mixed
success rate	success rate	success rate	success rate
max smoothed	max	max smoothed	max	max smoothed	max	max smoothed	max
MT-SAC	62.25%	68.75%	53.22%	62.50%	50.37%	52.50%	28.78%	31.50%
MT-SAC+TE	64.76%	70%	61.12%	68.75%	52.45%	54.75%	37.59%	40%
MTMH-SAC	65.21%	70%	62.06%	67.50%	47.67%	48.75%	39.65%	42.75%
SoftModu	51%	55%	51.34%	58.75%	26.23%	28.75%	21.50%	23.50%
CARE	68.03%	75%	61.35%	67.50%	55.47%	57.50%	45.00%	48.50%
CMTA(ours)	78.95%	83.75%	82.07%	87.5%	68.90%	71.00%	71.69%	74.5%
Single-SAC(upper bound)	64.33%	68.75%	71.11%	76.25%	/	/	/	/
To evaluate the performance, we compare CMTA with other baselines on MT10-Fixed, MT10-Mixed, MT50-Fixed and MT50-Mixed environments. The training curves averaged over 8 seeds are plotted in Figure 3. And Table 1 records the best evaluation results of all the methods, including both max smoothed (with smooth factor 0.8, details can be found in APPENDIXC) mean success rate and max mean success rate (large variance). It is noteworthy that our method is even better than CARE, which uses additional prior knowledge.
As shown in Figure 3, our approach consistently outperforms baselines in all environments, whether on asymptotic performance or sample efficiency. In the original environments MT10-Fixed and MT50-Fixed, it can be seen from Table 1 that CMTA is better than other baselines around  $15\%$  and  $24\%$ , respectively. And in the more difficult environment with changing positions, MT10-Mixed and MT50-Mixed, the superiority of our method is more obvious, which is about  $34\%$  and  $60\%$  better than other baselines respectively.
When changing from Fixed environment to Mixed environment, it becomes more difficult because the agent have to deduce patterns or objectives of tasks (e.g., moving an object to a goal) rather than merely memorizing fixed positions. All the baseline methods suffered this impact in MT10-Mixed and MT50-Mixed, but our method achieves even better performance, which implies good generalization and robustness of our method.
Significantly, we notice that only our approach outperforms learning each task individually (we only conduct this experiment on MT10, because it is too slow to train 50 policies simultaneously on MT50). This evidence proves that our approach can avoid negative transfer and improve the performance of tasks by correctly combining the modules. While all the methods can take advantage of multi-task learning to compress the model and improve sample efficiency, in contrast to ours, other methods come at the cost of performance.
As mentioned before, RL experimental result has large stochasticity, and the evaluation metric used in previous works ([48, 44, 34]) is max success rate during testing, which leads to a large variance and is influenced by evaluation times. Thus we use max smoothed success rate to mitigate the effects of large variances caused by max operations. Because of these randomnesses, it is reasonable that the results are somewhat different from the previous works. However, the result of soft modularization[44] in our experiment is much worse than previous work. In addition to the number of seeds and the randomness of the RL algorithm, we guessed that the most likely reason is that SoftModu uses a loss weighting trick in the optimization, which balances the individual loss for different tasks. For the sake of fairness, we don't use any loss weighting method. The ablation study of SoftModu[44] also shows that MTMH-SAC outperforms SoftModu without balance, which is consistent with our experimental results.
Figure 4: Effectiveness of temporal information(TI) on MT10-Fixed and MT10-Mixed environment, each curve has been averaged over 8 seeds.
Table 2: Effectiveness of contrastive loss(CL) on MT10 and MT50 tasks after training for 2.4 million steps (for each task). Results have been averaged over 8 seeds.
agent	MT10-Mixed	MT50-Mixed
success rate	success rate
max smoothed	max	max smoothed	max
CARE	61.35%	67.50%	45.00%	48.50%
CARE + CL	65.24%	71.25%	47.61%	49.75%
CMTA w/o CL	79.46%	85%	62.66%	65%
CMTA(ours)	82.07%	87.5%	71.69%	74.5%
Temporal Attention. We investigate the significance of temporal information and report the comparison results in Figure 4. It can be seen that the temporal information we employed is indeed highly effective, which indicates that the negative transfer within tasks is further alleviated.
Contrastive Modules. We conducted comparative experiments to validate the effectiveness of contrastive loss, as shown in Table 8. From the results, it can be seen that the CL loss consistently improves performance in the mixed environment, especially in MT50-Mixed where our method achieves an increase in success rate of nearly 10 percentage points. Additionally, we observed that the impact of CL loss in the fixed environment is not substantial. This could be attributed to the fixed environment being overly homogeneous, which makes it prone to overfitting and fails to demonstrate the robust generalization advantage of contrastive modules.
To verify whether contrastive learning can constrain modules to learn unique functions and reduce similarity between them, we use t-SNE[40] to visualize the output embeddings of K(6 in our setting) multiple modules in Figure 5. The visualization of CMTA and CMTA without CL are obtained by multiple embeddings of modules at each time step on MT10-Fixed for 10 episodes. It can be observed that without contrastive learning constraints, the encodings of multiple modules are mixed together.
(a) CMTA w/o CL
(b) CMTA
Figure 5: t-SNE visualization of multiple modules' encodings on MT10-Fixed environment.
By using CL, the encodings of different modules become more distinct, indicating that CL indeed reduces the similarity between modules.
One limitation is the computational cost of CMTA, the contrastive learning part is proportional to  $O(n^{2})$ , where  $n$  is number of experts. If task similarity is substantial, then even with an increase in the number of tasks, computational cost won't necessarily escalate as long as the number of experts remains constant. Another limitation is the dependence on task similarity, and it's the fundamental assumption in the field of mtrl, even humans struggle to extract mutually beneficial information from entirely unrelated tasks. For instance, attempting to learn from a combination of tasks like playing Go and playing Arai games would likely yield limited benefits.
In this paper, we present contrastive modules with temporal attention, which is a generalized way of learning distinct shared modules with finer granularity for multi-task RL. CMTA has the ability to constrain the modules to be different from each other by contrastive learning and the ability to learn the best combination of distinct shared modules dynamically at each time step across tasks by using temporal attention. The module combination in CMTA varies form step to step, so that modules can be reused when it's helpful and not shared when it's conflict, alleviating the negative transfer between within the task and shows great superiority especially for complex tasks. Experiments on Meta-World show that our method outperforms all baselines, both on sample efficiency and performance, and it's the first time achieved performance surpassing that of training each task separately. As the variety of tasks increases (from MT10 to MT50, from Fixed environment to Mixed environment), the advantages of our approach become more apparent. Our method has achieved good results in multi-task RL, and we will further extend it to meta learning in future work.
This work is partially supported by the NSF of China(under Grants 61925208, 62102399, 62222214, 62002338, U19B2019), CAS Project for Young Scientists in Basic Research(YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6429-6438, 2019.
[2] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, 2016.
[3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39-48, 2015.
[4] Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
[5] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 1:539-546 vol. 1, 2005.
[6] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In International Conference on Machine Learning, 2008.
[44] Ruihan Yang, Huazhe Xu, YI WU, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4767-4777. Curran Associates, Inc., 2020.
[45] Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Yunkai Gao, Kaizhao Yuan, Rui Chen, Siming Lan, Xing Hu, Zidong Du, Xishan Zhang, Qi Guo, and Yunji Chen. Online prototype alignment for few-shot policy transfer. In International Conference on Machine Learning, 2023.
[46] Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Xing Hu, Zidong Du, Qi Guo, Rui Chen, Ling Li, and Yunji Chen. Learning controllable elements oriented representations for reinforcement learning. Neurocomputing, 549:126455, 2023.
[47] Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Xing Hu, Zidong Du, Xishan Zhang, Qi Guo, and Yunji Chen. Object-category aware reinforcement learning. ArXiv, abs/2210.07802, 2022.
[48] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 1094–1100. PMLR, 30 Oct–01 Nov 2020.
[49] Amir Roshan Zamir, Alexander Sax, Bokui (William) Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3712-3722, 2018.
A Pseudo Code
Algorithm 1 CMTA
Initialize: replay buffer  $D$  with  $\emptyset$   
Initialize: initial hidden state  $h_0$  with zero tensor  
Initialize: policy  $\pi$  with  $\phi$ , Q-function Q, task encoder  $g$ , k experts  $f^1, \dots, f^k$ , LSTM, fully connected layer  $\mathcal{W}$   
Input: state  $s_t$  for each environment, one-hot task id  $z_{\tau}$   
1: for episode  $m = 1, 2, \dots$  do  
2: for time-step  $t = 1, 2, \dots$  do  
3: for each task  $\tau_i$  do  
4:  $z_{enc}^j = f^j(s_t), \forall j \in 1, \dots, k$   
5:  $z_{task} = g(z_{\tau})$   
6:  $h_t = lstm(s_t; h_{t-1})$   
7:  $\alpha_1, \dots, \alpha_k = softmax(\mathcal{W}(h_t; z_{task}))$   
8:  $z_{enc} = \sum_{j=1}^{k} \alpha_j \cdot z_{enc}^j$   
9:  $z = z_{task}||z_{enc}$   
10: sample action  $a_t \sim \pi(\cdot|z_{task}; z_{enc})$   
11: Perform action  $a_t$ , get reward  $r_t$  and next state  $s_{t+1}$ .  
12:  $D = D \cup \{s_t, a_t, r_t, s_{t+1}, h_{t-1}, h_t, z_{\tau}\}$   
13: end for  
14: randomly sample batch from  $D$   
15: compute  $L_{contrastive}$  by Eq 3  
16: compute  $L_{actor}$  and  $L_{critic}$  by Eq9 and Eq10  
17: update k experts with  $L_{contrastive}$   
18: update  $\pi_{\phi}$  with  $L_{actor}$   
19: update all components except  $\pi_{\phi}$  with  $L_{critic}$   
20: end for  
21: end for
B Libraries
We use the following open-source libraries: MetaWorld $^3$ , MTEnv $^4$ , MTRL $^5$  [33].
C Smooth factor
$$
s m o o t h e d \_ p o i n t [ i ] = \left\{ \begin{array}{l l} s m o o t h e d \_ p o i n t [ i - 1 ] * f a c t o r + p o i n t [ i ] * (1 - f a c t o r), & \text {i f} i > 0 \\ p o i n t [ i ], & \text {i f} i = 0 \end{array} \right.
$$
D Hyperparameter Details.
<sup>3</sup>https://github.com/rlworkgroup/metaworld, commit-id:af8417bfc82a3e249b4b02156518d775f29eb289
<sup>4</sup>https://github.com/facebookresearch/mtenv
$^{5}$ https://github.com/facebookresearch/mtrl
Table 3: Hyperparameter values that are common across all the methods
Hyperparameter	Hyperparameter values
batch size	128 × number of tasks
network architecture	feedforward network
actor/critic size	three fully connected layers with 512 units
non-linearity	ReLU
policy initialization	standard Gaussian
temperature	learned and distangled with tasks
exploration parameters	run a uniform exploration policy 1500 steps
num of samples / num of train steps per iteration	1 env step / 1 training step
evaluation frequency	3000 steps
replay buffer size	5000000
policy learning rate	3e-4
Q function learning rate	3e-4
optimizer	Adam
policy learning rate	3e-4
beta for Adam optimizer for policy	(0.9, 0.999)
Q function learning rate	3e-4
beta for Adam optimizer for Q function	(0.9, 0.999)
discount	0.99
Episode length (horizon)	150
reward scale	1
Table 4: Hyperparameter values of task encoder
Hyperparameter	Hyperparameter values
task encoder train from scratch	embedding layer with dim 64 + FC 128 + FC 64 + FC 64
pretrained	pre-trained embedding layer with dim 512 + FC 128 + FC 64 + FC 64
Table 5: Hyperparameter values of Soft Modularization
Hyperparameter	Hyperparameter values
task encoder type	train from scratch
routing network size	4 layers and 4 modules per layer with dim 64
Table 6: Hyperparameter values of CARE
Hyperparameter	Hyperparameter values
task encoder type	pre-trained embedding layer
encoder size	FC 64 + FC 64
number of encoders	6
Table 7: Hyperparameter values of CMTA
Hyperparameter	Hyperparameter values
task encoder type	train from scratch
encoder size	FC 64 + FC 64
number of encoders	6
β	2500
E Additional Experiment Results
Table 8: Ablations on experts number. Evaluation performance on MT10 tasks after training for 1.0 million steps (for each task). Results have been averaged over 8 seeds.
agent	MT10-Fixed	MT10-Mixed
success rate	success rate
max smoothed	max	max smoothed	max
CMTA with 2 experts	54.26%	62.5%	52.53%	57.5%
CMTA with 4 experts	67.68%	72.5%	72.19%	76.25%
CMTA with 12 experts	71.21%	75.0%	73.18%	78.75%
CMTA with 20 experts	68.12%	72.5%	75.01%	78.75%
CMTA with 6 experts	68.25%	73.75%	78.49%	82.5%
Figure 7: t-SNE visualization of CMTA attention weights on MT10-Mixed environment.