Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning
Authors: Siming Lan
Rui Zhang
Qi Yi
Jiaming Guo
Shaohui Peng
Yunkai Gao
Fan Wu
Ruizhi Chen
Zidong Du
Xing Hu
Xishan Zhang
Ling Li
Yunji Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
multi-task learning
contrastive learning

Abstract: In the field of multi-task reinforcement learning, the modular principle, which involves specializing functionalities into different modules and combining them appropriately, has been widely adopted as a promising approach to prevent the negative transfer problem that performance degradation due to conflicts between tasks. However, most of the existing multi-task RL methods only combine shared modules at the task level, ignoring that there may be conflicts within the task. In addition, these methods do not take into account that without constraints, some modules may learn similar functions, resulting in restricting the model's expressiveness and generalization capability of modular methods.
In this paper, we propose the Contrastive Modules with Temporal Attention(CMTA) method to address these limitations. CMTA constrains the modules to be different from each other by contrastive learning and combining shared modules at a finer granularity than the task level with temporal attention, alleviating the negative transfer within the task and improving the generalization ability and the performance for multi-task RL.
We conducted the experiment on Meta-World, a multi-task RL benchmark containing various robotics manipulation tasks. Experimental results show that CMTA outperforms learning each task individually for the first time and achieves substantial performance improvements over the baselines.

Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning
Siming Lan 1,2,3
Rui Zhang
Yi Jiaming Guo
Shaohui Peng
Yunkai Gao 1,233
Fan Wu
Ruizhi Chen Zidong Du Xing Hu
Xishan Zhang Ling Li45 Yunji 24 24
University of Science and Technology of China
2 State Key Lab of Processors, Institute of Computing Technology,
Chinese Academy of Sciences, Beijing, China Cambricon Technologies
University Chinese Academy of Sciences, UCAS, Beijing, China
5 Intelligent Software Research Center, Institute of Software, CAS, Beijing, China
Shanghai Innovation Center for Processor Technologies, SHIC, Shanghai, China
{lansm, [1ansm,gyk314, yiqilmmaaiimmmmlml....m ustu..uu.... edu.
(changruiguu.jkaminngg duzidong. huxing, zhangxishan, cy..aict.acc wwt.haaohhuuuuuaaa22222222 wufan2020, ruizhi, laiinnnisscsess ac.
{pengshaohui,
Abstract
In the field of multi-task reinforcement learning, the modular principle, which involves specializing functionalities into different modules and combining them appropriately, has been widely adopted as promising approach to prevent the negative transfer problem that performance degradation due to conflicts between tasks. However, most of the existing multi-task RI methods only combine shared modules at the task level, ignoring that there may be conflicts within the task. addition, these methods do not take into account that without constraints, some modules may learn similar functions, resulting in restricting the model's expressive- ness and generalization capability of modular methods. In this paper, propose the Contrastive Modules with Temporal AttentionCCMTA) method to address these limitations. CMTA constrains the modules to be different from each other by contrastive learning and combining shared modules at finer granularity than the task level with temporal attention, alleviating the negative transfer within the task and improving the generalization ability and the performance for multi-tase RL. We conducted the experiment Meta- World, multi-task RI. benchmark containing various robotics manipulation tasks. Experimental results show that CMTA outperforms learning each task individually for the first time and achieves substantial performance improvements over the baselines, Our code can be found https://gihhub.cc///iiieeng/g//
1 Introduction
Though deep reinforcement learning (RL) has made significant progress in multiple domains such playing Atari games [22] and robotic control[ 16,, most of these methods tackle different tasks in isolation, making difficult to leverage the previously learned skills for a new task. Multi-task RL aims to train multiple tasks simultaneously in sample efficient manner [6] by using shared
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and re-used components across different tasks. Since different tasks may have complementary information or act as a regularizer for each other, multi-task RL methods have the potential to get better performance and generalization [4].
There is vital challenge in multi-task RI. if two tasks essentially unrelated have conflicts, increasing the performance of the method on one task will hurt the performance on the other, referred as negative transfer problem [42, 37j. The occurrence of negative transfer is attributed to the utilization of the same model to learning different tasks or different functions. When conflicts arise, the model needs to compromise among multiple tasks, which often results in inferior performance compared to training each task individually. To tackle this problem, researchers propose to exploit the task relationships [35, 34] along with basic shared models, including task- task-specific decoding head[ [7], routing networks [10, 29], cross-stitch networks [21], compositional modules[8, 44, 2] and mixture of experts[9, 20, 34J. These methods use different combinations of shared modules or different parts of the shared model according to different tasks, which can be seen as an embodiment of modular approaches.
The main idea of modular methods is to separate and specialize the functionalities into different modules, allowing them to focus on their respective skills features without interference or con- flicting objectives. Different modules can collaborate and complement each other's strengths. By appropriately combining these distinct modules, it is theoretically possible to resolve the trade-off dilemma encountered or negative transfer in multi-task RL and enhance the model's expressive power and generalization ability. And it has been observed in neuroscience research that different regions of the brain possess completely different functions, aligning with the concept of modularity. This may be one of the reasons why humans are capable of rapidly learning multiple tasks.
Despite the widespread adoption of modular principle in most multi-task RL methods, these methods still underperform compared to single-task training. We argue that the inferior performance of these works may be attributed to the fact that they have not fully adhered to the principle of modularity, which hinders their ability mitigate negative transfer. One of the key aspects of modularity is ensuring that different modules specialize in distinct functionalities, but these methods only partition the network into multiple modules superficial way without further constraints which may result in some modules learning the same functionality. This goes against the modular principle and restricts the model's expressiveness and generalization capability of modular methods and leads to redundant computations and even overfitting. Another challenge lies in the appropriate combination of modules. Most methods only combine shared modules at the task level, using fixed combination of modules for each task during the whole episode. However, negative transfer exists not only between tasks but also within tasks. For instance, considering the task 'Push' in Meta-Worl (orld 1), the agent needs to reach the puck at first and then push it to a goal, these two purposes are not identical and not always be mutually supportive, so negative transfer may exist when using a same module combination at these time steps.
In this paper. we propose a novel approach called Contrastive Modules with Temporal Attention (CMTA) to address the limitations of existing multi-task RL methods and further alleviate negative transfer in multi-task RL. The key insight lies in constraining the modules to be different from each other by contrastive learning and combining shared modules at finer granularity than the task level with temporal attention. Roughly speaking. CMTA uses LSTM to encode the temporal information, along with task encoder to encode the task information. Then CMTA takes both temporal and task information to compute the soft attention weights for the outputs(encodings) of modules, and we apply contrastive loss to them, aiming to increase the distance between different encodings. Therefore, CMTA enforces the distinctiveness of modules and provides the agent with a finer-grained mechanism to dynamically select module combinations within episode, alleviating the negative transfer within the task and improving the generalization ability and the performance for multi-task RL.
We conduct experiments on Meta- World[48], a multi-task RL benchmark containing 50 robotics manipulation tasks. Experimental results show that the proposed CMTA significantly outperforms all baselines, both on sample efficiency and performance, and even gets better performance than learning each task individually for the first time. CMTA also shows great superiority especially for complex tasks. As the variety of tasks increases, the advantages of CMTA become more apparent.
2

---

Reach handla
Push handle to goal
Success
Figure I: Different purpose in Meta- World manipulation task Puush' Push the puck to a goal.
2
Related work
Multitask Learning. Multi-task learning (MTL) can learn commonalities and differences across different tasks. [4, 41, 30]. Researchers have shown MTL can make multiple tasks benefit from each other by sharing useful information and exploiting the commonality and structure of these tasks [18, 28, 19, 27]. [49] shows that different tasks have different similarities, and some unrelated tasks may have negative impact on each other when training together. A large portion of the MTL work is devoted to the design of multi-task neural network architectures that can avoid negative transfer,
[31, 17] use a shared encoder that branches out into task-specific decoding heads. [21] proposed Cross-stitch networks, which softly share their features among tasks, by using linear combination of the activations found in multiple single task networks. Multi-Task Attention Network [19] consists of single shared network containing global feature pool, together with soft-attention module for each task.
Compositional modules and Mixture of experts. In order to reduce the negative impact of gradient conflict, a feasible method is to use compositional modules, which makes the model have better generalization. [3] leverages the compositional structure of questions in natural language to train and deploy modules specifically catered for the individual parts of question. [8] decomposes policies into "task-specific" (shared across robots) and "robot-specific" (shared across tasks) modules, and the policy is able to solve unseen tasks by re-combining the pre-defined modules. Routing Networks [29] is comprised of router and set of neural network modules, and the composition of modules is learned in addition to the weights of the modules themselves. Instead of directly selecting routes for each task like Routing Networks[29) soft modularization 44] uses task-specific policy softly combine all the possible routes.
In the deep learning era, ensemble models and ensemble of sub-networks have been proven to be able to improve model performance[38, 15J. [9]uunn the mixture-of-experts model into basic building blocks (MoE layer). The MoE layer selects experts based on the input of the layer at both training time and serving time. [23] selects one specific expert at a time for each input. [32] use gating function to select experts. [20] use multi-gate method, and introduce a gating network for each task.
Task embedding. Task embedding a very general form of learning task relationships, and has been widely used in meta-lcarning, as well as multi-task learning. It uses task information to model the explicit relationships between tasks, such as clustering tasks into groups by similarity, and leveraging the learned task relationships to improve learning on the tasks at hand. [1] uses task embedding to generate vectorial representations of visual classification tasks which can be used reason about the nature of those tasks and their relations. [14] uses metric learning to create task embedding that can be used by a robot to learn new tasks from one or more demonstrations. [34] uses metadata, pre-trained task embedding obtained by task description and pre-trained language model, to help the robot make use of the relation between continuous control tasks.
Contrastive Learning. Contrastive Learning is framework that aims to learn representations that adhere to similarity constraints in a dataset organized into pairs of similar and dissimilar items. This can be thought of as performing dictionary lookup task, where the positive and negative pairs represent a set of keys relative query (or anchor). One of the most basic forms of contrastive learning is Instance Discrimination [43], in which a query and a key are considered positive pairs if they are data augmentations of the same instance (e.g. image) and negative pairs otherwise. However, choosing appropriate negative pairs can be significant challenge in contrastive learning, as it can greatly impact the quality of the learned representations. There are several loss functions available for use in contrastive learning, such InfoNCE [39], Triplet[36], Siamese [5], and others.
3

---

mixture of experts 3 Preliminaries
Markov Decision Process (MDP) is defined by tuple (8, A, R, where S is finite set of states, A is a finite set of actions, transition function P(s'|s, (s, a) is transition probability from state state s' after action executed, reward function R(ss, a)) is the immediate reward obtained after action a is performed, and discount factor Y E (0, 1). We denote T(ass) as the policy which is mapping from state action. The goal of MDP
find an optimal policy to maximize the reward function. More specifically, in finite horizon MDP, the objective is to maximize the expected
discounted total reward which is defined by max= Er- Fr 7 (s;, at)], where is trajectory following dt ~ T (a))):)) St+1 ~ P(s:+1|st,a;) The value function policy π defined as (*)) = Fm 1 R()),
Multi-task RL optimization. In general multi-task learning with N tasks, optimize the policy to maximize the average expected return across all tasks, and the total loss function is defined as:
N
Liotel Σ X,L.
This is the linear combination of task-specific losses L, with task weightings A₃-
Contrastive learning. Given query q. positive keys kt and negative keys k_ the goal of contrastive learning is to ensure that matches with k more than h An effective contrastive loss function, called InfoNCE[39), is:
explq- k+ /T) Lg -log exply k+ s+/T) + exply k- k-/T)²
where r a temperature hyper-parameter.
4
Method
We now introduce our novel multi-task RL architecture, Contrastive Modules with Temporal Attention, which combines different modules with finer granularity by using temporal attention and contrastive modules. CMTA has the ability to alleviate negative transfer within the task, improve performance, enhance generalization, and achieve state-of-the art results. Notably, surpasses the performance of learning each task individually for the first time in the Meta-Wone " environment[48]
4.1 Contrastive Modules
Employing modular approach to reuse various modules at different time steps can result in enhanced performance and better generalization. Ensemble models and the mixture of experts have been proven to be able to improve model performance [38, 15] and shared modules. However, without proper constraints, some modules may redundantly learn the same skill feature, which goes against the original intention of using the modular approach, limiting the generalization ability of modular methods and only adding computational overhead without improving performance.
To tackle this problem, we introduce contrastive learning to enforce the distinctiveness of modules. Most researchers use contrastive learning to obtain a more robust representation, whereas we use contrastive learning to encourage the modules to be different from each other.
At time step for i-th encoder's output 2 as query qu, we use other encoder's output 2/ (j * i) as its negative pairs k and the output the next time step of i-th encoder 2 is selected the positive pair k The total contrastive loss can be written as:
explg: kT/T) log explg: kt E Ceaply--k -T)))'
Leon
"This is not the partially observable environment, because the task id is available and can be used to uniquely discriminate the task.

---

contrastive
Expert
?
enc
Expert
Policy network
Letor
Attention
Z sonk
L Leritic
LSTM
h,
2
network
h,
2,
Embedding
MLP
2 aud
task encoder
Figure 2: Architecture of our CMTA model. Given the current state we use mixture of k experts get encodings, and extract the task embedding Stask by task encoder using one-hot task id Zr. And we use LSTM with - and 8, as inputs, output current temporal information hę. The soft attention weights a, which used compute weighted sum of multiple encodings, take hg and Stask as inputs.
where K denotes the number of experts.
4.2 Temporal attention
Most methods neglect the negative transfer within tasks by only combining shared modules the task level, using fixed combination of modules for each task throughout the entire episode. To address this issue, we propose combining the modules with finer granularity at each time step by using temporal attention, further alleviating negative transfer within tasks.
Mixture of experts. We mixture of experts as our shared modules, implemented by k individual encoders. Each representation 2 output by the corresponding expert, encodes different features of the state St, and can be represented as:
= - f'(s,,
where fó denotes the j-th expert.
Temporal Attention module. In order to combine the shared modules dynamically, we use attention module to output the soft combination weights.
Given the current state St, we encode temporal information h4 as:
h = listmms;; hx-1);
(5)
where lstm a long short-term memory [13], h is previous hidden state at time t 1, and hg initialized with zero vector.
Given the input one-hot task id Zr, the task information be computed
Stask g(2+),
(6)
where g is the task encoder. Notably, [34] description and language model, called metadata. They argue that knowing the task description such "open door' and 'opnn drawer' is helpful for modeling the task relationships.
pre-trained embedding layer trained by task
By concatenating the temporal information h, and task information Ztask, we can dynamically compute appropriate attention weights within the episode as:
"1 ag softmaz hz)),
5

---

where W is fully connected layer, and softmar is a normalize function which restricts the sum of a; 1.
And the final representation a weighted sum of k multiple representations using soft attention weights as:
Zene
Oj
Fencs
(8)
which can be seen as combination of different modules.
4.3
Training approach
We use Soft Actor-critic (2) train our policies. SAC an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. The optimization objectives of policy and Q-function are:
JQ(0)=
(o) ()(00(,,)) (r(ss)) (7o(s)))))((s a'merg
(z) E [oog 1 Qo(s,a)]
(10)
where a is a temperature parameter that controls how important the entropy term is. Note that we only use actor loss to update the policy network, and use critic loss to update the whole model. The algorithm is described in Appendix A.
For the multi-task optimization with N tasks, we optimize the policy without any loss weighting tricks, which means we treat each task equally. The total loss is computed as:
Liotal
Li
(11)
N
Li LRL B Leons
(12)
where T. denotes the loss function of task i, B is hyper-parameter.
Experiment
In this section, we evaluate our method on multi-task robotics manipulation. We introduce the environment in Section 5.1, propose appropriate evaluation metric in Section 5.3 and compare with baseline methods on WerldWorld in Section 5.2 and Section 5.4 and conduct ablation study in Section 5.5.
5.1 Environment
We evaluate the effectiveness of CMTA model on Meta-World environment(48), which is collection of robotic manipulation tasks designed to encourage research in multi-task RI. We use the mean success rate of the binary-valued success signal as our evaluation metric, which has been clearly defined in Meta- World[48]. Meta-World consists of 50 distinct robotic manipulation tasks with simulated Sawyer robotic arm, each task has its own parametric variations 50 sets of different initial positions of the object and the goal. The multi-task RL benchmarks within Meta- World are MT10 and MT50, which consist of simultaneously learning 10 and 50 tasks. the original setting, MT10 and MT50 only use one fixed position for each task during the training stage, which may cause agent to simply memorize the position and overfitting. To reflect the generalization ability of the model, we mix all 50 sets of positions together for tasks in MT10 and MT50 at training stage, and name them as MT10-Mixed and MT50-Mixed. The original MT and MT50 denoted as AT10-Fixed and MT50-Fixed

---

MT-SAC
MTT-SAC +TE
MTMH-SSC
Soft Modularization
cane
(MTAAours)
Single-SAC
Figure 3: Training curves of different methods on MT10 and MT50 benchmarks, each curve is averaged over 8 seeds(Shaded areas: the standard deviation over 8 seeds). Our approach consistently outperforms baselines in all environments, whether on asymptotic performance or sample efficiency, and the superiority is more obvious on MT1O-Mixed and MT50-Mixed. TE Task Encoder.
5.2 Baselines
We take the following multi-task RL methods as the object of our comparison, considered as the baselines: (i)Single-task SAC(upper bound): Using a separate network for each task, which is usually considered as the performance upper bound. Mii)Multitassk SAC(MTT-SACT48) Using shared network with the state as input. Muliituk SAC with task encoder(MT--AC+TT) Adding a task encoder with a one-hot task id as input based on MT-SAC. (iv) multi- head SAC (MTMH-SAC(48)), Using a shared network with task-specific heads for each task. (v)Soft
SaailariizationSStMMoolll [44]): It contains several compositional modules as the policy network, and uses a task-specific routing network softly combine the module in the policy network. Cvi)Connextual ceantion-based representation (CARE[34)): CARE uses mixture of encoders, and uses a context encoder with additional prior knowledge metadata to context- based attention.
5.3 Implementation details
The experimental results of reinforcement learning environments like Meta- World have great ran- domness, and the number of seeds plays an important part in the final performance of the model. Specifically, as mentioned in [34], MTMH-SAC achieved success rate of 88% (1 seed) in [48], our experiment, each agent trained over seeds, and we average the mean success rate of each seed to measure the performance. We train the agent with 2.4 million steps (per task) for both MT10 and MT50 environments. More implementation details of hyper- paremeeessssess can be found in Appendix D.
which is 44% different from 10seed's performance.
During training phase, we evaluate the agent every 3K steps (20 episodes), and we find that the smoothed testing curve and the smoothed training curve are almost identical. It might be the case that the temperature parameter a becomes very small in the training period, which discourages exploration, that is to say, the policy has the same behavior without stochasticity in the training and testing phase.
Even though the testing results have been averaged over 8 seeds, it is still very volatile. Previous work often the maximum of the mean success rate in evaluation, which is much higher than the average, since the max operation will bring much larger variances. In our experiments with seeds, there a big gap between the maximum of the mean success rate and the maximum of the smoothed (smooth factor 0.8) mean success rate, which is about 5% ~ 10%, bringing more uncertainty to the evaluation of model performance. Predictably, this uncertainty is much bigger in the case of fewer seeds. [44, 34] tries to reduce uncertainty by evaluating the agent several times at once, but it doesn't t really work. Because in the original MT10 and MT50, each task environment has fixed initial position, and the policy based on SAC is determined during evaluating. In other words, given the parameters of the model, no matter how many times we evaluate the agent, the result exactly the same as the first time in the testing stage. Thus, compared to the maximum of the mean success rate, we use the maximum of the smoothed mean success rate to measure the performance of models, which is more stable and realistic.
7

---

Table 1: Evaluation performance on MT10 and MT50 tasks after training for 2.4 million steps (for
each task), using the metric of max success rate and max smoothed success rate, where the prior has smaller variance. Results have been averaged over 8 seeds. Our CMTA outperforms other methods on all test environments, including the Single-SAC which is considered as the upper bound. TE = Task Encoder.
MT10-
MTS0-Fixed
MT50-Mixed
agent
success rale
success rale
Success Fale
success rale
maL smoothed
max
max smoothed
max
mIL smoothed
max
smoothed
max
MT-SAC
62.25%
68.75%
53,22%
62.50%
50.37%
52.50%
28, 73.7%%
31,50%
STCAAC+T
64.76%
61.12%
68.75%
54.75%
37.59%
MTMH--SA
65,21%
70%
62.06/
67.50%
47.67%
48.75%
39.65%
42.75%
MonMadu
51%
55%
51.34%
58.75%
26.23%
28.75%
21.50%
23.50%
CARE
68.03%%
75%
61.35%
67.50%
55,47%
57.50%
45.0015
48.50%
CMTA(our))
78.95%
83.75%
82.07%
87.5%
68.90%
71.00%
71.69%
74.5%
Single-SAClupeer bound)
64.33%
68.75%
71.11%
76.25%
5.4 Results
To evaluate the performance, we compare CMTA with other baselines on MT10-Fixed, MT10-Mixed, MT50-Fixed and MT50-Mixed environments. The training curves averaged over 8 seeds are plotted in Figure 3. And Table records the best evaluation results of all the methods, including both max smoothed (with smooth factor 0.8, details can be found in APPENDIXC) mean success rate and max mean success rate (large variance). It is noteworthy that our method is even better than CARE, which uses additional prior knowledge.
As shown in Figure 3, our approach consistently outperforms baselines in all environments, whether sample efficiency. In the original enviromments MT 10-Fixed and
on asymptotic performance MT50-Fixed, it can be seen from Table that CMTA is better than other baselines around 15% and 24%, respectively. And in the more difficult environment with changing positions, MT10-Mixed and MT50-Mixed, the superiority of our method is more obvious, which is about 34 % and 60% % better than other baselines respectieeeeeeee respectively.
When changing from Fixed environment to Mixed environment, becomes more difficult because the agent have to deduce patterns or objectives of tasks (e.g., moving an object to a goal) rather than merely memorizing fixed positions. All the baseline methods suffered this impact in MT10- and MT50-Mixed, but method achieves even better performance, which implics good generalization and robustness of our method.
Significantly, we notice that only our approach outperforms learning each task individually (we only conduct this experiment on MT10, because it is too slow to train 50 policies simultaneously on MT50). This evidence proves that our approach can avoid negative transfer and improve the performance of tasks by correctly combining the modules. While all the methods can take advantage of multi-task learning to compress the model and improve sample efficiency, in contrast to ours, other methods come at the cost of performance.
As mentioned before, RL experimental result has large stochasticity, and the evaluation metric used in previous works ([48, 44, 34J) is max success rate during testing, which leads to large variance and is influenced by evaluation times. Thus we use max smoothed success rate to mitigate the effects of large variances caused by max operations. Because of these randomnesses, it is reasonable that the results are somewhat different from the previous works. However, the result of soft modularization (44] in our experiment is much worse than previous work. In addition to the number of seeds and the randomness of the RL algorithm, we guessed that the most likely reason is that SoftModu uses loss weighting trick in the optimization, which balances the individual loss for different tasks. For the sake of fairness, we don't tt use any loss weighting method. The ablation study of SoftModu[44] also shows that MTMH-SAC outperforms SoftModu without balance, which is consistent with our experimental results.
8

---

MTTC-Fixed
MT10 1ix0-Mixed
13.4
0.2
0.2
CMTA wib 71
0.0
CMTA web T1
0.0
0.5
2.0
Milioo step
Milii Stop
Figure 4: Effectiveness of temporal information (TI) on MT10-Fixed and MT10-Mixed environment, each curve has been averaged over seeds.
Table 2:
Effectiveness contrastive loss(CL) MTI0 and MT50 tasks after training for 2.4 million steps (for each task). Results have been averaged over seeds.
MT10-Mixed success rate
MT50-Mixed success rate
agent
max smoothed
max
max smoothed
max
CARE CARE CL
61.35%
67.50%
45.00%
48.50%
65.24%
71.25%
47.61 %
49.75%
CMTA w/o CL
79.46%
85%
62.66%
65%
CMTA(ours)
82.07%
87.5%
71.69%
74.5%
5.5
Ablation Study
Temporal Attention. We investigate the significance of temporal information and report the compar- ison results in Figure 4. It can seen that the temporal information we employed is indeed highly effective, which indicates that the negative transfer within tasks is further alleviated.
Contrastive Modules. We conducted comparative experiments validate the effectiveness of contrastive loss, as shown in Table 8. From the results, it can be seen that the CL loss consistently improves performance
the mixed environment, especially in MT50-Mixed where our method achieves an increase in success rate of nearly 10 percentage points. Additionally, we observed that the impact of CL loss in the fixed environment is not substantial. This could be attributed the fixed environment being overly homogeneous, which makes it prone to overfitting and fails to demonstrate the robust generalization advantage of contrastive modules.
To verify whether contrastive learning can constrain modules to learn unique functions and reduce [-SNE[40] visualize the output embeddings of K(6 in setting)
similarity between them, multiple modules in Figure 5. The visualization of CMTA and CMTA without CL are obtained by multiple embeddings of modules each time step MT 10-Fixed for 10 episodes. that without contrastive learning constraints, the encodings of multiple modules are mixed together.
be observed
(b) CMTA Figure 5: t-SNE visualization of multiple modules' encodings on MT10-Fixee environment.
(a) CMTA w/o CL

---

By using CL, the encodings of different modules become more distinct, indicating that CL indeed reduces the similarity between modules.
Limitation
One limitation the computational cost of CMTA, the contrastive learning part is proportional to O(n2).wherr is number of experts. If task similarity is substantial, then even with an increase in the number of tasks, computational cost won't necessarily escalate as long as the number of experts remains constant. Another limitation is the dependence on task similarity, and it's s the fundamental assumption in the field of mtrl, even humans struggle to extract mutually beneficial information from entirely unrelated tasks. For instance, attempting to learn from combination of tasks like playing and playing Atrai games would likely yield limited benefits.
7 Conclusion
In this paper, we present contrastive modules with temporal attention, which is generalized way of learning distinct shared modules with finer granularity for multi-task RL. CMTA has the ability to constrain the modules to be different from each other by contrastive learning and the ability to learn the best combination of distinct shared modules dynamically each time step across tasks by using temporal attention. The module combination in CMTA varies form step to step, so that modules can be reused when it's helpful and not shared when it's conflict, alleviating the negative transfer between within the task and shows great superiority especially for complex tasks. Experiments on World show that method outperforms all baselines, both sample efficiency and performance, and it's the first time achieved performance surpassing that of training each task separately. As the variety of tasks increases(from MT10 to MT50, from Fixed environment Mixed environment), the advantages of our approach become more apparent. Our method has achieved good results multi-task RI,, and will further extend meta learning in future work.
Acknowledgements
This work is partially supported by the NSF of China(under Grants 61925208, 62102399, 62222214, 62002338, U19B2019). CAS Project for Young Scientists Basic Research(YSBRR-Oe Youth Innovation Promotion Association CAS and Xplore Prize.
References
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vee: Task embedding for meta-learning. 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pages 6429 6438, 2019.
[2] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference Machine Learning. 2016.
[3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 39-48, 2015.
[4] Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
[5] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. 2005 IEEE Computer Society Conference on Computer Vision and Pottern Recognition (CVPR'05) 1:539-546 vol. 1, 2005.
[6] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In International Conference on Machine Learning, 2008.
10

---

[7] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. ArXiv, (bss/200.00976, 2020.
[8] Coline Devin, Abhishek Gupta, Trevor Darrell, P. Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2169-216 2176, 2016.
[9] David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. CoRR, abs/1312.4314 2013.
[10] Chrisantha Fernando, Dylan S. Banarse, Charles Blundell, Yori Zwols, David R Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. ArXiv, abs/1 701.08734, 2017.
[11] Jiaming Guo, Rui Zhang, Xishan Zhang, Shaohui Peng, Qiaomin Yi, Zidong Du, Xing Hu, Qi Guo, and Yunji Chen. Hindsight value function for variance reduction in stochastic dynamic environment. In International Joint Conference on Artificial Intelligence. 2021.
[12] Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-po max- imum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018
[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9:1735-1780, 1997.
[14] Stephen James, Michael Bloesch, and Andrew Davison. Task-embedded control networks for few-shot imitation learning. ArXiv, abs/1810.03327,, 2018.
[15] Michael I. Jordan and Robert A. Jacobs. Hicrarchical mixtures of experts and the em algorithm. Neural Computation, 6:181-214, 1993.
[16] Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 0bss/1509.0997 2015.
[17] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Tak Wu Kwong. Pareto multi-task learning. In Neural Information Processing Systems, 2019.
[18] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. ArXiv, abs/ (bss/1605.00 001001,011112 2016.
[19] Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. 20/9 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1871-1880, 2018.
[20] Jiaqi Ma. Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018
[21] Ishan Misra, Abhinav Shrivastava, Abhinav Kumar Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3994-4003, 2016.
[22] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
[23] Giambattista Parascandolo, Mateo Rojas-Carully Carulla, Niki Kilbertus, and Bernhard Schölkopf. Learning independent causal mechanisms. International Conference on Machine Learning. 2017.
[24] Shaohui Peng, Xing Hu, Qi Yi, Rui Zhang, Jiaming Guo, Di Huang, Zikang Tian, Ruizhi Chen, Zidong Du, Qi Guo, al. Self-driven grounding: Large language model agents with automatical language-aligned skill learning. arXiv preprint 209.02230. 2023.

---

[25] Shaohui Peng, Xing Hu, Rui Zhang, Jiaming Guo, Qi Yi, Rui Chen, Zidong Du Ling Li, Qi Guo, and Yumji Chen. Conceptual reinforcement learning for language-condttoons tasks. In AAAI Conference on Artificial Intelligence, 2023.
[26] Shaohui Peng, Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Rui Chen, Xishan Zhang, Zidong Du, Ling Li, Qi Guo, and Yunji Chen. Causality-driven hierarchical structure discovery for reinforcement learning. ArXiv, abs/2210.06964, 2022.
[27] Lerrel Pinto and Abhinav Kumar Gupta. Learning to push by grasping: Using multiple tasks for effective learning. 2017 IEEE International Conference Robotics and Automation (ICRA), pages 2161-2168, 2016.
[28] Rajeev Ranjan, Vishal M. Patel, and Rama Chellappa. Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:121-135, 2019.
[29] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. ArXiv, abs/1711.012211 2017.
[30] Sebastian Ruder. An overview of multi-task learning in deep neural networks. ArXiv, abs/1706.05098, 2017.
[31] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization, In Neural Information Processing Systems, 2018.
[32] Satinder Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8:323-339, 2004.
[33] Shagun Sodhani and Amy Zhang. Mtrl multi task rl algorithms. Github, 2021.
[34] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9767-9779. PMLR, 18-24 Jul 2021.
[35] Anders Søgaard and Joachim Bingel. Identifying beneficial task relations for multi-task learning in deep neural networks. In Conference of the European Chapter of the Association for Computational Linguistics, 2017.
[36] Nitish Srivastava. Unsupervised learning of visual representations using videos. 2015.
[37] Yee Whye Teh, Victor Bapst, Wojeiech M. Czarnecki, John Quan, James Kirkpatrick, Raia Had- sell, Nicolas Manfred Otto Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. ArXiv, abs/ 707.7070000 700.04475, 2017.
[38] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Bonch, and Patrick Medaniel. Ensemble adversarial training: Attacks and defenses. ArXiv, abs/1775.072044 2017.
[39] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/ fbss/180.003743 2018.
[40] Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.
[41] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:3614-3633, 2020.
[42] Sen Wu, Hongyang Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. ArXiv, abs/2000500044, 2020.
[43] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance-level discrimination. ArXiv, has/1805.01972 2018.
12

---

[44] Ruihan Yang, Huazhe Xu, YI WU, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4767-4777. Curran Associates, Inc., 2020.
[45] Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Yunkai Gao. Kaizhao Yuan, Rui Chen, Siming Lan, Xing Hu, Zidong Du, Xishan Zhang, Qi Guo, and Yunji Chen. Online prototype alignment for few-shot policy transfer. In International Conference on Machine Learning, 2023.
[46] Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Xing Hu, Zidong Du, Qi Guo, Rui Chen, Ling Li, and Yunji Chen. Learning controllable elements oriented representations for reinforcement learning. Neurocomputing, 549: 126455, 2023.
[47] Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Xing Hu, Zidong Du, Xishan Zhang, Qi Guo, and Yunji Chen. Object-category aware reinforcement learning. ArXiv, abs/2210.07802, 2022.
[48] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 1094-1100 1100. PMLR, 30 Oct-01 Nov 2020.
[49] Amir Roshan Zamir, Alexander Sax, Bokui (William) Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3712-3722, 2018.
13

---

A
Pseudo Code
Algorithm 1 CMTA
Initialize: replay buffer with Ø
Initialize: initial hidden state ho with zero tensor
Initialize: policy т with @. Q-function Q task encoder g, experts ft, ,..... Sth lstm, fully connected layer W
Input: state 9; for each environment, one-hot task id 2+
1: for episode m 2, do for time-step
2:
1,2. do
3: for each task T; do
4: =/ = (s,), Vj E c., Stask = (
5:
6:
h, lssmssssss h1-1)
7:
01
s0f ooftmaa(W((h ztask))
8:
Zene
Σド= Kj Zennc
9:
2task Zenc
10:
sample action T--|Zenshi Zenc)
11:
Perform action ae, get reward ry and next state 8
12:
D = (s$ Tt,Tt, 8t+1, 14-1, h4, 2r}
13:
end for
14:
randomly sample batch from D
15:
compute Leontrastive by E compute Lactor and Leritic by Eq9 and Eq10
16:
17:
update experts with Leontrastive
18:
update Te with Luctor
19:
update all components except Fa with Leiiti
20: end for
21:
end for
B Libraries
We use the following open-source libraries: MetaWorld', MTEnv* MTRL [33].
C Smooth factor
smoothed ppinttid srroothed_poinfffe 1] factor + point(i] * (1 - factor), ifi> 0
point[i],
if
D Hyperparameter Details.
Phupp://githuh.c "htps://gituuh.co//twwrwr wtt.://tw.....ooooooooooootttttt
"haps://github.com/ com//fcchhwokreeeecch// mteer
14

---

Table 3: Hyperparameter values that are common across all the methods
Hyperparameter
Hyperparameter values
batch size
128 x number of tasks feedforward network
network architecture
actor/critic size non-linearity
three fully connected layers with 512 units
ReLU
policy initialization temperature
standard Gaussian
learned and distangled with tasks run uniform exploration policy 1500 steps
exploration parameters num of samples num of train steps per iteration
env step training step
evaluation frequency replay buffer size policy learning rate Q function learning rate optimizer
3000 steps
5000000
3e-4
3e-4
Adam
policy learning rate
3e-4
beta for Adam optimizer for policy
(0.9, 0.999)
Q function learning rate beta for Adam optimizer for Q function
3e-4
(0.9, 0.999)
discount
0.99
Episode length (horizon)
150
reward scale
Table 4: Hyperparameter values of task encoder
Hyperparameter
Hyperparameter values
task encoder train from scratch pretrained
embedding layer with dim 64 + FC 128 + FC 64 + FC 64
pre-trained embedding layer with dim 512 FC 128 FC 64 FC 64
Table 5: Hyperparameter values of Soft Modularization
Hyperparameter
Hyperparameter values
task encoder type routing network size
train from scratch
4 layers and 4 modules per layer with dim 64
Table 6: Hyperparameter values of CARE
Hyperparameter
Hyperparameter values
task encoder type encoder size number of encoders
pre-trained embedding layer
FC 64 FC
6
Table 7: Hyperparameter values of CMTA
Hyperparameter
Hyperparameter values
task encoder type encoder size number of encoders 6
train from scratch FC 64 + FC 64
B
2500
15

---

E
Additional Experiment Results
MT-SAC
MT-SAC +TE
MTMH--SAC
Soft Modularization
care
CMTAAOurs)
single-SAC
Figure 6: Training curves of different methods each task of M0-e-Mixed, each curve is averaged over 8 seeds. Our approach consistently outperforms baselines in all tasks, whether on asymptotic performance or sample efficiency.
Table 8: Ablations on experts number. Evaluation performance on MTI0 tasks after training for 1.0 million steps (for each task). Results have been averaged over seeds.
MT10-Fixed
MT10-Mixed success rate
agent
success rate
max smoothed
max
max smoothed max
CMTA with 2 experts
54.26%
62.5%
52.53%
57.5%
CMTA with 4 experts
67.68%
72.5%
72.19%
76.25%
CMTA with 12 experts
71.21
75.0%
73.18%
78.75%
CMTA with 20 experts
68.12%
72.5%
75.01 %
78.75%
CMTA with 6 experts
68.25%
73.75%
78.49%
82.5%
16

---

peg-inper-sste
stace
close
Figure 7: t-SNE visualization of CMTA attention weights on MT Mi-Med environment.
17


Reviews:
[{"review_id": "EOvtBvBzgd", "reviewer": "Reviewer_ouGh", "summary": "The paper focuses on multi-task reinforcement learning. Motivated by the negative task transfer within each task, the paper proposes the Contrastive Modules with Temporal Attention (CMTA) method, which utilizes temporal attention to modulate the weights of experts. Concretely, temporal attention takes recent history as input to capture local information. The proposed CMTA is evaluated in MetaWorld and shows superior performance than baselines.", "questions": "- What is the number of experts in the mixture for each experiment setting, including MT10-Fixed, MT10-Mixed, MT50-Fixed, MT50-Mixed?\n \n - How is the smooth curve calculated exactly? The meaning of the smooth factor in line 242 is unclear. \n\n - Why do the performances drop for MT50-Mixed after 1.5 Million steps (figure 3 the right most figure MT50-Mixed)?", "limitations": "- The limitation is not discussed in the conclusion. One trade-off is the increased model size due to experts and the improved performance.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The methodology part is clear and well-written.\n\n - The methodology is well-motivated.", "weaknesses": "- The paper poses the multiple skills that may be utilized in each task as negative transfer problems within a task. I believe such terminology is ok but I would highly recommend including a session in related work summarizing the skill-based RL literature and pointing out the deep connection between “negative transfer within a task” and “skill-based RL” in the introduction part. \n\n - The novelty of the paper is limited from my point of view.\n\n - The experiment part lacks necessary information and please refer to the questions. The ablation study can be further improved by analyzing the impact of history length in the temporal attention mechanism, as well as the number of experts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Our gratitude goes to the reviewer for their insightful comments, which have significantly enhanced the quality of our work.  Any further discussion will be appreciated.\n\n> W1:  The paper poses the multiple skills that may be utilized in each task as negative transfer problems within a task. I believe such terminology is ok but I would highly recommend including a session in related work summarizing the skill-based RL literature and pointing out the deep connection between “negative transfer within a task” and “skill-based RL” in the introduction part.\n\nWe appreciate the reviewer's suggestions. We think that \"skill-based RL\" indirectly addresses the issue of \"negative transfer within a task\" by employing distinct skills. However, its primary focus lies in using hierarchical reinforcement learning to tackle intricate tasks that are challenging for a single policy, or inducing diverse skills through reward shaping during pretraining to adapt swiftly to new tasks. We will incorporate this analysis in the forthcoming revised version of the paper.\n\n> w2:  The novelty of the paper is limited from my point of view.\n\nThe novelty of this paper is primarily twofold: contrastive learning and temporal attention. These address two aspects of non-modularity in current  MTRL methods. Both of these aspects were acknowledged in the \"Originality\" by Reviewer 4rda. In the paper, we conducted ablation experiments on the introduced components individually (as shown in Table 2 and Figure 4). These experiments demonstrated the effectiveness of both components, resulting in performance improvements of approximately 5% and 20%, respectively. Based on the summary part, it appears that the reviewer might have mainly focused on the second aspect, potentially leading to confusion regarding the novelty of the paper.\n\n> W3： The experiment part lacks necessary information and please refer to the questions. The ablation study can be further improved by analyzing the impact of history length in the temporal attention mechanism, as well as the number of experts.\n\n- number of experts:  The ablation result can be seen in the pdf of global response. The experimental results indicate that having fewer experts leads to performance degradation. However, increasing the number of experts beyond a certain point does not yield positive effects.\n\n- history length: During the data collection process, we store both the current hidden state and the next hidden state . Consequently, each sample comprises the elements (s, a, r, s', h, h'). This structure enables our temporal attention mechanism to effectively encompass the historical information of the entire trajectory up to the current time step. Notably, the history length is not a hyperparameter here, which makes conducting ablation experiments on it potentially unnecessary.\n\n> Q1: What is the number of experts in the mixture for each experiment setting, including MT10-Fixed, MT10-Mixed, MT50-Fixed, MT50-Mixed?\n\nWe use 6 experts for all settings, as mentioned in the Appendix D.\n\n> Q2: How is the smooth curve calculated exactly? The meaning of the smooth factor in line 242 is unclear.\n\nThanks for bringing this omission to our attention. We will certainly address this in the revised version by including the calculation method for the smooth curve:\n\n$$smoothed\\\\_point[i] =  \\\\begin{cases} smoothed\\\\_point[i-1]* factor + point[i] *(1-factor), & \\\\text{if $i$ > 0} \\\\\\\\  point[i], & \\\\text{if $i$ = 0} \\\\end{cases}$$\n\n> Q3: Why do the performances drop for MT50-Mixed after 1.5 Million steps (figure 3 the right most figure MT50-Mixed)?\n\nAll algorithms exhibit this phenomenon in the MT50-Mixed environment, and we believe this is an inherent issue with the mixed environment itself.  Most of the baselines experience performance degradation around 0.5 million steps, whereas our method's performance decline begins at 1.5 million steps, highlighting its robustness.  We speculate that the reason for this performance drop lies in the more diverse nature of the MT50-Mixed environment.  Overtraining can lead to model overfitting, causing it to prioritize simpler tasks over more challenging ones.  Our observations of individual task success rates support this, as some tasks indeed show declines in success rates during later stages. In MT10-mixed, among the 10 tasks, only one task experiences a decline in performance during the later stages(Appendix C, Figure 6, push-v1). However, in MT50-mixed, among the same set of 10 tasks, 5 tasks demonstrate performance drops, and these declines occur earlier in the training process compared to MT10-mixed.  Given that our x-axis represents steps for each task, the training data volume for MT50 is five times that of MT10, which can lead to quicker overtraining issues."}, {"review_id": "nhj4KvfRFy", "reviewer": "Reviewer_4rda", "summary": "This paper proposes an approach to multi-task RL called Contrastive Modules with Temporal Attention that aims to address the issue of negative transfer between tasks in multi-task RL.\n\nThe proposed method consists of two main components: contrastive learning and temporal attention. The contrastive learning component is used to ensure that the shared modules learned by the method are distinct from each other. This is achieved by applying a contrastive loss that encourages the modules to produce different outputs for the same input. The temporal attention component is used to dynamically combine the outputs of the different modules at each time step. This allows the method to adapt to the specific requirements of each task.\n\nThe authors evaluate their method on the Meta-World benchmark, a widely used benchmark for multi-task RL. They compare the performance of their method with several baselines, including methods that train each task separately and methods that share all modules across tasks. The results show that CMTA outperforms the baselines in terms of both sample efficiency and performance.", "questions": "Subsumed into Weaknesses and Limitations.\n\n1. What are three limitations of your work that have not been addressed in the reviews, and what are your thoughts about them? This isn't intended to diminish your work. Instead it's to show that you understand where and how your work shines and to highlight where it may not as future problems to be addressed or as problems that are insignificant for some reason(s).", "limitations": "Scalability: The proposed method might not scale well to tasks with a larger number of subtasks or more complex environments, even though the paper claims in Line 311-312 that \"as the variety of tasks increases, the advantages of our approach become more apparent\", they don't show this outside of Meta-World, and Meta-World is measure 0 subset of possible tasks. \n\nRegardless, the computational cost of the method, especially the contrastive learning part, could increase significantly with the complexity of the tasks.\n\nDependence on Task Similarity: The effectiveness of the proposed method might depend on the similarity of the tasks. If the tasks are very different from each other, the shared modules learned by the method might not be effective for all tasks.\n\nI did not find the paper addressing its limitations anywhere. One I will put forth is the lack of insight into the temporal attention module's relationship to different aspects of the problem, such as (1) how the attention weights vary over time, (2) whether they reach a steady state, (3) what do they focus on, for the Meta-World problems covered in the paper. \n\nI believe we should be wary of how this overall approach behaves, especially the temporal attention module and the assigned soft attention weights, in problems in which ethics, safety, fairness, bias, may be a concern. This is general concern for any algorithm that doesn't explicitly address and mitigate these issues, but it's especially one here because of the lack of insight into the temporal attention module's relationship to aspects of the problem.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality\n- The paper presents a novel approach to multi-task RL by introducing contrastive modules with temporal attention.\n- The method addresses the issue of negative transfer between tasks, which is a significant challenge in multi-task RL. The authors propose a novel solution to this problem by constraining the modules to be different from each other and using temporal attention to dynamically combine them.\n\nQuality\n- The authors provide adequate experimental results on Meta-World, a widely-accepted continuous control robotics benchmark, and some ablation studies that support the effectiveness of their method.\n- The paper is well-referenced, indicating a thorough understanding of the existing literature. The authors clearly position their work within the context of previous research.\n\nClarity:\n- The paper is well-organized and the writing is clear. The authors provide a clear explanation of their method and its advantages.\n- The figures and tables in the paper are informative and support the text well. They help to clarify the method and the experimental results.\n\nSignificance:\n- The proposed method addresses a critical challenge in the field and shows superior performance compared to existing methods.\n- The method proposed by the authors, in particular the temporal attention mechanism, has the potential to be widely adopted in the field of multi-task RL. It could also inspire future research in this area.", "weaknesses": "Limited insight into hyperparameter sensitivity: The paper would be stronger if it discussed the sensitivity of the proposed method to its hyperparameters. Understanding how changes in hyperparameters affect the performance of the model is crucial for reproducibility and for users who wish to apply the method to their own tasks.\n\nLacking insight into soft attention weights: It would be interesting to see the general relationships of soft attention weights and other aspects of the problem, such as its temporal nature, the tasks involved, etc.\n\nLimited Discussion on Failure Cases: While the paper presents a number of successful results, it could discuss in detail the scenarios where the proposed method fails or performs sub-optimally. Such a discussion could provide valuable insights into the limitations of the method and guide future improvements.\n\nLack of Comparison with Related Work: While the paper compares the proposed method with several baselines, it does not compare it with other multiple other methods that also use contrastive learning or attention mechanisms in the context of multi-task RL (*see first bullet point below). Such comparisons could provide a more comprehensive evaluation of the proposed method. \n  - However, the paper does add the contrastive loss to CARE and evaluate it, but CARE's attention mechanism is very different than CMTA's and only CARE was compared against in this way.\n  - It would be interesting to see how other proposed baselines or competitive related work would benefit from the proposed temporal attention module.\n\nGeneralizability: The proposed method has been evaluated on a specific benchmark (Meta-World). Its performance on other benchmarks or real-world tasks that differ from Meta-World type setup and tasks is not known. CMTA may not outperform certain baselines or other algorithms on suites of tasks in other benchmarks or real-world tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their thoughtful suggestions and comments that have greatly improved our manuscript.  Any further discussion will be appreciated.\n\n> W1: Limited insight into hyperparameter sensitivity\n\nThe ablation of experts number can be seen in the pdf of global response. The experimental results indicate that having fewer experts leads to performance degradation. However, increasing the number of experts beyond a certain point does not yield positive effects.\n\n>  W2:  Lacking insight into soft attention weights: It would be interesting to see the general relationships of soft attention weights and other aspects of the problem, such as its temporal nature, the tasks involved, etc.\n\n- temporal nature: see the answer of L4\n- tasks involved：see the t-sne visualization of attention weights for different tasks in the pdf of global response.  It is obvious that the attention weights of different tasks are clustered into different clusters, which indicates that CMTA will choose different module combinations for different tasks.\n\n> W3: Limited Discussion on Failure Cases\n\nSome discussion on failure cases including:\n\n- According to the Sec 5, the Mixed version of MetaWorld benchmark is supposed to be more difficult than the fixed version. Why the Single-SAC is performing worse in the Fixed MT10, and it seems the Single-SAC results (in Fixed setting) is much worse than the reported in previous works.\n- All baselines perform worse in Mixed (compared with in Fixed), while the proposed method works better in Mixed Version.\n- Why the performance of all methods significantly drops after a certain stage for MT50-Mixed (similar phenomenon did not appear in MT10-Mixed).\n\n Due to character limitations, the answers can be found in reviewer fVjH's W1. \n\n> W4: Lack of Comparison with Related Work\n\nCurrently, there are no other MTRL algorithms utilizing contrastive learning. Additionally, the employment of attention mechanisms is limited to the CARE and SoftModu. Our use of contrastive learning and temporal attention is built upon a mixed-of-experts (MOE) design, which restricts their applicability to methods utilizing MOE (specifically, only CARE employs MOE). To validate the temporal attention module on other baselines like MTSAC, it would indeed necessitate the introduction of an MOE. However, this modification would alter the original structure (e.g., SAC + temporal attention (+MOE) = CMTA w/o CL). In essence, this corresponds to a portion of our ablation experiments.\n\n> W5: Generalizability: The proposed method has been evaluated on a specific benchmark (Meta-World). Its performance on other benchmarks or real-world tasks that differ from Meta-World type setup and tasks is not known. CMTA may not outperform certain baselines or other algorithms on suites of tasks in other benchmarks or real-world tasks.\n\nMetaWorld provides a diverse task distribution with 50 different tasks involving objects like doors, cups, windows, drawers, etc. and skills like push, pull, open, close, etc.  Evaluating on a broad task distribution（Metaworld） provides a good estimate of the generalization capabilities of mtrl algorithms .\n\nCurrently, MetaWorld stands as the sole widely recognized benchmark in mtrl, and previous mtrl works (CARE(Sodhani et al., 2021), SoftModu(Yang et al., 2020)) also  only evaluate their methods on MetaWorld. Therefore, we need to first construct benchmarks before implementing the algorithm. So, we will leave it as future work. \n\n> L1: Scalability\n\nAs mentioned in W5,  MetaWorld stands as the sole widely recognized benchmark in mtrl and provides a good estimate of the generalization capabilities of mtrl algorithms.  In Meta-World,  we believe that the following changes in settings have contributed to increased task complexity and diversity:  \n\n- The number of tasks increasing from 10 (MT10) to 50 (MT50).\n\n- Transitioning from fixed positions to varying positions (mixed).\n\nIn both of these scenarios, our method exhibits greater advantages over other baselines (as indicated in lines 259-262). This outcome to some extent substantiates the claim that \"as the variety of tasks increases, the advantages of our approach become more apparent.\"\n\n> L2:   Regardless, the computational cost of the method, especially the contrastive learning part, could increase significantly with the complexity of the tasks. \n\nThe computational cost of the contrastive learning part is proportional to $O(n^2)$ , where n is number of experts.  If task similarity is substantial, then even with an increase in the number of tasks, computational cost won't necessarily escalate as long as the number of experts remains constant. For instance, in the case of MT50, we employed the same six experts as in MT10.\n\n> L3:   Dependence on Task Similarity\n\nTask similarity is indeed a fundamental aspect of multi-task learning. Even humans struggle to extract mutually beneficial information from entirely unrelated tasks. For instance, attempting to learn from a combination of tasks like playing Go and play atrai games would likely yield limited benefits.\n\n> L4:  I did not find the paper addressing its limitations anywhere. One I will put forth is the lack of insight into the temporal attention module's relationship to different aspects of the problem.\n\nBased on our observations, the trends in attention weights exhibit rapid changes during the early stages of each episode, followed by continuous fluctuations within a narrow range. It is plausible that the agent rapidly infers the necessary skills during the initial phase and subsequently refines the skills it employs based on the discrepancies between execution and prediction. Since our attention mechanism does not directly operate on input observations, it becomes challenging to directly infer what specific aspects the agent is focusing on."}, {"review_id": "7dn6xnzruV", "reviewer": "Reviewer_GQFL", "summary": "The paper introduces an approach for multi-task RL. Their approach is similar to CARE, which learns separate encoder modules, but they add a contrastive task loss on top of the encoders. They show this approach outperforms all reported baselines on Meta-World (MT-10 and MT-50) and a variant of Meta-World where the initializations are mixed throughout training.", "questions": "- On L143 \"Notably, it surpasses the performance of learning each task individually for the first time in the Meta-World environment\" Is this true in the case of MT10? E.g., Figure 3 of Gradient Surgery for Multi-Task Learning (Yu et. al., 2020).\n- What are the error bars in the figures?\n\n", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The proposed approach outperforms all reported baselines and independent training on the metaworld tasks in the fully observed setting. \n- The proposed solution is straightforward to implement.", "weaknesses": "- A lot of the methods section should be moved to a preliminary section because it is difficult to understand what is novel and not novel. The temporal attention section on L168-180 is one example. Overall I found the organization of the paper confusing.\n- There are not enough experimental results to fully validate the method (e.g., relevant baselines like PC-grad). In all of the comparisons to baselines the approach is still fairly overlapping with the error bars of other baselines and the meaning of the error bars is not described anywhere in the text.\n\n- It would help the reader to consolidate the terms. The modules are separately referred to as both experts and modules. I would find it more straightforward if the naming was consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for their thorough review and valuable recommendations that have strengthened our paper.  Any further discussion will be appreciated.\n\n> W1: A lot of the methods section should be moved to a preliminary section because it is difficult to understand what is novel and not novel. The temporal attention section on L168-180 is one example. Overall I found the organization of the paper confusing.\n\nBoth subsections 4.1 and 4.2 in our method section is novel and thus doesn't need to be placed in the preliminary section. Our innovation lies in two aspects: contrastive learning and temporal attention. The section on temporal attention (L168-180) is indeed one of our contributions, and it diverges from the previous approach of CARE. While CARE employs task information for attention, our attention module incorporates temporal information as well. It's possible that the reviewer overlooked this distinction, leading to confusion.\n\n> W2.1: There are not enough experimental results to fully validate the method (e.g., relevant baselines like PC-grad). \n\nThe experimental results of PC-grad can be compared using Table 1 and Table 3 in the CARE paper (Sodhani et al., 2021). From these tables, it becomes evident that CARE outperforms PC-grad and is indeed a stronger baseline:\n\n|        | MT10              | MT50             |\n| ------ | ----------------- | ---------------- |\n| PCGrad | 0.72  $\\pm$ 0.022 | 0.5 $\\pm$ 0.017  |\n| CARE   | 0.84 $\\pm$ 0.051  | 0.54 $\\pm$ 0.031 |\n\nThe landscape of existing multi-task RL methods spans multiple orthogonal directions, encompassing architecture design, gradient modulation (including PCgrad), and loss weighting. Methods from different directions can indeed be combined, such as PCgrad + our method, PCgrad + CARE, and so on. However, this fusion is not the primary focus of our study. Consequently, we haven't included PCgrad in our baseline comparisons, as the baselines we have chosen primarily reside in the architecture design direction.\n\n> W2.2:  In all of the comparisons to baselines the approach is still fairly overlapping with the error bars of other baselines and the meaning of the error bars is not described anywhere in the text.\n\nNot mentioning the significance of error bars in the paper was indeed an oversight on our part, and we appreciate the reviewer's observation. The error bars (shaded areas) represent the standard deviation across 8 different seeds. In RL experiments, results can be highly influenced by the choice of seeds, making it necessary to average the outcomes across multiple seeds to mitigate the impact of randomness-induced errors. The overlapping error bars among different baselines merely indicate the substantial effect of RL's randomness in MT10. As a result, our primary mode of comparison relies on the mean performance of different algorithms.  Furthermore, in the MT50, there is no overlap among different baselines. This distinct separation clearly highlights the superiority of our approach.\n\n> W3: It would help the reader to consolidate the terms. The modules are separately referred to as both experts and modules. I would find it more straightforward if the naming was consistent.\n\nThat's a great suggestion, and we will certainly rephrase the relevant sections in the forthcoming revised version to minimize reader confusion. We appreciate the reviewer's valuable advice.\n\n> Q1: On L143 \"Notably, it surpasses the performance of learning each task individually for the first time in the Meta-World environment\" Is this true in the case of MT10? E.g., Figure 3 of Gradient Surgery for Multi-Task Learning (Yu et. al., 2020).\n\nIn Figure 3 of the PC-grad paper (Yu et al., 2020), PC-grad appears to achieve performance close to that of single task training. However, the paper does not explicitly mention averaging results across multiple seeds in the experiments. Hence, we speculate that PC-grad might have utilized only a single seed for experiments on Meta-World.  The substantial randomness inherent in reinforcement learning, as discussed in W2.2, can weaken the persuasiveness of results from single seed. Based on our experimental findings, it is evident that only our approach outperforms single task training in both MT10-fixed and MT10-mixed.\n\n> Q2: What are the error bars in the figures?\n\nSee the answer of W2.2."}, {"review_id": "RgxBUlmumO", "reviewer": "Reviewer_fVjH", "summary": "This work proposes to enhance the expressiveness and generalization capability of the modular methods in multi-task reinforcement learning by applying contrastive loss over different task modules and encode the task related information with a temporal attention module. This work shows by applying both techniques, the proposed CMTA method can learn modules producing different learned embeddings and outperform baselines in different benchmarks.", "questions": "*  Some explanation regarding the experiment results mentioned in the weakness section would be appreciated.\n* Additional visualization regarding the attention weights for different tasks. And since different modules have different semantic meaning (powered by the contrastive learning), some investigation on what kind of skill a specific expert represents would be interesting as well.\n* Though this work claims the method works without any loss weighting trick in the optimization, it would be interesting to see some results from that end, \n\nI would raise my rating if the authors could reasonably address part (given the limited time for rebuttal) or full of my concerns.", "limitations": "This work propose a general multi-task RL method, in this case, I think no specific potential negative societal impact or similar things should be addressed. As the author indicated, the current method works well in multi-task RL (with a fixed number of tasks), and would like to see some extension in meta learning or open-vocabulary settings in the future.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Different modules of the method are well-motivated, aligned with intuition and described with detail. Details provided for the community to reproduce the results. Using contrastive learning to enforce different module learn different skills is reasonable.\n* The visualization of learned encoding of different modules shows the contrastive learning term encourages the different modules to learn skills with different semantic meanings.\n* According to the experiment results, the proposed method CMTA outperformed different baselines by a large margin in different settings.\n* The overall writing of the work is easy to follow.", "weaknesses": "1. Some experiment results are not well-explained: \n* According to the Sec 5, the Mixed version of MetaWorld benchmark is supposed to be more difficult than the fixed version.  Why the Single-SAC is performing worse in the Fixed MT10, and it seems the Single-SAC results (in Fixed setting) is much worse than the reported in previous works. \n* All baselines perform worse in Mixed (compared with in Fixed), while the proposed method works better in Mixed Version. \n*  Why the performance of all methods significantly drops after a certain stage for MT50-Mixed (similar phenomenon did not appear in MT10-Mixed). \n2. Some components of the methods could be better ablated like, how does the number of experts affect the performance of the method. Since this work propose to learn more meaningful skills for different experts via contrastive learning, more discussion on this part would make the work stronger.\n3. Visualization of the attention weight for different tasks is missing, which could help the audience understand the proposed method.\n4. The use of different skill-use is discussed in previous work (Soft Modularization, where the state information and the task information are used at the same time to output the module selection), and no specific training objective (the difference with previous here seems to be the LSTM ) in this work is addressing this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere appreciation to the reviewer for their valuable insights and constructive feedback. Any further discussion will be appreciated.\n\n> W1.1.1: According to the Sec 5, the Mixed version of MetaWorld benchmark is supposed to be more difficult than the fixed version. Why the Single-SAC is performing worse in the Fixed MT10 ?\n\nMixed version has varying positions. In theory, the agent is more likely to deduce patterns or objectives of tasks (e.g., moving an object to a goal) rather than merely memorizing fixed positions. Thus it is possible for Single-SAC performing better in Miexed MT10.\n\n> W1.1.2: it seems the Single-SAC results (in Fixed setting) is much worse than the reported in previous works.\n\nIn the CARE paper, the performance of Single-SAC reached 90%. However, in our reproduction, we could only achieve this level of performance with a single seed. We raised an issue on GitHub regarding our inability to reproduce the Single-SAC results, but the authors have not responded. In the SoftModu paper, after averaging over 3 seeds, the performance of Single-SAC was reported as 78.5%. Considering the difference in seed numbers, this result closely aligns with our experimental findings.\n\n> W1.2:  All baselines perform worse in Mixed (compared with in Fixed), while the proposed method works better in Mixed Version.\n\nThis might be indicative of our method's superior generalization ability, enabling it to capture invariant patterns across different positions and thus leading to improved performance. In contrast, other methods in the mixed environment could potentially shift from memorizing a single set of positions (as in Fixed) to remembering 50 sets of positions. This shift could contribute to a decline in performance due to increased complexity and task variation.\n\n> W1.3:   Why the performance of all methods significantly drops after a certain stage for MT50-Mixed (similar phenomenon did not appear in MT10-Mixed).\n\nAll algorithms exhibit this phenomenon in the MT50-Mixed environment, and we believe this is an inherent issue with the mixed environment itself.  Most of the baselines experience performance degradation around 0.5 million steps, whereas our method's performance decline begins at 1.5 million steps, highlighting its robustness.  We speculate that the reason for this performance drop lies in the more diverse nature of the MT50-Mixed environment.  Overtraining can lead to model overfitting, causing it to prioritize simpler tasks over more challenging ones.  Our observations of individual task success rates support this, in MT10-mixed, among the 10 tasks, only one task experiences a decline in performance during the later stages(Appendix C, Figure 6, push-v1). However, in MT50-mixed, among the same set of 10 tasks, 5 tasks demonstrate performance drops, and these declines occur earlier in the training process compared to MT10-mixed.  Given that our x-axis represents steps for each task, the training data volume for MT50 is five times that of MT10, which can lead to quicker overtraining issues.\n\n> W2:  Some components of the methods could be better ablated like, how does the number of experts affect the performance of the method. \n\nThe ablation of experts number can be seen in the pdf of global response. The experimental results indicate that having fewer experts leads to performance degradation. However, increasing the number of experts beyond a certain point does not yield positive effects.\n\n> W3:  Visualization of the attention weight for different tasks is missing, which could help the audience understand the proposed method.\n\nSee  the t-sne visualization of attention weights for different tasks in the pdf of global response.  It is obvious that the attention weights of different tasks are clustered into different clusters, which indicates that CMTA will choose different module combinations for different tasks.\n\n> W4:  The use of different skill-use is discussed in previous work (Soft Modularization, where the state information and the task information are used at the same time to output the module selection), and no specific training objective (the difference with previous here seems to be the LSTM ) in this work is addressing this issue.\n\nIn comparison to SoftModu, our method differs not only in the utilization of LSTM but also in the specifics of attention computation and their application. For instance, while we concatenate temporal information with task information to calculate attention, SoftModu employs a dot product between state information and task information, involving multiple layers of attention weight computation.\n\n> Q1: Some explanation regarding the experiment results mentioned in the weakness section would be appreciated.\n\n See the answer of W1.\n\n> Q2:  Additional visualization regarding the attention weights for different tasks. \n\n See the answer of W3.\n\n> Q3:  Though this work claims the method works without any loss weighting trick in the optimization, it would be interesting to see some results from that end.\n\nWe add a relatively simple loss weighting trick in our method:  let the task weightings $\\lambda_i$ (see euqation 1 in our paper)  is propotional to  exp(1/($su_i + \\delta$)), where $su_i$ is the current success rate of task i.  The evaluation perfomance(average on 8 seeds) after training 1 million steps is:\n\n|                            | smoothed SR on MT10-Mixed |\n| -------------------------- | ------------------------- |\n| CMTA                       | 78.5                      |\n| CMTA+ naive loss weighting | 73.6                      |\n\n\nInterestingly, the introduction of this trick resulted in a decrease in the performance of our method. This might suggest that the naive loss weighting we attempted may not be effective as initially thought. Additionally, there exist numerous studies on loss weighting in MTRL, all of which can potentially be integrated with our method or the baselines we've utilized."}, {"review_id": "T4H9tIzeSa", "reviewer": "Reviewer_sdV7", "summary": "The paper studies modular multi-task reinforcement learning to address negative transfer problem. The proposed method has two components: 1. contrastive learning on module outputs, to encourage model expressiveness and generalization. 2. use temporal information to combine module outputs, to address negative transfer within tasks. Experiments in Meta-World shows that the proposed method outperforms other multi-task baselines and learning tasks individually.", "questions": "1. Why are the results in Figure 3 quite different to the results in \"Soft Modularization\". In that paper, Soft Modularization outperforms multi-task variants of SAC a lot.\n\n2. Since most multi-task methods cannot outperform the individual-learning baseline, what are the benefits of multi-task RL? Can the learned multi-task model adapt to unseen tasks quicker?", "limitations": "The authors have addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for modular learning with temporal attention to address negative transfer is clear.\n\n2. The paper is very well-written, with relevant references and easy-to-follow narration.\n\n3. The method is the only one outperforming single-task RL in Meta-World.", "weaknesses": "1. The experiments don't reflect the claim that the method improves generalization.\n\n2. Some previous works (e.g., Multi-Task Reinforcement Learning with Soft Modularization) also select different modules in different time steps in a task. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback and would like to thank you for your time and effort in reviewing our manuscript.  Any further discussion will be appreciated.\n\n> W1: The experiments don't reflect the claim that the method improves generalization.\n\nWe believe that the following changes in settings have contributed to increased task complexity and diversity:\n\n1. The number of tasks increasing from 10 (MT10) to 50 (MT50).\n2. Transitioning from fixed positions to varying positions (mixed).\n\nIn both of these scenarios, our method exhibits greater advantages over other baselines (as indicated in lines 259-262). This to a certain extent underscores the superior generalization ability of our approach.\n\n> W2: Some previous works (e.g., Multi-Task Reinforcement Learning with Soft Modularization) also select different modules in different time steps in a task.\n\nIn comparison to SoftModu, our method differs not only in the utilization of LSTM but also in the specifics of attention computation and their application. For instance, while we concatenate temporal information with task information to calculate attention, SoftModu employs a dot product between state information and task information, involving multiple layers of attention weight computation.\n\n> Q1:  Why are the results in Figure 3 quite different to the results in \"Soft Modularization\". In that paper, Soft Modularization outperforms multi-task variants of SAC a lot. \n\nWe mentioned this point in Section 5.4 Lines 277-283. The original SoftModu paper employed a loss weighting trick, which we deliberately omitted in our experiments to ensure fairness in comparison. Additionally, Figure 6(b) in the SoftModu paper (Yang et al., 2020) displays the performance without utilizing this trick, and it aligns quite closely with our reproduction results.\n\n> Q2.1:  Since most multi-task methods cannot outperform the individual-learning baseline, what are the benefits of multi-task RL? \n\n**Better Sample Efficiency:** This advantage can be interpreted as each task benefiting from additional samples generated by auxiliary tasks. Consequently, during the initial training stages, the performance improvement of multi-task rl algorithms outpace that of Single-SAC.\n\n**Model Capacity Compression:** By employing a single network to address all tasks instead of using n separate networks.\n\n> Q2.2: Can the learned multi-task model adapt to unseen tasks quicker?\n\nThis outcome hinges on whether the multi-task model merely memorizes multiple tasks or is capable of extracting commonalities among tasks."}]

Author rebuttal:
The PDF here includes our ablation experiments on the number of experts and the t-SNE visualization of CMTA attention weights of different tasks.

Meta-review: The paper presents a novel solution to the negative transfer problem in multi-task reinforcement learning. It introduces the CMTA method, which employs contrastive learning to ensure diverse modules and uses temporal attention to combine them effectively. The paper's strengths include its novel approach, compelling experimental results outperforming benchmarks, simple implementation, and clear presentation. The author should aim to incorporate the suggestions put forward by reviewers, in particular, those around discussing limitations, additional visualisations, and clarifications.
Decision: Accept (poster)