Title: Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

multi-task learning

contrastive learning
Abstract: In the field of multi-task reinforcement learning, the modular principle, which involves specializing functionalities into different modules and combining them appropriately, has been widely adopted as a promising approach to prevent the negative transfer problem that performance degradation due to conflicts between tasks. However, most of the existing multi-task RL methods only combine shared modules at the task level, ignoring that there may be conflicts within the task. In addition, these methods do not take into account that without constraints, some modules may learn similar functions, resulting in restricting the model's expressiveness and generalization capability of modular methods.
In this paper, we propose the Contrastive Modules with Temporal Attention(CMTA) method to address these limitations. CMTA constrains the modules to be different from each other by contrastive learning and combining shared modules at a finer granularity than the task level with temporal attention, alleviating the negative transfer within the task and improving the generalization ability and the performance for multi-task RL.
We conducted the experiment on Meta-World, a multi-task RL benchmark containing various robotics manipulation tasks. Experimental results show that CMTA outperforms learning each task individually for the first time and achieves substantial performance improvements over the baselines.

Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning

Abstract

In the field of multi-task reinforcement learning, the modular principle, which involves specializing functionalities into different modules and combining them appropriately, has been widely adopted as a promising approach to prevent the negative transfer problem that performance degradation due to conflicts between tasks. However, most of the existing multi-task RL methods only combine shared modules at the task level, ignoring that there may be conflicts within the task. In addition, these methods do not take into account that without constraints, some modules may learn similar functions, resulting in restricting the model's expressiveness and generalization capability of modular methods. In this paper, we propose the Contrastive Modules with Temporal Attention(CMTA) method to address these limitations. CMTA constrains the modules to be different from each other by contrastive learning and combining shared modules at a finer granularity than the task level with temporal attention, alleviating the negative transfer within the task and improving the generalization ability and the performance for multi-task RL. We conducted the experiment on Meta-World, a multi-task RL benchmark containing various robotics manipulation tasks. Experimental results show that CMTA outperforms learning each task individually for the first time and achieves substantial performance improvements over the baselines. Our code can be found at https://github.com/niiceMing/CMTA.

1 Introduction

Though deep reinforcement learning (RL) has made significant progress in multiple domains such as playing Atari games [22] and robotic control[16], most of these methods tackle different tasks in isolation, making it difficult to leverage the previously learned skills for a new task. Multi-task RL aims to train multiple tasks simultaneously in a sample efficient manner [6] by using shared

and re-used components across different tasks. Since different tasks may have complementary information or act as a regularizer for each other, multi-task RL methods have the potential to get better performance and generalization [4].

There is a vital challenge in multi-task RL if two tasks are essentially unrelated or have conflicts, increasing the performance of the method on one task will hurt the performance on the other, referred as negative transfer problem [42, 37]. The occurrence of negative transfer is attributed to the utilization of the same model to learning different tasks or different functions. When conflicts arise, the model needs to compromise among multiple tasks, which often results in inferior performance compared to training each task individually. To tackle this problem, researchers propose to exploit the task relationships [35, 34] along with basic shared models, including task-specific decoding head[17], routing networks [10, 29], cross-stitch networks [21], compositional modules[8, 44, 2] and mixture of experts[9, 20, 34]. These methods use different combinations of shared modules or different parts of the shared model according to different tasks, which can be seen as an embodiment of modular approaches.

The main idea of modular methods is to separate and specialize the functionalities into different modules , allowing them to focus on their respective skills or features without interference or conflicting objectives. Different modules can collaborate and complement each other's strengths. By appropriately combining these distinct modules, it is theoretically possible to resolve the trade-off dilemma encountered or negative transfer in multi-task RL and enhance the model's expressive power and generalization ability. And it has been observed in neuroscience research that different regions of the brain possess completely different functions, aligning with the concept of modularity. This may be one of the reasons why humans are capable of rapidly learning multiple tasks.

Despite the widespread adoption of modular principle in most multi-task RL methods, these methods still underperform compared to single-task training. We argue that the inferior performance of these works may be attributed to the fact that they have not fully adhered to the principle of modularity, which hinders their ability to mitigate negative transfer. One of the key aspects of modularity is ensuring that different modules specialize in distinct functionalities, but these methods only partition the network into multiple modules in a superficial way without further constraints which may result in some modules learning the same functionality. This goes against the modular principle and restricts the model's expressiveness and generalization capability of modular methods and leads to redundant computations and even overfitting. Another challenge lies in the appropriate combination of modules. Most methods only combine shared modules at the task level, using a fixed combination of modules for each task during the whole episode. However, negative transfer exists not only between tasks but also within tasks. For instance, considering the task 'Push' in Meta-World(Figure 1), the agent needs to reach the puck at first and then push it to a goal, these two purposes are not identical and not always be mutually supportive, so negative transfer may exist when using a same module combination at these time steps.

In this paper, we propose a novel approach called Contrastive Modules with Temporal Attention (CMTA) to address the limitations of existing multi-task RL methods and further alleviate negative transfer in multi-task RL. The key insight lies in constraining the modules to be different from each other by contrastive learning and combining shared modules at a finer granularity than the task level with temporal attention. Roughly speaking, CMTA uses LSTM to encode the temporal information, along with a task encoder to encode the task information. Then CMTA takes both temporal and task information to compute the soft attention weights for the outputs(encodings) of modules, and we apply contrastive loss to them, aiming to increase the distance between different encodings. Therefore, CMTAenforces the distinctiveness of modules and provides the agent with a finer-grained mechanism to dynamically select module combinations within an episode, alleviating the negative transfer within the task and improving the generalization ability and the performance for multi-task RL.

We conduct experiments on Meta-World[48], a multi-task RL benchmark containing 50 robotics manipulation tasks. Experimental results show that the proposed CMTA significantly outperforms all baselines, both on sample efficiency and performance, and even gets better performance than learning each task individually for the first time. CMTA also shows great superiority especially for complex tasks. As the variety of tasks increases, the advantages of CMTA become more apparent.

Reach the handle

Push the handle to goal

Success

2 Related work

Multitask Learning. Multi-task learning (MTL) can learn commonalities and differences across different tasks. [4, 6, 7, 41, 30]. Researchers have shown MTL can make multiple tasks benefit from each other by sharing useful information and exploiting the commonality and structure of these tasks [18, 28, 19, 27]. [49] shows that different tasks have different similarities, and some unrelated tasks may have negative impact on each other when training together. A large portion of the MTL work is devoted to the design of multi-task neural network architectures that can avoid negative transfer, [31, 17] use a shared encoder that branches out into task-specific decoding heads. [21] proposed Cross-stitch networks, which softly share their features among tasks, by using a linear combination of the activations found in multiple single task networks. Multi-Task Attention Network [19] consists of a single shared network containing a global feature pool, together with a soft-attention module for each task.

Compositional modules and Mixture of experts. In order to reduce the negative impact of gradient conflict, a feasible method is to use compositional modules, which makes the model have better generalization. [3] leverages the compositional structure of questions in natural language to train and deploy modules specifically catered for the individual parts of a question. [8] decomposes policies into 'task-specific' (shared across robots) and 'robot-specific' (shared across tasks) modules, and the policy is able to solve unseen tasks by re-combining the pre-defined modules. Routing Networks [29] is comprised of a router and a set of neural network modules, and the composition of modules is learned in addition to the weights of the modules themselves. Instead of directly selecting routes for each task like Routing Networks[29], soft modularization[44] uses task-specific policy softly combine all the possible routes.

In the deep learning era, ensemble models and ensemble of sub-networks have been proven to be able to improve model performance[38, 15]. [9]turn the mixture-of-experts model into basic building blocks (MoE layer). The MoE layer selects experts based on the input of the layer at both training time and serving time. [23] selects one specific expert at a time for each input. [32] use a gating function to select experts. [20] use multi-gate method, and introduce a gating network for each task.

Task embedding. Task embedding is a very general form of learning task relationships, and has been widely used in meta-learning, as well as multi-task learning. It uses task information to model the explicit relationships between tasks, such as clustering tasks into groups by similarity, and leveraging the learned task relationships to improve learning on the tasks at hand. [1] uses task embedding to generate vectorial representations of visual classification tasks which can be used to reason about the nature of those tasks and their relations. [14] uses metric learning to create a task embedding that can be used by a robot to learn new tasks from one or more demonstrations. [34] uses metadata, a pre-trained task embedding obtained by task description and a pre-trained language model, to help the robot make use of the relation between continuous control tasks.

Contrastive Learning. Contrastive Learning is a framework that aims to learn representations that adhere to similarity constraints in a dataset organized into pairs of similar and dissimilar items. This can be thought of as performing a dictionary lookup task, where the positive and negative pairs represent a set of keys relative to a query (or anchor). One of the most basic forms of contrastive learning is Instance Discrimination [43], in which a query and a key are considered positive pairs if they are data augmentations of the same instance (e.g. image) and negative pairs otherwise. However, choosing appropriate negative pairs can be a significant challenge in contrastive learning, as it can greatly impact the quality of the learned representations. There are several loss functions available for use in contrastive learning, such as InfoNCE [39], Triplet[36], Siamese [5], and others.

3 Preliminaries

Markov Decision Process (MDP) is defined by a tuple ( S, A, P, R, γ ) where S is a finite set of states, A is a finite set of actions, transition function P ( s ′ | s, a ) is a transition probability from state s to state s ′ after action a is executed, reward function R ( s, a ) is the immediate reward obtained after action a is performed, and discount factor γ ∈ (0 , 1) . We denote π ( a | s ) as the policy which is a mapping from a state to an action. The goal of an MDP is to find an optimal policy to maximize the reward function. More specifically, in finite horizon MDP, the objective is to maximize the expected discounted total reward which is defined by max π E τ [ ∑ T t =0 γ t R ( s t , a t )] , where τ is trajectory following a t ∼ π ( a t | s t ) , s t +1 ∼ P ( s t +1 | s t , a t ) . The value function of policy π is defined as V π ( s t ) = E π [ ∑ T t =0 γ t R ( s t , a t )] .

Multi-task RL optimization. In general multi-task learning with N tasks, we optimize the policy to maximize the average expected return across all tasks, and the total loss function is defined as:

This is the linear combination of task-specific losses L i with task weightings λ i .

Contrastive learning. Given a query q , positive keys k + and negative keys k -, the goal of contrastive learning is to ensure that q matches with k + more than k -. An effective contrastive loss function, called InfoNCE[39], is:

where τ is a temperature hyper-parameter.

4 Method

Wenowintroduce our novel multi-task RL architecture, Contrastive Modules with Temporal Attention, which combines different modules with finer granularity by using temporal attention and contrastive modules. CMTA has the ability to alleviate negative transfer within the task, improve performance, enhance generalization, and achieve state-of-the-art results. Notably, it surpasses the performance of learning each task individually for the first time in the Meta-World 2 environment[48].

4.1 Contrastive Modules

Employing a modular approach to reuse various modules at different time steps can result in enhanced performance and better generalization. Ensemble models and the mixture of experts have been proven to be able to improve model performance [38, 15] and we use mixture of experts as our shared modules. However, without proper constraints, some modules may redundantly learn the same skill or feature, which goes against the original intention of using the modular approach, limiting the generalization ability of modular methods and only adding computational overhead without improving performance.

To tackle this problem, we introduce contrastive learning to enforce the distinctiveness of modules. Most researchers use contrastive learning to obtain a more robust representation, whereas we use contrastive learning to encourage the modules to be different from each other.

̸

At time step t, for i -th encoder's output z i t as query q i , we use other encoder's output z j t ( j = i ) as its negative pairs k -i , and the output at the next time step of i -th encoder z i t +1 is selected as the positive pair k + i . The total contrastive loss can be written as:

L

contrastive

Expert 1

Expert k

Policy

Lactor

network

Attention

Zenc

Lcritic

LSTM

ht

p

network

ht-1

Embedding

MLP

task

task encoder

where K denotes the number of experts.

4.2 Temporal attention

Most methods neglect the negative transfer within tasks by only combining shared modules at the task level, using a fixed combination of modules for each task throughout the entire episode. To address this issue, we propose combining the modules with finer granularity at each time step by using temporal attention, further alleviating negative transfer within tasks.

Mixture of experts. We use mixture of experts as our shared modules, implemented by k individual encoders. Each representation z j enc output by the corresponding expert, encodes different features of the state s t , and can be represented as:

where f j denotes the j -th expert.

Temporal Attention module. In order to combine the shared modules dynamically, we use attention module to output the soft combination weights.

Given the current state s t , we encode temporal information h t as:

where lstm is a long short-term memory [13], h t -1 is previous hidden state at time t -1 , and h 0 is initialized with zero vector.

Given the input one-hot task id Z τ , the task information can be computed as:

where g is the task encoder. Notably, [34] uses a pre-trained embedding layer trained by task description and language model, called metadata. They argue that knowing the task description such as 'open a door' and 'open a drawer' is helpful for modeling the task relationships.

By concatenating the temporal information h t and task information z task , we can dynamically compute appropriate attention weights within the episode as:

where W is a fully connected layer, and softmax is a normalize function which restricts the sum of α i to 1.

And the final representation is a weighted sum of k multiple representations using soft attention weights α as:

which can be seen as a combination of different modules.

4.3 Training approach

We use Soft Actor-critic (SAC)[12] to train our policies. SAC is an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. The optimization objectives of policy and Q-function are:

where α is a temperature parameter that controls how important the entropy term is. Note that we only use actor loss to update the policy network, and use critic loss to update the whole model. The algorithm is described in Appendix A.

For the multi-task optimization with N tasks, we optimize the policy without any loss weighting tricks, which means we treat each task equally. The total loss is computed as:

where L i denotes the loss function of task i , β is a hyper-parameter.

5 Experiment

In this section, we evaluate our method on multi-task robotics manipulation. We introduce the environment in Section 5.1, propose an appropriate evaluation metric in Section 5.3 , and compare with baseline methods on Meta-World in Section 5.2 and Section 5.4 and conduct ablation study in Section 5.5.

5.1 Environment

We evaluate the effectiveness of our CMTA model on Meta-World environment[48], which is a collection of robotic manipulation tasks designed to encourage research in multi-task RL. We use the mean success rate of the binary-valued success signal as our evaluation metric, which has been clearly defined in Meta-World[48]. Meta-World consists of 50 distinct robotic manipulation tasks with a simulated Sawyer robotic arm, each task has its own parametric variations 50 sets of different initial positions of the object and the goal. The multi-task RL benchmarks within Meta-World are MT10 and MT50, which consist of simultaneously learning 10 and 50 tasks. In the original setting, MT10 and MT50 only use one fixed position for each task during the training stage, which may cause the agent to simply memorize the position and overfitting. To reflect the generalization ability of the model, we mix all 50 sets of positions together for tasks in MT10 and MT50 at training stage, and name them as MT10-Mixed and MT50-Mixed . The original MT10 and MT50 is denoted as MT10-Fixed and MT50-Fixed .

MT10-Fixed

MT50-Fixed

0.8 f

MT50-Mixed

+ 8°0

0.8

0.7

0.6

0.7

0.6

ssaoons

0.6

ssaoons

0.6

ssaosns

0.4

0.5

ssasans

0.5

0.4

0.4

0.4

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1 -

0.0

0.0

0.0

0.0

0.0

0.5

1.0

Million step

1.5

2.0

2.5

0.0

0.5

1.0

Million step

1.5

2.0

2.5

0.0

0.5

1.0

Million step

1.5

2.0

2.5

0.0

0.2

0.4

0.6

0.8

1.0

1.2

MT-SAC

MT-SAC +TE

MTMH-SAC

SoftModularization

care

CMTA(ours)

single-SAC

5.2 Baselines

We take the following multi-task RL methods as the object of our comparison, considered as the baselines: (i)Single-task SAC(upper bound) : Using a separate network for each task, which is usually considered as the performance upper bound. (ii)Multi-task SAC(MT-SAC[48]) : Using a shared network with the state as input. (iii)Multi-task SAC with task encoder(MT-SAC+TE) : Adding a task encoder with a one-hot task id as input based on MT-SAC. (iv)Multi-task multihead SAC (MTMH-SAC[48]) : Using a shared network with task-specific heads for each task. (v)Soft Modularization(SoftModu[44]) : It contains several compositional modules as the policy network, and uses a task-specific routing network softly combine the module in the policy network. (vi)Contextual attention-based representation (CARE[34]) : CARE uses a mixture of encoders, and uses a context encoder with additional prior knowledge metadata to context-based attention.

5.3 Implementation details

The experimental results of reinforcement learning environments like Meta-World have great randomness, and the number of seeds plays an important part in the final performance of the model. Specifically, as mentioned in [34], MTMH-SAC achieved a success rate of 88% (1 seed) in [48], which is 44% different from 10seed's performance. In our experiment, each agent is trained over 8 seeds, and we average the mean success rate of each seed to measure the performance. We train the agent with 2.4 million steps (per task) for both MT10 and MT50 environments. More implementation details of hyper-parameters can be found in Appendix D.

During training phase, we evaluate the agent every 3K steps (20 episodes), and we find that the smoothed testing curve and the smoothed training curve are almost identical. It might be the case that the temperature parameter α becomes very small in the training period, which discourages exploration, that is to say, the policy has the same behavior without stochasticity in the training and testing phase.

Even though the testing results have been averaged over 8 seeds, it is still very volatile. Previous work often use the maximum of the mean success rate in evaluation, which is much higher than the average, since the max operation will bring much larger variances. In our experiments with 8 seeds, there is a big gap between the maximum of the mean success rate and the maximum of the smoothed (smooth factor 0.8) mean success rate, which is about 5% ∼ 10%, bringing more uncertainty to the evaluation of model performance. Predictably, this uncertainty is much bigger in the case of fewer seeds. [44, 34] tries to reduce uncertainty by evaluating the agent several times at once, but it doesn't really work. Because in the original MT10 and MT50, each task environment has a fixed initial position, and the policy based on SAC is determined during evaluating. In other words, given the parameters of the model, no matter how many times we evaluate the agent, the result is exactly the same as the first time in the testing stage. Thus, compared to the maximum of the mean success rate, we use the maximum of the smoothed mean success rate to measure the performance of models, which is more stable and realistic.

5.4 Results

To evaluate the performance, we compare CMTA with other baselines on MT10-Fixed, MT10-Mixed, MT50-Fixed and MT50-Mixed environments. The training curves averaged over 8 seeds are plotted in Figure 3. And Table 1 records the best evaluation results of all the methods, including both max smoothed (with smooth factor 0.8, details can be found in APPENDIXC) mean success rate and max mean success rate (large variance). It is noteworthy that our method is even better than CARE, which uses additional prior knowledge.

As shown in Figure 3, our approach consistently outperforms baselines in all environments, whether on asymptotic performance or sample efficiency. In the original environments MT10-Fixed and MT50-Fixed, it can be seen from Table 1 that CMTA is better than other baselines around 15% and 24%, respectively. And in the more difficult environment with changing positions, MT10-Mixed and MT50-Mixed, the superiority of our method is more obvious, which is about 34% and 60% better than other baselines respectively.

When changing from Fixed environment to Mixed environment, it becomes more difficult because the agent have to deduce patterns or objectives of tasks (e.g., moving an object to a goal) rather than merely memorizing fixed positions. All the baseline methods suffered this impact in MT10-Mixed and MT50-Mixed, but our method achieves even better performance, which implies good generalization and robustness of our method.

Significantly, we notice that only our approach outperforms learning each task individually (we only conduct this experiment on MT10, because it is too slow to train 50 policies simultaneously on MT50). This evidence proves that our approach can avoid negative transfer and improve the performance of tasks by correctly combining the modules. While all the methods can take advantage of multi-task learning to compress the model and improve sample efficiency, in contrast to ours, other methods come at the cost of performance.

As mentioned before, RL experimental result has large stochasticity, and the evaluation metric used in previous works ([48, 44, 34]) is max success rate during testing, which leads to a large variance and is influenced by evaluation times. Thus we use max smoothed success rate to mitigate the effects of large variances caused by max operations. Because of these randomnesses, it is reasonable that the results are somewhat different from the previous works. However, the result of soft modularization[44] in our experiment is much worse than previous work. In addition to the number of seeds and the randomness of the RL algorithm, we guessed that the most likely reason is that SoftModu uses a loss weighting trick in the optimization, which balances the individual loss for different tasks. For the sake of fairness, we don't use any loss weighting method. The ablation study of SoftModu[44] also shows that MTMH-SAC outperforms SoftModu without balance, which is consistent with our experimental results.

MT10-Fixed

0.8

0.8

0.6

0.6

success

success

0.4

0.4

0.2

0.2

CMTA(ours)

CMTA(ours)

0.0

CMTA w/o T1

0.0

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.5

1.0

1.5

2.0

2.5

Million step

Million step

5.5 Ablation Study

Temporal Attention. We investigate the significance of temporal information and report the comparison results in Figure 4. It can be seen that the temporal information we employed is indeed highly effective, which indicates that the negative transfer within tasks is further alleviated.

Contrastive Modules. We conducted comparative experiments to validate the effectiveness of contrastive loss, as shown in Table 8. From the results, it can be seen that the CL loss consistently improves performance in the mixed environment, especially in MT50-Mixed where our method achieves an increase in success rate of nearly 10 percentage points. Additionally, we observed that the impact of CL loss in the fixed environment is not substantial. This could be attributed to the fixed environment being overly homogeneous, which makes it prone to overfitting and fails to demonstrate the robust generalization advantage of contrastive modules.

To verify whether contrastive learning can constrain modules to learn unique functions and reduce similarity between them, we use t-SNE[40] to visualize the output embeddings of K(6 in our setting) multiple modules in Figure 5. The visualization of CMTA and CMTA without CL are obtained by multiple embeddings of modules at each time step on MT10-Fixed for 10 episodes. It can be observed that without contrastive learning constraints, the encodings of multiple modules are mixed together.

s

(a) CMTA w/o CL

(b) CMTA

By using CL, the encodings of different modules become more distinct, indicating that CL indeed reduces the similarity between modules.

6 Limitation

One limitation is the computational cost of CMTA, the contrastive learning part is proportional to O ( n 2 ) ,where n is number of experts. If task similarity is substantial, then even with an increase in the number of tasks, computational cost won't necessarily escalate as long as the number of experts remains constant. Another limitation is the dependence on task similarity, and it's the fundamental assumption in the field of mtrl, even humans struggle to extract mutually beneficial information from entirely unrelated tasks. For instance, attempting to learn from a combination of tasks like playing Go and playing Atrai games would likely yield limited benefits.

7 Conclusion

In this paper, we present contrastive modules with temporal attention, which is a generalized way of learning distinct shared modules with finer granularity for multi-task RL. CMTA has the ability to constrain the modules to be different from each other by contrastive learning and the ability to learn the best combination of distinct shared modules dynamically at each time step across tasks by using temporal attention. The module combination in CMTA varies form step to step, so that modules can be reused when it's helpful and not shared when it's conflict, alleviating the negative transfer between within the task and shows great superiority especially for complex tasks. Experiments on Meta-World show that our method outperforms all baselines, both on sample efficiency and performance, and it's the first time achieved performance surpassing that of training each task separately. As the variety of tasks increases(from MT10 to MT50, from Fixed environment to Mixed environment), the advantages of our approach become more apparent. Our method has achieved good results in multi-task RL, and we will further extend it to meta learning in future work.

References

Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6429-6438, 2019.

Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning , 2016.

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 39-48, 2015.

Rich Caruana. Multitask learning. Machine Learning , 28:41-75, 1997.

Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , 1:539-546 vol. 1, 2005.

Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In International Conference on Machine Learning , 2008.

Michael Crawshaw. Multi-task learning with deep neural networks: A survey. ArXiv , abs/2009.09796, 2020.

Coline Devin, Abhishek Gupta, Trevor Darrell, P. Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 2169-2176, 2016.

David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. CoRR , abs/1312.4314, 2013.

Chrisantha Fernando, Dylan S. Banarse, Charles Blundell, Yori Zwols, David R Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. ArXiv , abs/1701.08734, 2017.

Jiaming Guo, Rui Zhang, Xishan Zhang, Shaohui Peng, Qiaomin Yi, Zidong Du, Xing Hu, Qi Guo, and Yunji Chen. Hindsight value function for variance reduction in stochastic dynamic environment. In International Joint Conference on Artificial Intelligence , 2021.

Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , 2018.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9:1735-1780, 1997.

Stephen James, Michael Bloesch, and Andrew J. Davison. Task-embedded control networks for few-shot imitation learning. ArXiv , abs/1810.03237, 2018.

Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation , 6:181-214, 1993.

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR , abs/1509.02971, 2015.

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Tak Wu Kwong. Pareto multi-task learning. In Neural Information Processing Systems , 2019.

Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. ArXiv , abs/1605.05101, 2016.

Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1871-1880, 2018.

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2018.

Ishan Misra, Abhinav Shrivastava, Abhinav Kumar Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3994-4003, 2016.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv , abs/1312.5602, 2013.

Giambattista Parascandolo, Mateo Rojas-Carulla, Niki Kilbertus, and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning , 2017.

Shaohui Peng, Xing Hu, Qi Yi, Rui Zhang, Jiaming Guo, Di Huang, Zikang Tian, Ruizhi Chen, Zidong Du, Qi Guo, et al. Self-driven grounding: Large language model agents with automatical language-aligned skill learning. arXiv preprint arXiv:2309.01352 , 2023.

Shaohui Peng, Xing Hu, Rui Zhang, Jiaming Guo, Qi Yi, Rui Chen, Zidong Du, Ling Li, Qi Guo, and Yunji Chen. Conceptual reinforcement learning for language-conditioned tasks. In AAAI Conference on Artificial Intelligence , 2023.

Shaohui Peng, Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Rui Chen, Xishan Zhang, Zidong Du, Ling Li, Qi Guo, and Yunji Chen. Causality-driven hierarchical structure discovery for reinforcement learning. ArXiv , abs/2210.06964, 2022.

Lerrel Pinto and Abhinav Kumar Gupta. Learning to push by grasping: Using multiple tasks for effective learning. 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 2161-2168, 2016.

Rajeev Ranjan, Vishal M. Patel, and Rama Chellappa. Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41:121-135, 2019.

Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. ArXiv , abs/1711.01239, 2017.

Sebastian Ruder. An overview of multi-task learning in deep neural networks. ArXiv , abs/1706.05098, 2017.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Neural Information Processing Systems , 2018.

Satinder Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning , 8:323-339, 2004.

Shagun Sodhani and Amy Zhang. Mtrl - multi task rl algorithms. Github, 2021.

Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9767-9779. PMLR, 18-24 Jul 2021.

Anders Søgaard and Joachim Bingel. Identifying beneficial task relations for multi-task learning in deep neural networks. In Conference of the European Chapter of the Association for Computational Linguistics , 2017.

Nitish Srivastava. Unsupervised learning of visual representations using videos. 2015.

Yee Whye Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Manfred Otto Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. ArXiv , abs/1707.04175, 2017.

Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick Mcdaniel. Ensemble adversarial training: Attacks and defenses. ArXiv , abs/1705.07204, 2017.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv , abs/1807.03748, 2018.

Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9:2579-2605, 2008.

Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44:3614-3633, 2020.

Sen Wu, Hongyang Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. ArXiv , abs/2005.00944, 2020.

Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance-level discrimination. ArXiv , abs/1805.01978, 2018.

Ruihan Yang, Huazhe Xu, YI WU, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 4767-4777. Curran Associates, Inc., 2020.

Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Yunkai Gao, Kaizhao Yuan, Rui Chen, Siming Lan, Xing Hu, Zidong Du, Xishan Zhang, Qi Guo, and Yunji Chen. Online prototype alignment for few-shot policy transfer. In International Conference on Machine Learning , 2023.

Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Xing Hu, Zidong Du, Qi Guo, Rui Chen, Ling Li, and Yunji Chen. Learning controllable elements oriented representations for reinforcement learning. Neurocomputing , 549:126455, 2023.

Qi Yi, Rui Zhang, Shaohui Peng, Jiaming Guo, Xing Hu, Zidong Du, Xishan Zhang, Qi Guo, and Yunji Chen. Object-category aware reinforcement learning. ArXiv , abs/2210.07802, 2022.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning , volume 100 of Proceedings of Machine Learning Research , pages 1094-1100. PMLR, 30 Oct-01 Nov 2020.

Amir Roshan Zamir, Alexander Sax, Bokui (William) Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3712-3722, 2018.

A Pseudo Code

Initialize : policy π with ϕ , Q-function Q, task encoder g , k experts f 1 , · · · , f k , lstm , fully connected layer W

B Libraries

We use the following open-source libraries: MetaWorld 3 , MTEnv 4 ,MTRL 5 [33].

C Smooth factor

D Hyperparameter Details.

E Additional Experiment Results

reach-v1

push-v1

pick-place-v1

door-open-v1

1.0

1.0

0.6

mW

1.0

8°0

0.8

0.5

0.8

0.4

0.2

0.2

0.1

0.0

0.0

0.0

0.0

0.0

0.5

1.0

Million step

1.5

2.0

2.5

0.0

0.5

Million step

1.0

1.5

2.0

2.5

0.0

1.0

Million step

1.5

2.0

0.0

0.5

1.0

Million step

1.5

2.0

2.5

drawer-open-v1

drawer-close-v1

button-press-topdown-v1

peg-insert-side-v1

1.0

1.0

0.08

8°0

0.8

0.06

0.04

9.4

0.02

0.2

0.0

0.0

0.00

0.0

0.5

1.0

Million step

1.5

2.0

2.5

0.0

0.5

daps uo

1.0

1.5

2.0

2.5

00

0.5

1.0

Million step

1.5

2.0

2.5

0°0

0.5

1.0

Million step

1.5

2.0

2.5

window-open-v1

window-close-v1

0.8

0.8

0.6

0.6

0.2

0.0

0.0

0.5

1.0

Million step

1.5

2.0

0.0

0.5

1.0

Milion step

1.5

2.0

MT-SAC

MT-SAC+TE

MTMH-SAC

SoftModularization

care

CMTA(ours)

single-SAC

reach

ado-roop

button-press-topdown

window-open

push

drawer-open

peg-insert-side

window-close

pick-place

drawer-close

Peer reviews:
comments: SUMMARY:
The paper studies modular multi-task reinforcement learning to address negative transfer problem. The proposed method has two components: 1. contrastive learning on module outputs, to encourage model expressiveness and generalization. 2. use temporal information to combine module outputs, to address negative transfer within tasks. Experiments in Meta-World shows that the proposed method outperforms other multi-task baselines and learning tasks individually.

STRENGTHS:
1. The motivation for modular learning with temporal attention to address negative transfer is clear.

2. The paper is very well-written, with relevant references and easy-to-follow narration.

3. The method is the only one outperforming single-task RL in Meta-World.

WEAKNESSES:
1. The experiments don't reflect the claim that the method improves generalization.

2. Some previous works (e.g., Multi-Task Reinforcement Learning with Soft Modularization) also select different modules in different time steps in a task.

QUESTIONS:
1. Why are the results in Figure 3 quite different to the results in "Soft Modularization". In that paper, Soft Modularization outperforms multi-task variants of SAC a lot.

2. Since most multi-task methods cannot outperform the individual-learning baseline, what are the benefits of multi-task RL? Can the learned multi-task model adapt to unseen tasks quicker?

LIMITATIONS:
The authors have addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to enhance the expressiveness and generalization capability of the modular methods in multi-task reinforcement learning by applying contrastive loss over different task modules and encode the task related information with a temporal attention module. This work shows by applying both techniques, the proposed CMTA method can learn modules producing different learned embeddings and outperform baselines in different benchmarks.\n\nSTRENGTHS:\n* Different modules of the method are well-motivated, aligned with intuition and described with detail. Details provided for the community to reproduce the results. Using contrastive learning to enforce different module learn different skills is reasonable.\n* The visualization of learned encoding of different modules shows the contrastive learning term encourages the different modules to learn skills with different semantic meanings.\n* According to the experiment results, the proposed method CMTA outperformed different baselines by a large margin in different settings.\n* The overall writing of the work is easy to follow.\n\nWEAKNESSES:\n1. Some experiment results are not well-explained: \n* According to the Sec 5, the Mixed version of MetaWorld benchmark is supposed to be more difficult than the fixed version.  Why the Single-SAC is performing worse in the Fixed MT10, and it seems the Single-SAC results (in Fixed setting) is much worse than the reported in previous works. \n* All baselines perform worse in Mixed (compared with in Fixed), while the proposed method works better in Mixed Version. \n*  Why the performance of all methods significantly drops after a certain stage for MT50-Mixed (similar phenomenon did not appear in MT10-Mixed). \n2. Some components of the methods could be better ablated like, how does the number of experts affect the performance of the method. Since this work propose to learn more meaningful skills for different experts via contrastive learning, more discussion on this part would make the work stronger.\n3. Visualization of the attention weight for different tasks is missing, which could help the audience understand the proposed method.\n4. The use of different skill-use is discussed in previous work (Soft Modularization, where the state information and the task information are used at the same time to output the module selection), and no specific training objective (the difference with previous here seems to be the LSTM ) in this work is addressing this issue.\n\nQUESTIONS:\n*  Some explanation regarding the experiment results mentioned in the weakness section would be appreciated.\n* Additional visualization regarding the attention weights for different tasks. And since different modules have different semantic meaning (powered by the contrastive learning), some investigation on what kind of skill a specific expert represents would be interesting as well.\n* Though this work claims the method works without any loss weighting trick in the optimization, it would be interesting to see some results from that end, \n\nI would raise my rating if the authors could reasonably address part (given the limited time for rebuttal) or full of my concerns.\n\nLIMITATIONS:\nThis work propose a general multi-task RL method, in this case, I think no specific potential negative societal impact or similar things should be addressed. As the author indicated, the current method works well in multi-task RL (with a fixed number of tasks), and would like to see some extension in meta learning or open-vocabulary settings in the future.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper introduces an approach for multi-task RL. Their approach is similar to CARE, which learns separate encoder modules, but they add a contrastive task loss on top of the encoders. They show this approach outperforms all reported baselines on Meta-World (MT-10 and MT-50) and a variant of Meta-World where the initializations are mixed throughout training.

STRENGTHS:
- The proposed approach outperforms all reported baselines and independent training on the metaworld tasks in the fully observed setting. 
- The proposed solution is straightforward to implement.

WEAKNESSES:
- A lot of the methods section should be moved to a preliminary section because it is difficult to understand what is novel and not novel. The temporal attention section on L168-180 is one example. Overall I found the organization of the paper confusing.
- There are not enough experimental results to fully validate the method (e.g., relevant baselines like PC-grad). In all of the comparisons to baselines the approach is still fairly overlapping with the error bars of other baselines and the meaning of the error bars is not described anywhere in the text.

- It would help the reader to consolidate the terms. The modules are separately referred to as both experts and modules. I would find it more straightforward if the naming was consistent.

QUESTIONS:
- On L143 "Notably, it surpasses the performance of learning each task individually for the first time in the Meta-World environment" Is this true in the case of MT10? E.g., Figure 3 of Gradient Surgery for Multi-Task Learning (Yu et. al., 2020).
- What are the error bars in the figures?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to multi-task RL called Contrastive Modules with Temporal Attention that aims to address the issue of negative transfer between tasks in multi-task RL.\n\nThe proposed method consists of two main components: contrastive learning and temporal attention. The contrastive learning component is used to ensure that the shared modules learned by the method are distinct from each other. This is achieved by applying a contrastive loss that encourages the modules to produce different outputs for the same input. The temporal attention component is used to dynamically combine the outputs of the different modules at each time step. This allows the method to adapt to the specific requirements of each task.\n\nThe authors evaluate their method on the Meta-World benchmark, a widely used benchmark for multi-task RL. They compare the performance of their method with several baselines, including methods that train each task separately and methods that share all modules across tasks. The results show that CMTA outperforms the baselines in terms of both sample efficiency and performance.\n\nSTRENGTHS:\nOriginality\n- The paper presents a novel approach to multi-task RL by introducing contrastive modules with temporal attention.\n- The method addresses the issue of negative transfer between tasks, which is a significant challenge in multi-task RL. The authors propose a novel solution to this problem by constraining the modules to be different from each other and using temporal attention to dynamically combine them.\n\nQuality\n- The authors provide adequate experimental results on Meta-World, a widely-accepted continuous control robotics benchmark, and some ablation studies that support the effectiveness of their method.\n- The paper is well-referenced, indicating a thorough understanding of the existing literature. The authors clearly position their work within the context of previous research.\n\nClarity:\n- The paper is well-organized and the writing is clear. The authors provide a clear explanation of their method and its advantages.\n- The figures and tables in the paper are informative and support the text well. They help to clarify the method and the experimental results.\n\nSignificance:\n- The proposed method addresses a critical challenge in the field and shows superior performance compared to existing methods.\n- The method proposed by the authors, in particular the temporal attention mechanism, has the potential to be widely adopted in the field of multi-task RL. It could also inspire future research in this area.\n\nWEAKNESSES:\nLimited insight into hyperparameter sensitivity: The paper would be stronger if it discussed the sensitivity of the proposed method to its hyperparameters. Understanding how changes in hyperparameters affect the performance of the model is crucial for reproducibility and for users who wish to apply the method to their own tasks.\n\nLacking insight into soft attention weights: It would be interesting to see the general relationships of soft attention weights and other aspects of the problem, such as its temporal nature, the tasks involved, etc.\n\nLimited Discussion on Failure Cases: While the paper presents a number of successful results, it could discuss in detail the scenarios where the proposed method fails or performs sub-optimally. Such a discussion could provide valuable insights into the limitations of the method and guide future improvements.\n\nLack of Comparison with Related Work: While the paper compares the proposed method with several baselines, it does not compare it with other multiple other methods that also use contrastive learning or attention mechanisms in the context of multi-task RL (*see first bullet point below). Such comparisons could provide a more comprehensive evaluation of the proposed method. \n  - However, the paper does add the contrastive loss to CARE and evaluate it, but CARE's attention mechanism is very different than CMTA's and only CARE was compared against in this way.\n  - It would be interesting to see how other proposed baselines or competitive related work would benefit from the proposed temporal attention module.\n\nGeneralizability: The proposed method has been evaluated on a specific benchmark (Meta-World). Its performance on other benchmarks or real-world tasks that differ from Meta-World type setup and tasks is not known. CMTA may not outperform certain baselines or other algorithms on suites of tasks in other benchmarks or real-world tasks.\n\nQUESTIONS:\nSubsumed into Weaknesses and Limitations.\n\n1. What are three limitations of your work that have not been addressed in the reviews, and what are your thoughts about them? This isn't intended to diminish your work. Instead it's to show that you understand where and how your work shines and to highlight where it may not as future problems to be addressed or as problems that are insignificant for some reason(s).\n\nLIMITATIONS:\nScalability: The proposed method might not scale well to tasks with a larger number of subtasks or more complex environments, even though the paper claims in Line 311-312 that \"as the variety of tasks increases, the advantages of our approach become more apparent\", they don't show this outside of Meta-World, and Meta-World is measure 0 subset of possible tasks. \n\nRegardless, the computational cost of the method, especially the contrastive learning part, could increase significantly with the complexity of the tasks.\n\nDependence on Task Similarity: The effectiveness of the proposed method might depend on the similarity of the tasks. If the tasks are very different from each other, the shared modules learned by the method might not be effective for all tasks.\n\nI did not find the paper addressing its limitations anywhere. One I will put forth is the lack of insight into the temporal attention module's relationship to different aspects of the problem, such as (1) how the attention weights vary over time, (2) whether they reach a steady state, (3) what do they focus on, for the Meta-World problems covered in the paper. \n\nI believe we should be wary of how this overall approach behaves, especially the temporal attention module and the assigned soft attention weights, in problems in which ethics, safety, fairness, bias, may be a concern. This is general concern for any algorithm that doesn't explicitly address and mitigate these issues, but it's especially one here because of the lack of insight into the temporal attention module's relationship to aspects of the problem.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
The paper focuses on multi-task reinforcement learning. Motivated by the negative task transfer within each task, the paper proposes the Contrastive Modules with Temporal Attention (CMTA) method, which utilizes temporal attention to modulate the weights of experts. Concretely, temporal attention takes recent history as input to capture local information. The proposed CMTA is evaluated in MetaWorld and shows superior performance than baselines.

STRENGTHS:
- The methodology part is clear and well-written.

 - The methodology is well-motivated.

WEAKNESSES:
- The paper poses the multiple skills that may be utilized in each task as negative transfer problems within a task. I believe such terminology is ok but I would highly recommend including a session in related work summarizing the skill-based RL literature and pointing out the deep connection between “negative transfer within a task” and “skill-based RL” in the introduction part. 

 - The novelty of the paper is limited from my point of view.

 - The experiment part lacks necessary information and please refer to the questions. The ablation study can be further improved by analyzing the impact of history length in the temporal attention mechanism, as well as the number of experts.

QUESTIONS:
- What is the number of experts in the mixture for each experiment setting, including MT10-Fixed, MT10-Mixed, MT50-Fixed, MT50-Mixed?
 
 - How is the smooth curve calculated exactly? The meaning of the smooth factor in line 242 is unclear. 

 - Why do the performances drop for MT50-Mixed after 1.5 Million steps (figure 3 the right most figure MT50-Mixed)?

LIMITATIONS:
- The limitation is not discussed in the conclusion. One trade-off is the increased model size due to experts and the improved performance.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-24