Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to use masks in multi-task training so that only a part of network parameters are used for a specific task. The framework considers combining both online and offline RL methods. Experiments show that the proposed method significantly reduces the training parameters and achieves competitive performance compared with previous methods.
strength_and_weaknesses: Strengths:

The idea of introducing sparse training to multi-task RL is a promising direction.

The proposed framework is combined with both online and offline RL methods, showing the generality.

Weaknesses:

The literature review is not extensive, missing the discussion and comparison of related works [1-3].

1) Some technical details are not convincing: not considering Data distributional shift in offline RL setting. Data distributional shift is a common problem in offline RL since the data set is collected by some unknown policy. When the trained policy interacts with the environment, there exist some states/actions out of distribution where the policy may behave badly. 

2) The masks are fixed during training. However, it's more reasonable to adjust the mask since the sensitive measure is not static over the whole state space. Also, it is not static during different training stages. On contrary, if the mask is adaptive changing, it may cause unstable training.

3) The experimental results should focus more on the benefits of sparse training regarding the training time/resources saved. For example, the parameter reduction results in offline RL are not provided.

Experiments should consider the full MT50 task set and compare the proposed method with CAgrad [4].

[1] Sparse Multi-Task Reinforcement Learning.
[2] Multi-task Batch Reinforcement Learning with Metric Learning.
[3] Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models.
[4] Conflict-averse gradient descent for multi-task learning.

clarity,_quality,_novelty_and_reproducibility: The Clarity should be further improved. This paper has a lot of grammar issues, such as 'nonstationaritym',  'radient interference', and 'of group the parameters'.

The idea is composed of several existing techniques, which makes it somewhat not novel. The pruning technique used in this paper does not consider the difference from the single-task setting.

Open-sourced code is provided.  
summary_of_the_review: Overall, this paper points out a promising direction in MTRL and provides relatively extensive experimental results. However, as listed in the pros and cons part, the novelty, technical issues, and missing related works make this paper lean to reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims to learn multiple RL tasks with the same network. SNIP is used to prune all but the 5% most "relevant" parameters of the network, yielding a mask for every task. Tasks can share parameters, and therefore all tasks have to be trained simultaneously. The authors show that SNIP depends strongly on the data set, and propose to regularly change the task-masks based on the newest data for online RL scenarios. The new method NPF is evaluated on 3 Mujoco and the Meta-World multi-task benchmarks with the BCQ, IQL and SAC algorithms and compared with other baselines. Results show that NPF is competitive with other multi-task approaches.

strength_and_weaknesses: **Strength**

The paper presents an innovative approach to learn and represent multiple tasks in one network. The approach is simple and can be adapted into many algorithms. Results look generally good. 

**Weaknesses**

The idea is, to the best of my knowledge, novel, but not very deep. Following the lottery hypothesis, the authors just select parameter-masks and keep them (in the offline setting) during training fixed. In the online setting the authors identify that the change in loss requires them to recompute the mask based on newer data. However, the implementation (averaging the last $M$ scores, stopping criterion) sounds very simplistic and there is no analysis on the dynamics of selected masks, ways to deal with this non-stationarity and whether the mask eventually converges, just as the policy does. More analysis on this would strengthen the paper significantly. Furthermore, there are some questions I would like to have answered:
1. You emphasize that the training data changes the loss a lot, but in RL the loss is also changed e.g. by bootstrapping: at time $t$ the loss depends on the value function(s) with parameters $\theta_t$. Have you evaluated this? How is the effect of this on the mask selection? 
2. What is the "continual pathway configuration" (referred to later as "parallel learning") in Figure 4? Is it NPF without the pretraining stage or training without mask or something else?
3. How significant are the results? It looks as if the standard deviations overlap a lot. Can you make a statistical test whether your method is significantly better (p <= 0.05) than the next best baseline? 
clarity,_quality,_novelty_and_reproducibility: The paper is mostly well written and mostly correct (see comments below).
The evaluation is mostly solid and the appendix contains enough information to reproduce the experiments.

**Minor comments:**
- sec.3: "naivelt"
- sec.3: say explicitly whether sharing the same transition is a necessary assumption
- sec.4.1: $\frac{\partial\mathcal L}{\partial \theta_q}$ appears in different equations with different meanings. Clarify this by using $\frac{\partial\mathcal L(\theta_0)}{\partial \theta_q}$ and/or $\frac{\partial\mathcal L(\theta_0; D_n)}{\partial \theta_q}$
- eq.1: $m_n = S(...)$ appears incorrect. It should be something like $m_n = \text{argmax}_{q} S(...)$.
- sec.4.2: shortly explain what "synchronous parameter updates" are
- sec.4.2: "function 1" -> "Equation (1)"
- alg.1: clarify where the $n$ in the training loop comes from (e.g. run all tasks $n$ in parallel) and when exactly the optimizer is executed (after all gradients have been summed). Currently the parallelization part is more confusing than clarifying. Adding an inner loop might help.
- eq.2: it is completely unclear what the sum-variable $m$ refers to here, as it does not appear in the equation and overloads the mask $m$. Better define the sum over variable $k$ and (a) use $TD_n^{t-M:t}$ to denote that you are referring to replay buffer length or (b) define some $s_n^t$ as score at time $t$ and then select the mask based on the average last M scores $\frac{1}{M} \sum_{k=0}^{M-1} s_n^{t-k}$ or (c) define an online update $\bar s_n^t = (1-\alpha) \bar s_n^{{t-1}} + \alpha s_n^t$.
- alg.2: the algorithm does not reveal how "the last M iterations" affect the masks and does not show how $D_n$ and $TD_n$ differ (they both just get all the data, what is the difference?). 
- tab.2: there need to be other baselines (like in tab.3) for comparison of the Mujoco experiments.
- fig.5: the caption almost touches the main text
- p. 6: the paragraph "Online Multitask Training and Addressing Gradient Interference" is in a weird place. It would make more sense in the beginning of Section 4.  
- sec.5.1: the text says "the mean performance with 95% confidence interval", but the caption fo Table 3 says "mean and std over 10 seeds". Which is it?
- sec.5.1: the MT baseline is ambiguous: do you train one network for all tasks (which should not work without a task indicator) or one network per task?
- fig.6: figure almost touches the main text
summary_of_the_review: I really liked this paper. It is not totally clear whether the method will allow something that would otherwise be very hard in multi-task RL, but it is a nice idea and in my opinion well evaluated. The only weak point is a relatively low novelty and a lack of theoretical analysis. However, unless another reviewer finds something really wrong with it, I recommend to accept the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a multitask RL framework by learning multiple pathways through a single network, called Neural Pathway Framework (NPF), which aims to find a task-specific optimized mask. Started with a method from a network pruning literature, the authors further proposed to use replay buffer and average multiple iteratures to stablize training. Experimental results shows performance improvement with reduced parameter count.
strength_and_weaknesses: I do have some concerns regarding this paper, especially in terms of the clarity.

The clarity of the paper needs to be improved. 
1. How is global-model optimized in Algorithm 1? Providing an equation would be very helpful. 
2. The most important equation (1) in paper is not clear to me. By my understanding, m_n is the mask for n_th task, while S(theta; D) is the objective function. Why these two terms are equivalent?
3. Equation 2 is also unclear to me. It envolves two techniques to stablized the training, keep a temporary replay buffer TD_n and re-confriguring using the average of the previous M iteration scores. In paper, it seems that keeping a temporary replay buffer aims to keep only the most-recent samples from agent to update the mask, but averaging between several iteration is using samples span longer history, which seems contradictory. How would you interpret it?
4. For results in Table 3, how could variance of BCQ with NPF be 0? Also, since it is normalized, I'm curious of the original performance of BCQ and IQL algorithm.


clarity,_quality,_novelty_and_reproducibility: In terms of originality, Pathway is an interesting idea as an overall framework for AI system to handle discrepency between tasks, domains, etc, which is first proposed by Jeff Dean to my knowledge. That been said, it is still interesting to discuss possible ways to actually implement the idea to solve problems and address difficulties, like how to find the optimal pathways for each task or domain or even sample. 

In terms of concreteness, the proposed model aims at achieving similar performance using sparse sub-network in comparison with full dense network. Thus the experiment in Figure 3 shows inferior results with proposed model in comparison with baseline. It needs to further clarify that how significant it this performance difference.
summary_of_the_review: The paper studies an interesting problem and gives a potential solution. My major concerns are about clarity. I might will change my recommendation after things are more clearer.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on multitasking ability in reinforcement learning (RL). To do so, a multi-task learning framework, Neural Pathway Framework (NPF), is proposed by simultaneously training multiple specialized pathways where each pathway corresponds to a task. Finally, experiments were conducted on several continuous control tasks under both online and offline settings, including using Mujoco-based and MetaWorld environments, with results demonstrating its effectiveness in terms of performance and parameter size.

strength_and_weaknesses: Strengths:
- Multitasking is an important ability to achieve artificial general intelligence (AGI). This is an interesting work targets to multitask RL where the ideas build up from the neuro-science perspective on neural pathways in human brains and the machine learning perspective on finding sub-networks with pruning.

Weaknesses:
- The proposed methods are not presented in a principled way, although the main contribution of this work is primarily on the empirical side. For example, what's the formal definition of the problem? What are the loss functions or objectives for the offline or online settings? What's the mask used in the methods? What are the original networks and sub-networks? How do neural pathways/sub-networks interact with different tasks? How to guarantee or justify that the obtained sub-network is the right solution? How are the learned skills reused/transferred among tasks? What does the mask "$m$ is fixed" mean?
- In Figure 4 (a), why the gradient norm becomes larger? Does that mean there exists a convergence issue? How to define the success rate in (b)?
- This paper claims that the proposed method "is especially suitable for real-world applications in which large datasets need to be processed in real-time or which are deployed on low-resource devices". However, I didn't see any comparisons of time cost or computation cost shown in the experiments.


Minors:
- "In the online RL setting,." -> "In the online RL setting,".
- "nonstationaritym" on pp.5 should be "nonstationarity".
clarity,_quality,_novelty_and_reproducibility: - This paper could be improved by clearly showing more details of the proposed method as I mentioned in the main Weaknesses. Besides, there are some questions that could be clarified.
- The idea of finding neural pathways with pruning is not new. Since the current paper is not organized and written in a good form, especially on the methodology part, it's hard for me to evaluate its originality accurately.
summary_of_the_review: According to my comments in both the main Weaknesses and the section of "Clarity, Quality, Novelty", I feel this is an empirical paper where reasons to reject outweigh reasons to accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
This paper addresses the problem of multi-task learning using a single network where each task uses a subnetwork much smaller than they usually use. The authors use an existing method for calculating connection sensitivity, which is utilized to create separate masks for the tasks. I appreciate that the authors aim to address an important problem. However, the results and the presentation did not create a clear value. The message and many details of the work did not get across well. Moreover, a clear intuition needs to be included on why an initial computation of mask worked well and better than continuing to change the mask. I strongly encourage the authors to work on the exposition of the idea clearly and justify the steps of the method more carefully and rigorously for future submission.

justification_for_why_not_higher_score: The work has many deficiencies, failing to provide a clear value.
Decision: Reject