Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes an approach for composing parameters for solving multi-task RL problems and obtains promising results on MetaWorld benchmark
strengths_and_weaknesses: ## Strengths

1. It is a relatively straightforward approach that works well in practice. 
2. The paper is mostly well written (though see next section)
3. The baselines are both recent and relevant.

## Weakness

1. The paper should add the "one-model-per-task" baseline to their results, to get a sense of performance gap.
questions: 1. The paper mentions a "stop-relaunch" trick ion [23] but I could not find a reference to that trick in [23[.

2. In line 154, it says " in PaCo, there is a natural way to mitigate this issue without resorting to ad-hoc [23] or more expensive schemes [34]." Isnt the loss maskout another adhoc trick. Infact this trick also introduces a new hyper-parameter.

3. I am not sure I understand why w-reset is used. I understand that we can do it but isnt PaCO already ignoring the loss when it is greater than $\eps$. I didnt understand the need for both loss masking and w-reset

4. It would be helpful if the authors can comment on the "novelty" of the work. Is the main novelty that they are doing composition in the parameter space, instead of the function space? I am not considering this as a weakness - I just want to make sure I am not missing anything here.

5. Could the authors add details about the computation overhead of the different approaches?

6. The analysis related to Figure 4 is not very convincing. eg arguing that "peg-insert v.s. window-open/window-close/door open/drawer-open are related from the perspective of behavior, i.e., first interacting with an object (e.g., peg/window/door), and then taking a trajectory of motions to accomplish the subsequent operation, (e.g. insert/window/door)" does not sound convincing as the same argument applied to "pick-place" while it is quite far in the latent space. 
limitations: Covered in section 7
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes PaCo, a framework for multi-task learning in RL. It uses a set of parameters that are composed in a unique way for each task. A compositional vector w is learned for each task, which weights the items in the parameter set. This results in a weighted policy for each task that relies on shared underlying parameters. This can help with learning shared structure across tasks and improve sample efficiency.
strengths_and_weaknesses: The work is original and clear with strong results on a multitask benchmark. If my concerns are addressed (e.g. number of seeds, hyperparameters, and a couple of unsupported claims), I support the acceptance of this paper.
 
Strengths
* Parameter sharing is an interesting area for improving sample efficiency in MTRL, and I have not seen published work that composes policies in this way at the parameter level.
* The interpolated policy visualization in Figure 3 is unique and interesting.

Weaknesses
* One of the potential benefits of this framework is the ability to add new tasks after training the parameter set, which is mentioned on L144 and described in appendix section 3.2 (which says "in experiments, we do find successful extensions from existing skill set to a new skill when the skills are similar"). This claim is unfortunately unsupported. Please either provide evidence of such experiments or do not include the claim.

-------
Update: I thank the authors for their responses. The concerns (especially about number of seeds and hyperparameters) were sufficiently addressed; thus I am increasing my score from borderline accept to weak accept. I do recommend expanding the discussion/analysis of new results, i.e. "continual learning without forgetting" (what to compare these numbers to) and single-task SAC vs K=10 vs K=10 with one-hot initialization (what relative performance numbers mean for these three versions).
questions: * Are all experiments using 10 seeds? I see that table 1 is using 10 seeds, but I'm not sure about tables 2 and 3. For example, the std of PaCo-vanilla in table 2 seems very high and I'd expect it to be lower with 10 seeds.

* Page 7, footnote 3: If these baselines were implemented for a different setting, are there any hyperparameters that need to be re-tuned for MetaWorldV2? It would be helpful to know which hyperparameters from appendix table 4 were tuned for PaCo specifically and what values for each were included in the search.

* I'd like to see a baseline where each task has its own policy (no parameter sharing) and another baseline where the size of the parameter set (K) is the same as the number of tasks. In the K=10 case for MT10, for example, does each task essentially claim its own item in the parameter set and there is limited parameter sharing?

* I'm curious about the importance of assigning/tuning separate temperatures for each task (L89). Is this a crucial aspect of PaCo's success over baselines?

Minor (no need to comment on the below):
* Appendix section 3.1: I don't think this section is that important, but if it's going to stay in the paper, please add a figure with evidence of this experiment (e.g. reward curves for PaCo and the modified ratio PaCo).

* L313: "Overall, the increasing in parameter set size K grows with the task number n but essentially at a lower speed." I don't see enough evidence for this claim given that the parameter set size ablation is only shown for MT10 (10 tasks). I would probably remove the claim unless the parameter set size ablation can be repeated on MT20 or MT50 (to show how it scales).

Typos:
* L136: details about PaCo is --> details about PaCo are
* L159: compositional natural --> compositional nature
* L294: some task already explodes --> some task has already exploded
limitations: I appreciate the limitation discussion of the linear compositional form.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors propose a simple, yet effective way to learn parameters in multi-task RL setting, where parameters for a task are a task-specific linear combinations of task-agnostic parameters. Parameters of linear combinations as well as task-agnostic parameters are both learned end to end. This simplicity allows for resets of linear combination parameters for tasks with exploding losses, which should lead to more stable learning. To this end, authors propose to use a random convex-combination of non-exploding linear combination parameters as a stabilization scheme. 
strengths_and_weaknesses: I think that the paper is well written and presents a reasonably novel and interesting idea with surprising results. I do have some questions about experiments and baselines and I am a bit surprised that the algorithm performs worse with more parameters. Therefore I am not certain if more experimental evaluation is not needed. I also think that the contribution and the paper could be strengthened by some theoretical results/motivation or intuition about why the particular proposed reset scheme increases the performance by so much (10-15%). I do not yet have a very strong opinion about whether the paper should be accepted although I am leaning more towards acceptance. I am willing to change my score based on the discussion with authors and fellow reviewers.

**Strengths**:
- Paper is in general well written and clear, the setting and method are well explained. I liked section 3 in particular
- In my view the idea seems novel enough even though part of it is closely related to previous approaches (specific case of Soft-Modularization?). However, I am not very familiar with current Multi-task RL approaches so I am not 100% certain about this.
- The results indicate that proposed re-initialization is quite effective on a common Meta-world benchmark (MT10)

**Weaknesses**:
- Only one domain is considered which might make it hard to judge the significance and scalability of the results especially since one of the ablations is a bit counter-intuitive (see questions). Appendix seems to contain experiments from a larger MT50 domain. However, these are not part of the main text.
- Experiment are mostly sound and include ablations. However, I do have several questions about them and would like to (among others) see learning curves from the experiments or intermediate/max values of the performance metrics instead of just final values

**Comments/Suggestions/Typos (do not affect the score):**
- line 80: use former one instead of previous one?
- line 84: I am not sure if about "share similar dynamics (actions space)" in this case since I presume action space is shared in general. Maybe transition function could fit better in this context?
- line 156 $\eta$ is used again but denotes something different than in background section as far as I can tell
- line 159: "compositional natural" nature?
- line 331 "tasks covers" cover

_______________________________________________________________
Update: I believe that the concerns about the domain and missing intermediate scores were sufficiently addressed by the authors. I have therefore increased my score. However, after observing the results, I also agree with reviewer CCrp that seeing the results beyond 2 million would be helpful. This is because while some method (SAC+ Film, PCGrad) have already converged, others (Care, Paco, single-task SAC) seem to still be improving so it would be nice to see how training progresses further and if/how long Paco keeps an advantage. 

questions: - In lines 65-66 you mention that you use separate temperature for each task with different skills. Does that mean that you use 10 different values of alpha in MT10-rand? 
  - If yes: Why is this necessary? Is the performance sensitive to the choice of these parameters and would it degrade with single value?
  - If no: Can you please explain what you meant by this?
- I am wondering about the relation to soft-modularization. As far as I can tell it seems to be more general because it allows for dependency on s. Your method should then be a specific case of this ($z$ does not depend on s and are shared for all $m$). Why should it then be less flexible (line 189.)? 
- I am wondering about the performance of single-task model where you use separate network for each task (line 175). Have you tried to run such baseline?
- How does the value of parameter epsilon affect performance? Have you tried runs with different values? If yes what was the performance?
- I am surprised that in table 3 K=8 performs worse than K=5 since the model with K=8 should be able to represent anything that K=5 can. In the footnote you also mention that K=10 performs better than K=5. Do you have performance metrics for other K? Why does high K lead to worse results(you mention additional complexities from over-parametrization, could you elaborate on that)? 
- Are the networks used for other baselines the same? Do all methods have same amount of hyper-parameters?

limitations: I do not foresee any negative social impact. Limitations of the work seem to be addressed as far as I can tell.
review_title: null

Meta-review: metareview: This paper proposes a multi-task RL architecture that composes task-independent parameters and task-dependent parameters to construct a task-specific policy. The results on the MetaWorld multi-task learning benchmark shows that the proposed method outperforms relevant baselines. In general, the reviewers found the idea interesting and novel, and the results are solid. Several concerns about the lack of single-task baseline and the lack of scores given intermediate steps were addressed with updated results during the rebuttal period, which made all the reviewers lean towards acceptance. Thus, I recommend accepting this paper. In the meantime, there are still remaining comments about the result beyond 2M steps (suggested by Reviewer CCrp) and additional baselines (suggested by Reviewer DW33). I highly suggest the authors to include these results for the camera-ready version.
confidence: Less certain
Decision: Accept