Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an action representation scheme for multi-task training, which is relatively simple in that the action representation is decoupled with the sensory represenation. The proposed method allows for emergent action composition as well as interpolation in the action space. The validity of the method is demonstrated on computational experiments in a Mujoco environment, compared against several other RL schemes. 
strength_and_weaknesses: Strengths: 
- This work is well-motivated. There is certainly a lack of effective task representation for complex multi-task environment. The proposed framework is addressing an timely and important problem. 
- The proposed idea is interesting and looks making sense in general. 
- The method is evaluated, compared agasint a reasonable choice of state-of-the-art 

Weaknesses: 
- The methodological details are not clearly described/analyzed. Perhaps the paper is to talk about the arguably simplea idea, but other than the framework in Figure 1, additional description/investigation is needed on what details in action representation would make impact on the performance. Would the MLP be the best choice for this purpose? 
- The use of terminology is a bit abused. It is not very clear the task composition in this work can be called "emergent." 

clarity,_quality,_novelty_and_reproducibility: - The paper is relatively well-written and easy to follow, but more detailed invetigation on the detailed choice of action representation network would be useful. 
- The approach looks technically sound in general, but details would be benefitical. 
- The paper addresses a timely and important problem and presents a simple but new idea. 

summary_of_the_review: The reviewer thinks that this paper presents an interesting idea on an important problem. The paper will be benefited by adding analysis on the detailed algorithm choices/variations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a learning framework (Figure 1) for action abstraction in the scheme of multi-task reinforcement learning (RL). A latent action representation is assigned to each task, and the model tried to learn the tasks with state representation (shared among tasks) and action representations (specific to each task, task embedding) segregated. It was observed that the model self-developed interpretable and compositional action presentations via RL. The learned action representation can facilitate quick, gradient-free transfer learning. More interestingly, it was shown an interpolation in the latent action space led to an explainable "interpolation" in the behavior space. The main contribution is a proof of concept of a novel framework for action abstraction.
strength_and_weaknesses: ## **[Strength]**

- The methodology is simple and intuitive (no additional loss functions for action abstraction).
- The idea and result are quite interesting.
- Interpolation/composition in the latent action space surprisingly works.
- The proposed framework provides novel insights on how to leverage the task distribution for learning high-level motor skills


## **[Weakness]**

### Related work is incomplete
While the paper mentioned Representation learning in RL, multi-task/metaRL in Sec. 5, important pieces are missing. A clearly related batch of studies are skill discovery with deep RL, which is closely related. Some related references are:
- Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019
- Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
-  Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control primitives: Skill discovery via reset-games. In Advances in Neural Information Processing Systems, volume 33, 2020.

### The proposed framework heavily relies on the task set
To achieve the proposed results, there are quite a few requirements: e.g., a distribution of uni/multi-modal tasks with well-defined reward functions, the tasks shared the same state space. By contrast, skill discovery methods can achieve similar results without a reward function. Furthermore, people and animals can often self-develop motor primitives in a single task.

### Technical details are not clear
For example, model architecture, hyper-parameters, training schemes: Sec.A.3.2 is clearly not enough, not to say that source code was not provided. The authors should provide more details for reproducibility.

### Statistical significance of major claims is not well addressed
While the most interesting result --- interpretable interpolation and composition of LTEs, was well presented using an example agent (Figure 3, 4), it is unclear whether this result is common with different random seeds. That is to say, the authors did not demonstrate statistical reliability of the phenomenon showed in Figure 3,4, e.g., what is the percentage of random seeds that can achieve such a result?  More evaluations remain to be complete.

### Lack of ablation studies
Many design choices lack explanation. For example, the two techniques used to constrain the space, normalizing LTE and injecting random noise. Why normalizing LTE? What will happen if there is no such regularization? I guess these two techniques are essential to the success of the proposed framework, yet remains to be discussed.

### The writing quality is to be improved.
The paper seems not well polished. Belows are some issues I found. Nonetheless, I think the paper need a careful and throughout proofreading. Also, reference should also be manually checked.
- The first sentence in Sec.1: Deep RL can learn --> deep RL agents can learn
- Page 5: Specifically, We perform ... --> Specifically, we perform
- Page 5: We visualized the LSEs and and LTEs: duplicate "and"
- Page 9: the end, incorrect citation style
- Page 10: Duplicate citation of Chandak 2019
- Page 11: {Bayes}
- Page 14: control control


### **[Questions]**
1. Will the authors release source code?
2. How to initialize $\mathcal{Z}_\mathcal{T}$ ?
3. I am curious about the result of extrapolation in LTE space, e.g., if $\beta < 0$ or $\beta >1$, what will happen?


clarity,_quality,_novelty_and_reproducibility: The writing quality is moderate but can be improved through proofreading. The authors did a good job on presenting the ideas and experimental results.  The key idea is clear and intuitive, while implementation details need to be clarified. The proposed framework is novel to my knowledge. 
summary_of_the_review: Overall, I appreciate this paper as it proposes a simple method to achieve interesting results, though it poses some demands to the task set. The goodness is the latent action representation showed self-organization via RL without additional, explicit loss functions for this purpose, which provides some insights for both machine learning and cognitive science about how motor primitives could be autonomously acquired and leveraged for transfer learning. The insights are, however, limited because there are a lot of points for the studies to improve as I mentioned above. Overall, my recommendation is around the borderline as for now, and my rating may increase given a constructive rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an architectures that separately encodes sensory information and the task id in a multi-task setting. These are then concatenated to processed by a neural network that produces the action. The paper imposes constraints on the task representation (or action representation in the paper): the representation is restricted to the unit sphere and trained with some noise. The paper then evaluates the proposed architecture on a number of simple locomotion domains. Empirically the methods compares favourably to multi-task and meta-RL baselines. The paper also demonstrates that it is possible to interpolate in the task embedding space both within the same task (e.g. different velocities) and between different locomotion modes.
strength_and_weaknesses: Strengths:
* interesting results with a relatively simple methods

Weaknesses:
* From what I can tell the paper proposes essentially a well-regularized version of learned task embeddings. It would be nice to see an ablation of the two constraints (noise and restriction to the unit sphere). Are both of these necessary to achieve the results shown? Would other regularization ideas work as well (e.g. information bottleneck?). It is interesting to see that this methods appears to perform well but it is unclear to me what specifically makes it work.
* relatively toy environments. I think the paper would be substantially stronger if the similar results were shown in more complex settings.
* I think task representation might be a better term than action representation (since action already has a specific meaning in RL). At the moment I think readers might be confused. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and would be straightforward to reproduce. I think the novelty of the approach is somewhat limited (basically a well-regularized task embedding) but there are some intriguing empirical results. 
summary_of_the_review: Interesting paper showing that a well regularized task embedding is competitive with multi task and meta-RL baselines on a set of relatively toy locomotion domains. The embedding also allows some interesting interpolation. I would like to see some more analysis to understand what specifically makes this approach work. I would also like to see if it scales to less toy settings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies the action representations emerged in multi-task policy training. The authors study a two-stream policy architecture for multi-task reinforcement learning. One input stream encodes the observation and the other input stream embeds the task ID. Finally, the two embedding vectors are concatenated and fed into the policy network. When proper normalization and noise injection are applied, the authors observed that the task ID embeddings space emerges where each embedding in this space encodes a meaning behavior. Embeddings in the emerged space can be interpolated or composed to generate interesting behaviors. The authors then further demonstrate that optimization in this embedding space can yield fast adaptation to downstream tasks. Visualization of the learned state embedding space and task embedding space provides more intuitive understanding of the learned structures.
strength_and_weaknesses: # Strength
* Overall the paper is clearly written and easy to follow. A few flaws undermines the readability a bit, which are discussed below.
* The finding of the emergence of the action embedding space is interesting. The proposed architecture is simple and straightforward. Yet this simple architecture yields interesting findings. The strong empirical performance against meta-RL baselines could inspire us to rethink the fast adaptation problem.

# Weakness
* The tasks in the experiments are relatively simple. Each task can be effectively described by one or two factors - which is nice for visualization and understanding. However, it is unclear if the same phenomenon would be observed in more complicated tasks such as object manipulation.
* I find the experiment results in Section 4.6 rather confusing. I do not understand the purpose of this experiment. When the task IDs are one-hot encoded, the input layer essentially serves as an embedding layer. It does change the interaction between the state embedding and the task embedding from concatenation to addition, but a certain equivalence can be shown for these two different types of interaction [1]. The fact that this one-hot variant does not even work well during training seems rather confusing to me. I will appreciate if the authors could provide better clarity on this experiment.
* One question I am interested in asking is the importance of embedding normalization and noise injection. I think it will be good to conduct an ablation study for this purpose.
* The section paragraph of the related work section looks confusing to me. The authors list a few works that address multi-task RL. But I don't see their connections to this work that are worth highlighting.
* Writing and presentation can be improved. For example, what are $\theta_{1}$, $\theta_{2}$, $\bar{\theta_{1}}$, $\bar{\theta_{2}}$ in Algorithm 1? The sampled action $a_{t}$ in the second for-loop is never used. Overall, the role of Algorithm 1 is unclear. It presents a standard multi-task policy training procedure which does not highlight LTE. Figure 3 and Figure 4 need more explanation in either the caption or the text. The screenshots are incomprehensible in their current form.

# Questions and discussion
* The second paragraph of Section 4.3 says "The coefficient $\beta$ is searched to better fit the target task". Why is search needed here? The new task is a convex combination of two reference tasks. I would expect we can just interpolate the embeddings with the same coefficient. Did the authors try it? What did you observe?

# Minor issues
* In the 5th line in the second paragraph on page 5: "The newly composed task is usually lie in a new modality..." Please fix the grammar.
* In the x-axis of Figure 5: what does "steps" mean? Does it mean episodes?
* In the second paragraph in Section 5: reference to Jeon et al and Chandak et al should use \citep{}.

**References**
1. Dumoulin _et al_ 2018, Feature-wise transformations, https://distill.pub/2018/feature-wise-transformations/ 
clarity,_quality,_novelty_and_reproducibility: * Clarity: Fair. Suggestions for improvements on presentation are provided above.
* Quality: Good, though I think an ablation study on embedding normalization and noise injection could provide better understanding.
* Novelty: To the best of my knowledge, the findings presented in this paper are novel.
* Reproducibility: Fair. Sourcecode is not provided. But the appendix provides enough implementation details.
summary_of_the_review: Overall I think the findings in this paper are interesting and could benefit the community. But the presentation can be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper develops the use of a latent space for actions across multiple reinforcement learning tasks. Tasks are represented with a one-hot encoding, and the latent space for actions is a low-dimensional sphere.  The experiments demonstrate interpretability and fast learning from the latent action space across multiple Mujoco locomotion tasks. 

Strengths:
The reviewers found the work well-motivated (MYUq), simple (MYUq, oCDb, VRfp), with adequate experiments (MYUq) and interesting results (OCDb).  There was no major weaknesses in the results that were presented.

Weaknesses:
The experiments were adequate for the claims, but reviewers were unsure whether if similar results would hold for more complex domains (VRfP,YSuL,oCDb).  There were also concerns about the technical clarity of the paper, which were largely resolved after the author response.

justification_for_why_not_higher_score: The reviewers were not wholly convinced in the generality of the method.  The paper presents a "proof-of-concept", which merits additional validation in more general domains.
justification_for_why_not_lower_score: The proposed method is simple and novel.  The presentation is clear enough for other researchers to replicate the work.  Even reviewers who were not entirely convinced in the generality of the proposed method thought it could inspire followup research.
Decision: Accept: poster