Simple Emergent Action Representations from Multi-Task Policy Training
Authors: Pu Hua
Yubei Chen
Huazhe Xu
Venue: ICLR 2023 poster (2023)
Keywords: action representation
reinforcement learning
representation learning

Abstract: The low-level sensory and motor signals in deep reinforcement learning, which exist in high-dimensional spaces such as image observations or motor torques, are inherently challenging to understand or utilize directly for downstream tasks. While sensory representations have been extensively studied, the representations of motor actions are still an area of active exploration. Our work reveals that a space containing meaningful action representations emerges when a multi-task policy network takes as inputs both states and task embeddings. Moderate constraints are added to improve its representation ability. Therefore, interpolated or composed embeddings can function as a high-level interface within this space, providing instructions to the agent for executing meaningful action sequences. Empirical results demonstrate that the proposed action representations are effective for intra-action interpolation and inter-action composition with limited or no additional learning. Furthermore, our approach exhibits superior task adaptation ability compared to strong baselines in Mujoco locomotion tasks. Our work sheds light on the promising direction of learning action representations for efficient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space. Project page: https://sites.google.com/view/emergent-action-representation/

Published as a conference paper at ICLR 2023
SIMPLE EMERGENT ACTION REPRESENTATIONS FROM MULTI-TASK POLICY TRAINING
Pu Hua 'A, Yubei Chen Huazhe X3
'Tsinghua University, "Center for Data Science, New York University, 'Shanghai Al Lab, *Shanghai Qi Zhi Institute
ABSTRACT
The low-level sensory and motor signals in deep reinforcement learning, which exist in high-dimensional spaces such image observations motor torques, are inherently challenging to understand or utilize directly for downstream tasks. While sensory representations have been extensively studied, the representations of motor actions are still an area of active exploration. Our work reveals that space containing meaningful action representations emerges when a multi-task policy network takes as inputs both states and task embeddings. Moderate con- straints are added to improve its representation ability. Therefore, interpolated or composed embeddings can function as a high-level interface within this space, providing instructions to the agent for executing meaningful action sequences. Empirical results demonstrate that the proposed action representations are effec- tive for intra-action interpolation and inter-action composition with limited or no additional learning. Furthermore, our approach exhibits superior task adap- tation ability compared to strong baselines in Mujoco locomotion tasks. Our work sheds light the promising direction of learning action representations for effi- cient, adaptable, and composable RL, forming the basis of abstract action planning and the understanding of motor signal space. Project page: https: //sites. google com/ wtt./viiwwweeeeeeeeeeee wwww:/wwwwweeeeeeeeeeeeeeeeeeeee ewwereeeeeet....e act iontiion-ie representation/
INTRODUCTION
Deep reinforcement learning (RL) has shown great success in learning near-optimal policies for performing low-level actions with pre-defined reward functions. However, reusing this learned knowledge to efficiently accomplish new tasks remains challenging. In contrast, humans naturally summarize low-level muscle movements into high-level action representations, such as "pick up" or "turn left", which can be reused in novel tasks with slight modifications. As result, we carry out the most complicated movements without thinking about the detailed joint motions or muscle contractions, relying instead high-level action representations (Kandel et al. 2021). By analogy with such abilities of humans, we ask the question: can RL agents have action representations of low-level motor controls, which be reused, modified, or composed to perform new tasks?
As pointed out in Kandel et al. (002)), "the task of the motor systems is the reverse of the task of the sensory systems. Sensory processing generates an internal representation in the brain of the outside world or of the state of the body. Motor processing begins with an internal representation: the desired purpose of movement." In the past decade, representation learning has made significant progress in representing high-dimensional sensory signals, such as images and audio, to reveal the geometric and semantic structures hidden raw signals (Bengio et al.) 2013 Chen et al., 2018 Kornblith et al.) 2019; Chen et al. 2020, Baevski al. 2020; Radford et al.) 2021 Bardes et al.) 2021 Bommasani et al. 2021] He et al. 2022] Chen et al. 2022). With the generalization ability of sensory representation learning, downstream control tasks can be accomplished efficiently, as shown by recent studies Nair et al. (2022); ; Xiao et al. (2022); Yuan et al. (202)). While there have been sig- nificant advances in sensory representation learning, action representation learning remains largely unexplored. To address this gap, we aim to investigate the topic and discover generalizable action representations that can be reused efficiently adapted perform new tasks, An important concept
Denotes equal contributions.

---

Published as a conference paper at ICLR 2023
in sensory representation learning is pretraining with comprehensive task set of tasks, followed by reusing the resulting latent representation. We plan to extend this approach to action represen- tation learning and explore its potential for enhancing the efficiency and adaptability of reinforce- ment learning agents. We propose a multi-task policy network that enables a set of tasks to share the same latent action representation space. Further, the time-variant sensory representations and iime-invariant action representations are decoupled and then concatenated as the sensory-action rep- resentations, which is finally transformed by policy network to form the low-level action control. Surprisingly, when trained on a comprehensive set of tasks, this simple structure learns an emergent self-organized action representation that can be reused for various downstream tasks. In particular, we demonstrate the efficacy of this representation in Mujoco locomotion environments, showing zero-shot interpolationccomosittii and few-shot task adaptation the representation space, out- performing strong meta RL baselines. Additionally, we find that the decoupled time-variant sensory representation exhibits equivariant properties. The evidence elucidates that reusable and generaliz- able action representations may lead to efficient, adaptable, and composable RL, thus forming the basis of abstract action planning and understanding motor signal space. The primary contributions in this work listed follows:
1. We put forward the idea of leveraging emergent action representations from multi-task learners better understand motor action space and accomplish task generalization.
2. We decouple the state-related and task-related information of the sensory-action represen- tations and reuse them to conduct action planning more efficiently.
3. Our approach is strong adapter, which achieves higher rewards with fewer steps than strong meta RL baselines when adapting to new tasks.
4. Our approach supports intra-action interpolation as well as inter-action composition by modifying and composing the learned action representations.
Next, we begin our technical discussion right below and leave the discussion of many valuable and related literature to the end.
2 PRELIMINARIES
Soft Actor-Critic. In this paper, approach is built on Soft Actor-Critic (SAC) (Haarnoja et al., 2018). SAC is a stable off-policy actor-critic algorithm based on the maximum entropy reinforce- ment learning framework, in which the actor maximizes both the returns and the entropy. We leave more details SAC in Appendix
Task Distribution. We assume the tasks that the agent may meet are drawn from pre-defined task distribution p(7). Each task in p(T) corresponds to a Markov Decision Process (MDP). Therefore, task T can be defined by tuple (S,A, po, Rt), in which S and A are respectively the state and action space, P the transition probability, Po the initial state distribution and the reward function.
The concept of task distribution is frequently employed in meta RL problems, but have made some extensions on to better match with the setting in this work. We divide all the task dis- tributions into two main categories, the "uni-modal" task distributions and the "multi-modal" task distributions. Concretely, the two scenarios are defined as follows:
Definition 1 (Uni-modal task distribution): In uni-modal task distribution, there is only one modality among all the tasks in the task distribution. For example, in HalfCheetah- Vel, a Mujoco locomotion environment, we train the agent run at different target velocities. Therefore, running is the only modality in this task distribution.
Definition 2 (Multi-modal task distribution): In contrast to uni-modal task distribution, there are multiple modalities among the tasks in this task distribution. A multi-modal task distribution includes tasks of several different uni-modal task distributions. For instance, we design a multi- CetChhetah-RunJJunnn which contains two modalities including
modal task distribution called BaCCCheetah-BakkJmppp HalfCheetah- Back Vel and HalfCheetah- BackJump. The former has been defined in the previous section, and the latter contains tasks that train the agent to jump with different reward weight. In our implementation, we actually train four motions in this environment, running, walking, jumping ans standing. We will leave more details in Section and Appendix B.
2

---

Published as a conference paper at ICLR 2023
3 Sersory Representation
St
Sensory Information
Stute Encoder
Action Decoder
Moter Signal
(s
Semsury-Actin Representation
Zz Action Represcutation
Zy
Z
Zrgg
Action Representation Self-organization
Figure 1: Emergent action representations from multi-task training. The sensory information and task information encoded separately. When both are concatenated, action decoder de- codes them into a low level action.
EMERGENT ACTION REPRESENTATIONS FROM MULTI-TASK TRAINING
In this section, we first introduce the sensory-action decoupled policy network architecture. Next, discuss the multitask policy training details, along with the additional constraints to the task embedding for the emergence of action representations. Lastly, we demonstrate the emergence of action representations through various phenomena and applications.
3.1 MULTITASK POLICY NETWORK AND TRAINING
Decoupled embedding and concatenated decoding. abstract high-level task, e.g., "move for- ward", typically changes relatively slower than the transient sensory states. As simplification, we decouple the latent representation into a time-variant sensory embedding Ze. and time-invariant task embedding ZT, which is shown in Figure These embeddings concatenate to form a sensory- action embedding (s,,, 7) = [Z Zr]. which transformed by the policy network (action decoder) & output low-level action distribution p(a.) 1 Z,, Zr), e.g., motor torques. The action decoder & is a multi-layer perceptron (MLP) that outputs a Gaussian distribution in the low-level action space A.
Latent sensory embedding (LSE). The low-level sensory state information encoded by an MIP state encoder à into latent sensory embedding 2,, d(s:) Rm It includes the proprioceptive information of each time step. LSE is time-variant in an RL trajectory, and the state encoder is shared among different tasks. We LSE and sensory representation interchangeably in this paper.
Latent task embedding (LTE). A latent task embedding Z7 E Idd encodes the time-invariant knowledge of specific task. Let's assume we are going train N different tasks, and their embed- dings form an LTE set {2Tx }. These N different tasks share the same state encoder ó and action decoder t/: in other words, these tasks share the same policy network interface, except for their task embeddings being different. For implementation, we adopt fully-connected encoder, which takes as input the one-hot encodings of different training tasks, initialize the set {Z }. This task encoder is learnable during training.
After training, the LTE interface can be reused as a high-level action interface. Hence, we use LTE and action representation interchangeably in this paper.
Training of the multi-task policy networks. A detailed description of the multi-task training is
demonstrated in Algorithm [ When computing objectives and their gradients, we use policy и parameterized by to indicate all the parameters in the state encoder, action decoder, and [ZT, }- The overall training procedure is based on SAC. The only difference is that the policy network and Q networks additionally take input the LTE Z7 and task label, respectively. During training.

---

Published as a conference paper at ICLR 2023
also apply two techniques constrain this space: 1) normalize the LTEs so that they lie hypersphere; 2) we inject a random noise to the LTEs to enhance the smoothness of the space. An ablation study on these two constraints is included in Appendix B.7
Algorithm Multi-task Training
Input: Training task set {Tv) ~ p(T), 01, 02, 0, - 01, 02 + 02 - Initialize LTE set (TTx] J for {TN}
for each pre-train epoch do for Ti in (Tn) } do
Sample a batch B, of multi-task RL transitions with T.
B <- J B;
end for end for
for each train epoch do
Sample RL batch b B
for all transition data in b do
3x. (s,,)
TT, = normalize + n) and n ~ NN0 o2)
Sample action 2-112a, 2T,) for computing SAC objectives end for
for each optimization step do
Compute SAC objectives ((a),, J. (w)) (O) with b based on Equation Update SAC parameters end for end for
Output: The optimal model of state encoder o* and action decoder vi* and set of LTEs {2Tx) }
3.2 THE EMERGENCE OF ACTION REPRESENTATION
After train the multi-task policy network with comprehensive set of tasks, where the LTE vectors in {ZT, }o share the same embedding space, we find that {Ztx] } self-organizes into ge- ometrically and semantically meaningful structure. Tasks with the same modality are embedded in a continuous fashion, which facilitates intra-task interpolation. Surprisingly, the composition of task embeddings from different modalities leads novel tasks, e.g., "run" + "jump" = "jump run". Further, the action representation can be used for efficient task adaptation. Visualization also reveals interesting geometric structures in task embedding and sensory representation spaces. In this subsection, we dive into these intriguing phenomena, demonstrating the emergence of action representation and showing the generalization of the emergent action representation.
Task interpolation & composition. After training the RL agent to accomplish multiple tasks, select two pre-trained tasks and generate a new LTE through linear integration between the LTEs of the two chosen tasks. The newly-generaie task embedding is expected to conduct the agent to perform another different task. The generated LTE defined by:
z' = f( + (T,
where i.j are the indices of the selected tasks and ZT,- 2T, are their corresponding LTEs. 3 is hyperparameter ranging in (0,1). The function ʃ(-) is regularization function related to the pre- defined quality of the LTE Space. For instance, in this paper, f()) is normalization function to extend or shorten the result of interpolation to a unit sphere.
A new task is interpolated by applying the aforementioned operation on the LTEs of tasks sampled from a uni-modal distribution. The interpolated task usually has the same semantic meaning as the source tasks while having different quantity in specific parameters, e.g., running with different speeds. A new task is composed by applying the same operation on tasks sampled from a multi- modal distribution. The newly composed task usually lies in new modality between the source
4

---

Published as a conference paper at ICLR 2023
tasks. For example, when we compose "run" and "jump" together, will have a combination of an agent running while trying to jump.
Efficient adaptation. We find that agent trained with the multi-task policy network can adapt to unseen tasks quickly by only optimizing the LTEs. This shows that the LTEs learn a general pattern of the overall task distribution. When given new task after pre-training, the agent explores in the LTE Space to find a suitable LTE for the task. Specifically, we perform a gradient- free cross-entrops method (CEM) (De Boer al. 2005) in the LTE space for accomplishing the desired task. Detailed description can be found in Algorithm 2
Geometric Structures of LTESs and the LSEs. We then explore what the sensory representations and the action representation space look like after multi-task training. In order to understand their geometric structures, we visualize the LSEs and LTEs. The detailed results of our analysis are presented in Section
Algorithm Adaptation via LTE Optimization
Input: Adaptation task T ~ p(T). o*, s¹* capacity of clite set m, number of sampling n Initialize the elite set Z. with m randomly sampled LTEs from the LTT Space for each adapt epoch do
Initialize the overall test set by 2 + 0 for Z: in L. do
Sample LTES Z11 Bn near Z,
Z < E U (2,, Zi..... 3
end for
for Z, in Z do
while not done do
Ze = =d'(s:)
a: ~ (-)2, 2j)
rt R(s ar
St+1 p(5
end while
end for
Sort the task embeddings in Z by high cumulative reward in the trajectory Select the top LTEs in Z to update Z.
end for
4 EXPERIMENTS
In this section, we first demonstrate the training process and performance of the multi-task policy network. Then, we use the LTEs as high-level action interface instruct the agents to perform unseen skills through interpolation without any training. After that, we conduct experiments to evaluate the effectiveness of the LTEs in task adaptation. Lastly, we visualize the LSEs and LTEs to further understand the structure of the state and action representation. We use emergent action representation (EAR) refer to the policy using the LTEs.
4.1 EXPERIMENTAL SETUPS
Environments. We evaluate our method on five locomotion control environments (HalfCheetah- Vel, Ant-Dir, Hopper-Vel, Walker-Vel, MalfCheetah-Run-Jumee based on OpenAl Gym and the Mujoco simulator. Detailed descriptions of these benchmarks are listed in Appendix [B.T Be- yond the locomotion domain, also conduct simple test manipulation, please refer to Appendix (B6 for detailed results.
method in the domain of robot
Baselines. We compare EAR-SAC, the emergent action representation based SAC with several multi-task RL and meta RL baselines. For multi-task RL baselines, we use multi-head multi-task SAC (MHMT-SAC) and one-hot embedding SAC (OHE-SAC: for ablation). For meta RL baselines, we use MAML (Finn et al. 2017) and PEARL (Rakelly et al. 2019). Detailed descriptions of these baselines are listed in Appendix B.2
5

---

Published as a conference paper at ICLR 2023
HalfCheetah-Vel
Ant-Dir
Hopper-Vel
Walker-Vel
Million Steps
Millinn Steps
Million Steps
Million Steps
Figure 2: Training performance of our method and baselines. The x-axis represents the total training steps (in million steps) and the y-axis represents the average reward of all the training tasks. All the training are based on 3 seeds. The shaded area is one standard deviation.
Method
HalfCheetah- Vel
Ant-Dir
Hopper- Vel
Walker-Vel
EAR-SAC (Ours)
-136.9=1.29
1216.7:54.97
173.5=11.09
172.7::4.05
MAML
4500.9+44..3
222.8++99400
34.0++41.92
-4.9
PEARL
-155.0+15.50
635.9+19.21
170.0+7.80
163.9+1.007
MHMT-SAC
-145.2+2..00
1020.0:::4..0
172.9+1.17
160.6-15.21
OHE-SAC
-609.011.110
195.7:10.29
90.7+2.39
65.0+120.83
Table Comparison with baselines on the final performance. The metric is the return of the last epoch, and the mean and standard deviation is calculated among 3 seeds.
4.2 MULTI-TASK TRAINING FOR ACTION REPRESENTATIONS
In this section, we train the multi-task network and evaluate whether sensory-action representations boost training efficiency. EAR-SAC is compared with all the multi-task RL and meta RL base- lines on final rewards in [Td additionally compared with off-policy baselines on the training efficiency in Figure 2 We find that EAR-SAC outperforms all the baselines in terms of training efficiency. In environments with high-dimensional observations (e.g., Ant-Dir), EAR-SAC achieves large performance advantage against the baselines. We attribute this that the learned action repre- sentation space may provide meaningful priors when the policy is trained with multiple tasks.
4.3 ACTION REPRESENTATION AS A HIGH-LEVEL CONTROL INTERFACE
In this section, control the agent by recomposing the LTEs. We perform intra-action interpolation and inter- action composition in uni-modal task distributions and multi-modal task distributions respectively.
Direction
Intra-action interpolation. Intra-action interpolation conducted in HalfCheetah- Ant-Dir, Hopper- and Walker-Vel We interpolate the action representa- tions between two tasks using Equation] [ The coeffi- cient is searched better fit the target task. An inter- polation example in HalfCheetah- Vel is demonstrated Figure 3 We select the tasks of running at m/s and 2 m/s to be interpolated and get three interpolated tasks: run 1.2 m/s, 1.5 m/s, 1.7 m/s. We perform evaluation on each task for trajectory, and visualize them. In each task, we make the agent start from the same point (not plotted in the figure) and terminate at 100 time steps. Only part of the whole scene visualized in the figure We find that through task interpolation, the agent manages to accomplish these interpolated tasks without any training. We leave more detailed interpolation results in all environments as well as some extrapolation trials Appendix B.4
Figure 3: Interpolated tasks. The top and bottom rows are in-distribution. The middle rows show the interpolated tasks. and we mark the terminals with red flags.

---

Published as a conference paper at ICLR 2023
Moving Direction
Run Jump
Run
Jump
Moving Direction
Walk Stand
Walk
Stand
Figure 4: Visualization of task compositions. Two stop-motion animations of the proposed composition tasks are demonstrated. Animated results are shown in the project page: https:
7/sites poogle.com//veente
htom/view// aeteeeeee--cccioo--eeeeeeeeeeeee
HalfCheetah--e
Ant-Dir
Hopper-Vel
Walker-Vel
ANL S4C, Ierure
Mold, featrne
Steps
Steps
Steps
Steps
Figure 5: Adaptation results. The x-axis represents the adaptation steps and the y-axis represents the average reward of all the adaptation tasks. We fix the adaptation task set for all algorithms in the same environment.
Inter-action composition. Inter-action composition is conducted in BanCChhetaa-RunJJumnn in which we merge two uni-modal task distributions. Taking HalfCheetab-BackkW and HalfCheetah- BackJump as an example, we find that the agent has learned four motions: walking backward, running backward, standing with its front leg, and jumping with its front leg. We select walk- standing as a composition task pair and run-jumping as the other and compose them with Equation These two generated representations are evaluated
1000 time steps and part of their evaluation trajectories are visualized in Figure 4 We find that the walk-standing representation enables the agent to walk backward with a standing posture despite some pauses to keep itself balance, while the run- nun-jumping representation helps the agent to jump when rumning backward after some trials. These empirical results indicate that the LTES can be used as action representations by composing them for different new tasks.
4.4 TASK ADAPTATION WITH ACTION REPRESENTATIONS
this section, we assess how well an agent can adapt to new tasks by only updating the LTEs. We compare the agent's adaptation ability with the meta RL baselines (MAML.PEARL) and the ablation baseline (OHE-SAC). The results
that EAR can adapt to new tasks and achieve the converged performance in no more than three epochs, outperforming the baseline meta RL methods. We note that, with the help of the LTE space, we can adapt to new tasks using zero-th order optimization method.

---

Published as a conference paper at ICLR 2023
4.5 VISUALIZATION OF STATE AND ACTION REPRESENTATIONS
In this section, we further analyze and visualize the sensory-action representations based on the HalfCheetah environment.
Raw State Signals v.s, Sensory Representations
Action Representation Space
Time Stepe: 25
Time Stege: 13
10 L8: 321
555 230 129 Sugoo
66
The Stozy
1.5
413
1223 4ET
+30 Cb
Figure 6: Comparison between raw state sig- nals (left) and sensory representations (right) in Vel. To make the period of running motion easier to observe, we reduce the time interval between adjacent time steps. For now period the motion is 15 steps, while in the original setting it is only 6 steps.
Figure 7: Visualization of the action represen- tation space in HalfCheetah-Ve Vel. Each grid on the unit sphere represents the terminal of an ac- tion representation and colored based on how the agent performs when the policy is condi- tioned on the action representation. The redder the grid is, the faster the agent runs.
The sensory representations. In HalfCheetah- Vel, the halfcheetah agent is encouraged to at a specific velocity, thus making its motion periodical. Therefore, the ideal raw signals and the LSES in the trajectory should be periodical well. After conducting Principal Components Anal- ysis (PCA) (Abdi & Williams 2010) to reduce the dimension of all the raw signals and the sensory representations collected
trajectory, we visualize them for different time steps in Figure 6 We find that the raw state signals appear to be only roughly periodical due to the noise in the states. However, the LSEs show stronger periodicity than the raw signals, indicating that the sensory repre- sentations can reduce the influence of the noise in raw signals, thus helping the agent better under- stand and react to the environment.
The action representations. To better understand the intriguing fact that the agent can adapt to new tasks using the emergent action representations when only sparse training tasks are provided, we plot the LTE space in Figure 7 We find that the LTTES on the hypersphere automatically form a continuous space, constructing the basis of the composition of the representations it. This also explains why the interpolation and composition of the LTEs result in meaningful new behaviors: when interpolated and normalized, the new LTE would still lie on this hypersphere, leading to new and meaningful action sequences.
4.6 ABLATION STUDY
In this section, we ablate the two constraints mentioned in Section 3.1 as well as the overall representation-basee architecture of our method better understand what makes work in the proposed structure. We find that the random noise injection improves the stability of the multi-task training procedure, while the unit sphere regularization improves the quality of the representation space and thus better supports task interpolation and composition. Further, we compare EAR with a different policy network, which takes in a raw state vector and a one-hot task embedding rather than the LSE and LTE, in the multi-task training and task adaptation settings. We find that the one-hot embedded policy performs unsatisfactorily in all environments and fails completely in the Ant-Dir environment, echoing with the idea that simple emergent action representations mean- ingful and effective in task learning and generalization. A complete version our ablation study which contains more detailed analysis is in Appendix B.7
8

---

Published as a conference paper at ICLR 2023
5 RELATED WORK
Representation learning in reinforcement learning. Representation learning has been widely
applied in RL generate representations of sensory information (Laskin et al., 2020 Chandak
et al. 2019 Yarats et al. 202T). policies (Edwards et al., 2019), dynamics (Watter et al. 2015).
In recent years, action representation has attracted more attention. In previous works Dulac-Amoo et al. 2013). the action embeddings have been proposed to compress discrete actions to contin- uous space based on prior knowledge. Recently, many researchers focus on mapping between the original continuous action space and a continuous manifold to facilitate and accelerate policy learn- ing (Allshire et al., 202I) or simplify teleoperation (Losey et al. 2020). Morcover, such action representations are extended to high-dimensional real-world robotics control problems (Jeon et al.) 2020). In this area, a closely related work is Chandak et al. (2019). which enables a policy to output latent action representation with reinforcement learning and then uses supervised learning struct a mapping from the representation space to the discrete action space. However, in our work, discover emergent self-organized sensory-action representations from multi-task policy training without additional supervised learning, which can be further leveraged to generalize to new tasks.
con-
Multi-task reinforcement learning. Multi-task RL trains agent to learn multiple tasks simul- tancously with single policy. A direct idea is to share parameters and learn a joint policy with multiple objectives (Wilson et al.) 2007 Piinto Gupta 2017). but different tasks may conflict with each other. Some works tackle this problem by reducing gradient conflicts Yu et al. (022a)), design- ing loss weighting (Hessel et al.] [019 Kendall et al., 2018), or leveraging regularization (Duong et al. 2015). Modular network has also been proposed to construct the policy network with combi- nation of sub-networks (Heess et al., 2016 Yang et al.) 2020). our work, multiple tasks share the state encoder and action decoder; the action decoder is conditioned on LTE to reduce task conflicts.
Emergent representations are learned for continuous actions during multi-task training.
Unsupervised skill discovery. Unsupervised skill discovery employs competence- based methods to encourage exploration and help agents learn skills. These skills can be used in adapting to new tasks. The unsupervised methods maximize the mutual information between states and skills (Eysenbach) ct al. 2018 Liu & Abbeel, 2021]] Sharma et al., 2019] Xu et al.) 2020 Laskin et al. (022). while our method learns geometrically and semantically meaningful representation in multi-task setting. We believe the proposed method can be complementary to the unsupervised paradigm.
Meta reinforcement learning. The adaptation part of this work shares a similar goal as meta rein-
forcement learning (meta RL) (Finn et al. 2017 Xu et al. [20 8a, Rothfuss et al. 2018 1 Wang et al.,
2016 Levine et al. 2020). Meta RL executes meta-learning (Schmidhuber) 1987 Thrun & Pratt
2012 algorithms under the framework of reinforcement learning. which aims to train an agent to
quickly adapt to new tasks. Many meta RL methods are gradient- based, performing gradient descent on the policy (Finn et al. 2017 Xu et al.] [2018|] Rothfuss et al., (2018). the hyperparameters (Xu jet al. 20T8h), or loss functions (Sung et 2017 Houthooft et | (018) based on the collected experience. Other meta RL algorithms are context- based (Wang et al., 2016 Rakelly et al.) 2019). They embed context information into a latent variable and condition the meta-learning policy on such variables. Besides, recently offline meta RL algorithms (Levine al. 2020 Mitchell et al.) 2021) have also been widely explored, which enables the agent to leverage offline data to perform meta-training. However, we exploit the emergent high-level representations to abstract low-level actions for adaptation rather than introduce complicated new techniques.
6 CONCLUSION
In this paper, we present our finding that the task embeddings in multi-task policy network can automatically form a representation space of low-level motor signals, where the semantic structure of actions is explicit. The emergent action representations abstract information of sequence of actions and can serve as a high-level interface to instruct an agent to perform motor skills. Specifically, we find the action representations can be interpolated, composed, and optimized to generate novel action sequences for unseen tasks. Along with this work, a promising direction to learn the action representation via superviee learning. Another intriguing direction is to learn hierarchical action representations that may capture different levels of semantics and geometric information in motor signals, thus facilitating future applications such hierarchical planning (LeCun) 2022).

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
Yubei would like to thank Yann LeCun and Rob Fergus for sharing their visionary thoughts on action representation learning. We also thank Shaoxiong Yao for participating the early discussion of this work.
REFERENCES
Hervé Abdi and Lynne Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433-459, 2010.
Arthur Allshire, Roberto Martín-Martin Charles Lin, Shawn Manuel, Silvio Savarese, and Animesh Garg. Laser: Learning latent action space for efficient reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp- 6650-6656. IEEE, 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33: 124441144000 2020.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance- inaaeeee-iiaaanneecovaaaannee regullrizzaiiaiiioo for self-supervised learning. arXiv preprint trXiv:2105.04906, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1883 2013.
Rishi Bommasani, Drew A Hudson. Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108, 07258, 2021.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learning action representations for reinforcement learning. In International conference on machine learn- ing, pp. 941-950. PMLR, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems, 31, 2018.
Yubei Chen, Zeyu Yun, Yi Ma, Bruno Olshausen, and Yann LeCun. Minimalistic unsupervised learning with the sparse manifold transform. arXiv preprint arXiv:2209. 2022.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. tutorial the cross-entropy method. Annals of operations research, 134(1):19667, 2005.
Gabriel Dulac-Arnold, Richard Evans, Peter Sunehag, and Ben Coppin. Reinforcement learning in large discrete action spaces. corr abs/1512.07679 (2015). arXiv preprint arXiv: 1512::151,.07679, 2015.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross- lingual parameter sharing in neural network parser. Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 2: short papers), pp. 845 850, 2015.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent poli- cies from observation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 175511151111 1763, PMLR, 09-15 Jun 2019. RtLhtt:: h/tpr//eeocee.n.... mlr. wtt.preas//v77eedaarre 9a httm)
the 36th International Conference
10

---

Published as a conference paper at ICLR 2023
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa- tions, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- veta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1133 1135. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In P Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint 1610.5 2016.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Besa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Hong Jun Jeon, Dylan P Losey, and Dorsa Sadigh. Shared autonomy with learned latent actions. arXiv preprint 20XXx:20005032.000 2020.
E.R. Kandel, J.D. Koester, S.H. Mack, and S.A. Siegelbaum. Principles of Neural Science, Sixth Edition. McGraw-Hie Hill Education, 2021. ISBN 278422464
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 7482-7491, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114.22 2013.
arXiv preprint
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2661- 2671, 2019.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions reinforcement learning. Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 5639-6636666 5650. PMLR, 13-18 Jul 2020. URL https: http:////ccceeiessssee/e .mlr. preas/vi 19/1 fteas//v199/assknn22
html
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cie: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint 11X1v:2202.00161, 2022.
Y LeCun. path towards autonomous machine intelligence. preprini posted on openreview, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. CoRR, abs/2005.01643, 2020.
Hao Liu and Picter Abbeel. Aps: Active pretraining with successor features. In International Conference Machine Learning, pp. 6737. PMLR, 2021.

---

Published as a conference paper at ICLR 2023
Dylan P Losey, Krishnan Srinivasan, Ajay Mandlekar, Animesh Garg, and Dorsa Sadigh. Con- trolling assistive robots with learned latent actions. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2020.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng. Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In Marina Mcila and Tong Zhang (eds.). Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7780-7791. PMLR, 18-24 Jul 2021.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. arXiv preprint arXiv:2203. /2601, 2022.
Lerrel Pinto and Abhinav Gupta. Learning to push by grasping: Using multiple tasks for effective learning. In 20/7 IEEE international conference robotics and automation (ICRA), pp. 2161- 2168. IEEE, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 11448-7733. PMLR, 2021.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning. pp. 5331-5340. PMLR, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784, 2018.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta... hook. PhD thesis, Technische Universität München, 1987.
John Schulman, Sergey Levine Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 2015. PMLR.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- unsupervised discovery of skills. International Conference on Learning Representations, 2019.
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv: (xXiv:1706.09529 2017.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew M. Botvinick. Learning to reinforcement learn. CoRR, 0bs//1111.05666 2016.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Em- bed to control: A locally linear latent dynamics model for control from raw images. CoRR, abs/1506.07366 2015,
cashgg/abs11500 07365
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference Machine learning, pp. 1015-1022, 2007.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.0m 06100, 2022.
Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control prim- itives: Skill discovery via reset-games. Advances in Neural Information Processing Systems, 33: 4999-5010, 2020.
12

---

Published as a conference paper at ICLR 2023
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning explore via meta-policy gradient. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5463-5472. PMIR, 10-15 Jul 2018a.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Ad- vances neural information processing systems, 31, 2018b.
Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-tan reinforcement learnin with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.
Reinforcement learning with pro- totypical representations. Marina Meila and Tong Zhang (eds.). Proceedings of the 38th In- ternational Conference Machine Learning, volume 139 of Proceedings Machine Learning Research, pp. 11920-11931. PMLR, 18-24 Jul 2021. URL https: https::/ppoceeednggs aml....eedinggmm...... mlr..
gress/v133/yaratsslls .tmm
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020a.
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1108 PMLR, 2020b.
Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. First Workshop on Pre- Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022.
13

---

Published as a conference paper at ICLR 2023
A DETAILED DESCRIPTION OF SAC
SAC is a stable off-policy actor-critic algorithm based on the maximum entropy reinforcement learn- ing framework, in which the actor maximizes both the returns and the entropy. In SAC, there three types parameters to update during optimization: the policy parameter w, the soft Q-function parameter 0 and learnable temperature a. The objectives are:
J(a) = Earna [-a log Tue (a, st) - aH
(2)
Jx(W) ES Barnt [a log NTw (ar 5t) Qo (sc+ at)]]
(3)
1 (Qo (s:, at) (r (Sc: ax) + - a (Va (st+1))))
(0) = E )nD
(4)
where 7l in Equation is pre-defined minimum expected entropy. The value function in Equation is parameterized by the weights Vi in the target Q-network and can be defined by:
(S:)
Enyss IQ (St, at) a log (ar 8x)]
(5)
B EXPERIMENTAL DETAILS
B. ENVIRONMENTS
The detailed descriptions of the reinforcement learning benchmarks our experiments are listed as follows:
HalfCheetah- Vel (Uni-modal): In this environment, we train the halfcheetah agent to run at target velocity. The training task set contains 10 velocities. The target velocities during training range from m/s to 10 m/s. For every m/s, we set training task. The adaptation task set contains velocities that are uniformly sampled from [1,10]. The agent is penalized with the 1 error between its velocity and the target velocity.
Ant-Dir (Uni-modal): In this environment, we train the Ant agent to run in a target direction. The training task contains 24 directions which are consecutive integers from 10. The adaptation task set contains 5 integer directions which are uniformly sampled from (0,360) degrees. The agent is rewarded with the velocity along the target direction and penalized with the velocity along the direction perpendicular to our target.
Hopper/Walker- Vel (Uni-modal): Similar to HalfCheetah- Vel, Hopper/Walker- Vel contains 10 velocities in training task set and 3 in adaptation task set. The target velocities when training range from 0.2 m/s to 2 m/s and every 0.2 m/s we set a training task. The Hopper/Walker agent is penalized with the error between its velocity and the target velocity.
HalfCheetah- Run-Jump (Multi-modal): There are two different uni-modal task distributions in CeCChheetah-BakkJnnnn HalfCheetah-
this environment, respectively HalfCheetah-Bace Baraahe--eeeellll Vel and Back trains the halfcheetah agent to run backward a target velocity, and the agent is penalized with the 4 error between its velocity and the target velocity. HalfCheetah-Backk BackJump trains the agent raise its hind leg to jump, and the agent is rewarded with the height of its hind leg. HalfCheetah- Baattahhh-uJJuuupp contains 7 velocities for running backward (from m/s to m/s and every 1 m/s training task is set) and 3 tasks for jumping (3 different weights for the leg height in the total reward function) in the training task set. In our implementation. the jumping task with smallest weight finally turns be motion of "standing" instead of jumping.
B.2 BASELINES
The detailed descriptions of the multi-task RL and meta RL baselines are listed as follows:
MAML. MAML Finn et al. 2017) is gradient-based meta-RL algorithm based on Trust Region Policy Optimization (Schulman et al. 2015). It aims to learn easily adaptable model parameters for tasks in a specific task distribution. MAML conducts explicit training on model parameters so that small number of training data and gradient steps required to adapt to new task. In our implementation, we modify the range and density of task sampling, which will be discussed the following section.
14

---

Published as a conference paper at ICLR 2023
PEARL. PEARL (Rakelly al. 2019) is context-based off-policy meta-RL algorithm based on Soft Actor Critic Haarmoja et al. 2018) and variational inference method (Kingma & Welling PEARL tackles the problem of limited sample efficiency in meta-RL domain. PEARL integrates off-policy RI. algorithms with a latent task variable which infers how to adapt to a new task with small amount of history contexts.
2013).
Multi-head multi-task SAC (MHMT-SAC), MHMT-SAC is a multi-task RL architecture, in which each training task has an independent head input, and the body of the policy network is shared among all the tasks. We compare our method with MHMT-SAC mance of an agent with emergent action representation in multi task training.
evaluate the perfor-
One-hot embedding SAC (OHE-SAC). The sensory-action representation is introduced when training the agent to learn multiple tasks the training set. In OHE-SAC, We omit the represen- tation layer to make the policy network common multi-task RL structure, which concatenates a one-hot embedding and the raw state vector as the input of the policy network. Through com- parison between the final performance of the two methods, OHE-SAC helps identify the effect of action representations in training and adaptation.
B.3 MULTI-TASK TRAINING
B.3.1 NETWORK ARCHITECTURE
In this section, discuss the choice of network architecture. our method, we MLP the network structure of the encoders and decoders. Here we replace the first layer of MLP with an RNN layer to get the RNN encoder and decoder. Our MLP-based network is compared with the RNN-based network. A comparison in the HalfCheetah- Vel environment of the two networks is shown in Figure [ We find that if we simply make such changes with certain amount of tuning, MLP-based network achieves much higher performance than RNN- based one.
HalfCheetah-Vel
-193
1883
EAR RNN
4300
simple
0.
24
6.7
Million Steps
Figure 8: Comparison of MLP and RNN structure.
B.3.2 HYPERPARAMETERS
In this section, we provide detailed settings of our methods. We set up the hyperparameters, as shown in Table 2 for the environments and algorithms in the Mujoco locomotion benchmarks.
B.3.3 TASK SAMPLING DENSITY
In the experiments, we fix the range of task sampling be for different algorithms in the same environment. We find that in HalfCheetah-Ve Vel environment, multi-task RL methods get different final rewards for different tasks (in Figure 99. The reason is that when the agent is trained well enough, the penalty of velocity in the reward will become slight, thus making the control cost of the agent dominant in the reward function. Therefore, larger cost it should pay, leading to a relatively low reward. Thus to make our comparison valid and persuasive, the range of task sampling should be fixed. The detailed settings of the implementation in our paper are demonstrated in Table 3 Similarly, the adaptation tasks are also shared among all the algorithms in the same environment Section 4.4
natural that the faster the agent runs, the
B.4 INTRA-ACTION INTERPOLATION
Additional details for the "Intra-action interpolation". We have performed task interpolation between adjacent tasks in HalfCheetah- Vel in Section 4.3 where interpolate the LTEs of running at Im/s and 2m/s to control the agent to run at 1.2, 1.5, 1.7m/s. Some visualized results are provided in the text, and here demonstrate quantitative analysis based multiple seeds for
15

---

Published as a conference paper at ICLR 2023
Hyperparameters
HalfCheetah-Väl
Hopper/Walker-Ve Vel
Ant-Dir
max episode frames
200
200
200
sensory encoder
2-layer MIP
2-layer MLP
2-layer MLP
action decoder
4-layer MLP
4-layer MLP
4-layer MLP
task encoder
L-aeerr FC
1-layer FC
1-layer FC
Q-network
4-layer MLP
4-layer MLP
4-layer MLP
LSE shape LTE shape
16
16
16
3
3
optimizer
Adam
Adam
Adam
learning rate
3
3
3
discount
0.99
0.99
0.99
batch size replay buffer pretrain epochs training epochs
1280
1280
1920
le6
lc6
1.2c6
20
20
20
4k
5k
10k
optimization times
200
200
200
evaluation episodes
3
Table Hyperparameters of EAR in four benchmarks
HalfCheetah-Vel
HalfCheetah-Vet
-200
.25
-500
-000
-70
-750
100
185
1500
EML
2500
0.
65
0.3
Million Steps
Million Steps
Figure 9: Training curves of tasks with different target velocities with multi-task RL method.
the statistical reliability of our claims. During the experiments, we find that the agent may learn to move forward by hand-standing instead of rumning when faced with the Im/s task in some seeds. makes the postures of Im/s and 2m/s rather different and hard to achieve interpolation. Due to such phenomenon,
change the pre-trained tasks to 2m/s and 3m/s, and the interpolated tasks to 2.2m/s, 2.5m/s and 2.7m/s to fairly assess the stability of task interpolation on multiple seeds. The coefficients / of the task interpolation 3 seeds are demonstrated in Table 4 We can find that as the coefficient of interpolation decreases, the cheetah tends to run faster, which indicates that the LTEs between the embeddings of running 2m/s and 3m/s show continuity in the LTE Space.
Mid-task interpolation. Mcanwhile, to fully evaluate the performance of task interpolation, we conduct another experiment on all environments. We aim adjust the interpolation coefficients to achieve "mid-task interpolation" between pairs of tasks, where a "mid-task" has the average modality of the two base tasks. For instance, in the first seed of the experiment aforementioned, we find that when B 0.42, the agent can run forward at 2.5m/s, which is mid-task of running at 2m/s and 3m/s. The results of mid-task interpolation in 3 seeds are shown Table 11 13
1:1 task interpolation. In the previous parts, we first determine the desired interpolated tasks and find a suitable coefficient, for the task. this part, from another angle, we fix the interpolation co- efficient and observe the corresponding tasks. We perform 1:1 task interpolation in all environments. which means we fix the coefficients 3 to always be 0.5. The results are shown in Table 15 16 18 We summarize the results of the interpolation aforementioned in Table
Task extrapolation trials. Besides interpolation, we also perform extrapolation between tasks, which means make 3 > 1 or 3 0, in HalfCheetah- Vel and Ant-Dir. We demonstrate some of the results in Ta We find that, in HalfCheetah- Vel, the extrapolated behavior can also achieve the target speed as long as the speed is in the distribution of the training tasks. The exception happens
16

---

Published as a conference paper at ICLR 2023
Method
HalfCheetah- Vee
Hopper/Walker-Ve Vel
Ant-Dir
EAR-SAC (Ours)
10 tasks in [0,10] 10 tasks in [0.2]
24 tasks in [0,360)
MAML
100 tasks in [0, 10]
40 tasks in [0,2]
40 tasks in (0,360)
PEARL
100 tasks in [0,10]
20 tasks in [0,2]
100 tasks in [0,360)
MTMH-SAC OHE-SAC
10 tasks in [0,10]
10 tasks in [0.2]
24 tasks in (0,360)
10 tasks in [0,10]
10 tasks in [0,2] 24 tasks in [0,360)
Table 3: Density of task sampling the training procedure.
Task
B
Evaluation
Task
B
Evaluation
Task
B
Evaluation
Vel-2.0
1
2.01m/s
Vel-2.0
1
2.00m/s
Vel-2.0
2.00m/s
Vel-2.2
0.67
2.20m/s
Vel-2.2
0.64
2.20m/s
Vel-2.2
0.69
2.20m/s
Vel-2.5
0.42
2.50m/s
Vel-2.5
0.42
2.50m/s
Vel-2.5
0.44
2.51m/s
Vel-2.7
0.25
2.70m/s
Vel-2.7
0.30
2.69m/s
Vel-2.7
0.29
2.70m/s
Vel-3.0
2.98m/s
Vel-3.0
0
2.99m/s
Vel-3.0
3.00m/s
Table 4: Coefficients and evaluation results of task interpolation in HalfCheetah- Vel.
Metaworld-Reach
Metaworld-Reach
09000
15000
15000
EAR SAC finetune
FEARL_finetung
11000
EAR SAC
1350
MHMT
PEARL
12
Million Steps
Steps
Figure 10: Training performance of method and baselines in Metaworld-Reach.
Figure 11: Adaptation results of method and PEARL in Metaworld-Reade
when the target speed is out of the distribution(e.g. extrapolate Vel-1.0 and Vel-2.0 with >> 1 or Vel-9.0 and Vel-10.0 with / < 0) where the agent can only perform extrapolation within a relatively small range of B. In the higher-dimensional environment Ant-Dir, extrapolation still works in most cases but has higher failure rate.
B.5 INTER-ACTION COMPOSITION
We have performed task composition in the MalfChheetah-Run--le Jump environment. Two stop motion animations are shown in Section 4 and we provide more quantitative results here. The coefficients of the task composition in multiple seeds are demonstrated in Table I0 Composing walking and standing succeeds in all three seeds while composing running and jumping only succeeds two seeds. In the seed where the run-jumping task fails, discover new motion called move-jumping, in which the cheetah agent jumps backward with its hind leg raised.
B.6 APPLICATION IN MANIPULATION
In previous sections, we mainly tackle the classic and widely studied locomotion problems in rein- forcement learning. In this section, apply our method to the domain of manipulation to exam- ine the effectiveness of the proposed method in other domains. We propose another environment Metaworld Retach Reach based on the metaworld (Yu al.) 2020b), in which robot arm is trained to reach different points. We train the agent with multiple reach goals under the proposed multi-task setting and perform fast adaptation with the learned policy. The training and adaptation curves
17

---

Published as a conference paper at ICLR 2023
Success Rate
HalfCheetah- Vel
Hopper- Vel
Walker-
Ant-Dir
Mid-task interpolation
26/27
27/27
27/27
70/72
1:1 task interpolation
25/27
27/27
25/27
66/72
Table 5: Success rate of mid-task interpolation and 1:1 task interpolation on all environments. The definition of success in mid-task interpolation is whether suitable 3 (0.1, 0.9) be found, while in 1:1 task interpolation is whether the interpolated task lies between the two base tasks.
Tasks
Evaluation
Tasks
B
Evaluation
Tasks
Evaluation
1.8
1.04
1.8
3,03
1.8
8.12
Vel-1.0
1.5
1.01
Vel-4.0
1.5
3,50
Vel-9.0
1.5
8.45
1.2
0.99
1.2
3.84
1.2
8.76
&
&
&
-0.2
2.07
-0.2
5.21
-0.2
10.10
Vel-2.0
-0.5
2.19
Vel-5.0
-0.5
5.53
Vel-10.0
-0.5
10.29
-0.8
2.35
-0.8
5.87
-0.8
10.19
Tasks
B
Evaluation
Tasks
Evaluation
Tasks
B
Evaluation
1.8
288.73
1.8
107.03
1,8
255.95
Dir-O
1.5
308.72
Dir-120
1.5
111.38
Dir-270
1.5
259.95
1.2
342.61
1.2
116.81
1.2
268.35
&
&
-0.2
9,46
-0.2
135.53
-0.2
288.98
Dir-15
-0.5
355.72
Dir-135
-0.5
141.67
Dir-285
-0.5
288.36
-0.8
49.29
-0.8
145.02
-0.8
294.08
Table 6: Coefficients and evaluation results of task extrapolation in HalfCheetah-Väl Vel and Ant-Dir in
seed 3.
demonstrated respectively in Figure 10 We find that our method can still find suitable latent task embeddings in the action representation space for adaptation tasks in less than steps.
B.7 COMPLETE VERSION OF ABLATION STUDY
In this section, we ablate the two constraints and the overall representation-based architecture of our method to better understand what makes it work in the proposed structure.
Random Noise Injection. We inject random noise into the LTES during the training procedure enhance the smoothness of the space. To assess the effect of this injected noise, compare it with the case with- out the injected noise in the high- dimenssiil environ- ment Ant-Dir. The training curves are shown in Figure 12 We observe that, when injected with random noise, our method achieves comparable training performance the one without noise. Then we evaluate the policy we learn. For each task each seed, we evaluate the trained policy for multiple trajectories. We find that without the injected noise in training, the policy will be sensitive to perturbation in the system during evaluation. An example of such phenomenon shown in Table 7 In the task of "direction-240" in Ant-Dir, evaluate the policy for 5 trajectories and find that the second and third trials termi- nate before the maximal steps (200 steps), which means they fail halfway. We count the number of such failures respectively and summarize them in Table The policy trained without injected noise fails to perform over 20% of tasks stably, while the one with noise avoids failure.
No.
Terminate steps
Trajectory Trajectory
200
102
Trajectory
144
Trajectory
200
Trajectory
200
Table 7: An example: Direction-240 task in Ant-Din seed 5.
Failure Rate
Number of failures
Without noise With noise
16 in 72 tasks
0 in 72 tasks
Table 8: Failure rates among all training tasks with/without injected noise. There are 24 tasks in Ant-Dir for each seed and seeds in total.
18

---

Published as a conference paper at ICLR 2023
Ant-Dir
Ant-Dir
EARL SAC, withormnie
EAIL SAC whhousrem EAR_SAC
202
EAR_SAC
282
37%
133
2./8
2.55
2.9
540
Million Steps
Million Steps
Figure 12: The comparison of training curves of the policy trained with/without the injected random noise.
Figure 13: The comparison training curves of the policy with normalizednnone-nomaaa LTES and LSEs.
Success Rate
Mid-task interpolation
1:1 task interpolation
Without normalization
64/72
59/72
With normalization
70/72
66/72
Table 9: Success rates of mid-task and 1:1 task interpolation with/without normalization.
Unit sphere regularization. We also apply regulariza- tion on LTEs by normalizing them on the surface of unit sphere. We ablate this regularization by removing the normalization functions for LSEs and LTES and directly concatenating them together as the input of the action de- coder. The training curves are demonstrated in Figure
Ant-Dir
750
13 We find that our method achieves higher sample ef- ficiency and rewards than the one without normalization. Then we compare their performance downstream tasks.
250
direction,
We repeat the interpolation experiments with the new pol- icy without normalization and calculate the success rates
direction 133
201
500
in Table With the normalization, the success rates of interpolation experiments increase, which indicates that normalization improves the quality of the representation space and thus better supports interpolation. Furthermore, also try to perform task adaptation with the new pol- icy. An example is demonstrated in Figure [4] Results show that the policy can still adapt to some tasks while requiring more training steps.
Steps
Figure 14: An example: three random adaptation tasks in seed 1. Two achieve comparable results with our method while the remaining one requires much more steps.
Representatiin-basee architecture. Readers might be curious about whether the learned, ma- nipulable LTEs are better than a naive one-hot embedding. In this part, we compare EAR with different policy network that takes the LSE and LTE. Specifically, in multi-task training, we concatenate the raw state vector with the one-hot embedding as the input of the policy network form a multi-task RL architecture. From the results in Figure and [a we find that the one-hot embedded policy performs unsatisfactorily in all environments and fails completely the Ant-Dir environment. We attribute this the fact that, without the sensory-action representation, the agent fails to understand the underlying relation- ships among multiple tasks, especially in those environments with high high-dimensional observations and large training task set. We also compare both methods in the task adaptation setting in Fig- ure 5) When optimizing the one-hot embedding to adapt to new tasks, the agent can barely reach a reasonable performance in any of the new tasks. These facts echo with the idea that simple emergent action representations are meaningful and effective in task learning and generalization.
raw state vector and one-hot task embedding rather than
19

---

Published as a conference paper at ICLR 2023
Task
B
Task
3
Task
8
Move-jumping Walk-standing
0.41
0.52 Walk-standing
Run-jumping
Run-jumping
0.27
0.42
0.30
Walk-standing
0.45
Table 10: Coefficients of task composition in HalfCheetah-Run Beachheetah-nnn-JJuee Run-Jump.
Target
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
0.47
0.44
0.38
0.44
0.46
0.46
0.47
0,44
0.46
Evaluation
1.60
2.51
3.51
4.50
5.50
6.50
7.51
8.50
9.51
Target
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
8
0.42
0.42
0.38
0.39
0.43
0.45
0.46
0.51
0.48
Evaluation
1.51
2.50
3.50
4.50
5.50
6.50
7.50
8.50
9.50
Target
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
B
0.37
0.42
0.45
0.43
0.49
0.52
0.51
0.53
0.41
Evaluation
1.49
2.50
3.50
4.50
5.50
6.50
7.50
8.49
9.51
Table 11: Mid-task interpolation in HalfCheetah-Ve Vel.
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
0.59
0.50
0.47
0.39
0.51
0.64
0.55
0.64
0.60
Evaluation
0.30
0.50
0.70
0.90
1.11
1.30
1.50
1.70
1.90
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
0.59
0.44
0.45
0.54
0.63
0.54
0.53
0.60
0.61
Evaluation
0.30
0.51
0.70
0.90
1.09
1.30
1.50
1.70
1.90
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
0.51
0.52
0.52
0.58
0.48
0.72
0.63
0.58
0.04
Evaluation
0.30
0.50
0.70
0.91
1.10
1.30
1.50
1.70
1.90
Table 12: Mid-task interpolation in Hopper- Vel.
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
0.54
0.36
0.49
0.38
0.45
0.44
0.66
0.82
0.77
Evaluation
0.30
0.50
0.70
0.90
1.10
1.30
1.50
1.70
1.90
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
0.36
0.23
0.37
0.44
0.41
0.45
0.48
0.58
0.74
Evaluation
0.30
0.50
0.70
0.90
1.10
1.30
1.51
1.70
1.90
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
0.50
0.39
0.46
0.51
0.43
0.52
0.51
0.76
0.62
Evaluation
0.30
0.50
0.70
0.90
1.10
1.30
1.50
1.70
1.90
Table 13: Mid-task interpolation Walker- Vel.
20

---

Published as a conference paper at ICLR 2023
Target
7.5
22.5
37.5
52.5
67.5
82.5
97.5
112.5
0.45
0.82
0.46
0.46
0.49
0.50
0.34
0.40
Evaluation
7.48
22.51
37.81
52.57
67.56
82.50
97.62
112.29
Target
127.5
142.5
157.5
172.5
187.5
202.5
217.5
232.5
0.49
0.68
0.42
0.99
0.73
0.58
0.36
0.70
Evaluation
127.58
142.47
157.47
163.43
187.48
202.53
217.39
232.52
Target
247.5
262.5
277.5
292.5
307.5
322.5
337.5
352.5
0.26
0.67
0.58
0.31
0.36
0.37
0.57
0.47
Evaluation
247.50
262.48
277.50
292.61
307.56
322.49
337.55
352,42
Target
7.5
22.5
37.5
52.5
67.5
82.5
97.5
112.5
0.76
0.56
0.37
0.20
0.59
0.31
0.40
0.40
Evaluation
7.27
22.54
37.51
52.32
67.54
82.58
95.52
112.52
Target
127.5
142.5
157.5
172.5
187.5
202.5
217.5
232.5
0.34
0.22
0.52
0.48
0.41
0.47
0.38
0.27
Evaluation
127.69
142.80
157.56
172.39
187.50
202.50
217.44
232.61
Target
247.5
262.5
277.5
292.5
307.5
322.5
337.5
352.5
0.35
0.56
0.40
0.35
0.86
0.52
0.66
0.48
Evaluation
247.50
262.33
277.49
292.42
307.37
322.55
337.32
352.82
Target
7.5
22.5
37.5
52.5
67.5
82.5
97.5
112.5
0.75
0.50
0.53
0.29
0.46
0.43
0.39
0.99
Evaluation
7.58
22.48
37.44
52.44
67.63
82.53
97.61
112.62
Target
127.5
142.5
157.5
172.5
187.5
202.5
217.5
232.5
0.55
0.47
0.34
0.36
0.30
0.21
0.27
0.32
Evaluation
127.51
142.44
157.37
172.31
187.43
202.68
217.59
232.48
Target
247.5
262.5
277.5
292.5
307.5
322.5
337.5
352.5
B
0.26
0.30
0.30
0.38
0.15
0.55
0.47
0.41
Evaluation
247.44
262.54
277.45
292.84
307.19
322.44
337.74
352.33
Table 14: Mis-task interpolation in Ant-Dir.
Target
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
Evaluation
0.96
2.42
3.30
4.43
5.46
6.45
7.49
8.46
9.49
Target
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
Evaluation
1.23
2.38
3.35
4.40
5.41
6.46
7.47
8.44
9.02
Target
1.5
2.5
3.5
4.5
5.5
6.5
7.5
8.5
9.5
Evaluation
1.25
2.38
3.42
4.42
5.48
6.54
7.49
8.50
9.43
Table 15: 1:1 interpolation in HalfCheetah- Vel.
21

---

Published as a conference paper at ICLR 2023
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
Evaluation
0.31
0.51
0.68
0.91
1.08
1.36
1.54
1.72
1.96
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
Evaluation
0.31
0.46
0.69
0.90
1.14
1.31
1.50
1.73
1.95
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
Evaluation
0.38
0.51
0.69
0.89
1.15
1.34
1.51
1.73
1.90
Table 16: 1:1 interpolation in Hopper- Vel.
Vel.
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
Evaluation
0.32
0.46
0.69
0.87
1.09
1.29
1.82
1.83
1.97
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
Evaluation
0.26
0.46
0.67
0.89
1.08
1.28
1.51
1.72
1.97
Target
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
Evaluation
0.29
0.47
0.69
0.90
1.09
1.30
1.50
1.79
1.95
Table 17: 1:1 interpolation in Walker- Vel..
Target
7.5
22.5
37.5
52.5
67.5
82.5
97.5
112.5
Evaluation
8.65
22.86
38.86
52.07
67.76
79.79
175.41
113.61
Target
127.5
142.5
157.5
172.5
187.5
202.5
217.5
232.5
Evaluation
126.45
142.40
154.05
169.70
185.50
198.52
214.90
231.39
Target 247.5
262.5
277.5
292.5
307.5
322.5
337.5
352.5
Evaluation
243.37
258.17
273.74
290.81
316.52
323.48
338.66
351.11
Target
7.5
22.5
37.5
52.5
67.5
82.5
97.5
112.5
Evaluation
47.26
25.14
39.65
26.37
60.58
81.91
96.71
114.74
Target 127.5
142.5
157.5
172.5
187.5
202.5
217.5
232.5
Evaluation
125.47
133.68
160.27
171.79
190.62
201.69
217.45
229.55
Target 247.5
262.5
277.5
292.5
307.5
322.5
337.5
352.5
Evaluation
243.46
263.07
275.24
291.22
319.19
326.01
337.86
350.19
Target
7.5
22.5
37.5
52.5
67.5
82.5
97.5
112.5
Evaluation
8.60
21.04
37.14
51.72
68.20
84.26
98.73
112.72
Target 127.5
142.5
157.5
172.5
187.5
202.5
2.7.5
232.5
Evaluation
129.20
152.40
157.35
308.21
189.11
201.22
218.09
230.26
Target 247.5
262.5
277.5
292.5
307.5
322.5
337.5
352.5
Evaluation
245,85
238.00
283.31
296.63
308.51
322.86
334.91
352.17
Table 18: 1:1 interpolation in Ant-Dir.
22


Reviews:
[{"review_id": "rFjjGKKzPF", "reviewer": "Reviewer_VRfP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an architectures that separately encodes sensory information and the task id in a multi-task setting. These are then concatenated to processed by a neural network that produces the action. The paper imposes constraints on the task representation (or action representation in the paper): the representation is restricted to the unit sphere and trained with some noise. The paper then evaluates the proposed architecture on a number of simple locomotion domains. Empirically the methods compares favourably to multi-task and meta-RL baselines. The paper also demonstrates that it is possible to interpolate in the task embedding space both within the same task (e.g. different velocities) and between different locomotion modes.", "summary_of_the_review": "Interesting paper showing that a well regularized task embedding is competitive with multi task and meta-RL baselines on a set of relatively toy locomotion domains. The embedding also allows some interesting interpolation. I would like to see some more analysis to understand what specifically makes this approach work. I would also like to see if it scales to less toy settings. ", "strength_and_weaknesses": "Strengths:\n* interesting results with a relatively simple methods\n\nWeaknesses:\n* From what I can tell the paper proposes essentially a well-regularized version of learned task embeddings. It would be nice to see an ablation of the two constraints (noise and restriction to the unit sphere). Are both of these necessary to achieve the results shown? Would other regularization ideas work as well (e.g. information bottleneck?). It is interesting to see that this methods appears to perform well but it is unclear to me what specifically makes it work.\n* relatively toy environments. I think the paper would be substantially stronger if the similar results were shown in more complex settings.\n* I think task representation might be a better term than action representation (since action already has a specific meaning in RL). At the moment I think readers might be confused. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R_Fj2XLxkZw", "reviewer": "Reviewer_MYUq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an action representation scheme for multi-task training, which is relatively simple in that the action representation is decoupled with the sensory represenation. The proposed method allows for emergent action composition as well as interpolation in the action space. The validity of the method is demonstrated on computational experiments in a Mujoco environment, compared against several other RL schemes. ", "summary_of_the_review": "The reviewer thinks that this paper presents an interesting idea on an important problem. The paper will be benefited by adding analysis on the detailed algorithm choices/variations. ", "strength_and_weaknesses": "Strengths: \n- This work is well-motivated. There is certainly a lack of effective task representation for complex multi-task environment. The proposed framework is addressing an timely and important problem. \n- The proposed idea is interesting and looks making sense in general. \n- The method is evaluated, compared agasint a reasonable choice of state-of-the-art \n\nWeaknesses: \n- The methodological details are not clearly described/analyzed. Perhaps the paper is to talk about the arguably simplea idea, but other than the framework in Figure 1, additional description/investigation is needed on what details in action representation would make impact on the performance. Would the MLP be the best choice for this purpose? \n- The use of terminology is a bit abused. It is not very clear the task composition in this work can be called \"emergent.\" \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9P6SZSEniUI", "reviewer": "Reviewer_oCDb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a learning framework (Figure 1) for action abstraction in the scheme of multi-task reinforcement learning (RL). A latent action representation is assigned to each task, and the model tried to learn the tasks with state representation (shared among tasks) and action representations (specific to each task, task embedding) segregated. It was observed that the model self-developed interpretable and compositional action presentations via RL. The learned action representation can facilitate quick, gradient-free transfer learning. More interestingly, it was shown an interpolation in the latent action space led to an explainable \"interpolation\" in the behavior space. The main contribution is a proof of concept of a novel framework for action abstraction.", "summary_of_the_review": "Overall, I appreciate this paper as it proposes a simple method to achieve interesting results, though it poses some demands to the task set. The goodness is the latent action representation showed self-organization via RL without additional, explicit loss functions for this purpose, which provides some insights for both machine learning and cognitive science about how motor primitives could be autonomously acquired and leveraged for transfer learning. The insights are, however, limited because there are a lot of points for the studies to improve as I mentioned above. Overall, my recommendation is around the borderline as for now, and my rating may increase given a constructive rebuttal.", "strength_and_weaknesses": "## **[Strength]**\n\n- The methodology is simple and intuitive (no additional loss functions for action abstraction).\n- The idea and result are quite interesting.\n- Interpolation/composition in the latent action space surprisingly works.\n- The proposed framework provides novel insights on how to leverage the task distribution for learning high-level motor skills\n\n\n## **[Weakness]**\n\n### Related work is incomplete\nWhile the paper mentioned Representation learning in RL, multi-task/metaRL in Sec. 5, important pieces are missing. A clearly related batch of studies are skill discovery with deep RL, which is closely related. Some related references are:\n- Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019\n- Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.\n-  Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control primitives: Skill discovery via reset-games. In Advances in Neural Information Processing Systems, volume 33, 2020.\n\n### The proposed framework heavily relies on the task set\nTo achieve the proposed results, there are quite a few requirements: e.g., a distribution of uni/multi-modal tasks with well-defined reward functions, the tasks shared the same state space. By contrast, skill discovery methods can achieve similar results without a reward function. Furthermore, people and animals can often self-develop motor primitives in a single task.\n\n### Technical details are not clear\nFor example, model architecture, hyper-parameters, training schemes: Sec.A.3.2 is clearly not enough, not to say that source code was not provided. The authors should provide more details for reproducibility.\n\n### Statistical significance of major claims is not well addressed\nWhile the most interesting result --- interpretable interpolation and composition of LTEs, was well presented using an example agent (Figure 3, 4), it is unclear whether this result is common with different random seeds. That is to say, the authors did not demonstrate statistical reliability of the phenomenon showed in Figure 3,4, e.g., what is the percentage of random seeds that can achieve such a result?  More evaluations remain to be complete.\n\n### Lack of ablation studies\nMany design choices lack explanation. For example, the two techniques used to constrain the space, normalizing LTE and injecting random noise. Why normalizing LTE? What will happen if there is no such regularization? I guess these two techniques are essential to the success of the proposed framework, yet remains to be discussed.\n\n### The writing quality is to be improved.\nThe paper seems not well polished. Belows are some issues I found. Nonetheless, I think the paper need a careful and throughout proofreading. Also, reference should also be manually checked.\n- The first sentence in Sec.1: Deep RL can learn --> deep RL agents can learn\n- Page 5: Specifically, We perform ... --> Specifically, we perform\n- Page 5: We visualized the LSEs and and LTEs: duplicate \"and\"\n- Page 9: the end, incorrect citation style\n- Page 10: Duplicate citation of Chandak 2019\n- Page 11: {Bayes}\n- Page 14: control control\n\n\n### **[Questions]**\n1. Will the authors release source code?\n2. How to initialize $\\mathcal{Z}_\\mathcal{T}$ ?\n3. I am curious about the result of extrapolation in LTE space, e.g., if $\\beta < 0$ or $\\beta >1$, what will happen?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6DordS5eMDc", "reviewer": "Reviewer_YSuL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the action representations emerged in multi-task policy training. The authors study a two-stream policy architecture for multi-task reinforcement learning. One input stream encodes the observation and the other input stream embeds the task ID. Finally, the two embedding vectors are concatenated and fed into the policy network. When proper normalization and noise injection are applied, the authors observed that the task ID embeddings space emerges where each embedding in this space encodes a meaning behavior. Embeddings in the emerged space can be interpolated or composed to generate interesting behaviors. The authors then further demonstrate that optimization in this embedding space can yield fast adaptation to downstream tasks. Visualization of the learned state embedding space and task embedding space provides more intuitive understanding of the learned structures.", "summary_of_the_review": "Overall I think the findings in this paper are interesting and could benefit the community. But the presentation can be improved.", "strength_and_weaknesses": "# Strength\n* Overall the paper is clearly written and easy to follow. A few flaws undermines the readability a bit, which are discussed below.\n* The finding of the emergence of the action embedding space is interesting. The proposed architecture is simple and straightforward. Yet this simple architecture yields interesting findings. The strong empirical performance against meta-RL baselines could inspire us to rethink the fast adaptation problem.\n\n# Weakness\n* The tasks in the experiments are relatively simple. Each task can be effectively described by one or two factors - which is nice for visualization and understanding. However, it is unclear if the same phenomenon would be observed in more complicated tasks such as object manipulation.\n* I find the experiment results in Section 4.6 rather confusing. I do not understand the purpose of this experiment. When the task IDs are one-hot encoded, the input layer essentially serves as an embedding layer. It does change the interaction between the state embedding and the task embedding from concatenation to addition, but a certain equivalence can be shown for these two different types of interaction [1]. The fact that this one-hot variant does not even work well during training seems rather confusing to me. I will appreciate if the authors could provide better clarity on this experiment.\n* One question I am interested in asking is the importance of embedding normalization and noise injection. I think it will be good to conduct an ablation study for this purpose.\n* The section paragraph of the related work section looks confusing to me. The authors list a few works that address multi-task RL. But I don't see their connections to this work that are worth highlighting.\n* Writing and presentation can be improved. For example, what are $\\theta_{1}$, $\\theta_{2}$, $\\bar{\\theta_{1}}$, $\\bar{\\theta_{2}}$ in Algorithm 1? The sampled action $a_{t}$ in the second for-loop is never used. Overall, the role of Algorithm 1 is unclear. It presents a standard multi-task policy training procedure which does not highlight LTE. Figure 3 and Figure 4 need more explanation in either the caption or the text. The screenshots are incomprehensible in their current form.\n\n# Questions and discussion\n* The second paragraph of Section 4.3 says \"The coefficient $\\beta$ is searched to better fit the target task\". Why is search needed here? The new task is a convex combination of two reference tasks. I would expect we can just interpolate the embeddings with the same coefficient. Did the authors try it? What did you observe?\n\n# Minor issues\n* In the 5th line in the second paragraph on page 5: \"The newly composed task is usually lie in a new modality...\" Please fix the grammar.\n* In the x-axis of Figure 5: what does \"steps\" mean? Does it mean episodes?\n* In the second paragraph in Section 5: reference to Jeon et al and Chandak et al should use \\citep{}.\n\n**References**\n1. Dumoulin _et al_ 2018, Feature-wise transformations, https://distill.pub/2018/feature-wise-transformations/ ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster