Generalizing Reinforcement Learning to Unseen Actions
Authors: Ayush Jain*
Andrew Szot*
Jincheng Zhou
Joseph J. Lim
Venue:  (2020)
Keywords: reinforcement learning
unsupervised representation learning
generalization

Abstract: A fundamental trait of intelligence is the ability to achieve goals in the face of novel circumstances. In this work, we address one such setting which requires solving a task with a novel set of actions. Empowering machines with this ability requires generalization in the way an agent perceives its available actions along with the way it uses these actions to solve tasks. Hence, we propose a framework to enable generalization over both these aspects: understanding an action’s functionality, and using actions to solve tasks through reinforcement learning. Specifically, an agent interprets an action’s behavior using unsupervised representation learning over a collection of data samples reflecting the diverse properties of that action. We employ a reinforcement learning architecture which works over these action representations, and propose regularization metrics essential for enabling generalization in a policy. We illustrate the generalizability of the representation learning method and policy, to enable zero-shot generalization to previously unseen actions on challenging sequential decision-making environments. Our results and videos can be found at sites.google.com/view/action-generalization/

Under review as a conference paper at ICLR 2020
GENERALIZING REINfORCEMENt LEARNING TO UN- SEEN ACTIONS
Anonymous authors Paper under double-bling review
ABSTRACT
A fundamental trait of intelligence is the ability achieve goals in the face of novel circumstances. In this work, we address one such setting which requires solving a task with novel set of actions. Empowering machines with this ability requires generalization in the way an agent perceives its available actions along with the way it uses these actions solve tasks. Hence, we propose a framework enable generalization over both these aspects: understanding an action's 's functionality, and using actions to solve tasks through reinforcement learning. Specifically, an agent interprets an action's s behavior using unsupervised representation learning over a collection of data samples reflecting the diverse properties of that action. We employ a reinforcemen learning architecture which works over these action repre- sentations, and propose regularization metrics essential for cnabling generalization in a policy. We illustrate the generalizability of the representation learning method and policy, to enable zero-shot generalization to previously unseen actions on chal- lenging sequential decision environments. More training and testing videos can be found at sites. google. com wtw. watpo//ieww/ccce http://wwwwcciiiooneeeeeaaaaaaaa generalizationn/
INTRODUCTION
Imagine visiting your friend for the first time, and you decide to cook your favorite dish there. But since you have never been their kitchen before, there could be certain tools you have never seen, like an odd-shaped sponge. However, by looking at its porous texture observing its interaction with water, you can understand that this object can absorb liquid. Later during cooking when you want to clean the table, you select that sponge since you relate its absorbing characteristics with another tool you have used for cleaning. Just like this scenario, our tasks often involve making selections from novel or unseen entities. When encounter such choices, we examine them first. understand their functionality which informs our selection process while solving a task.
Can machines also understand previously unseen choices and subsequently them for solving tasks? From a reinforcement learning perspective, this brings an interesting question of how to enable generalization of discrete action policies solve tasks using unseen of actions. Prior work in deep reinforcement learning has explored generalization over environments (Cobbe et al.) 2018 Nichol al. 2018), and tasks (Finn et al.) 2017 Parisi al. 2018). However, action space generalization is relatively unexplored and is crucial for agents to be flexible in the face of novel circumstances, like selecting an unseen sponge for a known task of cleaning in above example.
In this work, our goal is to develop a framework that reflects the two phases of solving action general- ization: (1) general understanding of unseen discrete actions from their characteristic information (like appearance or behaviors), and (2) training a policy to solve tasks by utilizing this general understanding. However, an action can have diverse behaviors and hence requires a collection of data (e.g. different viewpoints, videos state trajectories of how it effects environment) to sufficiently express this diversity. Hence, the primary challenge is to develop a generalizable unsupervised learning method which extract action's 's characteristies from dataset constituting its diverse effects. To this end, we propose to embed actions' datasets by extending the work on hierarchical variational autoencoders (Edwards & Storkey 2017).
The obtained embeddings reflect an action's 's general utility, and can be used as action representations in the downstream task of reinforcement learning. However, conventional reinforcement learning

---

Under review as a conference paper at ICLR 2020
Initial State
Scenario
Scenario
Moving
Goal
Goal
Goal
(a) Chain Reaction Tool Environment (CREATE)
Initial State
Scenario A
Scenario B
(b) Shape Stacking
Figure 1: Generalizing the knowledge of solving a task to new set of actions. (a) CREATE is a sequential environment where the task help the green ball reach the goal (blue) by selecting tools and deciding where to place them. (b) In Shape Stacking the goal is to stack a tall tower by selecting the right shapes and their placements. Scenario depicts the training scenario when the agent learns to utilize given set of actions to solve the task. Scenario B presents an unseen set of actions to the agent which is expected to generalize to solve the task zero-shot.
algorithms utilize the available actions in way that best optimizes reward. This directly incen- tivizes policy to overfit to the actions seen during training, just like the problem of overfitting to training data in supervised learning. To address this challenge, we formulate this problem as risk minimization (Vapnik 1992) for reinforcement learning, and propose regularization objectives to enforce generalization of policy to unseen actions.
The main contributions of this paper are: (1) introducing the problem and proposed solution to enable action space generalization reinforcement learning, (2) representing an action with a dataset reflecting its diverse characteristics, and employing a generalizable unsupervised learning approach to embed these datasets. (3) a method to use learned action representations in reinforcement learning, and regularization methods enable learning of generalizable policies.
RELATED WORK
Generalization in reinforcement learning In typical deep reinforcement learning (RL) tings (Mnih et al., 2015) 2016 Lillicrap et al. 2015 Schulman et al. 2017), policy or value network learns to act over an action space of fixed dimensionality. By taking states or observations as input to neural networks, these methods are able generalize to unseen environment states drawn from similar distribution as training (Cobbe al., 2018 Nichol al.
2018). Similarly, prior works have explored generalization in RL for unseen instructions (Oh al.
2017 new sequences of
subtasks (Andreas et al. (017). manipulation of unseen tools (Fang et al. 2018 Xie et al.) 2019).
task demonstrations (Xu al. 2017). and agent morphologies (Wang al. 2018 Sanchez-Gonnza
et al. 2018; Pathak et al.) 2019). 9. In contrast, our framework enables zero-shot generalization of RL policies when the agent gets previously unseen action set.
Unsupervised representation learning for downstream tasks Bengio et al. (2013) state represen- tation learning of data makes it easier to extract useful information when building predictors. Prior
2

---

Under review as a conference paper at ICLR 2020
works show that such representations have been useful for variety of downstream tasks, like classifi- cation and video prediction (Denton et al.] (217), visually representing objects for relational reasoning tasks (Steenbrugge al. 2018), representing image-states for domain adaptation in RL (Higgins) et al. 2017), and, representing goals for better exploration (Laversanne- Finot et al.. 2018) and sample efficiency (Nair al. 2018) RI., In this paper, show how unsupervised representation learning over datasets (Edwards & Storkey) 2017) can be used for embedding discrete actions, and enable generalization in the downstream task of reinforcement learning.
Action Representations Using continuous representations of discrete actions, a policy can be trained through combined Q-function over state and action representations (He et al. (015), or in an actor-critic architecture by selecting the nearest neighbor action vector to the policy's 's continuous output (Van Hasselt & Wiering 2009 Dulac-Amold et al. 2015). Unlike our work, these prior works assume access to ground truth action representations, which are usually not readily available. In other related work, action representations are learned implicitly through inverse model on a fixed action space to ease learning in large discrete action spaces (Chandak et al. 2019) or for intrinsic reward (Kim et al. 2019). In contrast, we do not have the assumption of fixed action space as we learn action representations separately, and hence are able to incorporate same policy. While [Tennenholtz & Mannor (2019) pre-learn action representations explicitly using co-occurrence of actions in task-specific demonstrations, our generic embedding method applies to various modalities of datasets to represent actions, which are task-independent and hence suited for generalization to unseen actions.
actions for the
Skill and Trajectory Embeddings In reinforcement learning, variational autoencoders (VAE) (Kingma & Welling 2014) are often used for learning an abstraction for continuous en- tities like skills and state-action trajectories. Specifically, Co--Reyes et al. (2018) utilize a trajectory autoencoder for hierarchical RL, and Lynch et al. (2019) learn a latent space of trajectories and employ coal-conditioned planner over it. Hausman et al. (2018) learn an embedding space of skills through a shared policy for different tasks, and utilize this space for solving other related tasks. In this paper, we extend the framework of hierarchical VAE (Edwards & Storkey 2017 Achille et al.) 2019) to trajectories, so as to embed even sequential datasets which are better indicative action behavior. In general, an action can be discrete skill choice, and an action's 's behavior can be represented as the trajectory of effects it causes on the environment. Since individual trajectories are incapable of capturing the diverse effects of actions, propose to use datasets for representing actions.
3 GENERALIZATION TO UNSEEN ACTIONS
Our approach is based on the intuition that when humans encounter previously unseen discrete entities, we examine them understand their functionality through visual inspection or physical interaction, before deciding what to select for a task. Once the general functionality is inferred, these discrete objects can be used as actions in decision- making tasks. like selecting a tool for cooking or furniture assembly. In this paper, we incorporate these two phases (Figure 2 to enable agents to utilize previously unseen actions: (1) extracting representations of actions from datasets of unstructured information (e.g. image, videos), and (2) training reinforcement learning policy to utilize these action representations with the joint objective of generalization and reward maximization.
In order to represent actions, we note that an action can have diverse behaviors like how it interacts with its environment. Further, there can be various ways an agent observes this dataset. In the sponge example, the action exhibits diverse properties like absorption compression, and the agent can observe this through porous texture (image) or through interacting with it (states trajectory). Therefore, in its most general form, information about an action can be expressed in the form of a diverse collection of unstructured data like images, videos or trajectories. To learn action representations in an unsupervised and generalizable, we use hierarchical VAE and extend it to sequence data like videos (Section 3.2). Next, we show how a policy is trained to use these action representations as input, and propose training objectives for enabling generalization (Section 3.3)
3.1 PRELIMINARIES
For learning agent, denote the entire set of possible discrete actions as A. For evaluation, assume an episodic setting, where the agent only has a subset A C. A of actions available to it. Each action A has an associated dataset D (m),.... } } of observable samples Fn P(ala)
3

---

Under review as a conference paper at ICLR 2020
Action Datasets
HVAE
Action Embeddings
Action
D
Policy
4
Train
O
OOO
π
F +)
D
Inference
Figure 2: Framework for generalization to unseen actions. (1) Action datasets for all training actions are used to train Hierarchical VAE (HVAE) model. (2) The action encoder embeds each dataset to define the approximate posterior qu D)) over action latents c. (3) The instance encoder qa (2|2z, ja, c) encodes each data sample x, while conditioned on the action latent into a distribution over instance latents z. (4) The decoder p(a]= 2, e) reconstructs the action sample x based on the action embedding and sample latent (5) The policy π takes current state 8, and the inferred action embeddings ca for each of the given actions and produces a categorical distribution to represent the policy. Similar flow occurs inference, when new actions and their datasets are given.
which are characteristic of the action During training, the agent only has access to subset Ak C A of known actions. During evaluation, action set A can even be totally unseen for the agent.
The action set A constitutes the discrete action space of an episodic Markov decision process (MDP). Formally, {S, A, T, R. } defines the set of states, actions, transition probability, reward function, and discount factor of MDP. Given set of available actions A A at any time step t, the core problem is to learn parameters of a policy Te ( which defines a probability distribution over actions a, A for state 8t. Since the available action sets A are stochastically sampled and the environments are in general stochastic, we primarily consider stochastic policies in this paper. The performance of To is evaluated based on discounted return R = E T1 7'r(st, 4z) where r is the reward function and T the episode horizon. The aim is to train a policy which only has access to the known actions Ax and its datasets, but generalizes to maximize reward on unseen actions.
3.2 UNSUPERVISED LEARNING OF ACTION REPRESENTATIONS
We represent the diverse characteristics of an action with a dataset of observed information. To extract usable information from these action datasets, we propose an unsupervised representation learning method learn action embeddings. Our key insight is that the common information underlying different samples of an action's ' dataset best represents the general properties of that action.
Therefore, aim to learn action encoder map each discrete action's entire dataset continuous representation. For unsupervised learning of this encoder, we can use a variational autoencoder (VAE) with reconstruction objective (Kingma & Welling, 2014). However, since the input to VAE is in the form of a dataset, it should capture the information shared across multiple data samples. Therefore we encode both, the action datasets and the sample within each action's 's dataset into a hierarchy of connected latent spaces.
Such hierarchical VAE (HVAE) architecture has been explored by EEwwards Storkey (2017) for few-shot classification and clustering of datasets. We use it for the purpose of encoding action datasets and using them for generalization (Figure 2p. HVAE composed of an action VAE over datasets and an instance VAE over samples. The encoders and decoders of the instance VAE are conditioned on its parent action latent vector. For each action a and its associated dataset D = {æI..... wr. the action encoder ga (c) is used to sample action latent while regularized by an action prior Pa (c). For each action sample x E D, the instance encoder qs (2)22, c) is used to sample a latent z encoding the sample instance œ. while conditioned
The prior distribution pa (2|c) well as the decoder p(z) z, c) are also conditioned on the action latent. For each action dataset, ELBO comprises of reconstruction over data samples and the two KI divergence terms (Edwards & Storkey 2017):
4

---

Under review as a conference paper at ICLR 2020
CD (e)g)(11))
Σ B (1n,) (la,( DDla (c(a(c (c)] ZED
(1)
We further extend this framework to incorporate sequential data like state trajectories and videos, as that is more suitable to express behaviors of actions. For a dataset of trajectories we use Bi-LSTM encoder for qu D)),, and LSTM decoder p(7 22, 8) which also takes the initial state & of and reconstructs the rest of it (Schuster & Paliwal 1997 Wang et al. 2017/Co-Reyer et al. 2018). For the case of video datasets, we also incorporated temporal skip connections (Ebert et al. 2017)
from by predicting an extra mask channel, weigh contributions from the predicted frame and the first frame s.
For getting representations of any action E A (seen unseen) through trained HVAE, we use the action dataset encoder qa (c( output's mean as the representation Ca (Figure 2}. This choice of using mean representation follows prior work like Higgins et al. Steenbrugge et al. 2018). but one could also use sampling from the output distribution as representation, as done in Locatello et al. (019).. The generalizability of these representations to unseen actions depends the action's 's behaviors lie in the distribution of behaviors of known actions. Hence, the hierarchy in HVAE makes it expressive encoder for actions, since even seemingly new discrete actions can have characteristics which belong to the distribution of previously seen effects.
whether
3.3 LEARNING POLICIES OVER ACTION REPRESENTATIONS
While solving tasks with new actions, humans first form general interpretation of the behaviors of actions, and then utilize it to take appropriate actions. Similarly, once agent learns actions representations based on observed datasets (section 3.2), it should learn to utilize them for solving tasks. This involves not only extracting the task-specific information from the representations, but also doing so in a generalizable manner so that can utilize previously unseen action representations.
Here assume access for each a E A. Our aim is to learn a policy Ta (a) A, Φ) which maximizes the expected reward under any set of available actions A A. We propose utilize the action representations ca as inputs to the policy, which acts as a function approximator over action representations and states. Specifically, our policy consists of utility function fo x gd IR, which maps d-dimensional action embedding and a state to its utility. The probability distribution over actions is simply defined as the Softmax over the utilities of each available action a' E A.
embedder c, and hence the associated action representations C. o(Da)
(olsmna)
To s, A)
(2)
ENEE efaln,e))
We train the parameters 0 using policy gradient methods
3.4 ENABLING GENERALIZATION TO UNSEEN ACTIONS
The primary objective is to find parameters 0 of policy which maximizes rewards on unseen action sets A. We formulate this generalization problem with statistical learning theory (Vapnik] 1998. 2013), and propose regularization objectives which aim to satisfy its assumptions. The theory mainly deals with generalization in supervised learning problems with an assumption on training examples to be independent and identically distributed (i.i.d. sampled). In reinforcement learning setup with action representations ca, the objective becomes minimizing the theoretical risk of the policy:
min Risk(TT)) = min c, [L(fo(s, cu), max EA (.Is.) RR ca)]
(3)
Here a real-valued loss function which measures the optimality of policy hypothesis Ta (equiva- lently the utility function fa) with respect to the output y² of an optimal stochastic policy T² state s. While L, x or y" cannot be defined in closed form, the definition of optimal policy (Sutton & Barto 2018 SStton et al. (2000) makes this objective equivalent to maximizing the cumulative reward R, given an unseen action space A and their action representations c,, Note that the expectation in Eq. 3 is also over states drawn from environment, but dropped for readability.
5

---

Under review as a conference paper at ICLR 2020
During training. the agent only has access limited set of known actions Ak C A. The standard reward maximization objective in RL with training set of actions, Aς is equivalent to Empirical Risk Minimization (ERM) of the hypothesis To (Vapmik 1992). Hence, the ERM training objective is:
max E Baw (.1s.A)) RR ca)]
However, policy trained with ERM is prone to overfitting to data seen during training, just like in supervised learning. This problem becomes more severe for on on-policy RL because the distribution of input data, (a) used for training To is governed by the actions taken by Te itself. This means that the policy can bias its own training data distribution towards a small subset of actions, while ignoring other actions, which could actually be more informative about the actions available at test time. Since there is no prior information on the distribution over action space A C A at test time, it is assumed to be uniform. Therefore, this discrepancy between training and evaluation due to the non-stationarity of RL training, breaks the identical distribution (in i.i.d) assumption in statistical learning theory (Bousquet al., 2003). To address this con-uniformity in training data, the following regularizing techniques are proposed to augment the ERM objective in Eq.
(1) Maximum entropy regularization: Maximum entropy objective (Ziebart al.
2008 augments Eq. 4 with the stochastic policy's s entropy H|To (a[s)] with weight /3, as in Eq. 5 This makes the policy maximize environment reward, under the constraint of taking diverse actions. This helps generalization in two ways: (a) the input data distribution used for training the policy becomes more uniform over action representations, and (b) the policy outputs maximum entropy distributions which make the least assumptions about the possibly unseen set of available actions A A, and hence by the principle of maximum entropy (Jaynes 1957 Guiasu & Shenitzer 1985) overfits the least.
max Pa RR(s, ca) + (a|ss]
B
(2) Changing action spaces: The training data distribution can be made more uniform by sampling of available actions A C Ak. uniformly in every episode. This blocks certain actions, making the policy select appropriate actions only from the available set A. Hence, the experience collected by the policy is uniformly spread over the known actions Ak, making the training data distribution more identical to the assumed uniform distribution at test time. Eq, shows this training objective:
max E 6
c.) [a[mm[
(3) Clustering similar actions: The known action space AK can contain several groups of similar actions (e.g. various knives for cutting), and randomly sampled action space A may contain actions from each group. This can be exploited by the reward maximiiiii policy during training to overfit to actions from particular groups, but will fail generalize if similar actions are unavailable while testing. To avoid this, we propose to utilize the pre-learned action representations (section 3.2) to partition Ag into a set of k groups 9K [91 ... where i is hyperparameter. For every episode during training, an action set Ag is built by sampling subset of groups, G C 9K and then sampling actions from G only. Two-step sampling ensures that certain groups of actions are blocked every episode, encouraging the policy to utilize underused action groups as well, making training data more uniform over the action representation space. We use equal-sized variant of k-means for clustering. The overall objective is formalized in Eq. beloo:
max E Ea (...s [R(s, ca) + [3[[m(()sss]] where Ag C (ala E 9.9 E EG} and G c GK 0
(7)
In experiments (Section 5). we perform model selection based on validation set of actions. We further ablate each regularization techniques and analyze their contribution in different environments.
4
ENVIRONMENTS
4.1 GRID WORLD
In GRID WORLD environment CChevalier-Boisver et al.) 2018). agent navigates 2D 9x9 maze to reach a goal cell for a sparse reward. A column of lava is randomly placed in every episode, touching which ends the episode. The discrete action space consists of all 5-step maero actions, where each

---

Under review as a conference paper at ICLR 2020
macro-action is defined by 5-length sequence of left, right, up down movement. The entire action space of size 4 = 1024 actions is randomly split into a train and test set of 512 actions. The action datasets collected on empty grid where the agent is initialized at random locations. Two kinds of data types are used to represent the state sequence of agent one-hot vectors and continuous (x.y) grid coordinates.
4.2 RECOMMENDER SYSTEM
The RECOMMENDER SYSTEM environment (Rohde et al. 2018) simulates how users may respond to product recommendations, Every episode, the agent must recommend items to a new user with the objective of maximizing the click through rate (CTR) for the recommendations. This simulated environment uses randomly initialized embeddings for recommendations (actions), and we use the same demonstrate policy generalization to new actions. Action space of size 10,000 is randomly split equally into train and test actions.
4.3 CHAIN REACTION TOOL EN ENVIRONMETT (CREATE)
CREATE a physics- puzzle where the goal is to make a specified ball reach a goal position (blue), inspired by the popular video game The Incredible Machine. The agent must place tools in real time to manipulate the path of the ball to reach the goal position. The environment presents a challenging multi-step step task, requiring the agent to select the tool to place as well as its position (z. on the screen. The agent has access to a subset of diverse tools such as trampolines, see-saws, cannons, funnels, and conveyor belts (Appendix C.2). The position aspect makes this a parameterized action space Hausknecht & Stone (2015) with both discrete and continuous components. Our policy architecture consists of another head to output this continuous vector and it is trained jointly with the discrete action. We solve 3 different CREATE tasks: Push, Navigate and Obstacle. The tools evaluated at test time are completely unseen tool types from those seen during training.
4.4 SHAPE STACKING
In SHAPE STACKING the agent must drop blocks on a table to build the highest standing tower. Our objective is different from prior works (Groth et al., [2018] Lerer et al.) 2016) in that maximize the tower height in an RL setting, whereas the prior work predicts the stability of the tower. Similarly to CREATE, the action space in Object Stacking, consists of (a, y) coordinates of where the object should be dropped above the table. This environment is shows our ability to generalize problem solving ability
new action space in complex 3D task. The action dataset here are images of the objects from various angles (or viewpoints). In this case the visual appearance of the object is sufficient infer its functionality.
EXPERIMENTS
5.1 BASELINES & ABLATIONS
We compare against two policy architectures which can utilize action representations for generalization to unseen action sets. We also compare against a VAE-based non-hierarchical embedding learning method, to learn action representations from unstructured action data (see Fig. 3
Baselines:
Nearest Neighbor: During training, a policy is learned over all known actions. Given unseen actions, the policy's output is used to select the nearest available action in embedding space.
Distance Based: Based on "DulacAAmoo et al. (2015), a continuous action-space policy outputs in the action embedding space and the closest available action to this output is selected.
Non hierarchical VAE: A shared VAE is trained over the samples across all action datasets. An action's 's embedding is then computed as the mean over the embeddings of samples in its dataset.
Ablations: We individually ablate each of the three proposed regularization metrics in our method.
Ours: entropy: Trained without entropy regularization by setting entropy coefficient to zero.
Ours: no changing: Trained over the entire set of known actions without any action space sampling.
Ours: clustering: Training oction-space! is uniformly sampled (Eq. 6 no k-means clustering.
Alternate embeddings: We compare how the embedding learning method (HVAE) applies to various forms of unstructured data (Fig. 3. In CREATE, action datasets comprise of state trajectories for
7

---

Under review as a conference paper at ICLR 2020
CREATE Push
CREATE Obstacle
CREATE Navigate
Success Rate (N) 0
Success Rate (N)
Success Rate (N)
Oura
clusterang
bura: no entropy
burs: no changing
Durs [video]
hierardhizal
Nearest neighbor
Shape Stacking
Recommender
Grid World
Reward
Figure 3: Quantitative results: displayed are 3 of the CREATE tasks, the Block Stacking task, the Recommender task and the Grid World task. The performance displayed is measured generalization to the test set of actions across 3,200 episodes. All results are averaged across seeds. The legend describes ablations of our method (shades of red), embedding baselines (shades of blue), policy architecture baselines (shades of green), and alternate modalities in learning embeddings (yellow).
tool behavior, except Ours (video) where video datasets used instead. In Grid World, action datasets contain trajectories of states in one-hot representation, except in Ours (state) where states real-valued 2D coordinates. Ours (Ground Truth) representations are not learned, but instead uses manually engineered representations for the actions. Detailed descriptions are present in Appendix C
5.2 QUANTITATIVE RESULTS
The generalization performance of the policy to unseen actions across all environments and method variations is shown 3 As seen from the results our method or ablations all of our methods have the strongest ability to generalize to unseen actions across a variety of environments. The difference among our ablations smaller simpler environments like Grid World, Recommender systems and Shape Stacking, where the unseen action spaces are very similar to training actions. The effect of ctuttring--bsse sampling and entropy regularization can be seen for Obstacle and Navigate environments, which require solving the task with quite different tools at testing. CREATE Push is solvable with a wide variety of tools, and hence the no-entropy policy trains to a higher reward, and able to generalize well as many unseen tools solve the task easily. The performance of our method against its variant with non-hierarchical VAE embeddings shows the importance hierarchy in latent space to represent actions.
We test the generalizability of our embedder and policy for the task of zero-shot generalization to unseen actions. Specifically, our primary experiments across all four environments, discussed in section 4 train a policy on a fixed set of actions, tune hyperparameters on separate evaluation set, and then test the ability generalize analysis on cases where this generalization succeeds and fails. Finally, we evaluate how our method's generalizability varies with the degree of difference between seen and unseen.
new of actions. We further provide qualitative
5.3 FURTHER ANALYSIS
Qualitative results of the policy test performance are shown in The left and middle column contain success cases. In the left column for CREATE seen the policy, despite never having used of the tools before, still be able to solve the task. Likewise, for shape stacking we see the policy able to novel shapes to build tall and stable pile to maximize the height. We also show cases of failure
8

---

Under review as a conference paper at ICLR 2020
Angle Generalization
Embedding Generalization
Effect of Unseen Actions
Success Rate (S)
Successs Rutter (S)
Successs Rotes
Angle Difference (degrees)
Embedding Distance
Unseen
Figure 4: Varying difficulty of test action space: (i) Each test action at least a specific angle apart from all actions seen during training (ii) Each test action is at least specific distance in embedding space apart from all actions seen during training (iii) Test set contains seen/unseen ratio
generalize the right most column. both cases the policy chooses the right types of actions and barely misses the objective.
We also analyze the conditions needed for generalization to unseen actions. We perform all analyses on CREATE Push task because of the large diversity of tool functionalities. We show generalization across changing physical tool parameters with angle and the embeddings the policy is trained and tested on. Finally, we show the effect of unseen versus seen actions performance.
Figure 5: Qualitative analysis: shown are two success cases and one failure case for CREATE and Object Stacking. In CREATE the trace of the ball trajectory outlined. All of the tools or objects in these results the policy is generalizing to select and was not trained over these actions.
CONCLUSION
Generalization to novel circumstances is important ability have, for robust and widely applicable artificial agents. In this paper propose the problem of generalization of reinforcement learning policies unseen spaces of actions, with the use of action representations learned in an unsupervised manner. Our two-phase framework demonstrates how representation learning can be combined with downstream task of reinforcement learning, specifically to represent actions. demonstrate efficacy of our methods on four challenging environments, and discuss which variants work when. The key takeaway is that when unseen actions are quite different from known actions, then more regularization helps to train generalizable policies.
REFERENCES
Alessandro Achille, Giovanni Paolini, Glen Mbeng, and Stefano Soatto. The information complexity of learning tasks, their structure and their distance. arXiv preprint arXiv:1904.032000 03292, 2019.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pp. 166-175, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588) 2013.

---

Under review as a conference paper at ICLR 2020
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning theory. In Summer School on Machine Learning, pp. 169 209- 20..2077..P. Springer, 2003.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip S Thomas. Learning action representations for reinforcement learning. arXiv preprint arXiv:1902.00/1 2019.
Maxime Chevalier- Boisvert, Chevalier-Boisvert,, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: / / thub. com
www.//o.///ccocccoooooooo//ocoaa max metaximmeen www.e http///gggmmiiiiii.m.... 2018.
John Co-Reyes, YuXuan Liu. Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey
Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory
embeddings. In International Conference Machine Learning, pp. 1008-1017, 2018.
Karl Cobbe, Oleg Klimov. Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization 2018.
in reinforcement learning. arXiv preprint arXiv:
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pp. 4414-4423 2017.
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint 15/2.0777/ 2015.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learning, pp. 344-356, 2017.
Harrison Edwards and Amos Storkey. Towards a neural statistician. International Confer-
Learning Representations, 2017. URL httpss://operrevllww http:///wwweeeee/rmttmtee
HJDBUF51e
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, and Silvio Savarese. Learning task-oriented grasping for tool manipulation from simulated self-supervision. arXiv preprint arXiv:1806.092: 2018
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135, 2017.
Oliver Groth, Fabian Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised stacking. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 702-717, 2018.
Silviu Guiasu and Abe Shenitzer. The principle of maximum entropy. The mathematical intelligencer, 7(1):42-48, 1985.
Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. arXiv preprint
2015.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018. URL /ttL/ttts////tereee////
He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv preprint 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning, pp. 1480-1490, 2017.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
10

---

Under review as a conference paper at ICLR 2020
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. EMI: Exploration with mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3360-3369 Long Beach, California, USA, 09-15 Jun 2019 9. PMLR. UtkLhhttp://prcceeeiiie
ml press / httmlll
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 2014.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://github
com/ c http://kssr.kko//yttrrtttt a2c ppo acktr- gai 2018.
Adrien Laversanne-
Alexandre Pere, and Pierre- Yves Oudeyer. Curiosity driven explo- ration of learned disentangled goal spaces, In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 487-504, PMLR, 29-31 Oct 2018. URL f not 8a html
htttp://proceddii ngs mm htt.gg/......../...bb.e
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition block towers by example. In International Conference on Machine Learning, pp. 430-438, 2016.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2015.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint 100.005,, 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of dis- entangled representations. In International Conference on Machine Learning, pp. 41 4124, 2019.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. arXiv preprint 1933vv1903.01973, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning, pp. 1928-1937, 2016.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances Neural Information Processing Systems, pp. 9191-9200, 2018.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:1804.03720, 2018.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning, pp. 2661-2670, 2017.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. arXiv preprint 1rXiv:1802.07555 2018.
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A. Efros. Learning to control self-assembling morphologies: study of generalization via modularity. arXiv preprint 1rXiv:1902.05544 2019.
11

---

Under review as a conference paper at ICLR 2020
David Rohde, Stephen Bonner, Travis Dunlop, Flavian Vasile, and Alexandros Karatzoglou. Recogym: A reinforcement learning environment for the problem of product recommendation in online advertising. arXiv preprint arXiv:1808.000/ 2018.
Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg. Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning. 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions Signal Processing, 45(1 (5(11):2673-2 2):27332661, 1997.
Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generalization for ab- stract reasoning tasks using disentangled feature representations. arXiv preprint 1r.4444811.04784. 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057-1063, 2000.
Guy Tennenholtz and Shie Mannor. The natural language of actions. arXiv preprint 1r 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Hado Van Hasselt and Marco A Wiering. Using continuous action spaces to solve discrete problems. In 2009 International Joint Conference on Neural Networks, pp. 1149-1156. IEEE, 2009.
Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances in neural information processing systems, pp. 831-838, 1992.
Vladimir Vapnik. Statistical learning theory. 1998.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference Learning Representations, 2018. URL
https://openreviie
net/ wtt.pp/rwwreeee.eeeooooooaaaaaaa Sww.mmiid-Ssqiiiiii
Ziyu Wang. Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Guyon, U. Luxburg, S. Bengio, Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5320-5329, Curran Associates, Inc., 2017. http: paper/7116 robust imitation of of-diverse www.ddeere--bevvoosrs.... pdf
nips cc/
behavicrs
Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight, 2019.
Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In International Conference on Robotics and Automation, 2017.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008
12

---

Under review as a conference paper at ICLR 2020
A ALGORITHM
Algorithm 1 Training
Require: Set of known actions Ax {ar,... aN associated action datasets {D1, DN). number of clusters & and number of clusters sample from S Initialize HVAE model parameters
for epoch = 1, .... do
Sample 02 ~ AA
Compute loss in Eq. for D;
Update HVAE model with gradient of Eq. end for
Initialize empty set Cк {}
Infer action representations Ca. = 0% (c))), Va, AK {n denotes mean of distribution) Store action representations Cк + {Cay . Initialize policy parameters
Compute k-means clustering on Ck: Gr < (91.....9H.
4-00 0
Receive initial state Sy from ENV and initialize replay buffer while not done do for step in rollout buffer do
Sample clusters G GK
Sample action set Ag from clusters G as in Eq.
Compute fs Ca,). Vay Ag
Sample action ~ (s) using Eq. 2
St+1, F4 ENV(S., 0z)
Store experience (st, 8141, rτ) in replay buffer ++t+)
end for
Update based with PPO using rollout buffer end while
Algorithm 2 Testing
Require:
Set of test actions AT = {aN+1... aN+M) and action datasets (D DN Infer action representations Ca; - Lhd (c),, Va, Ar
while not done do Compute fo(st: Ca.), Va, E Ar Sample action ~ TO using Fq. 2
8t+1, P2 <- ENV EINV(st, at)
+ +-++1
end while
HVAE implementation is based the PyTorch implementation of Neural Stastician (Edwards & Storkey 2017). and we use RAdam optimizer (Liu et al.] 2019). For training our policy, we use PPO (Schulman et al. 2017 Kostrikov 2018) with Adam optimizer (Kingma & Ba 2015
13

---

Under review as a conference paper at ICLR 2020
B ANALYSIS OF TRAINING PROCEDURE AND MODELS
B.1 VISUALIZATION OF HIERARCHICAL EMBEDDING SPACES
Boany
Douncy, Recagen
Beancy Portagon
Beany Sauar
Food
Fond, Jeebece
Fioz Square
Food Thangle
Trenss
Hr.pp
State-based Action Embedding
Image-based Action Embedding
(a) CREATE
Figure 6: T-SNE Visualization of learned embedding space for CREATE environment. Tools in CREATE are labeled by their properties which define them to be floor, trampolines, high-frictional, etc. The action embeddings clearly group similar actions together.
State-based Action Embedding
One-hot prediction based Action Embedding
GRID WORLD
Figure 7: T-SNE Visualization of learned embedding space for Grid World environment.
14

---

Under review as a conference paper at ICLR 2020
Arth
Cont Hies
Dome
Ellosold
Hall Jectangle
Horisonaal Bectangle
Mikkk, Solkk_Tring Tranggle
Rectangle
Sphere
Terratedron
State-based Action Embedding
(a) SHAPE STACKING
Figure T-SNE Visualization learned embedding space for Shape-stacking environment.
B.2 PERFORMANCE CURVES FOR ABLATIONS
CREATE Push
CREATE Obstacle
CREATE Navigate
Succeec
Figure 9: Success rate curves comparing the learning of the primary method Ours ablations. The success rate shown being evaluated on the validation action set. Note that performance decreasing on the validation set indicates overfitting the training set of actions.
B.3 TUNE-TUNNG COMPARISON
CREATE Push
CREATE Obstacle
CREATE Navigate
1.
2.4
Enviormmnt Steps
Enstronmer Steps
Enviforment Steps
Figure 10: Fine-tuning the policy to adapt to new action space by re-initializing the final layer. Performance is computed across seeds trained from the same model.
One possible way to adapt policy an unseen set of actions is to spend time fine-tuning the learned policy for the new action space. This process keeps the earlier layers the policy but fine-tunes the actual action selection in the final layer by re-initializing the output layer. This process is able to adapt to an unseen action set of any size. However, this finettuning process can be time consuming which is why try achieve generalization in this paper. In Figure T0, we show the performance
15

---

Under review as a conference paper at ICLR 2020
of such fine tuning the three CREATE tasks. As from the figure, it takes millions of steps to fine-tune, proving prohibitive for adaptive policies. On the other hand our framework generalizes to the new action space without the need for expensive RL retraining.
C ENVIRONMENT DETAILS
C.I
GRID WORLD
GRID WORLD environment consists of agent and lava wall with opening as shown in figure The lava wall can be either horizontal or vertical. The agent is spawned in random position and can move in 4 directions (up, down, left, and right). The objective of the agent is reach the goal in the bottom-right right corner while avoiding lava.
Observation
Action
RDDDDD
DRRUUU
UUUDDA
RADDDD
Observation
Action
LLUUUU
DRRDLL
LLDDAR
RDDRAL
Figure 11: GRID WORLD environment. The agent is the red triangle and the goal the green cell. The environment contains one row or column lava wall with single opening. Each action consists of consecutive moves in 4 directions.
Observations: The observation space is 9x9x3 where three channels represent (1) object id (agent, wall, goal, lava). (2) color of an object, and (3) zero in each cell. The agent gets flattened vector of this 9x9x3 matrix.
Actions: An action of the agent is 5 consecutive moves in 4 directions. Hence, 4 1, 024 actions are possible in total. Once the agent selects an action, executes 5 sequential moves step-by-step. During action execution, if the agent hits the boundary, it will stay in the current cell. If the agent steps on lava, the game will be terminated. The whole action set is divided into 8:1:1 split of train, validation and test action sets.
Rewards: GRID WORLD provides a sparse reward, 1 0.9 x (step/maxx sttpssstepp.... only when the agent reaches the goal. The reward is discounted based the number of actions taken to encourage shorter path to the goal.
Termination: Each game is terminated when the agent takes max_stepp (64) actions or the agent reaches the goal or the lava wall.
Action Datasets:
The action datasets observations of an agent performing actions in 80x80 grid with no obstacles. The observations constitute the trajectory of states the agent sees during application of macro-action, starting from randomized initial states. A dataset of 1024 such trajectories is used to represent a single macro-action, and is encoded by the HVAE to an action representation which identifies the underlying action behavior. We consider three kinds of embeddings our experiments:
One-hot (default): State is represented by two 80-dimensional one-hot vectors of x and coordinates of the agent's 's locations on the 80x80 grid. Autoencoder reconstruction is based on a softmax cross-entropy loss over the states in the trajectory. The learned action embeddings 1 16-dimensional.
16

---

Under review as a conference paper at ICLR 2020
State: Two-dimensional continuous vector of and y coordinates of the agent in the 80x80 grid. Reconstruction loss is computed based on Gaussian log- eggliielihood over the states in the trajectory. The learned action embeddings are 16-dimensional.
Ground-truth: These are 5-dimensional embeddings containing the true knowledge of the five moves (up, down, left, right) that constitute a macro-aaciin.
C.2 CHAIN REACTION TOOL ENVIRONMENT (CREATE)
CHAIN REACTION TooL ENVIRONMENT (CREATE) is physics-based puzzle where the objective is to make a target ball reach goal position by placing variety of tools, inspired by the popular video game "The Incredible Machine". The environment contains two movable objects, marker ball (green) and target ball (red). When game starts, the marker ball is falling off from the top of the screen and an agent requires to place tools to redirect the kinetic energy of the marker ball to the target ball that the target ball reaches the goal position (blue) illustrated in Figure C.2
Observation
Action
Put ramps
Put tramppline
bump ramp Put buump ramp
Figure 12: CREATE environment. In CREATE, the green ball falling into the scene, which must push the red target ball into the blue goal location. The top and bottom rows show actual evaluation results when our model tested on CREATE UP and CREATE DowN, respectively.
Observations: An observation for each time-step is an 84x84x3 image of a game screen and 3 frame stack to provide information about velocity and acceleration the balls. Initially, the game contains 3 balls in the observation: marker ball (green), target ball (red), and goal ball (blue).
Actions:
CREATE contains 1,737 tools of type: ramps, trampolines, walls, balls, floor, conveyor belts, funnel, polygons of different shapes, cannons, fans and buckets. We also have action for No-Operation. The whole tool set divided into train (939), validation (400) and test sets (400). Every time-step, the agent outputs a parameterized action, i.e. discrete-continuous action which has three values (tool, x. v). where tool specifies which tool to place and (æ, y) represents the position of the tool in the screen.
The agent gets +1 reward when the marker ball hits the target ball, and +10 reward when the target ball passes the goal. In addition reward for success, the environment provides some intermediate rewards. For every time-step, +0.01 reward is given to encourage the agent to keep balls inside the screen, At the same time, an invalid action is penalized by giving -0.01 reward (e.g.. placing tool outside of the screen placing tool on top of other tools).
Rewards:
Termination: Each game is terminated when the agent takes 20 actions, or the marker ball goes out of the screen before hits the target ball, or the target ball goes out of the screen before it passes the goal.
The action dataset is constructed by testing the properties of each tool through scripted interactions with a probe ball. ball launched at a given tool from various angles and speeds and then interacts with the tool. The properties of the tool will determine the deflection path of the ball. Testing interaction from various angles and speeds helps to build a better understanding of the tool. The collected datasets consist of 1024 trajectories of length 10 of the ball's interaction with the tool. Each trajectory consists of observations in the form of either environment state (default) or 48x48 gray-scale images (Ours (image) in Fig. 3) We consider three kinds of embeddings in our experiments:
Action Dataset:
State trajectory (default): Each environment state in a trajectory is represented by 2D coordi- nates and 2D velocity of the probe ball, concatenated with 2D tool's location. Autoencoder reconstruction is based on Gaussian likeliiid over the states in the trajectory. The learned action embeddings 2i-diinns...
17

---

Under review as a conference paper at ICLR 2020
Video: Each state in a trajectory is represented by 48x48 gray-scale image, which makes the trajectory a video of probe ball's interaction with the tool. Autoencoder reconstruction is based on Gaussian oeg-likeihood over the images in the input video. The learned action embeddings are 128-dimensional.
Ground-truth: These are 32-dimensional embeddings containing the true knowledge of the tool in the form of one-hot encoding of the tool type along with its properties such as angle, length, elasticity, etc.
C.3 SHAPE STACKING
SHAPE STACKING a mujoco simulation environment (Todorov et al., 2012) where the agent is given a set of objects of different shapes of varying sizes, for instance, cubes, rectangles, spheres, round cylinders, archs, etc. The objective is to stack a tower as high as possible by choose the appropriate objects given in particular episode.
Observations: observation for each time-step is an 84x84x3 image of the shapes laying on the table.
Actions: total there are 900 distinct shapes in the environment. 675 are used for learning the policy. The remaining 225 are used for evaluating performance unseen actions. The same types of polygons do not appear both train and test. Like CREATE the action space includes making discrete selection over the shape to drop and the x, y coordinates to drop the shape above the scene.
Rewards: The agent receives reward for the positive difference in height of the tower. A penalty of -0.25 is given for every repeated shape.
Termination: The game terminated either after the tower of shapes exceeds in height or after 10 shape placements.
In Shape Stacking the functionality of each action is characterized by the physical appearance of the shape. For this reason, the action dataset consists of images of the shape from various angles and heights (viewpoints). 1,024 images of resolution 84x84 constitute the dataset to represent single shape.
Action Dataset:
18


Reviews:
[{"review_id": "rkxczGYx9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with the problem of how to enable the generalization of discrete action policies to solve the task using unseen sets of actions. The authors develop a general understanding of unseen actions from their characteristic information and train a policy to solve the tasks using the general understanding. The challenge is to extract the action's characteristics from a dataset. This paper presents the HVAE to extract these characteristics and formulates the generalization for policy as the risk minimization.\n\nStrengths:\n1. This paper shows us how to represent the characteristics of the action using the a hierarchical VAE.\n2. From the provided videos, we can directly observe the results of this model applied to different tasks.\n\nWeaknesses:\n1. In the paper, the authors mentioned that they proposed the regularization metrics. However, they didn't describe them in details. It is important to develop the proposed method in theoretical style.\n2. Analyzing the regularization metrics should be careful in the experiments.\n3. Since there are many previous works related to action representation, the experiments should contain the comparison with the other method to see how much improvement was obtained.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJlWHQoTYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the very interesting problem of generalising to new actions after only training on a subset of all possible actions. Here, the task falls into different contexts, which are inferred from an associated dataset (like pixel data). Having identified the context, it is used in the policy which therefore has knowledge of which actions are available to it. \n\nIn general, this paper is written very well, with some minor suggested tweaks in the following paragraphs. The key strategies used here (HVAE to identify contexts, ERM as the objective, entropy regularisation, etc) all make sense, and are shown to work well in the experiments carried out. \n\nWhile the experiments are sufficiently varied, it worries me that only 3 or 2 seeds were used. In some cases, such as NN and VAE in the CREATE experiments show large variances in performance. Perfects a few more seeds would have been nice to see. This is the key reason why I chose a 'Weak Accept' instead of an 'Accept'.\n\nSome of the results (the latent spaces) shown in the appendix are very interesting too, particularly since they show how similar actions spaces cluster together in most cases. \n\nMinor issues: \n\n1) In Figure 3, I am not clear about what 'im' and 'gt' settings are.\n2) In Figure 3, it would have been nice to have consistent colors for the different settings.\n3) It would have been nice to see the pseudocode of the algorithm used.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkg_dJ1RFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of generalization of reinforcement learning policies to unseen spaces of actions. To be specific, the proposed model first extracts actions’ representations from datasets of unstructured information like images and videos, and then the model trains a RL policy to optimize the objectives based on the learned action representations. Experiments demonstrate the effectiveness of the proposed model against state-of-the-art baselines in four challenging environments. This paper could be improved in the following aspects:\n1.\tThe novelty of the proposed model is somewhat incremental, which combines some existing methods, especially the unsupervised learning for action representation part that just combines methods such as VAE, temporal skip connections…\n2.\tSome components of the proposed methods are ad hoc, and are not explained why using this design, such as why Bi-LSTM for encoder and why LSTM for decoder.\n3.\tMore definitions about the model should be offered, such as “y^∗ is some optimal action distribution”, how to get the optimal action distribution?\n4.\tSome datasets are not sufficient enough for sake of statistical sufficiency, such as recommendation data with only 1000 action space.\n5.\tThe contributions of action regularizations are not validated on experiment section.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for reinforcement learning with unseen actions.  More precisely, the problem setting considers a partitioned action space.  The actions available during training (known actions) are a subset of all the actions available during evaluation (known and unknown actions).  The method can choose unknown actions during evaluation through an embedding space over the actions, which defines a distance between actions. The action embedding is trained by a hierarchical variational autoencoder. The proposed method and algorithmic variants are applied to several domains in the experiments section.

The reviewers discussed both strengths and weaknesses of the paper.  The strengths described by the reviewers include the use of the hierarchical VAE and the explanatory videos.  The primary weakness is the absence of sufficient detail when describing the solution.  The solution description is not sufficiently clear to understand the details of the regularization metrics.  The details of regularization are essential when some actions are never seen in training.  The reviewers also mentioned that the experiment analysis would benefit from more care.

This paper is not ready for publication, as the solution methods and experiments are not presented with sufficient detail.
Decision: Reject