Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper deals with the problem of how to enable the generalization of discrete action policies to solve the task using unseen sets of actions. The authors develop a general understanding of unseen actions from their characteristic information and train a policy to solve the tasks using the general understanding. The challenge is to extract the action's characteristics from a dataset. This paper presents the HVAE to extract these characteristics and formulates the generalization for policy as the risk minimization.

Strengths:
1. This paper shows us how to represent the characteristics of the action using the a hierarchical VAE.
2. From the provided videos, we can directly observe the results of this model applied to different tasks.

Weaknesses:
1. In the paper, the authors mentioned that they proposed the regularization metrics. However, they didn't describe them in details. It is important to develop the proposed method in theoretical style.
2. Analyzing the regularization metrics should be careful in the experiments.
3. Since there are many previous works related to action representation, the experiments should contain the comparison with the other method to see how much improvement was obtained.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies the problem of generalization of reinforcement learning policies to unseen spaces of actions. To be specific, the proposed model first extracts actions’ representations from datasets of unstructured information like images and videos, and then the model trains a RL policy to optimize the objectives based on the learned action representations. Experiments demonstrate the effectiveness of the proposed model against state-of-the-art baselines in four challenging environments. This paper could be improved in the following aspects:
1.	The novelty of the proposed model is somewhat incremental, which combines some existing methods, especially the unsupervised learning for action representation part that just combines methods such as VAE, temporal skip connections…
2.	Some components of the proposed methods are ad hoc, and are not explained why using this design, such as why Bi-LSTM for encoder and why LSTM for decoder.
3.	More definitions about the model should be offered, such as “y^∗ is some optimal action distribution”, how to get the optimal action distribution?
4.	Some datasets are not sufficient enough for sake of statistical sufficiency, such as recommendation data with only 1000 action space.
5.	The contributions of action regularizations are not validated on experiment section.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper addresses the very interesting problem of generalising to new actions after only training on a subset of all possible actions. Here, the task falls into different contexts, which are inferred from an associated dataset (like pixel data). Having identified the context, it is used in the policy which therefore has knowledge of which actions are available to it. 

In general, this paper is written very well, with some minor suggested tweaks in the following paragraphs. The key strategies used here (HVAE to identify contexts, ERM as the objective, entropy regularisation, etc) all make sense, and are shown to work well in the experiments carried out. 

While the experiments are sufficiently varied, it worries me that only 3 or 2 seeds were used. In some cases, such as NN and VAE in the CREATE experiments show large variances in performance. Perfects a few more seeds would have been nice to see. This is the key reason why I chose a 'Weak Accept' instead of an 'Accept'.

Some of the results (the latent spaces) shown in the appendix are very interesting too, particularly since they show how similar actions spaces cluster together in most cases. 

Minor issues: 

1) In Figure 3, I am not clear about what 'im' and 'gt' settings are.
2) In Figure 3, it would have been nice to have consistent colors for the different settings.
3) It would have been nice to see the pseudocode of the algorithm used.

review_title: Official Blind Review #2

Meta-review: This paper proposes a method for reinforcement learning with unseen actions.  More precisely, the problem setting considers a partitioned action space.  The actions available during training (known actions) are a subset of all the actions available during evaluation (known and unknown actions).  The method can choose unknown actions during evaluation through an embedding space over the actions, which defines a distance between actions. The action embedding is trained by a hierarchical variational autoencoder. The proposed method and algorithmic variants are applied to several domains in the experiments section.

The reviewers discussed both strengths and weaknesses of the paper.  The strengths described by the reviewers include the use of the hierarchical VAE and the explanatory videos.  The primary weakness is the absence of sufficient detail when describing the solution.  The solution description is not sufficiently clear to understand the details of the regularization metrics.  The details of regularization are essential when some actions are never seen in training.  The reviewers also mentioned that the experiment analysis would benefit from more care.

This paper is not ready for publication, as the solution methods and experiments are not presented with sufficient detail.
Decision: Reject