Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to learn a linear subspace for actions to control agents. It is well-known in the RL community that choice of action space has a dramatic effect on the learning process. For example, end-effector control in robotic manipulation is much easier than direct torque control for several tasks. Thus, learning better action spaces is an important research problem. This work specifically aims to learn local linear subspaces for actions, and utilizes NHT for efficient learning and inference. The paper also demonstrates that subspaces learned using NHT have desirable local smoothness properties.
strength_and_weaknesses: The paper proposes a nice algorithmic approach for the problem studied -- i.e. finding local linear sub-spaces for actions. However, the paper does not do a good job of motivating why this is an important problem. What are the specific advantages of linear subspaces? Why is it preferable to nonlinear per-timestep action-space mappings (nonlinear latent actions) and trajectory-level action-spaces (motor primitives)?
clarity,_quality,_novelty_and_reproducibility: - The main baseline, besides the simple SVD baseline, is LASER. However, the paper does not review or explain what this baseline is. I would encourage the authors to review this method to situate their work in better context. 
- Per my quick read, LASER seems to be a representative algorithm for the latent actions framework. The paper can compare against a representative algorithm from the motor program framework.
- The paper can better motivate the importance of the research question they study and specifically the advantages of a linear subspace for actions.

summary_of_the_review: It is unclear why this problem is important and relevant to the NeurIPS community. The authors can either motivate the problem statement better or submit to a more specialized conference where it might better appeal to the audience.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed neural household transforms (NHT) for learning a low-dimensional action representation for RL problem. Prior work in this area usually learns an autoencoder-style action representation using neural networks. In this work, the authors propose to learn a network that maps to an orthogonal matrix instead of the original action space. By doing this, it enables them to avoid the need of encoding the action into the latent space as it can be obtained by an analytical least square solution. To obtain a well-behaved mapping function, exponential map is used to map the neural network output onto a unit sphere, which is then formed into the orthogonal projection matrix. The authors provided theoretical justifications for the approximation error as well as the smoothness of the proposed method. The algorithm is evaluated on two manipulation tasks and a locomotion task in simulation and is demonstrated to outperform baseline methods.
strength_and_weaknesses: Strengths:
1. The proposed NHT algorithm is intriguing, with theoretical justifications, and without much compromise to make the method practical.
2. Compared to alternative methods that learn a latent action space, the proposed method appears to perform better.

Weaknesses:
1. The manipulation tasks were trained with DDPG, while newer methods like SAC or TD3 has shown notably better learning performance. It’s not clear how much impact the algorithm would still make given a stronger algorithm.
2. The comparison to using the raw action space is not entirely fair. For example in the half-cheetah case an optimal policy is already trained to collect data for learning the action representation. I think a more proper way of comparison is to separate the tasks for training the action space representation and evaluating the representation performance. For example for the two manipulation tasks one can learn the action space with one task and test on the other. For the half-cheetah task one may learn the action space with a running backward policy and test with a running forward policy.

clarity,_quality,_novelty_and_reproducibility: good
summary_of_the_review: As discussed in the weaknesses, my main concerns for the work is the comparison fairness to the baseline as well as using a stronger learning algorithm for the manipulation tasks. I think the algorithm itself is novel and interesting in my knowledge and if the concerns were addressed I believe this would be a good one for the venue.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new latent action model based on Neural Householder Transform, where context dependent linear actuation subspaces are the outputs of an encoder-like neural network. The encoder outputs can be transformed via the exponential map on the sphere to form Household reflectors, that can be used to compute the context-dependent subspace. These local subspaces when interfaced with standard RL approaches can lead to more efficient and robust (w.r.t hyperparameters) learning in some environments.
strength_and_weaknesses: The paper is well written and the introduced approach is quite nice in my opinion.

Strengths:

* The Neural Householder Transform and the way it is computed, can yield a flexible way to compute locally well approximating linear subspaces, as indeed shown by the paper. 

* By bounding the network with a Lipschitz bound, the authors can also show that the computed transform is smooth. This result can perhaps be used computationally as well, however the authors have not done so (e.g. during training).

Weaknesses:

* The claims of the paper about the contributions are not rigorously demonstrated, the authors themselves use "tend to", to describe the effect of the framework (when compared with competing approaches). This can be remedied by either more detailed experiments, and/or by a more stringent analysis of the failures/benefits of the approach. 

* It is not clear if some of the NHT is the author's contribution, e.g. the particular output representation utilizing the exponential map.
clarity,_quality,_novelty_and_reproducibility: The paper is unclear at times about the NHT and the author's contribution, please see above. Otherwise the paper is quite well written and the method is well introduced. 

Some minor comments:
* In the introduction: is the last paragraph meant specifically for NHT?
* Section 2.3: in some works -> please cite
* How do you fix k? Can it be estimated/learned? Any works that do so? 
* Is section 3.1.1. part of the contribution?
* In Figure 3, what happens after 100 epochs? Does the alternative approach catch up?
* How do you decide what is the right observation/context for the environments? (e.g. WAM wipe and grasp)
* Why would/should agents trained with NHT be more robust w.r.t. different hyperparameter configurations?
summary_of_the_review: I would recommend the acceptance of the paper, see above.

=== POST-REBUTTAL EDIT ===

Thanks to the authors for providing a careful rebuttal. As mentioned before, the method proposed in the paper is quite nice and novel in my opinion. However upon deeper reading the paper, the comments and the rebuttal, I am not convinced about the results of the paper. In particular relating to the hyperparameter-robustness, I am not convinced of the link between the NHT transform and increased robustness to hyperparameters. It's not clear to me why the context dependent linear subspaces emitted by the NHT transform would lead to such robustness, as opposed to the nonlinear dimensionality reduction achieved by LASER. I would rather expect improved learning curves, and better generalization (as I noted in my reply to the authors). 

As hyperparameter-robustness seems to be the main claim of the paper (as an improvement), I'm lowering my score to 5, but I'm happy to reconsider if the reviewers would like to give some feedback on my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of identifying an action representation appropriate for robot manipulation tasks. The paper notes that existing methods that learn a lower-dimensional latent representation of the actions typically involve a large number of parameters. Further, the learned action representations tend to sacrifice interpretability. In an effort to overcome these limitations, the paper proposes to learn local linear subspaces for actions that are then transformed using the exponential map on the sphere to give rise to Household reflectors. The framework is supported by the underlying theoretical justification and empirically demonstrated on different locomotion and manipulation tasks for which it outperforms contemporary baselines.


While there were some questions of relevance raised in the initial reviews, the problem of learning abstract action representations is relevant and of interest to the learning community. All three reviewers appreciate the proposed algorithmic approach to identifying local linear subspaces for actions. They also appreciate the fact that the proposed method does not require fundamental compromises in order to be effective, and note the empirical evidence in support of the advantages of NHT. However, the reviewers identify a few key weaknesses with the current approach that need to be addressed in order to clarify the significance of the contributions. In particular, the reviewers question the chosen baselines as appropriate algorithms to compare against. Similarly, one reviewer points out that, despite discussions with the authors as part of the reviewing phase, they remain concerned about the conclusions regarding the hyperparameter robustness claims. Unfortunately, these concerns were not resolved as part of the discussion between the authors and two of the reviewers. The reviewers and AC agree that the paper has the potential to provide a nice contribution to the 
problem of action representation learning, but the aforementioned concerns need to be addressed.
justification_for_why_not_higher_score: As the e reviewers note, the claims made in the paper are not supported by adequate experimental evaluation.
Decision: Reject