Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an approach to discover the feasible actions for a given state in an environment. The algorithm proposes actions, which end in a single step in either success or failure, as determined here by a hand-designed heuristic. The generated actions are multimodal in nature and show limited generalization to out-of-distribution tasks. The method is tested on a robotic gripping task with 3-4 shapes. The results demonstrate that, in this domain, the proposed method can discover more modes of the feasible action distribution than the baselines.
strength_and_weaknesses: The strengths of the paper are:
1. It is theoretically well motivated and sound. The authors explore various f-divergences to find the best one for this setting.
2. The analysis of the various modes and grasping success for different methods is thorough for the chosen domain.
3. As far as I can tell, their approach is novel.

The two biggest areas for improvement are:
1. Lack of comparison with any relevant literature - grasping is well studied problem and comparison with some method in literature besides author-crafted baselines is required. The authors explore a few variations of their own method as well as compare to some sanity-check baselines but  the grasping success % can be compared to other methods in literature. I also found the related works section lacking in references to grasping literature and instead referencing peripheral works such as LLMs.
2. Unclear writing - It is unclear which problem the method is solving, that of generating feasible actions or discovering skills or both. Skills are typically referring to a sequence of actions whereas the case considered here is closer to CB with a single state/action. It seems the terms "multimodal action distributions" and "skills" are being conflated and further clarification is needed. Moreover, while in the abstract the claim is to learn "reusable and transferrable skills", there is no discussion later in the paper about the reusability or transferability of the action modes discovered.

Smaller points of feedback:

3. It is unclear what happens in environments where the hand-designed heuristic for judging success/failure is not present. The method is only tested on a single environment of 2D grasping and could benefit from being tested on at least one other problem to get an idea of how well it works as a general method for feasible action generation.
4. The role of the "critic" network is not clearly explained. Why is it that the learned critic can generalize over environment samples better, and hence provide better gradients, than directly training the KDE on them?
5. What is the role of the imitation learning dataset used to bootstrap the critic? An ablation on how much data is needed there would be interesting to see.
6. A study of the sample complexity in terms of environment interactions would be a relevant result.
7. Results for the figure 8 shape are missing from figure 3.


clarity,_quality,_novelty_and_reproducibility: Quality and novelty are high as stated above.
The paper as it is is lacking in clarity of motivation and terminology as well as comparisons. 
The method is clearly explained with enough details to allow for reproducibility.
summary_of_the_review: This is a novel and sound method for multimodal action discovery and with proper comparison, clearer writing, and demonstrating the method's generality on at least one other domain it can be a strong contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to train a generative neural network to generate all feasible actions of a sub task by interacting with an environment. The trained generator provides a distribution that matches a uniform distribution over all feasible actions. The authors derive the general optimization target for arbitrary f-divergences using a combination of kernel density estimates, resampling, and importance sampling. An application of a 2D robotic grasping problem is presented in this experiment to show the proposed algorithm outperforms other related methods. 
strength_and_weaknesses: The problem considered in this paper is to generate all feasible actions for a certain task which is hard to obtain for complex tasks. The author propose a novel learning method for generative neural network in order to generate feasible actions. In order to solve the problem, the authors further propose a gradient estimator for the gradient of f-divergence. The proofs are rigorous and correct. 

Weakness: I found this paper a bit hard to follow since some theories are not well explained in the paper. It would be great if the authors can provide more comprehensive explanation for the result you used in the paper. 
clarity,_quality,_novelty_and_reproducibility: The proposed method is novel. The theoretical proofs look correct to me. 
summary_of_the_review: This paper considers a very important problem in real application. The proposed method is novel and well supported by theoretical and experimental results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an RL method designed to generate all possible actions. I assume this is done to help exploration of the space in order to learn a better policy. The approach is evaluated on a grasping task.
strength_and_weaknesses: The paper works on an interesting and important problem of finding/exploring all actions. The paper is well written and motivates the problem nicely.

The biggest weakness is the lack of experiments. There is only one experiment, and its setup, environment, action space, etc. are not defined. Further, there is no comparison to other methods, other than a few proposed in the paper. This makes it difficult to judge the work and its contribution.

It is also unclear how much the proposed approach helps vs other things used (imitation learning, etc.)
clarity,_quality,_novelty_and_reproducibility: I'm unsure of the novelty of the approach, lots of works have studied exploration ideas in RL, but this is not an area I am too familiar with.

The writing is well done and explains the idea and method well. I'm not sure if it will be reproducible from the paper, as it is heavy on equations but has no implementation details or experimental setup explanation. The paper states a github link will be given, but at the moment, only the paper is available.
summary_of_the_review: The paper proposes a method to generate all possible actions and is evaluated on a grasping setup. The paper is well written, but the experiments are weak/minimal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors address the problem of large solution spaces by decomposing problems into smaller subproblems and training agents to consider all feasible actions for the subproblems, hopefully enhancing transferability of solutions to new problems.  They develop a generative model to produce all feasible actions and evaluate it on a 2D grasping problem.

strength_and_weaknesses: Strengths:
 - The idea of generating all feasible actions is an interesting one, and has intuitive appeal to facilitate skill transfer.
 - The authors present novel approaches to estimate gradients in various sampling contexts.
 - The authors present novel approaches to reduce bias in training the critic.
 - Empirical results are encouraging. 

Weaknesses:
 - Typo on page 8: "RKL, has" -> "RKL has"
 - The legend of Figure 3(b) covers some numbers

clarity,_quality,_novelty_and_reproducibility: The work seems original and interesting, and after more development, could be broadly applicable. 

Questions:

(1) In Section 4.2, do you choose your high-uncertainty actions probabilistically, or take the one with the smallest margin?  On a related note, this approach sounds a lot like an active learning problem; could you substitute a different AL algorithm here? 

(2) When you contrast JS/FKL to ME, you note that they have similar failure rates but more diverse outputs from JS/FKL.  What kind of tradeoff do you think is acceptable between failure rate and diversity?  I.e., how much higher of a failure rate (over ME) would you accept to claim that the higher diversity of outputs worthwhile? 

(3) The idea of generating all actions sounds a little like the idea of "diverse density" from multiple-instance learning.  Are there any results from that family of algorithms that might be useful here? 

summary_of_the_review: I think that this is a paper worthy of publication in ICLR, though my confidence is not high.  It clearly defines a problem and concisely presents their approaches.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the problem of learning all feasible actions (i.e., covering all modes) in interactive settings with complex action spaces. The authors propose a generative approach trained using a general optimization target for f-divergences derived by the authors. The approach is evaluated on a 2D grasping task.

Reviewers found the problem tackled here well motivated and important. A novel, systematic approach is proposed, based on rigorous and correct proofs, and shows encouraging results in a grasping problem domain. Thorough analysis of the modes and success in this target domain complete the picture.

Initial reviews pointed out some weaknesses, some of which have been addressed by the authors during the rebuttal period. Reviewers noted a lack of discussion of related works specific to grasping, some lack of clarity about the problem setting and concepts such as "skills", and some aspects of the provided theory. Reviewers are satisfied with how these points were addressed.

The single biggest shortcoming noted by reviewers is the focus on a single domain for empirical validation. While compelling and important, the grasping problem studied here is just a single example, and does not go far enough to support the claim that this proposed method is generally applicable. A particular concern are hand-crafted components such as the grasping success heuristic - how easily could the overall approach be translated to other settings?

The final recommendation for this paper is taking into account discussion between reviewers. While all reviewers agree that this is high-quality work with novel contributions and potential for impact, they are uncomfortable recommending acceptance given the remaining concern about the evaluation setup and generality of the proposed approach. Therefore, the recommendation is not to accept the paper at the present stage.
justification_for_why_not_higher_score: The paper is strong overall, with clear novelty on the proposed approach. However, the remaining concern is an important one - the authors claim that their method is generally applicable, but only evaluate on a single domain. What is more, the evaluation setup includes a hand-crafted success heuristic, meaning that the authors would need to somehow show how easily similar results can be achieved in other domains. I could see a version of this paper that makes less general claims and instead focuses purely on grasping, in which all claims are supported. However, this version might be more suitable for a domain specific (e.g., robotics) conference.
Decision: Reject