Learning to Generate All Feasible Actions
Authors: Mirco Theile
Daniele Bernardini
Raphael Trumpp
Cristina Piazza
Marco Caccamo
Alberto Sangiovanni-Vincentelli
Venue: Submitted to ICLR 2023 (2023)
Keywords: Generative neural networks
feasible actions
f-divergence

Abstract: Several machine learning (ML) applications are characterized by searching for an optimal solution to a complex task. The search space for this optimal solution is often very large, so large in fact that this optimal solution is often not computable. Part of the problem is that many candidate solutions found via ML are actually infeasible and have to be discarded. Restricting the search space to only the feasible solution candidates simplifies finding an optimal solution for the tasks. Further, the set of feasible solutions could be re-used in multiple problems characterized by different tasks. In particular, we observe that complex tasks can be decomposed into subtasks and corresponding skills. We propose to learn a reusable and transferable skill by training an actor to generate all feasible actions. The trained actor can then propose feasible actions, among which an optimal one can be chosen according to a specific task. The actor is trained by interpreting the feasibility of each action as a target distribution. The training procedure minimizes a divergence of the actor's output distribution to this target. We derive the general optimization target for arbitrary f-divergences using a combination of kernel density estimates, resampling, and importance sampling. We further utilize an auxiliary critic to reduce the interactions with the environment. A preliminary comparison to related strategies shows that our approach learns to visit all the modes in the feasible action space, demonstrating the framework's potential for generating multimodal action distributions.

LEARNING TO GENERATE ALL FEASIBLE ACTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Several machine learning (ML) applications characterized by searching for an optimal solution to a complex task. The search space for this optimal solution is often very large, so large in fact that this optimal solution is often not computable. Part of the problem is that many candidate solutions found via ML are actually infeasible and have to discarded. Restricting the search space to only the feasible solution candidates simplifies finding an optimal solution for the tasks. Further, the set of feasible solutions could be re-used in multiple problems characterized by different tasks. In particular, we observe that complex tasks can be decomposed into subtasks and corresponding skills. We propose to learn reusable and transferable skill by training actor to generate all feasible actions. The trained actor then propose feasible actions, among which an optimal one can be chosen according to specific task. The actor is trained by interpreting the feasibility of each action a target distribution. The training procedure minimizes a divergence of the actor's s output distribution to this target. We derive the general optimization target for arbitrary I-divergences using a combination of kernel density estimates, resampling, and importance sampling. We further utilize auxiliary critic reduce the interactions with the environment. A preliminary comparison to related strategies shows that our approach learns visit all the modes in the feasible action space, demonstrating the framework's ss potential for learning skills that can be used in various downstream tasks.
INTRODUCTION
Complex tasks can often be decomposed into multiple subtasks, with corresponding skills that solve these subtasks. Learning reusable and transferable skills an active area of research (Kalashnikov et al. (202T): Chebotar et al. (202T) : Deisenroth et al. (2014)) However, given subtask, learning or even defining the corresponding skill is not straightforward. Consider a robotic scenario where a robot is tasked to grasp object and handle it in downstream tasks. Different downstream tasks can have different optimal grasps if the object has multiple feasible grasping poses. Therefore, a grasping skill cannot be trained based optimality definitions of individual tasks. However, grasping algorithm that learned all feasible grasps could support all possible downstream tasks even without explicit knowledge thereof during training. The downstream tasks then select their respective optimal grasp among the proposed feasible options. Therefore, we consider a skill to be defined by the set of all feasible actions of a subtask.
We propose a novel method to train a generative neural network to generate all feasible actions of subtask by interacting with an environment. The interaction loop adopted from Contextual Bandit (CB) (Langford et al. 2008)) and Reinforcement Learning (RL) (Sutton & Barto (001)): ): the environment presents a state for which the actor selects an action, which is tested in the environment, yielding either success or failure outcome. As in CB, we limit ourselves to one-step interactions as opposed to sequential multi-step interactions common in RL. However, we do not minimize regret, typically done in CB. Instead, optimize the final policy as in RL. Unlike CB and RL, the approach does not aim to find one optimal solution for a given problem but aims to learn all feasible ones.
By interpreting the feasibility of each action given state as a posterior probability distribution over the actions, a target probability density function (pdf) is defined. The actor is trained to minimize a divergence of its output distribution to this target pdf. The training algorithm in the method proposed can be used with any given f-divergence, including Reverse Kullback-Leible (RKL), Forward Kullback- Leibler (FKL., and Jensen-Shannon (JS). The possibility to use FKL and JS is instrumental

---

in visiting all the modes of the posterior distribution, RKL is known to collapse into single mode (Jerfel et al. (2021)). The training algorithm presented in this paper uses Kernel Density Estimation (KDE) estimate the pdf of the actor and Monte Carlo integration with importance sampling to estimate the normalization of the target. The divergences are estimated using samples from a proposal distribution which is separate KDE based the original samples of the actor. This resampling step is necessary for convergence, which is discussed in Section [.3] As interactions with the environment are typically costly, an auxiliary critic network imitating the environment is trained simultaneously. The critic's feasibility estimate of an action is then used to form the target distribution.
The learning algorithm has been tested on planar robotic grasping problem. We test FKL, RKL, and IS divergences and compare them to implementations of maximum entropy (ME) RL and Generative Adversarial Networks (GANs). Besides accuracy, we measure how many grasping modes, i.e., disconnected regions in the action space, are visited by each approach. Generating actions in all tasks.
grasping modes can ensure that the learned skill reusable and transferable for various downstream
The contributions of this paper are the following:
Design of a new learning method for generative neural network models to explicitly learn to generate all feasible actions.
Introduction of a novel gradient estimator for f-divergences that takes advantage of KDEs,
resampling, and importance sampling.
Application of the proposed learning algorithm to 2D robotic grasping problem, comparing the proposed gradient estimators for f-divergences with related methods.
The rest of this work is structured as follows. Section 2 discusses the related work. Section describes the optimization problem followed by the methodology in Section 4 The evaluation setup described in Section 5 and the results presented in Section [)] Section concludes and gives an outlook on future work.
2
RELATED WORK
CBs have been successfully applied to several interactive learning problems with discrete action spaces (Langford Zhang
et al. (2014); Foster & Rakhlin (2020);[Simch))) (Simchi-Levi & Xu (22TT)) In several cases, the context and action spaces have been embedded in a linear multidimensional action space. The embedding keeps the interaction linear while the action and context embeddings can be non-linear (Chernozhukov al. (2019); Foster et al. (2020); Zhu et al. (2022) ). Recently, there has been an increased interest in extending the approach to continuous action
spaces. However, most works are limited to ID actions (Chemozhukov al. (2019); Majzoubi et al.)
(020)
Zhu & Mineiro 2022)).
Learning from interactive environment is also the focus of RL (Sutton & Barto 2018 ). Many RL. approaches use Actor-Critke (AC) architectures, among which the Soft Actor-Critic (SAC) algorithm (Haarnoja al. (2018) is most related our work. In SAC, the state-action value function of the critic is transformed into an maaed distribution (Haarnoja et al. (2017)), yielding the target of the actor's distribution. SAC uses RKL the loss function for the actor, which yields maximum entropy RL. Through reparameterization trick, which usually uses the family of Gaussians, the RKL minimized through direct gradient from the critic.
GANs propose a similar architecture to AC, training a generator and discriminator adversarially. This adversarial training is equivalent to minimizing the JS divergence (Goodfellow al. (2014)) and has been extended to arbitrary f-divergences (Nowozin et al. (2016)). Conditional GANs (Mirza & Osindero (2014) offer an alternative solution to the posterior sampling problem, as a generator conditioned on given state can be trained to provide successful actions adversarially. However, the problem analyzed in our paper is not naturally adversarial, as actions that have not yet been tested in the interactive environment should not be implicitly rejected. The discriminator learns to discriminate between tested successful actions from untested ones, providing the generator with inconsistent gradients.
2

---

Expert knowledge is used in Imitation Learning (IL) to derive policy from demonstration data. The policy may be learned in a supervised manner in behavior cloning (Pomerleau ( (1988)) or as a combination of Inverse Reinforcement Learning (IRL) (Ng et al. (2000) to learn reward function and a subsequent RL procedure. HHo & Ermon 2016) introduced Generative Adversarial Imitation Learning (GAIL), mitigating the intermediate IRL. step by using a generative adversary. As discussed in Li et al. (2017). the policy learned by GAIL tends to interpolate between modes leading to erroneous behavior in multimodal settings. Using f-divergence minimization, the authors in Ke et all (2020); Ghasemipour et al. (2020) intentionally collapse modes to avoid interpolation. IL. and adversarial approaches require large amounts of expert data. However, expert data is limited in an interactive environment. Additionally, given that we aim to find all feasible actions, even more expert data representative of all feasible actions would be required.
Posterior sampling has been long-standing problem in statisties. State-of-the methods in Bayesian statistics rely on Markov Chain Monte Carlo (MCMC) algorithms (Hastings ( ( & Smith (1990)), eliminating the need normalize the distribution which is often an intractable problem k (2015)). Variational Inference (VI) relies instead on fitting the posterior with a
(1970); /Gelfand
family of parametric probability distributions that can then be sampled from (Jordan et al. (199));
Wainwright & Jordan (2008). Neural samplers offer another alternative by approximating the posterior with generative neural network (Nowozin al. (2016); [uu et al. (2018)). Normalizing flows also infer the pdf for each sample using invertible mappings (Rezende & Mohamed (2015);
Tabak & Turner (2013): Tabak & Vanden-Eijnden (2010). While this approach does not require density estimates, it limits its applicability to specific neural network designs.
For robotic grasping, Kalashnikov (aaaasnnk et al./ (2018) propose using Deep Reinforcement Learning (DRL) to find optimal grasps through interaction with multiple real- world robots. If the goal is to find grasping poses explicitly to be used as the target of a classical controller, supervised learning techniques are often utilized (Kleeberger et al. (2020)) To support various downstream tasks, it would be necessary to find all feasible grasps. To this end, the action space is typically discretized and grasping success is estimated for each discrete action through maps- This can be learned supervised (Kumra ctt al.) (2020): Morrison et al.] (2020) ) or self-supervised (Zeng et al. (2000). Zeng et al. (2020) explicitly utilize structure given by spatial equivariances. We aim to find solution that needs neither
discretization nor to make use of the structure as these requirements restrict applicability to planar
picking in carefully crafted environments.
3
OPTIMIZATION PROBLEM
3.1 PROBLEM FORMULATION
An interactive environment, simulated or physical, is defined as a function g : S x S 1], where S is the state space of the problem, and A is the corresponding action space. For all E S we associate a feasible action space A, such that g(s, a) = 1, Va E A and an infeasible action A. The goal of this work is to find
space A, such that g(s,a) a) 0, Va A, with AT U,A a state-dependent surjective map n. 2 - A5. referred to as policy, where 2 is a latent space of appropriate dimensionality. For given state and uniform sampling from the latent space 2, the pdf of n, is a function g. A + IR, which is the posterior distribution (a(a) q(als). For the same state, the distribution of the feasible actions according to the environment can be defined as
g(s, a)
(a) [Ag(s,a) a) da
(1)
which is the true posterior P=(a) p(als). The optimal solution satisfies D, (ps (1 0, where Dy is an f-divergence, for example from Ta This implies p. g.. therefore the support of q, is equal to the support of ps. which A by definition. Thus, the optimal policy is the solution to the
optimization problem:
न. = argming,~n1 Dy (ps ll
(2)
with II being an arbitrary family of distributions. To generalize over all states S, the policy can be modeled as a neural sampler Te : S x - A, with a corresponding pdf go (a]s), where indicates the parameters of the neural network. Assuming that the environment can be used efficiently for repeated controlled experiments, i.c., testing several actions for the same state, the above optimization problem can be solved directly the environment. If this is not possible, critic network can be
3

---

f(t)
f(())
Jensen-Shannon (JS)
(t log 1+1
+ log(t)
2o log
2t
i+1
Forward Kullback-Leibler (FKL)
log(t)
Reverse Kullback-Leibler (RKL)
log(t)
log(t) +1
Table 1: Non-exhaustive list of f-divergences and the corresponding first derivative for gradient estimators. The I-divergences are obtained by substituting the f functions above in equation 3 and setting = qo go/p. The conventions for p, q, FKL and RKL assume that p is the target distribution, is the model, and the FKL divergence is log(p/g).
used to imitate the environment, which further discussed in Section 4 Note that the system state is often only partially observable, and the action must be inferred from observation. For simplicity of notation in the following derivation of the gradients, we assume that the state directly observable, and omit the state and action dependence of q and p.
3.2 F- D-DIVERGENCES
The f-divergence between two pdfs and g is generalization of the Kullback-Leibler (KL) divergence and has the form (Liese & Vajda (2006)
Dy(0 go) / ps da,
where f oo) + convex function. Different choices of f lead to well known divergences as summarized in Table The gradients of the f-divergence w.r.t. 0 can be estimated commuting the derivative with the integral (L [L'Ecuyer (1995)) and using the score function gradient estimator (Kleijnen & Rubinstein
(1996) as
0 00 D 30 J
9o da SAP p (
④ p 1 4o da =
-A,0 go S'
p
log qo da, (4)
p
30
using the fact that does not depend on O. Since qo is normalized to 1 and thus Oo SA da SA Og log q da Lagrangian term À can be added the gradient estimator:
S go (5' (r(1) 98
00 D D
+
log go da.
(5)
00
the support of ge includes all of A the above formula can be rewritten as the expectation on go as
00 D = Ego (
g0
log qe
(6)
00
Sampling from a proposal distribution q', the expectation can be computed with importance sampling (Robert & Casella (2004);
Liu (2001)
90
θ
00 ~
+
A
log do
P
00
3.3 GRADIENT ESTIMATION
Given a sample a ~ qo, it not possible to directly evaluate qo (a) as it is not available in closed form. Therefore, qe needs to be estimated to compute the gradients of the f-divergence. Given N sampled actions a; ~ qo qe can be approximated with a KDE by
(a) (a)
Σ ka aa),
(8)
N
a,rigs
where he is a Gaussian kernel with a diagonal bandwidth matrix a.. The KDE makes the estimate of the expectation possible. Using equation [6] computing the expectation value the average over the samples yields
θ Dj 20
Σ
(s'
90.0 p
+
θ
log ao.0.
(9)
00
G, ruge
4

---

This gradient estimator turned out not to converge in our experiments. While systematic investigation of the convergence issue was not completed, we suspect two primary reasons for this. First, the support qo usually does not cover the whole action space A, which is necessary for the expectation formulation in equation 6 Second, evaluating (o(a;) based on a KDE, which uses aj as supports, has bias for j i.
Adding Gaussian noise to the samples gives full support in A and reduces the bias at the support points of the KDE, which lead convergence in the experiments. The new smoothed samples are given by aj = 04 + for mi S < m(i + 1) and e ~ (0, o'), where m indicates the number of smoothed samples drawn for each original sample. This is equivalent sampling from a KDE with a, as supports and o' as bandwidth. The gradient, using importance sampling in equation 7 can be
rewritten after resampling as follows
① 00
DI
1
Σ
q0.0
(5'
-0.00 p
+
log qo,0.
(10)
M 0;
90,0'
00
10 requires an estimate of p, which in turn requires an 00000
with M mN. Additionally, equation estimate of the volume in equation
J g(a) do
1 Σ g(a7) a0.0r (a))
(11)
M
This estimation is similar to self-normalized importance sampling (Murphy (2012) but uses the proposal distribution. The bandwidth o' of the proposal distribution a hyper-parameter. Setting o' co, experiments show that e helps convergence. Intuitively, larger bandwidth enables the exploration of nearby modes in the action space. Specific estimators for the different f-divergences can be obtained substituting f' from Table into equation 10, A summary of the gradient estimators used in this work is given in Table 2
4 METHODOLOGY
The derivation in Section 3.3 assumes that the training could be performed directly on the interactive environment. To train the actor, multiple actions have be evaluated for the same state. Typically, this is not possible, either because of reproducibility in real experiments or computational cost for simulations. An auxiliary neural network Es- S x A IR with parameters Φ, can be trained to imitate the environment g. The policy can then be trained to match the distribution of the feasible actions according to this auxiliary neural network. We refer to Ta and & as generative actor and critic, respectively. The neural network architectures are presented in Appendix B
The learning algorithm presented in this paper is inspired by RL and CB. At every training step, the environment generates a state for which the actor proposes an action. The action is evaluated in the environment yielding success or failure. The state, action, and feasibility form an experience stored in replay memory. The critic is trained on random samples of experiences from the replay memory with a cross- eross-yntropy loss on the outcome. The actor trains on batch of states from the memory. For each state, multiple actions are sampled from the actor, used as support for qe a and a New samples are drawn from the proposal distribution ao.--- These samples are evaluated by the critic Es, and the gradients interaction loop can be found in Appendix While the general interaction loop is standard in RL, two changes have proven beneficial to the convergence: balanced replay memory and maximum uncertainty collection. Additionally, an action optimization can take advantage of the density estimate improve performance after training.
computed according equation TO The algorithm of the
4.1 BALANCED REPLAY MEMORY
Since the environment yields either success or failure, the critic is a binary classifier that suffers from unbalanced data when being trained. Its memory dataset continuously grows through the interactions between the actor and the environment. In the beginning. the actor performs poorly, yielding primarily experiences with failure labels, with the opposite at the end of the training. This labeling bias prevented the critic from distinguishing between success and failure outcomes, making
5

---

Loss
Actor Gradient Estimator
A
JS
2M Laz
dd.r da.
log 200.-
log 90,0
0
p-46,-
FKI
M Enz No.
log 90.0
80
RKL
M E qa.a
log
log qu,0
-1
80
GAN
N
Σ da
log(1
ai
ME
N La;
8 86
log Q0.
Ba
e log Ee
8 36
44
Table 2: Gradient estimators of different losses and choice of Lagrangian multiplier X.
convergence impossible. To avoid the critic from biasing towards failure or success labels, we use two replay memories, for failures and one for successes. When training the critic, half of the experiences are sampled from the positive replay memory and the other half from the negative replay memory. With this strategy, the labeling bias can be mitigated. The potentially amplified classification bias (e.g., complicated shapes have more failure labels) did not appear to hinder convergence. This memory can be prefilled with imitation examples to bootstrap the critic learning. While is possible to minimize the use of expert knowledge, this paper focuses on the main learning method, while the impact of imitation learning will be analyzed future work.
4.2 MAXIMUM-UXCERTAIE COLLECTION
Given one state of the environment, the actor can generate several candidate actions. Depending on the training stage and the state, these proposed actions can have different degree of information for the critic. Selecting actions for which the critic predicts ss 0.5, i.c., it cannot decide between success and failure, provide higher information content. This strategy has proven to improve the convergence our tests.
4.3 ACTION OPTIMIZATION
Optimal performance an environment with multiple disconnected sets of feasible actions would require multimodal distribution with probability density of zero between the modes. Since the actor is a continuous neural network and the latent space is continuous, the actor cannot generate only positive actions. However, the actor aims to minimize the probability density for actions in the gaps between the modes. The probability density at each action can be estimated using the KDE, and the actions with the lowest corresponding density can rejected. The accuracy of actors with strong multimodal performance like FKL is significantly increased from action optimization as shown in Section
EXPERIMENTAL SETUP
5.1 ROBOTIC GRASPING SIMULATION
The proposed architecture was tested in simplified robotic grasping simulation. We assume vertical configuration of parallel gripper with three degrees of freedom x, and a and an object that is an extrusion of a shape. The simulator generates five different shapes with varying position, angle, color, and geometric parameters. The success of a grasp is determined by the relative position and alignment of the gripper to the outline of the object as seen from a camera positioned above the experiment. Details about the simulator
be found in Appendix A
The choice of this simulation, as opposed to existing robotic simulations, was motivated by efficiency while iterating through several combinations of models and parameters. The target distribution fidelity not primary concern this work. The focus is instead on the capability of the proposed method to learn all feasible actions.

---

(a) Problem Ground Truth
Critic (d) Actor
Figure I: Critic classification and actor distribution trained with JS compared with the ground truth. Five example grasps are shown in the problem and their associated locations in the ground truth. The figures show projections onto the x-y plane (top row) and the x-a plane (bottom row).
5.2 COMPARISON
In the evaluation, comparing different f-divergences with each other and with two other approaches. The analyzed f-divergences are the FKL, RKL, and JS divergences. The two other approaches an ME RL algorithm similar to SAC in Haarnoja et al. (2018). which trains the actor to minimize
min E,-,M [log (T =))s) Ea (,,,))(,)))) =))]
(12)
with M being the replay memory. The critic is trained described in Section 4 Instead of using the reparameterization trick with known distribution estimate the entropy, we use the KDE. The other approach an implementation of conditional GAN (Mirza Osindero (2014)) with a growing dataset. The min-max optimization problem is given through
min max Ba. (Log(E.s (s, a)) - log(1 - (o(s,
(13)
8
with the positive replay memory My. which grows through successful interactions with the environ- ment. An asterisk is added (e.g., JS*) when using action optimization, rejecting 10% of the proposed actions with the lowest probability density. The actor gradient estimators for all approaches listed in Table
In the following section, we only compare with approaches that do explicitly utilize the structure of the problem, as the intention of the proposed approach is to be generally applicable in a continuous CB problem setting. comparison with widely used approach from the grasping literature is conducted in Appendix
6 RESULTS
For each configuration, agents were trained for I million interaction steps with the environment, taking approximately 48 hours on a single NVIDIA A100 GPU. At the start of the training, 80k examples, including positives and negatives, for randomly generated shapes were added to the balanced replay memory to bootstrap the critic and discriminator learning.
Figure T shows the problem, the ground truth feasible picking positions, the critic estimate, and a heat-map of the actor's proposed actions. All figures are projections taking the maximum over the dimension that is not shown. In the problem visualization in Figure Ia five feasible picks are shown in different colors, which correspond to the markers in Figure lb These markers highlight the complex multimodality of the problem. While it appears that, e.g., red and purple in the same mode in the x-y projection, is visible in the x-a projection that they are not directly connected. Figure Tc shows that the critic has approximate understanding of the feasible regions of the action
7

---

H
8
Box
(a) Problem
(b) Truth
(c) JS
(d) FKL
(e) RKL
GAN
(g) ME
Figure 2: Qualitative comparison of the implemented algorithms, showing action heat-maps on three different states, with the last state never been observed during training.
space, showing five modes clearly in the x-y projection. The actor distribution in Figure ld also shows all five modes, while the output is significantly sharper in the centers of the modes. This is due to the use of the KDEs and the choice of bandwidth a.
In the qualitative comparison in Figure the actor distributions of the different algorithms shown for three different shapes. While the H and 8 shapes were trained on, the Box shape has not been seen during training. The different subfigures show the action heat maps of all implemented algorithms, showing only the x-y projections. The H-row shows that JS and FKL learned all five modes, with IS having the fewest samples in the connecting area. RKL and the GAN show two modes. The ME implementation collapses in single mode. The 8-row and the Box-row show similar pattern with the most pronounced spread of the action distributions JS and FKL and mostly collapsed action regions in the other approaches.
To quantify the multimodal capabilities, and thus the transferability of the learned skill, each algo- rithm's ss accuracy and shares of modes on all shapes were evaluated. For each shape, 1024 random states were generated that differ in pose, color, and geometry (example variations can be seen Appendix A). For each state, 1024 actions were sampled from the different actors. The actions were then evaluated, and the mode of each action was recorded. The modes were then ranked and averaged over all the states of that shape by frequency. By averaging the ranks instead of the modes, the last rank shows the average ratio of the least frequent mode for each state.
Figure 3 shows the shares of each rank for the H and Box shapes for all the algorithms, with the asterisk indicating that action optimization was applied. This figure presents the multimodal capabilities of the JS and FKL algorithms, which are the only ones with the last ranked mode present for the H/ and with significantly more pronounced modes than the others for the Box. Therefore, only IS and FKL are capable learning transferable skill according to our definition. The generalization capability of the GAN implementation is significantly lower than the others, as seen on the Box shape.
To quantify the performance, Table the accuracy (feasible actions generated over total actions generated) for each shape and the last ranked mode for the H, T. and Box shapes. The table shows that ME has solid performance performs well with some modes present, but overall it is weaker than the others. RKL has high scores but mostly fails at finding all the modes. FKL shows good performance in mode finding, with overall accuracy similar to RKL. JS is on par with the ME accuracy with the addition that repeatably finds all the modes. Generally, action optimization improves accuracy but does not help mode finding, slightly decreasing the least ranked mode for most approaches. The maximum deviations in the superscript show that all approaches learn reliably with the GAN having the highest performance deviations among runs.
shapes but fails to find the different modes. The GAN algorithm
8

---

100
100
16.4
16.5
80
161
13.5
13.2
94
322
17.5
13.1
6.9
13.1
25.3
41-2
472
14.3
Mode frequercy [%]
25.0
26.3
80
355
6.0
40
2L3
223
69.6
6L3
67.0
Bank
1ank
15.2
24.0
Rank
90.9
Rank
26.3
Rank
8
Ranks
10.3
15*
15
FXL
FKL
RKL
RKL
GAN
GAN
ME*
ME
J5*
J5
FKLL
FXL
RKL
AKL
GAN
GAN
ME*
ME
(a) H Shape
(b) Box Shape
Figure 3: Gripping rank comparison, with the ratio of picks for each ranked mode or failure in %.
IS*
JS
FKL*
FKL
RKL"
RKL
GAN* GAN
ME*
ME
H
96.01-0
91.2
92.5
85.4
86.9
83.63.5
85.82 83.5 96.6
95.91.2
T
96.5".b 93.31.0
92.5
88.0"9
97.6"9
95.71/2
81.62 79.71.7 93.9
93.6
Score
8
86.92.3
82.g17
83.4
77.70.7
82.83.
79.22.3
70.34.5
66.95.7 85.50.5
84.21.9
Spoon
9..40.6
96.90.
93.6
90.62.3
97.4
97.6"
91.72.1 95.12.3 92.1
02.2
Box
62.2
61.72.7
53.87.4
52.17.1
53.5
53.0
34.28
33.172
65.76.9
65.9%.8
Avg
87.8
85.2
83.20.7
78.80
83.62.1
81.82.1
73.3
71.72.7
86.82.1
86.4
H
9.70.7
10.70.3
9.89.
9.40.
0.0%.
0.0%.0
0,20.4
0,20.3
0,0
0.0%.
Mode
T
11.0
12.30.6
15.8
15.3a.
0.40.3
0.70.4
1.12.3
1.42.6
0.0%.0
0.0%.
Box
5.50.8
6.91.
6.8
7.2
0.70.8
1.31.3
0.0%.
0.0%.0
0.00.0
0.0%.
Table 3: Comparison on all shapes with the mean of the grasping success ratio in % on top and the least ranked mode in on the bottom, with the maximum deviations over the 3 runs in superscript.
7 CONCLUSION AND FUTURE WORK
This work proposes to learn skill by training generator to generate all feasible actions for subtask. To this end, the output distribution of the generator is learned match a uniform distribution over the feasible action space. While learning within 2D grasping simulation, the method shows stable convergence for FKL,, RKL, and JS. As expected, FKL is more efficient visiting all the modes. IS has the highest accuracy while reliably visiting all the modes. The proposed learning strategy expands the current oftt-off-heeatt training within multimodal interactive environments by showing competitive accuracy while visiting all the modes. Since the proposed approach visit all the modes, it learns the skill grasping independently of a specific downstream task. In future work, we will investigate how downstream tasks can choose their optimal action among the proposed feasible options. As it is not dependent on the structure of the problem, we will further utilize it for 6D grasping problem as well as for other applications.
Some limitations have emerged during experimentation. Currently, many imitation examples are required to bootstrap the critic's learning. A possibility to mitigate this could the progressive tuning of the KDEs or learning their parameters during training. This approach could favor exploration initially and divergence estimation later in training. A complementary strategy could be using curriculum learning techniques that start with simple problems where solutions are less sparse in the action space. Furthermore, the proposed approach may not be very effective in high-dimensional problems as the sampling requirements for the estimator grow exponentially. The limit on the degrees of freedom will be explored in future work. mitigation of this issue can come from the rich literature on non-pan-ee -onammrmetr density estimation in higher-de dimee--dmmiiionall problems and its applicability to the proposed method. Another approach could be to split higher-dimensionn problems into multi-step low dimensional problems and to learn generate all feasible trajectories.

---

8 REPRODUCIBILITY STATEMENT
The final version will include a GitHub link to the code for learning and evaluation.
REFERENCES
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. International Conference on Machine Learning, pp. 1638-1646. PMLR, 2014.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07744 2021.
Victor Chernozhukov, Mert Demirer, Greg Lewis, and Vasilis Syrgkanis. Semi-paramette efficient policy learning with continuous actions. Advances in Neural Information Processing Systems, 32, 2019.
Marc Peter Deisenroth, Peter Englert, Jan Peters, and Dieter Fox. Multi-task policy search for robotics. In 2014 IEEE international conference on robotics and automation (ICRA), pp. 3876-3881. IEEE, 2014.
Dylan Foster and Alexander Rakhlin. Beyond ueb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pp. 3199-3210, PMLR, 2020.
Dylan Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances Neural Information Processing Systems, 33:1 1478 1 1489,, 2020.
Alan Gelfand and Adrian FM Smith. Sampling- approaches to calculating marginal densities. Journal of the American statistical association, 85(410):398-409, 1990.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pp. 1259-1277.
PMLR, 2020.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep ensrgy-based policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1352-1361. PMLR, 06-11 Aug 2017. URL https::/proceedi:gg mlr Sreas/v70/haarnce falaa html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
W. K. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97-109, 04 1970. ISSN 0006-3444. doi: 10.1193/biomee/57..199 URL
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai. Mao Ye, and Guang Cheng. Stein neural sampler. ArXiv, abs/1810.03545, 2018.
10

---

Ghassen Jerfel, Serena Wang, Clara Nong-Fannjiang Katherine A Heller, Yian Ma, and Michael I Jor- dan. Variational refinement for importance sampling using the forward kullback-leibler divergence. In Uncertainty in Artificial Intelligence, pp. 1819-1829. PMLR, 2021.
Michael Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence Saul. An introduction to variational methods for graphical models. Machine Learning, A:1007665907178.
01 1999. doi: 10.1023/
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651-673.
PMLR, 2018.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. arXiv preprint urXiv:2104.08211 2021.
Liyiming Ke, Sanjiban Choudhury, Matt Bares,, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning f-divergence minimization. In International Workshop the Algorithmic Foundations of Robotics, pp. 313-329. Springer, 2020.
Kilian Kleeberger, Richard Bormann, Werner Kraus, and Marco F Huber. A survey on learning-based robotic grasping. Current Robotics Reports, 1(4):239-249, 2020.
J.P.C. Kleijnen and R.Y. Rubinstein. puter Simulation Models by the Score Function Method. Other publications Tilburg University, School of Economics and https://ideas repec. ottpp/tittttttiss/ URL
Optimization and Sensitivity Analysis of Com-
TiSEM 188-9692-5441-4877-2ddd1
Management, 1996.
John K. Kruschke. Chapter 5 bayes' rule. In John K. Kruschke (ed.), Doing Bayesian Data Analysis (Second Edition). pp. 99-120. Academic Press, Boston, second edition edition, 2015. ISBN
0.00005-0000002222000000000 URL https 978-0-12-4058888 doi: htwp://wwwwaaa..........a.aaaaaa
www.
Sulabh Kumra, Shirin Joshi, and Ferat Sahin. Antipodal robotic grasping using generative residual convolutional neural network. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9626-9633. IEEE, 2020.
John Langford and Tong Zhang. The ppoch-greedy algorithm for multi-armed ban- dits with side information. (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Asso- ciates, Inc., 2007. URL htttpss://poooceeeiiiiiiio www.gis..euurrs..oommmmm httpurippcccpppprr///eep pttp///e/e/e//illeeee (DD0E888bbaad33c33fffAE12111
In J. Platt, D. Koller, Y. Singer, and S. Roweis
John Langford, Alexander Strehl, and Jennifer Wortman. Exploration scavenging. In Proceedings of the 25th international conference on Machine learning, pp. 528-535, 2008.
Pierre L L'Ecuyer. On the interchange of derivative and expectation for likelihood ratio derivative
estimators. Management Science, 41(4):738-748, 1995. ISSN 00251909, 15265501. URL
http: / /wwww jet.r.storroooo www.sor..rr//staaeeee2/2//
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30, 2017.
F. Liese and I. Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):434-4411, 2006. doi: 10. (09//IT.2000.88173.)
Jun S. Liu. Monte carlo strategies in scientific computing. In Springer Texts in Statistics, 2001.
Maryam Majzoubi, Chicheng Zhang, Rajan Chari, Akshay Krishnamurthy, John Langford, and Aleksandrs Slivkins. Efficient contextual bandits with continuous actions. Advances in Neural Information Processing Systems, 33:349-360, 2020.

---

Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. ArXiv, 2014.
Douglas Morrison, Peter Corke, and Jürgen Leitner. Learning robust, real-time, reactive robotic grasping. The International journal of robotics research, 3902-3):13-201, 2020.
Kevin Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012. ISBN 0262018020.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, 2000.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran As- sociates, Inc., 2016. URL https:ttpppptooooo cedebb 8 E5 0ww5533be88cco 71t
http:/ttts////ccciinngg// nwwrips cc//paper/211//Ei... Paper pdf
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in Neural Information Processing Systems, 1, 1988.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
Christian P. Robert and George Casella. Monte carlo statistical methods. In Springer Texts in Statistics, 2004.
David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: an introduction. MIT Press, second edition, 2018.
Esteban G. Tabak and Cristina Vilma Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66, 2013.
Esteban G. Tabak and Eric Vanden-Eijnden Density estimation by dual ascent of the lieliioel Communications Mathematical Sciences, 8:217-233, 2010.
Martin Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1:1-305, 01 2008. doi: 10.1561/ 2200000001.
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4): 1309-1319, 2020.
Yinglun Zhu and Paul Mineiro. Contextual bandits with smooth regret: Efficient learning in continu- ous action spaces. International Conference on Machine Learning, pp. 27574-27590, PMLR, 2022.
Yinglun Zhu, Dylan J Foster, John Langford, and Paul Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning, pp. 27428-27453. PMLR, 2022.
12

---

A GRASPING SIMULATION
The grasping simulator generates four different shapes (H, 8, Spoon, T) for training and Box shape for testing. The shape position, orientation, color, and geometry parameters are randomly sampled, producing various observations. The observation space is 128 x 128 pixel RGB image. We assume a vertical configuration of a parallel gripper with three degrees of freedom x, y, and that the object an extrusion of the 2D shape in the observation. The action space is constrained to the center 78 78 pixel region to avoid undefined behavior at the border the RGB image. The angle of the grasp is in [0, TT) as the gripper is symmetrical, and thus a full revolution is not necessary.
assume
The success grasp is only determined by the relative position and alignment of the gripper to the outline of the object, seen from camera positioned above the experiment. We developed an algorithm that, given the alignment of the gripper, i.e., I, V. and a and a simulated picture of the object from fixed camera, provides success/failure outcome in deterministic and reproducible manner. Given the maximum aperture of the parallel gripper and the width of the gripper claws w, the simulation analyzes the cropped image content of dimensions / x between the gripper claws before the claws close on the object. The simulation checks if the object is sufficiently present and equidistant from the claws and aligned within parameterized margins. Figure 4 shows successful grasping poses and the respective gripper content for the objects that are trained on.
oO
(a)
b
(c)
(d)
(c)
(n)
(g)
(h)
Figure 4: Feasible gripper positions (red) for different variations of the shapes (H-shape, 8-shape, Spoon, and T-shape) used in training, with a detailed view of the area between the gripper to the right of each figure.
B NEURAL NETWORK ARCHITECTURES
Latent Input (d)
FC
Shared MLP (121x(128+d))
(11x11x128)
(22x22x64)
(43x43x32)
(128x128x5)
Figure 5: Before processing, the image is augmented with positional encoding resulting in 5 total channels (r, g, b, x, y). The network's input layer (in gray) is 5x5 embedding layer with stride followed by 7 residual blocks (in yellow) with a bottleneck. The output is processed by 3 layers of "pixelwise" shared MLPs (in brown), with the features being concatenated with latent input (in purple) of length d. The latent input is random sample from Z for the actor and the action to be evaluated for the critic. Four (for the actor) or three (for the critic) fully connected layers (in blue) output the action and the score, respectively.
The neural network design was guided by simplicity and inspired by GANs. Features that rely on domain-specifie knowledge are avoided to evaluate better the learning method presented in the paper.
13

---

The structure of the actor and critic neural networks are illustrated in Figure The residual feature extraction network (He et al. (2016)) shared between the actor and critic.
As peculiarity of the network and the loss, the actor's inferred action has four components, [x, y, sin at, cos ai), with the ratio the third and fourth action components. As the scale factor r does not change the angle, the critic receives the normalized action [x,, y, sin a, cos ai] input. To avoid the actor from reaching the singularity at r = 0 and the distribution g being spread along the radius, g a) and E(s, a) are scaled with unnormalized Gaussian the radius, centered at 0.5 with the standard deviation of 0.4.
[0, v2). The angle can be extracted trivially with the arctan of
C ALGORITHM AND HYPERPARAMETERS
Algorithm 1: Jenson-Shanme Shannon training loop
Initialize M with imitaition and random examples and initialize 0, for 1 to Training Steps do // Training steps are 1,000, 000 in experiments for to Interaction Steps do // Interaction steps are in experiments
< Generate a new problem
2, Sample uniformly in Z,
[
A2 21), Vi E [1
F + ao(s,a,),
e [1.
j arg min
|0.5 Agl // Get action with highest uncertainty
g(),
10
r == then
Store (8,0j:7) in Mp
12
else
Store ()) in Mn
14
end
15
end
16
for to Critic Steps do
/1 Critic steps are 2 in experiments (8i, 0i, r Sample half from Mp and half from Mn
- as v Σte ri (s),))(s;)a)) + - (1-r;) log(1 (s,,
18
19
end
29
for to Actor Steps do
11 Actor steps are 1 in experiments
21
for l to do
S Sample from M
23
z + Sample uniformly in 2, WiE[L,N]
0; (s(K) vi E N]
25
~ o'), Vj [1,
a < stop gradient(((//mm)) + [ // Resample from KDE Vj M1] // Evaluate KDE on samples
J + EN laj - a;),
Φ + N Ei kor -a), Vj [1, // Evaluate proposal pdf
28
29
f + Co(on, a)),
Vj [1,
30
+ N7 ΣØ o
// MC integration with importance sampling
P3 V
&
Vj M
32
9k f- EM
أ=
/
4 log
24j 4p
Va (gg(
// gradient trace
33
end
0 0
0gg Σ K
end
36
end
14

---

Parameter
Value
Description
N
128
Minibatch size
M
256
Resampling size
2
Samples per KDE support point (M/N)
U
64
Maximum uncertainty proposals
K
16
Actor batch size
L.
32
Critic batch size
diag(0.025, 0.025, 0.4, 0.4)
KDE bandwidth
o'
diag(0.075, 0.075, 1.2, 1.2)
Sampling KDE bandwidth
MM
160,000
Positive replay memory size
Mn I
160,000
Negative replay memory size
320,000
Total replay memory size
Table 4: Hyperparameters
D OBSERVATION VARIATION EXPERIMENT
D.I SETUP
Retated
Figure 6: Different distortions are applied, showing colored chess board for illustration and an example shape under all distortions.
To highlight the difference between the proposed approach and related work of robotic grasping, we investigate how distortions of the observation affect the performance. The distortions investigated are rotation, projection, and rotation projection as shown in Figure 6 These distortions correspond to different camera perspectives. We train a new agent for 10 training steps for each distortion and approach in the following comparison.
We compare with a common approach in the literature Zeng ct al. (2020)) that make use of spatial equivariances. The approach utilizes fully convolutional networks to output probability of success for each action of a discretized action space. We implement two variants. In the first one, just as in Zeng et al. (2020), the observation is fed into the neural network multiple times with different rotations. The neural network then only needs to output a one-channel image containing the probability of success of each discretized v, y action for the given rotation of the image. This approach thus makes use of translation equivariance by using convolutional neural network (CNN) and rotation equivariance. the experiments, we denote as the map approach (H). The second variant estimates for each observation the success for different rotations by outputting multi-channel image indicating the success estimate of each discretized æ, V, a action explicitly. Thus only takes advantage of translation equivariance. It is called stacked heat-map (SH) in the following.
The approaches are implemented using fully convolutional networks with an hourglass structure, adopting the beginning of the Resnet in Figure 5lad adding the same structure in reverse order with nearest- neighbt upsampling. Both approaches predict grasping success for 78x78 pixels with 16 rotation angles. They are trained the balanced replay buffer. The replay buffer is also filled with imitation learning examples, and maximum uncertainty sampling is applied. For evaluation, the success estimate of each discretized
cross-entropy loss on the grasping outcome sampled from
15

---

action is used its probability be sampled. To increase accuracy, inverted temperature factor increases the difference between higher and lower score actions. Specifically, the actions are sampled according to
exp(3 ((s,a) g(ass) EVeEA, exp(3 log (()
(14)
with being the fully convolutional network with s as input and as output shape the discretized action space Ad- The inverted temperature was set to / 100.
D.2
RESULTS
100
9.8
4.0
6.0
Ronk
Rank
Rank
Rank
80
Rank
Rank
Mode frequency [k]
Pank
21.0
21.
203
20.5
10.2
76-4
80
2012
20.6
24.5
40
23.2
23.3
40
00.0
15.3
65.7
35.0
32-3
414
27.4 25
31.4
23.3
15t Normal
JS+
15" H Projected
15
SH
JS*
SM Projected
(a) H Shape
(b) Box Shape
Figure 7: Gripping rank comparison, with the ratio of picks for each ranked mode or failure in %.
Normal
Rotated
Projected
Rotated Projected
*
H
SH
JS*
H
SH
JS*
H
SH
JS
H
SH
H
96.7
92.6
97.7
95.1
92.0
96.6
96.2
24.0
0.9
97.9
21.6
0.8
T
97.1
93.5
98.2 2
95.5
94.3
98.3
96.2
31.6
0.8
98.0
27.0
0.7
Score
8
86.4
90.4
97.2
85.5
87.4
95.5
85.3
15.4
0.9
88.8
14.3
0.8
Spoon
97.7
93.8
98.8
96.5
94.5
98.6
97.1
33.8
0.7
97.8
33.2
0.6
Box
62.3
90.7
96.2
55.9
88.4
89.4
54.7
21.1
1.2
53.5
19.3
1.1
Avg
88.0
92.2
97.6
85.7
91.3
95.7
85.9
25.2
0.9
87.2
23.1
0.8
H
9.3
10.0
8.6
7.5
4.6
8.0
0.2
0.0
8.5
0.0
Mode
T
11.5
18.7
18.5
10.4
14.8
9.8
11.8
0.5
0.1
14.3
0.2
0.1
Box
5.5
16.7
16.8
4.3
9.5
1.4
5.1
0.3
0.1
5.3
0.1
0.1
Table 5: Comparison of the proposed Jensen-Shannon approach with approaches from the literature.
The results are shown in Figure 7 and Table which compare the performance of the proposed Jensen-Shannon (JS) approach with the heat-map (H) and stacked heat- map (HS) approaches on the four different distortions. As expected, the specifically crafted and SH approaches perform better the original problem than the generic approach proposed in the paper. In that scenario. scene understanding is required, and only local features need be considered to estimate grasping success. Therefore, the approaches Box-Shape, since the grasping success depends only on gripper alignment. They only need to learn to imitate the grasping success heuristic shown in Figure 4
expected generalize well to unseen shapes, seen for the
Interestingly, rotating the observation does not seem to impact their performance. However, under projection and projection rotation, both approaches fail to learn to grasp. The heat-map approach
16

---

still learns grasp the objects occasionally, while the stacked heat-map approach does not learn anything. Our proposed approach learns well for all distortions. The generalization on the Box-Shape decreases bit but the general performance remains similar. Surprisingly, the results for rotation + projection on the training shapes are above the variance in the main results in Table An explanation for the performance increase could be that the region in the input observation that the object can be in is smaller than in the normal case, as can be seen by the area between the colored tiles in Figure The reduced region could slightly ease scene understanding, leading improved results.
In general, the performance of our proposed approach does not depend on the distortion as it does not explicitly use the spatial structure. Its design does not depend on the specifies of the experiment at all. It can, therefore, learn independently of the distortion applied as long as the object is still fully observable.
17


Reviews:
[{"review_id": "um-ZX_gSWF7", "reviewer": "Reviewer_bb3F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an RL method designed to generate all possible actions. I assume this is done to help exploration of the space in order to learn a better policy. The approach is evaluated on a grasping task.", "summary_of_the_review": "The paper proposes a method to generate all possible actions and is evaluated on a grasping setup. The paper is well written, but the experiments are weak/minimal.", "strength_and_weaknesses": "The paper works on an interesting and important problem of finding/exploring all actions. The paper is well written and motivates the problem nicely.\n\nThe biggest weakness is the lack of experiments. There is only one experiment, and its setup, environment, action space, etc. are not defined. Further, there is no comparison to other methods, other than a few proposed in the paper. This makes it difficult to judge the work and its contribution.\n\nIt is also unclear how much the proposed approach helps vs other things used (imitation learning, etc.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EBDmtUlJsr", "reviewer": "Reviewer_RMrn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of large solution spaces by decomposing problems into smaller subproblems and training agents to consider all feasible actions for the subproblems, hopefully enhancing transferability of solutions to new problems.  They develop a generative model to produce all feasible actions and evaluate it on a 2D grasping problem.\n", "summary_of_the_review": "I think that this is a paper worthy of publication in ICLR, though my confidence is not high.  It clearly defines a problem and concisely presents their approaches. \n", "strength_and_weaknesses": "Strengths:\n - The idea of generating all feasible actions is an interesting one, and has intuitive appeal to facilitate skill transfer.\n - The authors present novel approaches to estimate gradients in various sampling contexts.\n - The authors present novel approaches to reduce bias in training the critic.\n - Empirical results are encouraging. \n\nWeaknesses:\n - Typo on page 8: \"RKL, has\" -> \"RKL has\"\n - The legend of Figure 3(b) covers some numbers\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "CGgRiqejLgL", "reviewer": "Reviewer_JUVt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to train a generative neural network to generate all feasible actions of a sub task by interacting with an environment. The trained generator provides a distribution that matches a uniform distribution over all feasible actions. The authors derive the general optimization target for arbitrary f-divergences using a combination of kernel density estimates, resampling, and importance sampling. An application of a 2D robotic grasping problem is presented in this experiment to show the proposed algorithm outperforms other related methods. ", "summary_of_the_review": "This paper considers a very important problem in real application. The proposed method is novel and well supported by theoretical and experimental results. ", "strength_and_weaknesses": "The problem considered in this paper is to generate all feasible actions for a certain task which is hard to obtain for complex tasks. The author propose a novel learning method for generative neural network in order to generate feasible actions. In order to solve the problem, the authors further propose a gradient estimator for the gradient of f-divergence. The proofs are rigorous and correct. \n\nWeakness: I found this paper a bit hard to follow since some theories are not well explained in the paper. It would be great if the authors can provide more comprehensive explanation for the result you used in the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AiG_IQ3uCL", "reviewer": "Reviewer_fUjg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to discover the feasible actions for a given state in an environment. The algorithm proposes actions, which end in a single step in either success or failure, as determined here by a hand-designed heuristic. The generated actions are multimodal in nature and show limited generalization to out-of-distribution tasks. The method is tested on a robotic gripping task with 3-4 shapes. The results demonstrate that, in this domain, the proposed method can discover more modes of the feasible action distribution than the baselines.", "summary_of_the_review": "This is a novel and sound method for multimodal action discovery and with proper comparison, clearer writing, and demonstrating the method's generality on at least one other domain it can be a strong contribution.", "strength_and_weaknesses": "The strengths of the paper are:\n1. It is theoretically well motivated and sound. The authors explore various f-divergences to find the best one for this setting.\n2. The analysis of the various modes and grasping success for different methods is thorough for the chosen domain.\n3. As far as I can tell, their approach is novel.\n\nThe two biggest areas for improvement are:\n1. Lack of comparison with any relevant literature - grasping is well studied problem and comparison with some method in literature besides author-crafted baselines is required. The authors explore a few variations of their own method as well as compare to some sanity-check baselines but  the grasping success % can be compared to other methods in literature. I also found the related works section lacking in references to grasping literature and instead referencing peripheral works such as LLMs.\n2. Unclear writing - It is unclear which problem the method is solving, that of generating feasible actions or discovering skills or both. Skills are typically referring to a sequence of actions whereas the case considered here is closer to CB with a single state/action. It seems the terms \"multimodal action distributions\" and \"skills\" are being conflated and further clarification is needed. Moreover, while in the abstract the claim is to learn \"reusable and transferrable skills\", there is no discussion later in the paper about the reusability or transferability of the action modes discovered.\n\nSmaller points of feedback:\n\n3. It is unclear what happens in environments where the hand-designed heuristic for judging success/failure is not present. The method is only tested on a single environment of 2D grasping and could benefit from being tested on at least one other problem to get an idea of how well it works as a general method for feasible action generation.\n4. The role of the \"critic\" network is not clearly explained. Why is it that the learned critic can generalize over environment samples better, and hence provide better gradients, than directly training the KDE on them?\n5. What is the role of the imitation learning dataset used to bootstrap the critic? An ablation on how much data is needed there would be interesting to see.\n6. A study of the sample complexity in terms of environment interactions would be a relevant result.\n7. Results for the figure 8 shape are missing from figure 3.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject