Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions
Authors: Juan Del Aguila Ferrandis
Joao Moura
Sethu Vijayakumar
Venue: CoRL 2024 (2024)
Keywords: State Estimation
Reinforcement Learning with Tactile Sensing
Non-prehensile Manipulation
TL;DR: We propose a method to learn visuotactile state estimators and uncertainty-aware control policies for non-prehensile manipulation under occlusions and extensively validate it in simulation and robotic hardware.

Abstract: Manipulation without grasping, known as non-prehensile manipulation, is essential for dexterous robots in contact-rich environments, but presents many challenges relating with underactuation, hybrid-dynamics, and frictional uncertainty. Additionally, object occlusions in a scenario of contact uncertainty and where the motion of the object evolves independently from the robot becomes a critical problem, which previous literature fails to address. We present a method for learning visuotactile state estimators and uncertainty-aware control policies for non-prehensile manipulation under occlusions, by leveraging diverse interaction data from privileged policies trained in simulation. We formulate the estimator within a Bayesian deep learning framework, to model its uncertainty, and then train uncertainty-aware control policies by incorporating the pre-learned estimator into the reinforcement learning (RL) loop, both of which lead to significantly improved estimator and policy performance. Therefore, unlike prior non-prehensile research that relies on complex external perception set-ups, our method successfully handles occlusions after sim-to-real transfer to robotic hardware with a simple onboard camera.

Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions
Juan Del Aguila Ferrandis', João Moura Sethu Vijayakumar
School of Informatics, The University of Edinburgh, UK The Alan Turing Institute, UK
Abstract: Manipulation without grasping, known as non-prehensile manipula- tion, is essential for dexterous robots in contact-rich environments, but presents many challenges relating with underactuation, hybrid-dynamics, and frictional un- certainty. Additionally, object occlusions in a scenario of contact uncertainty and where the motion of the object evolves independently from the robot becomes a critical problem, which previous literature fails to address. We present method for learning visuotactile state estimators and uncertainy-aware control policies for pon-prehensile manipulation under occlusions, by leveraging diverse interaction data from privileged policies trained in simulation. We formulate the estimator within a Bayesian deep learning framework, to model its uncertainty, and then train uncertainty-awai aware control policies by incorporating the pre-learned estimator into the reinforcement learning (RL) loop, both of which lead to significantly im- proved estimator and policy performance. Therefore, unlike prior non-prehensile research that relies on complex external perception set-ups, our method success- fully handles occlusions after sim-to-real transfer robotic hardware with sim- ple onboard camera. See our video: https: http://oooouuooo//oo//eeeee be/hW-CBi_HWe
Keywords: State Estimation, Reinforcement Learning, Tactile Sensing, Non- prehensile Manipulation
Introduction
Non-prehensile manipulation is crucial skill for enabling versatile robots interact with ungras- pable objects, using actions such as pushing, rolling, or tossing. However, achieving dexterous non-prehensile manipulation in robots poses significant challenges. During contact interactions, dif- ferent contact modes arise such as sticking, sliding, and separation, and transitions between these contact modes lead to hybrid dynamics [1, 2, 3]. Furthermore, due to its underactuated nature, requires long-term reasoning about contact interactions as well as reactive control to recover from mistakes and disturbances [1, 2]. The frictional interactions between the robot, the object, and the environment difficult model, which creates uncertainty in the behavior of the object [4, 5].
The highly uncertain nature of the underactuated frictional interactions [4, 5] make the non- prehensile manipulation problem especially sensitive to occlusions. Previous non-prehensile works assume near pear-perfect visual perception from external systems, providing either point-cloud [6] or pose observations [7, 9, 10, 11]. However, moving towards more versatile onboard perception will make frequent occlusions unavoidable, either due obstacles the environment, self occlu- sions, or even human-induced occlusions, for instance in human-robot collaboration setting.
In this paper, we propose a learning-based system for non-prehensile manipulation that leverages tactile sensing to overcome occlusions in the visual perception. We decompose the learning task into a state estimation and a control task. We begin by training privileged policy with reinforce- ment learning (RL) in an occlusion-fred free environment and periodically storing checkpoints, i.e. the policy's 's parameters, various stages of training. Then, by rolling out wide range of optimal and suboptimal policy checkpoints, we able to collect diverse dataset of trajectories, which pro-
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

E
(a) = 0.0
(b) 9.0s
(c) f 11.0s
(d) t 16.5s
(c) f 19.0s
Figure 1: Snapshots from exemplar rohot motion pushing box to the target with occlusions. Columns correspond to different time-frames, with (h) and (d) corresponding cludes the reading the box pose. first row shows rohot setup, the second row shows the robot camera view for the tag/ohject detection, and the final row shows the rviz view with yellow and blue corresponding to the observation (from the camera) and the estimation of the box poses. The green area marks the target.
snapshots where the robot view
cess to introduce synthetic occlusions in the visual perception as well as sensory noise. We use this trajectory dataset train state estimator within Bayesian deep learning framework capture the uncertainty in the state prediction. We note that capturing the uncertainty within such 'black-box' deep learning based state estimators important for developing more robust downstream policies and enabling safer and more reliable real-world deployment. Finally, we train an RL control policy in an environment with visual occlusions, using the pre-learned state estimator in the loop.
We evaluate our proposed system on a challenging planar pushing task with frequent and prolonged visual occlusions, achieving 94% success rate. We also deploy the learned estimation and control framework robotic hardware with zero-shor sim-to-real transfer. Our contributions are:
We propose a novel approach to learn state estimators based on force interaction for non- prehensile manipulation under visual occlusions, leveraging privileged policies to generate diverse interaction data and a Bayesian deep learning framework to model the uncertainty,
We also propose learn uncertainty-avve control policies by integrating the pre-trained state estimator in the RL training loop.
We show that, by modeling the uncertainty, we are able to learn significantly more accurate estimators and, furthermore, training uncertainty-awarr control policies with the estimator in the loop leads to improved performance.
We validate that our approach exhibits good transferability a physical robot using purely on-board perception with both -occurll- and human- innuued visual occlusions.
2 Related Work
Non-prehensile Manipulation. Previous works successfully learn wide variety of non-prehensile manipulation skills, including planar pushing [7, 8, 9, 10, 11]. pivoting against the external envi- ronment to realize grasps [12, 13), retrieving objects cluttered environments [14, 15], and 6D rearrangement [6, 16, 17]. However, all of these methods avoid visual occlusions, generally by re- lying on complex perception schemes, external to the robot, that guarantee continuous tracking of the manipulated object. Some works rely on motion capture set-ups, such as Vicon, with multiple specialized cameras such that if some cameras experience occlusions, others continue providing liable readings [7, 8, 11]. Other works employ vision-based set-ups, with tracking tools such as

---

April'Tag [18]; however, they rely on external cameras strategically placed to avoid occlusions [9]. Works without explicit tracking systems also require specialized set-ups to avoid occlusions, for in- stance by using a camera under a transparent table to observe the object [10] or combining multiple external depth cameras to create a point cloud [6. 17]. In contrast, our proposed approach enables us to perform non non-prehensile manipulation tasks with simple on-board camera and leverage tactile sensing to keep solving the task during prolonged visual occlusions. We believe this is important step towards deploying robotic systems outside of the lab environment, without access to external perception set-ups and, hence, with likely occurrence of occlusions.
Dealing with Observation Uncertainty. There are model-based approaches for integrating uncer- tainty into planning, requiring either analytical or learned dynamics models [19, 20J. More relevant to this work, there are also multiple learning- baannd approaches for dealing with observation uncer- tainty. Many works train control policies directly in the simulated target environment and use domain randomization as well synthetic observation noise during learning achieve zero-shot sim-to-real transfer [7, 11, 21, 22]. Others combine multiple sensory modalities such as vision, haptic, and au- dio signals to handle scenarios where one or more modality might be unreliable [23, 24, 25, 26). Recently, a popular technique is to first learn a teacher policy assuming access to the ground-truth environment state and then train a student policy to imitate the teacher's actions given the observa- tions from the deployed environment [27, 28, 29]. While taking inspiration from these methods, we find that these alone are insufficient achieve good performance in the challenging non-prehensile manipulation task with prolonged visual occlusions.
3 Method
3.1 Problem Formulation
We formulate the non-prehensile task as a finite-horizon Partially Observable Markov Decision Pro- cess (POMDP) described by the tuple AS P, R, r). In particular, at every timestep t, the environment has a state 8; E S and the agent takes an action a, E A. resulting in a new state 81+1 according to the transition model P x A Pr(S). Due to the partially observable set- ting, the agent is unable to perceive 8+41 directly and instead receives an observation O Ω that depends 8c+1 and a, based the observation model O S A + Pr(St). The agent ceives a reward r; associated with the transition (8c, a,, 11+ which is given by the reward function R : AxS + R. The objective is to learn a policy : Ω - Pr(AA) that maximizes the discounted cumulative reward Ex
EnTo' y're]. with maximum horizon H and discount factor E (0, 1).
The partially observable nature of the task is critical since we are interested in learning non- prehensile manipulation policies that handle occlusions in the visual perception. Specifically, we assume that, in addition to correlated and uncorrelated Gaussian sensory noise, the environment observation On features occlusions in the pose of the manipulated object. We assume that robot end- effector pose and force measurements are available through proprioception without any occlusions.
We decompose the learning task into a state estimation and control task, where first learn state estimator f(o:) (&t, Σx) and then use it to learn control policy π(š,, Σe). In practice, since we consider occlusions in the object pose q. chi we only train f to predict q: oh Additionally, assume sequential access to states and observations, enabling the use of recurrent architectures for f and π with internal representations that capture the input evolution.
3.2 Learning the State Estimator
Data Collection. Training a model-free state estimator for non prehennse manipulation that ro- bust to visual occlusions demands large and diverse dataset of interactions to ensure high accuracy and reduce out-of-distribution scenarios. Collecting this data to the extensive data requirements, the cost of hardware, and the need for human supervision consequently, we utilize simulation instead. This has the added advantage of being able to ex-
real-world setting is difficult due
3

---

plore simulation environments with different physical characteristics, which is crucial to enable the learned state estimator to generalize effectively across different environments. We propose that an effective strategy is to first train a privileged policy prriu that has access to the ground truth state of the environment, and then rollout a set of n checkpoints stored during training { privs........................... .... T Trice) collect the interaction dataset D [T', T2...... .......... J. where I; = {so: $1 $1....} } denotes trajectory in the state space generated by one of the privileged policy checkpoints. The privileged checkpoints include untrained random policies well wide range of suboptimal and optimal policies. This approach allows to collect an arbitrary amount of task-relevant data, ensuring that a diverse set of behaviors is present, and thereby preventing the estimator from becoming biased towards a particular behavior. Note that it is straightforward to use RL for learning Tprio(S) (t)) without occlusions.
Data Processing. We want to train a state estimator ʃ that maps environment observations Or to estimated object poses q: q along with the associated uncertainty Σ, Therefore, we process the dataset DD using our assumed observation model O to produce dataset D' g.... where I = ((on; )o"),, (01, ( )) ...3 is trajectory in the observation space with the corresponding ground-truth trajectory of the object pose. We consider a state 8, = (a: hi qi fi) composed of the object Ttth and the robot end-effector of poses, and the end-effector force f(. We use an observation model o2 = (s;) ) that adds both correlated (sampled at the beginning of the trajectory), and uncor- related (sampled at each timestep) Gaussian noise to the observation the state. Additionally, O applies occlusions to the object pose, which can start at every timestep with a probability of p and have duration sampled from Gaussian distribution. Throughout an occlusion, the observed object pose remains the same as the last available pose prior to the start of the occlusion. The observation also includes a binary indicator Ç. which is 1 for occlusion and 0 otherwise.
Types of Uncertainty in Bayesian Deep Learning. Our state estimator f captures the two main types of uncertainty modelled under a Bayesian framework: aleatoric uncertainty and epistemic un- certainty [30]. Aleatoric uncertainty captures the noise inherent to the data. This arises for example due to sensory noise and it is irredueible even with additional data. Epistemic uncertainty captures the uncertainty in the model parameters, and it can be arbitrarily reduced with sufficient training data. Furthermore, we consider setting where the uncertainty varies depending on the input to the model, known as heteroscedastic uncertainty.
Modeling Aleatoric Uncertainty. We use the dataset D' to learn state estimator f(o.) (q.01 Eg(e) that predicts both object pose and alcatoric uncertainty. We formulate our estimator to predict the mean and covariance, such that p(q:t) |o)) ~ N(qq Sale), and use the negative log-likelihood [30, 31] as the loss function
obj
1 In (S ) + (1t")) - (6 )
T
Stale -1
(1.
= 2
In practice, we use a diagonal covariance matrix to reduce the output dimensionality.
Modeling Epistemic Uncertainty. To model the epistemic uncertainty of the learned state estima- tor we use Dropout variational inference [30, 31, 32). More specifically, dropout [33], common regularization technique for neural networks, can be interpreted as approximate variational infer- ence in Bayesian neural networks [30, 32]. Hence, we apply dropout to the state estimator f at both training and testing time, which allows us to perform stochastic forward passes, effectively sampling from the approximate posterior of f. This is known as Monte Carlo (MC) dropout. Suppose that we perform M stochastic forward passes of J, we can approximate the epistemic uncertainty
T
M M
M
EePt
1
bbj
1
Σ gouh)
Σ
q
obj
(2)
M
where qo0h) are the sampled outputs [31]. This formulation also allows to perform more accurate predictions of the object pose by using the approximate predictive mean
M
obj
1
9
obj
(3)
M
im1

---

Combining Aleatoric and Epistemic Uncertainty. We assume independence of the aleatoric and epistemic uncertainties, as in [30, 31j. Therefore, the estimated total uncertainty, using M Monte Carlo dropout samples of f(o)), is given by Σ, = En + Depi resulting in
M
1 M
Σ,
ool
bob)) qi'
(G qi )
1
Σ
qi
obs
Σ
)
obj qi
(4)
M
M
M
1=1
1=1
3.3 Learning the Control Policy
After training the state estimator, we can use it to learn an RL policy Test in an environment with visual occlusions. In particular, given an observation of the environment Or, we feed through our state estimator, using Monte Carlo sampling to approximate the predictive mean object pose qe and covariance Σ,, shown in Eq. (3) and Eq. (4). Since our state estimator only predicts the object pose, we construct the complete state estimate & by replacing the object pose in or with q, Finally, we provide this state estimate &, along with the estimated uncertainty Σ, as input to the RL policy, such that Test ($t E₁). By learning with the state estimator in the loop, Teat can leverage the uncertainty estimation and better adapt to the inaccuracies of the estimator.
4 Experiment Set-up
Simulated RL Environment. We develop planar pushing simulation environment for policy training and data collection using Isaac Sim [34]. In order to speed up the simulation and training, we abstract away the robot as a spherical pusher. We also assume fixed-size cuboid as the manipulated object. At the start of every episode, we randomly sample the starting object and pusher poses within the workspace. The goal of the episode is to push the object within 1 cm and 15 deg of the target position and orientation. We define the reward as Pt = Tterm + k- (1-derans) ) K2 drot) + kss (1 vp)). where Tterm is the termination reward (with 50 for reaching the target and 110 for violating the workspace boundaries), derans and orot are the normalized Euclidean and rotational distances the target, up is the normalized magnitude of the pusher velocity, and k1 0.1, k2 0.02, k 0.004. The control frequency is 15 Hz with maximum horizon of H = 300 steps or 20
Sim-to-real Transfer.
To enable zero-shot sim-to- real transfer, we use dynamics randomization on the mass, friction, and restitution. We also add correlated noise, sampled at the beginning of the episode, and uncorrelated noise, sampled at every timestep, to the observation. When considering scenarios with occlu- sions, they can start at any timestep with a probability of = 1/30. Table summarizes the occlusion duration and observation noise distributions.
Parameter
Distribution
Occlusion Duration
(100, (O 0.7%) N NO 0.0025²) m WO 0.05²) rad
Force noise
Position noise
Orientation noise
Table 1: Occlusion and noise distributions.
Privileged Policy (Tprio)) We train the privileged policy with Proximal Policy Optimization (PPO) [35] in the occlusion-free environment. The policy takes actions a; = [Ua,, vy] consisting of the and y pusher velocities, limited to a maximum 0.05 ms-1. We discretize the action space, with 11 bins for each velocity, to use categorical exploration, which improves RL performance for planar pushing [11]. We consider failure if the policy reaches the maximum horizon without com- pleting the task. We use recurrent architecture for the policy and value networks in PPO to capture the hidden dynamics of the environment.
State Estimator (f). To train the state estimator, we first deploy 300 uniformly spaced privileged policy checkpoints to collect 750,000 trajectories in the occlusion-free enviromment. We add ob- servation noise and synthetic occlusions to these trajectories, according to Table 1. For the state estimator, we use a recurrent architecture to capture the temporal dynamics of the environment. The architecture consists of a Long Short-Term Memory (L.STM) layer of size 1024 followed by three linear layers of sizes (512, 256, 8), with tanh non-linearities. We apply dropout with probability 0.2 before every linear layer. In line with [30], we estimate the epistemic uncertainty with 50 stochas- tic forward passes. The estimator outputs the predicted object pose and aleatoric uncertainty. We
5

---

represent the object pose as 9:"h = (x, y, sin(f), cos(8)). where (z. y) is the planar position and 0 the orientation with respect to the axis. For the uncertainty, the estimator predicts the natural logarithm of the diagonal elements the covariance matrix Eple.
Control Policy (Test). We use the same architecture and training procedure for the control pol- icy Test with Tprics but with occlusions in the training environment. We obtain the state and uncertainty estimates ($ Ex) through the pre-trained state estimator and input them to Test.
Simulation Experiments
5.1 Effect of Modeling Uncertainty
We study the effect of modeling aleatoric and epistemic uncertainty on the accuracy of the state estimator. In particular, we compare the performance of our proposed formulation against two alter- native formulations: one that removes the Monte Carlo (MC) dropout step, used model epistemic uncertainty, and another that removes MC dropout and replaces the likelihood loss, used to model aleatoric uncertainty, with a standard Mean Squared Error (MSE) loss. We also include a baseline that uses the visual pose observation without processing. We measure the translational and rotational error of each estimator in separate test dataset and summarize the results in Table 2.
The results show that modeling aleatoric and epistemic uncertainty significantly improves the state es- timator accuracy. The likelihood loss enables the network to predict the aleatoric uncertainty and hence reduce the effect of noisy inputs on the loss, an effect known as learned loss attenuation [30]. MC dropout enables us sample from the ap- proximate posterior and hence ob- tain a more accurate pose estimation through the approximate predictive mean. Finally, our state estimator obtains similar or better accuracy than previous model-based approaches [36, 37] in sig- nificantly more difficult scenarios. We consider much longer occlusions, with duration ( 5²) s, higher observation noise, and more complex trajectories involving large object rotations.
Estimator Method
Trans. Error (mm)
Rot. Error (deg)
Mean L.2
RMSE
Mean Abs.
RMSE
Vision Baseline
49.83
59.65
22.56
39.61
MSE Loss
7.62
6.49
2.51
3.37
Likelihood Loss
6.83
5.92
2.46
3.39
Ours
4.42
3.84
2.10
2.96
Table Accuracy of the state estimator without modeling uncer- tainty (MSE loss), modeling aleatoric uncertainty (likelihood loss), and ours, which models alcatoric and epistemic uncertainty (likeli- hood loss MC dropout). The baseline relies only visual input.
5.2 Comparison of Policy Training Approaches
We evaluate different techniques to learn the control policy. consider learning Test (8e, E.), taking as input the state and uncertainty prediction provided by the pre-trained state estimator, as well as Teee (8)) which only receives the state prediction. We also learn a policy using the variant of our estimator trained with MSE loss, without capturing uncertainty. Additionally, an end-to-end RI. approach, which attempts to learn the policy directly without explicitly learning a state estimator. Furthermore, we evaluate the performance of (&)) directly applying the optimal privileged policy with the pre- pre-trained state estimator. Finally, we experiment with a teacher- student behavior cloning approach [27, 28, 29]. We use our privileged policy as the teacher to collect optimal trajectories in the occlusion-fred environment, process them to introduce occlusions and noise, and train a student policy to imitate the optimal actions via behavior cloning.
consider
Fig. 2 shows the resulting performance. Our uncertainty-aware policy Test($r, Σ,) achieves the highest success rate (94%). Note that, without explicitly receiving the uncertainty as an input, (&)) achieves slightly lower final performance (92%). We believe this be- cause, through extensive environment interactions during training, with the state estimator in the loop, is possible to implicitly obtain information about its uncertainty. Nevertheless, Test (5t) still leverages our estimator that captures uncertainty. In fact, when we train the same policy using the MSE estimator, the performance considerably degrades (64%) due to the

---

lower estimator accuracy. Overall, by modeling aleatoric and epistemic uncer- tainty we obtain a significantly better esti- mator and, furthermore, providing the un- certainty estimate explicitly the policy leads to slightly improved performance.
MSE Extimotor
(ad Behnviee Claning
Rad(A]
End-to-end
90
80
Applying the optimal privileged policy directly with our state estimator already achieves good success rate (83%). How- ever, including the estimator in the RL training significantly improves perfor- mance (94%), indicating the advantage of learning policies that, explicitly implic- itly, account for estimator uncertainty. Fi- nally, end-to-end RI. and behavior cloning achieve low success rate, 12% and 38%, highlighting the need to explicitly learn state estimators.
Stoosss Rtae (96)
60-
80
30-
20
Training Step
Figure Performance with different training configurations of the control policy. For RL learning curves, we report mean and standard deviation across three training seeds.
5.3 Effect of Occlusion Duration
We evaluate the methods discussed in Sec- tion 5.2 with different occlusion durations and show the results in Fig. 3. We re- port success rate averaged over 2,000 runs with randomized environment configura- tions. We keep the probability = 1/30
Teali, Ex) (A,)
MSB Estimntor End-so-nn Rí.
TpnulA] Dehavtor Cloning
Suxios Hade (%6)
75-
that an occlusion starts at each step but fix the occlusion duration for each sce- nario and, in the case of full occlusions, lasting until termination. We find that (est Σ and (&)), the policies trained with our proposed estimator in the loop, achieve minimal performance degradation, 6.7% and 8.3%, between the no and full occlusions scenarios. End-to-end RL and behavior cloning, have the highest decline in performance, 45.8% and 37.6%.
No Occluniion
Go Ohclusions
10s Declusions
Full Ouclusions
Figure 3: Performance of different control policies with in- creasing occlusion durations.
5.4 Estimated State and Uncertainty Trajectories
We visualize in Fig. 4 estimated state and uncertainty trajectories generated by
Teelf, Ee) Trie Trijectory
Twosth) Thee Trajectory
Start
T..(h, E. Prot. Tiajenney
Swath) Ptod. Thjletory
Test (St, Σ,) and (S:) in full clusion scenario. For the uncertainty, we use the sum of the predicted vari- ances corresponding to the (I, y) posi- tion. Test (8t. Σx) manages to solve the task and Tprio fails due to violat- ing the workspace boundaries. While Tprin (&t) immediately makes contact and attempts rotate the box towards the get, Tear Σx) learns switch contact faces, allowing it to perform more grad- ual rotation, possibly using sticking rather than sliding contact, which creates less uncertainty in the object pose. (St) takes a sharp turn leading to a progressive increase in the estimator uncer- tainty and error. On the other hand, Test Ex) gradually re-orients the box and again switches contact faces before reaching the target. Notice the progressive decrease in uncertainty during the
Fueld, So) Pred.
Uncertliity
Ta) Uncertontt
EB
0.3
0
un
n.0
-02
-033
11.0
[m]]
Figure Trajectories generated under full occlusion.
7

---

initial rotation as the estimator obtains more interaction data, necessary to infer the observation noise and hidden environment dynamics. The final contact-face switch also leads to a dramatic decrease in uncertainty since making contact in a perpendicular face adds significant information about the object pose, Overall, we find that training the control policy with the estimator in the loop leads to different learned behaviors leveraging various strategies reduce the uncertainty. Further, we quantitatively evaluate the difference in contact switches, an effective uncertainty-reducti strategy, between Apriv (8t), the privileged policy in the occlusion-free environment, and Teat ($ Ex) in the standard environment with occlusions. We find that pprro(()) (5p) makes 2.68 = 1.53, while Test (st, Ex) makes 4.22 + 2.67 contact switches per episode, averaged across 20, 000 randomized episodes.
6 Hardware Experiments
We conduct hardware experiments using KUKA iiwa robot arm with F/T sensor at the wrist and pusher end-effector, as shown in Fig. 1. A camera mounted immediately above the pusher tracks the pose of the manipulated object through AprilTag [18] markers. We use OpTaS [38] to map policy actions (end-effector velocities) to robot joint configurations. Note that this set-up is naturally prone to occlusions as the markers might be outside the camera's ' field of view or obstructed by the pusher itself. Additionally, we observed significant force readings outside of contact interactions due to the dynamics of the pusher. During training, we neglected these dynamics and instead relied on adding large amounts of correlated and uncorrelated noise to the force observation to provide robustness. For future work, we aim to model and introduce the pusher dynamics in the learning process.
We deploy our proposed policy Test (8;, Σn) and state estimator f(or) in the robot with zero-shot sim-to-real transfer. We observe similar information gathering strategies as discussed in Section 5.4. For instance, the policy tends to perform a contact switch, which reduces uncertainty, right before aligning the object with the target, as illustrated in Fig. We evaluate two scenarios with random- ized initial configurations: (1) occlusions arising from the onboard perception set-up, and (2) longer human-inducee induced occlusions. In the first scenario, the second, our method achieves 10/11 10 with 5s occlusions and 7/10 with full occlusions. The sup- plementary video shows examples from those scenarios. We find that the state estimator and control policy exhibit good transfcrability to the physical robot, even in the presence of unseen dynamics.
method achieves 19/20 successful runs and, in
7 Limitations and Conclusion
Limitations. Compared to model-based approaches using analytical models, model-free state state estimators require vast amounts of interaction data providing sufficient coverage of the state space. In this work, we use a fixed-shape for the pusher and object, without exploring how to generalize the method to diverse geometries. Additionally, our current simulation set-up neglects any effects of the pusher dynamics on the force measurements, observable during robot experiments. We omit any quantitative evaluation on the accuracy of the estimated uncertainty, which would require a well- characterized system. Furthermore, despite the planar pushing task capturing the key challenges of other non-prehensile manipulation skills, the generalization of our findings to other tasks remains untested. Finally, we only use the estimated uncertainty as an additional input to the control policy; however, we could also apply it to predict failures, trigger fallback recovery mechanisms, and to augment the training dataset for the state estimator, improving the state space coverage.
Conclusion. We propose method to learn visuotactile state estimators for non- non-prehensile manipu- lation under occlusions by leveraging privileged policies, learned in simulation, to generate diverse interaction data. Furthermore, we formulate the estimator within a Bayesian deep learning frame- work, to model its uncertainty, and then train uncertainty- -aware RL control policies by incorporating the pre-learned estimator into the RL training loop. both of which lead sigmificantly improved esti- mator and policy performance in extensive simulation experiments. Unlike prior non-prehensile research that relies on complex external perception set-ups to avoid occlusions, our method achieves successful sim-to-real transfer to robotic hardware with a simple onboard camera, under occlusions.
8

---

Acknowledgments
This work was supported by the JST Moonshot R&D (Grant No. JPMJMS2031), the Kawada Robotics Corporation, and the RAICo Fellows Scheme.
References
[1] M. T. Mason. Mechanics and planning of manipulator pushing operations. The International Journal of Robotics Research, 5(3):53-71, 1986. 46::10.1777073333466000000000
[2] R. Hogan and A. Rodriguez. Reactive planar non-preheneee non-prehensile manipulation with hybrid model predictive control. The International Journal of Robotics Research, 39(7):755-773, 2020. doi: 10.1177/0278344920919999
[3] J. Moura, Stouraitis, and S. Vijayakumar. Non- Non-prehensile planar manipulation via trajectory optimization with complementarity constraints. In 2022 IEEE International Conference on Robotics and Automation (ICRA), pages 970-976. IEEE, 2022. doi:10.1 10/010//6///2443322222222 9811 994.
[4] J. Zhou, R. Paolini, A. M. Johnson, J. A. Bagnell, and M. T. Mason. A probabilistic planning framework for planar grasping under uncertainty. IEEE Robotics and Automation Letters, (4):2111-2118, 2017. 1. (09//.RA.201..277000
[5] M. Bauza and A. Rodriguez. A probabilistic data-driven model for planar pushing. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3008-3015. IEFE, 2017. doi:10. 101.1100ARA...1.7779888.
[6] W. Zhou, B. Jiang, Yang, C. Paxton, and D. Held. Hacman: Learning hybrid actor-critic maps for 6d non- non-prehensile manipulation. In Conference on Robot Learning, pages 241-265. PMLR, 2023.
[7] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robolics and Automation (ICRA), pages 3803-3810. IEEE, 2018. 000110.110//1RAA20888..6666
[8] K. Lowrey, S. Kolev, J. Dao, A. Rajeswaran, and E. Todorov. Reinforcement learning for non- prehensile manipulation: Transfer from simulation to physical system. In IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), pages 35 42, 2018. doi: N0.1100/SSMPARR 2018.837666886.
[9] R. Jeong, J. Kay, F. Romano, T. Lampe, T. Rothorl, A. Abdolmaleki, T. Erez, Y. Tassa, and Nori. Modelling generalized forces with reinforcement learning for sim-to-real transfer. arXiv preprint arXiv: 1910.0941, 2019.
[10] I. Cong, H. Liang, P. Ruppel, Y. Shi, M. Görner, N. Hendrich, and J. Zhang. Reinforce- ment learning with psssonpropriocception model for robot planar pushing. Frontiers in Neuro- robotics, 16, 2022. ISSN 1662-5218. doi: 0..3389/fnboo.22222829444
[11] J. Del Aguila Ferrandis, J. Moura, and S. Vijayakumar. Nonprehensile planar manipulation through reinforcement learning with multimodal categorical exploration. In 2023 JEFF/RSI International Conference on Intelligent Robots and Systems (IROS), pages 5606-5613, IEEE, 2023. doi: 10. 109/ 10341629.
[12] W. Zhou and D. Held. Learning to grasp the ungraspable with emergent extrinsic dexterity, In Conference on Robot Learning, pages 150-160. PMLR, 2023.
[13] S.-M. Yang, M. Magnusson, A. Stork, and Stoyano. Learning extrinsic dexterity with pa- rameterized manipulation primitives. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 5404-5410, IEEE, 2024. 160:10.1199/CCA457447.0000 1
101.111611110 1431.

---

[14] Y. Jiang, Y. Jia, and X. Li. Contact-aware non non-prehensile manipulation for object retrieval in cluttered environments. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023. doi:10.1 109/ 101.1220222201117666
[15] G. Liu, J. De Winter, D. Steckelmacher, R. K. Hota, A. Nowe, and B. Vanderborght. Saateerbbrggh.SSneggssitttttt task and motion planning with reinforcement learning-based non-prehensile actions. lEEE Robotics and Automation Letters, 2023. 10i:
[16] M. Kim, J. Han, J. Kim, and B. Kim. Pre-and post-contact policy decomposition for non- prehensile manipulation with zero-shot sim-to-real transfer. In 2023 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 10644-10651. IEEE, 2023. doi: 10.1109/1R055555222020
[17] Y. Cho, J. Han, Y. Cho, and B. Kim. CORN: Contact- Object Representation for Non- prehensile Manipulation of General Unseen Objects. In International Conference on Learning Representations (ICLR), 2024.
[18] J. Wang and E. Olson. Apriltag 2: Efficient and robust fiducial detection. In 2016 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 1993-419... IEEE, 2016. doi: 10. (09/IROS.2016 67766677
[19] S. Patil, Y. Duan, J. Schulman, K. Goldberg, and P. Abbeel. Gaussian belief space planning with discontinuities in sensing domains. In 2014 IEEE International Conference Robotics and Automation (ICRA), pages 1483-6490. IEEE, 2014. 100110.109//CRA.2144..000
[20] G. Chou, N. Ozay, and D. Berenson. Safe output feedback motion planning from im- ages via learned perception modules and contraction theory. In International Workshop on the Algorithmic Foundations of Robotics, pages 349-367. Springer, 2022. doi:10.1007/
[21] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipu- lation. The International Journal of Robotics Research, 39(1)3320,, 2020. doi:10.1117/
077366419887449
[22] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[23] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3307, 2018. doi: 109.100///...1118... 28827997..
[24] K. Zhang, M. Sharma, M. Veloso, and O. Kroemer. Leveraging multimodal haptic sensory data for robust cutting. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), pages 409-416 IEEE, 2019. doi: 9
[25] Y. Hao, R. Wang, Z. Cao, Z. Wang. Cui, and Sadigh. Masked imitation learning: Dis- covering environment-invarian modalities in multimodal demonstrations. In 2023 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 1-7. IEFE, 2023. doi: 10. 1t0:000000/R005552222222..03
[26] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L.. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 IEEE International Conference on Robotics and Automation (ICRA), pages 8950. IEEE, 2019. 110.:0.1110//CA22211997777
[27] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl. Learning by cheating. In Conference Robot Learning, pages 66-75, PMLR, 2020.
10

---

[28] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust percep- tive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62):eabk2822, 2022. doi: 200011166cciroboicsabbk2333.
[29] T. Chen, J. Xu, and P. Agrawal. A system for general in in-hand object re-orientation. In Confer- ence on Robot Learning, pages 297-307. PMLR, 2022.
[30] A. Kendall and Y. Gal. What uncertainties do need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
[31] R. L. Russell and C. Reale. Multivariate uncertainty in deep learning. IEEE Transactions on Neural Networks and Learning Systems, 33(12)77937-7943, 2021, doi:10.11 10w.1119//NNNS.... 3086757.
[32] Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning. In International Conference on Machine Learning, pages 1050-1059. PMLR, 2016.
[33] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 (1):1929-1958, 2014.
[34] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.104
2021.
[35] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633: 2017.
[36] K.-T. Yu and A. Rodriguez. Realtime state estimation with tactile and visual sensing. ap- plication to planar manipulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7778-7785, IEEE, 2018. 1o::10.1109/1C. 111001/AA2......831111111
[37] M. A. Lee, B. Yi, R. Martin-Martin, S. Savarese, and J. Bohg. Multimodal sensor fusion with differentiable filters. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10444-10451. IEEE, 2020. doi: 10. 101100/000055443220000.333
[38] C. E. Mower, J. Moura, N. Z. Behabadi, S. Vijayakumar, T. Vercauteren, and C. Bergeles. Op- tas: An optimization-basee task specification library for trajectory optimization and model pre- dictive control. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 91-9944. IEEE, 2023, (00.1109// 2022.1011127...
[39] N. Heess, D. Tb, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 000. 2017.
[40] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4/ 44826990, 2014.

---

Appendix
A Simulation Environment
We develop our planar pushing simulation environment using NVIDIA Omniverse Isaac Sim due its GPU parallelization capabilities, which significantly accelerate the training and data collec- tion process. Fig. 5 shows visualization of the simulation environment. The rectangular planar workspace has dimensions 0.6 m x 0.3 m. Note that designed this workspace based on the max- imum reachability in our robotic hardware set-up. For the manipulated object, we use cuboid of size 0212 m X 0.1 m x 0.07 m, and for the pusher we use sphere of radius 0.013 m. We enforce the workspace boundaries with respect to the centroids of the pusher and the object. During policy training and data collection, we randomize the dynamics of the environment, including the mass of the manipulated object as well friction and restitution coefficients of the table, the pusher, and the object. Table 3 summarizes the randomized dynamics parameters and their corresponding sampling distributions.
Figure 5: Planar pushing simulation environment Isaac Sim. The pusher is shown in red, the manipulated object in dark blue, and the target pose in light blue.
Parameter
Distribution
Statie friction
U(003, 0.5)
Dynamic friction
0(0.1,0.3)
Restitution
((0.1,0.7)
Object mass
((3.0,3.5) kg
Table 3: Dynamics randomization parameters and corre-
sponding sampling distributions.
B Additional Training Details
B.1 Reinforcement Learning Policies
We process the RL observation by scaling each component to the range [-1, In particular, we scale the (x, y) coordinates using the workspace dimensions and for the object orientation f we use si) and cos( For the pusher force ff we apply clip(ff. 110, 10)/10 10 component-wist wise. In practice, this means that we limit the magnitude of the force reading along each axis to N. When providing the predicted uncertainty from the state estimator as an RL policy observation, we the standard deviations, clipped to the range [0, 1). Note that in these cases where the RL policy receives the predicted uncertainty, keep the reward function unchanged.
12

---

During RL policy training, we use learning rate scheduler based on the KL divergence of the policy, as in [39, 22). The scheduler has a target KI. divergence of 7 10-3, a minimum learning rate of 1.5 10-4 and maximum learning rate of 10-2. For the policy function, we use a neural network architecture with the following layers and corresponding sizes: linear (128) + LSTM (256) linear (128) linear (22), with tanh nonlincarities. The output consists of 22 logits that define the categorical distributions neural network architecture but replace the final linear (22) layer with linear (1) layer that outputs the state value prediction. During training, when episodes reach the maximum horizon, thereby terminating, we use the value function prediction corresponding to the final observation to bootstrap the final reward. Table summarizes the remaining RL hyperparameters for PPO [35] training. Finally, Fig. 6 shows the training performance of the privileged policy Tyrio(81) in the occlusion- free environment.
the and y velocities. For the value function,
the same
Hyperparameter
Value
Rollout Steps
100
Parallel Environments
4000
Mini-batch Size
25000
Epochs
5
Clip Range (e)
0.2
Discount Factor (7)
0.993
GAE Parameter (A)
0.95
Entropy Loss Coefficient
0.01
Value Loss Coefficient
1.0
Table 4: PPO hyperparameters.
100
Suaoeia Rate: (6)
Training Step
Figure 6: Training performance of the privileged policy Tprii((s)) We report mean and standard deviation across three random seeds.
B.2 State Estimator
To train and evaluate the state estimators, we collect separate training, validation, and testing datasets containing 7.5 10a, 1.5 10s and 1.5 10a trajectories respectively. We process the trajectories scaling the pose and force measurements to the range [-1, 1] as discussed for the RL training in Ap- pendix B. Additionally, Table 5 summarizes the training hyperparameters for the state estimator.
13

---

Fig. 7 and Fig, 8 show the validation loss when training the state estimator with the likelihood loss and the MSE loss, respectively.
Hyperparameter
Value
Mini-batch Size
30000
Epochs
110
Learning Rate
10-3
Optimizer
Adam [40]
Sequence Length
300
Table State estimator hyperparameters.
Validatoon Los (Lllellhoo))
-100
-33-
-20
-125
Epoch
Figure 7: Estimator validation loss when training with the likelihood loss.
0,007
Validatio Los (MS3)
0.008
0.00%
0.000
Epech
Figure 8: Estimator validation loss when training with the MSE loss.
B.3 Behavior Cloning Baseline
To train the behavior cloning baseline discussed Section 5.2, we collect new training and valida- tion datasets using the last privileged policy checkpoint to provide optimal trajectories. We do testing dataset since evaluate the baseline directly in the planar pushing simulation environ- ment. The training and validation datasets contain 7.5- 7.5 and 1.5. 10 trajectories respectively. We
14

---

process the trajectories to scale the pose and force measurements, add sensory noise, and introduce occlusions using the same procedure as for the state estimator.
The neural network architecture for the behavior cloning policy the same as for the RL policy function, discussed in Appendix B. Hence, the output in both cases consists of 22 logits that define separate categorical distributions over the and velocities. We train the behavior cloning policy using loss function defined as the sum of the cross-entropy between the predicted and target distributions for the and velocities. The training hyperparameters are the same as shown in Table Finally, Fig. 9 shows the validation loss during training.
1182
0.61
Valiiatio Less (Croo-catrige)
0.00
0.57
0.56
0.56
20
50
Epocis
Figure 9: Behavior cloning policy validation loss.
15


Reviews:
[{"review_id": "8gHn8QtJap", "reviewer": "Reviewer_TfpZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "- The paper proposes a method for occlusion-robust visuotactile manipulation. The algorithm first trains a fully-observed policy with privileged information, renders imperfect/partial observations from this data, and uses this rendered data to train a state estimator, done via Bayesian deep learning. This state estimator is given as input to train a partially-observed RL policy. The method is evaluated on a planar pusher example with occlusions.", "summary_of_recommendation": "Overall, I recommend a weak accept: the paper is generally solid, but some more discussion on the limitations and comparison to existing uncertainty-aware visual control methods would strengthen the paper.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please see review", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81E5OzjdVR", "reviewer": "Reviewer_XGFC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The main idea of this paper is to develop a method that uses visuotactile fusion for state estimation and control to address the challenge of occlusions in non-prehensile manipulation tasks. The proposed approach leverages privileged policies trained in a simulation environment to generate diverse interaction data. This data is then used within a Bayesian deep learning framework to model uncertainty in state estimation. The pre-trained state estimator is integrated into the reinforcement learning (RL) training loop to develop control policies that can handle uncertainty effectively.", "summary_of_recommendation": "The method effectively addresses the challenge of occlusions in non-prehensile manipulation by using visuotactile fusion.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. The paper mentions using a Bayesian deep learning framework to capture uncertainty, which significantly improves the accuracy of the estimation. Could you explain the specific reasons for choosing the Bayesian framework? What are the distinct advantages of this method in paper's application compared to other possible choices, such as traditional neural networks or alternative uncertainty modeling methods?\n\n2. In the experiments, the paper shows performance differences between different control strategies, particularly those that incorporate the pre-trained state estimator in the RL training. Could you further explain the reasons for the performance differences, especially why the paper's strategy (e.g., $π_{est}(\\hat{s}_t, \\hat{Σ̂}_t)$) performs best in handling visual occlusions?\n\n3. The paper claims successful sim-to-real transfer, particularly under conditions of occlusions and sensor noise. Could you provide more detailed descriptions of the specific strategies employed to ensure this successful transfer? How does the method effectively maintain its performance when dealing with unknown dynamics and noise in real-world robotic hardware setups?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GxcK6Nwipd", "reviewer": "Reviewer_eXyy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method for learning prehensile manipulation that positions cubes in a table-top scenario. The method uses a Bayesian state estimator to assess uncertainty in the state and learns a policy that makes use of these state estimates including uncertainty. In comparison to non-probabilistic baselines the method performs superior and can cope with occlusions to the visual camera input.", "summary_of_recommendation": "Good paper. The paper makes a useful contribution and is sufficiently different from the related work, but should include further works and improve the way it contrasts against the related work.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The related work section on non-prehensile manipulation should be revised to more clearly contrast against related work and potentially signify why the switch to eye-in-hand is relevant.\n-- Rephrase: L106 -- collecting real world data is not prohibitive, others have done it! See, e.g.:\n\nBauza, Maria, Ferran Alet, Yen-Chen Lin, Tomás Lozano-Pérez, Leslie P. Kaelbling, Phillip Isola, and Alberto Rodriguez. \"Omnipush: accurate, diverse, real-world dataset of pushing dynamics with rgb-d video.\" In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4265-4272. IEEE, 2019.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Update: The reviewers have addressed important concerns that reviewers have raised. In particular, there was an important point about how privileged policy would have a different state distribution than the non-privileged policy. From what I have understood, the authors address this by keeping the states that were encountered by the intermediate checkpoints when training the privileged policy. I do not think this is the fundamental solution to the state OOD problem, it is an okay answer. I also believe that there several interesting aspects to the paper, such as Bayesian formalization and learning of state estimator, which was not covered in previous approaches.

======================

This paper presents a method for non-prehensile manipulation that is robust to occlusion, via the use of visuotactile sensing.

Overall, reviewers are positive about the hardware experiment, the problem it tackles, and the method it proposes. Some technical points need to be ironed out (i.e. how the privileged policy isn't encouraged to take information-gathering actions, justifications on certain technical decisions, such as why the use of Bayesian framework, etc.).

Several closely related works seem to have gone undiscussed. In addition to the ones reviewer eXyy suggested, I suggest readers to check the following papers:
- CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects
Cho et al., ICLR 2024
- Pre- and Post-Contact Policy Decomposition for Non-Prehensile Manipulation with Sim-to-Real Transfer.
Kim et al., IROS 2023
Decision: Accept