A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation
Authors: Bernardo Aceituno
Alberto Rodriguez
Shubham Tulsiani
Abhinav Gupta
Mustafa Mukadam
Venue: CoRL2021 Poster (2021)
Keywords: Manipulation
Visual learning
Differentiable optimization

Abstract: Specifying tasks with videos is a powerful technique towards acquiring novel and general robot skills. However, reasoning over mechanics and dexterous interactions can make it challenging to scale visual learning for contact-rich manipulation. In this work, we focus on the problem of visual dexterous planar manipulation: given a video of an object in planar motion, find contact-aware robot actions that reproduce the same object motion. We propose a novel learning architecture that combines video decoding neural models with priors from contact mechanics by leveraging differentiable optimization and differentiable simulation. Through extensive simulated experiments, we investigate the interplay between traditional model-based techniques and modern deep learning approaches. We find that our modular and fully differentiable architecture outperforms learning-only methods on unseen objects and motions.

A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation
Bernardo Accituno', Alberto Rodriguez', Shubham Tulsiani", Abhinav Gupta", Mustafa Mukadam
Massachusetts Institute of Technology, "aaceooo Al Research
Specifying tasks with videos is powerful technique towards acquiring novel and general robot skills. However, reasoning over mechanics and dexterous interactions can make challenging scale learning contact-rich manipulation. In this work, we focus on the problem of visual non- non-prehensile planar manipulation: given video of an object in planar motion, find contact-aware robot actions that reproduce the same object motion. We propose novel architecture, Differentiable Learning for Manipulation (DLM), that combines video decoding neural models with priors from contact mechan- ics by leveraging differentiable optimization and finite difference based simulation. Through extensive simulated experiments, we investigate the interplay between traditional model- basse techniques and modern deep learning approaches. We find that our modular and fully differentiable architecture performs better than learning-only methods unseen objects and motions. https://gitth:
Abstract:
com/baceiiee wenceiituo/ddi ddmm.
Keywords: Manipulation, Visual learning, Differentiable optimization
Model
Video
Robot Actions
We tackle the problem of visual non-prehensile planar manipulation where given pre-segmented video of an object in planar motion (left) the goal is to find the robot actions (middle) to reproduce the same object motion (right). In learning for manipulation, study the role of structure, intermediate representations, and end-to-ent differentiation.
Figure 1:
Introduction
Dexterous manipulation tasks can easily described through videos and have the potential to scale robot learning, but decoding relevant information from them remains challenging problem. While recent ap- proaches employing deep learning from images and videos have seen remarkable success various prehen- sile robotic tasks [1. 2, 3, 4], there has been limited progress in the direction of non-prehensile manipulation. An underlying struggle lies in building representations that can reason over object geometry, rigid-bode mechanics, and the combinatorial decisions of choosing contact-modes. Mechanics representations and pri- ors are ubiquitous in model- techniques and thus have been able to demonstrate working solutions [5, 6, 7, 8]. However, these methods be computationally expensive, unable to exploit past experience, and require hand engineering, as they assume access to known object shape, pose, and desired trajectory.
Can bringing mechanics representations and priors into deep learning models aid in decoding videos and learning dexterous manipulation? We study this question on the problem of video-based non-prehen planar manipulation. Here, given video of an object in planar motion in some environment (i.e. input video). the goal is to find contact and force trajectories for the robot fingers (i.e. output robot actions) that when executed will reproduce the same object motion in the same environment. This problem is illustrated in Fig. 1. Our approach is to build intermediate representations based on mechanics that serve as glue in combining upstream neural models that decode the video with downstream differentiable priors that solve for robot actions. We limit the problem scope to planar settings, with the input video being pre-segmented,
5th Conference on Robot Leaming (CoRL. 2021), London, UK.

---

and the robot having point-finges (standard in robot manipulation literature [9, 10, 11, 6, 8D. These simplifications allow us assess the impact of structured learning without the confounding effects of sensor or actuator noise, and enable us to make a reasonable first step towards general approaches that would be able learn dexterous manipulation by watching large scale human videos [12].
We present an architecture, Differentiable Learning for manipulation (DLM) (illustrated in Fig. 2), that works in three stages and can be progressively trained on each stage: (1) a neural model that derenders the input video to mechanical parameters i.e. object shape, object trajectory, discrete contact-mode decisions, and allowable forces between robot fingers and object, (2) a differentiable convex optimization module based on evxpylayers [13, 14] that solves inverse dynamics from the mechanical parameters to find contact and force trajectories for the robot fingers, and finally (3) a differentiable contact-mechanbe simulation module based on lep-physics [15] and finite- diffeerefffngg that executes the robot actions to simulate the object motion. We train the first two stages with supervision from an expert contact trajectory optimizer [6] while the third stage is trained end-to-end over the desired object trajectory.
To investigate the role structure plays in learning, we compare our approach with various architectures including neural models without any mechanics priors on visual non-prehensile planar manipulation tasks. In our experiments, we train the third stage of DLM on dataset that is smaller relative to the training dataset used for the first two stages to resemble a more realistic setting where pre-training is done on a large simulation dataset with easy to obtain supervision and fine-tuning is done on smaller real- real-world dataset that is expensive to collect and harder to label. We find that our structured model is able to outperform other learning-base based architectures on unseen objects and motions, while being more computationally efficient compared to the non-learning expert.
2 Related Work
Prior work in visual manipulation has involved learning predictive models from videos of robot interacting with its environment. These models are used to find actions with simulation roll-outs, in a moeel-predicctvv fashion [16, 17, 1, 18, 19, 20, 21]. These approaches have been successful at solving prehensile manipula- tion problems where the dynamics are hard model. When the dynamics have a known model, while these approaches remove the need to analyze visual data, mechanics or geometry, they struggle in generalizing to different tasks and geometries and can be data inefficient. Our work aims to address this limitation, under the planar non-prehensile manipulation task, by learning mechanical parameters from video and optimizing robot finger actions with a known rigid-body mechanics model, instead of learning a fully predictive model.
On the other side of the spectrum, model- -based techniques leverage the known contact-mechanee that govern the manipulation process for planning or control. The scope of these techniques is often focused in: (i) the planning/contro of primitives [22, 23, 9J, which can be combined sequentially to complete a task [7, 24), or (ii) in the optimization of general skills [25, 5, 6], which solve inverse rigid-body dynamics to solve task. These approaches have the potential to provide a certification of robustness [10, 26] and versatility to operate with different robot kinematics and task specifications. However, they also depend on accurate state-estimation and control to be implemented. Moreover, solving contact mechanics involves non- smooth optimization or large combinatorial search. This prevents the scalability of these methods to complex geometries and large time horizons. The main focus of the tfate-of-the-art art is in planar manipulation tasks over sagittal settings, where friction and contact forces are the primary source of challenges [5, 27, 6, 28).
Recent research has tried to bridge the gap between deep learning and moodel-basee techniques by introducing differentiable computational techniques as part of the learning pipeline. The first relevant line of work involves the introduction of differentiable optimization solvers [29, 30, 13], which introduce a parametric convex optimization program as a differentiable layer in neural network. We leverage one of these solvers [13] to solve quadratic program that takes as input mechanical parameters from a task (derendered from video) and outputs robot finger trajectories within our network. The second relevant line of work comes from the implementation of differentiable contact-mechanies simulators [15, 31J, which execute a parametric physical simulation and allow to back-propagate from a measured output. We modify an existing differentiable simulator [15), using -ifttediiffrrenees to handle intermittent contact dynamics and evaluate the error when exccuting the learned robot finger trajectories as part of a loss function.
3 Visual Non-Prehensile Manipulation Inference
In this section, we define our problem of inferving manipulation actions from a video which serves as task specification. We limit our focus to the context of 2D non-prehensile manipulation given 2D

---

pre-segmented videos of desired object motion. In the discussion section, we elaborate on possible avenues extending our formulation and approach to 3D, real videos, and real robots. We start by defining the main components of our problem, its assumptions, and our notation:
1. Task: a video V with T'' frames showing a sequence of object poses. The tasks involves a set of object poses r(t) E SE),), sampled in I' "time-steps t.
2. Object: a polygonal rigid-body O with mass matrix M and Np facets. Each facet F has corresonnding friction cone FCy-
Action Space: a set of N point-fingers' moving freely around space. We describe the position of finger time-step (t(.). The contact interaction between the object and the finger at Pt(t) applies a force (1)).
Environment: a polygonal environment described as planes with friction cones FCe. The contact interaction between the object and the environment occurs at the contact point P.(() and results in a reaction force (t).
We show high-level representation of our problem in Fig. 1. One of the challenges in solving this problem come from interpreting video data, in order to extract an implicit representation of the object shape, the object motion, and finding the appropriate contact interactions between the robot fingers and the object. Specifically, finding a contact interaction is divided in two steps: (i) select a sequence of contact modes, indicating where each contact is applied at each time-step, and (ii) invert the dynamics resolve the exact contact locations and forces to apply. In this context, inverse dynamics (ID) is an optimization problem that receives an object motion and outputs forces and locations, formulated in the form:
ID:
min P.A
Σ (A
(1)
subject to: MF(t) ((r))-)(r)-J() Pe(t) (()))))F((t) Ee(t)
(2)
where (z) is a cost function that typically penalizes actuation efforts, G(r) represents gravity and inertial effects, and J(r) is a Jacobian mapping contact forces A A- =AIAA,...ANN,A into wrenches. Solving ID, however, assumes knowledge of the exact shape of the object and the contact modes be applied [32, 27, 7], encoded in the facets Fc(() and friction cones (C). Solving this problem without assuming known contact modes is often intractable and a significant body of research has focused on studying it [25, 33, 6]. Our approach addresses this by extracting object shape and contact modes (as facets and friction cones) as parameters through deep neural model, which can be trained with ground-truth labeled data and augmented with self-supervision through differentiable optimization and differentiable simulation.
4 Differentiable Visual Non-Prehensile Manipulation
In this section, we present our approach Differentiable Learning for Manipulation (DLM), Each subsection describes stage of the model and provide technical and implementation details. Our approach leverages deep neural networks, differentiable optimization, and simulation, to construct the full pipeline, illustrated in Fig. 2.
4.1 Mechanical Derendering
One challenge of visual manipulation is to decode geometry and motion from video. While pre-segmented videos can be obtained from realistic videos with state-of-the-ar computer vision tools [34], it is challenging to translate explicit task information into an implicit latent space. This is due to discrete variables such as facets, vertices, and intersections. Hence, the first stage of our model encodes the video frames V into an implicit latent space L" To achieve this, we first encode each frame / through set of convolution layers C.N and combine these embeddings with a Long Short-Term Memory (LSTM) layer, to retain their temporal relation, as:
L - WV-LLS/MONNN()CCNWN)......
(3)
'The use of point- fingers does not account for robot kinematics, While this is a limitation, it is standard approach evaluate planning and leamingg pipelines for manipulation problems (see 28, 10, 26, 6, 8)).
3

---

Neural-bused
-lodel-based
ISTM
Deerendering,
q
Stomiation
P
Dediadd
Olgect Trajactery
Vidao
Lp
Lor
Our proposed fully differentiable pipeline, DLM (Differentiable Learning for Manipulation). In contrast to approaches that represent policies bod-forwwrd neural networks, approach leverages model-basee priors from mechanics to process mechanical parameters via differentiable quadratic program and evaluate the resulting policy through differentiable simulator.
Figure 2:
W
Given this encoding, we find all the parameters required for ID with differentiable optimization, in
similar fashion to [35]. This set of mechanical param- eters [6] includes: (i) the Jacobian matrix (rr) (t), (ii)
FC.
FC
the location of external contacts P. and (iii) param- eters that implicitly encode contact modes F., FC..
and FCc. We derender this set of mechanical parame- ters P= (=((p()))))) (r((),)) (t) (I.).....(C
(t.
Pe(t)
through an MI.2 and the loss function:
de(t
FC
(t) = MM Vt., Lp (p=1(P(()-P())
(4)
F
Pe(t)
(5)
As illustrated in Fig. these parameters are an implicit representation of the finger contact modes and the local object geometry for given task, since the contact mode only encodes which facet in contact and what is the friction cone.
Figure 3: Interactions in our setup (top-left) and the mechanical parameters from the scene (bottom). Differ- contact modes encoded implicitly via the friction cones and the object facet each contact (top-right). Note that the input video only demonstrates the object moving without any robot actions.
4.2 Differentiable Inverse Dynamics
Given the derendered parameters, recast ID the following quadratic program which solves for robot. actions Pa (t).
QP: Po.ee
min
subject to: MF (()-G(r))+e())) E(()) (.()) (a(() (6)
Under this linearization, QP has the property of being a convex optimization problem [36]. This has few benefits: (i) its solution is always the global optima, (ii) if (*()==P(() then its solution is guaranteed to match the ground-truth optima, and (iii) QP can be added as a differentiable layer of our model [29, 13), which inputs p* and outputs the optimal Pa,AE for such parameters. There are few considerations to make QP a differentiable layer. In particular, QP must have a solution-or be feasible- for any choice of parameters (*) [13]. To make the problem always feasible, we relax the dynamics constraints by adding a slackness term e(f) and penalize it in the loss function JQp, with weight After solving QP, we supervise its solution with ground -tuth labels, under known parameters, with the loss
LQp + Pell -xel13:
which drives the model towards a of actions that both consistent with the physical structure, such that e(!)- - the ground-truth physical parameters - Pay>A- After training, we set e = to ensure consistency.
Faaets If are represented by two vertices in the world frame, while friction cones FC are represented with two rays originating the contact point.

---

4.3 Evaluation
A drawback of learning directly from the action space is the dependence on ground-truth finger trajectories to assess the generality of the learned model. The existence of multiple solutions, e-g. as depicted Fig. 4, makes ambiguous to assess model with unseen labeled data. To resolve this am- biguity in the evaluation, we simulate the task using inferred actions PC (t). We then compare the simulated object trajectory with the desired object trajectory, and use this error metric [37]. Algebraically, simulator is represented by
W
r(t+1)=x(0+ATT(0)) ((++)==((++A710)
(t) (tmm(p(tt)
Figure 4: Example of four different valid solu- tions simple manipulation task of pushing a block from left to the right. Since the video does not show what actions to follow, our model may output any of these actions depending on its supervision.
ME()++Gr)=-()))
and:
A N-M.I
is functional representation of the contact forces. These forces need to be determined based on the inertial properties the system. resolve these forces we solve a Linear Complementary Problem (LCP), based on [15], which outputs the corresponding contact forces between the object, fingers, and the environment. These contact forces are activated discretely as:
1,x=0 0,
(lp)())()))000()))
where (( (t)) is the distance between point p (t) and the object at pose r(t). Similar relations are used represent the friction cones and contact modes, which we omit for simplicity. Then, we use the simulation for the evaluation metric using the loss function
Cyim (p)
(10)
where (t) are finger trajectories obtained by solving QP. It is important to note that Luim is not an injective mapping from the finger trajectory error ((f)))p(tt]le Therefore, having actions near the ground-truth might not necessarily lead to a lower Leim-
4.4 Supervision via Simulation
Having access to the loss function Crim can also tune our model subject to ground-truth -rund physics. This reduces (or potentially removes) the burden of having labeled data over the parameters P(l) and the finger- trajectories (4), leading to fully self-supervised approach. Training our model with Zvim requires the function Sim(-) (i.e. the simulator) to be differentiable in all its domain. However, the function 5(.) only returns gradients at the origin. While this is pervasive problem, many researchers have successfully included contact mechanies as differentiable functions by approximating d(-) with smooth function [25]. In our case, we approximate 8(-) as the smooth function:
where (-.k) is a sigmoidal function with factor k. This approximation can be made arbitrarily tight as w->00 oc. With this smoothing, the simulator gradients are computed via finite differences as:
Sim(pp Vr (t) ( (Sim(p) - (t))m,,))/2k
where hn - /13x3. While automatic differentiation a more efficient approach to obtain gradients, we found it incompatible with the smoothed contact model of our current implementation. We leave this enhancement for future work. Minimizing Lysm drives the learned finger trajectories to push the object through the desired object trajectory. A consequence of this approximation is that 5(.) Rs o()) leads to effects such as contact at distance and small penetration, although these are attenuated for large x in practice.
5

---

Mackaskal
Flngr Dugetodt
oajet mug
NN
MDR
DLM
MIQP
Training, fine-tuning and family-test sets Test set
Figure 5: Left: Examples shapes of objects our different datasets. Right: Architectures used to evaluate our framework. Blue segments are stages in the architecture with neural model, black segments are differentiable model-based stages. Dots represent points with labeled data used for pre-training and arrow ends the final loss function for the full architecture. The MIQP approach has privileged access to all the parameters of the problem.
5 Experiments
We implement all models in Py PyTorhh [38] with Python 3. To generate our datasets, we use MATLAB R2020b and solve optimization problems with the Gurobi [39] solver. For all experiments, we use Adam as our network optimizer and CVXPY layers [14, 40] as differentiable solver for QP. In all experiments we train over each loss function 100 iterations with learning rate of 10-4, We implement our simulator on top of the LCP-based scheme in [15] by modifying it to accommodate our problem setting. We run all training, testing, and timing experiments on a Macbook Pro with 2.9 GHz 6-Core Intel CPU.
Datasets: We built four datasets of non- con-prehensile manipulation tasks in the sagittal plane with randomized trajectories (2), all ground truth robot actions are found by solving Contact-Traecctory Optimization (CTO) [6]:
Training set of 20 tasks with randomly generated polygonal objects of 4 6 facets, including all mechanical parameters and robot actions for each task.
Fine Tunnng set of 20 tasks with new random objects of 4 facets, with robot actions for each task.
Test set of 40 tasks with new random objects of 12 facets, with robot actions for each task.
Family Test set of 40 tasks with new random objects of 4-6 facets (same shape and task distribution as training and fine-tuning sets), with robot actions for each task.
Each task has a video of = RGB frames of size 50 50, We find optimal finger trajectories for each task using a Mixed- Mixed-Integer Quadratic Program (MIQP) [6]. for N = 2 point fingers. We set к = 0.5, de 1.=0.1 0.1 and dim( ( = 75. We illustrate example objects we generate for each dataset Fig. (left). Also see Appendix for more details on data generation.
Architectures: We asses the performance of our approach, Differentiable Learning for Manipulation (DLM) against four different architectures and model-based oracle (see Fig. 5).
NN: The Neural Network (NN) architecture replaces QP in DLM, with 3-layer MLP. This is akin to pixel-to--un tocelltt- style policy network [41, 3] and trained to generate MIQP solutions with a loss function N = |Ip (t)(
NNM: The Neural Network Manipulation (NNM) architecture extends NN by also fine-tuning by minimizing Lvins akin the examples used in [15). This network pre-trained with 100 iterations of minimizing Lxv.
MDR: the Mechanical DeRendering (MDR) architecture, DLM is cut off when it trained extract parameters from video by minimizing the loss function Lp. At test time robot actions are obtained by solving QP. This akin the approach used to learn integer solutions to mixed- mixed-integer programs [42, 43].
CVX: In the ConVeX optimization (CVX) architecture, DLM is cut off after being trained by minimizing Cop. This network is pre-trained with 100 iterations of minimizing Lp.
DLM: Our proposed approach is trained by minimizing Lssi after it is pre-trained with 100 iterations of minimizing Lor which in turn is pre-trained with 100 iterations of minimizing Lp.
MIQP (Oracle): A model-based solution to each manipulation task, solved via CTO [6] and provided with all ground-truth parameters (such as object shape and desired object trajectory SE(2)).
side of the seene with gravity pointing down in the image plane.

---

Finger-trajectory loss in training set
Objjet-trajectory loss in training set
Bnd-to-End loss in fine-tuning set
NN
NNM
MDB
M.3
CNX
1. CVX
Finger-tmjortory loss
Object-trajccory loss
End-to- in test set
Parcentage of spochs (96)
Parcontage of opochs (56)
Parcentage spocha (56)
Figure 6: Left: In contrast NN, MDR and CVX closely match the finger-trajectory solution from MIQP (Oracle). Center: While each network is trained to match finger trajectories from MIQP, the more structured networks (MDR and CVX) generalize better to unseen object trajectorics and shapes. Right: Each network is trained by propagating gradients through the simulator. The DLM architecture performs better than the unstructured NNM without over-fitting.
Table 1: Object- Trajectory Loss and computation times (in seconds) for the different architectures each dataset.
Madel
Training
Fine Tuning
Test
Family
Backward Pass (s)
Forward Pass (s)
NN
8.5
N/A
9.5:22
10.0=1.5
0.00
0.06 02
NNM
15.0.15.00
11.0:12.00
1.00
40.00
0.06.10.02
MDR
4.5
N/A
7.5+
4.5:22.0
0.10=0.01
0.11
CVX
2.0+0.5
N/A
4.0+0.3
3.5+0.5
0.23=0000
0.11+0002
DLM
1.5
3.5
3,5
5..00=5.00
0.11
MIQP
0.57
Network details:
The C.) and DeConu(.) operators represent 3 (de)convolution operations with 6, 12, and 24 channels. The (.) function is 5-layer bi-directional recurrent LSTM. Each MLP(.) has 3 fully-connected hidden layers of equal size to their input. All the layers use ReLU activation functions. We add a 0.2 dropout and batch-normalizztion to each fully-connected layer of the network.
5.1 Learning Policies from Data
We first analyze the ability to infer finger trajectories from the ground truth data obtained via CTO.
Finger trajectory loss: We measure the ability of NN, MDR, and CVX methods to learn specific finger actions from the training set and evaluate their generalization unseen data from the test set. We present the results of rumning each network for 100 epochs in Fig. 6 (left). The NN network consistently falls in a local minima. In contrast, the MDR and CVX result in lower training loss closer to the MIQP (Oracle) reference. CVX further refines the learned weights for more accurate predictions, since it is pre-trained with learned weights from MDR.
Simulated object trajectory loss: As we mention in Section IV.D, when we presented new data, the networks struggle to generalize finger trajectories to new object shapes. This is a consequence of each problem having multiple solutions. To resolve this ambiguity, we measure the object-trajectrry loss by simulating the learned actions at each learning iteration. We show the object trajectory loss curves for each of our datasets in Fig. 6 (center). Fig. 6 (center) demonstrates how the inferred actions lead to better execution unseen objects, despite little improvement the enger-rraeectry loss. We note how training with CVX not only refines the loss of MDR, but also prevents over-fitee over-fitting. Examples of the qualitative performance of each model are shown in Fig. 7.
5.2
Learning Policies from Simulation
Next we study the impact of learning actions in a self-supervised fashion. We train DLM and NNM networks on the Fine-tuning set, leveraging the differentiability of our simulator. Both networks are pre-trained with the training set. We plot the resulting object trajectory loss in Fig. (right). The DLM architecture decreases the test loss without overfitting on the Fine-tuning set, while the NNM network oscillates around local minima. This test presents the most similarity a real-world experiment where, without ground-trup labels on the parameters, the network learns to execute a video motion by iterating over experiments. Examples of qualitative performance of DLM model shown in Fig. 7. We show an example of how this approach can be applied to real object, given segmented video, in Fig. 1. Also see Appendix for results on shapes from the OmniPush dataset [44].
5.3 Comparative Analysis
Finally, we compare the final performance of the different networks in terms of object-trajectory loss and computation times (per data-point), shown in Table 1. Inference time in neural model is smaller than
7

---

Input Video
MDR
CVX
DLM
Qualitative examples of each network applied to four trajectories from the family (top two) and test (bottom two) sets. These shapes and motions embedded mechanical structure (MDR, CVX, DLM) tend to perform better than structured ones (NN). DLM pipeline provides layer of supersupeviiion to the model which helps it to generalize better to unseen scenarios.
Figure 7:
during the training phase of each model. Networks with
MIQP (Oracle) [6], as our model involves simple operations and solving QP. We note that training the DLM model is slow, due to the use of finite difference in the backward pass.
6
Discussion
This work explores the problem of visual non non-prehensile planar manipulation, reconciling tools from model-based mechanics with deep learning. Our proposed Differentiable Learning for Manipulation (DLM) approach: (i) encodes the input video V in a latent vector L (ii) derenders mechanical parameters (t) for the task, (iii) solves QP [29] obtain robot finger actions Pt (t((), and (iv) evaluates the performance of the model by simulating these actions. We train this model by minimizing Lp, in order to match the ground-trun -ruth parameters, and by minimizing LQp, in order to match the ground-truft truth actions. Moreover, we can self-supervise this approach with new data by back pack-propagating through the simulator [15] by minimizing Leim, without the need for ground-truth labels on parameters or actions. assess this method by learning how solve planar manipulation tasks given pre-segmented video showing desired object motion. Our experiments suggest that, when compared to fully neural architectures, our approach generalize better unseen tasks and shapes with the same amount of training data.
Limitations: The first limitation of this approach comes from the differentiable simulation scheme. Since our implementation uses finite differences, the slow backward pass hampers the scalability of our model. Similarly, the approximations required make contact-mechane effects, such as rigid-bods penetrations and contact at distance, which can compromise the quality of the results. second limitation arises from our mechanical assumptions. Assuming that contacts occur at set of points disregards tasks involving multiple continuous contacts (e.g. pivoting against wall). Moreover, the representation of the friction cones brings challenges for extending to scenarios, as cannot appropriately represent sliding contact. Finally, since our setup uses unconstrained point fingers, we are unable to represent more realistic robot kinematics.
differentiable also lead undesired
Future work: The next step towards our vision is applying this framework in real-world system, requiring the capability handle real-world video data and robot kinematics. Our vision perform training on completely synthetic data generated with MIQP [6]. fine-tuned for robustness with the simulator, and execute task specified with real-world video. Exploring faster simulation schemes will also be essential to scale this model to larger datasets. Techniques that explicitly resolve the contact- mentacttmecaanics,, through differentiable solvers, can alleviate this issue [15, 31]. Extending this framework to more complex manipulation tasks might require an implicit representation for multiple environmental contacts. Finally, moving to tasks in (3) also possible by solving contact-trajectory optimization [6] in 3D environment [6, 8] also leveraging recent advances in large-scale 3D differentiable simulation [31].

---

Acknowledgments
We thank Felipe de Avila Belbute-Pers for help with the LCP simulator and Hongkai Dai for helpful discussions. Part of this work was done while Bernardo Aceituno was an intern Facebook Al Research. Bemardo Aceituno was supported by the Mathworks Engineering Fellowship.
References
[1] S. Tian, F. Ebert, D. Jayaraman, M. Mudigonda, C. Finn, R. Calandra, and S. Levine. Manipulation by feel: Touch-based control with deep predictive models. In ICRA. IEEE, 2019.
[2] A. Zeng, S. Song. J. Lee, A. Rodriguez, and Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. T-RO, 2020.
[3] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, (9(1),3-0,, 2020.
[4] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, Krasin, D. Duong. V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. CoRL, 2020.
[5] Chavan-Dafle, R. Holladay, and Rodriguez. In-hand manipulation via motion cones. In RSS, 2018.
[6] B. Aceituno- Cabnzas and A. Rodriguez. A global quasi-dynamic model for contact-traeectrry optimization. In RSS, 2020.
[7] F. R. Hogan and A. Rodriguez. Reactive planar non-prchensile manipulation with hybrid model predictive control. IJRR, 2020.
[8] Cheng, E. Huang, Y. Hou, and M. Mason. Contact mode guided sampling- based planning for quasistatic dexterous manipulation in 2d. In ICRA, 2021.
[9] J. Zhou, R. Paolini, J. A. Bagnell, and M. T. Mason. A convex polynomial force-motion model for planar sliding: Identification and application. In ICRA. IEEE, 2016.
[10] S. Sadraddini and R. Tedrake. Sampling-based polytopic trees for approximate optimal control of piecewise affine systems. In ICRA. IEBE, 2019.
[11] N. Fazeli, M. Oller, J. Wu, Z. Wu, B. Tenenbaum, and A. Rodriguez. See, feel, act: Hierarchical leamingg for complex manipulation skills with multisensory fusion. Science Robotics, 2019.
[12] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018
[13] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z.. Kolter. Differentiable convex optimization layers. 2019.
[14] S. Diamond and S. Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83)11-5, 2016.
[15] F. Avila Belbute-Peres, K. Smith, K. Allen, J. Tenenbaum, and J. Z. Kolter. End-to-end differen- tiable physics for learning and control. In NeurlPS, 2018.
[16] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. In NewrIPS, 2016.
[17] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physies. In NeurlPS, 2016.
[18] Kloss, M. Bauza, Wu, J. B. Tenenbaum, Rodriguez, and Bohg. Accurate vision-based manipulation through contact reasoning. In ICRA. IEEE, 2020.

---

[19] Z. Xu, J. Wu, A. Zeng, J. B. Tenenbaum, and S. Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. In RSS, 2019.
[20] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. WAFR, 2020.
[21] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforce- ment learning from visual demonstrations, arXiv preprint arXiv:2010.09034, 2020.
[22] M. Mason. Mechanics of robotic manipulation. 2001.
[23] D. Prattichizzo and J. C. Trinkle. Grasping. In Springer handbook of robotics, pages 955-988. Springer, 2016.
[24] P. S. Schmitt, F. Wirnshofr. K. M. Wurm, and W. Burgard. Planning reactive manipulation in dynamic environments. In IROS. IEEE, 2019.
[25] M. Posa, C. Cantu, and Tedrake. direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, 2014.
[26] B. Aceituno-Cabezas Ballester, and A. Rodriguez. Certified grasping. ISRR, 2019.
[27] N. Chhavan-aafl and A. Rodriguez. Sampling-bast based planning of in-hand manipulation with external pushes. In ISRR. 2017.
[28] Hou and M. T. Mason. Robust execution of contact-rich motion plans by hybrid force-velocity control. In ICRA. IEEE, 2019.
[29] B. Amos and J. Z. Kolter. Optnet: Differentiable optimization layer in neural networks. NeurIPS, 2017.
[30] B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter. Differentiable mpe for end-to-end planning and control. In NeurlPS, 2018.
[31] Y.L. Qiao, J. Liang, V. Koltun, and M. C, Lin. Scalable differentiable physics for learning and control. ICML, 2020.
[32] S. Feng, E. Whitman, X. Xinjilefu, and C. G. Atkeson. Optimization-based full body control for the darpa robotics challenge. Journal of Field Robotics, 2015.
[33] R. Deits, T. Koolen, and R. Tedrake, Lvis: Learning from value function intervals for contact-aware robot controllers. In ICRA. IEEE, 2019.
[34] K. Hc, G. Gkioxari, Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[35] M. Jaques, Burke, and T. Hospedales. mhysiex-asiiveers--aapiii Joint unsupervised learning of objects and physics from video. In ICT.R 2019.
[36] S. Boyd, P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[37] P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum. Simulation as an engine of physical scene understanding. PNAS, 2013.
[38] A. Paszke et Pytorch: imperative style, high-performance deep learning library. NeurIPS. Curran Associates, Inc., 2019.
[39] L. Gurobi Optimization. Gurobi optimizer reference manual, 2020. URL http:://pw/w.t wwwp:w/wurbbicoo com.
[40] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, pages
2019.
[41] P. R. Florence, Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning, pages 373-385, 2018.
10

---

[42] F. Hogan, E. Romo, and A. Rodriguez. Reactive planar manipulation with convex hybrid mpc. In ICRA, 2018.
[43] A. Cauligi, P. Culbertson, B. Stellato, D. Bertsimas, M. Schwager, and Pavone. Learn- ing mixed-integer convex optimization strategies for robot planning and control. arXiv preprint arXiv:2004.03736,, 2020.
[44] M. Bauza, F. Alet, Y-C. Lin, T. Lozano-Pérez, P. Kaelbling, Isola, and A. Rodriguez. Omnipush: accurate, diverse, real-world dataset of pushing dynamics with rgb-d video. In IROS. IEEE, 2019.


Reviews:
[{"review_id": "wLS9FUItWu", "reviewer": "Reviewer_5sxA", "summary": "This submission aims to identify robot actions that involve contacts to manipulate an object given a video of an object moving along a planar motion. To achieve this objective, it proposes an ML pipeline that decodes the video informed by contact mechanics and by using differentiable optimization and simulation. The pipeline has 3 differentiable steps: i) a neural model estimates mechanical parameters such as object shape, trajectory and allowable forces, ii) a convex optimization solver to find contact and force trajectories for the robot fingers, iii) a simulation that executes the robot actions. The first two steps are trained using supervision from a contact trajectory optimizer, while the last step is trained in an end-to-end fashion. The evaluation aims to show that the proposed pipeline outperforms ML approaches without knowledge of mechanics in terms of generalization to new objects and motions. It also argues computational efficiency against non-learning approaches, such as the contact trajectory optimizer. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The combination of machine learning primitives and reasoning about the underlying mechanics, which this work pursues, is appreciated and is reasonable for the target domain. The self-supervision step by using the differentiable simulator and avoid the need for ground truth labels for parameters or actions is also a strength of the proposed framework.\n\nAt the same time, the paper limits the scope to: i) animated planar motions, i.e., the object’s poses exist in SE(2) poses and the visual data are very clear, there are no occlusions due to a manipulator (or human demonstration), and ii) simulated, freely-moving point-fingers. These choices - a toy setup as it is called by the authors - remove sensing or actuation noise and obviously significantly simplify the challenge. The paper mentions that “The main challenges ... come from interpreting video data.. to extract an implicit representation of the object shape and the motion of the task” but one is left wondering how hard of a challenge this is given the animated, clear data available.  For instance, the objects considered also feel quite artificial and disconnected from applications. \n\nThe work is emphasizing at multiple points that the input data do not show the actions performed to achieve the object motion as one of the challenges, since it can result in an ambiguity regarding the set of robot actions that result in the same object motion. But what is the real-world setup that motivates this type of input? One can consider human demonstrations but then the contact points are part of the visual input and the key challenge arises due to occlusions. \n\nThe computational benefits of the proposed approach were not clear as the backwards pass seems to result in significant computational overhead. \n\nThe qualitative results show that there is still a significant gap between the output of the proposed approach and the desired object motion. \n      \nThe authors should consider better highlighting in Figure 5 (right) and in the “Architectures” section the incremental relationship of MDR, CVX and DDM (which ends up being the final proposed approach) as well as NN and NNM.\n\nSome minor issues:\nLine 120: “Hence, The first”\nLine 141: “QP it can be” \nLine  143: “QP must be have”     \nLine 194: “objects objects”                    \nLine 224: “with with”            \nLine 245: “as the our model”                ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is understandable that certain compromises in the realism of the challenge are considered in order to explore this interesting space. The reviewer was ambivalent on whether these compromises in the setup still allow us to be informed about the type of solutions that will address realistic problems or whether the gap is significant enough that this is not possible. Overall, however, it will be beneficial for those working in dexterous manipulation to learn about this differentiable pipeline and what it can achieve even in this simple setup. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Highlighting the challenge of the simplified setup, better supporting some of its choices (e.g., free flying object in input) and where the state-of-the-art is in this domain relative to the contribution will help. \n\nIs there a way to evaluate the impact of the approximations for the QP so that it is differentiable?    "}}, {"review_id": "m-7mrdjfvn", "reviewer": "Reviewer_teuP", "summary": "The authors suggested a new pipeline for learning 2D manipulation tasks. The paper is well written, however, the proposed method is limited to 2D cases. Even those 2D examples in the paper are not complicated. There is only one object and one blue floor. The only interaction between the object and the floor is friction and gravity. There is no comparison between the proposed method and other methods. The authors only compared different versions of their method. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written. The model seems to be promising. \nHowever,\n\n1) it is only evaluated with toy examples. I think that it is not easy to generalize the method for 3D or more complicated tasks.  \n\n2) The method also requires a differentiable simulator because the cost function is based on the simulator. It might work for these 2D examples. But it might not be easy to have a differentiable simulation for more complicated tasks. \n\n3) There is no comparison with other methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the authors are solving some problems which might not be challenging enough as a contribution. The method might not be generalizable to other complicated tasks. I would suggest that authors should prove that the method can be generalized to more complicated tasks.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "some issues:\n1) Is the model (two parts) trained end-to-end or separately?\n\n2) In the first row of Fig.7, two fingers are on one side of the object. The question is why the object can be raised. \n\nThe major problem of the paper is that the authors are working on simple toy examples. According to the authors themselves, the method cannot be easily generalized to 3D tasks. \"Implementing this framework in a real-world system is paramount, requiring the capability to handle real-world video data and robot kinematics\". But it is exactly what we should do.  In literature, people are using CNN+LSTM to handle real-world video data. As the authors mentioned, the mechanical parameters can be quite complicated for 3D tasks. It means that the generalization of the proposed method might be poor. "}}, {"review_id": "_XPEPv82bS4", "reviewer": "Reviewer_XGL2", "summary": "The paper presents a pipeline to go from 2D video input of physical interactions to a set of mechanical parameters that explains the sequence (via a learned LSTM model) and subsequently uses the parameters to optimize for finger manipulation actions and physically simulate resulting object trajectories. A number of ablations and comparisons are conducted with losses imposed on object trajectories, finger trajectories, and/or the mechanical parameters. Compared to purely learned architectures the approach shows favorable performance.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The presented problem is general and highly relevant for robotics.\n- The paper evaluates a lot of variations, ranging from purely learning based pipelines to those that interweave model-based components.\nLimitations section\n\nWeaknesses:\n- The “visual” data part of the method seems a little bit exaggerated. The input data doesn’t seem to have a lot of the characteristics of real data coming from a camera sensor. The input layer could equally be interpreted as an imposed representation rather than a raw measurement.  Is there any reason the approach could not be evaluated e.g. on the OmniPush dataset?\n- In its current form the paper is a little bit demanding when it comes to trying to extract insights/lessons learned. There is a lot to unpack, ranging from loss functions to architecture structures, run-time etc. Maybe re-structuring the Experiment section into a number of tested hypotheses could clarify this.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My main criticism (making the acceptance “weak”) is due to the nature of the data: It is not only limited to the plane but more importantly not based on real images.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- How exactly is the dataset generated? Can this be described in more detail in the appendix?\n- Multiple times the simulator is called “differentiable” when in fact finite differences are used. I’m sure this might technically still be the correct terminology but I think nowadays “differentiable simulation” is most often understood as automatic differentiation. Especially, since the “finite difference” part is only revealed in the limitation section at the end of the paper, this feels slightly misleading.\n"}}]

Author rebuttal:

Meta-review: Strengths:

- The proposed method combines learning and planning aspects in an interesting way. The combination of neural network methods with convex optimization methods was appreciated.

- The problem of learning contact rich manipulation from video is important.

- Comprehensive ablations.

Weaknesses:

- All the reviewers identified the simplistic evaluation as a major weakness, specifically the visually simplistic images and the planar manipulation setting. A successful evaluation of the method using more realistic video would make the work much more impactful (one reviewer suggests the OmniPush dataset).

- It is not clear that the qualitative performance demonstrated in fig 7 warrants the (apparently) significant computational overhead involved here and the overall complexity of the method.

Post-rebuttal:

Significant concerns remain about whether this approach will scale up to moderately realistic manipulation problems. However, the approach is very novel and executed well and is probably of interest to a significant body of researchers in the field.

Decision: Accept (Poster)