Robust Manipulation Primitive Learning via Domain Contraction
Authors: Teng Xue
Amirreza Razmjoo
Suhan Shetty
Sylvain Calinon
Venue: CoRL 2024 (2024)
Keywords: Robust policy learning
Contact-rich manipulation
Sim-to-real
TL;DR: We propose a bi-level approach to learn parameter-conditioned manipulation policies using multiple models and domain contraction.

Abstract: Contact-rich manipulation plays an important role in everyday life, but uncertain parameters pose significant challenges to model-based planning and control. To address this issue, domain adaptation and domain randomization have been proposed to learn robust policies. However, they either lose the generalization ability to diverse instances or perform conservatively due to neglecting instance-specific information. In this paper, we propose a bi-level approach to learn robust manipulation primitives, including parameter-augmented policy learning using multiple models with tensor approximation, and parameter-conditioned policy retrieval through domain contraction. This approach unifies domain randomization and domain adaptation, providing optimal behaviors while keeping generalization ability. We validate the proposed method on three contact-rich manipulation primitives: hitting, pushing, and reorientation. The experimental results showcase the superior performance of our approach in generating robust policies for instances with diverse physical parameters.

Robust Manipulation Primitive Learning via Domain Contraction
Teng Xuel?, Amirreza Razmjoo",, Suhan Shetty!.2 Sylvain Calinon 1.2 'diapp Research Institute "Ecole Polytechnique Fédérale de Lausanne (EPFL)
(teng. xue, amirreza. razejoo,,, suhan. sheaty,, sylvain.calinon) @idnn).diipp.. ch https://sites. http:////.eesggoglee....... com/viex/robustpl
Policy training
Parameter-conditioned policy retrieval
Hit
Push
Reorientation
Figure 1: Overview of the proposed bi-level approach. Left: Parameter-auggmented policy train- ing using multiple models. The state, action, and parameter variables are denoted in black, blue, and red colors, respectively. Right: Parameeer-condittoone policy retrieval through domain contraction. The retrieved policies perform well in terms of both generalization and optimality given a diverse of objects with different shapes, weights, and friction parameters.
Abstract: Contact-rich manipulation plays an important role in human daily ac- tivities, but uncertain parameters pose significant challenges for robots to achieve comparable performance through planning and control. To address this issue, do- main adaptation and domain randomization have been proposed for robust pol- icy learning. However, they either lose the generalization ability across diverse instances or perform conservatively due to neglecting instance-specific informa- tion. In this paper, we propose bi-level approach to learn robust manipulation primitives, including els, and parameter-condiiionee policy retrieval through domain contraction. This approach unifies domain randomization and domain adaptation, providing optimal behaviors while keeping generalization ability. We validate the proposed method three contact-rich manipulation primitives: hitting, pushing, and reorientation. The experimental results showcase the superior performance of our approach in generating robust policies for instances with diverse physical parameters.
augmentte policy learning using multiple mod-
Keywords: Robust policy learning, Contact-rich manipulation, Sim-to-real
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Robot manipulation usually involves multiple different manipulation primitives, such as Push and Pivot, leading to hybrid and long-horizon characteristics. This poses significant challenges to most planning and control approaches. Instead of treating long-horizon manipulation as whole, it can be decomposed into several simple manipulation primitives and then sequenced using PDDL planners [1, 2, 3] or Large Language Models [4, 5J. Although such manipulation primitives usually have low- to-medium --mmedu--dimeniioo dimensional state and action spaces, the breaking and establishment contact make tough for most motion planning techniques. Gradient-based techniques suffer from vanishing gradients when contact breaks, while sampling-base
techniques struggle with the combinatorial complexity of multiple contact modes, i.e., sticking and sliding. This leads to time-consuming online replanning in the real world for contact-rich manipulation, limiting the real-time reactiveness of robots in coping with uncertainties and disturbances.
Learning manipulation primitives that can quickly react to the surroundings, therefore, makes lot of sense. Since the Iearned manipulation primitives will be sequenced by symbolic planners, which have information about the geometric/motion level, the learned manipulation primitive should be robust to diverse instances with varied physical parameters, such as shape, mass, and friction coefficient. For example, once the push primitive is scheduled by the high-level symbolic planner, it should be able to push any objects with different shapes and friction parameters from any initial configurations towards their targets.
p(a)
Domain adaptation
Domain contraction
Domain rendomization
Figure 2: Illustration of DA, DR and DC. a is the physical parameter. p(a:) is the corre- sponding probability. DR conditions on the entire domain, DA a specific parameter (red line), and DC on a small set (blue lines).
Consider how humans perform manipulation tasks: we typically make rough estimates of object and en- vironmental parameters based on physical intuition [6], and our behaviors adapt accordingly. To enable robots achieve similar performance, the learned manipulation primitives should effectively generalize across diverse instances and adapt to the parameters of specific instances. Common techniques for robust policy learning include domain adaptation (DA) [7, 8, 9] and domain random- ization (DR) [10, 11]. DA aims for perfect match between simulation and reality but may sacrifice generalization ability, while DR generalizes well but lacks full knowledge of the target domain, leading to conservative behaviors. Combining DR and DA [11, 12] harness the benefits of both sensible strategy, but how to effectively balance them is still an open question. In this work, propose bi-level approach to address this challenge:
Level-1: Parameter-augmentte policy learning using multiple models. We augment the state space with physical parameters of multiple models and use Tensor Train (TT) to approximate the state-value function and advantage function.
Level-2: Parameter-conditionel policy retrieval through domain contraction. At the stage of execution, we can obtain rough estimate of the physical parameters in the manipulation domain. This instance-specifee information can be utilized to retrieve parameter-conditioned policy, ex- pected to be much more instance-optimal.
We summarize our contributions follows:
1) We propose domain contraction, a unification of domain adaptation and domain randomization, which results in optimal behavior while maintaining generalization.
2) We propose using tensor approximation for robust policy learning with multiple models and leveraging products of tensor cores for parameter-conditoons policy retrieval.
3) We provide both theoretical proof and numerical comparisons for the proposed approach.

---

2 Related Work
Manipulation policy learning. Many approaches have been proposed to learn manipulation poli- cies for mor-horizon manipulation, including Behavior Cloning (BC) [13, 14], Deep Reinforcement Learning (DRL) [15, 12, 16). and Approximate Dynamic Programming (ADP) [17, 18]. In our work, to ensure subsequent parameter-conditiona policy retrieval, it is essential to explore the full parameter domain. The reliance on the provided dataset in BC limits its ability to explore the broad state space and learn the full parameter-augee auggeetedd policy. DRL offers better exploration capability compared BC, but its policy training is usually time-consuming due to sample inefficiency, and policy retrieval is often suboptimal due to gradient-based optimization and local exploration. Con- versely, ADP aims to cover the full state space but faces the curse of dimensionality. Recently, new technique called Tensor Train Policy Iteration (TTPI) [19] has emerged to address this challenge by employing tensor approximation. Our work builds upon TTPI, extending it for robust primitive learning by augmenting the state space with model parameters.
Sim-to-real transfer. Obtaining large amounts of real-world data for primitive learning is challeng- ing. Therefore, robot learning in simulation and transferring to the real world is promising idea [20]. However, the reality gap between simulation and the real world poses a significant challenge to such idea. If the target domain is known and specific, Domain Adaptation (DA) [7, 8, 9] can be an effective method, but its reliance on domain-specific data can limit generalization to other scenarios without additional fine-tuning, On the other hand, Domain Randomization (DR) [21] seeks to de- velop a robust policy by introducing random variations into the simulation parameters. While this method offers good generalization, it often results in suboptimal and high-variance behaviors due to the restrictive assumptions about environment parameter distribution (e.g., normal or uniform). To overcome this, combining DA and DR can be promising strategy to balance generalization and optimal performance. The basic idea is to adapt the parameter distribution by leveraging differences between simulated and reference environment [10, 11, 22]. However, policies developed through such methods are often tailored to the reference environment or target domain. Recently, a method called Rapid Motor Adaptation (RMA) [23] has demonstrated remarkable success in learning ro- bust locomotion [16. 24] and manipulation primitives [12]. In this approach, a teacher policy is trained in simulation with various domain parameters, and student policy learns to replicate this behavior, with low-dimensional embedding of proprioceptive history as input. Our work closely follows this paradigm but introduces a more efficient way for learning the parameter-augmentt teacher policy. Additionally, our method eliminates the need to learn a student policy afterward, as the parameter-conditeee
robust policy can be easily retrieved from the teacher policy. also offers a flexible way to leverage domain knowledge, making it effective in both closed-loop and open-loop scenarios, whereas RMA struggle in open-loop scenarios due to the lack of privileged information.
Tensor Train for function approximation. A multidimensional function can be approximated by tensor, where each element in the tensor is the value of the function given the discretized inputs. The continuous value of the function can then be obtained by interpolating among tensor elements. However, storing the full tensor for a high-dimensional function can be challenging. To address this issue, Tensor Train (TT) was proposed to approximate the tensor using several third- order cores. The widely used methods include TT-SVD [25] and TT-cross [26]. Furthermore, TTGO [27] was proposed for finding globally optimal solutions given functions in TT format. Thanks to the low-rank embeddings of the original function, this method significantly enhances computation efficiency and reduces the risk of getting trapped in local optima. TTPI [19] was then introduced to learn control policies through tensor approximation, showing superior performance on several hybrid control problems. Logic-Skill Programming (LSP) [2] expands the operational space of TTPI by incorporating first-order logic to sequence policies. Building on TTGO and TTPI, our work extends these methods to learn robust policies, which further enhances the capabilities of LSP. We additionally demonstrate that the TT format is suitable structure for domain contraction, allowing for efficient parameter-comnitionee policy retrieval.
3

---

parameter distribution
P:log) A
paloy Rase
parameter
state
action
Figure 3: Overview of domain contraction in TT format. Using TTPI, we can obtain the parameter- augmented advantage function in TT format. It includes separate 3rd-order cores for different di- mensionality, such as parameter, state and action. In this figure, we demonstrate the advantage function for Hit primitíve. Given the parameter distribution (either by human knowledge or by system identification), we can retrieve the parameter-condittonel policy by making product of pa- rameter distributions and corresponding TT cores.
Method
3.1 Problem formulation
Consider discrete-time dynamical system defined by:
Et+1 ~ f(x,, uz)a), Ue ~ (a) a~ p(a),
(1)
where a, E P. and u, E Pu represent the state and action at time step I, respectively. The dynamics model f O2 x S4 x Qa + Ω2 is conditioned on the domain (or environment) parameters a E Qa (such as masses, shapes, or friction coefficients). These parameters are assumed to be random variables satisfying an unknown probability distribution p. Together with the reward function R P2 Q2 I and the discount factor 7 [0, 1], this system forms Markov Decision Process (MDP) M {Sz, Pu, Na. P, R.
In this work, we aim to find the optimal policy that maximizes the expected cumulative reward for distribution domain parameters a ~ p(a)), namely
V(zaa) -amp(a)
E Σ
2' R(x, u.) Tg -a-=)
(2)
A(x, u/) R(a, u) (V ula)) V(ala)),
Tx) arg max A(x, u/aa). uEf,
To learn such parameter-conditionee policies, it impractical to train them individually since there can be infinite instances for one single primitive. Instead, we propose to learn a parameter- augmented full policy first and then retrieve the parameter-condiitoond optimal policy for specific domain at runtime.
3.2 Robust policy learning through domain contraction
We define domain contraction as retrieving the parameter-condittoned policy from the weighted sum of parameter-specific advantage functions. The theoretical proof is provided Sec. 3.3. These functions are learned jointly by augmenting the state space with parameters, similar to the multi-gon setting in DRL. [28], resulting parameter-augmenteee advantage function.
TTPI [19] is used to approximate the parameter-augmenter value function
V(a x)
V(a Lim) V
Dd yd+1 dmm
idaa idetts
(3)

---

and the parameter-augmentt advantage function
A(a, 2, u) RJ (('idd 1 11m)
(4)
A .h
A
d+1
d+m
A
ddm--1 stam+t1
A ddm+n
mi
TT format. We define the tensor cores related and u as A 1imm U1m)
A
tm fitmm
adm++
Adtmtn fdommto."
(5)
Without loss of generality, we assume each subspace Qo. of the parameter space Qa Ga. X X.... x Das is discretized by N; points. The ceanneter-speefftee advantage function can then be extracted as
Aa, u) = A(rim Wim ax)) = A JJs: A A(zi:m Wi:n),
(6)
where a, = (a (ga) represents a at the discretization index j across all dimensions.
We define Py = pi (2( ... pa(aj as the probability of az and pt the probability dis- tribution at dimension i. Given the TT approximation P (it---ja) = P rough parameter distribution Py. we can then compute the parameter-conditione advantage function by computing the weighted sum of parameter-speciffe advantage functions in a separable form
: P......PP.. P.j. P of the
N₂
Na
A(x, a =
Σ
As (x. u)
===
3a=1
N pd
ddai A.j.i
A(21.m U1nn)
11=1
The computation of weighted sum is at the tensor core level, which is much more efficient than at the function level. After obtaining the parameter-conditioned advantage function, we can retrieve parameter--oonditoone policy using TTGO [27]. specialized method for efficiently finding globally optimal solutions given functions in TT format.
To retrieve such coannetter-coditioneed policies, knowing the parameter distribution is crucial. DR and DA rely on assumed distributions during training, while DC offers better way to leverage domain knowledge during execution, enabling optimal behaviors while preserving generalization capability. In Section 3.3, we demonstrate that DR and DA are two special cases of DC.
3.3 Theoretical proof
Theorem 1. Given domain parameters and the distribution p. the parameter-condttione policy can be retrieved from the weighted sum of paramfter-speciffc advantage functions.
Proof.
Given that each subspace Dap of the parameter space Q2 is discretized by N; points, the parameter-conditioned policy can be written as
N1
N Ps
A(æ, ula) = R(x, u) +
Σ
Σ J==1
-
===1
(8)
N;
Na
[(x, u) (V -
===1
j==1
and the parameter-specific advantage function defined as
Any (xz, un) R(x, u) + 7 (Zz, (aoorr)))V(((aaaaa V(xlaa;))
(9)
Given (8) and (9), we can derive that the parameter-conddtioned advantage function is the weighted sum of parameter-specifii advantage functions, namely
A(x, ula)
Σ
pj Aay (x, u),
(10)
5

---

and the parameter-conditione primitive policy can then be computed by
π <
arg max w/)). u
(1I)
Note that the parameter-conditioned policy cannot directly computed the weighted sum of parameter-specific policies, since the arg max operation does have an associative property with respect to addition.
Theorem 2. Domain randomization and domain adaptation are two special cases of domain traction.
Proof. The policies of DR, DA and DC are obtained by
N
Na
N2 Dj (x,,
(x) arg max MEfu
Pj Aa, (x. u),
TD) = arg max uEll
Σ Ja=1
u),
d1
31
TDC(2)) arg max wE
Pj Ao, u.)
where Pj. P3 and Py are the probabilities of the parameter instance a at index j. satisfying the distributions P. P and p. which are used for policy retrieval in DR, DA and DC, respectively.
Note that the policy training level, have no information about the real distribution p". DR assumes p* uniform or Gaussian distribution (which can be wrong assumption) for policy training. DA aims for closed match between simulation and the target domain by using partly the real data, resulting in P = p" Conversely, DC enables the use of domain knowledge during the execution stage, where instance-specific information becomes available. We assume the estimated rough parameter distribution as p. If p matches p", the policy aligns with DA: no instance knowl- edge present, p equals p, resulting in DR-like policy. DC therefore unifies DA and DR, while allowing flexible compromise between these two extreme cases.
4 Experimental Results
We validate the effectiveness of the proposed method on three contact-rich manipulation tasks: Hit, Push, and Reorientation, as shown in Fig. 1. All the primitives are highly dependent on the physical parameters between the object, the robot, and the surroundings. The environmental details, including the task setup and implementation details, are presented in the appendix.
4.1 Robust policy learning and retrieval
We first learn the aeaameter-augmenter policies using multiple models through tensor approxima- tion. The physical equation of Hit fully known, analytically. For Push and Reorientation. TTPI is used to approximate the parameter-augmenttd value functions and advantage functions.
therefore compute the control policy
Hit: Given the mass m, the friction coefficient μi the initial state xy and the target des, the advantage function is
A(x, I) (||æ - des 1|2 0
(12)
where 20g + I 0.5 Computing the impact I is find the value that maximizes (12).
Push: The reward function for Push primitive learning is defined as
r - (pcp + + 0.01 + 0.01c,);
(13)

---

0.040
state error
0.033
state error
0.03
0,3
0.030
0.00
0.2
6.1
0.01
0.0
0.00
X220
105
9
W
w-
1620
w-
(a) Hit
(h) Push
(c) Reorientation
Figure 4: Comparison of final state error given different estimated parameter distributions
with
-p= 12p 2 11/pp Co 66=1|ag 16-
cy = lffll. c. -1ppl:
(14)
where Tp = [82, sy] and Ho 0 denote the object's position and orientation. Without loss of generality, we set des = 0 as the target configuration. f the control force, while up is the velocity of the robot's end-effector. lp and lo are set to 0.005 and 0.01 respectively.
Reorientation: The reward function for Reorientation primitive learning defined as
= - (Bcg + ccy,
(15)
with Co = ||_ - Cj ll 2, is the orientation angle 9, and is the normal force between the gripper and object. xges is set to as the reorientation goal, and 3 is set to 10
We then retrieve the parameter-conditione policy through domain contraction. Note that any dis- tribution of the parameters is allowed in our framework. Without loss of generality, we assume the parameters satisfy uniform distribution, within range the discretization indices of each dimen- sionality (denoted as w), shown in Table and Fig. 4. w means we know the exact value of the physical parameter, corresponding domain adaptation. In contrast, N means have prior knowledge about the model parameters, corresponding to domain randomization.
Table and Fig. demonstrate the comparisons of cumulative reward and final state error, respectively. The state error is quantified as the L2 norm of the difference between the final state and the target state. The cumulative reward is normalized us- ing the value obtained through do- main adaptation. We can observe that Hit primitive depends more on the accuracy of model pa- rameters, as resembles an open-loop control. Once the impact given from the robot to the object, there is no way to adjust control inputs influence the object movements further. How- the comparison indicates, there is need to have precise parameter estimation. A rough range (w N/20) is sufficient to achieve the target. In contrast, Push and Reorientation can be considered more akin to closed-loop control. Therefore, the requirement for accurate parame- ter estimation can be relaxed further. As depicted Table and Fig. 4, a rough distribution with w = N/5 is adequate. Moreover, based on Table we observe that w = N results in the lowest cumulative reward. This is consistent with our assertion that domain randomization typically leads conservative behaviors. Although domain adaptation (w 1) yields the highest cumulative ward, obtaining precise parameter values be challenging in the real world. Domain contraction bridges the gap between domain adaptation and domain randomization, offering greater flexibility to generate optimal behaviors while utilizing instance-specio rough parameter distribution. This is much practical for real-world contact-rich manipulation tasks.
Table 1: Cumulative reward of three manipulation tasks
w =
w = N/20
w = N/5
w = N
Hit
1.0
0.65
0.21
0.01
0.01
0.02 1 0.05
Push
1,0
0.99 0.01
0.99 0.03
0.93 0.11
Reori.
0.99 0.04
0.99 0.07
0.85 + 0.19
4.2 Comparison to other primitive learning methods
7

---

Table 2: Comparison of robust primitive learning approaches (time in [s]econd or [m]inute)
RL+DR
RMA
Ours
time
time
teacher _erorr
student_etror
time
teacher_error
DC_error
Hit
5,76s
0.633 I U.53
4.366
0.036 I 0.030
NA
0,206
0,011 T 0.007
0.015 I 0,010
Push
41 m1
0.061 +: 0.003
54.72m
0.053 +: 0.103
0.039 + 0.126
5.83m
0.035 + 0.023
0.036 0.024
Reori.
21.78m
0.034 0.026
23.23m
0.086 0.065
0.104 0.087
1.32m
0.010
0.014
0.011 0.014
Moreover, we compared our method with two widely used robust policy learning approaches: rein- forcement learning with domain randomization (RL+DR) [21] and rapid motor adaptation (RMA) [23, 12]. The quantitative results are presented in Table including the time required for policy training and the final state error. Note that in RMA, teacher error and student_ _eroor correspond to the errors of the teacher policy given the true parameter and the student policy, respectively, while in our method, teacher error and DC error correspond to the errors of the parameter-augme augmented policy given the true parameter and the parameter-conditionee policy obtained through domain con- traction (DC). Our method requires significantly less time for policy training and results in much lower final state error compared to the other methods. Furthermore, when comparing the error of the teacher policy in RMA with our parameter- -uggeette policy, our method shows better accu- racy, highlighting its potential one-shot manipulation tasks where only action be executed (as shown in the Hit primitive in the table), RMA does not work due to the lack of privileged information.
teacher policy in the general RMA framework. Additionally, in
4.3 Real-robot experiments: planar push
We validated our proposed method for the planar pushing task using a 7-axis Franka robot and a RealSense D435 camera. The manipulated objects included a sugar box and a bleach cleanser from the YCB dataset [29], each with different shapes and masses, as shown in Fig. The friction coefficients between the objects and the table were varied by using metal surface and plywood, respectively. Note that it is casier control the robot kinematically rather than using force control. We leverage the ellipsoidal limit surface to convert the applied force to velocity, resulting in the motion equations shown in [30, 31]. We trained a purameter-augmented policy in simulation and then applied in the real world through domain contraction. The experimental results showcase the effectiveness of the obtained parameter-conditione policies in manipulating instances with di- verse parameters. Additionally, external disturbances were introduced by humans, demonstrating the reactiveness of the retrieved policy. Additional results are presented in the accompanying video.
5 Conclusion and Future Work
In this paper, propose bi-level approach learning robust manipulation primitives. Multiple models and tensor approximation are used to train the parameter-augmentt policy, which can be directly applied to diverse instances through domain contraction, outputting instance-dependent op- timal behaviors. Theoretical proof and numerical results demonstrate the efficiency of the proposed method for robust primitive learning. Real-world experiments further validate its effectiveness in contact-rich manipulation under uncertainty and disturbance.
Since we leverage TTPI to learn the assumes low-rank structure of the value function and advantage function. Therefore, the state space should have to-tmdiumm dimensionality (approximately less than 35), limiting its application for image-based policy learning. In the future, this limitation can be addressed by combining TT with neural networks in a data-driven manner [32, 33].
policy, we also adopt its limitations. TTPI
Moreover, assume that rough distributions of uncertain physical parameters are provided by hu- man knowledge. This requirement can be eliminated by integrating Large Visual Language Models or system identification into the framework. As shown in experiments, the parameter estimation does not need to be very precise, thanks to the proposed domain contraction technique. Additionally, the policy can be iteratively improved as the belief about parameter estimation is updated.

---

Acknowledgments
This work was supported by the China Scholarship Council (grant No.202106230104 and by the SWITCH project fattps://switch-pprgt .iitgggtthbbb .io)), funded by the Swiss National Sci- ence Foundation. We thank Jiacheng Qiu for suggestions about the implementation of RL basclines.
References
[1] C. Aeronautiques, A. Howe, C. Knoblock, D. McDermott, A. Ram, M. Veloso, D. Weld, D. W. Sri, A. Barrett, Christianson, et al. PDDL- the planning domain definition language. Technical Report, Tech. Rep., 1998.
[2] T. Xue, A. Razmjoo, S. Shetty, and S. Calinon. Logic-Skill Programming: An Optimization- based Approach to Sequential Skill Planning. In Proc. Robotics: Science and Systems (RSS), 2024.
[3] League: Guided skill learning and abstraction for long-horizon manipulation. IEEE Robotics and Automation Letters, 8(10):645166458 2023.
[4] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pages 8469-8488. PMLR, 2023.
[5] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, Jang. R. Julian, al. Do can, not as say: Grounding language in robotic affordances. In Conference robot learning, pages 287-318. PMLR, 2023.
[6] O. M. Groth. Learning physical intuition for robotic manipulation. PhD thesis, University of Oxford, 2021.
[7] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, Pastor, K. Konolige, al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), pages 2018
[8] K. Arndt, M. Hazara, A. Ghadirzadeh, and V. Kyrki. Meta reinforcement learning for sim-to- real domain adaptation. In Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), pages 2725 2731, 2020.
[9] Y. Chebotar, Handa, V. Makoviychuk, M. Macklin, Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), pages 8973-8979, 2019.
[10] B. Mchta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1176. PMLR, 2020.
[11] Muratore, T. Gruner, Wiese, B. Belousov, M. Gienger, and J. Peters. Neural posterior domain randomization. In Conference on Robot Learning, pages 1542-1542. PMLR, 2022.
[12] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, pages 1722-1732. PMLR, 2023.
[13] Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L.. Downs, A. Wong, J. Lee, I. Mor- datch, and Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158 158. PMLR, 2022.
[14] F. Torabi, G. Wameell, and P. Stone. Behavioral cloning from observation. In Proc. Intl Joint Conf. Artificial Intelligence (ICCAI), pages 4950-4957, 2018.

---

[15] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188-204. PMLR, 2021.
[16] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47):cabc5986, 2020.
[17] W. B. Powell. Approximate Dynamic Programming: Solving the curses of dimensionality, volume 703. John Wiley & Sons, 2007.
[18] P. Werbos. Approximate dynamic programming for real-time control and neural modeling. Handbook of intelligent control, 1992.
[19] S. Shetty, T. Xue, and S. Calinon. Generalized policy iteration using tensor approximation for hybrid control. In Proc. Intl Conf. Learning Representations (ICLR), 2024.
[20] X. B. Peng, M. Andrychowiez, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In Proc. IEEE Intl Conf. Robotics and Automation (ICRA), pages 3803-3810 10, 2018.
[21] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In Proc. IEEE/RSI Inil Conf. on Intelligent Robots and Systems (IROS), pages 23-30, 2017.
[22] F. Ramos, R. C. Possas, and D. Fox. Bayessim: adaptive domain randomization via probabilis- tic inference for robotics simulators. In Proc. Robotics: Science and Systems (RSS), 2019.
[23] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy with online system identification. In Proc. Robotics: Science and Systems (RSS), 2017.
[24] A. Kumar, Z. Fu, D. Pathak, and Malik. RMA: Rapid motor adaptation for legged robots. In Science and Systems (RSS), 2021.
Proc. Roboties:
[25] I. V. Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):
2295-22 2217, 2011.
[26] Oseledets and E. Tyrtyshnikov. Ti-cross approximation for multidimensional arrays. Linear Algebra and its Applications, 432(():70-88,, 2010.
[27] S. Shetty, T. Lembono, T. Löw, and S. Calinon. Tensor train for global optimization problems in robotics. International Journal of Robotics Research (IJRR), 43(6):811 8118839, 2024. doi: 10. 10.1177/07836692211177777
[28] M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In Proc. Intl Joint Conf. on Artificial Intelligence (IJCAI), 2022.
[29] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[30] R. Hogan and A. Rodriguez. Feedback control of the pusher-slider system: A story of hybrid and underactuated contact dynamics. In Algorithmic Foundations of Roboties XII: Proceedings of the Twelfth Workshop on the Algorithmic Foundations of Robotics, pages 800-815. Springer, 2020.
[31] T. Xue, H. Girgin, S. Lembono, and S. Calinon. Demonstration-guidee optimal control for long-term non-peehe planar manipulation. In Proc. IEEE Intl Conf. Robotics and Automation (ICRA), pages 4999-5005, 2023.
[32] S. Dolgov, D. Kalise, and L. Saluzzi. Data-driven tensor train gradient cross approximation for jamilton-jacoii-elllee equations. SIAM Journal Scientific Computing, 45(5):A2153- A2184, 2023,
10

---

[33] S. N. Shetty. Robot Learning using Tensor Networks. PhD thesis, EPFL, Lausanne, 2024.
URL. btattpss//infsseeee ence. www..ppcf..aaa.eee wwwpaanlee/0..5000000 14299/208211
[34] I. Oseledets and E. Tyrtyshnikov. TT-cross approximation for multidimensional arrays. Linear Algebra and its Applications, 432(():70-88, 2010.
[35] D. Savostyanov and I. Oseledets. Fast adaptive interpolation of multi-dimennional arrays in tensor train format. The 2011 International Workshop on Multidimensional (nD) Systems, pages 1-8, 2011.
[36] J. Zhou, R. Paolini, J. A. Bagnell, and M. T. Mason. A convex polynomial force-motion model for planar sliding: Identification and application. In Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), pages 372-377. 2016.
[37] F. R. Hogan and A. Rodriguez. Reactive planar non non-prehensile manipulation with hybrid model predictive control. International Journal of Robotics Research (IJRR), 39(7):755-775, 2020.
[38] F. Viña Barrientos, Y. Karayiannidis, C. Smith, and D. Kragic. Adaptive control for pivoting with visual and tactile feedback. In IEEE International Conference on Robotics and Automa- tion, Stockholm, Sweden 16-21 May 2016. Institute of Electrical and Electronics Engineers (IEEE), 2016
[39] T. Haarnjja. A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[40] A. Raffin, A. Hill, Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. URL http://jmml wwgppapprrs/22222011220 1364. html.

---

A Background of Tensor Train
A.1 Tensors as Discrete Analogue of Function
A multivariate function P(æ defined over a rectangular domain constructed with the Cartesian product of intervals (or discrete sets) I x ... x Ia can be discretized by evaluating it de I E Ix, E mu.).., This gives us ten- sor P. discrete version of P. where P (G)....AA)) Ix {(d1....,.d.
at points in the set X' = {(=''....,,."" x'd))
n43).
P(r ....,,,,, xx v(i dd) Ix, and is {1...., m. {l.....d]} The value of P at any point in the domain can then be approximated by interpolating between the elements of the tensor P.
A.2 Tensor Networks and Tensor Train Decomposition
Naively approximating a high-dimensionee function using tensor is intractable due the combi- natorial and storage complexities of the tensor (O(n))). Tensor networks mitigate the storage issue by decomposing the tensor into factors with fewer elements, akin to using Singular Value Decom- position (SVD) to represent large matrix. In this paper, we explore the use of Tensor Train (TT), type of Tensor Network that represents high-dimensional tensor using several third-order tensors called cores, as shown in Fig.
2D
PL
Figure TT decomposition generalizes matrix decomposition techniques to higher- dimensional arrays. In TT format, an ele- ment in a tensor can be obtained by multi- plying specific slices the core tensors. The figure presents examples of second-order, third-order, and fourth-order tensors. Image adapted from [27].
P P.
P!
P SAx
TE
p2
3D
P Panists
P! SH12
P Mal
P
Xis Kagkeng
P
R Xauier
X
P PebAN
P
P.
PI
We can access the element (f.,...,ia) id) of the tensor in this format simply given by multiplying matrix slices from the cores:
P P(i .. P Lia.
pd Saaa
(16)
where Pk. S. E - represents the 4g frontal slice (a matrix) of the third-order tensor pk For any given tensor, there always exists a TT decomposition [25]. This low-rank structure further facilitates sampling and optimization for robot planning and control.
There are several ways to acquire TT model, including TT-SVD [25] and TT-Cross [34, 35]. TT- SVD extends the SVD decomposition from matrix level to nigh-dimensional tensor level. How- ever, it needs store the full tensor first, which is impractical for high-dimensional functions. TT- Cross solves this issue by selectively evaluating the function P on a subset of elements, avoiding the need to store the entire tensor.
A.3 Function approximation using Tensor Train
Given the discrete analogue tensor P of function P, we obtain the continuous approximation by spline-based interpolation of the TT cores corresponding the continuous variables only. For example, we linear interpolation for the cores (i.e., between the matrix slices of the core) and define valuex function corresponding to each core A: (1.....d).
p p**xx)=
Fh
P
Fk pk
(17)
in+1 5N
k
11:
to
x
12

---

where zx I I 2t continuous approximation of P given by
=x and P Is C R + R 18 with rd = 1. This induces
P(æ....., Fa) R P (x1) (=a))
(18)
This allows us to selectively do the interpolation only for the cores corresponding to continuous variables. and hence we can represent functions in TT format whose variables could be mix of continuous and discrete elements.
A.4 Global Optimization using Tensor Train (TTGO)
In optimization problems involving task parameters æ and decision variables u, the goal is to find the optimal u that minimizes the cost function c(x, u). TTGO [27] frames this problem as maximizing an unnormalized probability density function (PDF) P(x, u), which is derived from c(z, u) through a monotonically non-increasing transformation. For example, P(æ, u) can be defined as -0((,,)) with 3 >
The TT-Cross algorithm is then used to compute the discrete analogue approximation of the un- normalized PDF, i.e., P, in the TT format. After approximating the joint distribution, pär can be obtained by conditioning on the given task parameter æ = Ωze
Given the TT model pa: the optimal u is obtained through iterative sampling, which determines the optimal solution for each dimension at the core level. Due to the low-rank nature of the TT model, this sampling process is highly efficient. A number of prioritized samples, N 2 1, are selected, and the sample(s) with the highest density (or lowest cost) are chosen as candidate solutions. These near-optimal solutions can then be further refined using local optimization methods, such as Newton- type optimization for continuous variables. Overall, this technique can efficiently find the globally optimal solution given any low-rank function, without requiring a convex structure. Moreover, the computation process is gradient-free and can handle a mix of continuous and discrete variables.
A.5 Generalized Policy Iteration using Tensor Train (TTPI)
Optimal control of dynamic systems with nonlinear dynamics presents a significant challenge in robotics. To address this, Generalized Policy Iteration using Tensor Train (TTPI) was proposed, leveraging tensor approximation and approximate dynamic programming [19]. This method ap- proximates state-value and advantage functions using Tensor Train (TT), effectively mitigating the curse of dimensionality. The low-rank structure of TT enables the use of TTGO to find near-global solutions during policy retrieval from the advantage function, even under complex nonlinear system constraints, surpassing the capabilities of existing neural network-based algorithms [33]. This ap- proach does not require knowledge the dynamics model; black-box simulator suffices, akin to model-free reinforcement learning. Shetty al. [19] demonstrated TTPI's superior performance on several hybrid control problems compared to state-of- the-art hybrid RL algorithms. In this paper, we apply this approach to learning manipulation primitives for contact-rich tasks involving many contact parameters. The typical TTPI algorithm cannot cope with the diverse contact-rich instances. Therefore, we propose domain contraction to retrieve the parameter-conditionel policy that can achieve robust manipulation.
B Domain information for each manipulation primitive Hit
B.1
Hitting is widely used manipulate objects through impact. In this work, we focus on planar hitting primitive. The state is the object position, denoted [z, vl. The control input is the applied impact I = [T,, Ty]- The physical parameters include the object mass m and friction coefficient μ. The motion equation is
des
2o + m
ugt2.
(19)
13

---

Given the initial state and the target ædes, the advantage function is
A(x, I) = - 1 0.01
(20)
We applied TT to approximate the advantage function A(x, I), and the correct impact is computed by maximizing (20).
B.2 Push
Pushing is challenging for robot planning and control due to its hybrid and under- actuated nature. The state is characterized by [Sa, Sy. sg, 1b, ø), and the action is denoted as [fz. fy. 1h ø]. Here, Sy: E S de- notes the position and orientation of the object the world frame. z is the relative angle the contact point in the object frame. represents the distance between the contact point and the object surface. exerted on the object, while Up = [&), Φ) rep- resents the angular and translational velocities of the robot's end-effector. The physical pa- rameters include the object mass m, radius r, and the friction coefficient between the object and table.
(sz: Sys Sp)
[Sz, Sy] are the forces
W
Figure 6: Illustration of pushing dynamics.
The applied force on this object can be mapped to its resulting velocity through a convex limit surface convex approximation [36], resulting in a sub-level set
H (ww) = 2" w Lww,
(21)
where = diag(fmis Smax: mmxx) with fmax as the maximum friction force between object and table, and Maax as the maximum torsional friction.
The robot dynamics is defined based the Quasi-Static approximation and the limit surface, result- ing in similar expression [37], namely
Rt
RLw Up
(22)
Up
with
cos 0
sin 0
R =
sin 0
cos 0
0
(23)
1
0
f
w
J
f
0 Py
1
f
(24)
Psl
where R is the rotation matrix and ur denotes the applied pusher wrench. J is the Jacobian matrix of the contact point the body frame. The contact position Pyl, in the object frame can be computed by
Ps = (r(b) + Φ) cos(v), Py = (r(1) + o) sin(v),
(25)
for any shape that can be parameterized radially with the radial distance described as r(v).
14

---

We parameterize the object shape using a concatenation of Bézier curves, with the weight matrix C defined as
1100
11111
C
(26)
0
11
[
0
0 is repeated for each junction of two consecutive Bézier curves. For two
where the pattern
0
concatenated cubic Bézier curves, each composed of Bernstein basis functions, we can see locally that this operator yields a constraint of the form
Wt
u/4
0
(27)
W6
which ensures that W55 These constraints guarantee that the last control point and the first control point of the next segment are the same, therefore enforcing Co continuity of the reconstructed shape. Fig. shows an example of reconstructing the shape of mustard bottle from the YCB dataset.
Reference radius
Reconstrurted
0.0
600
134
rahun
225"
10
Figure 7: Shape parametrization of mustard bottle using basis functions.
B.3
Reorientation
In this task, we aim enable the robot reorient an object using parallel fingers. An initial velocity given the object by swinging the robot arm. The state is the orientation angle 8, and the control input is the normal force fn between the gripper and object. The physical parameters are the object
15

---

Table 3: Time required for parameter-connditonee policy retrieval
Policy retrieval (core-level)
Policy retrieval (function-level)
Hit
0.016s = 0.002s
0.720s + 0.236s
Push
0.075s 0.003s
18.56s 4.748s
Reorientation
0.018s = 0.003s
8.494s + 0.561s
mass length I and torsional friction coefficient μ.. The gravitational torque and normal force fn are used as braking mechanisms to slow down the object motion. We build the dynamics model of the reorientation primitive based on [38] as
TÔ og + 2Tz,
(28)
0o- Ü
where Ty = f4 f1+7 the torsional sliding friction between robot gripper and the object. In this work, we set y μe is the torsional friction coefficient, which related the materials and normal force distribution. Tg mgllsi(0) is the gravity torque. We therefore include μes object mass and length the model parameters. The task is to rotate the object from vertically downward to vertically upward position. To achieve this, the object is given initial angular velocity &g by swinging the robot arm.
C Experimental details
C.1 Implementation details
In our experiments, we employed NVIDIA GeForce RTX 3090 GPU with 24GB of memory. For TTPI, the accuracy parameter Tmax and discount factor were set to 100 and 0.99, respectively. The continuous variables of state, action and parameter domains were discretized as 50 to 500 points using uniform discretization.
10-3 for TT-Cross approximation. The maximum rank
The baseline algorithms, DC+RL and RMA, utilize Soft Actor-Critic (SAC) [39] for policy learning- Our implementation is based Stable-Baselines [40], using a Multilayer Perceptron (MLP) with 64 64 x 64 architecture the policy network. The discount factor is 0.99, and the learning rate 0.001. Additionally, RMA employs MLP with a 256 128 x 64 architecture to embed privileged information.
C.2 Extra experiments
We compared the time used for retrieving the parameter-conditione policy in either core level or function level. Table shows that TT structure allows for much more efficient retrieval via core-level products compared with function-level products.
16


Reviews:
[{"review_id": "N7MlyCV8B7", "reviewer": "Reviewer_8Guh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Problem Studied by Authors: The authors of this work focus on the problem of learning robust primitives to execute contact-rich manipulation tasks. The main problem solved by the authors is to generate robust policies that can generalize to a diverse range of tasks. In this work, the authors have focused on 3 of such tasks.  Proposed Solution: Specifically, the authors propose a bi-level approach where they first learn a policy using multiple models and use a tensor train to approximate a value function and then retrieve a policy via domain contraction that extends domain randomization and domain adaptation.", "summary_of_recommendation": "Given that there are no benchmarks provided, I can change my weak accept to a strong accept if the authors can provide some sort of benchmarking results and compare their method with relevant primitive learning methods.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. The current method needs to be benchmarked with at least one of the well-known methods that learn primitives for a contact-rich manipulation task", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pt2PCXHsMt", "reviewer": "Reviewer_q6Sa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In the context of generalization in manipulation planning, the paper proposes domain contraction as a tradeoff between domain adaptation and domain randomization, where a policy is synthesized that is robust to a subset of possible parameters, as opposed to all possible (randomization) or a single parameter (adaptation). To efficiently compute such a policy, the tensor-train framework is used. The method is evaluated on a number of simple manipulation tasks, including on hardware.", "summary_of_recommendation": "I recommend weak accept: an interesting concept, clearly-written, and good experimental evaluation.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "please see review", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KJF6RyuIZ7", "reviewer": "Reviewer_aceS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes \"domain contraction\" with parameter-conditioned policies as a method for learning robust manipulation primitives. The state space is augmented to include latent model parameters (e.g., friction, mass), and the resulting parameter-conditioned value function and policy are derived. At deployment time, given a rough distribution over the latent properties, a corresponding robust policy can be efficiently retrieved. This approach unifies domain randomization and domain adaptation, with both being extreme cases of the proposed domain contraction. Rapid value learning and policy retrieval is enabled by tensor train decomposition and recent work applying it for control.", "summary_of_recommendation": "Overall, the paper needs significant re-writing for clarity and proper contextualization.", "technical_quality": 3, "clarity_of_presentation": 1, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please see above, especially the writing of the main method section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting extension of the Tensor Train framework, generalizing domain randomization and domain adaptation. The theory is solid, and meaningful evaluation is provided, including on real robot hardware. However, understanding the paper requires reading prerequisite papers first, and core concepts are introduced with insufficient context and prior motivation. Readability here is really a serious issue. In particular, Reviewer aceS gives good advice on how to improve the writing. Technical issues raised by multiple reviewers include scalability to higher-dimensional observations, and (theoretical and experimental) comparison to simpler baselines.

The rebuttal addressed most of the reviewers' concerns, including all mentioned in the preceding paragraph.
Decision: Accept