Robust Manipulation Primitive Learning via Domain Contraction

Authors: teng  xue
amirreza  razmjoo
suhan  shetty
sylvain  calinon
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 22:45:44

Abstract: Contact-rich manipulation plays an important role in everyday life, but uncertain parameters pose significant challenges to model-based planning and control. To address this issue, domain adaptation and domain randomization have been proposed to learn robust policies. However, they either lose the generalization ability to diverse instances or perform conservatively due to neglecting instance-specific information. In this paper, we propose a bi-level approach to learn robust manipulation primitives, including parameter-augmented policy learning using multiple models with tensor approximation, and parameter-conditioned policy retrieval through domain contraction. This approach unifies domain randomization and domain adaptation, providing optimal behaviors while keeping generalization ability. We validate the proposed method on three contact-rich manipulation primitives: hitting, pushing, and reorientation. The experimental results showcase the superior performance of our approach in generating robust policies for instances with diverse physical parameters.

Main paper: 
 introduction: 
 Robot manipulation usually involves multiple different manipulation primitives, such as Push and Pivot, leading to hybrid and long-horizon characteristics. This poses significant challenges to most planning and control approaches. Instead of treating long-horizon manipulation as a whole, it can be decomposed into several simple manipulation primitives and then sequenced using PDDL planners [1, 2, 3] or Large Language Models [4, 5] . Although such manipulation primitives usually have lowto-medium-dimensional state and action spaces, the breaking and establishment of contact make it tough for most motion planning techniques. Gradient-based techniques suffer from vanishing gradients when contact breaks, while sampling-based techniques struggle with the combinatorial complexity of multiple contact modes, i.e., sticking and sliding. This leads to time-consuming online replanning in the real world for contact-rich manipulation, limiting the real-time reactiveness of robots in coping with uncertainties and disturbances.Learning manipulation primitives that can quickly react to the surroundings, therefore, makes a lot of sense. Since the learned manipulation primitives will be sequenced by symbolic planners, which have no information about the geometric/motion level, the learned manipulation primitive should be robust to diverse instances with varied physical parameters, such as shape, mass, and friction coefficient. For example, once the push primitive is scheduled by the high-level symbolic planner, it should be able to push any objects with different shapes and friction parameters from any initial configurations towards their targets.Consider how humans perform manipulation tasks: we typically make rough estimates of object and environmental parameters based on physical intuition [6] , and our behaviors adapt accordingly. To enable robots to achieve similar performance, the learned manipulation primitives should effectively generalize across diverse instances and adapt to the parameters of specific instances. Common techniques for robust policy learning include domain adaptation (DA) [7, 8, 9] and domain randomization (DR) [10, 11] . DA aims for a perfect match between simulation and reality but may sacrifice generalization ability, while DR generalizes well but lacks full knowledge of the target domain, leading to conservative behaviors. Combining DR and DA [11, 12] to harness the benefits of both is a sensible strategy, but how to effectively balance them is still an open question. In this work, we propose a bi-level approach to address this challenge:Level-1: Parameter-augmented policy learning using multiple models. We augment the state space with physical parameters of multiple models and use Tensor Train (TT) to approximate the state-value function and advantage function.Level-2: Parameter-conditioned policy retrieval through domain contraction. At the stage of execution, we can obtain a rough estimate of the physical parameters in the manipulation domain. This instance-specific information can be utilized to retrieve a parameter-conditioned policy, expected to be much more instance-optimal.We summarize our contributions as follows:1) We propose domain contraction, a unification of domain adaptation and domain randomization, which results in optimal behavior while maintaining generalization.2) We propose using tensor approximation for robust policy learning with multiple models and leveraging products of tensor cores for parameter-conditioned policy retrieval.3) We provide both theoretical proof and numerical comparisons for the proposed approach. 
related work: 
 Manipulation policy learning. Many approaches have been proposed to learn manipulation policies for long-horizon manipulation, including Behavior Cloning (BC) [13, 14] , Deep Reinforcement Learning (DRL) [15, 12, 16] , and Approximate Dynamic Programming (ADP) [17, 18] . In our work, to ensure subsequent parameter-conditioned policy retrieval, it is essential to explore the full parameter domain. The reliance on the provided dataset in BC limits its ability to explore the broad state space and learn the full parameter-augmented policy. DRL offers better exploration capability compared to BC, but its policy training is usually time-consuming due to sample inefficiency, and policy retrieval is often suboptimal due to gradient-based optimization and local exploration. Conversely, ADP aims to cover the full state space but faces the curse of dimensionality. Recently, a new technique called Tensor Train Policy Iteration (TTPI) [19] has emerged to address this challenge by employing tensor approximation. Our work builds upon TTPI, extending it for robust primitive learning by augmenting the state space with model parameters.Sim-to-real transfer. Obtaining large amounts of real-world data for primitive learning is challenging. Therefore, robot learning in simulation and transferring to the real world is a promising idea [20] . However, the reality gap between simulation and the real world poses a significant challenge to such idea. If the target domain is known and specific, Domain Adaptation (DA) [7, 8, 9] can be an effective method, but its reliance on domain-specific data can limit generalization to other scenarios without additional fine-tuning. On the other hand, Domain Randomization (DR) [21] seeks to develop a robust policy by introducing random variations into the simulation parameters. While this method offers good generalization, it often results in suboptimal and high-variance behaviors due to the restrictive assumptions about environment parameter distribution (e.g., normal or uniform). To overcome this, combining DA and DR can be a promising strategy to balance generalization and optimal performance. The basic idea is to adapt the parameter distribution by leveraging differences between simulated and reference environment [10, 11, 22] . However, policies developed through such methods are often tailored to the reference environment or target domain. Recently, a method called Rapid Motor Adaptation (RMA) [23] has demonstrated remarkable success in learning robust locomotion [16, 24] and manipulation primitives [12] . In this approach, a teacher policy is trained in simulation with various domain parameters, and a student policy learns to replicate this behavior, with a low-dimensional embedding of proprioceptive history as input. Our work closely follows this paradigm but introduces a more efficient way for learning the parameter-augmented teacher policy. Additionally, our method eliminates the need to learn a student policy afterward, as the parameter-conditioned robust policy can be easily retrieved from the teacher policy. It also offers a flexible way to leverage domain knowledge, making it effective in both closed-loop and open-loop scenarios, whereas RMA struggle in open-loop scenarios due to the lack of privileged information.Tensor Train for function approximation. A multidimensional function can be approximated by a tensor, where each element in the tensor is the value of the function given the discretized inputs. The continuous value of the function can then be obtained by interpolating among tensor elements. However, storing the full tensor for a high-dimensional function can be challenging. To address this issue, Tensor Train (TT) was proposed to approximate the tensor using several thirdorder cores. The widely used methods include TT-SVD [25] and TT-cross [26] . Furthermore, TTGO [27] was proposed for finding globally optimal solutions given functions in TT format. Thanks to the low-rank embeddings of the original function, this method significantly enhances computation efficiency and reduces the risk of getting trapped in local optima. TTPI [19] was then introduced to learn control policies through tensor approximation, showing superior performance on several hybrid control problems. Logic-Skill Programming (LSP) [2] expands the operational space of TTPI by incorporating first-order logic to sequence policies. Building on TTGO and TTPI, our work extends these methods to learn robust policies, which further enhances the capabilities of LSP. We additionally demonstrate that the TT format is a suitable structure for domain contraction, allowing for efficient parameter-conditioned policy retrieval.  
problem formulation: 
 Consider a discrete-time dynamical system defined by:EQUATIONwhere x t ∈ Ω x and u t ∈ Ω u represent the state and action at time step t, respectively. The dynamics model f : Ω x ×Ω u ×Ω α → Ω x is conditioned on the domain (or environment) parameters α ∈ Ω α (such as masses, shapes, or friction coefficients). These parameters are assumed to be random variables satisfying an unknown probability distribution p. Together with the reward function R : Ω x × Ω u → R and the discount factor γ ∈ [0, 1], this system forms a Markov Decision Process (MDP)M = {Ω x , Ω u , Ω α , p, f, R, γ}.In this work, we aim to find the optimal policy that maximizes the expected cumulative reward for a distribution of domain parameters α ∼ p(α), namelyEQUATIONTo learn such parameter-conditioned policies, it is impractical to train them individually since there can be infinite instances for one single primitive. Instead, we propose to learn a parameteraugmented full policy first and then retrieve the parameter-conditioned optimal policy for specific domain at runtime.We define domain contraction as retrieving the parameter-conditioned policy from the weighted sum of parameter-specific advantage functions. The theoretical proof is provided in Sec. 3.3. These functions are learned jointly by augmenting the state space with parameters, similar to the multi-goal setting in DRL [28] , resulting in parameter-augmented advantage function.TTPI [19] is used to approximate the parameter-augmented value functionEQUATIONand the parameter-augmented advantage functionEQUATIONin TT format. We define the tensor cores related to x and u asEQUATIONWithout loss of generality, we assume each subspace Ω αi of the parameter spaceΩ α = Ω α1 × • • • × Ω α d is discretized by N i points.The parameter-specific advantage function can then be extracted asEQUATIONwhereα j = (α j1 , • • • , α j d) represents α at the discretization index j across all dimensions.We definep j = p 1 (α j1 )p 2 (α j2 ) • • • p d (α j d )as the probability of α j , and p i is the probability distribution at dimension i. Given the TT approximation P (j1,...,j d ) = P 1 :,j1,: P 2 :,j2,:• • • P d :,j d ,:of the rough parameter distribution p j , we can then compute the parameter-conditioned advantage function by computing the weighted sum of parameter-specific advantage functions in a separable formEQUATIONThe computation of weighted sum is at the tensor core level, which is much more efficient than at the function level. After obtaining the parameter-conditioned advantage function, we can retrieve the parameter-conditioned policy using TTGO [27] , a specialized method for efficiently finding globally optimal solutions given functions in TT format.To retrieve such parameter-conditioned policies, knowing the parameter distribution p is crucial. DR and DA rely on assumed distributions during training, while DC offers a better way to leverage domain knowledge during execution, enabling optimal behaviors while preserving generalization capability. In Section 3.3, we demonstrate that DR and DA are two special cases of DC.Theorem 1. Given domain parameters α and the distribution p, the parameter-conditioned policy can be retrieved from the weighted sum of parameter-specific advantage functions.Proof. Given that each subspace Ω αi of the parameter space Ω α is discretized by N i points, the parameter-conditioned policy can be written asEQUATIONand the parameter-specific advantage function is defined asEQUATIONGiven ( 8) and ( 9), we can derive that the parameter-conditioned advantage function is the weighted sum of parameter-specific advantage functions, namelyEQUATIONand the parameter-conditioned primitive policy can then be computed byEQUATIONNote that the parameter-conditioned policy cannot be directly computed as the weighted sum of parameter-specific policies, since the arg max operation does not have an associative property with respect to addition.Theorem 2. Domain randomization and domain adaptation are two special cases of domain contraction.Proof. The policies of DR, DA and DC are obtained byπ DR (x) = arg max u∈Ωu N1 j1=1 • • • N d j d =1 p j A αj (x, u), π DA (x) = arg max u∈Ωu N1 j1=1 • • • N d j d =1 p j A αj (x, u), π DC (x) = arg max u∈Ωu N1 j1=1 • • • N d j d =1 p j A αj (x, u).where p j , p j and p j are the probabilities of the parameter instance α at index j, satisfying the distributions p, p and p, which are used for policy retrieval in DR, DA and DC, respectively.Note that at the policy training level, we have no information about the real distribution p * . DR assumes p * as a uniform or Gaussian distribution p (which can be a wrong assumption) for policy training. DA aims for closed match between simulation and the target domain by using partly the real data, resulting in p ≈ p * . Conversely, DC enables the use of domain knowledge during the execution stage, where instance-specific information becomes available. We assume the estimated rough parameter distribution as p. If p matches p * , the policy aligns with DA; if no instance knowledge is present, p equals p, resulting in a DR-like policy. DC therefore unifies DA and DR, while allowing flexible compromise between these two extreme cases. 
experimental results: 
 We validate the effectiveness of the proposed method on three contact-rich manipulation tasks: Hit, Push, and Reorientation, as shown in Fig. 1 . All the primitives are highly dependent on the physical parameters between the object, the robot, and the surroundings. The environmental details, including the task setup and implementation details, are presented in the appendix.We first learn the parameter-augmented policies using multiple models through tensor approximation. The physical equation of Hit is fully known, we can therefore compute the control policy analytically. For Push and Reorientation, TTPI is used to approximate the parameter-augmented value functions and advantage functions.Hit: Given the mass m, the friction coefficient µ, the initial state x 0 and the target x des , the advantage function isEQUATIONwhere x = x 0 + I m t -0.5µgt 2 . Computing the impact I is to find the value that maximizes (12) . Push: The reward function for Push primitive learning is defined as EQUATIONEQUATIONwhere x p = [s x , s y ] and x o = θ denote the object's position and orientation. Without loss of generality, we set x des = 0 as the target configuration. f is the control force, while v p is the velocity of the robot's end-effector. l p and l o are set to 0.005 and 0.01π, respectively.Reorientation: The reward function for Reorientation primitive learning is defined asEQUATIONwithc g = ∥x o -x des o ∥, c f = ∥f n ∥.x o is the orientation angle θ, and f n is the normal force between the gripper and object. x des o is set to π as the reorientation goal, and β is set to 10 4 . We then retrieve the parameter-conditioned policy through domain contraction. Note that any distribution of the parameters is allowed in our framework. Without loss of generality, we assume the parameters satisfy a uniform distribution, within a range of the discretization indices of each dimensionality (denoted as w), as shown in Table 1 and Fig. 4 . w = 1 means we know the exact value of the physical parameter, corresponding to domain adaptation. In contrast, w = N means we have no prior knowledge about the model parameters, corresponding to domain randomization. norm of the difference between the final state and the target state. The cumulative reward is normalized using the value obtained through domain adaptation. We can observe that Hit primitive depends more on the accuracy of model parameters, as it resembles an open-loop control. Once the impact is given from the robot to the object, there is no way to adjust control inputs to influence the object movements further. However, as the comparison indicates, there is no need to have a precise parameter estimation. A rough range (w = N/20) is sufficient to achieve the target. In contrast, Push and Reorientation can be considered more akin to closed-loop control. Therefore, the requirement for accurate parameter estimation can be relaxed further. As depicted in Table 1 and Fig. 4 , a rough distribution with w = N/5 is adequate. Moreover, based on Table 1 , we observe that w = N results in the lowest cumulative reward. This is consistent with our assertion that domain randomization typically leads to conservative behaviors. Although domain adaptation (w = 1) yields the highest cumulative reward, obtaining precise parameter values can be challenging in the real world. Domain contraction bridges the gap between domain adaptation and domain randomization, offering greater flexibility to generate optimal behaviors while utilizing instance-specific rough parameter distribution. This is much practical for real-world contact-rich manipulation tasks. Moreover, we compared our method with two widely used robust policy learning approaches: reinforcement learning with domain randomization (RL+DR) [21] and rapid motor adaptation (RMA) [23, 12] . The quantitative results are presented in Table 2 , including the time required for policy training and the final state error. Note that in RMA, teacher error and student error correspond to the errors of the teacher policy given the true parameter and the student policy, respectively, while in our method, teacher error and DC error correspond to the errors of the parameter-augmented policy given the true parameter and the parameter-conditioned policy obtained through domain contraction (DC). Our method requires significantly less time for policy training and results in much lower final state error compared to the other methods. Furthermore, when comparing the error of the teacher policy in RMA with our parameter-augmented policy, our method shows better accuracy, highlighting its potential as a teacher policy in the general RMA framework. Additionally, in one-shot manipulation tasks where only one action can be executed (as shown in the Hit primitive in the table), RMA does not work due to the lack of privileged information.We validated our proposed method for the planar pushing task using a 7-axis Franka robot and a RealSense D435 camera. The manipulated objects included a sugar box and a bleach cleanser from the YCB dataset [29] , each with different shapes and masses, as shown in Fig. 1 . The friction coefficients between the objects and the table were varied by using a metal surface and plywood, respectively. Note that it is easier to control the robot kinematically rather than using force control. We leverage the ellipsoidal limit surface to convert the applied force to velocity, resulting in the motion equations shown in [30, 31] . We trained a parameter-augmented policy in simulation and then applied it in the real world through domain contraction. The experimental results showcase the effectiveness of the obtained parameter-conditioned policies in manipulating instances with diverse parameters. Additionally, external disturbances were introduced by humans, demonstrating the reactiveness of the retrieved policy. Additional results are presented in the accompanying video. 
conclusion and future work: 
 In this paper, we propose a bi-level approach to learning robust manipulation primitives. Multiple models and tensor approximation are used to train the parameter-augmented policy, which can be directly applied to diverse instances through domain contraction, outputting instance-dependent optimal behaviors. Theoretical proof and numerical results demonstrate the efficiency of the proposed method for robust primitive learning. Real-world experiments further validate its effectiveness in contact-rich manipulation under uncertainty and disturbance.Since we leverage TTPI to learn the parameter-augmented policy, we also adopt its limitations. TTPI assumes a low-rank structure of the value function and advantage function. Therefore, the state space should have low-to-medium dimensionality (approximately less than 35), limiting its application for image-based policy learning. In the future, this limitation can be addressed by combining TT with neural networks in a data-driven manner [32, 33] .Moreover, we assume that rough distributions of uncertain physical parameters are provided by human knowledge. This requirement can be eliminated by integrating Large Visual Language Models or system identification into the framework. As shown in our experiments, the parameter estimation does not need to be very precise, thanks to the proposed domain contraction technique. Additionally, the policy can be iteratively improved as the belief about parameter estimation is updated. 
 
 Appendix: A multivariate function P (x 1 , . . . , x d ) defined over a rectangular domain constructed with the Cartesian product of intervals (or discrete sets) I 1 × • • • × I d can be discretized by evaluating it at points in the set X = {(x i1 1 , . . . , x i d d ) :x i k k ∈ I k , i k ∈ {1, . . . , n k }}.This gives us a tensor P, a discrete version of P , where P (i1,...,i d ) = P (x i1 1 , . . . , x i d d ), ∀(i 1 , . . . , i d ) ∈ I X , and I X = {(i 1 , . . . , i d ) : i k ∈ {1, . . . , n k }, k ∈ {1, . . . , d}}. The value of P at any point in the domain can then be approximated by interpolating between the elements of the tensor P.Naively approximating a high-dimensional function using a tensor is intractable due to the combinatorial and storage complexities of the tensor (O(n d )). Tensor networks mitigate the storage issue by decomposing the tensor into factors with fewer elements, akin to using Singular Value Decomposition (SVD) to represent a large matrix. In this paper, we explore the use of Tensor Train (TT), a type of Tensor Network that represents a high-dimensional tensor using several third-order tensors called cores, as shown in Fig. 5 .Figure 5 : TT decomposition generalizes matrix decomposition techniques to higherdimensional arrays. In TT format, an element in a tensor can be obtained by multiplying specific slices of the core tensors. The figure presents examples of second-order, third-order, and fourth-order tensors. Image adapted from [27] .We can access the element (i 1 , . . . , i d ) of the tensor in this format simply given by multiplying matrix slices from the cores: P (i1,...,i d ) = P 1 :,i1,: P 2 :,i2,:EQUATIONwhere P k :,i k ,: ∈ R r k-1 ×r k represents the i k -th frontal slice (a matrix) of the third-order tensor P k . For any given tensor, there always exists a TT decomposition [25] . This low-rank structure further facilitates sampling and optimization for robot planning and control.There are several ways to acquire a TT model, including TT-SVD [25] and TT-Cross [34, 35] . TT-SVD extends the SVD decomposition from matrix level to a high-dimensional tensor level. However, it needs to store the full tensor first, which is impractical for high-dimensional functions. TT-Cross solves this issue by selectively evaluating the function P on a subset of elements, avoiding the need to store the entire tensor.Given the discrete analogue tensor P of a function P , we obtain the continuous approximation by spline-based interpolation of the TT cores corresponding to the continuous variables only. For example, we can use linear interpolation for the cores (i.e., between the matrix slices of the core) and define a matrix-valued function corresponding to each core k ∈ {1, . . . , d},EQUATIONwherex i k k ≤ x k ≤ x i k +1 k and P k : I k ⊂ R → R r k-1 ×r k with r 0 = r d = 1.This induces a continuous approximation of P given byEQUATIONThis allows us to selectively do the interpolation only for the cores corresponding to continuous variables, and hence we can represent functions in TT format whose variables could be a mix of continuous and discrete elements.A.4 Global Optimization using Tensor Train (TTGO)In optimization problems involving task parameters x and decision variables u, the goal is to find the optimal u that minimizes the cost function c(x, u). TTGO [27] frames this problem as maximizing an unnormalized probability density function (PDF) P (x, u), which is derived from c(x, u) through a monotonically non-increasing transformation. For example, P (x, u) can be defined as e -βC(x,u) 2 with β > 0.The TT-Cross algorithm is then used to compute the discrete analogue approximation of the unnormalized PDF, i.e., P, in the TT format. After approximating the joint distribution, P xt can be obtained by conditioning on the given task parameter x = x t ∈ Ω x .Given the TT model P xt , the optimal u is obtained through iterative sampling, which determines the optimal solution for each dimension at the core level. Due to the low-rank nature of the TT model, this sampling process is highly efficient. A number of prioritized samples, N ≥ 1, are selected, and the sample(s) with the highest density (or lowest cost) are chosen as candidate solutions. These near-optimal solutions can then be further refined using local optimization methods, such as Newtontype optimization for continuous variables. Overall, this technique can efficiently find the globally optimal solution given any low-rank function, without requiring a convex structure. Moreover, the computation process is gradient-free and can handle a mix of continuous and discrete variables.Optimal control of dynamic systems with nonlinear dynamics presents a significant challenge in robotics. To address this, Generalized Policy Iteration using Tensor Train (TTPI) was proposed, leveraging tensor approximation and approximate dynamic programming [19] . This method approximates state-value and advantage functions using Tensor Train (TT), effectively mitigating the curse of dimensionality. The low-rank structure of TT enables the use of TTGO to find near-global solutions during policy retrieval from the advantage function, even under complex nonlinear system constraints, surpassing the capabilities of existing neural network-based algorithms [33] . This approach does not require knowledge of the dynamics model; a black-box simulator suffices, akin to model-free reinforcement learning. Shetty et al. [19] demonstrated TTPI's superior performance on several hybrid control problems compared to state-of-the-art hybrid RL algorithms. In this paper, we apply this approach to learning manipulation primitives for contact-rich tasks involving many contact parameters. The typical TTPI algorithm cannot cope with the diverse contact-rich instances. Therefore, we propose domain contraction to retrieve the parameter-conditioned policy that can achieve robust manipulation.Hitting is widely used to manipulate objects through impact. In this work, we focus on planar hitting primitive. The state is the object position, denoted as x = [x, y]. The control input is the applied impact I = [I x , I y ]. The physical parameters include the object mass m and friction coefficient µ. The motion equation isEQUATIONWe parameterize the object shape using a concatenation of Bézier curves, with the weight matrix C defined asEQUATIONwhere the pattern which ensures that w 4 = w 5 . These constraints guarantee that the last control point and the first control point of the next segment are the same, therefore enforcing C 0 continuity of the reconstructed shape. Fig. 7 shows an example of reconstructing the shape of a mustard bottle from the YCB dataset.Figure 7 : Shape parametrization of a mustard bottle using basis functions.In this task, we aim to enable the robot to reorient an object using parallel fingers. An initial velocity is given to the object by swinging the robot arm. The state is the orientation angle θ, and the control input is the normal force f n between the gripper and object. The physical parameters are the object 


Decision: This paper proposes an interesting extension of the Tensor Train framework, generalizing domain randomization and domain adaptation. The theory is solid, and meaningful evaluation is provided, including on real robot hardware. However, understanding the paper requires reading prerequisite papers first, and core concepts are introduced with insufficient context and prior motivation. Readability here is really a serious issue. In particular, Reviewer aceS gives good advice on how to improve the writing. Technical issues raised by multiple reviewers include scalability to higher-dimensional observations, and (theoretical and experimental) comparison to simpler baselines.

The rebuttal addressed most of the reviewers' concerns, including all mentioned in the preceding paragraph.

Reviews: review: title: title: The given paper is a good extension of the Tensor Train framework to approximate value functions and learn parameter-conditioned policy through domain contraction. The work has good theoretical insights however, lacks a similar rigor in real-world experiments and no benchmarks.
paper_summary: paper_summary: Problem Studied by Authors: The authors of this work focus on the problem of learning robust primitives to execute contact-rich manipulation tasks. The main problem solved by the authors is to generate robust policies that can generalize to a diverse range of tasks. In this work, the authors have focused on 3 of such tasks.  Proposed Solution: Specifically, the authors propose a bi-level approach where they first learn a policy using multiple models and use a tensor train to approximate a value function and then retrieve a policy via domain contraction that extends domain randomization and domain adaptation.
main_review: main_review: Overall, the authors propose a very interesting framework to unify the domain randomization and domain adaptation methods to learn robust manipulation primitives. However, the manuscript can be difficult to follow since a lot of work is extended from prior work. Given the length restriction, it is understandable that its difficult, but authors can try to include a more detailed preliminary section in the appendix than what they have currently in Appendix Section B. Furthermore, there is no comparison with current well-established methods.

Strengths:
1. The theorems mentioned are well-defined and demonstrate how domain contraction is a unification of domain randomization and domain adaptation.
2. It is a good extension of the TTPI work for learning a parameter augmented value function and the advantage function in TT format.
3. The TT architecture definitely improves the overall efficiency of learning a value function and the corresponding advantage function.


Weaknesses:
1. The reader needs to understand the old literature on tensor train to understand the proposed work. Thus, authors should definitely consider adding more information on TTPI and the Tensor-Train framework relevant to their proposed approach. 
2. A lot of the key algorithms and frameworks used are just cited by the authors rather than explaining their relevance in their aforementioned task and learning framework.
3. The paper currently lacks insights into why certain design choices were made. For example, why use TTGO to retrieve the policy?
It is clear that the TT method is better in terms of training time. However, currently, as a researcher/reader, I am not convinced as to why one should use this method over imitation learning or model-based learning that have shown good results even on more complex manipulation tasks.
questions: questions: 1. The current method needs to be benchmarked with at least one of the well-known methods that learn primitives for a contact-rich manipulation task
limitations: limitations: 2
review_summary: review_summary: Given that there are no benchmarks provided, I can change my weak accept to a strong accept if the authors can provide some sort of benchmarking results and compare their method with relevant primitive learning methods.
title: title: Review
paper_summary: paper_summary: In the context of generalization in manipulation planning, the paper proposes domain contraction as a tradeoff between domain adaptation and domain randomization, where a policy is synthesized that is robust to a subset of possible parameters, as opposed to all possible (randomization) or a single parameter (adaptation). To efficiently compute such a policy, the tensor-train framework is used. The method is evaluated on a number of simple manipulation tasks, including on hardware.
main_review: main_review: Strengths: Overall, I quite like the idea of efficiently mixing parameter-conditioned policies at runtime, and I think the TT framework is a nice way to achieve this mixing efficiently. The paper is overall well-written and technically solid.

I only have minor concerns:
- It would help to provide some intuition on the scalability of such approaches to higher-dimensional systems or parameter spaces - how high is too high? It is discussed briefly in the conclusion that dimensionality is an issue due to discretization, but some concrete numbers would help to ground this claim.
- Where does the scalability challenge arise? Is this at training time, or at policy retrieval time? Some discussion would help here.
- In Table 2, what is the relation between training time and policy retrieval time?
- For the considered tasks, how important is the effect of parameter discretization density to computation time, and how sensitive is policy performance to a coarse parameter discretization?
- Section 3 would benefit from an overview of the method, before jumping directly into a theorem and proof
- In Eq(1), f is not defined (I suppose it is the dynamics)
- Figure 2 could be clearer - perhaps write explicitly in the caption that the blue and red lines represent explicitly the parameters that each policy is conditioned on.
questions: questions: please see review
limitations: limitations: 3
review_summary: review_summary: I recommend weak accept: an interesting concept, clearly-written, and good experimental evaluation.
title: title: DC
paper_summary: paper_summary: The paper proposes "domain contraction" with parameter-conditioned policies as a method for learning robust manipulation primitives. The state space is augmented to include latent model parameters (e.g., friction, mass), and the resulting parameter-conditioned value function and policy are derived. At deployment time, given a rough distribution over the latent properties, a corresponding robust policy can be efficiently retrieved. This approach unifies domain randomization and domain adaptation, with both being extreme cases of the proposed domain contraction. Rapid value learning and policy retrieval is enabled by tensor train decomposition and recent work applying it for control.
main_review: main_review: On the whole, the intuition and theoretical results seem solid, but the paper is very difficult to follow. Some additional text at the start of the method section should be added rather than immediately stating a theorem.  The problem should be stated clearly, i.e., we're in an MDP, the system dynamics are conditioned on the parameters $\alpha$, the goal is to recover the optimal policy $\pi^*$ for a specific distribution at runtime. Similarly, Theorem 2 regarding domain contraction is defined before providing a definition of DC. I would recommend stating a definition of the term and providing some intuition before proving its properties. And again, the definition of the various probability distributions is provided in the paragraph after the proof.  It feels like this whole section is backward from top to bottom. I would recommend introducing the problem, and at least define the parameter-conditioned policies and DC before providing their proofs. At present it is frankly difficult to follow without context or intuition. 

Some related work in the context of adaptation for sim-to-real is notably omitted. For example, related work [1,2,3] discusses learning with privileged information to enable rapid adaptation at runtime, yet only vanilla DR and DA are discussed. It seems that these papers are actually the most closely related works. Other works, like BayesSim [4], similarly estimate a distribution of sim parameters given real data and retrieve policies robust to that distribution through RL training. I'm not entirely sure the notion of domain contraction is really novel. 

The approach has some obvious limitations that are hinted at. The path toward high-dimensional observations is not obvious; the dynamics models need to be known a priori; the dynamic parameters need to be defined a priori. It isn't clear to me if this is a scalable approach or how it would be applied for more general, complex sim-to-real settings. 

Finally, the writing is also generally not up to the standards of the conference with some grammatical mistakes and awkwardness throughout. The manuscript should be proofread for grammar and style. 

[1] Yu, W., Tan, J., Liu, C. K., & Turk, G. (2017). Preparing for the unknown: Learning a universal policy with online system identification. arXiv preprint arXiv:1702.02453.
[2] Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020). Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), eabc5986.
[3] Kumar, A., Fu, Z., Pathak, D., & Malik, J. (2021). Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034.
[4] Ramos, F., Possas, R. C., & Fox, D. (2019). Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint arXiv:1906.01728. (in RSS 2019).
questions: questions: Please see above, especially the writing of the main method section.
limitations: limitations: 2
review_summary: review_summary: Overall, the paper needs significant re-writing for clarity and proper contextualization.