Title: Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Imitation Learning

Optimal Transport

Sim-to-Real
Abstract: Behavior cloning has shown promise for robot manipulation, but real-world demonstrations are costly to acquire at scale. While simulated data offers a scalable alternative, particularly with advances in automated demonstration generation, transferring policies to the real world is hampered by various simulation and real domain gaps. In this work, we propose a unified sim-and-real co-training framework for learning generalizable manipulation policies that primarily leverages simulation and only requires a few real-world demonstrations. Central to our approach is learning a domain-invariant, task-relevant feature space. Our key insight is that aligning the joint distributions of observations and their corresponding actions across domains provides a richer signal than aligning observations (marginals) alone. We achieve this by embedding an Optimal Transport (OT)-inspired loss within the co-training framework, and extend this to an Unbalanced OT framework to handle the imbalance between abundant simulation data and limited real-world examples. We validate our method on challenging manipulation tasks, showing it can leverage abundant simulation data to achieve up to a 30\% improvement in the real-world success rate and even generalize to scenarios seen only in simulation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a unified co-training framework that trains a single policy on both simulation and real-world demonstrations. The core innovation is the use of an Unbalanced Optimal Transport (UOT) loss to align joint distributions of observations and actions (or proprioceptive states) across domains in a shared latent space, rather than aligning marginal distributions. This action-aware alignment, augmented with temporally aligned sampling, preserves task-relevant features and handles domain imbalance. \n\nThe authors evaluate across six manipulation tasks using both RGB and point cloud modalities. They compare against Source-only, Target-only, MMD-based, and naive co-training baselines. Metrics include task success rates in standard and out-of-distribution (OOD) scenarios in both simulated and real environments. Key tools include MimicGen for simulation data and Robosuite for simulation environments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- **Use of UOT:** The integration of UOT into a co-training framework for sim-to-real transfer is methodologically sound and innovative. OT provides a structured and geometry-aware way to align feature-action distributions, which is superior to marginal feature alignment techniques like MMD. The author should cite [1] since it is very relevant on latent-space alignment with UOT without the marginal constraints.\n- **Data Imbalance Handling:** UOT’s ability to handle domain imbalance is crucial in practical sim-to-real settings where simulated data vastly outnumbers real-world examples. This mitigates overfitting to the dominant domain and avoids spurious alignments. \n- **Improved OOD Generalization:** The method shows consistent performance gains in out-of-distribution (OOD) settings, highlighting its strength in extrapolating from simulated experiences to unseen real-world scenarios. Real-world experiments with the Franka Panda robot validate that the method is not only simulation-effective but deployment-ready for physical systems.\n\n**Weaknesses**\n\n- **Limited Addressing of Dynamics Gap:** The method only tackles the visual observation gap and assumes quasi-static, prehensile tasks. It does not address the sim-to-real dynamics mismatch (e.g., surface friction of push block), which can be substantial in more dynamic or contact-rich scenarios.\n- **Dependence on Demonstration Alignment:** The method assumes the overhead of behavioral similarity between simulated and real demonstrations, enforced via DTW. This may not hold in scenarios where real demonstrations deviate significantly in strategy or execution from simulated ones. Moreover, the approach relies on action or proprioceptive similarity across domains. This limits its application to settings where such consistency is hard to guarantee (e.g., different control schemes or hardware platforms).\n- **Limited Theoretical Development:** The paper does not present theoretical guarantees or convergence analysis for the proposed UOT-guided co-training. While empirical results are strong, formal properties are not fully explored.\n- **Hyperparameter Sensitivity Not Explored:** The paper does not provide an ablation or sensitivity analysis on key UOT parameters (e.g., entropy regularization, KL divergence penalties), which may influence performance stability.\n\n[1] Nguyen, Duy Minh Ho, et al. \"Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model.\" Asian Conference on Machine Learning. PMLR, 2025.\n\nQUESTIONS:\n1. How sensitive is the performance to the choice of UOT hyperparameters (e.g., entropy regularization, KL penalties)?\n2. Could the approach generalize to tasks with significant dynamics gaps, such as high-speed or contact-rich manipulation? It seems all evaluated tasks are transferred kinematically. \n3. How does the performance scale with even fewer real-world demonstrations (e.g., <5 per task)?\n4. Would alternative domain alignment methods (e.g., contrastive learning or adversarial domain alignment [2, 3]) offer competitive or complementary benefits?\n\n[2] Raychaudhuri, Dripta S., et al. \"Cross-domain imitation from observations.\" International conference on machine learning. PMLR, 2021.\n\n[3] Kim, Kuno, et al. \"Domain adaptive imitation learning.\" International Conference on Machine Learning. PMLR, 2020.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unified sim-and-real co-training framework for learning generalizable policies for robotic manipulation. It aims to address the challenges of costly real-world robot demonstration data acquisition and the sim-to-real domain gap by leveraging both a small number of real-world demonstrations and abundant simulated data. A key contribution is the proposal of an Optimal Transport (OT)-based loss function to align the joint distributions of observations and actions across domains, extended to an Unbalanced Optimal Transport (UOT) framework to handle imbalanced data. Furthermore, a temporally-aware sampling strategy is introduced to enhance the efficiency of mini-batch OT training. Through experiments on various robotic manipulation tasks, the proposed method intends to demonstrate improved real-world success rates and superior generalization capabilities, even for scenarios learned exclusively in simulation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Addressing the sim-to-real transfer problem in robot learning remains a significant challenge, and the attempt to overcome the constraint of limited real-world data is relevant.\n- The conceptual idea of applying Optimal Transport (OT) to align joint observation-action distributions, utilizing Unbalanced Optimal Transport (UOT) to manage data imbalance, and introducing a temporally aware sampling strategy is interesting.\n\n### Weaknesses\n\n- The proposed methodology appears to be primarily a combination of existing research on Optimal Transport and Unbalanced Optimal Transport for domain adaptation. There is a lack of clear explanation and sufficient theoretical justification for how this combination offers fundamental new theoretical or methodological contributions beyond existing methods. The temporally aware sampling appears to be an auxiliary technique for solving a specific problem rather than contributing core originality.\n- While the presented experimental results show performance improvements in specific tasks, they lack rigor in several critical aspects.\n    - Comparisons to other state-of-the-art and more robust sim-to-real or domain adaptation techniques are very limited. The current baselines are insufficient to adequately prove the superiority of the proposed method, which could lead to an overstatement of the paper's contributions.\n    - While the paper emphasizes \"a few real-world demonstrations,\" there is insufficient detailed information about the scale, diversity, and complexity of the actual datasets used. The OOD generalization results might be attributable to the specific dataset characteristics or the simplicity of the simulation environment rather than the intrinsic strength of the methodology. Validation across a wider range of domain gap scenarios is necessary.\n- Unbalanced OT and the temporally-aware sampling can involve complex mathematical backgrounds, and their implementation and tuning in real systems are expected to be considerably complex. There is insufficient discussion on whether this added complexity sufficiently justifies the observed performance gains, and what potential issues might arise during real-world deployment.\n\nQUESTIONS:\n1. Optimal Transport and Unbalanced Optimal Transport are well-known for domain adaptation. Please clearly explain what new insights or contributions, methodologically or theoretically, this paper offers compared to existing OT/UOT-based domain adaptation methodologies. Additional theoretical analysis is needed to demonstrate that this is not merely a combination of existing techniques.\n2. To solidify the superiority of the proposed methodology, please conduct comparative experiments against at least three more recent and relevant sim-to-real or domain adaptation techniques. Furthermore, quantitatively analyze the method's robustness to real-world uncertainties (e.g., varying lighting conditions, object deformations, robot pose errors).\n3. The paper claims to require \"only a few real-world demonstrations.\" Please provide a detailed analysis of the minimum threshold for \"a few real-world data\" used, and how the diversity of this data (e.g., different angles, lighting, object instances) influences the results. Can you clearly demonstrate the method's effectiveness even with extremely limited real-world data (e.g., fewer than 5 demonstrations)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes an algorithm for sim and real co-training of robotic manipulation skills using Temporally Aligned Sampling and Unbalanced Optimal Transport. It is assumed that we have access to a large sim training dataset and a small real-world training dataset. The goal is to generalize to novel states in the real-world that are covered in the sim dataset but not in the real-world dataset. This is achieved by training an image or point cloud encoder with a regularization loss computed using optimal transport. It is shown that performing behavior cloning with this encoder leads to positive sim-to-real and sim-to-sim transfer and the learned policies generalize to states previously unseen in the target domain. The proposed method outperforms baselines that perform co-training on sim and real data and a simpler version of visual feature alignment between sim and real. The method is tested in a real-world robot experiment with a Franka arm.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**Novel combination of known methods.** While sim and real co-training, unbalanced optimal transport and temporally aligned sampling are known methods, their combination to solve visual sim to real alignment that is done in this submission is novel. In particular, unbalanced optimal transport is useful because not every state in the source domain might have a corresponding state in the target domain. \n\n**Excellent video.** The video provided in the supplementary material clearly communicates the idea that we are moving from a wide reset distribution in sim to a narrow reset distribution in real-world demonstrations and then shows examples of real-world policy inference with an OOD reset distribution. I think this is a great showcase of the idea of learning joint sim and real embeddings using temporally-aligned optimal transport.\n\n**Evaluation with both images and point clouds.** The submission includes evaluation of diffusion policies with image encoders and 3D diffusion policies with point cloud encoders. Both evaluations highlight the usefulness of optimal transport in sim and real co-training. Using both images and point clouds increases the impact this paper could have in the robot learning community.\n\n**Evaluation of OOD transfer.** The experiments specifically test transfer to OOD states but in the sim and in the real. Showing positive transfer in this setting is a step towards learning generalist real-world manipulation policies.\n\n## Weaknesses\n\n**No strong real-world baseline / oracle.** The performance of all methods in the real-world robot experiments is relatively weak. For example, all methods are limited to a 40% success rate for the task of stacking two blocks. This is not entirely unexpected, as the methods are provided with only 25 real-world demonstrations and the distribution shifts between training and testing. This set of experiments is missing an oracle that provides an upper bound on the real-world robot performance. Since we do not know how well a diffusion policy would perform with a few hundred demonstrations, it is not clear if the low performance is due to a lack of data or other problems, such as incorrect controller gains or bugs in real-world data collection. Providing an oracle performance would better contextualize the real-world results.\n\n**Unclear if method could be reproduced from its description.** The method section contains frequent references to prior works for practical algorithms and an algorithmic description of the full method is delegated to Appendix C, Algorithm 1. Even Algorithm 1 lacks details needed to fully understand the computation done during training. I would suggest adding more details into Algorithm 1 and providing more guidance for reproducibility.\n\n**Unclear sensitivity to hyper-parameters.** Each training step includes temporal alignment computation and optimal transport computation. It is unclear how the hyper-parameters of these two steps are tuned and how sensitive the method is to small changes in these hyper-parameters. This could be clarified in an additional ablation study.\n\nQUESTIONS:\nI do not fully understand the motivation in Section 4.3. It is stated that naively sampling pairs of transitions from the source and the target dataset “may yield pairs from different stages of tasks”. However, I thought that the optimal transport was computed over the entire episodes; wouldn’t the OT computation already take care of aligning different stages of the task across different trajectories?\n\nLIMITATIONS:\nThe limitations of the method are addressed in the conclusion. In particular, the section mentions lack of sim to real dynamics alignment and limited dexterity of skills generated by MimicGen-style demonstration cloning.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the problem is using simulator data to train a behavioral cloned policy that can be transferred into the real world or a separate target environment which might have significant differences in perceptual space compared to the source simulator domain. It addresses this problem by assuming some data can be collected in the target domain with expert demonstration. Then it uses the idea of co-training on source and target data to train a policy that can leverage the large amount of data in the source and the grounded data in the target domain.\n\nThe contribution of this paper is to add in an unbalanced optimal transport loss that will pull together the representations for observations in the source and target domains that are behaviorally similar (based on  observation, action pairs) while ensuring that the representations of observations that are not close to these few demonstrations don't collapse due to the unbalanced data volumes.\n\nEvaluation is done in both sim-to-sim as well as sim-to-real settings, along with checking if adding more simulation data helps this approach learn better policies. t-SNE plots are also used to validate if the learned representations are close to each other.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The paper has a clear aim, a clear contribution, and experiments for the most part are designed to validate that contribution\n* Section 3 describing the technique is fairly well written and should not leave an intelligent reader with too many doubts. Appropriate references are made for a diligent reader to follow up and understand details.\n* Practical implementation details are called out in Section 3 instead of hidden in the appendix, which is appreciated.\n* Figures are clear\n* Most relevant work is cited. See some caveats in weaknesses\n\n## Weaknesses\n* The main reason for my on-the-fence recommendation is that this paper depends a lot on empirical evaluation, and there doesn't seem to be any statistical guarantee on any of the results in the paper. None of the tables discusses the number of trials, variance in the results, confidence bounds, or statistical significance in the results. Values are made bold in tables without clarifying what the criterion is for this formatting; are these values the highest numbers, or are they significantly higher? While I forgive forgoing these for real robot experiments, I would require some statistical guarantee from the sim-to-sim experiments to approve this paper. Below I now go in order from the greatest weaknesses to nitpicks.\n* The hypotheses laid out at the beginning of Section 5 are not the same ones mentioned later on page 8. Same or similar wording when mentioning the hypotheses being validated will help a reader to ground the experiments. \n* The phrasing of the hypotheses at the beginning of Section 5 is not clear enough: when mentioning words denoting comparison such as \"improves\" and \"enables generalization\", it is useful to point out what it is improving over (I am assuming baseline co-training performance) or generalizing better than (I am assuming other baselines). \n* `Our method is applicable to different observation modalities` is vague. It is better to say `multiple observation modalities`\n* On line 297, the paper points to the appendix for the validation result of the first hypothesis. First, please reference the particular figure or table a reader should reference. Second, if an result is essential to validate the presented approach, it should be part of the main paper and not the appendix.\n* On line 314, the paper says `our approach effectively learns domain-invariant features that capture task-relevant information regardless of the sensory input.` This seems like over-claiming. Better to say that it does so `on multiple sensory modalities`.\n* In the paragraph starting line 315, results are talked about without referencing the actual table being referenced.\n* In line 336, Figure 5 is no longer the correct figure being referenced (It is again in the appendix, and is now Figure 10).\n* Line 126, I am assuming $S_{src}$ is the number of states covered in the source domain. This distinction from the source state space $\\mathcal{S}$ should be better clarified.\n* Emission functions used in line 113 should be defined earlier when defining observation spaces.\n* I understand that overcoming dynamics mismatch is not an aim of this paper, but related work should acknowledge that dynamics mismatch is one of the challenges in sim-to-real transfer, but clarify that this paper does not deal with it. The authors should reference some seminal papers in this approach. I refrain from suggesting any to avoid biasing the authors.\n* Line 156, ` ... while preserve detailed information ...` should be `... while preserving detailed information ...`\n\nQUESTIONS:\n* Main questions are about validity of results in paper. Those are clarified in the weaknesses section above. Sufficient answers to them would be good enough for me to recommend acceptance.\n* Assuming the dynamics of the environment is the same in source and target, could $ f(o^i\\_{src}), f(o^{i+1}\\_{src})$ be compared to $f(o^j_{tgt}), f(o^{j+1}_{tgt})$ (i.e. transitions instead of state action pairs)to get a similar objective to the one proposed in the paper? Or does the action/proprioception act as an oracle that can be used by the objective to decide which observations to align?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}