Emergent Dexterity Via Diverse Resets and Large-Scale Reinforcement Learning
Authors: Patrick Yin
Tyler Westenbroek
Zhengyu Zhang
Ignacio Dagnino
Eeshani Shilamkar
Numfor Mbiziwo-Tiapo
Simran Bagaria
Xinlei Liu
Galen Mullins
Andrey Kolobov
Abhishek Gupta
Venue: ICLR 2026 Poster (2026)
Keywords: Robotics; sim-to-real; reinforcement learning
Primary area: applications to robotics, autonomy, planning
TL;DR: We develop simple resetting strategies which enable off-the-shelf RL algorithms to scale to long-horizon dexterous tasks with zero task-specific engineering

Abstract: Reinforcement learning in massively parallel physics simulations has driven major progress in sim-to-real robot learning.  However, current approaches remain brittle and task-specific, relying on extensive per-task engineering to design rewards, curricula, and demonstrations. Even with this engineering, they often fail on long-horizon, contact-rich manipulation tasks and do not meaningfully scale with compute, as performance quickly saturates when training revisits the same narrow regions of state space. We introduce \Method, a simple and scalable framework that enables on-policy reinforcement learning to robustly solve a broad class of dexterous manipulation tasks using a single reward function, fixed algorithm hyperparameters, no curricula, and no human demonstrations. Our key insight is that long-horizon exploration can be dramatically simplified by using simulator resets to systematically expose the RL algorithm to the diverse set of robot-object interactions which underlie dexterous manipulation. \Method\ programmatically generates such resets with minimal human input, converting additional compute directly into broader behavioral coverage and continued performance gains. We show that \Method\ gracefully scales to long-horizon dexterous manipulation tasks beyond the capabilities of existing approaches and is able to learn robust policies over significantly wider ranges of initial conditions than baselines. Finally, we distill \Method \ into visuomotor policies which display robust retrying behavior and substantially higher success rates than baselines when transferred to the real world zero-shot. Project webpage: https://omnireset.github.io

Published as a conference paper at ICLR 2026
EMERGENT DEXTERITY VIA DIVERSE RESETS AND LARGE-SCALE REINFORCEMENT LEARNING
Patrick Yin
Tyler Westenbroek
Zhengyu Zhang
Ignacio Dagnino
Eeshani Shilamkar
Numfor Mbiziwo-Tiapo'
Simran Bagaria
Xinlei Liu
Galen Mullins
Andrey Kolobov
Abhishek Gupta
University of Washington
2NVIDIA
Microsoft Research
"Equal contribution
Drawer Insertion
Screwing Table Leg
7
Insertion
Robust Transfer Visuomotor Policy
Figure 1: Omn Reset automatically generates diverse reset states, enabling complex, multi- manipula- tion behaviors to emerge from large-scale reinforcement learning. Using the same generic reset procedure for each task, OmniResed learns robust, -pec-iie policies. The first row a drawer before wiggling it in. The second row shows the robot picking up a table leg. adjusting its grip using the table, and twisting the leg into the hole. The final two rows show peg insertion in simulation and distilled RGB policy attempting the task on real robot, which recovers from failed insertion to successfully complete demonstrating robust emergent retrying behavior.
robot pushing flipping

---

Published as a conference paper at ICLR 2026
ABSTRACT
Reinforcement learning in massively parallel physies simulations has driven major progress in sim-to-real robot learning. However, current approaches remain brit- lle and task-specific, relying on extensive per-task engineering to design rewards, curricula, and demonstrations. Even with this engineering, they often fail on long- horizon, contact-rich manipulation tasks and do not meaningfully scale with com- pute, performance quickly saturates when training revisits the same narrow re- gions of state space. We introduce Omn i Reset, a simple and scalable framework that enables on-policy reinforcement learning to robustly solve broad class of dexterous manipulation tasks using single reward function, fixed algorithm hy- perparameters, no curricula, and no human demonstrations. Our key insight is that long-horizon exploration can be dramatically simplified by using simulator resets to systematically expose the RL algorithm to the diverse set of robot-object inter- actions which underlie dexterous manipulation. Omn iesstt programmatically generates such resets with minimal human input, converting additional compute directly into broader behavioral coverage and continued performance gains. We show that Omn Reset gracefully scales to long- long-horizon dexterous manipulation tasks beyond the capabilities of existing approaches and is able to learn robust policies over significantly wider ranges of initial conditions than baselines. Fi- nally, we distill Cmn Reset into visuomotor policies which display robust retry- ing behavior and substantially higher success rates than baselines when transferred the real world zero-shot. Project webpage: https:/ommiirset..hhhe
INTRODUCTION
Reinforcement learning (RL) in massively parallelized simulation environments (Mittal et al., 2023; Todorov et al., 2012) has driven recent successes in sim-to-real robotics (Akkaya et al., 2019; Hwangbo et al., 2019). In principle, these algorithms can automatically acquire complex, contact- rich behaviors through repeated interaction with the environment. Yet in practice, obtaining robust and performant policies still requires extensive per-task engineering, limiting the scalability of this paradigm particulllllll for long-horizon manipulation tasks. This sharply contrasts with domains such as language modeling, where simply scaling data and compute has yielded dramatic gains with similarly simple RI algorithms Guo et al. (2025). How can we achieve comparable scalability for robotic manipulation?
The central bottleneck in robotic RL is that standard exploration techniques Schulman et al. (2017); Haarnoja et al. (2018) saturate as parallelism and compute are scaled, repeatedly sampling a narrow state-action distribution and becoming trapped in local minima Singla et al. (2024). While numer- ous advanced exploration methods have been proposed Pathak et al. (2017); Burda et al. (2018), their added algorithmic complexity makes them difficult to scale in practice. As a result, practi- tioners often rely on human intuition to reduce the exploration burden, injecting structure through task-specific reward design Westenbroek et al. (2022); Ng al., hand-designed curricula Lee et al. (2020a), or user-provided demonstrations Bauza et al. (2025); Peng et al. (2018); Nair et al. (2018). Although effective in many settings, these approaches are fundamentally limited by the amount of human effort they require. A complementary strategy is to simplify the learning problem through additional system scaffolding-usinn
RL only for contact-rich phases while relying on motion plan- ning or trajectory optimization for the remainder Lee et al. (2020b); Tang et al. (2024). While this reduces exploration demands, it comes at the cost of increased system complexity. These approaches reflect a prevailing assumption in robotics: that dexterous manipulation is too complex to emerge from large-scale RI. alone and must instead be scaffolded with additional task-specific structure.
In contrast, we argue for systematically exposing RL to superset of the interactions it will encounter when manipulating the scene, and then allowing dexterous task-specific behaviors to emerge from large-scale compute and optimization. Although the space of possible behaviors is vast-flipping, screwing. insertion, and other contact-rich motions- catiions-uucesssiii successful policies reuse a relatively small set of recurring interaction modes, such approaching objects. initiating contact, and forming stable grasps. These modes can be densely covered through generic resets that do not encode task- specific solutions. By sufficiently randomizing object poses and sampling these interaction states,
2

---

Published as a conference paper at ICLR 2026
substantially reduce the exploration burden and ensure the agent encounters meaningful success signals. This coverage allows rewards to propagate smoothly through the state space, enabling the agent to identify high-value regions and learn how to stitch together multiple distinct behaviors to reach these goals, without task-specific shaping our guidance.
Concretely, we introduce Omn i Reset, scalable framework for robotic manipulation that automat- ically generates diverse initial-state distributions to densely cover the contact-rich interactions the robot may encounter. This coverage allows PPO Schulman et al. (2017) to fully leverage large-scale compute, improving performance as the number of parallel environments increases. When scaled sufficiently, OmniReset learns to combine multiple interaction modes-such as pushing, flipping, and insertion (Fig. i)-into coherent, multi-stage strategies without task-specific reward shaping, curricula, or demonstrations. Across diverse contact-rich tasks, this approach solves long-horizon problems that are out of reach for existing methods, producing robust policies that succeed from wide range of initial states rather than narrow distributions typical of baseline approaches. Finally, we demonstrate that Omn i Reset can be used to train visuomotor policies via student-teacher dis- tillation which robustly transfer to the real-world and substantially outperform real-world baselines.
2 RELATED WORK
Exploiting Resets and in Reinforcement Learning: Exploiting simulator resets for RL a natural idea which has been explored in many contexts. Prior theoretical works (Kakade & Langford, 2002) have suggested more uniform sampling over initial states, but do not provide practical algorithms. The primary focus of many works is to make learning more tractable by generating an explicit curriculum over resets(Tang et al., 2023; Dennis et al., 2020; Bauza et al., 2025), for instance through nrevers--curriculmm of states going backwards from the goal (Florensa al., 2017) using learned dynamics model to propose viable resets (Edwards et al., 2018; Ivanovic et al., 2019). In contrast, second category of methods leverage demonstrations (whether human generated) to generate feasible pathways to the goal (Tao et al., 2024; Resnick et al., 2018; Salimans & Chen, 2018; Bauza al., 2025; Tang et al., 2024). In contrast these prior works, we show that neither human demonstrations, nor a curriculum is needed, but rather that the simple approach for generating diverse resets in Omn Reset naturally scales various nonghooizon manipulation problems without added algorithmic complexity.
automatically
Exploration Strategies for Reinforcement Learning: RL practitioners have designed a variety of exploration strategies to effectively uncover goal-reaching paths for fixed set of initial condi- tions, with uninformative rewards. major line of work is bonus-based exploration, where agents receive intrinsic rewards for visiting novel or unpredictable states. Count-based methods reward visits to rarely seen states (Ostrovski et al., 2017; Bellemare et al., 2016; Martin et al., 2017), while curiosity- based methods provide bonuses based on prediction errors (Burda al., 2018; Sancaktar et al., 2022: Pathak et al., 2017). Other approaches (Osband et al., 2016; 2019; Russo et al., 2018) promote temporally correlated exploration by injecting stochasticity the policy value-function level. Finally, diversity-drives
methods optimize for behavioral variety (Eysenbach et al., 2019; Rajeswar al., 2023). These approaches are complimentary our work; our main contribution is demonstrating that large scale-scale parallelization and resetting schemes lead to the emergence of surprising levels of dexterity without the need for advanced exploration incentives.
Leveraging Demonstrations: An alternative is to increasingly rely on human demonstrations and imitation learning to overcome difficult hong-horizon exploration. Approaches include adding aux- iliary BC loss terms to RL objectives (Nair et al., 2018; Hester et al., 2018; Rajeswaran et al., 2018), simply adding demonstrations to the replay buffer (Vecerik et al., 2017), and introducing reward shaping terms which encourage RL. agent to follow demonstrations (Tang et al., 2024; Reddy et al., 2019; Koprulu et al., 2024; Peng et al., 2018). Other works have sought to squeeze more information out demonstrations by automatically translating existing demonstrations to new initial conditions and scenes (Mandlekar et al., 2023) or by robustifying BC policies by promoting recovery behavior (Ke et al., 2023; Ankile et al., 2024). While our work complementarily pushes the limits of what behaviors can be learned entirely from scratch, we expect that demonstrations (when available) can also be incorporated into our framework to further accelerate learning.
3

---

Published as a conference paper at ICLR 2026
Collect Reset States
2
Train RL Expert
Distill to Perception
Zero-Shot Transfer Real Robot
Reaching
Near Oiject
State Base RL with Diverse Resets and Massively Parallel Simulation
Large Scale RGB Dataset with Visual Randomization
Robust Palicies With Zaro Real-World Data
Grasped
Near Coal
Figure 2: Sim-to-Rew Pipeline with Oesit (1) After generating partial assemblies and grasps from object, grasped, and near goal. (3) We then train state-based RL policy initialized from these reset states, which is used to (4) train student-teacher distillation to get a RGB policy. (5) By finetuning this RGB policy on a mix of simulation data and small set of real demonstrations, (6) we deploy the RGB based policy in the real world.
the simulator, (2) collect reset states: reaching,
3 GENERATING DIVERSE RESETS FOR LEARNING DEXTEROUS MANIPULATION
This section introduces Omn MmniRt,tt, scalable framework for robotic manipulation that automati- cally constructs RI. problems by generating diverse, manipulation-centrio reset distributions. Rather than relying task-specific curricula, demonstrations, carefully tuned rewards to guide explo- ration, Omn Reset exposes standard RI. algorithms to broad superset of key interaction states that would be rarely encountered under naïve exploration. When paired with large-scale simulation and compute, this reset design continually exposes the algorithm to diverse interaction states, pre- venting convergence narrow, suboptimal behaviors and enabling complex multi step manipulation skills to emerge from large-scale optimization.
3.1 PROBLEM SETTING
Reinforcement Learning Problem: We formalize the RL problems synthesized by OmniResse as a Markov decision process (MDP) defined by the tuple (S,A, 7, P), where s E denotes the state, E AA denotes the robot's action, s' ~ P(-|s, a) denotes the next state sampled from the transition dynamics, r is the reward function, y the discount factor, and 8g ~ p the initial state distribution. We optimize for the discounted sum of rewards: J(T) =pp--p 21= 7'r(Se, at)], where a (-|ss) denotes actions sampled from the policy. We focus our RL. training on compact state representations (i.e., Lagrangian states), only moving vision-based distillation for transfer to the real world (Sec. 5).
Task Scope and User Inputs: To effectively leverage task structure when automatically designing RL problems we make several practical assumptions. First, we focus on manipulating rigid bodies where the goal is to move single user-specified object to target configuration (potentially relative to other objects). For example, the assembly task in Figure 1 requires picking up a table leg, moving it to the desired hole, and screwing the pieces together-a long-horizon task requiring diverse behav- iors, yet framed as moving one object to a goal. Specifically, OmniReset requires the following input from the user:
Requirement 1 target object dar C 8 to be manipulated.
Requirement 2 A set of goal configurations G C S for Sar
Requirement 3 A workspace W C S for the robot.
At code level, this requires that the user identify slar in the environment definition, provide means for sampling goal states, and select and operating region for the robot (such as the area over table top). These pieces of information place minimal burden the yet nonetheless contain rich information about problem structure which OmniReset exploits to design easily solvable RL problems.
4

---

Published as a conference paper at ICLR 2026
Key Problem Parameters: OmniReset automatically generates the initial state distribution So with the right diversity which enables PPO to solve each of the tasks we consider with a single task agnostic reward function and zero tuning of algorithm hyper parameters.
3.2 AUTOMATICALL GENERATING RL PROBLEMS WITH DIVERSE RESETS
Reinforcement learning fails solve long-horizon manipulation tasks when training only encoun- ters a narrow slice of potential object configurations or robot-object interactions. To prevent this collapse, Reset systematically expands coverage along cover the space of pathways along which the target object can be transported to the goal by densely resetting far on the tabletop. at random point in the air, and at and near the goal G. This exposes the RL algorithm to difficult-tor toodssovee success signals, and allows these signals to smoothly propagate throughout the saae-sppce as the value function is updated. Next, we cover the different ways the robot can interact with gaar by resetting the arm in configurations where it is reaching towards gear, making contact with gar from a wide variety of points, and with stable grasps on far This exposes the RL algorithm to the different ways the robot can interact with g/or move it towards the goal. Altogether, this broad coverage enables the RL algorithm to discover high-value regions of the state space and the behaviors required to reach those states.
axes. First, we approximately
Specifically, we generate these resets as follows. First, we use the grasp sampler from Mittal et al. (2023) to caleulate a set of 1000 feasible grasp points on the target object star as depicted in Figure 3a. Next, we generate a set of feasible offsets for saor relative to the goal g. This is accomplished by spawning saar at G, and then applying small random forces dislodge the target from the goal, as in Tang et al. (2023). For example, for the insertion task in Figure 1, this process generates a continuum of relative configurations between the peg and the hole where the peg is only partially inserted. With these pre-computed quantities, we generate the following resets over the space of robot-object configurations:
1) Reaching Resets: S C S cap- ture states where the robot is mov- ing towards the target object. For our tabletop manipulation problems, this corresponds to resetting the target ob- ject at diverse poses on the table top, with the gripper spawned at random poses throughout the workspace W.
(a) Grasp Sampling, We display the grasp poses sampled the table leg. The grasp sampling ranges are broad, moderate, and narrow from left-to-right. Our methad uses the broad range.
SNO also spawns stor across the tabletop, but then resets the end effector to one of the pre-computed grasp points with a small random offset and randomly set the gripper to either be open or closed. This provides broad coverage over states where the robot is initiating contact with s star from a wide distribution of directions, providing coverage over both non non-prehen interactions and the initiation of stable grasps.
2) Near-Object Resets:
3) Stable Grasp Resets:
SG C S covers states where prehensile manipulation occurs and the robot has secured a stable grasp on stor We spawn far randomly in the air throughout W, then spawn the gripper at a feasible grasp point.
4) Near-Goal Resets: behavior such as insertion or twisting occur. We spawn glor at one of the pre-computed offsets from the goal G, then spawn the gripper to be in contact with ffr as with the Near-Object resets.
NG C provides dense coverage over near goal states where contact rich
Altogether, these different reset regions provide dense, approximate coverage over the space of path- ways to the goal, without prescribing a priori which behaviors are needed to solve the task. Indeed, as we see from our wide range of examples (Fig. 1) the RL algorithm is free to select completely dif- ferent pathways through the state space when solving different tasks, utilizing the reset states which useful while ignoring those which convergence the RL algorithm does not obtain stable grasps on the drawer, but instead repeatedly flips the drawer and then pushes it into the cabinet. In contrast, for the Leg Twi st ng tasks, the
not. For example, in the Drawer Insertion task, at
5

---

Published as a conference paper at ICLR 2026
robot picks the table leg, pushes it against the table obtain a more favorable grasp, and then twists the leg into the hole.
Practical Implementation: In practice it can be difficult to efficiently sample feasible resets which respect the contact constraints of the physics simulator. Sampling invalid initial conditions can lead to pathological and non-physical behavior which destabilize learning. Moreover, reset states must be sampled with minimal overhead to maximize GPU-paralletism. Thus, we first sample feasible resets during an offline phase, sampling proposed resets from the four regions defined above and rejecting invalid samples using a combination of collision checking and stepping the simulator for few steps allow object to settle. This yields four validated datasets corresponding to the four reset distributions described above: D DNO, D and DNG, During RL training, we sample from Uniform(D) where D D DNO U UUNNG We cache these resets on-GPU to ensure
efficient sampling during training.
3.3 ALGORITHMIC DECISIONS FOR RL TRAINING
Next we discuss key algorithmic decisions which are required for PPO to leverage the diversity of resets states and scale to the complexity of tasks we consider.
Task-Agnostic Reward Structure: Leveraging the design choices described above, we use a sim- ple, common reward function shared across all tasks:
Stt a.) Tauceess (5t) (St) + Treach (5t) + F'mooth (st, 0z) Fier (st).
(1)
Here, Tsuccess is a sparse binary reward indicating task completion, raist encourages minimizing the distance of the target object sa=a to the goal, Freach encourages the gripper to be near star, ssmooth penalizes large or rapidly changing actions, and Fiem penalizes unsafe or physically invalid states that trigger terminations. Importantly, this reward does not encode task-specific strategies all components and weights are kept fixed across experiments. We find that this generic structure is sufficient for stable training across diverse manipulation tasks, and performance largely insensitive to the precise weighting of individual terms. See the Appendix for additional details.
Scaling Parallel Environments: When combined with increasing reset diversity, we found that scaling the number of parallel environments used by PPO stabilized and accelerated learning. This enables Omn sese to obviate curricula and task dependent rewards, significantly reducing the amount of tuning which is required to solve a new task.
Asymmetric Actor-Critic: We asymmetric actor-critic approach (Pinto et al., 2017) for our learning architecture. The actor observations include history of the five previous time-steps for the state of the robot, the poses of objects the scene, and the previous actions taken by the policy. The critic takes in these observations as well as additional privileged parameters of the environment. We found that conditioning the actor on larger observation spaces led to less stable training and led us to only provide this information to the critic.
Generalized DependDe Exploration Noise: We employ the policy noise parameterization from (Raffin et al., 2022). In short, gSDE has separate prediction head which determines the gaussian exploration noise at each time-step and is conditioned on the features of the final layer the exploration
policy network. This approach enables the actor to learn different cemporally-coreeaate strategies in different regions of the state-space, crucial for solving heterogeneous multi-stage tasks.
4 SIMULATION EXPERIMENTS
We aim to address the following questions experimentally (Q1) Does outperffor baselines and scale to tasks beyond current methods? (Q2) How do our key design decisions affect performance? (Q3) Can our experts be used to generate diverse data for sim-to-real transfer?
4.1 TASK DESCRIPTIONS
We use the following tasks to demonstrate the effectiveness of the Omn Reset framework. The Leg Twisting tasks is replication of the the square_table task from (Heo et al., 2023), and involves screwing in a single table leg. The Drawer Insertion task is based on the task by the same name from (Heo et al., 2023) and requires inserting a drawer into dresser. The

---

Published as a conference paper at ICLR 2026
Peq Insertion task requires inserting a peg into hole. The Cube Stacking task requires stacking one cube on top of another with a desired orientation. The Wall SLide task requires non-prehensile motion to push block up against wall into desired orientation. Finally, the Cupcake Pl aeaeeeett task requires placing cupcake on a plate at a desired orientation. For each task, consider both Hard and Easy variants. For the Ha rd variants, the target object star is distributed on the table with x y coordinates in (æ, y) E [-0.2, 0.2] x [-0.15,0, 0, 15] and task-specific randomization over the goal position, whereas the Easy version of each task uses highly restricted set of initial conditions with (z, y) [0.1, 0.12] x 10.1, 0.12] with single fixed goal location. We refer the reader to our public code release for additional task specific parameters, but summarize each task below. Snapshots from the tasks are depicted in Figures and 4
Finally, to fully demonstrate the utility of Omn i Reset, we solve the Four Leg task depicted in Figure by a) training independent Omn Reset policies to screw in each of the four legs then b) using a simple scripting policy to switch between the policies to complete the overall long-horizon task. The demonstrations highlights how OmniReset can be combined with high-level planning to push the horizon tasks that can be solved to even greater extremes.
Figure 4: Additional tasks. Visualizations of the new manipulation tasks solved with OmniReset. From left to right: Four-Leg Table Assembly, Cube Stacking, Cupcake on Plate, Block Reorientation on Wall
4.2 SIMULATION EXPERT BASELINE COMPARISONS
We compare to the following baselines, which bootstrap learning with expert demonstrations, pro- viding them more prior information about how to solve the task than OmniReset. For each of the methods, we supply 100 demonstration generated by the final OmniReset policy, effectively giving the baselines access to the optimal behavior. The initial conditions for these demonstrations are drawn from the reaching region 5 which corresponds to the set of initial conditions the robot will encounter when solving the full task.
1) BC-PPO: We add a Behavior- Clonnig (BC) loss to the PPO objective to construct a baseline emblematic of numerous works combining BC and RL objectives (Hester et al., 2018; Rajeswaran et al., 2017). When training this algorithm, the environment is always reset from the reaching region SR which reflects the 'standard' reset distribution typically used solve such tasks.
2) Deep.Mmmic: We use Deep-imic-stylee reward augmentation (Peng et al., 2018) top of our generic rewards. During resets, a random demonstration is chosen, and the agent is reset from random point along the demonstration and is given an auxiliary reward which provides bonuses for following the demonstration.
3) Demo Curriculum: This baseline is constructed in the spirit of method such as (Bauza et al., 2025) and uses success- autocurriculum to samlple reset states from the demonstra- tions. We use PPO as the base RI. algorithm to ensure a fair comparison.
Learning Curves and Success Rates: We plot learning curves for the Hard variants of each task and the Easy variants of the Peg Insertion on, Leg Twisting, and Drawer Insert i on tasks in Figure 8. Success rates are reported for initial conditions sampled from the Reaching re- sets SR which correspond to states from which the robot must solve the full task. We see that OmniReset is able to consistently obtain high success rates on each of the tasks, substantially outperforming baselines. While the baselines of the tasks, they consistently struggle to scale to the wider distributions of initial conditions which defines the Ha rd variants. Figure 5 provides additional insight into the failure modes of baselines.
able to make some progress
the Easy variants
7

---

Published as a conference paper at ICLR 2026
Twisking [Beay] Reaching Rlssets
Log Tokkting (Caay) Near Goal Rlssets
40
330
60
130
760
140 311
(Oung)
Figure 5: Success Rates on Different Stages of Task. We plot the success rates for the Leg Twi st ng task when starting from states that are in the Near-Goal region and also the Reaching Region of the state space. When evaluating these success rates, we sample resets from the demonstrations used by the baseline algorithms to ensure the resulting policies start from in-distribution states. We see that the baselines can achieve moderate success rates when starting close the goal (Near Goal), but struggle to make meaningful progress the full long-horizon task (captured by the reaching resets).
Here we plot success rates during training for initial conditions drawn from the demonstrations which fall into the Near-Goal and Reaching regions of the state-space. This plot demonstrates how the baselines are able to solve part of the task (i.e. when starting near the goal) but fail to scale to the full long-horizon task (i.e. starting from a reaching position).
We conduct a robustness analysis on the learned policies (Fig 10). We sample an initial condition from one of the demonstrations, perturb the initial condition with forces of different magnitudes and report policy success from these perturbed initial conditions. We find that baseline performance quickly degrades under small perturbations, while the performance of OmniReset is barely affected under large perturbations.
Robustness of Policies:
We also analyze the ability of policies to solve the task from a wide range of initial conditions in the scatter plots in Figure (Fig 9). These plots show success rates over various initial conditions for both Omn Reset Demo Curriculum (the most successful baseline) on the Leg Screw Easy task. For each plot we show the success rate over 1000 sampled initial conditions from the full task distribution. This plot demonstrates how the baseline struggles to achieve achieve consistent success across the distribution of initial conditions it was trained while Ressitestse is able to succeed across the entire workspace.
4.3 ABLATING KEY DESIGN DECISIONS
We ablate 1) the number of parallel environ- ments (and PPO batch size) and 2) the range of reset randomization used by Cmn Reseton the Leg Twisting Hard task. Figure/ ablates the number of parallel environments and shows success rates during training from the four different reset distributions used by Reset. The reset distributions in Figure 7 are roughly ordered from left to right from the end of the task (Near-Goal) to the beginning of the task (Reaching). While runs with a smaller number of environments can make progress at solving the task from Near-Gool states, we ob- serve that a large number of parallel environ- ments are essential for scaling to the complexity of the full multi-stage task (i.e. from reaching states). Similarly, we see that increased diver- sity in the grasps used by OmniReset has substantial effect on sample efficiency, highlighting how densely covering the different modes of robot-object interactions is crucial for efficient and scalable RL training.
Success Entrrre Task
Seralie harge
Figure 6: Ablation on grasp sampling range. For this ablation on the screwing task, we find that training RL narrower grasp sampling ranges leads to worse sample efficiency and lower converged success rate.
8

---

Published as a conference paper at ICLR 2026
65536 environmetts
32766 environments
16984 environments
8192 endrenemnts
4096 environments
Success From Near-Goal Resets
Success from Near-Object Resets
Success Entire Task
Grasped Resets
6.8-
9,8
0.8
Success
R.6-
2.5-
0.5
0.6
0.4
1000
3250
1550
2000
1360
3800
Updatess
Up-atess
Figure 7: Ablation on number of environments. We plot the success rates over course of RL training using different number of environments. We find that the number of environments significantly impacts training performance.
Peg tracrtion (Mard)
Twoding (Hand)
Drawer Insertin (Hard)
Updats
Upditte
Updktes
Cupoake Pladement (Hard)
Cubee Stacking (Hard)
Wall Slide (Hir)))
Uppaae
Updittea
Upddaes
Pog Iracction (tur))
Terading (baxy)
Dranier Insertion (baa3)
Updars
Upditez
Updder
Orafficdd 1DL
BC-FO
Figure 8: Success rates during RL training. We plot success rates over learning process for tasks described in Sec.4. We see that scales to task where baselines struggle to make meaningful progress, espe- cially with the wide range of initial conditions specified in the Hard variants of tasks.
5 DISTILLATION AND REAL- WORLD TRANSFER
We demonstrate the utility of our learned setta-genrrtion policies by distilling them into visuomotor policies deployable directly on hardware from RGB inputs. The robot observes 224 224 images from three cameras (front, side, wrist). Using the photorealistic rendering capabilities of LsaacLab (Mittal et al., 2023), we collect 10,000 expert rollouts with synchronized images and actions for stan- dard student-teacher distillation Chen et al. (2021). The student policy uses ImageNet-prect ResNet- 18 encoder and a Gaussian MLP head conditioned on the five most recent observations.
Visual randomization. To mitigate the sim-to-real visual gap, apply extensive domain random- ization following DextrAH-G (Singh et al., 2025), varying lighting, backgrounds, object and robot appearance, and workspace textures. Camera extrinsics are calibrated to the real setup with addi- tional pose and FOV jitter for robustness. We also apply standard image augmentations including color jitter, blur, grayscale, and noise.

---

Published as a conference paper at ICLR 2026
Demo Curriculum, Narrow Distribution
Demo Curriculum, Wide Distribution
Ours, Wide Distribution
Goai Gasy Resst Disiibution
622
Figure 9: Success states of RL policy. For the screwing task, we plot the xy configurations from which RL polices trained with Demo Curriculum and Omn i Reset succeed when trained on the full reset distribution and on a narrow reset distribution. We find that Omn ReniReesttuuceed from much broader range of initial conditions.
Screar ng Teek Sutccwse Dnechew Liseer Partarasaams
Drawer Task Suczees Dedine Under Perturtatione
Pog Task Buccoss Croline Under Perturtalons
Deethe boc.ess
Orrefeeet Ovno
Deej Mimic
DemoCamiccum
2
Perturaation foice (b)
Pertarbation
Perforbation force (1)
Figure 10: Success rate over perturbations. We plot the decline in success rate (measured by ratio of success rate between no perturbations and current level of perturbations). We find that imniResetis robust to perturbations while performance of baselines drops significantly.
Dynamics randomization. To reduce the control gap, we calibrate kinematics to hardware and deploy the same task-space operational space controller in simulation and reality, with the policy predicting end-effector pose deltas that are converted to torques using identical Jacobian-based con- trol. We perform system identification of key actuator parameters (e.g., friction and delay) to match real joint behavior, then randomize controller gains and physical parameters around identified values during RL training with a stability curriculum. We additionally apply an action-scaling curriculum encourage smaller motions and randomize object mass and friction to promote contact robustness.
Real-World Transfer.
We successfully deploy distilled Omn i Reset policy for the Peg Insertion Task, depicted in Figure 1. Evaluating our distilled RGB policy trained on 10,000 simulation trajectories zero-shot in real achieve a success rate of 25% from a wide range of initial conditions. We compare this Diffusion Policy Chi et al. (2025) trained on 100 demonstrations, which achieves only 4% success rate. Qualitatively, the Omn Reset visuomotor policy displays robust retrying behavior and able recover and successfully insert the peg even after initial failed attempts caused by the sim-to-real gap. Altogether, this demonstrates how Omn i Reset provides a foundation for scalable data generation for robust sim-to-real policies which are able to solve tasks from much wider ranges of initial conditions than existing methods.
CONCLUSION
In this work we presented Omni Reset, a simple and scalable system for data generation in simu- lation for complex, dexterous tasks. The primary insight in OmniResetis showing that a diverse, minimally structured set of reset states paired with large-batch on-policy reinforcement learning in simulation can lead to the emergence of surprisingly complex dexterous behavior. We provide general purpose recipe to instantiate data generators across a variety of manipulation tasks, and demonstrate both the efficacy of this paradigm in simulation and it's ability to bootstrap real world policy learning. The presented work is currently still single-task and object, making multi-task extension in future work a promising direction of investigation. Furthermore, we are excited to ex- tend Omn Reset to higher-dimension for generating grasps Xu et al. (2023). Furthermore, Omn Rennieeeepppn. up the possibility of more systematically studying scaling laws for policy learning.
settings such as dexterous hand using existing techniques
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We have made efforts to ensure reproducibility of our results by describing the steps of our data generation and training pipeline (Sec.3), our distillation and transfer pipeline (Sec.5), and our ex- perimental results (Sec.4). Additional ablation studies are provided in the Appendix.
REFERENCES
Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07110 2019. 2
Lars Ankile, Anthony Simeonov, Idan Shenfeld, and Pulkit Agrawal. Juicer: Data-efficient imitation learning for robotic assembly. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5096-5103. IEEE, 2024.
Maria Bauza, Jose Enriaue Chen, Valentin Dalibard, Nimrod Gileadi, Roland Hafner, Murilo Martins, Joss Moore, Rugile Pevceviciute, Antoine Laurens, Dushyant Rao, et al. Demostart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 6756-6763, IEEE, 2025. 3,
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1810. 2018.1 2,
Tao Chen, Jie Xu, and Pulkit Agrawal. system for general in-hand object re-orientation. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 297-307. PMLR, 2021. URL html. 9
wwweeediiingmm........mmmm.mm.mm mll. www.mp...sss//mmmmmmmmmmmmmm.sse /vlsss/vle 64 chen22a.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 44(10-1 110-1111:6644100000 2025. 10
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances neural information processing systems, 10-13049-13000 2020.
Ashley D Edwards, Laura Downs, and James C Davidson. Forward-backward reinforcement learn- ing. arXiv preprint arXiv: 100.2 2018. 3
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representa- fions (ICLR), 2019.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on robol learning, pp. 482-495. PMLR, 30
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025. 2
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. Pmlr, 2018.

---

Published as a conference paper at ICLR 2026
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J Lim. Furniturebench: Reproducible real- world benchmark for long-horizon complex manipulation. The International Journal of Robotics Research, pp. 02783649241304789, 2023. 6
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations, In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018. 3,7
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26))eaau5872, 2019.
Boris Ivanovic, James Harrison, Apoorva Sharma, Mo Chen, and Marco Pavone. Bare: Backward reachability curriculum for robotic reinforcement learning. In 2019 International Conference on Roboties and Automation (ICRA), pp. 15 25-21. IEEE, 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning. pp. 267-274, 2002.
Liyiming Ke, Yunchu Zhang, Abhay Deshpande, Siddhartha Srinivasa, and Abhishek Gupta. Ccil: Continuity- based data augmentation for corrective imitation learning. arXiv preprint arXiv:231012977, 2023. 3
Cevahir Koprulu, Po-han Li, Tianyu Qiu, Ruihan Zhao, Tyler Westenbroek, David Fridovich-Kee Keil, Sandcep Chinchali, and Ufuk Topcu. Dense dynamics-aware reward synthesis: Integrating prior experience with demonstrations. arXiv preprint arXiv:2412.01114, 2024. 3
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020a. 2
Michelle A Lee, Carlos Florensa, Jonathan Tremblay, Nathan Ratliff, Animesh Garg. Fabio Ramos, and Dieter Fox. Guided uncertainty-awarr policy optimization: Combining learning and model- based strategies for sample-efficient policy learning. In 2020 IEEE international conference on robotics and automation (ICRA), pp. 7505-7512. IEEE, 2020b. 2
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimiegen: A data generation system for scalable robot learning using human demonstrations, arXiv preprint arXiv:2310.15500 2023.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Mareus Hutter. Count-based explo- ration in feature space for reinforcement learning. arXiv preprint arXiv:1706.0000 080900 2017. 3
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg, Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 86):3740-3747, 2023. doi: 2,5
N0.1109//RA
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp.
IEEE, 2018. 2,
Andrew Y Ng al. Algorithms for inverse reinforcement learning. 2
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems (NeurIPS), 2016.
Ian Osband, Benjamin Van Roy, Daniel J. Russo. and Zheng Wen. Deep exploration via randomized value functions. J. Mach. Learn. Res., 20:124:1-124:6. 2019. URL attps: :/ /jmmr....////////////////// orgg papers/v20/11 339. html.
12

---

Published as a conference paper at ICLR 2026
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017. 3
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018. 3,
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542, 2017. 6
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. In Conference on robot learning, pp. 1634-1644. PMIR, 2022.
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron C. Courville, and Alexandre Lacoste. Mastering the unsupervised reinforcement learning benchmark from pixels. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 28699-28667.. PMLR, 2023. URL https: peatttss//proceeeee httc///oceedieeeeooooooo eigeeeddiig... ml prrsss/ htmm
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint
2017. 7
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018. URL https: / rxiv, orgg abss//////////// / 1709.1.. 1087,33 3
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint 1905.: 1008, 2019. 3
Cinjon Resnick, Roberta Raileanu, Sanyam Kapoor, Alexander Peysakhovich, Kyunghyun Cho, and Joan Bruna. Backplay:" man muss immer umkehren". arXiv preprint 1807v: 2018.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Found. Trends Mach. Learn., 11(1):1-96, 2018. doi: 10.1561/2200000070. URL https: / htt///oiooorrrr/ 10. 10e.1151122222 221.6//0222000000..
Tim Salimans and Richard Chen. Learning montezuma's revenge from single demonstration. arXiv preprint 18/2.0388/ 2018. 3
Cansu Sancaktar, Sebastian Blaes, and Georg Martius. Curious exploration via structured world models yields zero-shot object manipulation. Advances in Neural Information Processing Sys- tems, 35:24170-244 70-24183, 2022. 3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X.v:1707.06347, 2017. 2,
Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratliff, and Karl Van Wyk. Dextrah-rgh: Visuomotor policies to grasp anything with dexterous hands, 2025. URL https://arxiv:
arg//bbs/2412 01791. 9
Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg: split and aggregate policy gradients.
arXiv preprint arXiv:2407.202333 2024. 2
Bingjie Tang, Michael A Lin, Iretiayo Akinola, Ankur Handa, Gaurav S Sukhatme, Fabio Ramos, Dieter Fox, and Yashraj Narang. Industreal: Transferring contact-rich assembly tasks from simu- lation to reality. arXiv preprint arXiv:2305. 17/10, 2023. 3.
13

---

Published as a conference paper at ICLR 2026
Bingjie Tang, Iretiayo Akinola, Jie Xu, Bowen Wen, Ankur Handa, Karl Van Wyk, Dieter Fox, Gaurav S Sukhatme, Fabio Ramos, and Yashraj Narang. Automate: Specialist and generalist assembly policies over diverse geometries. arXiv preprint arXiv:2407.08028. 1(2), 2024. 2,
Stone Tao, Arth Shukla, Tse-kai Chan, and Hao Su. Reverse forward curriculum learning for extreme sample and demonstration efficiency in reinforcement learning. arXiv:2405.00339, 2024. 3
arXiv preprint
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint (oXivv1707.088/7 2017. 3
Tyler Westenbroek, Fernando Castaneda, Ayush Agrawal, Shankar Sastry, and Koushil Sreenath. Lyapunov design for robust and efficient robotic reinforcement learning. arXiv preprint
2622212008.672,,, 2022. 2
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang,
Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasp- ing via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4737-4746, 2023. 10
14


Reviews:
[{"review_id": "VQEojh3dYh", "reviewer": "Reviewer_DGvL", "summary": "This paper presents a simple method to enable RL to learn long-horizon manipulation tasks. The key idea is to initialize the scene in a diverse set of initial states relevant to the task. The authors argue that \"attempting to cover this space of behaviors may initially\nappear intractable,\" but \"the space of ‘reasonable’ manipulation behaviors is actually surprisingly small\". To this end, the authors propose resetting to reaching, near object, stable grasp, and near goal. A user aids the process by providing near goal states. The user needs to provide a set of goal configurations, a workspace for the robot, and a set of near goal states, including contact-rich and goal states.\n\nThe authors show results in 3 challenging versions of manipulation tasks - drawer, screw, and peg in simulation. Additionally, the authors show the transfer of the screw task to the real world. To achieve real robot results, a state-based policy is distilled into a vision policy. Additionally, the authors found that it's necessary to co-train this distilled policy with a small amount of real robot data.", "questions": "1. Line 372: \"standard reset distributions $ρ^S$ described above\" - what exactly is it, and which line is it defined in?\n2. What happens if we initialize around demo states with added perturbations? With enough perturbations, do you expect the state distribution to be sufficient and work similarly to the proposed method?\n3. Since the core idea is having sufficient reset coverage in relevant states, is it possible to gather them without human assistance? E.g., via disassembly and perturbations around it?\n4. How were the 100 real-world demos collected?\n5. Several related complementary works are cited in the Related Work section. What happens when Omni Reset is combined with some of them?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Impressive long-horizon manipulation task\n2. Empirically, significant improvement over baseline", "weaknesses": "1. Only one task is shown on the real robot. What was the reason for not showing real-world results on all 3 tasks?\n2. The authors claim \"OmniReset automatically generates these resets with minimal human input\". In that regard, I would expect to see results on more tasks.\n\n**Minor**\n1. Line 186: \"We will let $s ∈ S$ denote the state space\" -> \"We will let $s ∈ S$ denote the state\".\n2. \"transition dynamics of the simulator by $s′ ∼ P(·|s)$\" -> \"next state sampled from the transition dynamics of the simulator by $s′ ∼ c. P(·|s)$\"\n3. \"$a ∼ π(·|s)$ to denote control policies\" -> \"$a ∼ π(·|s)$ to denote action sampled fromm control policy\"\n4. Line 211: Suggestion $J(π) = E_{s_0∼ρ,a∼\\pi}[Σ_{t=0}^∞ γ^t r(s_t, a_t)]$ and remove \"expection is alos taken w.r.t. the actions...\"\n5. In Figure 2 caption & on line 342: space after OmniReset\n6. Line 359: Grammar \"Both setting the orientation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QTJOvu2PvX", "reviewer": "Reviewer_BWSv", "summary": "This paper introduces OmniReset, a simple and scalable framework for training reinforcement learning policies for complex, long-horizon, and contact-rich manipulation tasks.\n\nThe framework's key insight is to cleverly bypass the difficult exploration problem in RL. It automatically generates a large-scale, diverse distribution of reset states that covers all \"reasonable\" points along the path to the goal. This allows complex, dexterous behaviors to \nemerge naturally from standard large-scale RL optimization. OmniReset is designed to operate without human demonstrations, auto-curricula, or extensive task-specific reward/hyper-parameter engineering.\n\nThe method constructs its diverse initial state distribution(beta ρ) by composing four sub-datasets: Reaching Resets (D^R), Near-Object Resets (D^NO), StableGrasp Resets (D^G), and Near-Goal Resets (D^NG). This only requires minimal high-level user specifications: a set of goal configurations (G), a workspace bounding box (W), and a bounding box for near-goal/contact-rich states (NG).\n\nThe Experimental results show that OmniReset significantly outperforms demonstration-based baselines (BC-PPO, DeepMimic, Demo Curriculum) on Easy task variants and successfully scales to the Hard variants of Screw, Drawer, and Peg tasks, where baselines struggled to make meaningful progress. Furthermore, the resulting policies exhibit superior robustness to perturbations. Finally, the paper shows that policies trained in simulation can be successfully transferred to a physical robot through vision-based distillation and co-training with a small amount of real-world data.", "questions": "While the authors achieve strong results on grasping and insertion, can this design generalize to other tasks, and can you quantify the labeling effort (annotation cost) required to design the task-specific annotations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty in Solving Exploration: The main contribution is the proposal and validation of using a diverse, minimally structured set of resets as an alternative to complex auto-curricula or reliance on expert demonstrations to solve long-horizon RL exploration challenges.\n\n2. Scalability and Performance: OmniReset successfully enables RL (PPO) to master complex, contact-rich tasks (e.g., Screw Hard) that were previously beyond the reach of existing techniques. Achieving success rates of over 97% on these benchmarks is a significant \nperformance breakthrough\n\n3. Robustness: Policies trained with OmniReset show exceptional robustness to perturbations and succeed over a much broader range of initial conditions compared to baseline methods like Demo Curriculum.\n\n4. Effective Sim-to-Real Pipeline: The work demonstrates a practical and high-value use case by distilling the learned expert policy into a visuomotor policy that achieves a 30% real-world success rate after co-training with only 100 real demonstrations, vastly outperforming a policy trained only on the real demonstrations (0% success).", "weaknesses": "1. Shifts Complexity from Algorithm to Task Configuration: While the paper minimizes algorithmic complexity, it shifts the burden to non-trivial task-specific configuration requirements, which are the basis of the diverse resets. This challenges the central claim of \"minimal human input”\n\nNear-Goal State Bounding Box (NG): Defining the bounding box for contact-rich states requires expert knowledge about the precise geometry and interaction points necessary to solve the task (e.g., the threads and corresponding hole for the Screw task). This step essentially encodes a critical part of the task solution as a manual configuration, introducing a significant, task-specific engineering step.\n\nPre-computed Grasps: The method relies on a pre-computed dataset of feasible grasps from a sophisticated grasp sampler. This is a powerful but unstated dependency. For novel objects or robot hands where such a tool is unavailable, generating this grasp dataset would be a substantial engineering effort in itself, acting as a form of implicit prior knowledge that is not universally available.\n\n2. Uncertain Generalizability Beyond Assembly Tasks: The effectiveness of the proposed four-part reset strategy (R,NO,G,NG) is demonstrated on three tasks that, while complex, all fall within the category of rigid object assembly and insertion. The framework's generalizability to manipulation paradigms with different structures is unproven", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kt3678TNaX", "reviewer": "Reviewer_anU6", "summary": "This paper proposes a method to generate diverse initial states to enable more efficient reinforcement learning of long-horizon manipulation tasks. This is demonstrated on three contact-rich manipulation tasks, compared with baseline methods that utilize demonstration data. Sim-to-real transfer is achieved via distillation and cotraining with real data.", "questions": "1. The details of the baseline are missing, e.g., how many demonstrations are used? \n\n2. Given the success of imitation learning using a diffusion policy with demonstration, it is not clear to me why a pure diffusion policy wouldn't work with demonstration. Can the authors provide some explanation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed idea is simple and well executed on the proposed tasks.", "weaknesses": "I think the paper lacks enough evaluation to turn this into a really strong paper, for example:\n\n1. The number of tasks used is quite limited. Given the rich set of tasks introduced in furniture-bench, where two of the tasks are coming from, it would be nice to evaluate the method on more tasks and report how many tasks are successful. This can give an idea of the limitations of the proposed method and what follow-up work can do to improve the results. \n\n2. Four different sources of reset distributions are proposed. However, this paper lacks ablations to verify how important those distributions are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f1ru1LNr5k", "reviewer": "Reviewer_n5Rw", "summary": "This paper focuses on general manipulation skill learning through reinforcement learning in simulation. The authors propose a method using large-scale generation of reset states based on user-specified points of interest, eliminating the need for complex reward or curriculum design while improving training efficiency and robustness. Experiments conducted in both simulation and the real world validate the effectiveness of the proposed approach.", "questions": "1. How exactly are the goal configurations defined? What happens if defining them is difficult, for example, in the case of a dexterous hand?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper clearly identifies a critical problem in current reinforcement learning methods for learning general manipulation skills within simulation and proposes using large-scale, valuable reset states to constrain the exploration space.\n2. The paper is clearly written, concise, and includes a comprehensive evaluation.", "weaknesses": "1. The current tasks are not long-horizon enough. For example, real-world assembly often involves multiple-part assembly. How should the points of interest be designed in such cases? For a chair with four legs, the set of points could be very large, making sampling potentially time-consuming.\n2. The current experiments use only a robotic arm. How would the approach scale to a high-DoF dexterous hand? How can valid grasps be sampled? \n3. For manipulation tasks that involve interaction with the environment, such as using a wall to adjust an object’s pose, current method seems can not effectively restrict the exploration space.\n4. Since the points of interest may vary across objects, scaling to diverse objects might still require object-specific selection and significant user trial-and-error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)