Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy
Authors: Inkook Chun
Seungjae Lee
Michael Samuel Albergo
Saining Xie
Eric Vanden-Eijnden
Venue: NeurIPS 2025 poster ()
Keywords: stochastic interpolants
test-time compute scaling
generative robot control
diffusion policy
flow matching
robotic manipulation
difficulty classification
adaptive computation
Primary area: probabilistic_methods
TL;DR: A framework that dynamically adjusts computational resources for robot controllers based on real-time task difficulty, reducing computation time by 2.6-4.4× while maintaining success rates, using the Stochastic Interpolant (SI) framework.

Abstract: Diffusion- and flow-based policies deliver state-of-the-art performance on long-horizon robotic manipulation and imitation-learning tasks. However, these controllers employ a fixed inference budget at every control step, regardless of task complexity, leading to computational inefficiency for simple subtasks while potentially underperforming on challenging ones. To address these issues, we introduce Difficulty-Aware Stochastic Interpolant Policy (DA-SIP), a framework that enables robotic controllers to adaptively adjust their integration horizon in real-time based on task difficulty. Our approach employs a difficulty classifier that analyzes RGB-D observations to dynamically select the step budget, the optimal solver variant, and ODE/SDE integration at each control cycle. DA-SIP builds upon the stochastic interpolant formulation to provide a unified framework that unlocks diverse training and inference configurations for diffusion- and flow-based policies. Through comprehensive benchmarks across diverse manipulation tasks, DA-SIP achieves 2.6-4.4× reduction in total computation time while maintaining task-success rates comparable to fixed maximum-computation baselines. By implementing adaptive computation within this framework, DA-SIP transforms generative robot controllers into efficient, task-aware systems that intelligently allocate inference resources where they provide the greatest benefit.

Dynamic Test-Time Compute Scaling in Control
Policy: Difficulty-waare Stochastic Interpolant Policy
Inkook Chun
Seungjae Leef
Michael Albergo
Saining Xie
Eric Vanden-Eijnden
New York University
Univeesitty of Maryland
Harvard University
$Capital Fund Management
Abstract
Diffusion- and flow-based policies deliver state-of-the-art performance on long- horizon robotic manipulation and imitation learning tasks. However, these con- trollers employ a fixed inference budget at every control step, regardless task complexity, leading to computational inefficiency for simple subtasks while potentially underperforming on challenging ones. To address these issues, we introduce Difficulty-Aware Stochastic Interpolant Policy (DA-SIP), a framework that enables robotic controllers to adaptively adjust their integration horizon real time based task difficulty. Our approach employs difficulty classifier that analyzes RGB-D observations to dynamically select the step budget, the optimal solver variant, and ODE/SDE integration each control cycle. DA-SIP builds upon the stochastic interpolant formulation to provide a unified framework that unlocks diverse training and inference configurations for diffusion- and flow-based policies. Through comprehensive benchmarks across diverse manipulation tasks, DA-SIP achieves 2.6-4.4x reduction in total computation time while maintain- ing task success rates comparable to fixed maximum compuuaatoomttttiionnn baselines. By implementing adaptive computation within this framework, DA-SIP transforms generative robot controllers into efficient, task-aware systems that intelligently allocate inference resources where they provide the greatest benefit.
(a) Training
Compute Efficiency Gains
Performance
Anmotatted dilfinully data
135
100%
100%
1012%
Speedup Factor
Performance (96)
3,22
DA-SIP
2.814
Dumorstaaiin sttition dlata
Medium
Hang
Squar Trmmport
Hang Gan
Puih
SquanSrrnipert
[b) Dynamic inference
Speedup
Max Compute
Performance
Compute (%)
Figure 1: Overview of the DA-SIP framework with computational efficiency gains and performance retention
Introduction
Diffusion 2 and flow-matching 35 models have recently shown strong performance in imitation learning for robotic policies, producing long-horizon action plans. By iteratively denoising trajectories (diffusion) or integrating probability-flow ODEs (flows), they capture the multi-modal distributions present in human demonstrations. These policies can be coupled with Vision-Language Models
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(VLMs) to form Vision-Lamguage-Actiinn models (VLAs) that map natural language goals to low-level control 7
Despite their success, existing controllers adopt uniform inference budget: every control cycle executes the same solver, step schedule, and interpolant-all chosen to handle the hardest subtasks. This wastes computation on easy subtasks while ignoring a key advantage of flow-based generative models: the solver type, step count, and ODE/SDE formulation can all be adjusted retraining.
time without
Large language models already employ adaptive computation: they more reasoning steps for diffi- cult queries and fewer for simple ones [8]. Physical manipulation tasks exhibit similar heterogeneity: in free space) can be planned with minimal computation, while
coarse motions (e.g., moving sub-millimeter placements demand more computational resources for precision.
To address this gap, we develop Difficulty-Awarr Stochastic Interpolant Policy (DA-SIP), which uses difficulty classification to dynamically allocate computational resources within the stochastic interpolant framework. Unobstructed object approaches receive minimal computational budgets, while precision manipulations command substantially higher budgets to ensure optimal performance.
TThis adaptive mechanism incorporates dataset of demonstrations annotated with subtask difficulty levels. These annotations are used train three complementary difficulty classifiers: lightweight CNN designed for efficiency, (ii) a few-shot vision- model (VLM) prompted with exemplar images, and (iii) few-shot VLM fine-tuned on our collected data. During each control cycle, the selected classifier evaluates the current RGB-I) observation and predicts a difficulty level.
Stochastic Interpolant
P
CNN or VLM dificullyy classifier
checely
Messured Diffioulty
Stochastic Interpolant (SI) Palkyy
Traniin volesty)
Human difiicully dataset
Interpslatt (Linazr, DVP
Human snnotation
Emnilomment
Nest Obsenation
Gee
traject
Robot trglctory dataset
Figure 2: High-level overview of our difficulty-awand aware stochastic interpolant policy (DA-SIP) frame- work. During training we choose (1) prediction target (noise, score, or velocity), (2) an interpolant (Linear, VP. GVP, etc.), and this configuration yields a single generative policy network that can perform both ODE and SDE integration. At inference time, learned difficulty classifier adaptively selects an inference configuration triple (step count, solver type, ODE/SDE formulation) based on the current state etatlnnnalingg context-dependent "System 1 vs. System 2" compute that maximizes success while minimizing latency.
To implement this adaptive computation effectively, we ground approach in the stochastic inter- polant (SI) framework [9], which unifies flow-based (ODE) and diffusion- based (SDE) generative modeling. This formulation enables critical capability: translating difficulty levels into optimized inference configurations. Specifically, each difficulty level determines an inference configuration (step count, solver type, ODE/SDE integration), allowing the system dynamically adjust computa- tional resources (Fig. 2h.
Contributions.
1. SI-grounded generative policy framework. We unify diffusion and flow policies under stochas- tic interpolants, exposing a spectrum of training and inference choices in continuous time.
2. Difficulty-awar adaptive compute. The difficulty classifier selects solver type and step count using a lightweight CNN, few-shot VLM, or fine-tuned VLM,, yielding a 2.6 6-4.4x compute reduction without sacrificing success rate.

---

3. Comprehensive benchmark study. We demonstrate that our sefficulty-guiidee approach reduces task completion time while matching the success rate of the maximum-come computte baseline, and we analyze the resulting latency-accuracy trade-off.
Related Work
Generative models Modern generative modeling techniques include diffusion models 1 10
flow matching [H5). and stochastic interpolants that unify both frameworks [9] I.. Diffusion models iteratively denoise data using score-matching objectives. while flow matching directly trains neural ODEs along flexible interpolation paths [3]. Recent distillation techniques further enhance computational efficiency [TH155]. The stochastic interpolant framework uncouples the interpolation trajectory from the choice of deterministic (ODE) or stochastic (SDE) dynamics, clarifying the design space and enabling independent control over sample quality, speed, and robustness.
Generative modeling for decision making Diffusion Policy adapts diffusion-based generative modeling to learn robot control policies from human demonstrations [16). Generating action se-
quences rather than single actions has demonstrated strong performance 17 Flow-matching policies
[18]. Moreover, distillation methods for control policies have been introduced [19 [20]. AdaFlow measures the multimodality of the action space in imitation learning by estimating the variance of possible actions at each state [21]. While it introduces adaptation by measuring multimodality in the action space, it contrast, our approach recognizes multiple dimensions of state difficulty and provides a more comprehensive adaptation framework built on the stochastic interpolant framework, adjusting the full inference triplet: step count, solver type, and ODE/SDE integration mode.
model the flow from noise to expert behavior through continuous transformations
addresses only single dimension of state complexity-variait
LLMs and VLMs for Robotic Control Recent work has integrated large language and vision- language models into robot control, creating Vision-LanguageAcciii Actiinn (VLA) models that map natural language and visual observations to control actions 7 22 23 SayCan 24 grounds LLMs by constraining language outputs to feasible robot skills, while Inner Monologue (25] enables closed- loop reasoning where environmental feedback guides task execution. Our difficulty-awarr approach in DA-SIP implements adaptive computation at the action generation level, conceptually similar to how LLMs allocate more "thinking" steps complex queries g but applied the visual control
domain.
3 Methods
3.1
Generating Flow Policies with Stochastic Interpolants
Flow- and diffusion-based policies transform standard Gaussian noise E e~N(0,11 into a sequence of robot actions x, xx o), where observations. Here do so within the stochastic interpolant framework [9) To this end, we first introduce the stochastic interpolant
I = a,X, + OiE, t [0,
(1)
where a decreases from ag = to on = and a, increases from do = 0 to σ 1, thereby interpolating between the action sequence In x. at = 0 and pure noise I E at 1. Associated with the stochastic interpolant
we define the velocity field
v(x, 0, = E[i, I = x,
(2)
where E[- I, o) denotes the expectation over x., conditional on I, x and fixed. This velocity can be efficiently learned by minimizing the loss
L.(v] = [ (I,, i, where the expectation is taken over x., E, and ~~ (O,
The score of the probability density of the interpolant (i.e. the gradient of the logarithm of this density) is given, via Stein's 's relation, as
s(x,t,o) = -o, 'EEE x
(4)

---

which can be learned by minimizing the loss
T [S] (1,, + o oi-ee e
(5)
Alternativlly. the score can be expressed in terms of the velocity s(x, t, o) o 02 (x, d₂
(2) and vice-versa as
Ty v0 agdt) at
v(x,t,o)=
s(x, 1, 0)
(6)
ag
These equations can be obtained by combining (2) and (4) with the relation x decomposing the conditional expectation of the sums into sum of conditional expectations, and solving for s(x, 4, o) or v(x, ().
= E I I =,
Finally, we introduce the reverse-time SDE:
dx.- v(X,,t,o) [ t, o) dt + Vw, dW
X1 E
where 2 0 is diffusion coefficient that can be adjusted post-training and W, is a reverse Wiener process. The probability distribution of the solution (7) conditional on o coincides at all times with the probability distribution of the interpolant conditional on o. In particular, the distribution of Xo is the target distribution p(x o), indicating that 7 can be used as a generative model of sequence of robot actions given observations. The unconditional case is recovered by dropping the conditioning on o.
In practice we instantiate the equations above using three stochastic interpolants with different choices of a and or:
Linear
Variance-Presserving (VP)
generalized VP (GVP)
a₁ = t,
0r 1 exp (-f B. ds
at = sin 2
0: = 1 - t.
= cos 211
exp (-3 3, ds
These stochastic interpolants correspond different ways interpolate between noise and action sequences. Linear interpolation provides a direct, computationally efficient path: VP interpolation, up to time rescaling, recovers the formulation used in diffusion models; and GVP's trigonometric formulation ensures analytical normalization (a + of 1).
3.2 Difficulty Classification Methods
We explore three approaches for classifying subtask difficulty from RGB-D observations, ranging from lightweight neural networks to Asionnlanguage models.
1. Lightweight CNN Classifier: A ResNet-18 backbone with classification head processes RGB-I) observations to predict difficulty levels. This supervised approach is trained on 300 annotated images per task (from total dataset of 2000 timesteps per task), providing efficient real-time classification with minimal computational overhead (~220ms per inference).
2. Few-Shot VL. Classifier. This zero-training approach leverages pre-trained vision-language models prompted with 1-3 exemplar images per difficulty category. The VLM classifies the current RGB-D observation by comparing
the provided examples, requiring no specifci training but incurring higher inference latency (~5000-1000ms),
3. Fine-Tuned VLM Classifier. We fine-tune pre-trained vanion-language model on our annotated difficulty dataset (300 training images per task). This approach combines the strong representa- tional capacity of VLMs with improved inference speed classification accuracy across tasks.
and achieves the highest
3.3 Adaptive Computation Allocation
At each control step, the difficulty classifier outputs a difficulty level d(o::) E (1,2,3) 2) for the current observation oz, This difficulty level is mapped to an inference configuration triple
(Ne, solver, type,) M(d(o:)),

---

where N₂ E {5, 10, 20} is the number of integration steps, solver, E (Euler, Heun, RK4) specifies the numerical integrator, and type, E {ODE, SDE) determines the integration mode. The mapping M is determined empirically by evaluating performance--fficiengg trade-offs validation set. The configuration triple then used by the Stochastic Interpolant Policy (SIP) to generate the next action sequence:
x ~ SIP Or; solver, type,).
Computational efficiency. For our three-level difficulty system, we map difficulty levels to config- urations that balance performance and compute cost:
Easy (d = 1): (N = 5, Euler, ODE) minimal computation for coarse motions
Medium (d = 2): (N 10, Heun, ODE) - moderate resources for approach phases
Hard (d = 3): (N 20, RK4, SDB) maximum computation for precision tasks
This adaptive allocation enables the system to use minimal resources for simple subtasks while reserving expensive high-fidelity integration for challenging manipulation phases.
4
Experiments and Results
We evaluate our adaptive flow policy framework across various robotic manipulation tasks to demon- strate its versatility and effectiveness.
4.1
Simulation Environments
Our evaluation spans diverse simulation environments:
RoboMinic: Benchmark suite for imitation learning in manipulation (Can, Lift, Square, Tool Hang tasks) [26]
Block Push: A non-prehensile manipulation task (from the Fetch suite) in which 7-DoF arm must push cubic block across a tabletop to target pose, requiring precise contact planning and continuous control
27
Push-T: Precision manipulation task requiring continuous control
28
Kitchen: Multi-stage environment with sequential task completion
29
Multimodal Ant: Locomotion tasks requiring complex coordination
30
These environments represent spectrum of complexity, from simple locomotion intricate multi- object manipulation.
4.2 Optimizing Solver Configurations for Diverse Robotic Tasks
For uniformity and fair comparison across baselines, focus velocity prediction, although framework also supports score prediction as an alternative training target 22.. Our experiments reveal that different robotic tasks require distinct solver configurations to achieve optimal performance, as summarized in Table Our adaptive computation mechanism draws inspiration from adaptive inference in neural networks and builds upon recent advances in flow-based policy learning [18]
20 31 33
Our systematic evaluation revealed a key finding: there is no single best configuration that universally outperforms others across all environments. Initially, we hypothesized that evaluating all possible training and inference configurations of the stochastic interpolant framework would yield one optimal configuration outperforming others across tasks. Instead, we discovered that different tasks require fundamentally different configurations to achieve optimal performance.
Simple manipulation tasks. Simple tasks like Can and Lift achieve near-optimal performance with minimal computation, with even 1-step configurations reaching 99-100% success. Even single-step flow model inference without distillation achieves the highest maximum performance, indicating that additional inference computation is wasted.
5

---

Table 1: Task characterisation and optimal configurations.
Block
Tool
Multimodal
Task
PushT
Lift
Can
Transport
Square
Push
Hang
Ant
Solver
Heun
Heun
Fuler
Fuler
Fuler
Fuler
Fuler
Fuler
Interpolation
VP
GVP
VP
VP
Linear
Linear
VP
VP
Integration
SDE
SDE
ODE
ODE
ODE
ODE
SDE
SDE
Inference
100
100
50
25
100
100
steps
Last step Success (%)
Tweedie
Tweedic
None
None
None
None
Euler
Euler
92.6
22.8
38.1
45.3
100.0
100.0
85.0
95.2
Each configuration is trained for 5,000 epochs with three random training seeds. The model uses a U-Net transformer architecture. During evaluation, we three inference seeds per training seed (ninc per configuration). Checkpoints are saved every 50 epochs, and we report the mean success rate of the final 10 checkpoints, with each checkpoint evaluated over 50 independent episodes. Training is performed on NVIDIA L40S GPUs.
Precision manipulation tasks. Push-T and Block Push demonstrate strong dependency on the inte- gration method. These tasks require substantially more computational resources because millimeter- precision actions produce meaningful differences in success rates. Heun approximation outperforms Euler by slight margin (.0.1%) consistently across different training and inference seeds, highlight- ing the importance of accurate numerical integration for high-precision control tasks.
Exploratory manipulation tasks. Most notably, Tool Hang and Multimodal Ant tasks achieve peak performance at intermediate step counts. Increasing from the optimal step count (50 for Tool Hang, 25 for Multimodal Ant) to 100 steps actually decreases performance by 0.5-2.9%, suggesting that excessive computation can be counterproductive. We attribute this phenomenon to the need for controlled stochasticity in exploratory manipulation- similar to threading needle, where some variation aids success but excessive randomness becomes detrimental.
Transport and placement tasks. Square and Transport tasks represent a middle ground, requiring neither the minimal computation of simple tasks nor the extreme precision of complex ones. While they involve multiple sequential actions, they don't 't demand the millimeter-level precision or carefully controlled stochasticity of the more challenging environments.
This diversity in optimal configurations reinforces our understanding that different robotic tasks have distinct computational requirements, motivating adaptive methods that adjust computational resources based on task characteristics.
4.3 Categorizing Tasks by Difficulty
In all tasks, we identify distinct phases through which the robot progresses. When the robot is initialized far from interaction objects, we categorize this as the Initial state (I). As the robot approaches within approximately 10 em of object, designate this the Near state (N). Subsequently, the robot engages in one of three interaction types: Grasping (G), where the robot grasps and places object; Stochastic attempts (S), which require exploratory variability for precise placement; or Continuous pushing (C), where the robot pushes objects without grasping.
By categorizing tasks based on manipulation type to assign an appropriate difficulty level, we avoid need to find optimal configurations for each individual task, which would be impractical. We use standardized categorization system that can be applied uniformly across all tasks. This one-time categorization eliminates any additional per-task tuning at deployment.
4.4 Difficulty Classification Performance
We developed methodology to reliably categorize states across our environments. We collected annotations from different human labelers, accumulating approximately 20,000 labeled states across the six difficulty categories (I, N, G. S, C, E). Our ablation studies show that 300 images per task prove sufficient (Appendix E. For each state, we assigned the final label based on majority voting, selecting the category with the highest frequency of votes among annotators. This donsensus-driven

---

greb
lift
8IRO
grab
transport
piade
UFF
Eif
pre
place
(A)
(B)
(D)
Precision
Precision
Stochastic attempts
needed
Stochastic attompts
push
(G)
Figure 3: Robot manipulation tasks across complexity categories. (A-B) Simple manipulation tasks (Can and Lift) require minimal computational steps while maintaining high success rates. (C-D) Transport and placement tasks (Transport and Square) show greater sensitivity to configuration choices, representing mediun-complexity challenges. (E-F) Precision manipulation tasks (Push and Block Push) demonstrate significant benefits from Heun integration and variance-preservingg diffusion models for Multimodal Ant) highlight that adaptive resource allocation outperforms maximum computation, addressing complex challenges with varying difficulty levels.
control. (G-H) Exploratory manipulation tasks (Tool Hang and
approach helped mitigate individual biases while establishing comprehensive ground truth dataset. Additionally, we selected a small representative subset (I-3 images per category) for few-shot prompting with vision-language models (VLMs). We fine-tuned several VLMs on our dataset and selected the best-performing model for classification (Table
(a) Lightweight CNN accuracy
(b) Few-shot VLM average
(c) Qwen- VL 2.5 per-env.
Env.
Acc. (%)
VLM
Avg. (%)
Env.
Few-shot (%)
Fine- tuned (%)
Can
88,9
Lift
92.8
LLAVA
25.0
Can
37.5
47.5
Push-T
94.6
Gemma-3
23.0
Lift
83.3
86.7
Square
87.4
Phi-4
24.6
Push-T
42.0
56.7
Tool Hang
81.3
Qwen2.5
45.3
Square
26.0
52.0
Transport 91.5
Tool Hang
50.0
40.0
Qwen2.5
53.9
Transport
30.0
40.0
(Fine-tuned)
Table Difficulty classifier accuracy. We test different models 36 For CNN, our ablation stud- ies in Appendix E. show that approximately 300 images per task achieve near-plateau performance, requiring only about 15 minutes of annotation effort per task. The CNN classifier also demonstrates robustness to sensor noise (Appendix E.4). For the few-shot VLM, the impact of varying numbers of exemplar images is described Appendix E.3.
Our lightweight CNN classifier demonstrated the strongest performance across all environments in
both accuracy and inference time. Among the tested VI.Ms, Qwen-VI. VI. 2.5 achieved the highest average accuracy (45.3%) for few-shot multi-image prompting. We fine-tuned this VL.M for epochs, and fine-tuning with minimal number of images further boosted accuracy (Table 2
7

---

4.5 Assigning inference triplets based on difficulty label
We assign minimal computational resources (1 inference step, Euler solver, ODE integration) to Initial and End states, as these phases require basic positioning unobstructed space. The Near category receives inference steps with Euler solver and ODE integration, sufficient for approaching objects while avoiding collisions. For Grabbing interactions, we allocate 10 inference steps with Euler and ODE, as these do not require maximal computational resources. Stochastic attempts benefit from controlled variability, so we assign 50 inference steps with SDE integration, which produced optimal performance in the Tool Hang task. Finally, Continuous pushing, requiring the highest precision, receives 100 inference steps with Heun solver and SDE integration to achieve millimeter-preciss control.
Inference Steps
Category
Characteristics
Solver
Integration
Initial state (I)
Robot positioned away from targets
Euler
ODE
Near (N)
Approaching within 30cm of targets
Euler
ODE
Grabbing (G)
Grasping and placing objects
10
Euler
ODE
Stochastic attempts (S)
Precise alignment requiring variability
50
Euler
SDE
Continuous pushing (C)
Sustained precise manipulation
100
Heun
SDE
End state (E)
Objectives achieved
Euler
ODE
Table 3: State difficulty categories and their computational requirements
4.6
Adaptive Computation Efficiency
Based on our empirical results across task categories, we mapped state difficulty optimal computa- tional configurations for each task phase.
Overall, DA-SIP achieved substantial reductions in task completion time compared maximum computation baselines: an average 3.3-fold reduction with adaptive CNN, 2.7-fold with few-shot VLM, and 2.3-fold with fine-tuned VLM. Performance remained robust across most methods, with average performance differences of only 1.3%, 4.7%, and 1,8% from maximum computation, re- spectively. Notably, the fine-tuned VLM achieved an excellent balance between the computational efficiency of CNN-based classification and the deployment flexibility of few-shot VLM approaches. With minimal training data, it achieved consistent performance across environments, even demon- strating significant performance improvements (up to 8% in Tool Hang) while maintaining the adaptability benefits of vision-language models. These results confirm that dynamically allocating computational resources based on state difficulty substantially improves efficiency while preserving task performance, with fine-tuned VLMs offering particularly appealing balance between reliability and adaptability.
SIP
DA-SIP Few-shot VLM
Simulation
Min Compute
Max Compute
Lightweight CNN
Fine-tuned VLM
Tool Hang
91.02
342.13
106.37 (-3.22x)
150.32 (-2.28x)
139.53 (-2.45x)
Square
25.86
146.68
56.20 (-2.61x)
89.04 (-1.65x)
108.76 (-1.35x)
Transport
73.04
342.54
77.36 (-4.43x)
178.29 (-1.92x)
163.38 (-2.10x)
Push
1.44
121.89
45.35 (-2.69x)
21.96 (-5,55x)
45.34 (-2.69x)
Lift
57.04
263.28
64.31 (-4.09x)
101.00 (-2.61x)
89.50 (-2.94x)
Can
63.70
214.81
74.32 (-2.89x)
94.94 (-2.26x)
86.44 (-2.49x)
Table 4: Time comparison across different simulations showing computational efficiency gains. Values indicate computation time (seconds), with fold reduction relative maximum compute shown in parentheses as negative multiples (e.g.. -3.2x indicates 3.2x faster than maximum compute). The adaptive configurations achieve notable efficiency, with minimum compute delivering up to 84.6x reduction in the Push T environment, and Few-shot VLM offering up to 5.6x speedup. A detailed timing breakdown is provided in Appendix E.2.

---

SIP
DA-SIP Few-shot
Simulation
Min Compute
Max Compute
Lightweight CNN
Fine-tuned VIM
VIM
Tool Hang
25%
25%
27% (+2%)
33% (+8%)
33% (+8%)
Square
88%
90%
92% (+2%)
88% (-2%)
89% (-1 (1%)
Transport
62%
80%
77% (-3%)
62% (-18%)
79% (-1%)
Push T
81%
86%
87% (+1%)
86% (0%)
87% (+1 %)
Lift
100%
100%
100% (0% 9)
100% (0%)
100% (0%)
Can
98%
99%
99% (0%)
99% (0%)
99% (0%)
Table 5: Performance comparison showing generally minimal success rate degradation despite computational savings. Values indicate success rate, with percentage difference relative to maximum compute shown in parentheses. Most configurations maintain performance close to maximum compute, with three environments (Push T, Lift, Can) showing no performance loss with few-shot VLM approach. The CNN classifier maintains strong performance in Transport (-3% from maximum), while few-shot VLM shows more substantial performance tradc-off (-18%) in this environment.
Conclusion
We introduced DA-SIP, a Difficuly-Aware Stochastic Interpolant Policy that brings adaptive test-time computation to generative robot control. By casting diffusion and flow policies in the stochastic interpolant framework and attaching difficulty classifier, DA-SIP selects the solver depth, interpolant, and ODE/SDE mode on the fly. Across our simulated manipulation suite, this approach cuts total computation by 2.6 2.6-4.4x while matching the task success rates of fixed, uniform-budget baselines.
This work contributes (i) a unified SI formulation that exposes a rich design space for generative policies, (ii) an adaptive inference system that dynamically selects computational parameters without modifying the underlying policy, and (iii) empirical evidence demonstrating significant computational savings without performance degradation. While primary evaluation focuses simulation environments, preliminary experiments with real robot data show that SIP achieves lower mean squared error than diffusion policy action prediction (Appendix E.5), suggesting promise for physical robot deployment. Future directions include applying this adaptive inference framework larger robotics foundation models and coupling with safery monitors for contact-rich tasks. Together, these advances move generative controllers toward efficient, deployable autonomy in resource-constraaned physical environments.
Acknowledgments
We thank the authors of Stochastic Interpolant and the SiT for their paper and codebase, which served the foundation of our research (9 IT. We also grateful for the authors of the Diffusion Policy codebase [66,, which was instrumental for our implementation.
We thank Sanghyun Woo for valuable early discussions research methodology and comparative analysis, and Nur Muhammad Mahi Shafiullah for insightful discussions on robotics experiment design and ensuring fair comparisons with bascline methods. We thank Lerrel Pinto and Mark Goldstein for helpful feedback on the draft and experiments. We are grateful to the annotators, including authors, Manish Chowdhury, Elvis Fiador, Jiwon Hwang, Yeonwoo Kim, Bruna Sampaio, and Aditya Singh for their assistance with data annotation. We thank Nanye Ma for helpful discussions about the SiT concepts and codebase, and for valuable feedback on our research approach.
This work was supported by compute resources provided by NYU High Performance Computing resources and services.
We also thank the anonymous reviewers for their constructive feedback that helped improve this paper.
References
[1] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances Neural Information Processing Systems (NeurIPS), pages 6840-6851, 2020.

---

[2] Y. Song. C. Meng, and S. Ermon. Score-based generative modeling through stochastic differen- tial equations. In International Conference on Learning Representations (ICLR), 2021.
[3] Y. Lipman, R. T. Q Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.027477 2022.
[4] Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations (ICLR), 2023.
[5] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. orXiv:2209.03000 2022.
[6] Johan Bjorck, Fernando Casta neda, Nikita Cherniadev, Xingye Da. Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint 2rXiv:2503.14734 2025.
[7] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, et al. mo A vision-languaae-action flow model for general robot control. arXiv:2410.244 2024. Conference on Robot Learning 2024.
[8] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824-24837, 2022.
[9] M. Albergo, N. M. Boffi, and E. Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. 0rXiiv2303.08797, 2023.
[10] Tero Karras, Miika Aittala, Timo Aila, et Elucidating the design space of Liffusion-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[11] Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden- Fijnen,,,,nde,, and Saining Xie. SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision (ECCV), pages 23-40. Springer, 2024.
[12] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2033v2330..01469 2023.
Consistency models.
[13] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. meXiv:2410.125571 2024.
[14] Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden-Eijnden. Flow map matching. a7Xfv:2406.07507, 2024.
[15] Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden-Eijnden. How to build a consistency model: Learning flow maps via self-distillation, 2025. URL httpp:://arxv.orr//aaaaaaa 18825
[16] Chi, Z. Xu, S. Feng. E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. International Journal of Robotics Research, 2023. Originally presented ICLR 2023.
[17] Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv:2304. 13705. 2023.
[18] M. Braun, N. Jaquier, L. Rozo, and T. Asfour. Riemannian flow matching policy for robot motion learning. IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5144-5151, 2024.
[19] Aaditya Prasad, Kevin Lin, Jimmy Wu, et al. Consistency policy: Accelerated visuomotor policies via consistency distillation. 2rXiv:2405.07503 2024. 2024.
10

---

[20] X. HU, B. Liu, X. Liu, et al. RF-POLICY: Rectified flows are computation -aeputattioonaaatiieeee decision makers. International Conference on Learning Representations (ICLR), 2024, Earlier version: NeurlPS 2023 Workshop "Foundation Models for Decision Making".
[21] X. Hu, Q. Liu, X. Liu, and B. Liu. AdaFlow: Imitation learning with variance-adaptive flow-based policies. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[22] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 2023.
[23] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, al. VLe: An open-source Iasionnlagguage action model. axXi:22406.09246, 2024. Accepted at NeurIPS 2024.
[24] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169/ 2022.
[25] Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint SaxXiv:227.056600 2022.
[26] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, et al. What matters in learning from offline human demonstrations for robot manipulation. In Robotics: Science and Systems (RSS), 2021.
[27] Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning modes with one stone. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 22955-22968 2022.
[28] Pete Florence, Corey Lynch, Andy Zeng, Oscar A. Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168, 2022.
[29] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1910. 10.10.11950 11956, 2019.
[30] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint orXiv:/606.01544 2016.
[31] N. Funk, J. Urain, J. Carvalho, V. Prasad, G. Chalvatzaki, and J. Peters. ACTIONFLOW: Efficient, accurate, and fast policies with spatially symmetric flow matching. In RSS Workshop "Structural Priors for Learning Robot Dynamics", 2024.
[32] Q. Zhang, Z. Liu, H. Fan, G. Liu, B. Zeng, and S. Liu. FlowPolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 14754-14762, 2025.
[33] F. Zhang and M. Gienger. Affordance-based robot manipulation with flow matching. arXiv:2409.01083 2024.
[34] H. Liu, C. Q. Wu, and Y. Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 34892-34916. 2023.
[35] S. Bai, K. Chen, X. Liu, J. Wang, et al. Qwen 2.5-vl technical report. arXiv:2502. 139.233 2025.
[36] A. Kamath, J. Ferret, S. Pathak, N. Vicillard, and Gemma Team. Gemma 3 technical report. ar0iv:2503. 19786, 2025.
11

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The three key contributions stated in Section 1 (SI-grounded flow framework, diffieulty-wwar adaptive compute, and benchmark study) are thoroughly validated through
experiments in Sections 4-5.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are addressed in the Conclusion (Section 5), which mentions future work to "transfer DA-SIP to real robots, extend the scheduler to larger foundation models, and couple adaptive compute with safety monitors for contact-rich tasks."
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Section 3.1 provides detailed mathematical formulations for the stochastic interpolant framework, including explicit equations for velocity fields, score functions, and the reverse-time SDE with all necessary assumptions stated.
Justification:
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Tables 1-2 provide detailed task configurations, solver settings, and inference steps. Section 4.2 includes training details (5,000 epochs, three random seeds, checkpoint frequency), and evaluation methodology (nine runs per configuration, 50 episodes per checkpoint).
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
The paper uses publicly available benchmark environments (RoboMimic, Block Push, Push T. Kitchen, Multimodal Ant) with proper citations to their sources in Section 4.1 and the references.
Justification:
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Table specifies all solver configurations, inference steps, and integration methods. Section 4.2 states "Each configuration is trained for 5,000 epochs with three random training seeds" and details the evaluation protocol with "three inference seeds per training seed."
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
12

---

Answer: [Yes]
Justification: Tables 3-4 report comprehensive performance metrics across all methods. The paper states that results are based on "the mean success rate of the final ten checkpoints, with each checkpoint evaluated over 50 independent episodes."
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification: The footnote of Table | mentions "NVIDIA L40S GPUs are used to train the policy model." Additionally, we provide experimental configuration information in- cluding training epochs, seeds, evaluation protocol, and network architecture to ensure reproducibility.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp////oorrpp....p.....taaaaaa
Answer: [Yes]
Justification: The research focuses on computational efficiency for robotic control with no apparent ethical concerns. All cited works are properly attributed.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The Conclusion mentions that DA-SIP "transforms generative robot controllers into efficient, task-aware systems" which addresses positive impacts, but the paper could benefit from more detailed discussion of potential negative impacts.
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The model only controls simulated robots in standard research environments, presenting minimal risk. No deployment on physical systems that could pose safety concerns is discussed.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets, simulation environments, and baseline methods are properly cited in Sections 2, 4.1, and the comprehensive bibliography with 53 references.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Section 4.3 describes the creation of a difficulty classification dataset with "20,000 labeled states across the six difficulty categories" collected from "8 different human labelers," documenting the new asset created for this work.
14. Crowdsourcing and research with human subjects
13

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: Section 4.3 mentions data collection from different human labelers" but doesn't t provide detailed instructions compensation information. This could be expanded in the final version.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The human labeling described in Section 4.3 appears to be standard data annotation work with minimal risk, not requiring IRB approval.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Section 4.3 clearly describes the use of vision-language models (VLMs) for zero-shot and fine-tuned classification methods, including specific models (LLAVA, Gemma-3, Phi-4, Qwen2.5) with appropriate citations.
14

---

Appendix A
Implementation Details
This appendix provides key implementation details of DA-SIP framework.
A.1
Stochastic Interpolant Policy Implementation
Table 6: Stochastic interpolant policy hyperparameters
Parameter
Value
Parameter
Value
Optimizer
AdamW
Weight decay
1c-6
Learning rate
lc-4
Schedule
Cosine decay
Batch size
256
Gradient clipping Checkpointing
1.0
Training epochs
5,000
Every 50 epochs 0.9999
Loss function
MSE
EMA rate
Prediction targets
Velocity/Score/Noise
LR Scheduler
Cosine decay
Interpolants
Lincar/VP/GVP
warmup steps
500
A.2 Lightweight CNN Classifier
Our CNN classifier uses three convolutional blocks followed by fully connected layers:
Table Lightweight CNN architecture and training parameters
Architecture
Training Parameters
Input
RGB image 32 x 32 x 3
Optimizer Learning rate
Adam
Conv1
32 filters, 3 x 3, BatchNorm, ReLUU
le-3
MaxPooll
x stride 2
Batch size
64
Conv2
64 filters, 3 BatchNorm, ReLU
Epochs
100
MaxPool2
x stride
Early stopping
patience=15
Conv3
128 filters, x 3, BatchNorm, ReLU
Loss function
Categorical cross-entropy
MaxPool3
2 x stride 2
Class weighting
Inverse frequency
Flatten
2048 units
Validation split
20%
FCI
256 units, ReLU, Dropout(0.5)
Augmentation
Flip, rotation, color jitter
FC2
units, Softmax
We apply histogram equalization per color channel and use weighted random sampling to handle class imbalance with:
samples
weights
(8)
n. classes class counts
A.3 Few-shot VLM Classification
We few-shot in-context learning with the following configuration:
Models evaluated: QW 2.5, LLAVA-Next, Gemma 3
Method: Present reference images with labels alongside target image
Prompt: "simply pick the label from the sample images you see in prompt and tell me which one is closest to what you see"
initialize
s stochastic exploration
Category mapping:
near
end state
grab
c continuous pushing
15

---

A.4 Fine-tuned VLM Implementation
Table 8: VLM fine-tuning configuration
Model Configuration
Value
Training Parameters
Value
Base model
Dwen2.55-V--7B-lnsss
Learning rate
2e-5
Quantization
8-bit, NF4 type
Batch size
32
Language LoRA rank
16
Epochs
12
Vision LoRA rank
8
Optimizer
AdamW
LoRA alpha
32
Weight decay
0.01
Vision LoRA alpha
16
Warmup ratio
3%
LoRA dropout
0.05
Precision
FP16
Target modules
g///loogaae//updoww proj 224x224
Gradient checkpointing
Enabled
Image resolution
Max sequence length
200 tokens
16

---

Appendix B Performance comparison
Table Performance on the "Lift" task comparing DDPM, DDIM, and Linear (SI Policy with linear interpolation) under varying inference steps. We used Euler ODE approximator for trained models for SIP. Note that this without distillation for SIP
Inference Steps
Diffusion Policy (DDPM) Diffusion Policy (DDIM)
SI Policy (ours)
100
1.00
1.00
1.00
50
1.00
0.99
1.00
10
0.04
0.03
1.00
5
0.03
0.02
1.00
0.00
0.03
1.00
Table 10: Success rates the "Can" manipulation task: Comparing conventional diffusion methods (DDPM, DDIM) with our SI Policy using linear interpolation across different inference step counts. All models employ Euler ODE approximation without distillation.
Inference Steps
Diffusion Policy (DDPM) Diffusion Policy (DDIM)
SI Policy (ours)
100
1.00
1.00
1.00
50
0.97
0.98
1.00
10
0.00
0.00
1.00
I
0.00
0.00
1.00
Table 11: Representative PushT results with VP interpolation, using either Euler or Heun, in ODE SDE mode, and varying numbers of steps. Performance (success rate) improves with more sophisticated solvers, showing that each configuration must be carefully tuned for optimal results.
Interpolation
Steps
ODE SDE
Approximator
Final Step
Success Rate
VP
10
ODE
Euler
Euler
0.786
VP
25
ODE
Euler
Euler
0.897
VP
50
ODE
Euler
Euler
0,912
VP
100
ODE
Euler
Euler
0.918
VP
100
SDE
Euler
Linear
0.918
VP
10
ODE
Heun
Tweedie
0,881
VP
25
ODE
Heun
Tweedie
0,916
VP
50
ODE
Heun
Tweedie
0,920
VP
100
ODE
Heun
Tweedie
0.922
VP
100
SDE
Heun
Tweedie
0.926
Comparison of Diffusion Policy
Inference Steps
DP (DDPM)
DP (DDIM)
SI Policy (ours)
0.125
0.810
0.881
25
0.214
0.813
0.916
50
0,801
0.812
0.920
100
0.807
0.810
0.926
17

---

Table 12: BlockPush task success rate with GVP interpolation, comparing different approximators (Euler vs. Heun), ODE vs. SDE approaches, and final step methods. Results show that Heun with Tweedie correction achieves the highest success rates, while ODE performance scales with step count. The bottom section provides direct comparison with standard Diffusion Policy.
Interpolation
# Steps
ODE / SDE
Approximator Final Step
Success Rate
GVP
ODE
Euler
0.026
GVP
10
ODE
Euler
0.155
GVP
25
ODE
Euler
0.199
GVP
50
ODE
Euler
0.195
GVP
100
ODE
Euler
0.200
GVP
100
SDE
Euler
Euler Tweedie
0.204
GVP
100
SDE
Euler
0.205
GVP
100
SDE
Heun
Euler
0.215
GVP
100
SDE
Heun
Tweedie
0.227
Comparison with Diffusion Policy
Method
Configuration
Success Rate
Diffusion Policy (DDPM)
100 steps
0.210
SI Policy (ours)
GVP, Heun, Tweedie, 100 steps
0.227
Performance on the Transport task with VP interpolation, comparing different numbers of steps, ODE vs. SDE approaches, and approximators. Results show that performance is highly dependent on sufficient step count, with SDE approaches achieving slightly better performance.
Table 13:
Interpolation # Steps
ODE SDE
Approximator
Final Step
Success Rate
ODE
Euler
0.006
VP
25
ODE
Fuler
0.758
VP
50
ODE
Euler
0.796
VP
100
ODE
Euler
0.784
VP
100
SDE
Fuler
Buler
0.850
VP
100
SDE
Heun
Tweedie
0.850
Comparison with Diffusion Policy
Diffusion Policy (DDPM)
SI Policy (ours)
0.852
0.850
Table 14: Performance on the Square task with VP interpolation, comparing different numbers of steps, ODE vs. SDE approaches, and approximators.
Interpolation # Steps
ODE SDE
Approximator
Final Step
Success Rate
VP
10
ODE
Fuler
0.892
VP
25
ODE
Fuler
0.950
VP
50
ODE
Euler
0,942
VP
100
ODE
Euler
0.926
VP
100
SDE
Euler
Euler
0.952
Comparison with Diffusion Policy
Inference Steps
DP (DDPM)
DP (DDIM)
SI Policy (ours)
10
0.000
0.928
0.892
50
0,916
0.943
0,942
100
0.944
0.962
0.952
18

---

Table 15: Tool Hang performance with VP interpolation across different configurations (averaged
across 3 seeds)
Interpolation
Steps
ODE/SDE
Solver
Success Rate
VP
ODE
Euler
0,000
VP
10
ODE
Euler
0.115
VP
ODE
Fuler
0.345
VP
50
ODE
Fuler
0.381
VP
100
ODE
Fuler
0.351
VP
10
ODE
Heun
0.342
VP
25
ODE
Heun
0.330
VP
50
ODE
Heun
0.371
VP
100
ODE
Heun
0.364
VP
100
SDE
Euler
0.357
VP
100
SDE
Heun
0.368
Comparison with Diffusion Policy
Step Count
DP (DDPM)
DP (DDIM)
SI Policy (ours)
0.000
0.482
0,000
10
0.000
0.484
0.342
25
0.000
0.486
0.371
50
0.534
0.490
0.381
100
0.484
0.480
0.368
Note: Surprisingly, Diffusion Policy methods perform much better on Tool Hang than SI Policy consistently. DDPMM achieves the highest overall success rate (0.534) at 50 steps, while DDIM
shows remarkable consistency across all step counts, maintaining 0.48 success even at very low step counts.
19

---

Appendix C Data Collection and Annotation
This appendix details our methodology for collecting and annotating the dataset of 20,000 labeled robot states used for training and evaluating difficulty classification models.
We developed systematic approach to collect data of robot states across various manipulation tasks. Our data collection protocol was as follows:
C.1 Episode Recording and Frame Extraction
We recorded complete episodes across six simulation environments (Can, Lift, Push T. Square, Tool Hang, and Transport). We extracted frames at regular intervals (every 5th frame) with additional adaptive sampling to ensure representation of critical states (e.g.. precise grasping moments). This approach yielded approximately 20,000 frames for annotation.
C.2 Annotation Process
Eight volunteers participated in the annotation process. Each annotator received tutorial videos for how annotate each image for each task. custom web-based annotation platform was used to help them annotate and each volunteer given a distinct username.
C.3 Difficulty Categories
The following categories were used to classify robot states based on computational requirements:
1. Initial (I): The robot is positioned away from targets and objects, performing gross position- ing movements free space. No precise control is needed.
2. Near (N): The robot is approaching within approximately 10em of target object but has not yet initiated contact or grasping. Some care in motion planning is required.
3. Grabbing (G):
The robot is in the process of grasping an object, has grasped an object and moving it to a new location. Moderate precision is required.
4. Stochastic (S):
The robot is attempting a precise placement or alignment task that requires controlled variability (e.g., inserting a tool, threading a needle). High precision with some exploration needed.
5. Continuous (C): The robot is pushing or manipulating an object without grasping, requiring continuous fine control with millimeter precision (e.g., pushing block along specific path).
6. End (E): Task objectives have been achieved, and the robot is in terminal state moving away from completed objectives.
C.4 Dataset Finalization
To ensure reliability, each state was labeled by multiple annotators. For the final dataset, we assigned category labels using majority voting.
The most common boundary cases occurred between Near (N) and Grabbing (G) categories
These patterns reflect the continuous nature of difficulty transitions during task execution, with some states occupying boundary regions between categories.
The final dataset contains approximately 20,000 labeled states spanning all six difficulty categories across the different simulation environments, providing a comprehensive foundation for training our difficulty classifiers.
20

---

Appendix D
Robustness/Effiiinncy Analysis
D.1 Data Efficiency
To address the scalability concern of training the difficulty classifier, especially with the CNN approach, we conducted ablation studies to determine the minimum training data required to achieve comparable performance to our full dataset. We measured the accuracy of models trained on varying numbers of images, evaluating their predictions against majority-vor -vote labels from human annotators.
Table 16: CNN classifier accuracy with varying training data sizes. Test set sizes shown in parentheses.
Environment
Training Images
Test Set Size
100
200
300
500
2000
Square
8%
48%
84%
85%
84%
910
Tool Hang
2%
23%
19%
29%
39%
1584
Lift
87%
89%
90%
92%
92%
900
Can
2%
85%
83%
88%
520
Transport
1%
1%
36%
36%
37%
1584
Average
20.0%
32.6%
62.8%
65.0%
68.0%
The results show that 300 training images achieve near-plateau performance for Square, Can, and Lift, while Tool Hang peaks at around 200 images and Transport at 300 images. Therefore, approximately 300 images per task can be used as a reasonable baseline amount of training data needed train an accurate classifier model, especially when training from scratch with architectures such as CNNs.
D.2 Computational Overhead Analysis
We re-rolled out the policy for Tool Hang (88 iterations, per-iteration average):
Table 17: Per-cycle computational breakdown for different difficulty classifiers (in seconds)
Component
CNN
Fine-tuned VLM
Classifier inference
0.023
0.362
SIP policy inference
0.301
0.127
Simulation execution
0.977
0.991
Data processing
0.057
0.055
Total per cycle
1.357
1.535
D.3 Few-shot VLM Learning
To investigate the impact of the number of exemplar images on VLM classification performance, we tested few-shot learning with varying numbers of images per category. The following results demonstrate the trade-off between classification accuracy and inference time.
Table 18: Few-shot VLM performance and inference time with varying exemplar images
Environment
1 img/cat
2 img/cat
3 img/cat
Square
20% (0.55s)
20% (0.89s)
24% (1.26s)
Tool Hang
16% (0.56s)
32% (0.91s)
52% (1.26s)
Lift
57% (0.48s)
83% (0.64s)
83% (0.82s)
Can
25% (0.52s)
43% (0.75))
38% (1.05s)
Transport
25% (0.50s)
25% (0.72s)
33% (1.05s)
Average
28.6% (0.52s)
40.6% (0.78s)
46.0% (1.09s)
21

---

D.4 CNN Robustness to Noise
We also conducted additional environment evaluating the robustness of our CNN difficulty clas- sifier under noisy sensor conditions. We added varying levels of gaussian noise to test images before classifying. This addresses the concerns about real-world deployment where sensor noise unavoidable.
Experimental setup:
Training samples per environment: 300
Test set: 20% data (stratified split)
Random seed: 42 (ensures reproducible noise across runs)
Noise levels: {0.0, 0.05, 0.15, 2.03
Table 19: CNN classifier accuracy (%) under different noise levels
Environment
Clean
Low Noise
Medium Noise
High Noise (a=0.30)
Extreme Noise
(a=0.0)
(a=0.05)
(a=0.15)
(o=2.0)
Square
84.51
85.60 (+1.10)
83.08 (-1.43)
78.68 (-5.82)
12.86 (-71.65)
Tool Hang
20.52
20.27 (-0.25)
19.95 (-0.57)
19.63 (-0.88)
1.83 (-18.69)
Lift
91.67
90.67 (-1.00)
89.00 (-2.67)
81.89 (-9.78)
75.11 (-16.56)
Can
81.92
82.50 (+0.58)
82.12 (+0.19)
82.69 (+0.77)
77.69 (-4.23)
Transport
34.66
34.34 (-0.32)
34.60 (-0.06)
35.10 (+0.44)
22.85 (-11.81)
Average 62.66
62.68 (+0.02)
61.75 (-0.91)
59.60 (-3.06)
38.07 (-24.59)
This experiment demonstrates that our CNN models still maintain reasonable performance under high noise conditions (o=0.30) with only 3.06% average accuracy drop. a=2.0 represents extreme noise conditions beyond typical real-world scenarios.
D.5 Real Robot Training Validation
To address concerns about sim-to-real transfer, conducted experiment with the 'real robot pushT" that has publicly available training data. We were able to train diffusion policy and SIP with linear interpolation and velocity prediction and found a decreasing pattern of training action MSE, revealing that SIP can mimic the expert human demonstrator, which suggests it will be useful when deployed the real world. Due time constraints, we were unable to roll out actual robot trajectories.
Experimental setup:
Dataset: Real robot PushT human demonstrations
Policy configuration: SIP with linear interpolation and velocity prediction
Baseline: Diffusion Policy with identical training setup
Metrics: Training loss and action MSE over training steps
Table 20: Real robot training results
Training Loss
Action MSE
Step
Diffusion Policy
SIP
Step
Diffusion Policy
SIP
420
0,0760
0.183
420
0.0250
0.0136
2520
0,0238
0.085
2520
0.00582
0.00472
4614
0.0382
0.131
4614
0.00467
0.00294
22


Reviews:
[{"review_id": "hOmz34bChc", "reviewer": "Reviewer_XJPo", "summary": "This research introduces the Difficulty-Aware Stochastic Interpolant Policy (DA-SIP), a novel framework designed to improve the computational efficiency of robot controllers. Current state-of-the-art policies, based on diffusion and flow models, use a fixed amount of computational power for every action, regardless of the task's complexity. This can be inefficient, as simple movements receive the same resources as complex, precise maneuvers. DA-SIP addresses this by dynamically adjusting its computational budget in real-time. It uses a difficulty classifier, which can be a lightweight CNN or a Vision-Language Model (VLM), to analyze the robot's current situation from sensor data and determine how challenging the next action is.\n\nBy integrating this difficulty assessment with a flexible stochastic interpolant framework, DA-SIP can select the most appropriate solver, integration method, and step count for the immediate task. For instance, an easy motion like approaching an object might use a single computational step, while a delicate grasping action would be allocated a much higher budget to ensure success. This adaptive approach allows the system to conserve resources during simple phases and deploy them intelligently for more demanding parts of a task.\n\nThe authors validated DA-SIP across a variety of simulated robotic manipulation tasks, from simple lifting to complex placements. The results demonstrate significant efficiency gains, with the system reducing total computation time by a factor of 2.6 to 4.4 compared to traditional methods that always operate at maximum computational load. Crucially, this optimization was achieved while maintaining comparable or even slightly improved task success rates. The study concludes that adaptive computation is a highly effective strategy for making generative robot controllers more efficient and practical for real-world applications.", "questions": "1.  **Regarding the \"Last Step\" Method in Table 1:** In Table 1, the \"Last step\" for the *PushT* and *Block Push* tasks is listed as \"Tweedie\". Could you elaborate on what the Tweedie step refers to in this context and explain its function and importance for these specific precision manipulation tasks?\n2.  **On the Cost Function Formulation:** The compute-cost model is defined as $C(s_{t})=k_{t}\\gamma_{solv\\mathfrak{er_{t}}+c_{0}$, which accounts for the number of inference steps ($k_t$) and the solver type (Euler vs. Heun). However, the choice between ODE and SDE integration, which is a key part of the adaptive inference triplet, does not appear to be factored into this cost model. How is the computational cost of selecting an SDE over an ODE accounted for, or is its impact considered negligible?\n3.  **Design Rationale for the Lookup Module:** The mapping of difficulty categories to specific inference triplets in Table 3 (e.g., 'Stochastic attempts' mapping to 50 steps, Euler, SDE) is critical to the success of the DA-SIP framework. What was the methodology or heuristic used to design this lookup table? Was this mapping determined through systematic experimentation, or was it based on domain knowledge about the tasks and solvers?\n4.  **On Performance Degradation in Exploratory Tasks:** The paper notes that for exploratory tasks like *Tool Hang*, increasing computation from 50 to 100 steps surprisingly *decreases* performance. The \"threading a needle\" analogy is provided as a potential explanation. Could you provide a more detailed, data-driven explanation for this phenomenon? For instance, does excessive computation reduce beneficial stochasticity or cause the policy to overfit to specific, suboptimal trajectories?\n\n5.  **Investigating Classifier-Policy Mismatch:** The difficulty classifier is trained on human-annotated labels of \"difficulty,\" while the policy learns from robot trajectories. Is there a risk of a mismatch between what a *human perceives* as difficult versus what is *computationally challenging* for the policy? How sensitive is the overall task performance to classification errors (e.g., the 18% performance drop in the Transport task with the Zero-shot VLM)?\n6.  **Generalizability of Difficulty Categories:** The framework relies on six predefined difficulty categories (Initial, Near, Grabbing, etc.) derived from the evaluated tasks. How would this approach generalize to entirely new robotic tasks that may not fit neatly into this existing categorization? Would this require a full redesign of the difficulty labels and the lookup table, and what would that process entail?", "limitations": "### 1. Reliance on Simulation\nThe entire evaluation of the DA-SIP framework is conducted within simulated environments. This presents a significant limitation, as there is no evidence of how the system would perform on a physical robot. The \"sim-to-real\" gap can be substantial in robotics, involving challenges that are not fully captured in simulation, such as:\n* **Sensor Noise:** Real-world RGB-D cameras have noise, artifacts, and are affected by lighting conditions, which could degrade the performance of the difficulty classifier.\n* **Physical Dynamics:** The complexities of real-world physics, friction, and object interactions might require different computational allocations than what was found to be optimal in the simulated tasks.\n* **System Latency:** The reported computational speedups do not account for real-world system latencies, such as network communication and actuator response times, which could diminish the overall efficiency gains.\n\n### 2. Manual and Task-Specific Design\nThe framework's effectiveness appears to hinge on several manually designed components that may not generalize well to new tasks or environments.\n* **Fixed Difficulty Categories:** The system uses a predefined set of six difficulty categories (e.g., \"Initial,\" \"Grabbing,\" \"Continuous pushing\"). This discrete categorization was tailored to the specific manipulation tasks in the benchmark. For a new task that doesn't fit this mold (e.g., dexterous in-hand manipulation, pouring), this entire taxonomy would likely need to be redesigned.\n* **Hand-Tuned Lookup Table:** The mapping between a difficulty category and a specific computational triplet (inference steps, solver, and integration mode) seems to be based on heuristics and prior experiments rather than a learned or optimized process. This makes the system brittle; adapting it to a new robot, new objects, or new tasks would require a labor-intensive process of re-tuning this critical lookup table.\n\n### 3. Data Annotation Bottleneck\nThe best-performing difficulty classifier is a supervised lightweight CNN, which required a dataset of approximately 20,000 human-labeled states. This introduces a major practical limitation:\n* **Scalability:** The need to collect and manually annotate a large dataset for every new environment or set of tasks is a significant bottleneck. It undermines the goal of creating an efficient, easily deployable system.\n* **Subjectivity of Labels:** The \"difficulty\" labels are based on human consensus. However, what a human perceives as difficult may not perfectly align with what the robot policy finds computationally challenging, potentially leading to suboptimal resource allocation.\n\n### 4. Lack of Explicit Safety Mechanisms\nThe paper mentions that future work will involve coupling the system with safety monitors. This implies that the current framework lacks explicit safety guarantees. An incorrect classification from the difficulty module could lead to the system allocating insufficient computation for a critical, high-precision action, potentially resulting in task failure or unsafe behavior, especially in contact-rich scenarios. The system's performance is highly dependent on the classifier's accuracy, but there is no fallback or safety layer discussed if the classifier is wrong.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1.  **Novel and Relevant Problem Formulation:** The paper addresses a significant and practical problem in modern robotics: the computational inefficiency of generative policies. By framing the solution as dynamic, test-time compute scaling, it draws a creative and powerful parallel to adaptive computation in large language models and applies it effectively to the domain of robot control.\n\n2.  **Strong Theoretical Grounding:** The approach is built upon the Stochastic Interpolant (SI) framework, which unifies diffusion and flow-based models. This provides a principled and flexible foundation for the method, allowing it to manipulate various parameters like solver type, integration mode (ODE/SDE), and step count without retraining the core policy. This is far more elegant and robust than an ad-hoc or purely empirical solution.\n\n3.  **Comprehensive Experimental Validation:** The authors test their DA-SIP framework across a diverse suite of simulated manipulation tasks with varying characteristics (simple, precision, exploratory, and transport). This rigorous evaluation demonstrates the versatility of the approach and provides strong evidence that the benefits are not limited to a single, narrow task type.\n\n4.  **Thorough Exploration of Classifier Methods:** The paper thoughtfully evaluates three different methods for difficulty classification: a data-efficient lightweight CNN, a flexible zero-shot Vision-Language Model (VLM), and a balanced fine-tuned VLM. This analysis provides valuable insights into the trade-offs between performance, data dependency, and ease of deployment, which is crucial for real-world applicability.\n\n5.  **Clear and Impactful Results:** The research delivers impressive quantitative results, demonstrating a 2.6-4.4x reduction in computation time while maintaining, and in some cases even improving, the task-success rates of fixed, maximum-compute baselines. It also uncovers the counter-intuitive insight that excessive computation can sometimes degrade performance, adding a valuable new dimension to the understanding of these models.\n\n### Weaknesses\n\n1.  **Exclusive Reliance on Simulation:** The primary weakness is that all experiments are conducted in simulation. There is no validation on a physical robot, which leaves the critical \"sim-to-real\" gap unaddressed. Real-world challenges like sensor noise, unexpected physical interactions, and system latencies could significantly impact the performance and efficiency of both the difficulty classifier and the policy itself.\n\n2.  **Manual Design and Generalization Issues:** The framework's core logic relies on manually crafted components that may not scale or generalize easily. The six difficulty categories are task-specific, and the lookup table that maps these categories to computational settings appears to be hand-tuned. Adapting the system to new, unseen tasks would likely require a significant manual effort of re-defining categories and re-tuning this crucial table.\n\n3.  **Data Annotation Bottleneck:** The best-performing and most reliable difficulty classifier presented is a supervised CNN, which required a substantial dataset of 20,000 human-annotated states. This reliance on large-scale, manual annotation is a major practical bottleneck that hinders the framework's scalability and ease of adoption in new environments.\n\n4.  **Lack of Explicit Safety Guarantees:** The system's behavior is contingent on the accuracy of the difficulty classifier. An incorrect classification could cause the robot to use insufficient computation during a delicate maneuver, leading to failure or unsafe actions. The paper does not discuss or implement any safety layers or fallback mechanisms to mitigate the risks of such classification errors, which is a critical concern for physical deployment.", "was_revised": true, "final_justification": "I acknowledge the effort the authors made and decided to maintain my score", "rebuttal": "We sincerely thank the reviewer for the constructive review.\n\n## #1 Addressing the Weaknesses\n\n### 1. Exclusive Reliance on Simulation\n\n1. We conducted an experiment with the 'real robot pushT' that has publicly available training data. We were able to train a diffusion policy and SIP with linear interpolation and velocity prediction and found a decreasing pattern of training action MSE, revealing that SIP can mimic the expert human demonstrator, which suggests it will be useful when deployed in the real world. Due to time constraints, we were unable to roll out actual robot trajectories.\n\nTrain Loss Comparison\n\n| Step | Diffusion Policy | SIP |\n|------|------------------|-----|\n| 420 | 0.0760 | 0.183 |\n| 2520 | 0.0238 | 0.085 |\n| 4614 | 0.0382 | 0.131 |\n\nTrain Action MSE Comparison\n\n| Step | Diffusion Policy | SIP |\n|------|------------------|-----|\n| 420 | 0.0250 | 0.0136 |\n| 2520 | 0.00582 | 0.00472 |\n| 4614 | 0.00467 | 0.00294 |\n\n2. Simulation-based research has historically driven significant advances in robotics and RL. Notable examples include: DQN (Mnih et al., 2015) and PPO (Schulman et al., 2017), which established foundational RL algorithms entirely through simulated environments before widespread real-world adoption. Similarly, we believe our work introduces an initial exploration of test-time compute scaling for robotic control—a concept that may serve as a starting point for broader applications in real-world systems as the field continues to develop physical task reasoning capabilities in real world.\n\n### 2. Manual Design and Generalization Issues\n\n1. We believe this is a valid concern regarding generalization: when the user wants the robot arm to complete a new task, it may require an additional category beyond the 6 categories, and retuning the table. Before we delve into the details, we would like to point out that if there is in fact a new category, this will not require retuning of the entire table but simply adding a new row with a corresponding triplet for the category.\n2. We tested the generalization capability of VLM by fine-tuning it without Lift data. Without being exposed to Lift data, VLMs needed to make predictions on Lift:\n\n| Train Setup | Test Setup | Accuracy | Baseline |\n|----------------|------------------|----------|----------|\n| 6500 imgs (5 games excluding Lift) | 500 Lift imgs (never seen) | 46.4% | 33.3%  |\n\nWith the current rate of progress for VLMs, we believe the generalization ability will only improve.\n\n3. Moreover, we want to point out that this is precisely the reason why we introduced VLM instead of CNN as a difficulty classifier. In fact, we only used 3 images per category in the prompt before using the VLM. This means if the user wants the robot arm to behave differently during some sub-task, they only need to take three pictures per category to make that decision. We conducted an additional experiment to see the impact of differing numbers of images for VLMs:\n\n| Environment | 1 img/cat | 2 img/cat | 3 img/cat |\n|-------------|-----------|-----------|-----------|\n| Square      | 20%       | 20%       | 24%       |\n| Tool Hang   | 16%       | 32%       | 52%       |\n| Lift        | 57%       | 83%       | 83%       |\n| Can         | 25%       | 43%       | 38%       |\n| Transport   | 25%       | 25%       | 33%       |\n\nVLMs were still effective enough to show performance improvement and time savings as shown in tables 4,5.\n\n### 3. Data Annotation Bottleneck\n\n1. In our paper, we used a total of ~20,000 collected images to train CNN. We conducted an additional experiment to see the impact of a lower amount of training data on the same validation data for CNN accuracy:\n\n| training imgs | 100 | 200 | 300 | 500 | 2000 | Test Set |\n|-------------|-----|-----|-----|-----|------|----------|\n| Square      | 8%  | 48% | 84% | 85% | 84%  | 910      |\n| Tool Hang   | 2%  | 23% | 19% | 29% | 39%  | 1584     |\n| Lift        | 87% | 89% | 90% | 92% | 92%  | 900      |\n| Can         | 2%  | 2%  | 85% | 83% | 88%  | 520      |\n| Transport   | 1%  | 1%  | 36% | 36% | 37%  | 1584     |\n\nAs you can see, we could achieve accuracy level that tend to plateau around 100-300 images for the above games. If we decide to use only 300 images per task, it will only take ~15 minutes to collect (an average of 3 seconds to label one image), which we believe is a reasonable time. The above pattern of improvement also aligns well with Table 2(a).\n\n\n### 4. Lack of Explicit Safety Guarantees\n\n1. The reviewer raised an important point about safety. Our approach provides safety guarantees through its design. As shown in Table 5, misclassifications never result in performance below min-compute levels, since any classification yields at least 1 inference step. Users can set min-compute as their baseline safety threshold for their risk tolerance level.\n2. For deployment in high-stakes environments, we recommend the users to benchmark their policy across the full compute spectrum to identify the minimum inference configurations required for safe operation. This threshold becomes the min-compute threshold. Even under worst-case misclassification scenarios, the system maintains safety-critical performance levels.\n \n\n## #2 Addressing the Questions\n\n### Q1: \"Tweedie Step\" in Table 1\n\n1. The Tweedie step refers to a denoising method that directly estimates the clean state from noisy input using Tweedie's formula for posterior mean estimation. In the stochastic interpolant framework, this is implemented as:\n\nx̂₀ = (xₜ + σₜ² · score(xₜ)) / αₜ\n\nWhere:\n- xₜ is the noisy state at time t\n- σₜ is the noise scale\n- αₜ is the signal scale\n- score(xₜ) is the score function (∇log p(xₜ))\n\nThis can also be expressed using the noise prediction formulation as:\n\nx̂₀ = (xₜ - σₜε̂) / αₜ\n\nWhere ε̂ is the predicted noise, related to the score by: score ≈ -ε/σₜ\n\n\n2. For precision tasks like PushT and Block Push, the Tweedie step generates the final action in one computation. It also provides the minimum mean squared error estimate of the clean data. Finally, the direct estimation avoids accumulation of errors from multi-step integration. This makes it an efficient choice when high precision is needed with minimal computational cost, which aligns perfectly with our adaptive computation framework where we aim to match computational resources to task requirements.\n\n### Q2: ODE vs SDE Cost Consideration\n\n1. The reviewer correctly identified that our cost model doesn't explicitly account for ODE/SDE differences. In practice, the computational difference is minimal that both require the same neural network forward passes. SDE adds sampling overhead (≈5% additional time). Lastly, the main cost comes from the number of integration steps. We simplified the model for clarity but acknowledge this could be made more precise in future work.\n\n### Q3: Lookup Table Design Methodology\n\n1. The lookup table was designed through systematic grid search:\n- **Step counts:** {1, 4, 10, 25, 50, 100, 200}\n- **Solvers:** {Euler, Heun}\n- **Integration:** {ODE, SDE}\n\nWe evaluated all combinations on validation trajectories for each task type, selecting configurations that optimized the success-rate/compute-time balance. This one-time offline optimization provides practical deployment guidelines.\n\n### Q4: Performance Degradation in Exploratory Tasks\n\n1. To provide a data-driven explanation, we conducted a theoretical experiment of finding a hidden ball.\n\n- True ball position: [5.0, 0.0] (unknown to policy)\n- Ball radius: 0.5 units\n- Perception noise: σ = 0.5\n- Maximum attempts: 10 per episode\n\n```python\n    ...\n    def simulate_attempt(self, t, observed_hole, attempts):\n        ...\n        elif self.n_steps == 50:\n            last_attempt = attempts[-1]\n            if np.random.random() < self.adaptation_rate:\n                away_direction = observed_hole - last_attempt\n                away_direction /= np.linalg.norm(away_direction) + 1e-6\n                return observed_hole + away_direction * self.attempt_std + \\\n                       np.random.normal(0, self.attempt_std * 0.5, 2)\n            else:\n                return last_attempt + np.random.normal(0, self.attempt_std, 2)\n        else:  \n            # for 100 steps\n            return observed_hole + np.random.normal(0, self.attempt_std, 2)\n```\n\n**Result:**\n| Inference Steps | Success Rate | Avg Attempts | Cells Explored | Adaptation Score |\n|----------------|--------------|--------------|----------------|------------------|\n| 1              | 6.0%         | 4.7          | 560            | 6.90             |\n| **50**         | **83.0%**    | **3.4**      | **60**         | **1.07**         |\n| 100            | 58.0%        | 2.3          | 24             | 0.18             |\n\nThe experiment demonstrates that controlled stochasticity enables intelligent exploration, while 100 steps over-smooth the policy, causing it to repeatedly attempt the same failed approaches.\n\n### Q5: Investigating Classifier-Policy Mismatch\n\n1. We investigated this through our VLM experiments, which had lower accuracy than the CNN classifier, allowing us to observe the impact of classification errors. In Table 5, the performance never dropped below the min-compute baseline. This provides an important guarantee: if users can tolerate min-compute performance, they can safely use VLM-based difficulty classification, knowing this represents the worst-case performance floor. The 18% drop in the Transport task represents our most challenging scenario, yet, performance remains acceptable. We believe as VLM capabilities continue to improve, this classifier-policy gap will naturally narrow, making the approach increasingly practical while maintaining bounded performance degradation.\n\n### Q6: Generalizability of Difficulty Categories\n\n1. Please check #1.2\n\n\n## #3 Addressing the Limitations\n\n1. Please check #1.1.\n\n2. Please check #1.2.\n\n3. Please check #1.3.\n\n4. Please check #1.4."}, {"review_id": "Nz0vYm97Vc", "reviewer": "Reviewer_Ub6t", "summary": "The paper proposes a test-time scaling approach, Difficulty-Aware Stochastic Interpolant Policy (DA-SIP), which adaptively chooses the interpolation configuration during inference. Specifically, DA-SIP first trains a difficulty classifier to assign a difficulty score for each state per step, and then selects a predefined inference configuration based on the score. The authors conduct experiments on diverse manipulation tasks to show that DA-SIP can significantly reduce the total computation time while maintaining the performance as maximum-computation baselines, showing the potential of efficient policy deployment.", "questions": "There are some questions and concerns, which I have outlined in the previous section.", "limitations": "The authors briefly discuss the limitation in the conclusion. In addition, some questions listed in *Strength And Weaknesses* may need further discussion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "1. Originality: Test-time scaling is one important research area for large models. While various works have explored test-time scaling methods in LLMs, their application in robotic manipulation with diffusion- and flow-based policies is, to the best of my knowledge, less explored. As the authors remark, the SI-grounded flow framework is one of their contributions. I'm wondering if this is a direct application of stochastic interpolant on robotic policy learning or if there are any major differences compared to the original work.\n2. Quality: Applying stochastic interpolant on test-time scaling for diffusion- and flow-based policies seems a natural choice. The experiments clearly show that the proposed DA-SIP can achieve lower latency than max-compute baselines. However, I have the following questions:\n   1. Could authors justify why max-compute is a good baseline to compare with? As the author mentioned in Section 4.2, different tasks require fundamentally different configurations to achieve optimal performance, so max-compute does not seem a perfect choice in general. Moreover, Table 4 and Table 5 illustrate that max-compute achieves very similar performance as min-compute in many tasks (Tool Hang, Square, Lift, Can) but needs substantially more time. In these tasks, DA-SIP seems to use a much longer time than the min-compute baseline.\n   2. How is time measured? Is this the wall-time or estimated by the compute-cost model defined in Section 3.5? Is the overhead of the classifier included?\n3. Clarity: The paper is well organized and clearly written, with a sufficient introduction of the background.\n4. Significance:\n   1. The idea of using a classifier to adaptively adjust the inference configuration is reasonable. However, the current implementation does not seem scalable. It seems that both difficulty categories and their corresponding inference triplets are manually designed with some heuristics, and the training data is also labelled by humans. Therefore, it might be hard to extend to broader scenarios where more categories need to be specified and each category requires a grid search over the best inference triplets. I'm wondering if there is a scalable way to do this, such as clustering categories with some unsupervised methods.\n   2. One merit of test-time scaling in LLMs is that the performance keeps increasing when we scale up the inference budget. While the paper presents some impressive results on how DA-SIP can be used to save computation, I'm also curious about whether such a trend exists for robotics manipulation. In Table 1, it seems that Tool Hang and Multimodal Ant are not using the maximum budget to achieve the optimal result. Does this imply that increasing the computation budget cannot further boost the performance?\n\nOverall, while the work has made some interesting exploration in test-time scaling for robotic manipulation, some parts of its results need further clarification. Given the number of open questions, I am unable to recommend acceptance at this stage.", "was_revised": true, "final_justification": "The authors' explanation and results addressed most of my concerns, and I think the additional results are good complements to the current paper and should be included in the revision.", "rebuttal": "We sincerely thank the reviewer for the thoughtful analysis of our work.\n\n### 0. \"I'm wondering if this is a direct application of stochastic interpolant on robotic policy learning ...\"\n\n1. Yes, this is a direct application of the Stochastic Interpolant (SI) framework to robotic policy learning. Our work systematically investigates the flexibility in training and inference configurations of SIP especially applicable for the domain of robotic manipulation.\n\n### 1. \"Could authors justify why max-compute is a good baseline to compare with?\"\n\n1. We thank the reviewer for this insightful observation. We selected max-compute as a baseline specifically because it reflects the current practice in the field. To our knowledge, most diffusion/flow matching policy works employ either fixed max-compute configurations (typically 100 steps) or focus on distillation methods using 1-2 steps. Therefore, we adopted the commonly used min and max compute settings  with SIP as our baseline comparison. We will include the per‑task ‘best static’ configuration column in the appendix so that readers can directly compare DA‑SIP against both min‑ and best‑static baselines.\n2. Initially, we hypothesized that we could identify one single best configuration with max-compute for SIP that would maximize performance across all environments. However, our investigation in Section 4.2 revealed that this assumption was incorrect. We discovered that different tasks indeed require fundamentally different configurations to achieve optimal performance. Through careful analysis of rollout videos across multiple environments, we identified distinct sub-task characteristics that explain why each environment benefits from different configurations.\n\n### 2. \"Table 4,5 illustrate that max-compute achieves very similar performance as min-compute in many tasks (Tool Hang, Square, Lift, Can) but needs substantially more time. In these tasks, DA-SIP seems to use a much longer time than the min-compute baseline.\"\n\n1. This is true, and this observation actually validates our key finding. In simpler games like Lift and Can, when using linear interpolation with ODE in SIP, an inference step of 1 can achieve a near perfect score (which is not the case with Diffusion Policy as shown in tables 9 and 10). While existing methods typically require secondary distillation to reduce inference steps, our results demonstrate that our linear interpolation with ODE eliminates this need entirely for these tasks.\n\n2. For Square, max-compute yields only marginal improvements (2%) from min-compute while substantially increasing computation. Our dynamic classification approach addresses this inefficiency, reducing runtime from 146.68s to 56.2s using CNN classification while achieving +4% performance improvement. \n\n3. Tool Hang presents a unique case where increased computation fails to improve performance. Our analysis reveals that higher inference steps lead to trajectory oversmoothing and reduced exploratory behavior, both harmful when stochastic attempts are crucial for success. With our VLM classification, we can achieve 8% gain with -2.28x reduced run time.\n\n### 3. \"How is time measured? Is the overhead of the classifier included?\"\n\n1. All reported times are wall-clock measurements that include every component of the system:\n\n- Classifier inference time\n- SIP inference time\n- Simulation execution time\n- Data pre/post-processing time\n\nWe re-rolled out our policy for Tool Hang (cumulative run of 88 iterations):\n\n| Component | CNN Configuration | Finetuned VLM Configuration |\n|-----------|------------------|---------------------------|\n| Classifier inference | 1.98s | 31.90s |\n| SIP inference | 26.50s  | 11.17s  |\n| Simulation execution | 85.95s  | 87.19s  |\n| Data processing | 5.03s  | 4.82s  |\n| Total time | 119.45s | 135.07s |\n\n\n2. The theoretical model in Section 3.5 serves to provide analytical understanding of computational complexity rather than predicting actual runtime.\n\n### 4. \"... the current implementation does not seem scalable. It seems that both difficulty categories and their corresponding inference triplets are manually designed with some heuristics.\"\n\n1. We appreciate the reviewer's concern. The inference triplets in Table 3 were systematically derived from our comprehensive task-level analysis in Table 1. By analyzing failure modes across tasks, we identified that similar sub-task types (e.g., precision placement) benefit from similar computational configurations regardless of the specific task. This insight allowed us to map difficulty categories to appropriate inference settings.\n\n2. The robustness of our categorization is validated by strong consensus among 8 human labelers, and the 6 categories capture fundamental manipulation types (approaching, grasping, precision placement, etc.) that generalize across tasks. The high classification accuracy (Table 2) validates that these categories are meaningful and recognizable, not arbitrary.\n\n3. While the current mapping from categories to configurations is derived from empirical analysis rather than end-to-end learning, this demonstrates the viability of adaptive computation for robotics. Future work can learn these mappings directly, but our contribution—showing that adaptive computation yields 2.6-4.4× speedups—remains valid regardless of how the mapping is determined\n\n4. We also note that human labeled difficulty classifications are very well in line with precedent in influential robotics and RL research. Firstly, Diffusion Policy (Chi et al., 2023) requires extensive human demonstrations, and RT-1 (Brohan et al., 2022) used 130k human-annotated episodes. OpenAI’s Automatic Domain Randomization for the Rubik’s‑Cube hand still begins from expert‑set parameter bounds (OpenAI et al., 2019), and Automatic Curriculum Learning surveys catalogue multiple of works that first discretize task difficulty by hand before learning to select among bins (Portelas et al., 2020). Similar to them, our study provides a simple, interpretable method that enables systematic study and can later be refined or automated once its practical benefits are understood.\n\n### 5. \"... the current implementation does not seem scalable. ... the training data is also labelled by humans.\"\n\n1. Instead of using total ~20,000 collected images to train CNN, we conducted an additional experiment to see how lower amount of training data affects accuracy:\n\n| training imgs | 100 | 200 | 300 | 500 | 2000 | Test Set |\n|-------------|-----|-----|-----|-----|------|----------|\n| Square      | 8%  | 48% | 84% | 85% | 84%  | 910      |\n| Tool Hang   | 2%  | 23% | 19% | 29% | 39%  | 1584     |\n| Lift        | 87% | 89% | 90% | 92% | 92%  | 900      |\n| Can         | 2%  | 2%  | 85% | 83% | 88%  | 520      |\n| Transport   | 1%  | 1%  | 36% | 36% | 37%  | 1584     |\n\n\nMost tasks achieve near-plateau performance with just 300 labeled images, requiring only ~15 minutes of annotation (3 seconds per image). The above pattern of improvement also aligns well with Table 2(a).\n\n2. VLMs are used precisely to address the scalability issue in our paper. We only used 3 images per category in the prompt for VLM and used total 1049 training images to finetune VLMs. We conducted an additional experiment to see the impact of differing numbers of images for VLMs:\n\n| Environment | 1 img/cat | 2 img/cat | 3 img/cat |\n|-------------|-----------|-----------|-----------|\n| Square      | 20%       | 20%       | 24%       |\n| Tool Hang   | 16%       | 32%       | 52%       |\n| Lift        | 57%       | 83%       | 83%       |\n| Can         | 25%       | 43%       | 38%       |\n| Transport   | 25%       | 25%       | 33%       |\n\nWhile VLM classification accuracy remains below that of CNNs, VLMs were still effective enough to show performance improvement and time savings as shown in tables 4 and 5.\n\n3. We hypothesized that after finetuning the VLMs with enough data, they may learn generalization ability that they do not need to be finetuned with new data but are able to make meaningful predictions. We conducted an additional experiment to test generalization:\n\n| Train Setup | Test Setup | Accuracy | Baseline |\n|----------------|------------------|----------|----------|\n| 6500 imgs (5 games excluding Lift) | 500 Lift imgs (never seen) | 46.4% | 33.3%  |\n\nWe believe with the current rate of VLM progress, its generalization ability will only improve in the future. \n\n### 6. \"One merit of test-time scaling in LLMs is that the performance keeps increasing when we scale up the inference budget... I'm also curious about whether such a trend exists for robotics manipulation. In Table 1, it seems that Tool Hang and Multimodal Ant are not using the maximum budget to achieve the optimal result. Does this imply that increasing the computation budget cannot further boost the performance?\"\n1. This is an exceptionally thoughtful observation that also highlights a crucial finding of our work. Yes, generally, we believe there is a trend where increased compute budget increases performance at least marginally. However, the trend does not seem to apply for Tool Hang and Ant where controlled noise aids intelligent exploration.\n2. We show a systematic evaluation across multiple inference steps on Tool Hang, varying inference steps k ∈ {1, 4, 10, 25, 50, 75, 100} while holding solver and dynamics configurations constant. \n\n| Steps  | 1 | 4 | 10 | 25 | 50 | 75 | 100 |\n|-----------|---|---|----|----|----|----|----|\n| Success  | 25.0% | 28.3% | 31.7% | 36.2% | 38.1% | 32.5% | 25.0% |\n\nThe data shows success rates increasing from 25% to peak performance of 38.1% at 50 steps. Interestingly, applying additional computation after 50 steps decreases performance, returning to baseline levels at 100 steps. This empirical validation strongly supports our theoretical insight that excessive integration steps over-smooth trajectories, suppressing the stochastic exploration essential for \"thread-the-needle\" manipulation tasks."}, {"review_id": "pGWuAFi48Z", "reviewer": "Reviewer_cLDi", "summary": "The paper presents Difficulty-Aware Stochastic Interpolant Policy (DA-SIP), which adatively adjusts the inference budget based on predicted task difficulty. The method is built upon the stochastic interpolant policy (SIP), which unifies diffusion-based and flow-based generative models. This enables the method to incorporate a difficulty classifier that estimates the task difficulty to select the appropriate step budget, the optimal solver variant, and ODE/SDE integration at each control cycle to balance the accuracy and efficiency. Across a suite of robotic manipulation tasks, the proposed method can achieve comparable success rates to fixed-budget baselines with 2.6-4.4× reduction in total computation time.", "questions": "1. In equation 3, could you explain in detail why $s(\\mathbf{x}, t, \\mathbf{o})=\\sigma_t^{-1} \\mathbb{E}\\left[\\mathbf{\\varepsilon} \\mid \\mathbf{x}_t=\\mathbf{x}, \\mathbf{o}\\right]$?", "limitations": "1. The difficulty categories are too task-specific and may not be directly extended to other more complicated tasks, e.g., bimanual tasks, fine-grained in-hand manipulation tasks, articulated or deformable object manipulation tasks, etc.\n2. The use of manually labeled data for learning the difficulty model is not scalable. Considering the limitation of the manually defined difficulty categories, the dataset would have to be re-labeled whenever the inputs to the difficulty model are changed or the difficulty categories have to be re-defined.\n3. The method has yet to be tested on real robots and more complex tasks.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method to unify diffusion and flow policies under stochastic interpolants introduces the possibility to switch between different configurations of generative models during training and inference.\n2. The method decreases the total computation time by 2.6-4.4× while maintaining comparable success rates to fixed-budget baselines.\n3. The difficulty model and the policy model architecture can be easily upgraded to more complex tasks without fundamental design changes.\n\n\nWeaknesses:\n1. The difficulty categories are too task-specific and may not be directly extended to other more complicated tasks, e.g., bimanual tasks, fine-grained in-hand manipulation tasks, articulated or deformable object manipulation tasks, etc.\n2. The use of manually labeled data for learning the difficulty model is not scalable. Considering the limitation of the manually defined difficulty categories, the dataset would have to be re-labeled whenever the inputs to the difficulty model are changed or the difficulty categories have to be re-defined.", "was_revised": true, "final_justification": "The authors response addressed my concerns and questions. Therefore, I am keeping my original score as \"Accept\".", "rebuttal": "We sincerely thank the reviewer for the positive assessment and feedback. \n\n## Addressing the Weaknesses\n\n### 1. \"The difficulty categories are too task-specific and may not be directly extended to other more complicated tasks, e.g., bimanual tasks, fine-grained in-hand manipulation tasks, articulated or deformable object manipulation tasks, etc.\"\n\n1. We appreciate the reviewer raising this important consideration. Through systematic analysis of our results in Table 1, we identified that task failures could be consistently attributed to specific sub-tasks that characterize each main task. While we acknowledge that new categories may emerge for different task types or robot platforms, our extensive grid search of SIP configurations revealed that these six categories comprehensively capture the fundamental challenges for robot arms with limited precision and frequency constraints.\n\n2. Importantly, these six categories successfully explained all sub-task variations across our diverse set of 6 tasks, which we believe provides comprehensive coverage for this robot class. The strong consensus among our 8 human annotators (enabling accurate classifier training) demonstrates that these categories represent genuine, recognizable patterns rather than arbitrary divisions. The classifier's high prediction accuracy further validates that our categorization captures meaningful and consistent difficulty dimensions.\n\n3. Regarding the specific examples of complex tasks mentioned, we note that our evaluation already encompasses both bimanual coordination (Transport task) and fine-grained manipulation (PushT task, which requires precise control despite not being in-hand). The effectiveness of our dynamic difficulty classification in improving performance while reducing completion time for these challenging tasks suggests that our framework can indeed handle sophisticated manipulation scenarios beyond simple pick-and-place operations.\n\n\n### 2. \"The use of manually labeled data for learning the difficulty model is not scalable. Considering the limitation of the manually defined difficulty categories, the dataset would have to be re-labeled whenever the inputs to the difficulty model are changed or the difficulty categories have to be re-defined.\"\n\n1. We conducted an additional experiment to investigate the minimal data requirements for effective difficulty classification:\n\n| Training Images | 100 | 200 | 300 | 500 | 2000 | Test Set Size |\n|----------------|-----|-----|-----|-----|------|---------------|\n| Square         | 8%  | 48% | 84% | 85% | 84%  | 910          |\n| Tool Hang      | 2%  | 23% | 19% | 29% | 39%  | 1584         |\n| Lift           | 87% | 89% | 90% | 92% | 92%  | 900          |\n| Can            | 2%  | 2%  | 85% | 83% | 88%  | 520          |\n| Transport      | 1%  | 1%  | 36% | 36% | 37%  | 1584         |\n\nMost tasks achieve near-plateau performance with just 300 labeled images, requiring only ~15 minutes of annotation (3 seconds per image).\n\n\n2. We hypothesized that after finetuning the VLMs with enough data, they may learn generalization ability that they do not need to be finetuned with new data but are able to make meaningful predictions. We conducted an additional experiment to test generalization:\n\n| Training Setup | Test Environment | Accuracy | Baseline |\n|----------------|------------------|----------|----------|\n| 5 environments (excluding Lift) | Lift (never seen) | 46.4% | 33.3% |\n\nWhile there is room for improvement, we believe with the current rate of VLM progress, its generalization ability will only improve in the future.\n\n3. Going further with VLMs, they are used precisely to address the scalability issue in our paper. We only used 3 images per category in the prompt for VLMs and used a total of 1049 training images to finetune VLMs. We conducted an additional experiment to see the impact of differing numbers of images for VLMs:\n\n| Environment | 1 img/cat | 2 img/cat | 3 img/cat |\n|-------------|-----------|-----------|-----------|\n| Square      | 20%       | 20%       | 24%       |\n| Tool Hang   | 16%       | 32%       | 52%       |\n| Lift        | 57%       | 83%       | 83%       |\n| Can         | 25%       | 43%       | 38%       |\n| Transport   | 25%       | 25%       | 33%       |\n\nWhile VLM classification accuracy remains below that of CNNs, VLMs were still effective enough to show performance improvement and time savings as shown in tables 4 and 5.\n\n4. **Comparison to existing approaches**: We respectfully note that manual annotation is standard practice in robotics research:\n   - Diffusion Policy (Chi et al., 2023) requires extensive human demonstrations\n   - RT-1 (Brohan et al., 2022) used 130k human-annotated episodes\n   - Our approach requires orders of magnitude less annotation while enabling adaptive efficiency\n\n\n## Addressing the Question\n\n### 1. In equation 3, could you explain in detail why \n$$\ns(\\mathbf{x},t,\\mathbf{o})=\\sigma_t^{-1}\\mathbb{E}[\\boldsymbol{\\varepsilon}\\mid\\mathbf{x}_t=\\mathbf{x},\\mathbf{o}]\\ ?\n$$\n\n1. Forward process\nFrom the stochastic interpolant formulation:\n$$\\mathbf{x}_t=\\alpha_t\\mathbf{x}_0+\\sigma_t\\boldsymbol{\\varepsilon},\\quad \\boldsymbol{\\varepsilon}\\sim\\mathcal{N}(\\mathbf{0},\\mathbf{I})\\tag{1}$$\n\n2. Posterior of the noise\nKey insight: Due to the linearity of the interpolant (1) in $(\\mathbf{x}_0, \\boldsymbol{\\varepsilon})$, we can express:\n$$\\boldsymbol{\\varepsilon} = \\frac{\\mathbf{x}_t - \\alpha_t\\mathbf{x}_0}{\\sigma_t}$$\n\nTaking conditional expectations given $\\mathbf{x}_t = \\mathbf{x}$ and $\\mathbf{o}$:\n$$\\mathbb{E}[\\boldsymbol{\\varepsilon}\\mid\\mathbf{x}_t=\\mathbf{x},\\mathbf{o}] = \\frac{1}{\\sigma_t}\\left(\\mathbf{x}-\\alpha_t\\mathbb{E}[\\mathbf{x}_0\\mid\\mathbf{x}_t=\\mathbf{x},\\mathbf{o}]\\right)\\tag{2}$$\n\nThis follows directly from the linearity of the interpolant—no joint Gaussianity assumption needed.\n\n3. Connection to the score\nBy definition, the score function is:\n$$s(\\mathbf{x},t,\\mathbf{o}) = \\nabla_{\\mathbf{x}}\\log p_t(\\mathbf{x}\\mid\\mathbf{o})$$\n\nUsing Stein's identity (also known as Gaussian integration by parts), the score can be expressed as:\n$$s(\\mathbf{x},t,\\mathbf{o}) = -\\sigma_t^{-1}\\mathbb{E}[\\boldsymbol{\\varepsilon}\\mid\\mathbf{x}_t=\\mathbf{x},\\mathbf{o}]\\tag{3}$$\n\nThis matches Lemma 3 of Song et al. (2021) and Eq. (38) of the SiT appendix.\n\n4. Sign convention\nIf you define $\\boldsymbol{\\varepsilon}':=-\\boldsymbol{\\varepsilon}$ or adopt the alternative convention $\\tilde{s}:=-\\nabla_{\\mathbf{x}}\\log p_t$, then:\n$$s(\\mathbf{x},t,\\mathbf{o})=\\sigma_t^{-1}\\mathbb{E}[\\boldsymbol{\\varepsilon}\\mid\\mathbf{x}_t=\\mathbf{x},\\mathbf{o}]\\tag{★}$$\n\n\n\n## Addressing the Limitations\n\n### 1. \"The method has yet to be tested on real robots and more complex tasks\" \n\n1. To address this concern, we conducted an experiment with the 'real robot pushT' that has publicly available training data. We were able to train a diffusion policy and SIP with linear interpolation and velocity prediction and found a decreasing pattern of training action MSE, revealing that SIP can mimic the expert human demonstrator, which suggests it will be useful when deployed in the real world. Due to time constraints, we were unable to roll out actual robot trajectories.\n\n \n**Train Loss Comparison**\n\n| Step | Diffusion Policy | SIP |\n|------|------------------|-----|\n| 420 | 0.076043 | 0.18309 |\n| 2520 | 0.023841 | 0.085425 |\n| 4614 | 0.038257 | 0.13193 |\n\n**Train Action MSE Comparison**\n\n| Step | Diffusion Policy | SIP |\n|------|------------------|-----|\n| 420 | 0.024996 | 0.013631 |\n| 2520 | 0.0058178 | 0.0047186 |\n| 4614 | 0.0046736 | 0.0029387 |\n\nThe comparable or superior trajectory matching on real robot data provides confidence in sim-to-real transfer potential."}, {"review_id": "bZyVIf0rIw", "reviewer": "Reviewer_9NDW", "summary": "**Summary**  \nThis paper proposes **DA-SIP (Difficulty-Aware Stochastic Interpolant Policy)**, a novel framework for adaptive test-time computation in diffusion- and flow-based robotic controllers. The authors identify a key inefficiency in existing approaches: they use a fixed inference budget for all control steps, regardless of the underlying task complexity. To address this, DA-SIP introduces a **difficulty classifier** that dynamically assesses the complexity of a subtask from RGB-D inputs and selects the appropriate solver configuration, integration horizon, and ODE/SDE mode on the fly.\n\nThe method is grounded in the **stochastic interpolant (SI)** framework, which unifies diffusion and flow-based models and allows flexible inference configuration without retraining. \nThrough comprehensive benchmarking on robotic manipulation tasks, DA-SIP demonstrates **2.6–4.4× reductions in computation time** while maintaining **task success rates comparable to fixed-budget baselines**. The approach provides an efficient and scalable path forward for resource-aware generative robot policies.", "questions": "**Questions and Suggestions**\n\n1. **Difficulty Classification Accuracy and Impact**  \n   - How sensitive is the overall system performance to errors in the difficulty classification stage?  \n   - Have you conducted experiments where the classifier is intentionally degraded (e.g., with noise or randomization) to test robustness?  \n   - Clarifying this would help evaluate how reliable DA-SIP is under real-world noise or partial observability.\n\n2. **Computational Overhead of Difficulty Classifiers**  \n   - What is the runtime overhead of the different classifiers (lightweight CNN, zero-shot VLM, fine-tuned VLM)?  \n   - In high-frequency control scenarios, even lightweight models could become bottlenecks. Providing profiling data would help assess real-time applicability.\n\n3. **Transfer to Real-World Settings**  \n   - Although the paper mentions future plans to transfer to real robots, have the authors considered sim-to-real strategies or domains where DA-SIP could already be viable (e.g., industrial pick-and-place)?  \n   - Any partial evaluation on physical hardware would strengthen the practical impact.", "limitations": "**Limitations**\n\nThe authors have partially addressed the limitations of their work by acknowledging that their approach has only been evaluated in simulation and may not directly transfer to real-world robots without further adaptation. This is a fair and important caveat, given the challenges of sim-to-real transfer in robotic control.\n\nHowever, the paper does not explicitly discuss the **potential societal impacts** of adaptive compute in autonomous systems. While the work is primarily technical, the deployment of adaptive inference in robotics could have broader implications—for example:\n\n- **Resource allocation bias**: Adaptive inference could inadvertently prioritize certain tasks over others based on incorrect difficulty estimation, which might be problematic in safety-critical applications.\n\n**Suggestion**:  \nIt would strengthen the paper if the authors briefly commented on these broader considerations, particularly regarding deployment in human-centric environments or industrial systems where incorrect difficulty assessments could cause harm or inefficiency.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths and Weaknesses**\n\n**Quality**  \n**Strengths:**  \n- The paper presents a well-motivated and clearly articulated problem: fixed inference budgets in flow- and diffusion-based policies result in inefficiencies across tasks of varying complexity.  \n- The proposed solution, DA-SIP, is technically sound and thoughtfully designed, leveraging the stochastic interpolant framework to enable adaptive inference configurations.  \n- Experiments are comprehensive and demonstrate strong empirical results: DA-SIP achieves a 2.6–4.4× reduction in computation without sacrificing performance.  \n\n**Weaknesses:**  \n- The method has only been evaluated in simulation. Real-world robot validation, though understandably out of scope, would further solidify the impact.  \n- The robustness of the method under noisy sensor inputs or misclassified difficulty levels is not fully explored.\n\n**Clarity**  \n**Strengths:**  \n- The paper is well-organized and clearly written, with good use of figures and examples to illustrate key components.  \n- Technical exposition is rigorous yet accessible, and contributions are clearly delineated.\n\n**Significance**  \n**Strengths:**  \n- This work addresses a highly practical and underexplored issue in generative robotic control: test-time computation efficiency.  \n- The proposed approach could inspire future research on resource-adaptive policies across other generative architectures and tasks.\n\n**Weaknesses:**  \n- While the method is promising, the lack of real-world deployment raises questions about generalizability beyond the simulation benchmarks.\n\n**Originality**  \n**Strengths:**  \n- The integration of adaptive compute via difficulty classification in a flow/diffusion-based framework is novel.  \n- The unification under the stochastic interpolant framework is conceptually elegant and technically innovative.\n\n**Weaknesses:**  \n- Some components (e.g., zero-shot and few-shot VLMs for difficulty classification) are based on prior ideas, although their integration into this specific context is novel.", "was_revised": true, "final_justification": "The author’s response addressed the majority of my questions and concerns; I am maintaining my original score.", "rebuttal": "We thank the reviewer for the thoughtful assessment and constructive feedback.\n\n## Addressing the Weaknesses\n\n### 1. \"The method has only been evaluated in simulation. Real-world robot validation, though understandably out of scope, would further solidify the impact\"\n\n\n1. To address this concern, we conducted an experiment with the 'real robot pushT' that has publicly available training data. We were able to train a diffusion policy and SIP with linear interpolation and velocity prediction and found a decreasing pattern of training action MSE, revealing that SIP can mimic the expert human demonstrator, which suggests it will be useful when deployed in the real world. Due to time constraints, we were unable to roll out actual robot trajectories.\n\n**Train Loss Comparison**\n\n| Step | Diffusion Policy | SIP |\n|------|------------------|-----|\n| 420 | 0.0760 | 0.183 |\n| 2520 | 0.0238 | 0.085 |\n| 4614 | 0.0382 | 0.131 |\n\n**Train Action MSE Comparison**\n\n| Step | Diffusion Policy | SIP |\n|------|------------------|-----|\n| 420 | 0.0250 | 0.0136 |\n| 2520 | 0.00582 | 0.00472 |\n| 4614 | 0.00467 | 0.00294 |\n\n\n\n### 2. \"The robustness of the method under noisy sensor inputs or misclassified difficulty levels is not fully explored.\"\n\n1. We conducted an additional experiment where we gradually added same-random-seed noise to the test data images only.\n\n- Training samples per environment: 300\n- Test set: 20% of data (stratified split)\n- Random seed: 42 (ensures reproducible noise across runs)\n- Noise levels: σ ∈ {0.0, 0.05, 0.15, 0.30, 2.0}\n\n### Test Accuracy (%) Under Different Noise Levels\n\n| Environment | Clean (σ=0.0) | Low Noise (σ=0.05) | Medium Noise (σ=0.15) | High Noise (σ=0.30) | Extreme Noise (σ=2.0) |\n|-------------|---------------|--------------------|-----------------------|---------------------|------------------------|\n| Square      | 84.51         | 85.60 (+1.10)      | 83.08 (-1.43)        | 78.68 (-5.82)      | 12.86 (-71.65)        |\n| Tool Hang   | 20.52         | 20.27 (-0.25)      | 19.95 (-0.57)        | 19.63 (-0.88)      | 1.83 (-18.69)         |\n| Lift        | 91.67         | 90.67 (-1.00)      | 89.00 (-2.67)        | 81.89 (-9.78)      | 75.11 (-16.56)        |\n| Can         | 81.92         | 82.50 (+0.58)      | 82.12 (+0.19)        | 82.69 (+0.77)      | 77.69 (-4.23)         |\n| Transport   | 34.66         | 34.34 (-0.32)      | 34.60 (-0.06)        | 35.10 (+0.44)      | 22.85 (-11.81)        |\n| **Average** | **62.66**     | **62.68 (+0.02)**  | **61.75 (-0.91)**    | **59.60 (-3.06)**  | **38.07 (-24.59)**    |\n\nThis experiment demonstrates that our CNN models still maintain reasonable performance under high noise conditions (σ=0.30) with only 3.06% average accuracy drop. σ=2.0 represents extreme noise conditions beyond typical real-world scenarios.\n\n### 3. \"...the lack of real-world deployment raises questions about generalizability beyond the simulation benchmarks\"\n\n1. Simulation-based research has historically driven significant advances in robotics and RL. Notable examples include: DQN (Mnih et al., 2015) and PPO (Schulman et al., 2017), which established foundational RL algorithms entirely through simulated environments before widespread real-world adoption. Similarly, we believe our work introduces an initial exploration of test-time compute scaling for robotic control—a concept that may serve as a starting point for broader applications in real-world systems as the field continues to develop adaptive computation strategies.\n\n2. While sim-to-real transfer remains an important direction for future work, our current results provide valuable insights for the community on integrating human cognitive assessments into robot learning systems.\n\n### 4. \"Some components (e.g., zero-shot and few-shot VLMs for difficulty classification) are based on prior ideas.\"\n\n1. While we acknowledge that VLMs themselves are established technologies, we believe our work makes important contributions by demonstrating their novel application to adaptive compute in robotic control. Determining task difficulty for dynamic compute allocation in robotic policies is a fundamentally different problem from typical VLM applications. Although VLMs have shown success in image understanding and captioning tasks, their ability to assess manipulation difficulty for compute optimization had not been explored. To our knowledge, our work is the first to show that VLMs can effectively predict the computational requirements of robotic control subtasks, enabling significant efficiency gains.\n\n2. Similar to how successful RL works like CURL [ICML'20], RAD [NeurIPS'20], and DrQ [ICLR'21] made significant contributions by adapting existing CV techniques (contrastive learning, image augmentation) to RL settings, our work demonstrates that VLMs can be repurposed for a novel task—difficulty assessment for adaptive inference. The key contribution lies not in the VLM architecture itself, but in showing that visual-language understanding can guide computational resource allocation in robotic control.\n\n## Addressing the Questions\n\n### 1. \"How sensitive is the overall system performance to errors in the difficulty classification stage?\"\n\nTable 5 demonstrates the impact of classification accuracy by comparing zero-shot VLM, fine-tuned VLM, and CNN classifiers. While the VLMs achieve lower accuracy than CNN, we included them to address scalability concerns—they require significantly less training data and leverage pre-trained generalization capabilities. Crucially, even with misclassifications, performance never drops below the min-compute baseline because any classification yields at least one inference step. This provides an important safety guarantee: users can set their minimum acceptable performance level through the min-compute configuration, ensuring that classification errors may reduce efficiency but will never compromise this performance floor. Thus, the system degrades gracefully under classification errors rather than failing catastrophically.\n\n### 2. \"Have you conducted experiments where the classifier is intentionally degraded to test robustness? Clarifying this would help evaluate how reliable DA-SIP is under real-world noise or partial observability\"\n\n1. We tested both aspects: (1) Degraded classifiers: Table 5 shows that even with lower-accuracy VLMs, performance never drops below min-compute baseline. (2) Noisy sensors: As detailed in Weakness #2 above, we added Gaussian noise to test images and found only 3.06% average accuracy drop at σ=0.30 under realistic noise conditions. Both experiments confirm DA-SIP degrades gracefully, with performance always bounded by the user-configurable min-compute threshold.\n\n### 3. \"What is the runtime overhead of the different classifiers? In high-frequency control scenarios, even lightweight models could become bottlenecks. Providing profiling data would help assess real-time applicability.\"\n\n1. We re-rolled out the policy for Tool Hang (88 iterations, per-iteration average):\n\n| Component | CNN Configuration | Finetuned VLM Configuration |\n|-----------|------------------|---------------------------|\n| Classifier inference | 0.023s | 0.362s |\n| SIP inference | 0.301s | 0.127s |\n| Simulation execution | 0.977s | 0.991s |\n| Data processing | 0.057s | 0.055s |\n| **Total time** | **1.357s** | **1.535s** |\n\n2. Moreover, we tested the impact of increasing number of images as examplars in VLM prompt.\n\n| Environment | 1 img/cat | 2 img/cat | 3 img/cat |\n|-------------|-----------|-----------|-----------|\n| Square      | 20% (0.55s) | 20% (0.89s) | 24% (1.26s) |\n| Tool Hang   | 16% (0.56s) | 32% (0.91s) | 52% (1.26s) |\n| Lift        | 57% (0.48s) | 83% (0.64s) | 83% (0.82s) |\n| Can         | 25% (0.52s) | 43% (0.75s) | 38% (1.05s) |\n| Transport   | 25% (0.50s) | 25% (0.72s) | 33% (1.05s) |\n\n   Even though higher number of images would increase the accuracy, we fixed on 3 img/cat as a good balance of performance, time, and memory usage.\n\n### 4. \"Have the authors considered sim-to-real strategies or domains where DA-SIP could already be viable? Any partial evaluation on physical hardware would strengthen the practical impact.\"\n\n1. Yes, we conducted a partial evaluation by training SIP on 'real robot pushT' data (discussed in Weakness #1). We specifically chose pushT because it directly corresponds to one of our simulation environments. The key finding is that SIP achieves comparable train action MSE error to the proven Diffusion Policy (0.00294 vs 0.00467 at step 4614), demonstrating that SIP can accurately reproduce human demonstrator trajectories. This low MSE indicates SIP learns the mapping from observations to actions as effectively as Diffusion Policy, strongly suggesting it would perform similarly when deployed on the actual robot. \n\n## Addressing the Limitations\n\n### 1. Lack of discussion on societal impacts for resource allocation bias and for safety-critical applications.\n1. Our VLM experiments directly address this concern by demonstrating bounded performance degradation under classification errors. As shown in Table 5, even with lower-accuracy VLMs, performance never drops below the min-compute baseline—providing a guaranteed performance floor that users can configure based on their risk tolerance. The worst case (18% drop in Transport task) still maintains acceptable performance. This ensures that classification errors reduce efficiency but never compromise safety thresholds. As VLM capabilities improve, this classifier-policy gap will naturally narrow, making adaptive compute increasingly practical for safety-critical applications.\n2. Similar to LLMs that adjust behavior based on context, we envision implementing safety filters that automatically enforce higher compute thresholds when detecting humans nearby or operating near critical objects. The key insight is that adaptive compute should enhance, not replace, existing safety protocols."}]

Author rebuttal:

Meta-review: The paper introduces the Difficulty-Aware Stochastic Interpolant Policy (DA-SIP), a test-time scaling approach for robotic control policies that adaptively allocates compute based on task difficulty using a difficulty classifier and the stochastic interpolant framework.

Strengths include a solid technical foundation built on the SI framework, addressing the inefficiency of fixed inference budgets in diffusion and flow-based robot policies, and demonstrating significant improvements in computation time without sacrificing task success rates. 

Several weaknesses, such as reliance on simulation, annotation bottlenecks, and task-specific difficulty categories, were acknowledged by reviewers. 
In rebuttal, the authors provided preliminary real-robot results (PushT) and additional experiments showing reduced data requirements and VLM-based generalization to unseen tasks. These responses were well received and alleviated many of the original concerns.

The paper received generally positive evaluations, with final scores of 5 (Accept) and 4, 4, 4 (Borderline Accepts). Given the novelty, technical soundness, strong empirical results, and the constructive rebuttal that improved reviewer confidence, I recommend Accept.
Decision: Accept (poster)