Demystifying Robot Diffusion Policies: Action Memorization and a Simple Lookup Table Alternative
Authors: Chengyang He
Xu Liu
Gadiel Mark Sznaier Camps
Joseph Bruno
Guillaume Adrien Sartoretti
Mac Schwager
Venue: ICLR 2026 Poster (2026)
Keywords: Action Memorization
Action Lookup Table
Diffusion Policy Analysis
Primary area: applications to robotics, autonomy, planning

Abstract: Diffusion policies for visuomotor robot manipulation tasks achieve remarkable dexterity and robustness while only training on a small number of task demonstrations.  However, the reason for this performance remains a mystery. In this paper, we offer a surprising hypothesis: diffusion policies essentially memorize an action lookup table---\emph{and this is beneficial}. We posit that, at runtime, diffusion policies find the closest training image to the test image in a latent space, and recall the associated training action  (i.e. action chunk), offering reactivity without the need for action generalization. This is effective in the sparse data regime, where there is not enough data density for the model to learn action generalization. We support this claim with systematic empirical evidence, showing that even when conditioned on highly out of distribution (OOD) images, Diffusion Policy still outputs an action chunk from the training data. We evaluate and compare three representative policy families on the same data set: Diffusion Policy, Action Chunking with Transformers (ACT), and GR00T, a pre-trained generalist Vision-Language-Action (VLA) model.  We show that Diffusion Policy gives strong action memorization giving surprising robustness in OOD regimes, ACT shows action interpolation with poor robustness in OOD regimes, and GR00T (benefiting from substantial pre-training) shows both action interpolation and OOD robustness. As a simple alternative to Diffusion Policy, we introduce the Action Lookup Table (ALT) policy, showing that an explicit lookup table policy can perform comparably in this low data regime. Despite its simplicity, ALT attains Diffusion Policy–level performance while also providing faster inference and explicit OOD detection via latent-distance thresholds. These results reframe diffusion policies for robot manipulation as reactive memory retrieval under data sparsity, and provide practical tools for interpreting, evaluating, and monitoring such policies. More information can be found at: \url{https://stanfordmsl.github.io/alt/}.

Published as a conference paper at ICLR 2026
DEMYSTIFYING ROBOT DIFFUSION POLICIES: Ac- TION MEMORIZATION AND A SIMPLE LOOKUP TABLE ALTERNATIVE
Chengyang H Xu Liu Gadiel Sznaier Camps'
Joseph Bruno
Guillaume Sartoretti
Mac Schwager'
Stanford University 2National University of Singapore
Temple University "Equal Contribution
hecy2stanford.eduu http::eeddddxxxixiiseseeee liwwujss@sttafoordeeee
brunof edu
guillaume, ott.://eeeee..toooooooo..o.ooooa
edu sg, sctwegerrrrtaafordd..d
ABSTRACT
Diffusion policies for visuomotor robot manipulation tasks achieve remarkable dexterity and robustness while only training on a small number of task demon- strations. However, the reason for this performance remains mystery. In this paper, we offer a surprising hypothesis: diffusion policies essentially memorize an action lookup table-and this is beneficial. We posit that, runtime, diffusion policies find the closest training image to the test image in latent space, and recall the associated training action (i.e. action chunk), offering reactivity with- out the need for action generalization. This is effeective in the sparse data regime, where there not enough data density for the model to learn action generaliza- tion. We support this claim with systematic empirical evidence, showing that even when conditioned on highly out of distribution (OOD) images, Diffusion Policy still outputs action chunk from the training data. We evaluate and compare three representative policy families on the same data set: Diffusion Policy, Action Chunking with Transformers (ACT), and GROOT, pre-trained generalist Vision- Language-Action (VLA) model. We show that Diffusion Policy gives strong ac- tion memorization giving surprising robustness in OOD regimes, ACT shows ac- tion interpolation with poor robustness in OOD regimes, and GROOT (benefiting from substantial pre-training) shows both action interpolation and OOD robust- ness. As simple alternative to Diffusion Policy, we introduce the Action Lookup Table (ALT) policy, showing that an explicit lookup table policy can perform com- parably in this low data regime. Despite its simplicity, ALT attains Diffusion Pol- icy-level performance while also providing faster inference and explicit OOD de- tection via latent-distance thresholds. These results reframe diffusion policies for robot manipulation as reactive memory retrieval under data sparsity, and provide practical tools for interpreting, evaluating, and monitoring such policies. More information can be found at: /tttp:://stafoott www.doddd............b.....
github. io/alt/
INTRODUCTION
Imitation learning for robot manipulation requires training policy map from image inputs to action sequence outputs given relatively small number of demonstrations. Recently, the Diffusion Policy Chi et al. (2023) has emerged as powerful approach to this problem by modeling the robot's policy as a denoising diffusion probabilistic model Ho et al. (2020). The Diffusion Policy infers an action chunk (a short action sequence, typically 16 time steps long) conditioned on the robot's camera views and joint angles. To infer an action chunk, the policy iteratively denoises a random action chunk using a learned denoising filter. The denoised action chunk is applied to the robot (typ- ically executing 8 of the 16 actions in the chunk), and the loop repeats. The architecture is derived directly from denoising diffusion model for image generation Saharia et al. (2022b); RRmbacc et al. (2022). adapted to produce action chunk outputs. The primary advantage of the Diffusion

---

Published as a conference paper at ICLR 2026
Policy lies in its ability to model multi-modas action distributions, scale to high-dimensional robot action spaces, and produce ann-horizon action sequences. Indeed, recent studies have shown that diffusion policies outperform many existing methods on challenging manipulation benchmarks Ze
et al. 2024); Wang et al. (202aa).
The performance of the Diffusion Policy is unquestionable, however the explanation for this per- formance remains elusive. In particular, typical diffusion policies are trained on between 50 to 200 task demonstrations (small amounts of data), while maintaining the same number of parameters (typically over 100 million) as image generation models trained on billions of images Saharia el al. (2022b): Wikipedia contributors (202)). Furthermore, the common practice is to train diffusion policies until the training loss is low, but the test loss is high the classic signal for over over-fitting fitting. Typically overfitting is associated with poor test-time performance and poor generalization. Yet, it observed that this overfitting is actually necessary for strong test time performance of the Diffusion Policy. The natural question arises:
Why do diffusion policies trained to overfit small data sets appear give strong test-time perfor- mance in robot manipulation?
In this paper we show that, indeed, diffusion policies highly overfit the training data, such that they essentially recall training action chunks inference. They exhibit little generalization in the action space, neither interpolation nor extrapolation. They effectively perform lookup table that maps runtime images to training action chunks. Combined with online closed-loop execution with run- time images, this action chunk memorization appears be winning recipe for strong manipulation policies obtained from small amounts of demonstration data. We show that for inference in in- terpolation regimes, in extrapolation regimes, and in highly out-of-distribution (OOD) regimes the diffusion model always recalls action chunks seen at training, giving the policy a surprising robust- ness in these OOD regimes. We verified this through pick-and-place experiments on real robot and Can and Square experiments in Robomimic (more details can be found in Appendix D).
It is then reasonable to wonder whether action memorization a property shared by all imitation learn- ing architectures when trained on small data sets, which motivates our second research question:
Do other imitation learning architectures for robot manipulation also exhibit action memorization when trained on the same dataset?
We find that an Action Chunking Transformer Zhao et al. (2023); Wu et al. (2024) (ACT) policy trained on the same dataset exhibits action generalization, often producing action chunks that are blend of several of the training action chunks. However, in OOD regimes ACT infers wild action chunks quite different from the training data, and therefore lacks the robustness of the Diffusion Policy. We also find that GROOT Bjorck et al. (2025). generalist pre-trained Vision- Laniuge- Action policy, when fine-tuned on the same data also exhibits action generalization, but maintains robustness in OOD regimes, likely benefiting from the large volume of pre- pre-training data.
A key drawback of the Diffusion Policy is slow inference time, which leads slow robot execution punctuated by pauses as the model recomputes the next inference at the end of each action chunk. This motivates third research question:
Can the same action chunk recall behavior be accom- plished with simpler, faster model architecture to yield faster runtime performance?
Imagts
Goaft
Imagee
To answer this question, we propose a simple lookup table policy with a trained image encoder to map from images to action chunks, which we call the Ac- tion Lookup Table (ALT) policy, illustrated in Fig. [ The ALT policy has similar task success and robust- ness properties as the Diffusion Policy, while being 300 times faster at inference, and requiring less than 1/100th the memory footprint. Our ALT policy is sim- pler than previous parametric interpolation- -based imi- tation learning architectures Pari et al. (2021); (Sridhar ct al. 22023 2024b), as it does not attempt any interpo-
afe
Image;
ahe
Cl. embedding space
Figure Resulting latent space of our con- strastive learning (CL.) based ALT on our train- ing data, illustrating the distribution of train- ing examples and example in- (1.2) and out-of- distribution (3) test images.
2

---

Published as a conference paper at ICLR 2026
lation in action space. We obtain multi- mudti-mo similar to the diffusion policy, through stochastic k-nearest neighbor action chunk lookup.
Concretely, we design lightweight image-joint pose encoder that maps each provided observation into a low-dimensional feature representation (LFR). LFRs seen during training are labeled with their associated action chunks, which are stored in a lookup table. At deployment time, new ob- servations are encoded to their LFRs, the nearest training LFR is found (with a k-d tree), and the associated action chunk is recalled from the lookup table. The encoder is trained using a contrastive learning objective CCen et al. (2020), encouraging positive sample pairs to be closer while push- ing negative pairs apart in the latent space. ALT avoids the costly iterative denoising steps of the Diffusion Policy, enabling faster much inference and much smaller memory footprint.
Our contributions are as follows: (1) We hypothesize the Diffusion Policy implicitly memorizes an action lookup table. We provide conceptual intuition for this hypothesis and support it through ex- tensive empirical validation. For baselines, we compare to two other imitation learning models, ACT and GROOT-N1.5, 5,, and show they provide comparatively more action generalization than Diffuion Policy. (2) We propose a simple Action Lookup Table (ALT) policy (Fig. I that explicitly indexes the nearest training action chunk from an observation embedding. We show that ALT delivers simi- lar performance to a Diffusion Policy, while giving faster inference and requiring less memory. We also propose an explicit OOD monitor for ALT using distance threshold in the latent space.
2 RELATED WORK
Diffusion models, trained by gradually adding Gaussian noise to data during training Ho et al. 20200; Ramesh et al.] (2021); Blattmann et al.) (2023), were originally developed for high- dimensional data generation tasks such as image, video, audio synthesis Nichol & Dhariwal (002]): Rombach et al. (2022). These models can produce seemingly novel high-quality images and
videos in a variety of different styles through simple text Ramesh et al. 12 Ruiz et al. (2023);
Balaji et al. y Saharia et al. (002bb and image Saharia et al. (2022a): [Tumanyan et al. 2023 ;
Ceylan al. (2023) prompt conditioning. In order to capture the complex multi-modal distributions
inherent in visual and auditory data, these models are often large, containing from hundreds of mil-
lions to billions of parameters Saharia et al. (2022b): Wikipedia contributors (2024), and are trained
over large datasets with hundreds of millions to billions of examples Rombach et al. (022)..
Leveraging the strong performance of diffusion models, the Diffusion Policy Chi et al. (2023) achieves sfate-of-the-art performance in visuomotor control for single skill imitation learning. Trained with a limited number of expert demonstrations, the model learns to predict a sequence
of robot actions Chi et al. (2023); Ren et al. (2024a): Lu et al. (0024): Lee & Kuo (202)) condi-
tioned on given observation. This observation can be images Chi et al. (2023). point clouds Zel
et al. (2024), semantic labels Wang et (2024))) Li et al. (2024)
potential fields Mizuta & Leung
(2024). Due to its apparent robustness to perturbations, diffusion policies have been deployed for wide range of robotics tasks, including manipulation Black et al. (2023); Kim et al. (2022); Chi cet al. (2023), multi-skill learning Chen et al. (2023)); Xu et al. (2023). 3. and motion planning Shaoul al. (204); (Serifi et al. (2024) ; (Sridhar et al. (2024a). Diffusion models have also been used in
robotics for data augmentation Kapelyukh et al. 2023) Chen et al. (2023b) to aid in the training of
other models.
The phenomenon of memorization in diffusion models has been well-studied in image generation, but not in robotics, our knowledge. Gu et al. (2023) observed that smaller datasets are prone to cause memorization, especially when conditioned with uninformative labels, while (Somepalli et al.) (2023a) discovered that reconstructive memorization occurs even for models trained on enormous datasets, with as much as 2% of the generated images being duplicates of the training data. Simi- larly, Carlini et al, (2023) demonstrated way to extract known training examples from -fate-of-the- art models, such as DALL-E 2 Ramesh et al. (2022). Meanwhile, Gu et (2023) notes that the tra- ditional denoising score matching objective used during training has a closed-form optimal solution that can only replicate training images. Although, this can be mitigated with synthetic data augmen- tation Xue et al. (0255) Jain et al. (2024) posits that the denoising process causes diffusion models generated images towards memorized data. Wenn et al. (2024) corroborates this by noting that diffusion models tend to converge known training sample regardless of initialization, suggesting memorization of
to learn attraction basin for each training sample, thereby guiding prompt-condtioons
3

---

Published as a conference paper at ICLR 2026
Training Samples Random seeds
for di disiusin Denoising
direction
Final inferenie
(b) low-capacity model, large data (LD) SD (e.g. Diffusion Policy) LD (e.g. Image Diffusion) Figure/ 2: Training a diffusion model from 2D points uniformly distributed on a star-shaped ID manifold.
(a) low-capacity model, small data (SD)
(c) high- model. (d) high-capacity model,
Each subplot shows different training regime: (a) A low-capacity model (~>400 parameters) trained small dataset (3k samples) gives erratic inferences. (b) The low-capacity model trained large dataset (100k samples) generalizes to the wrong manifold. (c) high-capacity model (~9..5 million parameters) trained on a small dataset (the Diffusion Policy regime) approximately memorizes the dataset, but does not generalize. All the inferred outputs (blue) overlay the training data (orange) points, essentially implementing lookup table. (d) A high-capacity model trained large dataset shows strong generalization to the correct data manifold (regime of large scale image diffusion models). See for further discussion.
both the prompt and the denoising trajectory. Similarly, both (oohhpalllll et al. (2023b) CheC al. (0224b) indicate that, although less prevalent than conditioned models, memorization still occurs in unconditioned models and Hintersdorf al. (2024) finds that memorized data are often associated with corresponding individual neurons.
Modifying the loss function Chen et al. 2024a), gradients Chen et al. (2024a) conditioning ap- proach Jain et al. (2024), keyword prompts Wen al. (2024); Somepall al. (2023b); Ren al. (2024b) during training and inference are all typical methods for reducing model memorization. However, we propose that while memorization is undesirable for image generation due to privacy and copyright concerns, it is actually beneficial for robotics. When the diffusion model is used in domains with rich input space (e.g., images) but limited output space (e.g., robot actions), the gap between model capacity and output dimensionality, combined with the use of imitation learn- ing (which inherently lacks task-level supervision), makes overfitting via memorization a plausible explanation for its strong performance in in on-distribution settings. Along these lines, previous ap- proaches have explored reusing past demonstrations in conjunction with parametric generalizations. They obtain the next action using frame-level nearest-neighhor action retrieval Pari et al. (202T). ac- tion prototypes obtained based on bounded residual (Srihhar al. (2023), or by utilizing learned context embedding composed of multiple trajectory fragments /rriharr et al. (2024b). In compar- ison, our proposed ALT method directly obtains the closest action sequence from a lookup table via contrastive image encoder, bypassing any parametric policy entirely, thereby allowing it to be lightweight and fast unlike more compute-intensive intensive hybrid alternatives.
3
DIFFUSION POLICY ANALYSIS
3.1 PRELIMINARIES
The output, x², of a diffusion model, Eg, is obtained by iteratively removing noise (i.e. denoising) from starting value, x², sampled from a Normal Distribution, (0, rT)). The denoising process evolves according to
x" --11= xee((" .k) +N(00001)
remove the noise in I steps based on predetermined noise schedule that specifies the values of a, and at each iteration. This procedure can be thought of as single stochastic gradient descent step x' r E()),, where the model Eœ is used predict the gradient field V (z)). more detailed explanation of the denoising process can be found in Chi et al. (003) and Ho et al. (020).
3.2 DIFFUSION MODEL GENERALIZATION REGIMES
We illustrate four generalization regimes for simple Multi-Layer Perceptron (MLP)-based diffu- sion denoising model trained to leam a ground-trume distribution consisting of 2D points uniformly sampled ID manifold shaped star (Fig. 2). We show qualitative model performance with
4

---

Published as a conference paper at ICLR 2026
low-capacity vs high-capacity MLP, trained with small vs large data sets. As expected, when low-capacity model is trained on a small dataset (Fig. 2)), it fails to fit the data adequately. Simi- larly, due to its limited capacity, when such model is given sufficient data (Fig. 2p). it is only able to learn an approximation that oversimplifies the data manifold (here, approximating a star shape as hexagon). In comparison, when a high-capacity model is trained the diffusion model tends to memorize the individual training samples rather than generalizing or interpolating between them. This memorization allows accurate fitting of the limited training points (good for robot manipulation tasks), but results in the model failing to capture the broader under- lying data manifold, behavior that is consistent with our findings for diffusion policies for robot manipulation. This phenomenon is related to manifold overfitting Loaiza- et al. (2022): when a powerful generative model is trained on data lying on narrow sub-manifold, it might fit the data too closely while struggling outside that sub-manifold. When the model is provided sufficient data (Fig. 21), it is now able to effectively fit both the data and the true underlying distribution, repre- senting the regime common in large scale image generation models. However, acquiring large-scale expert demonstrations for robot manipulation that evenly and densely cover the action sequence space remains a significant practical challenge. As Diffusion Policies are trained on larger and larger datasets, they may move toward the large data regime (Fig. Rii) with true generalization on the action manifold, but this seems be beyond the current state of the art.
small dataset (Fig. (2),,
3.3 HYPOTHESSI AND EXPERIMENTS
The core hypothesis of this paper is that the impressive reactivity, multimodality, and robustness exhibited by the Diffusion Policy stems not from deep understanding of the physical task, but from the simple ability to memorize training action chunks and recall appropriate action chunk when prompted with an image. To evaluate this hypothesis, we designed a series of cup grasping ex- periments We trained diffusion policy for cup grasping using the standard codebase from Chi et al. (2023). trained with 30/120 demonstrations of cup locations evenly spaced throughout the workspace, with held-out square in the middle, as indicated by the green circles and blue tape in Fig. 3] The robot has a third-person view fixed camera and a wrist mounted camera, both used to condition the policy. For each position, we performed one demonstration (to remove the con- founding effect of multi-modal action generation). We then validated the learned policy on the 30 in-distribution cases, confirming its ability to reproduce the training demonstrations.
To further investigate the action generalization be- havior of the policy, we systematically introduced a variety of interpolation and extrapolation in- puts, ranging from in-distribution (InD) to out-of- distribution (OOD) and analyzed the resulting behav- ior. Specifically, we designed four scenarios: (1:InD- Interpolate) Placing the cup at evenly spaced test po- sitions located between the original training positions (Fig. 3 green border); (2:00D-Interpolat, Slowly moving the cup from one in-distribution position through an O0I) region (blue tape square) to another in-distribution position (Fig. bllue border); (3:0OD- Extrapolate) Gradually moving the cup from an in- distribution position to an OOD location outside the fixed camera's 's field of view (Fig. 3 purple border). (4:00D-Distracte)
The red outlined panel shows InD tests. The green outlined panel shows InD in- terpolation tests, with cups evenly placed be- tween training positions. Purple and blue out- lined panels illustrate cases where the cup gradually moved from an in-distribution loca- tion to an OOD position. The remaining yellow outlined panels introduce OOD image distrac- tors assess the model's robustness.
Figure 3:
Introducing OOD visual dis- tractors of varying difficulty levels (Fig. (Fig.3yellow yellow bor-
der), including wildly OOD images of a cat and a dog; These settings allowed us to explore and analyze the generalization behavior and potential memory-driven characteristics of the Diffusion Policy. If the Diffusson Policy were performing action generalization, one would expect the fol- lowing in each scenario: (I:tnD-Interpolate)) interpolation in the action space; (2:0OD-Interpplltte) some action interpolation with degraded performance in the middle, where it is far from the train-
We also verified the hypothesis in the simulation environment provided by Rohomimie. For details, see Section 3.5 Appendix [D.2
We also conducted training with the standard scale set of 120 demonstrations, but for visualization we present results using smaller subset.
5

---

Published as a conference paper at ICLR 2026
ing examples; (00D-pxxtapolate) progressively degraded action performance the object moves farther from the training set; and (4:0OD-Distras) Distracttrs degraded action performance as the number and severity of distractors grows, with dog and cat inducing erratic action sequences.
In fact, all of these behavioral expectations are incorrect. every case, the Diffiusion Policy almost exactly reproduces one of the training action sequences explained below. This is consistent with our action lookup table hypothesis.
3.4 REAL-WORLD RESULTS
In this subsection, we introduce custom metric as a memory- audit, designed to quantify how closely an inference action sequence resembles sequences from the training set. It is defined as:
S = 1- =(,("),,_000 where s(r(") 7()) denotes the average Euclidean distance between the matched ((=0),(())
points the current action sequence and its closest training sequence, and s(r(1, x()) denotes the distance between the second-closes and the closest training action sequence. If an action sequence closely follows specific training sequence while maintaining clear separation from other nearby sequences, this provides strong evidence of memory-based retrieval rather than action generaliza- tion. Note that this similarity metric does not measure action quality, just action recall. For example, the robot may infer an ineffective action chunk, but if it closely matches one of the training action chunks, the similarity score will be high. We make no claims on the effectiveness of the actions generated by the policies; they are simply recalled from the training data.
3.4.1 DIFFUSION POLICY
SOMTI (rem)
R2
F2
R2
F2
sumtuntt Score
74
10
218
21
62
22
254
13
15
24
17
610
42
82
190
Stntee Sture
"4
14
12
130
12
Siestent
222
29
Figure 4: Similarity and distance statistics between each inference trajectory and training demonstrations for Diffusion Policy, ACT, and GROOT-N1.5 15. Bars indicate similarity scores, where sharper (more one-hot-like) distributions reflect stronger alignment with specific training example, signaling stronger memorization be- havior. From this comparison, Diffusion Policy exhibits the strongest action memorization, pletely OOD conditions.
under com-
Our experimental results provide compelling evidence supporting the hypothesis that the Diffusion Policy exhibits memory-base action cloning behavior. We first validate this in the in-distribution (InD) setting, where cups are placed exactly as they were during training. In this case, each action sequence almost perfectly overlaps with the corresponding training sequence (see Table [ where the memory-audit S close to 1. This indicates that the model is essentially replaying action quence memorized during training when presented with familiar inputs. This behavior persists even under visual OOD scenarios, where introduced distractors to the environment as shown in Fig. by the yellow border. Fig. 4s.44ipovide global view of similarity scores across all training trajectories under OOD scenarios. In the presence of distractors, almost all high-similarity

---

Published as a conference paper at ICLR 2026
matches are sharply concentrated on single training trajectory, indicating surprising OOD de- fault behavior. The diffusion model seems to revert to one or two fallback action sequences when presented with OOD images. Even when the in- put is entirely unrelated to the task, for example, an image of cat or dog, the diffusion model still produces an action sequence that closely resembles one from the training set. We believe these results show that the Diffusion Policy's 's decision- decision-making is largely governed by memory retrieval, rather than by generalized reasoning over the action space. In both clean and distractor scenarios, the model demon- strates consistent action replay behavior, support- ing our hypothesis that its decision- decision-making is funda- mentally memory-driven. Additional analyses and results also support this observation. In the InD- Interpolate cases, the Diffusion Policy outputs tra- jectory that closely matches one of the four corresponding nearest-neighbor training trajectories, while in the OOD-Extrapolate cases the generated trajectories are still similar the training trajec-
Table I: Average maximum similarity scores for each method across the various scenarios. Highest possible value is 1 (perfect match with training action chunk) and the lowest is O (far from all training action chunks). In OOD- Extrapolate (side) the cup is moved leftward out of the workspace and in OOD- DOD-Extrapolate (back) is when the cup is moved backwards. Note that since ALT directly selects a trajectory from the dataset, it always has a similarity of 1.
Sccnarios InD
ALT
DP
ACT
GROUT
I
0.935
0.465
0.783
OOD-Distractors
I
0.837
0.278
0.798
OOD-Interpolate
I
0.690
0.406
0.725
OOD-Extrapolate (side)
0.838
0,428
0.463
AOD.Fstmpolate (hack)
0.875
0.355
0.745
tories even once the cup is no longer in view of the 3rd person camera (see Table See Appendix D for more detailed InD visualization results and other OOD results.
3.4.2 ACTION CHUNKING WITH TRANSFORMERS
We evaluate Action Chunking with Transformers (ACT) Zhao et al. (2023) on the same datasets, test scenarios (including InD and OOD cases). ACT already exhibits notable degree of interpolation in InD scenarios, which are typically where memorization effects are most evident. In scenarios that visually in distribution, but require OOD action generalization, the ACT policy seems to sensibly compose action chunks to generalize its behavior. The 3D plot reveals clear evidence of interpolation: several inference trajectories fall cleanly between nearby training trajectories in the correct region.
However, in visual OOD regimes, when there are distractors, ACT produces almost random behav- ior, which is quite different from Diffusion Policy. As shown in Fig. 4 the model failed to reproduce the correct trajectory (traj15). Surprisingly, with distractor inputs (e.g., cats dogs), the Dif- fusion Policy returns a trajectory closely resembling some training demonstration (albeit not the correct traj15). In comparison, ACT's outputs are essentially random. For additional visualiza- tions, including detailed InD results and further OOD analyses, see Appendix D
3.4.3 GROOT-NI.5
Under full InD conditions, GROOT-NI.5Bjore et al. (2025) performs correct inference, consistently matching the correct trajectory. However, GROOT-NL.3 displays mixed behavior in InD-Interpolate: it sometimes interpolates trajectories between trained spots, while at other times it reverts to mem- orization. In more challenging OnDD-nnteeollae cases, GROOT-N1.5 does not memorize like the Diffusion Policy; instead, it produces trajectories that lie between training demonstrations. In OOD- Extrapolate. when the cup moves outside the workspace and beyond view, GROOT-NI.5 initially tracks correctly but then collapses average-like predictions, indicating lack of systematic ex- trapolation capability.
In the OOD-Distractors scenario, GROOT-NI.5 demonstrates strong robustness, likely stemming from its extensive vision- laniua--angug--acttion pre-training. Unlike the Diffusion Policy, GROOT-NI.5 successfully produces the correct trajectory (traj15) under such conditions. It does so by relying on consistent task-relevant cues from the first-person camera while ignoring irrelevant third-person distractor images. Fig. 4 |illustrates GROOT-NI.5's behavior under OOD-Distractors: the close align- ment between the predicted trajectory and traj15, together with the consistent similarity score distribution across distractors, provides strong evidence of robustness. In contrast, both ACT and the Diffusion Policy failed to produce the correct inference trajectory in some of these OOD cases. This indicates that GROOT-NI.5 relies on task-relevant cues (c.g., first-person view of the cup), likely thanks to its large-scale pretrained Rision-Laaguuae Model (VLM), rather than overfitting to
7

---

Published as a conference paper at ICLR 2026
spurious correlations, making it more reliable in visually perturbed environments. For other more detailed results, see Appendix D
3.4.4 ANALYSIS
Across the same pick-and-place task, we observe three distinct behaviors from the Diffusion Policy, ACT, and GROOT-N1.5. Diffussion Policy acts like retrieval controller in low-data settings: on InD scenes it effectively replays specific demo, and in the case of OOD, even when faced with a completely incomprehensible scenario, will still consider selecting one or two demonstrations from the training data as the output. ACT shows less action memorization compared to the Dif- fusion Policy: its chunked decoder with a smooth latent prior blends action segments, which can appear reasonable in InD-Interpolate and OOD-Interpolant but, under strong distractors, drifts to- ward prior or average chunks, essentially a hallucinated interpolation. GROOT-NI.S exhibits a mix of different behaviors: its diffusion- action head retains manifold- effect (drawing outputs toward training-ser set modes, which can cause memorization), while its VLM (pretrained on internet-scale data) provides task-relevant invariances that resist spurious cues, resulting in better OOD performance. Freezing the high-capacity Diffusion Transformer during fine-tune fine-tuning on small datasets avoids strong action memorization while keeping the model performant and robust, since the pretrained VLM anchors it with stable conditioning. However, once the Diffusion Transformer itself fine-tuned, the model shifts toward memorizing specific actions. We speculate that score-based models (e.g., DDPM Ho et al. (2020)) denoise by moving samples toward high-density regions of the conditional data distribution. Mathematically, with De 0,20 + ore, Tweedie's formula EEron 2011 gives
Zz (2e,)
(0(+)) = (ax + of V logp((xx|cond)) R
(2)
Qt
and the reverse-time SDE Song et al. (2020) is
dx = [f(x, t) - g² Vz log pr pc(cconddd) + g(t)dw.
Iterating these steps yields a manifold attraction effect: samples drift toward modes the diffusion policy's s conditional distribution p(A,1 Oz). In small-data regimes, those modes can degenerate into mixtures of narrow kernels around demonstrations, so generations tend stick near particular demos (a memory/replay effect).
Using the same hardware setup as for ACT and Diffusion Policy (RTX 4090), we kept the Diffusion Transformer frozen to avoid out-of-memory issues, following the official GROOT-N1.5 recommen- dation (--no-tune ddifffffuu fusiiiiin on m))ee.. In the above experiments, only the small adapter lay- and embodiment-specifie action heads (a few million parameters) were fine-tuned, while both the VI.M and the action expert remained frozen. However, we also fine-tuned the entire action expert (which includes the diffusion transformer), which led to much stronger action memoriza- tion compared to runs where the diffusion transformer was kept frozen. We believe that even large diffusion-based architectures tend memorize when fully tuned trained from scratch on small datasets. For pretrained VLA models, freezing most of the system and tuning only small set of parameters can achieve a good balance between generalizationn/tterplaaoo and -ndistribution performance especially when fine-tuning on small datasets.
3.5 SIMULATION RESULTS
In addition to our physical robot experiments, we further validated our hypothesis on the Robomimic benchmark using the official training demonstrations, evaluation trajectories, and pretrained check- points provided in the original Diffusion Policy release. As summarized in Table 2 we observe that, except for the Lift task, the rollout trajectories generated by Diffusion Policy exhibit a high degree of similarity to their training demonstrations. This consistent overlap strongly supports the model's impressive performance often arises from memorization rather than true gemeralization across unseen states. The comparatively lower similarity observed in the Lift task can be attributed to its simpler nature and the fact that it was trained for substantially fewer epochs in the official setup (only 300 and 450 epochs).
claim that
Overall, our findings reveal a systematic tendency of Diffusson Policies to reproduce familiar action patterns encountered during training, rather than synthesizing novel behaviors. Importantly, this
8

---

Published as a conference paper at ICLR 2026
Epochs
# Demos
Image Obs.
Epochs
# Demos
Low Dimensional
Can
11500
200
0.828 (4.032)
750
200
0.765 (5,408)
Square
2600
200
0.885 (2.086)
1750
200
0.799 (3.538)
Lift
300
200
0.578 (4.196)
450
200
0.580 (4.092)
Tool Hang
26550
200
0.962 (0.563)
3750
200
0.932 (1.016)
Transport 2750
200
0.965 (0.860)
2800
200
0,904 (2.356)
Block Push
4800
1000
0.963 (0.322)
Kitchen
4600
566
0.704 (14.231)
Table 2: Image Obs. models conditioned using image observations. Low Dimensional are models conditioned on low dimensional states. number is average highest similarity respect the training traeectories. Second number is the average euclidean distance to the nearest trajectory. Entries with are for tasks with no available checkpoints and therefore could not he evaluated. Epochs specify at what epoch the model weights were frozen. All demonstrations and model checkpoints were obtained from Chi et al. (023).
behavior was consistent across both simulation and real-robot evaluations, reinforcing the robustness of our observations. We acknowledge that extending this analysis to more complex, multi-skill, longer-horizon tasks, such sequential, multi-step missions or dexterous hand manipulation, would further strengthen the generality of our conclusions. However, these tasks typically require additional infrastructure and substantially larger demonstration datasets that are difficult obtain. We therefore leave this analysis for future investigations.
4 ACTION LOOKUP TABLE
Fission Model
Randam
Caloruiher
Building upon these results, we de- sign lightweight alternative method while still achieving comparable functionality show the bound of memorization mechanism. Our pol- icy, functioning similarly to a lookup table, uses an image encoder trained with contrastive learning as hash function to retrieve demonstration trajectories (as shown in Fig. [ our hypothesis holds, this method should deliver performance on par with the Diffusion Policy, while also offering more predictable fallback behaviors in the presence of out-of- distribution (OOD) inputs, therefore improving safety and robustness.
Ce
Ransom Reeraokkop
Model
Orgensl
Caloraller
Normilao
Resient
Corsaned
Now
Pdy
Kay
TOM
low-dim teature rep conerrataaiion dataoet
000
Fisly Trained
Insut
Model
Contrastive training (top, above the yellow dashed line) and inference (bottom) phases our ALT policy. The inference process has two stages: the green arrows indicate building the ALT latent space with the trained model, while blue arrows represent real-time inferenee.
Figure 5:
4.1 METHIO
At each timestep, the dataset includes a first-person end-effector view, a third- view, and the end-effector pose, denoted as: {(I), (.)) where I and I are the hand- and third-view images, and pi is the end-effector position and orientation. We employ fusion encoder to integrate these inputs into a unified embedding for contrastive learning (Fig (Fig.5)). ResNet- et al. (001)) pretrained on a large-scale dataset, serves as the image encoder backbone.
We adopt a contrastive learning framework to extract robust and discriminative representations from each frame using alignment across multiple modalities. We generate two different augmented views for each sample d; = (I I p:). Specifically, we apply a composed image augmentation pipeline A and Ag that transforms each input d, into augmented version views v. and e We feed of")
and is into the fusion encoder to obtain their embeddings ((1) and 2 These two embeddings form a positive pair, and train the network using the normalized temperature-scaled cross-entropy (NT-Xent) loss Chen et al. (2020) as our contrastive loss function:
exp (sim( (1)
2(21)/7)
C. 2B
Σ log
+ (1 ++ 2)),
(4)
Σ
2B kiti exp (sim( ()) 2k)/7)

---

Published as a conference paper at ICLR 2026
Table 3: Experimental Results. InD and OOD denote in- and out-of-distribution cases. The first column reports trajectory retrieval success rale, the second real-robot task success rale, and the remaining columns correspond to Fig. 3 MIT indicates model inference time (S). w/p and w/op refer to using or omitting the
end-effector pose as input.
Methods
Recall
InDs
In-1
OODI
OOD2
00D3
00D4
OODS
0OD6
00D7
MIT
K-D Tree
100%
63.3%
~00099
Diffusion Policy
100%
100%
/
/
/
x
4
~2.65
ALT w/ 7 = 0.9
100%
00D
0OD
0OD
0OD
00D
/
~40,00
ALT w/o p. 7 0.9
100%
00D 00D 00D 00D 0OD
0OD
00D
~0.009
ALT w/o p. 0.75
100%
100%
V
V /
OOD
0OD
~0.000
where sim(:.-) denotes the cosine similarity, and both inputs are L.2 12-normalized prior to computa- tion. The parameter is the temperature (set as 0.4 in practice), which controls the sharpness of the similarity distribution, effectively scaling the logits to adjust the contrastive loss sensitivity.
After training, we construct low-dimensional latent space to enable trajectory matching and pre- diction (Fig. 5). Each frame in the dataset is encoded by the fusion encoder, and the resulting embeddings, along with trajectory IDs and frame indices, are stored in a database. At inference, the current observation (views and pose) is encoded and compared via cosine similarity against this space. If similarity falls below a threshold y, the input is treated as OOD and triggers a safe fallback. Otherwise, the system retrieves the matched trajectory ID and frame index for real-time real-time prediction and policy execution. Although ALT is not a universally scalable solution in its most basic form, as larger datasets can result in higher memory usage and potentially slower lookup, its scalability can be improved through various strategics, including utilizing better data structures, approximate nearest neighbor search, state or action clustering Pertsch et al. (2025): ;; Rothfuss et al. (2018). or through action based contrastive learning Lee el al. (2025).
4.2 RESULTS
We evaluate the ALT policy through two experiments: task execution on a real robot under InD conditions, and performance under OOD-Distractors. Comparisons with KD-Tree retrieval and Dif- fusion Policy validate the explicit action memorization mechanism and the role of representation learning. Results in Fable successful matches (V) vs. failures (x). Green OOD cases indi- cate robust OOD detection with correct matches, while red OOD cases denote detected OOD inputs that lead to incorrect trajectories. Our ablations disentangle what enables memorization from how A naive KD-tree nearest neighbor baseline underperforms, showing that memo-
it is implemented 3 rization is not just feature proximity- -iithout task- embeddings and temporal alignment, trieval fails. In contrast, ALT, an explicit action lookup indexed by contrastive embeddings, matches diffusion policy performance These results suggest that representation-drivee policy acts an implicit retrieval system, while ALT exposes the same mechanism explicitly, with lower latency and an interpretable OOD flag.
InD and OOD tasks while using far less compute and memory.
memory is key to closed-loop success: diffusion
5 CONCLUSION
We investigated why diffusion policies are able to obtain strong performance with few task demon- strations in Visuomotor robot manipulation. We found that much of their closed-loop success in the low-data regime stems from retrieval-style action memorization, not sequence-level action general- ization. Our comparative study revealed that diffusion policies consistently recall verbatim action sequences from the training data, ACT exhibits interpolation across action segments showing better action generalization, and GROOT-NI.5 balances action memorization from the fine-tuning phase with robustness likely stemming from the large data volume pretraining of the policy. To probe the limits of simplicity of action memorization, we introduced ALT, a minimal, explicit lookup base- line, which reproduces diffusion-level behavior on small datasets while running orders of magnitude faster and offering a simple OOD flag. These findings reframe diffusion policies as powerful action retrieval memory systems that can attain proficiency in single task, but do not achieve generalizable manipulation skills.
Additional details the encoder architecture and the ablation study of the latent dimension are provided in the Appendix F
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by NSF grant 2342246, ONR grant N00014-23-1-2355 gift from Meta, and the ARPA-H HEART project. We are grateful for this support. This research was also partly supported by the Singapore Ministry Education (MOE), as well by an NUS Overseas Research Immersion Award.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
Johan Bjorck, Fermando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503. 14734, 2025.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310. 10639, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint ar arXiv:2311.5517,, 2023.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253 5270,, 2023.
Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2videa: Video editing using image diffusion. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 23206-23217, 2023.
Chen Chen, Daochang Liu, and Chang Xu. Towards memorization-frre diffusion models. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8425- 8434, 2024a.
Lili Chen, Shikhar Bahl, and Deepak Pathak. Playfusion: Skill acquisition via diffusion from language-annotated play. In Conference on Robot Learning, pp. 2012-2029, PMLR, 2023a.
Shang-Fu Fu Chen, Hsiang- Chhun Wang, Ming-Has Hao Hsu, Chun-Mao Lai, and thao-Hua Sun. Diffusion mmodll-augmenter behavioral cloning. arXiv preprint arXiv:2332.13335, 2023b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597-1607. PmLR, 2020.
Yunhao Chen, Xingjun Ma, Difan Zou, and Yu-Gang Jiang. Extracting training data from uncondi- tional diffusion models. arXiv preprint arXiv:2406. 12752, 2024b.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273666 2023.
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching with- out wn-the-wild robots. arXiv preprint arXiv:2402.1033 2024.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- tion, 106(496):1002116144 2011.
Xiangming Gu, Chao Du Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:23/0.02664, 2023.

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dominik Hintersdorf, Lukas Struppek, Kristian Kersting, Adam Dziedzic, and Franziska Boenisch. Finding nemo: Localizing neurons responsible for memorization in diffusion models. Advances in Neural Information Processing Systems, 37:88236-88278, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Anubhav Jain, Yuya Kobayashi, Takashi Shibuya, Yuhta Takida, Nasir Memon, Julian Togelius, and Yuki Mitsufuji. Classifier-free guidance inside the attraction basin may cause memorization. arXiv preprint arXiv:2411.167588 2024.
Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to roboties. IEEE Robotics and Automation Letters, 8(7):3956- 3963,, 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2426-2435, 2022.
Sung-Woof Lee and Yen-Ling Kuo. Diff-dagger: Uncertainty estimation with diffusion policy for robotic manipulation. arXiv preprint arXiv:2410.14868 2024.
Sung-War Wook Lee, Xuhui Kang, Brandon Y Yang, and Yen-Ling Kuo. Class: Contrastive learning
via action sequence supervision for robot manipulation. In Conference on Robot Learning, pp- 4743-4766. PMLR, 2025.
Hang Li, Qian Feng, Zhi Zheng, Jianxiang Feng, and Alois Knoll. Language-guided object-centric diffusion policy for collision-aware robotic manipulation. arXiv preprint arXiv:2407.00453 00451,, 2024.
Gabriel Loaiza- Ganzem Brendan Leigh Ross, Jesse C Cresswell, and Anthony Caterini. Diagnos- ing and fixing manifold overfitting in deep generative models. arXiv preprint 22Xiv:220..07172, 2022.
Guanxing Lu, Zifeng Gao, Tianxing Chen, Wenxun Dai, Ziwei Wang. Wenbo Ding, and Yansong Tang. Maniem: Real-time 3d diffusion policy via consistency model for robotic manipulation.
arXiv preprint arXiv:2406.01586. 2024.
Kazuki Mizuta and Karen Leung. Cobl-diffusion: Diffusion-basee conditional robot planning in dynamic environments using control barrier and 1yаpunov functions. In 2024 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 13801-13808. IEEE, 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171, PMLR, 2021.
Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint arXiv:21 2021.
Karl Pertsch, Kyle Stachowiez, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language--ction models. arXiv preprint arXiv:2501.09747, 2025.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International conference on machine learning, pp. 8821-8831. Pmir, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2244:2220400622, 1(2):3, 2022.
12

---

Published as a conference paper at ICLR 2026
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint 1XXx:244990008888 2024a.
Jie Ren, Yaxin Li, Shenglai Zeng. Han Xu, Lingjuan Lyu, Yue Xing, and Jiliang Tang. Unveiling and mitigating memorization in text-to text-to-image diffusion models through cross attention. In European Conference on Computer Vision, pp. 340-356. Springer, 2024b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Jonas Rothfuss, Fabio Ferreira, Eren Erdal Aksoy. You Zhou, and Tamim Asfour. Deep episodic Encoding, recalling, and predicting episodic experiences for robot action execution. IEEE Robotics and Aulomation Letters, 3(4):4007-4014, 2018.
memory:
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494 2022b.
Agon Serifi, Ruben Grandia, Espen Knoop, Markus Gross, and Moritz Bächer. Robot motion dif- fusion model: Motion generation for robotic characters. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-9, 2024.
Yorai Shaoul, Itamar Mishani, Shivam Vats, Jiaoyang Li, and Maxim Likhachev. Multi-robot motion planning with diffusion models. arXiv preprint arXiv:2410.03077 2024.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6048 00386,55, 2023a.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Under- standing and mitigating copying in diffusion models. Advances in Neural Information Processing 2023b.
Systems,
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456,, 2020.
Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 63-70. IEEE, 2024a.
Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, James Weimer, and Insup Lee. Memory- consistent neural networks for imitation learning. arXiv preprint 10.00:2310.06/71, 202.
Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, and Insup Lee. Regent: A retrieval- augmented generalist agent that can act in-context in new environments. arXiv preprint arXiv:2412.044 2024b.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image- tongee-t--nme to-image translation. In Proceedings of the IEEE/CVF Conference on Com- pufer Vision and Pattern Recognition, pp. 1921-1930, 2023.
13

---

Published as a conference paper at ICLR 2026
Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang, Haibo Zhao, Mark Yeatman, Jiuguang Wang, Robin Walters, and Robert Platt. Equivariant diffusion policy. arXiv preprint arXiv:2407.01812 2024a.
Yixuan Wang, Guang Yin, Binghao Huang, Tarik Kelestemur, Jiuguang Wang, and Yunzhu Li. Gendp: 3d semantic fields for category-level generalizable diffusion policy. In 8th Annual Con- ference on Robot Learning, volume 2. 2024b.
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In The Twelfth International Conference Learning Repre- sentations, 2024.
Wikipedia contributors. Stable diffusion. httpos//ea..wkkiloolaaarrr/////s Diffusion] 2024. Accessed: 2025-04-07.
Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: general, low-cost, and intuitive teleoperation framework for robot manipulators. In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 12166-12163. IEEE, 2024.
Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference robot learning, pp. 3536-3555. PMLR, 2023.
Zhengrong Xue, Shuying Deng, Zhenyang Chen, Yixuan Wang, Zhecheng Yuan, and Huazhe Xu. Demogen: Synthetic demonstration generation for data-efficient visuomotor policy learn- ing. arXiv preprint wiiv:2502.116322 2025.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations, arXiv preprint N4KW::2003309544 2024.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
14

---

Published as a conference paper at ICLR 2026
A DATA COLLECTION PIPELINE
A. ROBOT ARM DATA COLLECTION
Choosing an efficient, cheap, and safe method for data collection, crucial for robot imitation learn- ing, remains open problem. One of the most common solutions is to remote controllers, such as VR, 3D space mouse, or smartphones. However, due to high latency and indirect operation, the data collected in this way is often messy and low-quality, making it difficult accurately capture human skills. Fully synchronized systems with human operators, such as ALOHA AZoo et al.) (2023) and GELLO Wu et al. (2024), can solve this problem by allowing humans teleoperate the robot in a more intuitive way while tracking the actions of this system in real time. But, these methods require an additional specialized puppeting system, which incurs an additional cost. In comparison, UMI-gripper Chi et al. (2024) is a cheap, intuitive, and robot-agnostic solution for data collection. Yet, cannot be used our work it incompatible with situations where third-person perspec- tives are necessary, and limits the robot to a single manipulator that requires an expensive hardware interface. Thus, to collect the necessary data, we utilized a motion capture data collection method, MoDA, to capture high-quality action sequences with low latency.
Synchronized Data
Rystes
low-dinaoco
pizcer
haman hand relctwe
Figure 6: The data collection pipeline of MoDA (Motion-captured Demonstration for Arms). The green arrows indicate the process of aligning the relative positions of human's 's hand and the robotic arm, and the blue arrows indicate the data of collecting the robotic arm.
A.2 MOTION-CAPTURED DEMONSTRATION FOR ARMS
Data collection plays critical role imitation learning, as the quality and generalizability of the learned policy depends heavily on the fidelity of the demonstrations. In this work, we introduce MoDA (Motion-capturee
Demonstration for Arms), a streamlined and cost-effective data collection pipeline built upon motion capture (MoCap) systems that are commonly available in robotics labo- ratories (see Fig. (). This pipeline provides high-fidelity human demonstrations for the robot, where human demonstrator performs the cup grasping motion while wearing specialized trackers, and the system translates these motions into corresponding joint targets for a 6-DoF robot arm. MoDA can be extended to any other robot arm system with almost negligible cost, because our data col- lection pipeline is both task-agnostic and robot-agnostic. To collect the necessary expert training demonstration data, real time and map it directly to the end-effector of a robotic arm. Simultaneously, we estimate the inter-finger distance control the opening and closing of the gripper, thereby allowing nal when to grasp the cup. We then synchronize these actions with the corresponding in-hand and 3rd person camera views. Compared expensive telcoperation interfaces, our method does not need any active electronics or specialized wearables. Instead, the setup requires only a few 3D-printed brackets to attach passive IR reflective markers to the palm and fingers, making it an extremely low-cost and accessible solution when MoCap system is already available in the lab. Furthermore, unlike UMI Gripper, which requires direct human interaction during data collection, our setup allows human demonstrators to operate out of frame, thereby ensuring clean third-person video demonstrations. Compared to systems such ALOHA, which rely on external equipment like telcoperation interfaces instrumented gloves, our approach avoids the need for expensive or specialized hardware. In contrast to the UMI Gripper generated data, which often involves complex scenes with human demonstrators visibly present in
OptiTrack system to track the 6-DoF pose of the human palm in
sig-
systems such ALOHA, which rely on specialized and
15

---

Published as a conference paper at ICLR 2026
the frame, setup enables the collection of clean third-person video demonstrations where human demonstrators are minimally visible. This is particularly beneficial for training diffusion policies, it minimizes noise and ambiguity in both the action and visual observation spaces, reducing the risk of learning failures due to poor-quality data. In summary, unlike alternative setups that rely on specialized grippers, force sensors, teleoperation rigs, our system can be assembled in-house with minimal resources and negligible additional expense. Moreover, MoDA is not only task-agnostic, but also robot-agnostic, does not rely any specific type or model of robotic arm, making highly adaptable across different hardware platforms and manipulation scenarios. This flexibility enables seamless integration into wide range of experimental setups with minimal modification.
B EARLY STOPPING EXPERIMENT
To further support our hypothesis, we conducted an early stopping experiment. Early stopping is common technique used to prevent potential overfitting during training, with the goal of improving model's generalization ability. In this experiment, we reserved one-third of the dataset as a validation set and used the remaining two-thirds for training. During training, recorded both the validation loss and the mean squared error (MSE) between the predicted actions and ground-truth actions on the training set. The first metric, validation loss, is used to determine when stop training, thus preserving the model version with the best generalization. The second metric, actions MSE on training set, is used monitor the model's performance on the training set. As shown in Fig. 7 although the validation loss reaches its minimum at a certain point, the corresponding action MSE remains high, around 1800. This result indicates that overfitting a diffusion policy model to the training data is a necessary requirement for producing accurate trajectories, as choosing the best model (chosen based on the validation loss) results policy that cannot reproduce the correct in-distribution trajectories.
Validation Loss
Actiona MSE
action_mss msc-
m57mm111..8 1871.69
val locs=0.0467
44
slep
step
Figure 7: Early Stopping Experiment. The validation loss (left) reaches its minimum around step 650 before beginning to rise, indicating the onset of overfitting. However, at this point, the training action MSE (right) has not yet converged and remains as high 1800. This suggests that more extensive training is necessary for the Diffusion Policy to output effective actions, even the input in-distribution.
C DIFFUSION MECHANISM ANALYSIS
In this section, we present additional examples further illustrate the behavior of diffusion models under varying model capacities and dataset scales, as discussed in Section 3.2 Specifically, we examine three additional 2D manifolds: an ellipse g rectangle and a heart shape
Consistent with the observations from Section 3.2 low-capacity models trained on small datasets fail to accurately reconstruct manifolds, often producing noisy or collapsed outputs. Even when trained on large datasets, these models are limited by their representational capacity: simple shapes like ellipses can be approximated reasonably well (albeit still worse than with high-capacit, models), more complex structures suffer significant distortion. For example, due limited model expressive- ness, low- cow-cipacity models approximate the heart shape as a crude triangle and smooth out the sharp corners of the rectangle.
Memorization behaviors are obvious when high-capacity models are trained on small datasets. In- terestingly, although global structure reconstruction fails, local smoothness still emerge. For
16

---

Published as a conference paper at ICLR 2026
instance, while the overall manifold may not be recovered, segments of an ellipse still be accu- rately captured. This suggests that memorization in diffusion models is not absolute: when the train- ing data is locally dense, models may still interpolate between nearby points, preserving some local structure. However, when larger gaps exist between segments, interpolation fails, and the model in- stead memorizes discrete samples without capturing the broader underlying manifold. When high- capacity models are instead trained on large datasets, all manifolds are accurately reconstructed, almost perfectly matching the true geometry. In practice, though, acquiring such large-scale ex- pert demonstrations that sufficiently cover the state space remains a significant challenge in robotic manipulation tasks.
Small Data Simple Model (3k demas, 16x2)
Big Data Simple Model (100k demos, 16x2)
Small Data + Complex Model (3k demos, 1024x10) Big Data Complex Model (100k demos, 1024x10)
Input Points Ground Iuth
Input Points Ciccnd trub
Figure 8: Training a generative model from 2D points on a ellipse-shaped ID manifold. Orange points indicate training samples, gray points are noisy inputs, blue points are denoised outputs, and cyan lines shows the denoising directions.
D ADDITIONAL RESULTS: DIFFUSION POLICY
D.1 REAL-WORLD
Fig. II Shows the trajectory similarity between the predicted trajectory and the nearest ground-truth trajectory for the real-world cup placement experiment. The distance to the nearest neighbor (yellow polyline) is near zero (implying high similarity). while the distance the second nearest trajectory is substantially larger, resulting in similarity score that is close to 1 (the blue bar).
Fig. 12 presents additional supplementary visualization of the trajectory matching results under in- distribution conditions. In the left panels, for each case, the first blue bar represents the similarity score the most closely matched training trajectory, while the second blue bar the similarity score of second closest trajectory. The consistently high top-1 similarity scores, combined with significant gaps to the second closest match, indicate clear and confident retrieval from the training data. As shown in the right panels, the closest trajectory (blue) in the training dataset almost perfectly over- laps with the inference trajectory in all examples, showing that the Diffusion Policy can accurately retrieve the correct demonstration when the input remains within the training distribution. These
17

---

Published as a conference paper at ICLR 2026
Small Data Simple Model (3k demes, 16x2)
Big Data Simple Madcl (100k, demos, 16x2) Irpct
Small Dota Complex Model (3k demos, 1024x10) Big Data Complex Model (100k demos, 1024x10)
Irset Nu
Irpul Tarta
Figure 9: Training generative model from 2D points on a rectangle-shape ID manifold.
Small Data Simple Model (3k demes, 18x2)
Big Data Simple Madel (100k demos, 16x2) Ingut
Small Dota Complex Model (3k demos, 1024x10) Big Data Complex Model (100K demos, 1024x10)
Insot nu
Frput Narts
Figure 10: Training a generative model from 2D points on a heart-shaped ID manifold.
sults strongly support our hypothesis that the Diffusson Policy depends on memory-based retrieval mechanism achieve its compelling results. The sharp similarity peaks and trajectory overlaps provide strong evidence that the model not merely approximating the behavior, but is explicitly recalling memorized training trajectories under in in-distribution conditions.
We further analyzed several additional -iitt--dittiiuution (OOD) scenarios, including: placing the cup evenly between three four in-distribution positions (as shown in Fig. 13). gradually moving
18

---

Published as a conference paper at ICLR 2026
Figure 11: Similarity and distance statistics between inference and training trajectorics. Each sub- plot shows the similarity scores (blue bars) and average distances (orange lines) between the Diffu- sion Policy inference and training trajectories. The large gap between the closest and second-closest neighbors indicates strong alignment with specific training examples.
the cup out of the field of view from the edge of an in-distribution position (Fig. 14 and slowly tran- sitioning the cup between two distant in-distribution positions (Fig. [5]. is evident that under these OOD conditions, the Diffusion Policy continues to produce trajectories that closely resemble those seen during training. These experimental results provide further support for our core hypothesis regarding the memory-driven behavior of diffusion policies.
D.2 SIMULATIONS
We also see similar results for the two different simulated manipulation tasks: Can, and Square. Both tasks drawn from the Robomimic simulation benchmark and contain 196 training trajectories and four validation trajectories. In the Can task, the model is taught to transfer a can from one table to another table while in the "Square" task, the model is trained place square ring on square peg. We observed that these experiments also display action memorization (see Fig. 16 and Fig. 20). as each training observation was found closely align with the corresponding ground-truth demonstration. We observe similar memorization when examining the diffusion policy's 's rollouts from observations taken from the validation set (see Figs. 17 through 19 and Figs. Figs.[T] through 23) for can and square experiments respectively. Overall, in the Can task, we observed average similarity score of 0.88 and an average deviation of 2.0 mm. In comparison, in the "Square" task, observe similarity score of 0.82 with average deviation of 4.4 mm.
ADDITIONAL RESULTS: ACT AND GROOT-NI.5
We provide further results for experiments discussed in 3 for both the ACT and GROOT-NI.5 poli- cies on the following scenarios being: (I:Inn--Inerrpllate) (2:00D-Interpolate) and (3:0OD- Extrapolate).
E. ACT
For OOD-Interpolation in 24 we see trajectories that lie cleanly between training trajectories,
demonstrating successful interpolation.
The DOD-Extrapolation case in 25 often extrapolates the wrong direction leading to incorrect trajectories.
19

---

Published as a conference paper at ICLR 2026
Encoder-Dimm
InDs
InD-1
OODI
OOD2
OOD3
OOD4
OODS
OOD6
O0D7
ResNet-64
100%
K
X
ResNet-128
100%
X
ResNet-256
100%
/
/
X
SimpleCNN-64
19.35%
X
X
X
X
X
SimpleCNN-128
22.58%
X
X
X
X
X
X
X
X
SimpleCNN-256
22.58%
X
X
X
X
X
X
X
X
VIT-64
12.9%
V
J
J
X
VBT-128
12.9%
V
V
/
SMKIK
VYT-256
16.13%
X
X
X
X
X
XXXXXX
X
CLIP-64
22.58%
V
CLIP-128
100%
CLIP-256
51.61%
/
Swin-64
77.42%
Swin-128
83.87%
Swin-256
90.32%
x
X
Table 4: Performance of ALT under Different Encoder Architectures and Feature Dimensions.
E.2 GROOT-N1.5
From the InD-Interpolation 26 we see high similarity scores to single trained trajectory over many samples. This demonstrates high action-memorization while sometimes interpolating.
The OOD-Interpolatioo in 27 show some successful interpolation, but we see lower action-
memorization through the increased number of higher similarity score candidate trajectories.
For OOD-Extrapolation, highly OOD inputs can lead to incorrect extrapolations, There is some minor-action memorization for successful extrapolations.
F ENCODER ARCHITECTURE AND LATENT DIMENSION
To further understand role representation quality in ALT, we evaluated the system under five different encoder architecturs-RReeee--88 SimpleCNN, ViT, CLIP, and Swin-while varying their feature dimensions. The results in Table 4 reveal several consistent trends. First, ResNet- 18 achieves uniformly high performance across all dimensions, reflecting the stability and strong in- ductive biases of convolutional networks, which are well suited for manipulation-oriintee visual inputs. In contrast, SimpleCNN and ViT perform significantly worse: the former lacks sufficient representational capacity, while the latter requires large-scale pretraining and offers limited spatial inductive bias, making its embeddings unstable in our domain. CLIP produces strong features but is highly sensitive to output dimensionality- dimensions manifold, whereas excessively large dimensions distort it, leading degraded contrastive align- ment. Finally, Swin exhibits clear performance improvement as dimensionality increases, consis- tent with its hierarchical, local-window design that combines CNN-like locality with transformer expressiveness. Overall, these results highlight that encoder choice and representation quality have a substantial impact on ALT performance, confirming that representation quality is critical factor for contrastive training.
dimeeiss that are too small under-represent the feature
20

---

Published as a conference paper at ICLR 2026
Tarpet Thaj
Target Traj (Scatter)
Closest Traj (Full
Closest The (Matchhd)
2nd Closest Tha (Matched)
340
120
Target (Line)
Target Traj (Scatter))
2nd closest Traj (Full)
Closest (Matched)
2nd Closest Traj (Matched)
Start
Z
280
250
275
270
325
260
350
40
100
Target Traj (Line)
Target
Closest Thaj (Pull)
330
Closest Traj (Matched)
320
2nd Closest (Matched)
Start
Ooal
290
280
270
260
-20
40
60
80
100
380 320
300
X
Figure 12: The Diffusion Policy inference result analysis under in-distribution conditions. On the left, each subplot shows the similarity distribution between the query inference trajectory and all stored trajectories in the database. On the right, the three figures provide 3D visualizations of repre- sentative matching cases. The green line represents the inference trajectory, while the blue and red dots show the closest and second-closest trajectories retrieved from the training set, respectively.
21

---

Published as a conference paper at ICLR 2026
Figure 13: Inference trajectory analysis when the cup is placed between multiple In-distribution positions.
22

---

Published as a conference paper at ICLR 2026
Figure 14: Inference trajectory analysis when the cup is gradually moved out of view from an in- distribution boundary.
Figure 15: Inference trajectory analysis when the cup is gradually moved between two distant in- distribution positions.
23

---

Published as a conference paper at ICLR 2026
Similartty Train Trajectories (Per Normalized)
2000 600000
Figure 16: subset the total analysis done for the roll-outs of Diffusion Policy model from the training set with respect to all ground-truth training trajectories. The model was trained on the "Can" pickup benchmark.
Similarity to Ground-Truth Trajectories (Per-Rollout Normalized)
0.
-0.2
Palluu
Pallact
Sintarry Score
04
Figure 17: Analysis of a Diffusion Policy model roll-outs on the validation set with respect to all ground-truth validation trajectories. The model was trained on the "Can" pickup benchmark.
24

---

Published as a conference paper at ICLR 2026
Similarity Ground-Truth Trajectoriss (Pwe-Roitout Normaliowd)
Polloc
MO
Figure 18: Analysis of Diffusion Policy model roll-outs on the validation set with respect both the ground-trun truth training and validation trajectories. The model was trained on the "Can" pickup benchmark.
Similarity to Ground-Truth Trajectories (Nar-Roliout Normalizo))
Figure 19: Analysis of a Diffusion Policy model roll-outs the validation set with respect to the ground-truth training trajectories. The model was trained on the "Can" pickup benchmark.
25

---

Published as a conference paper at ICLR 2026
Simiarity to Train Trajectories (Per Rollout Nomalizedi
02
Figure 20: A subset of the total analysis done for the roll-outs of a Diffusion Policy model from the training set with respect to all ground-truth training trajectories. The model was trained on the "Squurre" pickup benchmark.
Simllaity to Train Trajectories Normaizze))
Figure 21: Analysss of Diffusion Policy model roll-outs on validation set with respect to all ground- truth validation trajectories. The model was trained on the "Square" benchmark.
26

---

Published as a conference paper at ICLR 2026
Similarity (Per-Rollout Normalzee)
AM
Figure 22: Analysis of a Diffusion Policy model roll-outs on validation set with respect to both the ground-truth training and validation trajectories. The model was trained on the "Square" pickup benchmark.
Similarity to Train Trajectortes (Per Rolbout Nomalized)
Figure 23: Analysis of a Diftusion Policy model roll-outs on training set with respect to all ground- truth training trajectories. The model was trained on the "Square" benchmark.
Figure 24: ACT Out of Distribution Interpolation Distance and Similarity Scores Across Six Roll- outs.
27

---

Published as a conference paper at ICLR 2026
Figure 25: ACT Out of Distribution Extrapolation Distance and Similarity Scores Across Six Roll- outs.
Melee
Motert
Figure 26: GROOT-NL.5 In Distribution Interpolation Distance and Similarity Scores Across Six Rollouts.
Meleet
Meleed
Figure 27: GROOT-NI.5 Out of Distribution Interpolation Distance and Similarity Scores Across Six Rollouts.
Figure 28: GROOT-NI.5 Out of Distribution Extrapolation Distance and Similarity Scores Across Six Rollouts.
28


Reviews:
[{"review_id": "winVEG8DiE", "reviewer": "Reviewer_aGTM", "summary": "This paper investigates why diffusion policies perform remarkably well in robot manipulation despite being trained on very few demonstrations. The authors hypothesize that diffusion policies do not truly generalize actions, but instead memorize and retrieve action sequences-similar to a nearest-neighbor lookup table-based on the closest-matching observation in latent space. Through real and simulated pick-place experiments, they show that Diffusion Policy consistently reproduces training trajectories even under out-of-distribution inputs, while ACT interpolates actions but fails in OOD settings, and GR00T combines some generalization with robustness due to large-scale pretraining. Motivated by these findings, they introduce a simple Action Lookup Table (ALT) approach using contrastive visual embeddings, which matches Diffusion Policy performance in low-data regimes while being faster and explicitly flagging OOD cases. The key takeaway is that the strength of Diffusion Policy in low-demo robot learning stems largely from memorization, not generalization, and that simple retrieval-style methods can provide similar performance with far lower cost and greater interpretability.", "questions": "1. How much does ALT generalize, or is capable of generalizing to novel object placement arrangements, lighting, etc.? How does lookup table size scale with generalization capabilities?\n2. As the number of demonstrations grows, how does ALT’s retrieval latency, memory footprint, and OOD detection accuracy scale? Would a nearest-neighbor search remain feasible for larger datasets or multi-task scenarios?\n3. How would the ALT policy perform on tasks that require longer-horizon reasoning, contact-rich dynamics, or multi-step planning? Would its lookup-based retrieval break down when task stages are less visually correlated?\n4. The paper uses a contrastive encoder to map images to latent representations. How sensitive is ALT’s performance to the choice of encoder architecture or the quality of contrastive learning?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organized, progressing logically from hypothesis to experimental validation and practical takeaway.\n2. The paper performs a sound analysis of Diffusion Policy and other single-task imitation learning methods that are known to perform well in the low-data regime. Diffusion Policy is indeed recommended to train to overfitting, and the paper tries to make the effects of that trend very clear.\n3. The authors design structured experiments (In-distribution, OOD-interpolate, OOD-extrapolate, and OOD-distractor) to dissect memorization versus generalization behavior, making the conclusions grounded and interpretable.\n4. The introduction of the Action Lookup Table (ALT) policy provides a transparent, computationally efficient alternative that matches diffusion policy performance while being ~300x faster and offering explicit OOD detection.", "weaknesses": "1. While I appreciate the analysis, the paper, in my opinion, does not provide a novel viable solution. It just presents a simple alternative to Diffusion Policy, which is not scalable to something that current large behavior models aim for. While I agree that Diffusion Policy is prone to overfitting, then so is any high-capacity supervised learning model trained on less data. Diffusion Policy did not just present one specific architecture or loss function, but provided a starting point for future approaches, which might be able to have less overfitting and more generalization while maintaining the multi-modality and dexterity that Diffusion Policy has. As such, I did not see anything in this paper which I did not expect from low-data supervised learning models, but it does not mean that researchers should replace them with non-scalable methods such as presented in this paper.\n2. The experiments in this paper are primarily conducted on simple manipulation tasks (e.g., cup grasping, pick-and-place), making it unclear whether the findings generalize to more complex, multi-skill or long-horizon robotic tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "235Jf6tbvz", "reviewer": "Reviewer_zjbw", "summary": "The main claim of the paper is that diffusion policy in low-data regime memorize demonstration actions during training and looks it up during inference, and proposes Action Lookup Table policy. Experiments with small datasets show that diffusion outputs memorized action chunks even when given out of distribution inputs. ALT also introduces out-of-distribution detection and stochastic kNN sampling, and observes that overfitting can be beneficial in manipulation setups.", "questions": "- Does this claim hold on a higher dim action space, and environments with multiple objects? Does it hold on a larger scale dataset?\n- Ablation: how much does the representation learning matter?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear claim with sound evidence from sim and real experiments.\n- ALT performs well in the low data regime and is orders of magnitude faster than diffusion / ACT.\n- Out-of-distribution experiments show that diffusion policy replays action chunks from the training set.", "weaknesses": "- My main concern is novelty. The idea that kNN lookup as a policy class on contrastively learned features works well in the low-data regime has been well-established in literature [1], and it is also well-known in the field that overfitting can help with downstream rollouts in behavioral cloning.\n- Limited experiments. Main claims are mostly validated with cup grasping experiments, with low effective state and action space dims. It would be good to see whether the diffusion memorization holds in higher dim action / state spaces (e.g. dexterous manipulation for higher dim actions, or benchmarks with more objects like LIBERO), as well as a sweep over dataset scale.\n\n[1] The Surprising Effectiveness of Representation Learning for Visual Imitation. Pari et al., 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dPNfQ2mqjm", "reviewer": "Reviewer_p3Mt", "summary": "The authors hyphothsize that visuomotor policies, e.g., diffusion policies (DP) memorize the image-action pairs instead of learn to generlize on them. To verify this, they conduct a systematic emprical study including analysis and experiments ranging from toy data to real-robot data. Moreover, they evaluate on three representative policy architectures, namely DP, Action Chunking with Transformers (ACT), and GR00T and found that they exhibit different behaviors. Based on this, a simple alternative called action lookup table implemented as observation encoder for action retrievial is introduced and requires less memory while providing greater run-time efficiency and OOD monitoring capability.", "questions": "- The influence of the feature dimensionality of ALT on the task performance seems relevant. Is this a core desgin question?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- An interesting and comprehensive emprical study for a highly relevant but under-explored problem of current visuomotor imitation learning framwork. I believe that this investigation is inspiring for both researchers and practioners.\n\n- The presentation of this submission is clear, well-organized and easy-to-follow along with illusrative figures and visualization;\n\n- The toy experiments and visuliazation provide a great example for the reader to under the problem at the beginning;\n\n- The analysis of results across three architectures is presented clear and concise, while also making connection to the manifold attraction effects. \n\n- The experiment is extensive from the perspective of hyphothesis validation. Morever, the experimental design (ID and OOD setups), policy architectures and evaluation metrics selection further consolidate the strictness of this empirical investigation.\n\n- This study is practical-relevant and highly informative for practioners, because in the real-world scenarios a large amount of robot data may be costly to collect.", "weaknesses": "- The current study and robot-related experiments are mainly done under the low-data regime. It remains unclear what implications this study can bring for the visuomotor imitation learning at larger scale. It would be valuable to provide insights (discussion or experimental study) about the validity for a large scale of data, e.g., 200, 2000 or 20k demostrations are provided.\n\n- Though the presention is clear and great, I would still suggest to shorten the part of analysis for the three architectures or to present the difference or similarites of them in a more visual way. Then more space can be left for a slightly proper experiment part, including setup introduction and more results. \n\n- In the experiment, the performance of ACT and GR00T for reference is missing; In table 2., success rates on OOD scenarios would be nice to have to investigate the expected behaviors described in section 3.3.\n\n- Regarding the proposed ALT, besides the improved computation efficiency, its OOD detection capability is highly desirable for robustness enhancement. It would be highly valuable to perform more experiments for this part to understand how effective it can be. For example, a comparison with other density-based approaches such as normalizing flows, which can be trained on the latent features. \n\n[1] Feng, J., Lee, J., Geisler, S., Günnemann, S., & Triebel, R. (2023, December). Topology-Matching Normalizing Flows for Out-of-Distribution Detection in Robot Learning. In Conference on Robot Learning (pp. 3214-3241). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G9bsGOg88a", "reviewer": "Reviewer_kN3g", "summary": "The work investigates the effectiveness of Diffusion Policy (DP) for visuomotor robot manipulation, proving that their performance in low-data regimes stems from action memorization rather than any sort of generalization. Empirical evidence demonstrates that the DP tends to recall action sequences from its training data, even when faced with highly OOD inputs, providing robustness in those scenarios. The authors compare this behavior with other architectures like ACT, which shows action interpolation but lacks robustness, and GR00T N1.5, which shows generalization due to pre-training. To validate their hypothesis on DP, the researchers introduce a simpler, faster alternative called the Action Lookup Table (ALT) policy, which explicitly relies on nearest neighbour lookup in an embedding space, to match DP's performance while also providing OOD detection. The (image & joint pose) encoder that creates the embedding is trained via contrastive learning.\n\nBasically, this work does a great job reframing DP, in data-sparse regimes, as a memory retrieval system.", "questions": "1. \"The diffusion model seems to revert to one or two fallback action sequences when presented with OOD images.\" Based on FIg-4, it seems like that trajecotry is traj4 ? Why do you think there is a single fallback demo? And why is that the chosen traj? Is it the one earliest in the training? What characteristic does traj4 have that sets it apart from the others. A deeper discussion into might provide insight into the importance of each demo in the training pipeline.\n\n2.Fig 5: Please improve visibility of the lines in this figure. The green and blue arrows are too similar, especially for such a small figure. Consider higher-contrast colours, or use dots and dashes.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Well explained idea. Strong paper to put forward the point of memorization.\n2. The example of grasping the cup, along with the memory-audit metric S, is simple and effective.", "weaknesses": "1. Sec 3.3 & Fig 3. More OOD variations could be added. Specifically, what if you change the cup object itself?\n2. Fig 4 is quite confusing at first sight. I had to read it again to understand that the lines indicate distance scores. Considering removing information to make it clearer. Do you need these absolute distance scores, when you already have a similarity metric? in that metric, you can see the one-hot like nature of DP anyway.\n3. ALT does not attempt to interpolate between training data actions. However, this may be a design choice by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)