Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos
Authors: Yufan Zhou
Zhaobo Qi
Lingshuai Lin
Junqi Jing
Tingting Chai
Beichen Zhang
Shuhui Wang
Weigang Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: procedure planning
diffusion
U-Net
temporal logic interpolation
action prediction
mask
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A masked temporal interpolation predictor for procedure planning in instructional videos

Abstract: In this paper, we address the challenge of procedure planning in instructional videos, aiming to generate coherent and task-aligned action sequences from start and end visual observations. Previous work has mainly relied on text-level supervision to bridge the gap between observed states and unobserved actions, but it struggles with capturing intricate temporal relationships among actions. Building on these efforts, we propose the Masked Temporal Interpolation Diffusion (MTID) model that introduces a latent space temporal interpolation module within the diffusion model. This module leverages a learnable interpolation matrix to generate intermediate latent features, thereby augmenting visual supervision with richer mid-state details. By integrating this enriched supervision into the model, we enable end-to-end training tailored to task-specific requirements, significantly enhancing the model's capacity to predict temporally coherent action sequences. Additionally, we introduce an action-aware mask projection mechanism to restrict the action generation space, combined with a task-adaptive masked proximity loss to prioritize more accurate reasoning results close to the given start and end states over those in intermediate steps. Simultaneously, it filters out task-irrelevant action predictions, leading to contextually aware action sequences. Experimental results across three widely used benchmark datasets demonstrate that our MTID achieves promising action planning performance on most metrics.

Published as a conference paper at ICLR 2025
MASKED TEMPORAL INTERPOLATION DIFFUSION FOR PROCEDURE PLANNING IN INSTRUCTIONAL VIDEOS
Yufan Zhou-
Zhaobo Qi Lingshuai Lin' Junqi Jing'
Tingting Chai" Beichen Zhang' Shuhui Wang? Weigang Zhang
Harbin Institute of Technology, Weihai
KKyy Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS
(yīzhou, Isiin, jqjing]estu.hie hit. edu. en, wangshuhui pict... ac. cn, {gizb, beiczhang, ttchai, wgzhang) whinnnggghhhh....... @hit edu. en
ABSTRACT
In this paper, address the challenge of procedure planning in instructional videos, aiming to generate coherent and task-aligned action sequences from start and end visual observations. Previous work has mainly relied on text-level su- pervision to bridge the gap between observed states and unobserved actions, but it struggles with capturing intricate temporal relationships among actions. Build- ing on these efforts, we propose the Masked Temporal Interpolation Diffusion (MTID) model that introduces a latent space temporal interpolation module within the diffusion model. This module leverages learnable interpolation matrix to generate intermediate latent features, thereby augmenting visual supervision with richer mid-state details. By integrating this enriched supervision into the model, we enable end-to-end training tailored to task-specific requirements, significantly enhancing the model's capacity to predict temporally coherent action sequences. Additionally, we introduce an action-aware mask projection mechanism to restrict the action generation space, combined with task-adaptive masked proximity loss to prioritize more accurate reasoning results close to the given start and end states over those intermediate steps. Simultaneously. it filters task-irrelevant ac- tion predictions, leading to contextually aware action sequences. Experimental results across three widely used benchmark datasets demonstrate that our MTID achieves promising action planning performance on most metrics. The code is available
INTRODUCTION
Recently, procedure planning has exhibited critical reasoning capability for solving real-world chal- lenges complex domains, such as robotic navigation (Sermanet et al. 20224] Bhaskara et al. 2024) and autonomous driving (Wang et al., 2024 Liao et al.) (022)). Among them, procedure planning in instructional videos (Zhao et al., 2022] Wang et al. 2023b Li et al. 2023) has been widely con- cerned because of its wide application scenarios, which involve identifying and generating coherent action sequences that align with the task's objectives, given the start and end visual observations.
In the field of procedure planning in instructional videos, the primary challenge lies in modeling the temporal evolution mechanism among actions and identifying pertinent conditions that can ef- fectively steer the generation of intermediary actions in scenarios where information scarce. As depicted in Figure aa), many scholars have resorted to capturing different forms of auxiliary in- formation about the intermediate states bridge the gap between observed states and unobserved actions. For example, event- based supervision (Wang et al. (202aa) leverages key task events to help the model learn temporal action structures, while task label supervision (Wang al. (023b) uses task-specific labels for better alignment with the task objective. The probabilistic procedure knowl- edge graph (Nagasinghe et al. (2024) provides structured knowledge enhance the understanding of action dependencies. Additionally, Niu et al. (2024) leverage large language models (LLMs) to describe state changes, improving the model's 's grasp of causal relationships by combining visual
"Corresponding Authors.

---

Published as a conference paper at ICLR 2025
Obsrrotion
Supervision Scope
Predicted Action
Latent Feature
pour
whist
put meringue
600
sught
misture
into oven
task Instructions..
Text-Level Supervision
(a) Supervised by condition supervision obtained from various preprocessing methods.
pour
add
whise
put meringue
sugar
misture
into oven
addrectionaftrrmocoooo
interpolate
interpllaee
Supervised by latent visual features obtained by interpolation.
Figure I: The core idea to solve procedure planning with previous methods and ours.
and language descriptions. However, all these methods are limited providing text-level supervi- sion, resulting in less detailed and comprehensive information, and failing to precisely capture the temporal relationships between actions. Furthermore, these methods decouple the acquisition of supervisory information from the intermediate action reasoning process, hindering effective collab- oration and interaction between the two. Consequently, becomes challenging to fully integrate and adapt current action reasoning tasks.
Based on the above analysis, we propose the Masked Temporal Interpolation Diffusion (MTID) model for procedure planning in instructional videos. As shown in Figure [b), the core concept is to leverage intermediate latent visual features, generated synchronously by a latent space temporal interpolation module. to provide comprehensive visual-level information for mid-state supervision. In the meanwhile, the generated visual features are directly injected into the action reasoning model, ensuring the generation of intermediate supervision information can be effectively applied to the current action reasoning task through end-to-end training.
Specifically. MTID comprises three core components: task classifier, latent space temporal in- terpolation module, and a diffusion model framework that integrates the DDIM strategy. In the first stage. transformer-based classifier predicts the task class label for the entire instructional video, given the start and end observations. This prediction serves as the foundation for subsequent ac- tion generation. The latent space temporal interpolation module is designed to capture and model temporal relationships. It employs an observation encoder to transform the visual features of the ob- servations into latent features that maintain temporal dependencies. A latent space interpolator then generates intermediate features using a learnable interpolation matrix, which dynamically adjusts the interpolation ratio to task-specific requirements. These interpolated features are refined through transformer blocks, enhancing their temporal coherence and capturing the dependencies between action sequences. In the third stage, during the denoising phase for generating action sequences, the input matrix is constructed by concatenating the task class label, observed visual features, and Gaussian noise. A masked projection is applied to exclude irrelevant actions, ensuring that the gen- erated actions remain within the desired range. To accelerate inference, DDIM is used throughout the iterative process. To further ensure task relevance, we adopt a task-adaptive masked proxim- ity loss which gradually decreases its focus toward the central features, reinforcing supervision on intermediate latent features while penalizing irrelevant actions, thereby constraining the generation process. By leveraging both the start and end observations, our model accurately predicts target ac- tion sequences, as demonstrated by experimental results on the Cross Task, COIN, and NIV datasets.
The main contributions of this paper are as follows:
We propose a Masked Temporal Interpolation Diffusion model with a mask to limit action initial- ization and task-adaptive masked proximity loss enhance accuracy.
We use a latent space temporal interpolation module to extract intermediate visual features with temporal relationships from the start and end states to guide the diffusion process.
Extensive experiments are conducted on several widely used benchmarks, showing significant performance improvements multiple tasks using the proposed method.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Procedure Planning in Instructional Videos. Procedure planning involves generating goal-
directed action sequences from visual observations in unstructured videos, like Qi et al. (2021
2024). Our work builds on PDPP (Wang et al.) 2023b), which models action sequences using dif- fusion processes. Earlier approaches focus on learning sequential latent spaces (Chang et al. 2020)
and adversarial policy learning (Bi et al. 2021). Recent methods introduce linguistic supervision for step prediction (Zhao et al. 2022). mmask-and-pree
strategies for step relationships (Wang et al., 2023a), and breaking sequences into sub-chains by skipping unreliable actions (Li al.) 2023)
sequeneeeee KEPP Nagasinghe et al. 2024) incorporates probabilistic knowledge for step sequencing, while
SCHEMA (Niu et al., 2024) tracks state changes at each step. However, none of these methods cus on the visual-level temporal logic between actions. Our approach introduces mid-state temporal supervision to capture these relationships, resulting in more accurate and efficient predictions.
Diffusion Probabilistic Models for Long Video Generation. Recent advances in diffusion proba- bilistic models (Croitoru et al.) 2023), originally popularized for image generation (Rombach et al. 2022). have achieved significant progress in generating long video sequences (Weng et al. 2024 Zhou et al. 2024a Jiang et al.) 2024). StreamingT22V (Henschel et al.) 2024) excels in produc- ing temporally consistent long videos with smooth transitions and high frame quality, overcoming the typical limitations of short video generation. Story DoryyDfffsion (Zhou et al. 2024b) further en- hances sequence coherence through consistent self-attention, enabling the creation of detailed, visu- ally coherent stories. The success of these models has inspired us to incorporate auxiliary temporal coherence mechanisms in our approach, which we believe are critical to achieving accurate and high-quality action prediction with our method.
3
METHOD
3.1 OVERVIEW
Following Chang et al. (2020), given an initial visual observation V, and target visual observation Vg, both are short video clips indicating different states of the real-world environment extracted from an instructional video, the procedure planning task aims to generate sequence of actions a1. that transforms the environment from V, to Vg. where T denotes the number of planning time steps. This problem can be formulated p(a1:1 Vs, Vg).
Considering the weak temporal reasoning ability caused by the absence of intermediate visual states, especially in long video scenarios, we propose the Masked Temporal Interpolation Diffusion (MTID) framework, which employs a denoising diffusion model to rapidly predict the intermedi- action sequence d1.7- As outlined in the following formula, MTID decomposes the procedure planning task into three sub-problems,
lf p(arr U1:M,C, c, Vs, Vy)p(ur. V., Vy)p(c Vss Vy)dur (I)
p(arr Vs, Vgy
The first sub-problem entails capturing information about the task be completed about the whole instructional video, serving as the basis for subsequent reasoning. As shown in Figure 2 this task supervision stage solves a standard classification problem using transformer encoder to extract features from observation pair {V., Vg and transform them into task class label c.
The second sub-problem focuses on reconstructing M intermediate visual features V1.M from V,, and Vg to address the lack of mid-state visual supervision and reveal hidden temporal logic within the action sequences, which achieved through our latent space temporal interpolation module.
The final sub-problem involves generating action sequence 01:2" based on the task information and interpolated intermediate features. Specifically, we first construct the input matrix &N for the de- noising steps, which consists of three dimensions. The task class dimension contains the captured task information e for each reasoning step. The observation dimension contains the visual observa- tions of the start and goal states {V.. Vg), where the intermediate states set to zero. The action dimension contains â which represents the intermediate state action sequence, and initialized by (0, 7) and further constrained by action mask mechanism to reduce the action space
3

---

Published as a conference paper at ICLR 2025
Additon
eatran
Supervision
Interploltdd Lakent Fastures Supervisio
Latent Space
Tempoca
Mdlliiiaiio
extract
Concalenation
and Tome
Tast Classifor
Propect
Lett
dmmenstons
Action
Re
Re-t
Upowggrat
Projected Denoising Step
Otsaeivation
Condition Projection
3). With a transformer classifier to guide from observations, 2 is processed a U-Net with latent space module for temporal supervision. Starting with denoising through masked projection, we derive the final actions needed to compute the task-adaptive masked proximity loss for iterative optimization.
Figure 2:
Overview of our Masked Temporal Interpolation Diffusion (prediction horizon T
to be predicted. Hence, the iteration matrix En E E (TT((++++0) is expressed as,
C
f
42
Qr-1
0
where n ranges from to N, is the number of task classes, A is the number of actions and O the observation visual feature dimension. Next, the intermediate latent features generated from the latent space temporal interpolation module will injected into the diffusion model iteratively optimize the matrix at. During the iteration process, we adopt DDIM to accelrrate the sampling process with fewer steps while maintaining strong performance. Lastly, we introduce the task-adaptive masked proximity loss Cair to enhance the reliability of the reasoning results.
3.2
MTID: MASKED TEMPORAL INTERPOLATION DIFFUSION
3.2.1
PRELIMINARIES
Denoising Diffusion Implicit Model (DDIM) (Song el al. 202T) improves sampling efficiency by making the reverse process deterministic, which reduces stochastic noise and establishes a direct mapping between the initial noise matrix y and the final output matrix ig across N non-Markovian steps. This approach reduces the number of steps needed while preserving sample quality.
Based on these advantages, we adopt the DDIM sampling strategy with the U-Net model (Ron- neberger al.) al,, 2015) for its ability to accelerate sampling with fewer steps while maintaining strong performance. This is especially useful in scenarios where the quality of results remains comparable to that of Denoising Diffusion Probabilistic Model (DDPM) (Ho et al. 2020; Nichol Dhariwal, 2021). despite its deterministic approach.
The forward process is parameterized as:
åN VaN 3t /1 āN
where ØN = II a2 ~~ No,, and ag = - 3., which represents the noise schedule controlling the amount of Gaussian noise added each step The forward process starts with the original data i and progressively adds noise, resulting in the final noisy matrix f.
In DDIM, the reverse process is defined as:
In
vI VI-anfe (Rm)) 21-1/111 ān-1 fo (in).
- vān-1
(4)
Van
where fo (fx) is the neural network's 's prediction of the noise component added to 2 This reverse process reconstructs the original data 2n from & by iteratively removing the noise introduced dur- ing the forward diffusion process. Unlike DDPM, DDIM's deterministic reverse process improves
4

---

Published as a conference paper at ICLR 2025
Observation Spoce Temporal Interpplation Mocue
Transfomer (TF)
Irekk
Uoeer
Man
1D coeredili
Coss Morcinn
RGL
Liner
Val
Lecer
Den
MaA
1
Dragghooe
UW
NN
resiiua contection
(a) Latent space temporal interpolation module.
(b) Residual temporal block & cross-attention module.
Figure Module in Figure generates temporally coherent latent features from observations, while the block in Figure 3b refines these features with temporal dependencies, guiding coherent action sequence generation.
sampling efficiency by directly mapping the noisy input ēy to the final output Sp. This makes DDIM an efficient method for generating or enhancing samples with fewer steps.
3.2.2 LATENT SPACE TEMPORAL, INTERPOLATION
As shown in Figure Ba the Latent Space Temporal Interpolation Module consists of three components: an observation encoder, which transforms observation visual features into latent fea- tures; latent space interpolator, which generates multiple features fill the intermediate supervi- sion; and transformer encoder blocks, which refine the generated features and enhance the module's ability model the temporal correlations among action sequences.
Specifically, this module first employs an observation encoder E, consisting of convolutional layers, transform the visual observations V, and Vg into their respective latent features Lg and Lg:
L.y Ln E(V., Vg)).
(5)
In the latent space, we perform linear interpolation between L. and Lg to generate a sequence of method dynamically
interpolated features (/1, I2...., [M] Unlike fixed linear interpolation, adjusts the interpolation through the learnable interpolation matrix & E M& to facilitate smooth and task-specific transitions. This matrix, restricted between 0 and I, is responsible for weighting the latent features L, and Lg to generate the intermediate latent features as follows:
& Sigmoid(W ++++) k): 1, = (1 - Φ3) T. + Φ5 T.g.
(6)
where W MRO and E IRMx are the parameters of the linear layer, and O represents the observation dimension. The matrix E M is learnable matrix initialized with a constant value, controlling a variable ratio between L, and Lg. This method requires no parameter tuning and offers better adaptability different tasks. The number of latent features M depends the number residual temporal blocks in the model.
The sequence of interpolated features {11, .2..... ... IM} is then passed through series of transformer encoder blocks TF to obtain the enhanced latent features:
F1, F2, FM FM = TF(,, , IM).
The self-attention mechanism in the transformers captures dependencies between latent features at different time steps by computing attention scores across latent features. Stacking multiple transformer blocks allows the model to iteratively refine these features, ensuring that temporal and contextual relationships are effectively learned.
To integrate the interpolated latent features {Fi, FM. } into the model during the denoising process, we incorporate cross-attention layers (Khachatryan et al. (202)) into the residual temporal blocks of the U-Net, shown in Figure 3b) This allows the model to dynamically focus relevant latent features through a learnable matrix, enhancing its ability to capture complex relationships with more latent temporal information and improving the quality of action predictions.
In this setup. the latent feature F,. processed through a linear layer, serves as the key and value, while Sn, processed through convolutional block and combined with (sampled from random
5

---

Published as a conference paper at ICLR 2025
integers), acts the query. The cross-attention is computed as:
pn softmar ( LFF
LF
(8)
VO
where CB denotes a 1D convolution block, TM represents a time MLP, and LFj refers to the result obtained from passing F; through a linear layer. This design enables the model to effectively capture temporal relationships, thereby improving the overall quality of action prediction.
3.2.3 MASKED PROJECTION FOR INITIALIZATION
While our model handles procedure planning tasks effectively, the denoising sampling process in diffusion models does not always guarantec that the generated actions fall within the desired range. To mitigate this issue, we introduce a masked projection that constrains the action space during the training denoising process. This approach is inspired by the masked latent modeling scheme proposed by Gao et al. (2023).
Since each task has specific action scope. we activate only the actions associated with the current task class label e and deactivate the others. When constructing the input matrix nx for the denoising process, we add the initial Gaussian noise solely the active action positions, while setting the non-active action positions to zero. This process can be expressed as follows:
āe, (0, C, if E Task(c)
if & Task(c)
where d denotes the action spanning the action dimension A, and c (00 1). The function T (csk(c) represents the set of actions corresponding to task c. By restricting the initial noise in the input matrix N to the relevant action scope, the model ensures that the procedure planning is confined to the active actions.
3.3 TASK-ADAPTIVE MASKED PROXIMITY LOss
Our training process consists of two main stages: (a) training a task classifier to extract task-related information based the given start and goal visual observations. (b) utilizing masked temporal interpolation diffusion model fo to fit the distribution of the target action sequence.
In the first stage, we minimize the cross-entropy loss between the predicted and true task classes to optimize the transformer-base task classifier.
In the second stage, we employ diffusion-basse training scheme and introduce task-adaptive masked proximity loss to model the target action sequence, defined as follows:
Caif
we - ma,d - (ar.d az,a)²
(10)
where dnd refers to the predicted action ID extracted from the final output d.. and äz,d denotes the ground truth action. This loss function computes the weighted mean squared error (MSE) between the predicted and ground truth actions at each planning time step. The term is a time-dependent gradient weight that controls the contribution of each time step, and my,d is a mask matrix that highlights specific action dimensions or planning time steps according to the task requirements.
The weight wa of gradient weighted loss is defined as:
W/- = wp + (1 - (1- min (1, - -1+1))-1
(11)
[ -1
where Wo is the initial weight. Since the task only observes the start and goal features, V, and Vg higher weights are assigned to predictions near these endpoints, thereby enhancing performance at ai and ar. Lower weights are assigned to the intermediate steps, allowing the model to balance the endpoints and middle states without placing too much emphasis on the endpoints. Unlike Wang et al. (2023b), who weights both start and end actions called both sides weighted loss, approach uses intermediate latent features for continuous supervision. This provides more comprehensive guidance, allowing us apply gradient weights for better alignment of the entire action sequence.

---

Published as a conference paper at ICLR 2025
Table Comparison with other methods on CrossTask dataset. Features extracted by the NowToo000-trriiee encoder and settings of PDPP are marked with i, while other features are provided directly by CrossTask. Note that we compute mloU by calculating average of every IoU of single action sequence rather than mini-batch for all datasets.
T=3
T=4
T=5
T
Models
SRT
mAcef
mloUT
SRT
mAcel
mloUt
SR1
SR1
Random
<0.01
0.94
1.66
<0.01
0.83
1.66
<0.01
<0.01
Retrieval- -aadev
8,05
23.30
32,06
3.95
22.22
36.97
2.40
1.10
WLTDO (Ehsami et al.
2018
1.87
21.64
31.70
0.77
17.92
26.43
UAAA
Abb Farha & Gall 2019
2019
2.15
20.21
30.87
0.98
19.86
27.09
UPN Srinivas et al 2018
2.89
24.39
31.56
1.19
21.59
27.85
DDN
Chang ct al. 20220)
12.18
31.29
47.48
5.97
27.10
48.46
3.10
1.20
PlaTe Sun ef al. 2022
16.00
36.17
65.91
14.00
35.29
55.36
Exi-GAIL
al..
2021
21.27
49.46
61.70
16.41
43.05
60.93
PIV
ZZho et al 2022
23.34
49,96
73.89
13.40
44.16
70.01
7.21
4.40
EGPP
Wang èl
2023a
26.40
53.02
74.05
16.49
48.00
70.16
8.96
5.76
PDPPf
Wang cl al. 20223b
37.2
64.67
66.57
21.48
57.82
65.13
13.45
8.41
KEPP Nagasinghe al.
2024
38.12
64.74
67.15
24.15
59.05
66.64
14,20
9.27
SCHEMAT (Nnu et al. 2024]
38.93
63.80
79.82
24.50
76.48
14.75
10.53
MTID (Ours)f
40.45
67.19
69.17
24.76
60.69
67.67
15.26
10.30
Additionally, a mask matrix m is applied to selectively emphasize certain planning time steps and action dimensions. This matrix is defined as:
P; if ăe,a is active
Me,d 1, otherwise
(12)
where p is scaling coefficient applied when the action is active, thereby increasing the penalty for unrelated actions. By this mechanism, actions that are unrelated to the current task are discouraged from appearing in the output, ultimately enhancing planning accuracy.
4
EXPERIMENTS
4.1 EVALUATION PROTOCOL.
Datasets and Settings. We evaluate our MTID method on three instructional video datasets: CrossTask (Zhukov et al. 2019), COIN (Tang al. 2019). and NIV (Alayrac et al. 2016).
CrossTask consists of 2,750 videos across 18 tasks, covering 105 actions, with an average of 7.6 actions per video. COIN contains 11,827 videos spanning 180 tasks, with average of 3.6 actions per video. NIV includes 150 videos from 5 tasks, with an average of 9.5 actions per video. We randomly split each dataset into training (70% of videos per task) and testing (30%), following pre- vious works (Sun et al.) 2022, Wang et al.] 20236) Niu et al. 2024). Furthermore, we conduct all experiments using the setting of PDPP, except for Table 3 where we adopt the setting of KEPP to ensure fair comparison. For details, please refer to Appendix for the comparison with PDPP.
Metries. Following previous works (Sun et 2022 Zhao al. 2022 Wang al. 2023b Niu
et al. 2024 Nagasinghe et al. (024). we evaluate the models using three key metries: (1) Success
Rate (SR) is the strictest metrie, where procedure considered successful only if every predicted
action step exactly matches the ground truth. (2) mean Accuracy (mAcc) computes the average
accuracy of predicted actions each time step, where an action deemed correct if it matches the ground truth action at the corresponding time step. (3) mean Intersection over Union (mloU) quantifies the overlap between the predicted procedure and the ground truth by calculating mloU (or (n(]]) as (a, jU[á;]] where } represents the set of ground truth actions, and {ā,} denotes the set of predicted actions.
4.2 RESULTS
Results for Task Classifier. The first stage of our approach involves predicting the task class based on the given start and goal observations. We implement this using transformer models, replacing
7

---

Published as a conference paper at ICLR 2025
Table 2: Classification results on all datasets.
CrossTask How
COIN
NIV
Models 3=
7'= 4
7=
7=
7=
7=
T=3
7=4
PDPP
92.43
92.98
93.39
93.20
79.42
79.42
100.00 100.00
100.00
MTID (Ours)
93.67
94.03
94.02
94.26
81.47
81.47
100.00
Table 3: Comparisons on COIN and NIV datasets. Note: only this table the KEPP's settings.
COIN
NIV
Models
=
T=4
T=
T= 4
SR1
mAcef
mloUt
SR1
mAcef
mloUf
SR1
mAcef
mloUf
SR1
mAcef
mloU?
Random
<0.01
<0.01
2.47
<0.01
<0.01
2.32
2.21
4.07
6.09
1.12
2.73
5.84
DDN
13.90
20.19
64.78
11.13
17.71
68.06
18.41
32.54
56.56
15.97
27.09
53.84
PVIV
15.40
21.67
76.31
11.32
18.85
70.53
24.68
49.01
74.29
20.14
38.36
67.29
EGPP
19.57
31.42
84.95
13.59
26.72
84.72
26.05
51.24
75.81
21.37
41.96
74.90
PDPP
19.42
43.44
13.67
42.58
22.22
39.50
86.66
21.30
39.24
84.96
KEPP
20.25
39.87
51.72
15.63
39.53
53.27
24.44
43.46
86.67
22.71
41.59
91.49
SCHEMA
32.09
49.84
83.83
22.02
45.33
83,47
27.93
41.64
76.77
23.26
39.93
76.75
MTID
30.44
51.70
59.74
22.74
49.90
61.25
28.52
44.44
56.46
24.89
44.54
57.46
the two-layer Res-MLP architecture employed Wang al.) (2023b). and train using a simple
cross-entropy (CE) loss. The classification results are presented in Table 2
Our method consistently
outperforms previous approaches all evaluated aspects.
Comparisons on CrossTask. We evaluate performance CrossTask using four prediction hori- zons. The results in Table [ demonstrate that our method outperforms all other approaches across all metrics, except for the SR at T = 6, where our model ranks second. These improvements are consistent across longer prediction horizons (T = 4, 5, 6) and other step-level metrics, including mAce and mloU.
Comparisons on NIV and COIN. Table 3 presents our evaluation results on the NIV and COIN datasets, demonstrating that approach consistently outperforms matches the best-performing methods across both datasets. These results highlight that our model performs robustly across datasets of varying sizes.
4.3 ABLATION STUDIES
Task-Adaptive Masked Proximity Loss. Ta- ble 4 demonstrates the effectiveness of our proposed loss strategy with T 5 on the CrossTask dataset, using Mean Squared Er- ror (MSE) as base loss. The results show that both the task-adaptive mask and gradi- ent weighted loss improve performance. While MSE alone results in lower scores, adding masks and fixed weights provides moderate im- provement. Our approach, which incorporates gradient weighted loss and intermediate super- vision, significantly boosts performance by leveraging richer task-relevant features.
Table 4: Ablation studies loss function.
Note: W: Both Sides Weighted Loss, GW: Gra- dient Weighted Loss, M: Mask.
ID
MSE
W
GW
M
SR1
11.89
13.90
15.10
13.26
5
13.93
6
15.26
Masked Projection. Table demonstrates that our masked projection (MP) on f as input to the U-Net enhances performance by filtering out irrelevant actions, allowing the model fo- cus on task-relevant actions. We also experi- ment with applying the mask during denoising iterations, but this approach proved ineffective. During the denoising process, the input matrix contains both positive and negative logits, and in some cases. negative values can improve the final score. Masking at this stage disrupted the natural behavior of the logits and the diffusion denoising process. Furthermore, applying the mask
Table 5: Ablation study on projection and phase when
on CrossTask dataset. Note: "CP" denotes the condition projection.
Models
SR
mAcet
mloUt
w/o MP
39.17
66.49
68.38
MP on iteration MP on initialization
3.38
10.17
9.66
40.45
67.19
69.17
8

---

Published as a conference paper at ICLR 2025
before the condition projection led to sub-optimal results due inaccrrtee task labels. Therefore, we apply the mask only after the condition projection for optimal performance.
and milo
random
copvll,)
(a) Ablation studies different simple initializa- tion coefficient values т.
(b) Ablation studies using different interpolated features for transformer blocks.
(c) Ablation studies selection with different in- terpolated features.
(d) Ablation studies different fusion methods.
Figure 4: Ablation Studies for Interpolation Strategy. Figure illustrates the features generated by the interpolator, where "copyy(Lg)" Lg)' means I, = Lg. and "copy( indicates I L, for S M In Figure 4c I; to T- indicates that we use interpolated features from the
and 1 = l.g for j >> T. i-th to the last i-th for the second interpolation. In Figure|4d| "all F denotes that each cross-attention module receives FI.M as input, "step F" means we input one Fy at the start and gradually increase the number of features inputted towards both sides, and "-all" indicates that we omit inputting Fj during the upsampling, middle-sampling. and downsampling processes in U-Net.
Latent Space Temporal Interpolation Module. To evaluate the impact of var- ious components within the module, we conduct ablation studies on the CrossTask dataset with T = 3. The results in Ta- blediind that the observation encoder effectively transforms visual observations into latent features, enhancing causal in- ference and capturing temporal logic. The interpolator enriches these features by generating multiple interpolated versions, which aids in rea- soning. The transformer encoder further retines these features, ensuring both mathematical interpo- lation and logical consistency, thereby improving the model's inference capabilities.
Table 6: Component ablation. Note: Int: Interpolation, Enc: Encoder, Trans: Transformer.
ID
Int
Ene
Trans
SR1
mAccf
mloUf
37.86
65.42
67.32
2
4
39.23
66.62
68.44
39.49
66.81
68.68
40.45
69.17
Interpolation Strategy. Figure illustrates our adapted interpolation strategies for contrast. Ini- tially, we test different initialization 7 values in Figure 4a The highest score occurs when é is initialized 1, which proves be the most stable and achieves the best overall performance, indi- cating that & converges close to 1. Further experiments in Figure compare the use of L,, Lg. or combination of both through copying and direct return. The results show that directly returning T.g performs well, suggesting that Vg may play critical role in action sequence inference. Although L, and L, are unadjusted features, the transformer encoder blocks can refine them to capture richer temporal logic and filter out irrelevant details, resulting in relatively reasonable scores. Additionally, we perform a second interpolation using the obtained Fj (Figure 4ch. This experiment shows that the features shift towards the center (from F, to F_), performance declines due to the loss of original information, aligning with intuition. In Figure/4d we examine different strategies for cross- attention. The "all F" approach results in sub-optimal performance, potentially due to the inclusion

---

Published as a conference paper at ICLR 2025
of numerous features being limited by the capacity of the layers, which diminishes the amount of information each feature can carry and introduces disorder. Similarly, the "step F" strategy may suffer from the same issue. When cross-attention is removed from various stages of the U-Net, observe that the more layers are removed, the worse the results become. However, when all layers removed, the results improve slightly, suggesting that introducing latent features into the U-Net without sufficient information disturbs the original distribution, leading to sub-optimal outcomes.
4.4 UNCERTAINTY MODELING
We present the uncertainty modeling results for CrossTask and NIV in Table Two baselines from Wang al. (202b) used for comparison: the Noise baseline, which samples from random noise and uses the given observations and task class condition to obtain results in single step without the diffusion process, and the Deterministic baseline, where d 0 and the model predicts fixed outcome. We evaluate performance using KI. divergence, NL.I,, ModeRec, and ModePrec, as outlined in Zhao et al. (2022).
Our approach outperforms the baselines CrossTask, particularly in modeling uncertainty and generating diverse plans. Compared to Wang et al. (2023b), our model excels the deterministic setting with 7 = 3, demonstrating its ability to capture latent temporal relationships even with fewer time steps. For the NIV dataset, we observe that despite small size, our diffusion-basee process still delivers improvements. Additional visualizations are provided in the appendix.
Table 7: The results of uncertain modeling on the CrossTask and NIV datasets.
CrossTask
NIV
Metric
Method
7 3
T=
T=
T
7=3
3
Deterministic
3.12
3.88
4.39
5.40
5.29
KL.-Div 4
Noise
2.75
3.16
4.37
4.74
5.36
6.03
Ours
2.66
2,81
2.12
1.97
4.65
5.47
Deterministic
3.70
4.45
4.98
5.34
5.48
5.42
NLL
Noise
3.33
4.04
4.95
5.32
5.44
6.12
Ours
3.24
3.69
3.22
3.27
4.74
5.56
Deterministic
52.76
41.13
31.46
18.65
27.77
26.48
ModcPree
Noisc
54.30
46.15
22.52
19.09
23.19
32.39
Ours
56.19
47.05
32.75
22.98
30.75
35.88
Deterministic
31.71
20.55
18.70
4.63
26.48
21.57
ModeRec 1
Noise
43.92
22.35
21.53
17.53
32.39
23.75
Ours
47.34
37.97
39.64
35.03
35.88
29.90
CONCLUSION
In this paper, we introduce the Masked Temporal Interpolation Diffusion (MTID) model, specifi- cally designed for procedure planning in instructional videos. Our model employs a latent space temporal interpolation module within a U-Net architecture to capture intermediate states and tem- poral relationships between actions. By incorporating a task-adaptive masked strategy during both inference and loss calculation, MTID improves the accuracy and consistency of generated action sequences. Extensive experiments across the CrossTask COIN, and NIV datasets demonstrate that our model consistently outperforms existing methods on key metrics. For future work, we aim to further optimize the memory efficiency of the model to handle larger datasets more effectively. Ad- ditionally, refining the mask mechanism to enhance control over intermediate state generation and exploring more diverse interpolation strategies remain promising directions. We also plan to extend the application of the temporal interpolation module to broader procedural learning tasks, including more complex conditional planning scenarios.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide general details on the datasets and experi- mental settings in Section 4.1 Comprehensive information on the model architecture, datasets, and training strategies can be found in Appendix A
ACKNOWLEDGEMENT
This work is partially supported by the National Natural Science Foundation of China under Grants 62441232, U2IB200217, 62306092, 62476068, and 62236008, and projects ZR2024QF066 and ZR2023QF030 supported by Shandong Provincial Natural Science Foundation.
REFERENCES
Yazan Abu Farha and Juergen Gall. Uncertainty-aware anticipation of activities. In ICCV Work- shops, 2019.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In CVPR, 2016.
Rashmi Bhaskara, Hrishikesh Viswanath, and Aniket Bera. Trajectory prediction for robot naviga- tion using flow-guided markov neural operator. In ICRA, 2024.
Jing Bi, Jieho Luo, and Chenliang Xu. Procedure planning in instructional videos via contextual modeling and model-based policy learning. In ICCV, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Chien-Ya Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. ECCV, 2020.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 45(9); 10850-10869, 2023.
Kiana Ehsani, Hessam Bagherinezhad, Joseph Redmon, Roozbeh Mottaghi, and Ali Farhadi. Who let the dogs out? modeling dog behavior from visual data. In CVPR, 2018.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. In ICCV, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint wrXiv:2403.14773 2024.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif Saurous, Bryan Seybold, al. Cnn architectures for large-scale audio classification. In ICASSP, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In CVPR, 2024.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero, Text-to-image diffusion models zero-shot video generators. In ICCV, 2023.

---

Published as a conference paper at ICLR 2025
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXi::1412.6980, 2014.
Zhiheng Li, Wenjia Geng, Muheng Li, Lei Chen, Yansong Tang, Jiwen Lu, and Jie Zhou. Skip-plan: Procedure planning in instructional videos via condensed action space learning. /CCV, 2023.
Haicheng Liao, Zhenning Li, Huanming Shen, Wenxuan Zeng, Dongping Liao, Guofa Li, and Chengzhong Xu. Bat: Behavior-aware human-like trajectory prediction for autonomous driving. In AAAI, 2024.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Hlotto100m Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint
arXiv: 0868,, 2019.
Kumaranage Ravindu Yasas Nagasinghe, Honglu Zhou, Malitha Gunawardhana, Martin Renqiang Min, Daniel Harari, and Muhammad Haris Khan. Why not your textbook? knowledge- enhanced procedure planning of instructional videos. CVPR, 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Yulei Niu, Wenliang Guo, Long Chen, Xudong Lin, and Shih-Fu Chang. SCHEMA: State CHanges MAtter for procedure planning instructional videos. In ICLR, 2024.
Zhaobo Qi, Shuhui Wang. Chi Su, Li Su, Qingming Huang, and Qi Tian. Self-regulated learning for egocentric video activity anticipation. IEEE transactions on pattern analysis and machine intelligence, 45(6):6715-6733) 2021.
Zhaobo Qi, Shuhui Wang, Weigang Zhang, and Qingming Huang. Uncertainty- -booste robust video activity anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- image segmentation. In MICCAI, 2015.
Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrish- nan, Christine Chan, Gabriel Dulac-Amold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In ICRA, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Aravind Srinivas, Allan Jabri, Picter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks: Learning generalizable representations for visuomotor control. ICML, 2018.
Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, and Animesh Garg. Plate: Visually- grounded planning with transformers in procedural tasks. IEFE Robotics and Automation Letters, 7(2):492449930. 2022.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang. Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019.
An-Lan Wang, Kun- Yu Lin, Jia-Run Du, Jingke Meng, and Wei-Shi Zheng. Event-guided procedure planning from instructional videos with text supervision. In ICCV, 2023a.
Hanlin Wang, Yilu Wu, Sheng Guo, and Limin Wang. Pdpp: Projected diffusion for procedure planning in instructional videos. In CVPR, 2023b.
12

---

Published as a conference paper at ICLR 2025
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In CVPR, 2024.
Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, et al. Art-v: Auto-regressive text-to-video generation with diffusion models. In CVPR, 2024.
Yuxin Wu and Kaiming He. Group normalization. ECCV, 2018.
He Zhao, Isma Hadji, Nikita Dvomik, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. P3iv: Probabilistic procedure planning from instructional videos with weak supervision. In CVPR, 2022.
Shangchen Zhou, Peiqing Yang. Jianyi Wang, Yihang Luo, and Chen Change Loy. Upscale-a-video: Temporal-consister diffusion model for real-world video super-resolution. In CVPR, 2024a.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng. and Qibin Hou. Storydiffu- sion: Consistent self-attention for long-range image and video generation. arXiv preprint arXiv:2405.01444, 2024b.
Dimitri Zhukov, Jean-Baana Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivie. Cross-task weakly supervised learning from instructional videos. In CVPR, 2019.
APPENDIX
A IMPLEMENTATION DETAILS
A.1 1 MODEL ARCHITECTURE DETAILS
In the first learning stage, we aim to predict the task class label given the observations (V,, Vg)- We employ a simple 4-layer transformer model for this task and use cross-entropy loss to train the model by comparing its output with the ground truth task class labels.
The classifier is neural network based on the transformer architecture. It first embeds the input data through a linear layer, after which the embedded data processed by multiple stacked transformer encoder layers. The output of the encoder layers is averaged and then passed through series of fully connected layers with ReLU activation functions. Finally, the processed data is passed through linear layer to generate the final output. Dropout layers are applied throughout the model to prevent overfitting.
Next, our main model is based 3-layer U-Net (Ronneberger et al. 2013). similar Wang et al. (2023b), but adapted for temporal action prediction. Each layer consists of two residual tem- poral blocks (He al., 2016). followed by either downsampling
upsampling. Each residual temporal block includes two convolutional layers, group normalization (Wu & He 2018). Mish ac- tivation (Misra 2019). and cross-attention module for feature fusion. Temporal embeddings are generated via a tully connected layer and added to the output of the first convolution. To handle the short planning horizon (T = (3,4,5, 4, 5, 6]), we employ 1D convolutions with kernel size of 2, stride of 1, and no padding for downsampling/ wwwnsamping/upsaapligg used by Wang et al. (2023b), instead of the max-pooling approach, ensuring the horizon length remains unchanged. The middle block consists of only two residual temporal blocks.
The input matrix é. a concatenation of the task class, action sequences, and observation features, with dimension of f fusion_dim C' + A + O where C, A, and O represent the number of task classes, action labels, and visual features, respectively. During the downsampling phase, the input 512 1024], with the reverse process occurring
is embedded through [fusion_dinn dim 256 during the upsampling phase.
The latent space temporal interpolation module consists of three main components: an observation encoder, a latent space interpolator, and transformer encoder blocks. The observation encoder re- duces the input dimensionality using DD convolutional layers with ReLU activations. The latent
13

---

Published as a conference paper at ICLR 2025
space interpolator generates intermediate features between two encoded representations via linear interpolation, guided by a learnable linear layer initialized with matrix т. The generated matrix is passed through a Sigmoid function to compute 1j- Finally, standard transformer encoder blocks apply attention mechanisms to enhance the temporal relationships in Pj. ultimately producing trans- formed latent features with shape of [M, OJ, where M refers the number of residual temporal blocks in the U-Net.
For the diffusion process, we employ a cosine noise schedule to generate J. which controls the amount of noise added at each step. These values correspond to the variance of the Gaussian noise introduced at each stage of diffusion.
A.2 DATASET DETAILS
Each video in the dataset is annotated with action labels and their corresponding temporal bound- aries, denoted as {8s, (;) for the i-th action, where 8; and e; represent the start and end times, respectively. The total number of actions in the dataset is denoted N Wum. We extract step quences (1: from the videos, with the horizon T' ranging from 3 to 6. Following the method
in previous work (Wang et al. 2023b), action sequences {lat..... a++r-1 -)m are generated by sliding a window of size over the Num actions. For each sequence, the video clip feature at the start time of action a₂ is used as the starting observation Va, and the clip feature the end time of action 7++7-1 is used as the goal state Vg, Both clips are 3 seconds in duration. The start and end times of each sequence are rounded to [s,]] and [ec+r-1]: 1. respectively. with the clip features between these times used as V, and Vg.
For the CrossTask dataset, we consider two types of pre-extracted features: (1) the 3200-dimensional features provided by the dataset, which combine I3D, ResNet-152, and audio VGG features (Car- reira Zisserman 2017 He et al., 2016 Hershey al. 2017), and (2) features extracted using an encoder trained on the How 1000M dataset (Miech et al. 2019). as used in (Wang et al. 20236). We utilize the latter due to its smaller size. For COIN and NIV datasets, we also use HowTo100M features (Wang et al. 2023b) to maintain consistency and ensure fair comparison.
A.3 DETAILS OF METRICS
2022) computed the mIoU metric over mini-batches, averaging the results across the batch size. However, this method introduces variability depending on the batch size. For instance, if the batch size equals the entire dataset, all predicted actions may be considered correct. In contrast, using batch size of one penalizes any mismatch between predicted and ground-truth sequences. To address this issue, we follow [Wang] et al.) (2023b) by standardizing mloU calculation, computing it for each individual sequence and then averaging the results, effectively treating the batch size one. However, this approach may result in our mloU scores being lower than those reported by others.
Previous works (Chang et al. 2020 Bi et al., 2021 Sun et al.
A.4 TRAINING DETAILS
Following Wang et al. 2023b), we employ a linear warm-up strategy to train our model, with specific protocols adjusted for different datasets. For the CrossTask dataset, we set the diffusion steps to 250 and train for 20,000 steps. The learning rate is linearly increased to 5 x 10 over the first 3,333 steps, then halved steps 8.333, 13.333, and 18,333. For the NIV dataset, with 50 diffusion steps, training lasts for 5,000 steps. The learning rate ramps up to 3 x 10-4 over the first 1,000 steps and is reduced by 50% at steps 2,666 and 4,332. In the larger COIN dataset,
use 300 diffusion steps and train for 30,000 steps. The learning rate increases to x 10 10-5 in the first 5,000 steps and is halved steps 12,500, 20,000, and 27,500, stabilizing at 2.5 10 10-6 for the remaining steps. Training is performed using ADAM (Kingma 2014) on 8 NVIDIA RTX 3090 GPUs.
A.5 DETAILS OF UNCERTAINTY MODELING
In the main paper, we investigate the probabilistic modeling capability of our model on the CrossTask and COIN datasets, demonstrating that our diffusion-based model can generate both di-
14

---

Published as a conference paper at ICLR 2025
verse and accurate plans, Here, provide additional details, results, and visualizations to further illustrate how our model handles uncertainty in procedure planning.
Additional Results COIN. Results the COIN dataset presented in Table g dataset, our model underperforms relative to the Deterministic baseline. We attribute this to the shorter action sequences, where reduced uncertainty is more advantageous but less critical for long- horizon procedural planning.
On the COIN
Visualizations for Uncertainty Modeling. In Figures 7a to 7 present visualizations of various plans with the same start and goal observations, generated by our masked temporal interpolation diffusion model CrossTask for different prediction horizons. We have observed that some results contain repeated actions, which is due to the probabilistic nature of our model's prediction method, making repeated action predictions unavoidable. The top five predicted logits for the actions are passed through a softmax function, and the action with the highest probability is selected to form the prediction figures.
Details of Evaluating Uncertainty Modeling. For the Deterministic baseline, we sample once to obtain the plan, as the result is fixed when the observations and task class conditions are given. For the Noise model, sample
Table 8: The results of uncertain modeling the COIN dataset.
Metric
Method
T
3
T =
4
baseline and our diffusion-baat 1,500 action sequences to calculate the uncertainty metrics. To efficiently perform this process, ap- ply the DDIM (Song et al.] 2021) sampling method our model, enabling each sampling process be completed in 10 steps. This accelerates sampling by 20 times for CrossTask and COIN, and by times for NIV. It is important to note that multiple sam- pling is only required when evaluating probabilistic modeling our model can generate a good plan with just single sample.
Deterministic
4.47
4,40
KL-Div
Noise
5.12
4.88
Ours
4.74
4.47
Deterministic
5,42
5,81
NLL J
Noise
6.07
6.28
Ours
5.69
5.87
Deterministic
34.04
32.47
ModePrec 1
Noise
23.16
22.18
Ours
28.83
26.91
Deterministic
27.41
20.88
ModeRec
Noise
21.06
15.24
Ours
23.27
18.14
B BASELINE METHODS
In this section, we describe the baseline methods used in our study.
Random Selection. This method randomly selects actions from the available action space within the dataset to generate plans.
Setrieval-Based Approach. Given the observations (V., VgJ, this method retrieves the nearest neighbor by minimizing the visual feature distance within the training dataset. The action sequence associated with the retrieved neighbor is then used the plan.
WLT DO (Ehsani et al., 2018). This method employs a recurrent neural network (RNN) to predict action steps based on the provided observation pairs.
UAAA (Abu Farha & Gall, 2019). UAAA two-stage approach that uses an RNN-
HMM model to predict action steps in auto-regressive manner.
UPN (Srinivas et al.) 2018 UPN is path planning algorithm for physical environments that learns a plannable representation to generate predictions. To produce discrete action steps, softmax layer is appended to the model's output, described in (Chang et al. 2020
DDN (Chang et al. 2020). DDN is an auto-regressive framework with two branches de- signed learn abstract representation of action steps and predict transitions in the fea- ture space.
PlaTe (Sun et al. 2022). PlaTe extends DDN by incorporating transformer modules into its two branches for prediction tasks. PlaTe uses different evaluation protocol compared to other models.
Ext-GAIL (Bi et al. 2021). Ext-GALL addresses procedure planning using reinforcement learning. Unlike our approach, Ext-GAIL divides the planning problem into two stages:
15

---

Published as a conference paper at ICLR 2025
the first provides long-horizon information, which is then used by the second stage. In contrast, our approach derives sampling conditions directly.
P'IV (Zhao et al. 2022). P'IV is transformer-base single- brancc model that incorpo- rates learnable memory bank and an additional generative adversarial framework. Similar model, IV predicts all action steps simultaneously during inference.
EGPP (Wang et al. 2023a). EGPP proposes an event-guided paradigm for procedure planning, where events are inferred from observed visual states guide the prediction of intermediate actions. The E3P model uses event-aware prompting and Action Relation Mining improve action prediction accuracy, significantly outperforming existing meth- ods in experiments.
PDPP (Wang al., 2023b). PDPP is two-branch framework that models temporal de- pendencies and action transitions using diffusion process. Like our model, PDPP predicts all actions simultaneously, refining predictions over multiple stages to enhance logical con- sistency during inference.
KEPP (Nagasinghe al., 2024). KEPP is knowledge-enhanced procedure planning system that leverages a probabilistic procedural knowledge graph (P2KG) learned from training plans. This graph acts as "textbook" guide step sequencing in instructional videos. KEPP predicts all action steps simultaneously with minimal supervision, achieving leading performance.
SCHEMA (Niu et al., 2024). SCHEMA focuses procedure planning by learning state transitions. employs a transformer- based architecture with cross-modal contrastive learn- ing to align visual inputs with text-based state descriptions. By tracking intermediate states, SCHEMA predicts future actions using a large language model to capture temporal depen- dencies and logical transitions, improving action planning in instructional videos.
C ADDITIONAL ABLATION STUDIES
Model Size. Due the large size of the COIN dataset, adjust the model size by modifying the U-Net architecture. As shown in Table 9 increasing the model size results in higher scores for the COIN dataset. We believe that optimizing the model to be more memory-efficie
could further increasing the size to 512 does not improve the scores on CrossTask. We believe this suggests overfitting, indicating that a model size of 256 is sufficient for this task.
improve performance, which we plan to explore in future work. In Table 9
Table 9: Ablation study on the role of model size on COIN and CrossTask datasets.
T=
T=
Dataset
Size
SRT
mAcct
mloUt
SRT
mAce?
mloU?
128
23.01
45.44
51.93
19.69
45.32
55.06
COIN
256
28.84
50.44
57.86
21.64
48.06
59.52
512
30.90
52.17
59.58
23.10
49.71
60.78
128
23.01
45.44
51.93
19.69
45.32
55.06
CrossTask
256
40.45
67.19
69.17
24.76
60.69
67.67
512
37.94
66.16
67.43
21.97
58.30
66.
Components of Observation Encoder. Table TO presents the impact of different encoder compo- nents. Based on this ablation study, the optimal model consists of two 1D convolution layers with ReLU activation, which achieves the best balance between depth and activation, resulting in the highest scores across all metrics. Adding more layers does not consistently improve performance, and activation functions like ReLU play a key role in enhancing model effectivvness. We believe that ReLU introduces non- non-linearity, enabling the network to capture temporal latent features more effectively. Moreover, by setting negative values to zero, ReLU promotes sparse activation, which may aid in the extraction and construction of latent features.
16

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study on the observation encoder components (T'
Cross Task).
Models
SR
mAccf
mloUt
1 conv. layer w/ ReLU
39.71
66.91
69.05
2 Id conv. layers w/ ReLU (Ours)
40,45
67.19
69.17
ld conv. layers w/ ReLU
36.04
64.41
66.55
1d conv. layer w/o ReLU
39.32
66.70
68.91
1d conv. layers w/o ReLU
39.38
66.73
68.65
3 ld conv. layers w/o ReLU
37.77
65.06
67.61
Table 11: Ablation study on the role of classifier type on CrossTask dataset.
==
==
T=5
T
Models
SRT
mAceT
mloUf
SRT
mAcc1
mloUf
SR
SR*
PDPP (Res-MLP)
37.2
55.35
66.57
21.48
57.82
65.13
13.58
8.47
PDPP (Transformer)
39.08
66.32
68.47
22.48
60.72
66.13
13.77
8.63
MTID (Transformer)
40.45
67.19
69.17
24.76
60.69
67.67
15.26
10.30
Transformer Classifier Type. We conduct ablation studies on the CrossTask and COIN datasets to evaluate the impact of transformer-bast based classifier. As shown in Tables I and 12 the in- clusion of the transformer-based classifier significantly boosts the performance of PDPP. Although the improvements
modest for longer horizons, this highlights the effectiveness of our temporal interpolation module on Cross Task compared to PDPP with the same transformer classifier. How- ever, the classifier's performance may also limit further improvements. Additionally, we observe that even with incorrect task class labels during supervision, the model still achieves strong scores, demonstrating its robustness, fault tolerance, and error correction capabilities.
maAce and moo
2000
always
linear
linear
square square
1&4
always
linear
linear
square
164
(a) Ablation studies on different complex initial- ization methods with max value 6.
(b) Ablation studies on different complex initial- ization methods with max value 1.
Figure 5: Ablation studies for interpolation strategy. Note: "always I" indicates that r initialized to '1': "linear 1" denotes that the values in the matrix increase linearly, with the first column initialized to '1' and the last column set '6' in Figure with gradual linear increase in between, and from '0' to '1" in Figure 55] "linear 4" represents the reverse process. "square 1" indicates that the value of r increases following a square progression. "t & 1" refers to a variation similar to our gradient loss weights, where the value first increases linearly and then decreases.
More Interpolation Strategies. We experimented with both linear and non-linear strategies, as shown in Figure 5) In Figure 5a we found that using a maximum value of '6' led to poor results, particularly for "square 1st and "square 4" strategies, indicating significant deviation from the desired final value. When reduced the maximum value to '1', the results still remained unsatisfactory, suggesting that the final value of r consistently converged around 'I', resulting in sub-optimal performance.
Number of Transformer Encoder Layers. Figure 6a shows the scores of three metrics across different numbers of layers in the transformer encoder blocks. The results indicate that using fewer layers (1 or 2) results in significant drop in performance, with SR being the most adversely affected. As the number of layers increases, the metrics stabilize, with notable improvements, especially in
17

---

Published as a conference paper at ICLR 2025
Table 12: Ablation study on the role of classifier type on COIN dataset.
T
4
Models
SR
mAcc1
mloUt
SR
mAcc
mloUt
PDPP (Res-MLP)
21.33
45.62
51.82
14.41
44.10
51.39
PDPP (Transformer)
24.02
48.03
55.21
17.36
46.12
55.82
MTID (Transformer)
28.84
50.44
57.86
21.64
48.06
59.52
SR, which shows significant positive shift at 6 and 7 layers. In contrast, mAce and mloU show more subtle variations, with slight positive changes as the number of layers increases, reflecting steady trend. These results suggest that optimal configuration of 6 or 7 layers delivers the best overall performance.
Table 13: Performance comparison of T and M.
Method
SR
mAcet
mloUt
T
38.64
66.13
68.05
M
40.45
67.19
69.17
Table 14: Ablation studies on different components on CrossTask. This ablation study effectively demonstrates the predictive capability of method. Note: The results of ID are from PDPP.
ID
Interpolation Module
Mask Projection
Proximity Loss
SR
mAcet
mloUt
37.20
64.67
66.57
2
39.03
66.49
68.26
38.88
66.36
68.35
4
38.57
66.02
68.17
39.64
66.74
68.77
6
V
39.71
66.65
68.83
39.17
66.49
68.38
8
40.45
67.19
69.17
Ablation for Our Different Methods. Table 14 presents the effects of our proposed methods. results demonstrate that each component signifcantly enhances the model's performance.
The
D MORE ANALYSIS FOR METHODS
More Explanation for M. Our MTID diffusion model takes as input a matrix containing action se- quences with T timesteps and is based U-Net, which contains M residual temporal blocks in the downsampling, upsampling, and middle layers for directly diffusing and generating intermediate target actions. To ensure that each intermediate layer contains valid auxiliary information, our Latent Space Temporal Interpolation Module needs to generate M intermediate auxiliary features. Subse- quently, apply cross-attention in residual temporal blocks across the M interpolated features and the entire input matrix rather than individual timesteps, enabling better temporal integration. We also conducted experiments demonstrate the effect of M. Our results Table 13 showed that using interpolated features only for T steps led to suboptimal performance. This also supports our decision to use interpolated features across all M modules.
In PDPP, for handling the loss, coefficient w with value of 10 is multiplied at the beginning and end positions. The paper believes that both sides more important because they are the most related actions for the given observations. Experiments have shown that this approach indeed effective. However, in my opinion, it doesn't 't better conform to the pattern of action accuracy. From my observations, found that the accuracy tends to decrease for actions closer to the middle, but not identical the middle as shown in Figure 6b This is because the supervision from real visual features is stronger on both ends, while,
More Explanation for Weighted Gradient Loss.
toward
18

---

Published as a conference paper at ICLR 2025
Accumcy (%)
(a) Ablation studies on the number of transformer en- coder block layers.
(b) Accuracy changes different actions at vari- ous time steps as epochs progress when T=5.
Figure 6: Combined ablation studies of different coefficients. Note: "tl" refers the number of transformer encoder layers in Figure 6a
the middle, there actually less available real information. Inspired by this, designed loss function that imposes stricter supervision constraints on both ends while relaxing the constraints in the middle, achieving better results.
Upper Bound of Visual Features Supervision. The comparison presented in Table 15 reveals that results vary depending on dataset characteristics, particularly size, task types, and average action sequence lengths. To explain this, categorize interpolated features into two parts: simple memory and hard temporal relationships. For instance, COIN, which has the largest dataset size but the shortest sequences, demonstrates that interpolated features excel in tasks focused simple memory. In contrast, NIV, being the smallest dataset with the longest sequences, shows comparable performance between real and interpolated features. Meanwhile, CrossTask, characterized by large size and long sequences, exhibits a significant performance gap favoring real features. These find- ings highlight trade-off where interpolated features perform well in simpler datasets but struggle with complex temporal relationships in larger, more diverse datasets. This underscores the necessity for improved interpolation methods to effectively manage complex, temporally diverse datasets in future research.
Table 15: Combined results for CrossTask, COIN, and NIV datasets with interpolated features and original real features.
T
T 4
T=5
T 6
Dataset
Method
SR1
mAcef
mloUt
SR1
mAcef
mloUt
SR
SR
Interpolated
40.45
67.19
69.17
24.76
60.69
67.67
15.26
10.30
CrossTask
Real
49.05
73.62
73.23
36.55
70.42
72.09
24.88
24.02
Interpolated
30.90
52.17
59.58
23.10
49.71
60.78
COIN
Real
27.07
49.07
57.53
20.01
47.35
58.24
Interpolated
29.63
48.02
56.49
25.76
46.62
58.50
NIV
Real
32.59
50.25
56.40
24.02
48.36
58.92
Comparison with PDPP. The results on COIN and NIV under the PDPP settings, as presented in Table [6] indicate that our performance on NIV is slightly lower due two main factors. First, the dataset size of NIV is significantly smaller than that of CrossTask and COIN, which leads to the model excessively learning detailed patterns from the training data and consequently reducing its generalization ability. Second, there are differences in experimental settings: PDPP defines states as the window between start and end times, while KEPP uses a 2-second window around start and end times. This difference allows PDPP to access more step information, particularly for short- term actions, which may weaken the impact of our interpolation feature supplementation. Despite these challenges with NIV under PDPP settings, our model demonstrates strong capabilities on the larger CrossTask and COIN datasets, highlighting its effectiveness in temporal logic and memory utilization.
19

---

Published as a conference paper at ICLR 2025
Table 16: Comparisons between PDPP and MTID under the setting of PDPP.
COIN
NIV
Models
T=
T=4
T=
T T=
SRT
MAAceT
mloUr
SRT
mAcof
mloUT
SRT
mAcef
mloUT
SRT
mAcef
mloUT
PDPP
21.33
45.62
51.82
14.41
44.10
51.39
30.20
48.45
57.28
26.67
46,89
59.45
MTID
30.90
52.17
59.58
23.10
49.71
60.78
29.63
48.02
56.49
25.76
46.62
58.50
E FURTHER DISCUSSIONS
Limitations. The limitations of our method are as follows. First, while the logical consistency between the actions generated by our model is generally strong, there is no guarantee of perfect alignment with the task. Mismatches were observed during the experiments, which common issue in procedure planning models. This challenge arises because the labels for multi-task and multi-action tasks the dataset are replaced by data IDs, which may lead to issues with numerical calculations.
Comparison Across Supervision Strategies and Mid-State Handling. Our MTTD model intro- duces several innovations that set it apart in terms of how it handles supervision and mid-state action prediction: (1) Supervision Approach: Weak vs. Full Supervision: DDN, PlaTe, and Ext-GAIL rely on fully supervised learning, requiring extensive annotations model temporal dynamies. In contrast, MTID uses a weakly supervised approach with a latent space temporal interpolation mod- ule, capturing mid-state information without detailed annotations. Its diffusion process and latent interpolation offer finer-grained supervision for intermediate steps, outperforming Ext-GAIL and DDN in long-term predictions. (2) Intermediate State Supervision and Logical Structure: PDPP uses task labels to bypass intermediate state supervision, and Skip- reduces uncertainty by skip- ping uncertain intermediate actions. However, both methods struggle fully capture the logical structure of intermediate steps. MTID addresses this by explicitly supervising mid-state actions through latent space interpolation, ensuring that the generated sequences are both temporally logical and well-aligned with the task requirements. (3) Handling of External Knowledge and Probubilistic Guidance: P3IV leverages natural language instructions for weak supervision, while KEPP uses probabilistic procedural knowledge graph (P2KG) to guide the planning process. While both methods aim to improve action prediction through external guidance, MTID distinguishes itself by focusing direct mid-state supervision via intermediate latent features from a diffusion model. This approach provides more precise control over action generation, ensuring logical consistency across the entire sequence. (4) State Representation and Visual Alignment: SCHEMA relies on large language models (LLMs) to describe and align state changes with visual observations, focus- ing on high-level state transitions. MTID, in contrast, directly uses mid-state supervision through latent space temporal interpolation, which improves visual- supervision and enhances temporal reasoning, resulting in more accurate action sequence predictions.
Generalization Capabilities. Our MTID model demonstrates strong generalization across varia- tions in action steps, object states, and environmental conditions. For action step variations, the model was evaluated with sequences of different lengths, ranging from 3 to 6 steps. The results consistently showed that MTID outperforms sfate-of-thh-ann models, leveraging its latent space tem- poral interpolation capture temporal relationships across various step lengths. In terms of object states and environmental contexts, the benchmark datasets used for evaluation cover a wide range of topics, such as cooking, housework, and maintenance, featuring diverse objects like fruits, drinks, and household items. For instance, the CrossTask dataset includes 105 step types across 18 tasks, while the COIN dataset features 778 step types over 180 tasks. These tests highlight MTID's ability to generalize effectively, capturing the nuances of varying object states and environmental conditions, due to its robust interpolation and diffusion framework.
20

---

Published as a conference paper at ICLR 2025
whisk
put beg
spreed
whisk
put mixtura
spreed mixture
0.3902
0,2061
0.2259
whisk
spread mixture
0.2900
0.2058
0.2075
V/2
whisk mixture
whisk mixture
spread mixture
V
0.3881
0.2862
0.2051
Fallure
whisk mixture
whisk mixture
add sugar
0.2344
0.3872
0.3032
(a) Horizon T
add onion
add rice
pour sesame oil
stir mixture
edd onion
add
pour
mixture
0.2475
0.3030
0.3493
add onion
stir mixture
add kimchi
stir mixture
0.3475
0.3215
0.3218
V/2
edd onion
stir mixture
add rice
stir mixture
V
0.2185
0.3738
0.2491
0.2721
Failure
add
0.3723
0.2201
0.3659
0.3287
(b) Horizon
GT
add onion
add lettuce
add meat
add tomato
add cheese
add
add lettuce
add
add
adi chease
0,2882
0.3039
0.3332
0.2718
0.2909
onion
0.2436
0.2803
0.3005
0.2510
0.2729
V,
add onion
edd taco
add lettuce
add tomato
add cheese
Vg
0,2773
Fallure
add onion
add lettuce
add
add cheese
add tomaao
0.2212
0.2353
0.2454
0.2591
(c) Horizon T =
top
SB8SOn Shepk
pal Steak ON grl
Mp steak
she wheck art
top steak
cut steak
0.2986
0.2593
0.2512
0,2811
0.2857
0,2661
SCason STOUR
pat KTEAK (H1)
Mp steak
creck senpartave
top steak
cut
0.2795
0.2757
0,2053
0,2053
0,2825
aeasOn STMAR
pol steak
Nip steak
top steak
cut steuk
Vg
0.2215
0.2103
0.3094
0.2633
0. 0.2029
0.2129
Fallure
stenh
App steak
top steak
cut steak
0,2044
0.2310
0,2454
0,2031
(d) Horizon T
Figure 7: Visualization diverse plans produced by our model with different horizons. Note: each figure includes images depicting the start and goal observations, the first row labeled "GT" showing the ground truth actions, the last row labeled "Failure" illustrating a plan that does not achieve the goal, and the middle rows displaying multiple reasonable plans produced by our model. These decimals represent the probability values obtained from action prediction.
21


Reviews:
[{"review_id": "4C3JUQqxEQ", "reviewer": "Reviewer_pfah", "summary": "This paper proposed a diffusion-based method, i.e. Masked Temporal Interpolation Diffusion (MTID) model for procedure planning. The diffusion model part is based on PDPP which takes the start and goal visual feature, action sequence and task class as input. The author proposed a Latent Space Temporal Logical Interpolation Module to generate the intermediate visual features given the start and goal visual features. Additionally, the authors proposed a masked projection and a task-adaptive masked proximity loss. the masked projection excludes the actions that do not belong to the tasks. The task-adaptive masked proximity loss puts more weight on the actions closer to the start and end actions and puts more penalties on the excluded actions. The experiments were performed on Crosstask, Coin, and NIV datasets, and SOTA results were achieved on most metrics of Crosstask and Coin datasets.", "questions": "In the paragraph starting from line 81, the authors need to clarify which parts in MTID are from PDPP (e.g. using predicted task class in the input matrix for the diffusion model, mask intermediate visual features, etc.)\n\n- In equation 1, why introduce M? v1:M should just be v2:T-1, i.e. the number of intermediate steps.\n\n- The notations for actions are inconsistent. Sometimes \\hat{a_t} is the ground truth action, sometimes \\bar{a_t} is (e.g. Eq 10), and sometimes a_t is (e.g. Figure 2). Please make them consistent throughout the full text.\n\n- The caption of Figure 2 needs to be elaborated. And the part to exclude actions that do not belong to a class is not shown. The fonts are too small to read.\n\n- The caption of Figure 3 needs to be elaborated. The fonts are too small to read.\n\n- Line 316, PDPP weights both start and end actions.\n\n- Put What does the dagger symbol mean in the caption of Table 1 too.\n\n- Line 359, Crosstask has 105 action classes.\n\n- Merge Table 13 in the appendix to Table 3 in the main manuscript to show results on all metrics on Coin and NIV.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper addresses the limitation of the previous diffusion model approach, i.e. the temporal dependencies of actions, by using generated intermediate visual supervision.\n- The Latent Space Temporal Logical Interpolation Module seems to be helpful as the intermediate visual supervision.\n- The task-adaptive masked proximity loss seems to contribute to the increase of performance.", "weaknesses": "- The Latent Space Temporal Logical Interpolation module does not have anything to do with 'logic'. It only has temporal dependencies between actions. Linear interpolation between start and goal visual observations with weights is not logic. \n\n- The experiment setting needs to be specified clearly in the main manuscript. The results on Coin and NIV datasets are misleading. This paper follows the setting of PDPP, however the results of baseline models (KEPP, SCHEMA) follow a different experiment setting. This needs to be clarified. Furthermore, the NIV dataset misses the results of PDPP, the proposed method does not outperform PDPP on all metrics for both time horizons.\n\n- Although the proposed ideas are interesting, it is not clear how the Latent Space Temporal Logical Interpolation module and the task-adaptive masked proximity loss contribute to the improvement on performance as they are built on top of PDPP. It will be beneficial to see the ablation studies of different combinations of model components on all datasets, i.e. diffusion + Latent Space Temporal Logical Interpolation, diffusion + masked project + task-adaptive masked proximity loss.\n\n- The presentation of the work has ambiguities and needs to be clarified. For details see comments below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wi4grFLlgE", "reviewer": "Reviewer_qUxm", "summary": "This paper introduces the Masked Temporal Interpolation Diffusion model to improve procedure planning in instructional videos. MTID integrates a latent space temporal logical interpolation module into a diffusion framework, generating richer visual supervision for intermediate states. The model is evaluated across benchmarking datasets and has been shown to achieve promising results.", "questions": "Does the task-adaptive masked proximity loss sufficiently balance intermediate state supervision with endpoint accuracy, or are there alternative loss functions that could enhance performance?\n\nThe classification results in NIV datasets seems to be way better than that of the COIN dataset. However, as shown in Table 3, the procedure planning results in terms of success rate and mAcc do not show then same trend. It seems like your model perform slightly better or generally the same on COIN rather than on NIV. How can you explain this discrepancy?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed MTID model is a novel application of diffusion models for procedural planning, introducing an innovative interpolation mechanism to enrich visual supervision, which is rarely addressed in the field.\n\nThe paper provides strong experimental evidence demonstrating that MTID outperforms existing models on multiple benchmark datasets. The results on CrossTask, COIN, and NIV datasets suggest significant improvements in most evaluation metrics. Plus the ablation studies are detailed and informative.", "weaknesses": "The model's architecture, especially the use of latent space interpolation combined with diffusion processes, may be difficult for readers to grasp fully. More intuitive visualizations or a simplified explanation could aid in better understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lgrUdMId1A", "reviewer": "Reviewer_DGZV", "summary": "This paper proposes a Masked Temporal Interpolation Diffusion model to tackle the problem of procedure planning in instructional videos, including a latent space temporal logical interpolation module. The interpolated mid-state visual feature is used to guide the intermediate process within the diffusion model. The MTID achieves SOTA performance on several datasets.", "questions": "Refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Based on previous works, this paper additionally uses the interpolated mid-state latent visual feature to supervise the model.\n2. The proposed MTID model achieves SOTA performance on several datasets.", "weaknesses": "1. The core concept is using the interpolated visual feature to provide visual-level mid-state supervision. However, the motivation of this operation is unclear. Using linear interpolation to interpolate the mid-state visual feature based on the start and goal visual feature is challenging. Additionally, an experiment that directly uses the visual features of mid-state for supervision (like previous works DDN) should be added, which can be seen as an upper bound.   \n2. In this paper, the task classifier uses a transformer, howerer,  in previous papers (EGPP, PDPP), a MLP is enough for this simple classification task.\n3. The masked projection introduces too much prior (i.e., the action space of each task).\n4. More analysis should be provided about the Task-Adaptive Masked Proximity Loss ablation, (MSE+GW achieves 15.1, while MSE+W+GW+M achieves 15.26). What's the weighted gradient loss, the authors should give more detail in the Method part. \n5. The novelty of the proposed method is limited compared with previous work PDPP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)