Customized Procedure Planning in Instructional Videos

Authors: Ali Zare, Constantine Tsibouris, Xudong Lin, Li Zhang, Ming-Hsuan Yang, Shih-Fu Chang
Primary area: applications to robotics, autonomy, planning
Keywords: Customized Procedure Planning, multi-modal models, vision-language models

Abstract:
Generating customized procedures for task planning in instructional videos poses a unique challenge for vision-language models. In this paper, we introduce Customized Procedure Planning in Instructional Videos, a novel task that focuses on generating a sequence of detailed action steps for task completion based on user requirements and the task's initial visual state. Existing methods often neglect customization and user directions, limiting their real-world applicability. The absence of instructional video datasets with step-level state and video-specific action plan annotations has hindered progress in this domain. To address these challenges, we introduce the Customized Procedure Planner (CPP) framework, a causal, open-vocabulary model that leverages a LlaVA-based approach to predict procedural plans based on a task's initial visual state and user directions. To overcome the data limitation, we employ a weakly-supervised approach, using the strong vision-language model GEMINI and the large language model (LLM) GPT-4 to create detailed  video-specific action plans from the benchmark instructional video datasets (COIN, CrossTask), producing pseudo-labels for training. Discussing the limitations of the existing procedure planning evaluation metrics in an open-vocabulary setting, we propose novel automatic LLM-based metrics with few-shot in-context learning to evaluate the customization and planning capabilities of our model, setting a strong baseline. Additionally, we implement an LLM-based objective function to enhance model training for improved customization. Extensive experiments, including human evaluations, demonstrate the effectiveness of our approach, establishing a strong baseline for future research in customized procedure planning.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730018517078

thread_type:
official_review

<thread_object>
creator: reviewer_pgdi

<content>
summary:
This paper investigates a more practical formulation of PPIV that considers user directions, called customized procedure planning in instructional videos.To overcome data limitations, the authors built a novel pipeline to collect customizations from existing PPIV datasets. Finally, a Customized Procedure Planner (CPP) framework (based on Llava) with a customization loss is proposed.

strengths:
1.  The article investigate an interesting problem, extend conventional procedure planning to open-vocabulary, varied, and detailed instructional plans.
2. The authors propose a weakly supervised training approach that addresses the lack of customization annotations for CPPIV model training.

weaknesses:
1.  This paper builds a pipeline for generating Customized Plans. The authors need to provide more examples and statistical results to demonstrate the effectiveness of the generated plans. Additionally, most of the keywords provided as examples in the manuscript are materials used in the production process, which does not quite align with the notion of customization.
2. In the data collection pipeline, does the VLM input include a Generic Plan? The prompt in Figure 2 does not seem to contain this information. Does the VLM model have such capabilities or knowledge?
3. In this task, the ground truth (GT) usually consists of gerund phrases, while the model generates full sentences. Could adding related prompts to constrain the model's output improve performance?
4. The authors should provide more examples to demonstrate the effectiveness of the proposed model and modules outlined in the text, Instead of just using numerical results.
5. The impact of the Customization Loss is marginal.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730530562494

thread_type:
official_review

<thread_object>
creator: reviewer_m1et

<content>
summary:
To address the potential challenges of lacking details in action steps in procedure planning in videos, the authors propose Customized Procedure Planner (CPP) framework to predict detailed action steps. They also used foundation models to create detailed action labels for benchmark dataset COIN and CrossTask. They also propose automated LLM-based metrics to evaluate the proposed models, therefore setting baselines.

strengths:
-The authors pinpoint the problem of lacking detailed action steps in textual form that could distinguish the completion of task in procedure planning. 

-To address this problem, they proposed the CPP framework and leveraged foundation models to train the model with pseudo action labels. The experiments in work is extensive.

weaknesses:
-The authors failed to advocate the gravity and scientific significance of the problem (e.g. lack of detailed action steps or user requirements) that they were trying to solve. 

-It seems that both are achievable just expanding the input/output spaces of previous tasks. 

-The proposed framework lacks novelty. It is a combination of foundation models, designed to solve a very specific task. 

-The authors only compare their proposed model with two foundation model baselines. 

-Using foundation models to do the evaluation are not robust because the definition of task success is not based on ground-truth.

questions:
-How is user requirements and detailed action steps fundamentally different from previous task instruction and action outputs? Can I view them as merely adding more detail to the data in the same modality?

-What solving this task matter? Can you prove or is there evidence that it might have impact other fields (e.g. in real world robotics?)

-Have you tried other combination of foundation models to solve your task?

-How should you reproduce your experiment results since commercial foundation model outputs are not reproducible?

-Why choose COIN and CrossTask datasets? There are more recent datasets (e.g. Ego4D).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573049825

thread_type:
official_review

<thread_object>
creator: reviewer_uhfc

<content>
summary:
The paper introduce a new task called customized procedure planning (CCP) as an extension to the task of procedure planning in instructional videos. This task generates action plans in natural languages conditioned on user-specific requirements and task objectives, utilizing a weakly supervised approach to overcome the lack of detailed customization annotations in existing datasets. The authors propose a training method, leveraging Large Language Models (LLMs) like GPT-4 for generating pseudo-labels and for enhancing customization through a novel objective function. The paper also introduces new LLM-based metrics to evaluate open-vocabulary, user-specific plans.

strengths:
1. The introduction of CPPIV is a valuable extension of traditional procedure planning tasks, addressing the limitations of existing models that do not output natural language plan. 

2. The development of LLM-based metrics to evaluate open-vocabulary plan customization and quality is innovative.

weaknesses:
1. While the new metrics are interesting, the reliance on LLM-based evaluation could be perceived as less interpretable and overly dependent on the LLM’s performance and biases.

2. Rather than simply framing this task as catering to user-specific needs, the primary distinction lies in how the goal is represented. Traditional procedure planning approaches are goal-oriented, often defining the goal using a single image. In contrast, this approach defines the goal using an Objective along with specific Conditions, providing a more nuanced and customizable representation.

3. The paper does not thoroughly address the potential limitations and biases introduced by pseudo-labeling, especially given that human-annotated datasets remain scarce.

4. The results on CrossTask and COIN are also somewhat difficult to interpret. Since these datasets lack ground truth action plans expressed in natural language, the evaluation relies on pseudo-labels generated by LLMs. This introduces a challenge: comparing model outputs, which are also generated by LLMs, against pseudo-labels from the same or similar models raises questions about the objectivity and robustness of the evaluation process.

questions:
Could you elaborate on the potential biases introduced by using GPT-4 and GEMINI for pseudo-labeling and how they may affect the quality of the generated plans? How does the model handle cases where the user-specified conditions conflict with one another or with the task objective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731125344310

thread_type:
official_review

<thread_object>
creator: reviewer_wx1o

<content>
summary:
The paper addresses the issue of generating customized procedures for task planning in instructional videos. Existing methods face challenges like overlooking customization and lacking proper datasets. The contributions are significant. It presents a novel setting for customized procedure planning, emphasizing user - specific needs. The Customized Procedure Planner (CPP) framework is proposed, which utilizes LlaVa - based models and is trained with pseudo - labels generated through a weakly - supervised approach. New evaluation metrics are introduced to assess planning and customization quality. Experimental results on CrossTask and COIN datasets show CPP's superiority over baselines like GPT - 4o. The integration of customization loss further enhances performance. Overall, this research lays a strong foundation for future work in customized procedure planning.

strengths:
1. The paper shows a novel task and CPP framework, using models creatively for generating customized procedures for task planning.

2.It is well-written and clear. The introduction motivates the problem, and the technical approach is detailed.

weaknesses:
1. The CPP model is trained and evaluated on a specific set of instructional video tasks (mostly related to cooking and DIY activities in the used datasets). It is unclear how well the model would generalize to other types of tasks or domains that have different characteristics and action requirements. 

2. The process of creating pseudo - labels using GPT - 4o and GEMINI might introduce some biases or inaccuracies. 

3.The interpretation of the "relevance score" for customization quality assessment could be more straightforward. The rubric used to measure customization is somewhat subjective, and it might not be clear how different users would rate the relevance of a plan.

4.The human evaluation seems to focus mainly on validating the model's performance rather than exploring potential areas for improvement. A more in - depth qualitative analysis of the human feedback could uncover additional insights into the strengths and weaknesses of the CPP model and guide further refinements.

questions:
See details in 'Weakness' section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732467118172

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We deeply appreciate the insightful feedback provided by the reviewers. We are pleased that Reviewer wx1o acknowledged the novelty and creativity of our CPP framework, highlighting its well-written problem definition, motivating introduction, and clear technical approach. Reviewer UHFc commended CPPIV for its contribution to extending traditional procedure planning and recognized the innovation of our LLM-based metrics for evaluating open-vocabulary plan customization. Reviewer m1et appreciated our identification of the gap in detailed action steps and the thoroughness of our experiments, while Reviewer Pgdi praised the significance of our weakly supervised training approach in addressing the challenge of lacking customization annotations. We are grateful for the recognition of the impact and importance of our work by all reviewers.

We have addressed their concerns  below, and hope they will update their scores if they find our responses clarifying.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper works on a novel task and develops a Customized Procedure Planner (CPP) framework for generating customized procedures for task planning.

After rebuttal and discussion, this paper receives review scores of 5,5,6,6. So it is a paper on the fence. 

Reviewers raised many concerns, such as lack of novelty, more solid results to demonstrate the effectiveness of the generated plans, more robust evaluation, and more comparisons. After the discussion, reviewers acknowledged that some of the concerns have been addressed, but many concerns remain, and the submission needs further refinement. 

AC has checked the submission, the reviews, the rebuttal, and the discussions between the authors and the reviewers. AC appreciated the authors' effort in improving the submission in the discussion, as many new results were added. However, AC agreed with the reviewers that better evaluations of the generated plan need to be done beyond just using foundation models. For example, authors can evaluate the generated plan by executing it in closed-loop simulation, such as simulated robotic manipulation tasks. Also, AC echoed the comment from the reviewer that this submission lacks novelty and theoretical foundation in the sense that it uses a foundation model to achieve some specific tasks. There is no guarantee of the output. Thus, a rejection is recommended.