Towards Multi-modal Procedure Planning in Instructional Videos
Authors: Ali Zare, Yulei Niu, Hammad Ayyubi, Shih-Fu Chang


Reviewer paper summaries: This work proposes a novel multi-modal procedural planning for instructional videos aiming to generate a series of action steps and image states, conditioned on initial and final observations.  It utilizes two transformers for generating action and state sequences showcasing good results on some planning datasets. It also proposes a weakly-supervised model that trains intermediate visual states on diffusion models with weakly labeled state from Gemini.
Summary of strengths: 1.This work proposes an extended PP problem into multi-modal domain and takes one step further than dual-modality PP by conditioning on both initial and end visual states.
2.This work showcasing promising results on CrossTask and COIN datasets.
3.This work is easy to read and well presented with a framework consisting of two auto regressive transformers each for action and visual states with conditional diffusion models for intermediate states.
4.This work proposes a weak supervision methods for intermediate visual states generation outperforming previous language-only supervision.
Summary of weaknesses: 1.For the weak visual supervision, the removal and avoidance of action-irrelevant visuals through Gemini is under-explained. Are there ablation studies used for captioning models except for Gemini? Does this weak visual supervision also outperform previous pure visual supervision ain Table 4?

2. The claim of this work that “expanding PP to the multi-modal domain,” is not correct. Task-wise, dual PP proposes the multi-modal PP task [Lu et al, 2023]. Besides, dual PP also bridges the text and image with two bridges, specifically the T2I bridge and I2T bridge, considering the intermediate states. This work seems incremental by adding a pair of start and goal states as generation conditions.

3.The input to the action decoder is the concatenation of all intermediate action step embeddings causing exploding generation complexity to the final state as the number of action steps goes up. The results of action horizon bigger than five are missing on CrossTask which has averaged 7.6 actions. Does it mean this model has a problem with longer action horizon planning?

4.Sec. 3.3 Utilizing Intermediate Visual Information is confusing. First, ”We predict the actions’ end states in the language domain and use them as conditions for editing the initial image states of the actions.” -> why are textual end states used for editing the initial visual states of the actions? Second, “the relevant visual knowledge generated by GenHowTo” is unclear.

5. The details of the newly constructed dataset for weak visual supervision are not delivered.
Comments, suggestions and typos: See weakness.
Overall assessment scores: 3