Visualizing Entity States in Recipes by Generating Step Images
Authors: Chengang Hu, Yansong Feng


Reviewer paper summaries: This paper proposes a novel task of step image generation in recipes, using step images as visual supervision for tracking entity states in procedural text without relying on manually annotated data. For this task, the authors collect high-quality multimodal recipe datasets, theSpruceEats. Addressing the limitation of existing two-stage methods in achieving deep interaction between text and image, this paper introduces an explicit state modeling approach based on multi-modal generative models. Experiments on theSpruceEats dataset demonstrate that the method enhance entity state tracking and image generation quality compared to existing methods.
Summary of strengths: This article is well written and easy to understand.
This article proposes a new dataset, which will be helpful to the community if it is open sourced in the future.
Summary of weaknesses: 1. The author did not visualize the entity states in each step in the paper.
2. The author only conducted experiments on the proposed dataset. It should also be compared with SOTA methods in the RecipeQA dataset.
3. Generating step images will lead to a decrease in efficiency. The author should compare the model complexity and training and inference time with the SOTA methods.
Comments, suggestions and typos: see "Summary Of Weaknesses"
Overall assessment scores: 2