Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new task as well as a dataset that was collected from instructional video datasets. Apart from this, they also have a model proposed for this task. They analyze the proposed data and achieve good performance with their model.
strength_and_weaknesses: The difference in state representations or difference between visual features has been widely used in video understanding tasks. The difference in sensitive encoding in this paper is not new since lots of video captioning also has a close idea.

The masking mechanism in this paper is not to predict the specific ground truth, such as words or attributes; instead, they mask the input features and predict the representation. However, this paper lacks details, and the authors should describe more about the masking and loss function design.


Also, since the feature keeps changing, masking in the feature space makes the MTM harder. Instead of masking 15%, did the authors try other masking ratios?

In their setting, they miss the SPICE score, which is also widely used in the image captioning domain. Also, considering the gap between machine evaluation and human judgment, it is necessary to have humans evaluate the predictions captions. Since the visual description in this paper is much longer than image captioning, simply replying to language rules may not well reflect the real performance of predictions.

CLIP model was pretrained on image-text data, while the data in this paper is video data; considering the gap between images and videos, what are the effects of the pretrained model for this setting? Did the authors consider some video-text pretrained model for this task?
clarity,_quality,_novelty_and_reproducibility: The writing of this paper is clear, and their main contribution is the data and the proposed method. The model contribution of this paper is limited, and some data analysis is not solid also. Some implementation details are not described clearly in this paper. 
summary_of_the_review: Generating descriptions for video has been a long time, and this paper proposes a new task called Visual Transformation Telling. This task is new and interesting, and the baseline models reported in this paper cover most of the recent frameworks. However, considering the dataset is their major contribution, I still think this paper still has some room to improve.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new visual reasoning task, Visual Transformation Telling (VTT), to evaluate models' understanding on action-state associations. The authors leveraged instructional videos in existing datasets to form image sequences and ask models to predict the corresponding transformation text in a captioning manner. The authors also proposed an end-to-end transformer-based model for solving this task and show compelling performance results with captioning baselines.
strength_and_weaknesses: [+] The proposed visual transformation telling task, sharing the same spirit with text-guided visual prediction or dynamics learning, studies important problems including action grounding and world dynamics learning. However, evaluating the description of transformation can be a more direct evaluation compared with image generation metrics since it could be made more structuralized in a quantitative evaluation setting. Therefore I believe, this task could be a good evaluation approach (or checklist) to test models' capabilities on fine-grained world dynamics learning and activity understanding.

[+] The authors provided sufficient ablative studies showing the limitation and strengths of current models and also their proposed TTNet. This could provide new insights into the model design (e.g. masking for autoregressive decoding), incorporating state difference features as context for better transformation learning, etc.

[-] One key concern of this paper also lies in the task design. With the authors evaluating the final transformation as a captioning task, I do not feel that the clarity of text for a direct evaluation has been fully utilized. More specifically, the authors argued that the ultimate goal of the transformation-telling task is to provide a logically consistent explanation for a sequence of observations (i.e. images), however, I do not see any evaluation regarding this part. Can the captioning scores reflect the quality of transformation telling generated by the models? or to what extent should the model elaborate to have a good enough transformation description (e.g., should it be a simple action verb? verb+noun phrase? phrase with adverbs, or what?). Therefore, I do feel that the current task does not suffice as a thorough evaluation benchmark for fine-grained action grounding or action-state association understanding, limiting its significance.

[-] Next, I do have questions regarding the data. The authors select start and end images as states and chain all such states into an image sequence to form the input data for transformation telling given an instructional video. However, I do feel that there are concepts that can not be simply represented by a single image (e.g. reflected by temporal patterns); then did the authors check if there exists (or could be visually distinguished) transformation between each two images so that the transformation could be reasoned out? As I believe the current data sources CrowdTask and COIN contain various instructional activities, I do feel that this could potentially be a problem. Following the same though, several end states could be the result of several previous steps and not only a single step (e.g., soup being red not only because of adding tomato but also adding ketchup), the transformation might not necessarily follow a chain structure, causing ambiguous states for visual transformation telling. These all contribute to the concerns of task quality.

[-] Finally, despite the good performance of the proposed TTNet, I feel that its significance is somewhat limited as there is no specific representation bound to the transformation and therefore incapable of being used as a transformation representation for forward dynamics simulation. The authors could potentially connect with text-guided generation to see if this loop from transformation to text and text to transformation could be closed.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and clear despite some difficulties in reading (e.g. TVR in abstract without full illustration, Figure 3 could be equipped with better captions and in-figure illustrations, the current one is vague for the autoregressive part though could be understood with text). The proposed Dataset and task are novel but incremental to existing works regarding understanding action-state transitions in content and captioning in task format. The results seem reproducible given the descriptions in the paper.
summary_of_the_review: This paper proposes an interesting and potentially significant task, visual transformation telling, for vision-language/video-language learning. However, given the current status of the paper, there are several unclear or unjustified aspects of the paper that still need to be clarified by the authors or considered in future revisions (see weakness). The authors might also want to connect with other works (e.g. text-guided video generation, visual dynamics learning) to better address the significance of their model. Therefore, I'm recommending a reject for this paper and hope the authors could improve in future revisions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary:
- The paper proposes a new task called Visual Transformation Telling (VTT), which is defined as predicting the intermediate steps ('steps' are referred to as ‘transformations’ in the paper) in forms of natural language given the start and end state of each step. 
- Two existing instructional video understanding datasets, CrossTask and COIN, are combined to form the VTT benchmark. 
- A Transformation Telling Net (TTNet), consisting of the CLIP image encoder, a transformer based context encoder and a transformer based decoder for text generation, is proposed to tackle the VTT benchmark problem. 
- Three strategies are further proposed to address the challenges of VTT and thus allow the TTNet to be difference sensitive, context aware and context consistent. 
- In-depth experiments demonstrate the effectiveness of the proposed model and strategies. 

strength_and_weaknesses: Strength:
1. Experiments and analysis are quite thorough with ablation studies clearly demonstrating the impact of each proposed component of the paper.
2. The proposed model TTNet significantly outperforms the baselines and the proposed strategies are intuitive, reasonable and effective.
3. Interesting qualitative results were presented. 
4. Failure cases were shown and discussed.
5. The paper is well-written and easy to follow. 

Weakness:
1. The necessity or importance of the proposed new VTT task is not clear. The concerns are:
-  There are similar tasks such as Procedure Planning [1], Walkthrough Planning [1], and Assistant Writer [2] which is an abstract visual storytelling task. 
    * Comparison between the VTT task and these tasks should be discussed. 
    * The proposed VTT task seems to be a simplified version of the Procedure Planning task. Procedure Planning requires a model to predict the intermediate steps given only the start and end state of a *video*, whereas the VTT task sort of asks a model to predict the step given the start and end state of each *step*. Therefore, Procedure Planning seems to be more complex than VTT. In addition, the application-wise impact of the Procedure Planning task is quite obvious - an agent that is learned to perform Procedure Planning basically learned how to achieve a certain goal/task through some ordered steps. The application-wise importance of the VTT task is currently not clear to me. 
2. Whether the current way of constructing the VTT benchmark is sensible is not clear. The reasons are:
    * According to the paper, the first frame of the first transformation (i.e., step) is used as the first state and the last frame of all transformations are used as the remaining states of the video. However, can the start frame and the end frame really display the start state and end state of a step/transformation? Based on my experience with the CrossTask and COIN datasets, very often that the first frame and the end frame of a step cannot truly show the start state and end state of a step. For example, unlike what’s shown in Figure 8, the last frame of the step “Dig a pit with proper size” could be *not* actually displaying the digged pit - the outcome of the transformation, but rather to be a frame just showing the narrator, some tool, etc. There are existing works like "Look for the Change" [3] that aims to find out which 3 frames in the video correspond to the time depicting the object initial state, the state-modifying action and the object end state. Using the method in [3] or designing a method to identify the frames that truly showcase the start and end state of a step, in my opinion, is more sensible.  Otherwise, I’m concerned about how clean the start and end states are in the current VTT benchmark, and how this kind of noise would affect the benchmark. 
    * Does the current VTT benchmark really require sophisticated generalizability and reasoning skills from a model? How challenging is the benchmark really? Right now the train/val/test splits of the benchmark were randomly splitted. It is not really clear whether there are any types of biases in the current benchmark, and whether a model can leverage that potential bias to bypass reasoning skills (this is especially important because the authors claimed the proposed benchmark is a reasoning benchmark). Also because the dataset splits were randomly generated, it is not clear to me if the natural language generation of the transformation is necessary, or how important it is for the TTNet to have the natural language generation capability. Since the TTNEt has the natural language generation capability, this means that TTNet should be able to generate unseen verb-noun combinations, unseen order of steps for seen or unseen tasks. After checking the qualitative results, I’m curious about whether the ordered list of transformation descriptions appears in training and whether it is possible that the model simply learns to memorize these descriptions. If the set of transformation descriptions is the same in training and testing, then maybe there is no need to define the VTT task to output the natural language of the transformations; using the step classes like the traditional approaches would be sufficient. 
    * One possible improvement to the current VTT benchmark might be providing annotations for a small fraction of the videos from the “Related Task” of the CrossTask dataset and use these videos as testing videos (I’m assuming only videos from the “Primary Task” of the CrossTask dataset were used to form the VTT benchmark since only those videos were provided temporal step boundary annotations). In this way, the VTT training data will have a step transformation like “pour water”, and a step transformation like “pour coffee into glass” will be absent in training but present in testing. Such a testing set would allow one to demonstrate the power of the natural language generation complement. 

[1] Chang, Chien-Yi, et al. "Procedure planning in instructional videos." European Conference on Computer Vision. Springer, Cham, 2020.

[2] Ravi, Hareesh, et al. "AESOP: Abstract Encoding of Stories, Objects, and Pictures." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Souček, Tomáš, et al. "Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is well-written. 
- Quality: the quality of the paper is good.
- Novelty: the authors may need to justify the novelty as well as the application-wise impact of the proposed task.
- Reproducibility: the proposed model and strategies in this paper should be easy to reproduce. In addition, the authors have provided their code in Supplementary Materials but I did not really check. 

summary_of_the_review: The paper is really well-written, and the performance of the proposed methods is well-supported by experiments. However, the authors may need to further justify the VTT task and how they construct the VTT benchmark. 

*After rebuttal*: I appreciate the new analysis added by the authors which to some extent addressed my major concerns on whether the current VTT benchmark really requires generalizability from models and whether the VTT benchmark was reasonably constructed. Therefore I raised the score from 5 to 6. However, there are indeed several aspects of this manuscript that can be improved, e.g., technical novelty of the proposed model - as the authors admitted, the generalizability of the proposed model is not satisfactory. Overall, I think the idea/direction of the paper is worth pursuing, so I encourage the authors to continue improving the paper no matter what the final outcome is.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new visual reasoning task, called Visual Transformation Telling (VTT). For a given sequence of images, the VTT task is to provide natural language descriptions of the transformations between every two neighboring images. Unlike previous related tasks, which focus on just linking the states, VTT focuses on the transformational processes that link the states. The VTT task is operationalized by building on existing instructional video datasets, namely CrossTask and COIN. The paper proposes a new end-to-end learnt model, Transformation Telling Net (TTNet), which consists of an image encoder, context encoder (to link images) and a decoder (to generate text). The base TTNet is enhanced to boost performance through several ways, such as usage of difference features to boost sensitivity to changes. Comparisons with captioning baselines demonstrate good performance by the enhanced TTNet. The evaluations include both automatic evaluations, as well as human evaluations by 25 evaluators looking at fluency, relevance and logical soundness.


-- STRENGTHS --

1) The proposed task is conceptually interesting and novel.

2) The experiments and analyses (including ablations, qualitative results, failure cases) are fairly thorough.

3) The proposed model outperforms the various baselines.


-- WEAKNESSES --

Putting aside relatively minor weaknesses and those that have been adequately addressed or clarified, the remaining key weaknesses are as follows.

1) Evaluation of what really matters. In response to initial reviews raising issues about absent / missing evaluation of the logical consistency of the explanations (a purported key aspect of the VTT Task), the authors managed to add in subjective evaluations by 25 human evaluators. The subjective evaluations included fluency, relevant and logical consistency. However, there are remaining reasonable doubts about the feasibility of VTT due to necessity of using human evaluators -- logical consistency is a key aspect of VTT, and this can only really be evaluated by a panel of humans. While not a fatal issue, the paper would have been much stronger if a reasonable alternative had been developed. The authors did not satisfactorily address this in the rebuttals, ultimately. Even putting aside this issue, their human evaluation of logical consistency is a very coarse-grained, uni-dimensional one; the VTT task is most interesting because it holds the potential to address complex issues involving causality, contextual dependencies, commonsense knowledge, etc., but these are all folded into a single number.

2) While the initial premise of the VTT task is very interesting, the actual definition of it comes across as overly simplified, not well thought through, and with many unstated assumptions that are not very general. Roughly speaking, VTT assumes a few things such as meaningful transformations between steps (but what if they are just adjacent frames in a 120 FPS video, with little meaningful change? or take some extreme opposite), a linear chain of transformations, that the transformations are one-off, that the transformations nicely complete between two adjacent frames, etc. etc. It so happens that VTT works okay for the instructional video datasets chosen (because of their nature), but that severely limits the generality and applicability of the task in practice.

3) Technical novelty of the proposed model is weak, putting together existing components and known "bells and whistles".



justification_for_why_not_higher_score: Ultimately, the overall recommendation to reject is centred around two key aspects.

1) Interesting premise, but operationalization / implementation of the VTT task raises doubts because several key aspects of the task definition and design did not come across as well thought-out.

2) Instead of choosing to do a "dataset paper" and focusing on the quality and characteristics of the task and data, etc. typical of such papers, this paper had in equal parts a new task and a new model, neither of which were done particularly well.

This is certainly not a bad paper at all, with no obvious fatal flaws, but reviewers gave only lukewarm support (both numerically and in writing). Overall, there seems to be unanimous feeling (among reviewers and myself) that the paper is "not quite there yet" for the various reasons stated.

Personally though, I highly agree with reviewer "tWf2" when they state that "overall, I think the idea/direction of the paper is worth pursuing, so I encourage the authors to continue improving the paper no matter what the final outcome is."

justification_for_why_not_lower_score: n/a
Decision: Reject