Visual Transformation Telling

## Appendix A Datasheet for Visual Transformation Telling

### A.1 Motivation

#### A.1.1 For what purpose was the dataset created?

This dataset was created for a new visual reasoning task to test transformation reasoning ability in real-world scenarios. Given a series of states (i.e. images), VTT is required to describe the transformation between every two adjacent states. Distinct from existing visual reasoning tasks that focus on surface state reasoning, this dataset focuses on capturing the underlying causes, e.g. actions or events, behind the differences among states. (see paper for details).

#### A.1.2 Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?

This dataset was created by Wanqing Cui, Xin Hong, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. At the time of creation, Wanqing Cui and Xin Hong was the Phd students at Key Laboratory of Network Data Science and Technology (Institute of Computing Technology, CAS). Jiafeng Guo, and Xueqi Cheng are Professors there. Liang Pang is a Professor of Data Intelligence System Research Center (Institute of Computing Technology, CAS). Yanyan Lan is a Professor of Institute for AI Industry Research, Tsinghua University.

#### A.1.3 Who funded the creation of the dataset?

This work was supported by the National Key R&D Program of China under Grant 2022YFB3103704, in part by the National Natural Science Foundation of China (NSFC) under Grant 62276248, and in part by Beijing Academy of Artificial Intelligence (BAAI) under Grant BAAI2020ZJ0303

#### A.1.4 Any other comments?

None.

### A.2 Composition

#### A.2.1 What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?

Each instance is a set of $N+1$ images, annotated with $N$ transformation descriptions, an event topic, and an event category.

#### A.2.2 How many instances are there in total (of each type, if appropriate)?

The dataset consists of 13,547 samples with 55,482 transformation descriptions. The overall dataset statistics are shown in Table 1.

**Table 1: VTT dataset statistics.**
|  | CrossTask | COIN | Train | Val | Test | Total |
| --- | --- | --- | --- | --- | --- | --- |
| Categories | 4 | 12 | 12 | 12 | 12 | 12 |
| Topics | 18 | 180 | 198 | 198 | 198 | 198 |
| Samples | 1825 | 11722 | 10759 | 1352 | 1436 | 13547 |
| States | 12860 | 56169 | 54716 | 6974 | 7339 | 69029 |
| Trans. | 11035 | 44447 | 43957 | 5622 | 5903 | 55482 |
| Unique Trans. | 105 | 749 | 853 | 812 | 806 | 853 |

#### A.2.3 Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?

We will provide all instances in our GitHub repository.

#### A.2.4 What data does each instance consist of?

Each instance consists of a set of images of an event, the transformation description between every two adjacent states, and the topic/category of the event.

#### A.2.5 Is there a label or target associated with each instance? If so, please provide a description.

The targets are the transformation descriptions of the image set.

#### A.2.6 Is any information missing from individual instances?

No.

#### A.2.7 Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?

No.

#### A.2.8 Are there recommended data splits (e.g., training, development/validation, testing)?

We randomly split the data into Train / Val / Test sets with samples of 10759 / 1352 / 1436 in the level of topic.

#### A.2.9 Are there any errors, sources of noise, or redundancies in the dataset?

we randomly sampled 200 examples and found only a few flaws caused by careless annotators, such as black or text-transition screens. In most cases, humans can predict transformations from extracted states.

#### A.2.10 Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?

The dataset is self-contained.

#### A.2.11 Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)?

No. All raw data in the dataset is from public sources.

#### A.2.12 Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?

No.

#### A.2.13 Does the dataset relate to people?

The images in the dataset are all about daily activities.

#### A.2.14 Does the dataset identify any subpopulations (e.g., by age, gender)?

No.

#### A.2.15 Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?

Yes, some of the images contain faces.

#### A.2.16 Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)?

No.

#### A.2.17 Any other comments?

None.

### A.3 Collection Process

#### A.3.1 How was the data associated with each instance acquired?

The data was all collected from two public instructional video datasets: CrossTask and COIN. CrossTask’s and COIN’s precise temporal segment annotations. For the first transformation, we used the first frame of the corresponding step segment as its start state and the last frame as its end state. For the remaining transformations, the end state is extracted in the same way, while the start state shares the end state of the previous transformation. For transformation descriptions, we directly used original text labels as transformation labels.

#### A.3.2 What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)?

We mainly used Python to collect and process the data.

#### A.3.3 If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?

See answer to question in Section A.3.1.

#### A.3.4 Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?

All collection and annotation was done by the authors.

#### A.3.5 Over what timeframe was the data collected?

The dataset was collected in 2023, which does not necessarily reflect the timeframe of the data collected.

#### A.3.6 Were any ethical review processes conducted (e.g., by an institutional review board)?

No review processes were conducted with respect to the collection of this data is based on public datasets.

#### A.3.7 Does the dataset relate to people?

The images in the dataset are all about daily activities.

#### A.3.8 Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?

Other sources: CrossTask and COIN.

#### A.3.9 Were the individuals in question notified about the data collection?

N/A.

#### A.3.10 Did the individuals in question consent to the collection and use of their data?

N/A.

#### A.3.11 If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?

N/A.

#### A.3.12 Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted?

Yes. See the Conclusion and Discussion in the main paper.

#### A.3.13 Any other comments?

None.

### A.4 Preprocessing/cleaning/labeling

#### A.4.1 Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?

N/A.

#### A.4.2 Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?

N/A.

#### A.4.3 Is the software used to preprocess/clean/label the instances available?

N/A.

#### A.4.4 Any other comments?

N/A.

### A.5 Uses

#### A.5.1 Has the dataset been used for any tasks already?

No.

#### A.5.2 Is there a repository that links to any or all papers or systems that use the dataset?

No.

#### A.5.3 What (other) tasks could the dataset be used for?

The dataset could be used for visual transformation telling, which predicts the transformation between two states of an event.

#### A.5.4 Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?

N/A.

#### A.5.5 Are there tasks for which the dataset should not be used?

N/A.

#### A.5.6 Any other comments?

None.

### A.6 Distribution

#### A.6.1 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?

Yes, the dataset is freely available.

#### A.6.2 How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?

The dataset is free for download at [https://github.com/hughplay/VTT](https://github.com/hughplay/VTT).

#### A.6.3 When will the dataset be distributed?

The dataset is distributed as of June 2024 in its first version.

#### A.6.4 Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?

The dataset is released under CC BY-NC License.

#### A.6.5 Have any third parties imposed IP-based or other restrictions on the data associated with the instances?

No.

#### A.6.6 Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?

No.

#### A.6.7 Any other comments?

None.

### A.7 Maintenance

#### A.7.1 Who is supporting/hosting/maintaining the dataset?

Author Xin Hong is maintaining, who is hosting on github.

#### A.7.2 How can the owner/curator/manager of the dataset be contacted

By submitting github issue or e-mail. E-mail addresses are at the top of the main paper.

#### A.7.3 Is there an erratum?

Currently, no. As errors are encountered, future versions of the dataset may be released (but will be versioned). They will all be provided in the same github location.

#### A.7.4 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances’)?

Same as previous.

#### A.7.5 If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?

No.

#### A.7.6 Will older versions of the dataset continue to be supported/hosted/maintained?

Yes. All data will be versioned.

#### A.7.7 If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?

Errors may be submitted via the bug tracker and issue on github. More extensive augmentations may be accepted at the authors’ discretion.

#### A.7.8 Any other comments?

None.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 1
questions: Please see the weaknesses section.
soundness: 3
strengths: Significance: This work has its significance since it focuses on spatiotemporal processing, the authors discuss that some advanced MLLMs struggle to describe the given transformations (that being said, the evaluation procedure should still be revised).
summary: This work creates a new benchmark which tests how well the current multimodal LLMs (MLLMs) process visual transformations (i.e. change of states) between two adjacent visual scenes, like opening/closing a door, or breaking some glass. In this benchmark, a set of open-source and closed-source models are evaluated via both human validators and automated metrics.
weaknesses: The paper has a poor presentation.
- Fig. 1 could be improved by showing explicitly what are the inputs and the outputs in this task.
- Sec 3.1 should be linked to Fig. 1.
- The most importantly, the main text lacks details about TTNet, which makes following the paper very difficult.
- Fig. 2(d) completely could be put into the appendix (that could save some space for the TTNet).

The evaluation procedure.
- Some models achieve very high CIDEr scores. Could the authors please double check?
- While I appreciate the number of different evaulation metrics, I am skeptical about a couple of them.
- Why do we have two task-related human evaluation criterias (relevance/logical soundness) where they could be combined? If some generation is irrelevant, could the logical soundness matter for that generation? (could you please provide a counter-example if there is one?)
- Why do the authors evaluate fluency? The most of the transformation descriptions are relatively short (Fig 2.(c)) and the literature already evaluated the fluency enough previously.
- Is there a reason to avoid some evaluation procedure similar to LLM-as-judge in this work?
- It is not clear to me, how they evaluate the models. Is the evaluation per-transformation or per-scene?
- How are the images being processed by the models? For instance, as far as I know some models could process input images subsequently, are all of the tested models capable of doing the same? If not, how do the authors process input images? Do they concatenate input images?

The models.
- The number of open-source models could be increased (e.g. PaliGemma, OpenFlamingo, Idefics, InstructBLIP there are some suggestions, a newer suggestion could be Pixtral).
- Some video models could be added. Actually some of these models are pretrained using both video and image data, so they could be adapted easily (e.g. [Frozen-in-Time](https://arxiv.org/abs/2104.00650), [Video-LLaMA](https://aclanthology.org/2023.emnlp-demo.49/) and many others).

Missing some relevant literature.
- Video-Text models should be included.
- [The ViLMA benchmark's change-of-state task](https://openreview.net/forum?id=liuqDwmbQJ) is quite related to the proposed benchmark. It should be mentioned and discussed.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. How did you input multiple images if the model only supports single-image input?

2. What's the performance difference between single image and multiple image inputs? For example, will models' output differ when giving $s_i$ and $s_{i+1}$ versus only using $s_{i+1}$?

3. In table 2, why Gemini gets worse performance when using multi-turn?

4. What's the fine-tuned performance of the baselines?
soundness: 2
strengths: The experiments are comprehensive. The presentation of the paper is good.
summary: This paper proposes a new task called Visual Transformation Telling (VTT), which requires the model to generate a caption between two images in an instructional video. This paper evaluated the performance of existing models on this new task, which shows
weaknesses: 1. **Motivation.** The motivation to create this task is vague. It is unclear why describing the transformation between two images is more important than a single image. From the qualitative examples in the paper, it seems that only using one image is enough to generate the caption. The evidence for real-world or downstream applications is not clearly justified in the paper. Here are some questions that authors can think about when evaluating the motivation of this work: Will this new task benefit models on other tasks? What are the real-world use cases of such a task/model? Do I need a robot to understand the state transformation in **natural language format** to perform daily tasks? If the answer is only "current models cannot perform well on our task", then you need to really think about whether to do such research.

2. **Distinction with other video-language tasks.** Basically, this paper converts the existing video action recognition tasks like COIN into a video captioning task. The concept of state transformation is ambiguous, and it is hard to justify this in a better format. Furthermore, compared to the original format of COIN, the generation format of VTT poses additional difficulties in evaluation, as indicated by the bad performance of humans when using automatic metrics (Table 2).

3. **Limited Contribution.** Overall, this paper creates a task without strong motivation and claims the existing model cannot perform well on this new task. There is not much technical content in the paper, and even the data is reused from previous work. The overall contribution of this work is below the standard of ICLR.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How do you ensure whether a change between two images is explainable and deterministic? How do you evaluate and address the multiple possibilities of changes between two images?
2. Could more models trained on image-text interleaved data, such as InternVL-2, be evaluated?
soundness: 2
strengths: 1. The writing of this paper is clear.

2. The experiments in this paper are comprehensive, including various quantitative tests and visual result analyses.
summary: This paper introduces a task called Visual Transformation Telling (VTT), which aims to evaluate a model’s reasoning ability about the cause of state changes between two images, especially in natural language. The authors collect a large-scale dataset related to this task from instructional video datasets, conduct human verification, and create a new benchmark. This paper also tests the performance of existing open-source and commercial models and proposes a new model to address the task, which outperforms existing methods.
weaknesses: 1. The difference between VTT and the existing Procedure Planning task seem not significant. Typically, Procedure Planning tasks are done on CrossTask and COIN datasets, and there is already a lot of related work on this. Moreover, both tasks are based on pre-collected temporal segment annotations from CrossTask and COIN, so the textual content should be the same. It seems like this paper simply transforms a classification problem into an open-ended text generation problem.
2. If the focus of VTT is on explanation, why not use two images instead of a reasoning chain? How does the previous reasoning process affect subsequent reasoning? When the model makes errors, are they due to the accumulation of prior mistakes or the inability to infer the current two states?
3. Specific details regarding testing models like GPT-4o need to be provided. For example, GPT-4 is a purely text-based model—how did you test it, or did you test GPT-4V? Additionally, the specific prompts and the details such as model version should be thoroughly explained.



**It is recommended to add a figure that clearly describes the differences between the task proposed in this paper and all previously related tasks.**

Meta-review: Summary: This work proposed a visual reasoning task, Visual Transformation Telling (VTT), to tell the differences between visual states transformation. A dataset with 13k samples is provided to support the task.

Strengths: The experiments are comprehensive.

Weaknesses: (1) The differences between the proposed visual transformation telling and procedure planning are not clear, challenging the motivation and novelty of the proposed task. (2) The setting of telling the differences of two images is not convincing. (3) The evaluation and results are not convincing and need further clarification.

This paper received three negative ratings as 5, 3, 3. The authors did not respond to reviewers' comments.

Decision: Reject