Visual Transformation Telling
Authors: Xin Hong
Yanyan Lan
Liang Pang
Jiafeng Guo
Xueqi Cheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: visual reasoning
transformation
captioning

Abstract: In this paper, we propose a new visual reasoning task, called Visual Transformation Telling (VTT). Given a series of states (i.e.~images), a machine is required to describe what happened (i.e.~transformation) between every two adjacent states. Different from most existing visual reasoning tasks, which focus on state reasoning, VTT concentrates on transformation reasoning. Moreover, describing the transformation in the form of language is more natural and closer to the real application than the property change way in the previous TVR task. We collect 13,547 samples from two instructional video datasets, i.e.~CrossTask and COIN, and extract desired states and transformation descriptions to form a suitable VTT benchmark dataset. After that, we introduce an end-to-end learning model for VTT, named TTNet. TTNet consists of three components to mimic human's cognition process of reasoning transformation. First, an image encoder, e.g. CLIP, reads content from each image, then a context encoder links the image content together, and at last, a transformation decoder autoregressively generates transformation descriptions between every two adjacent images. This basic version of TTNet is difficult to meet the cognitive challenge of VTT, that is to identify abstract transformations from images with small visual differences, and the descriptive challenge, which asks to describe the transformation consistently. In response to these difficulties, we propose three strategies to improve TTNet. Specifically, TTNet leverages difference features to emphasize small visual gaps, masked transformation model to stress context by forcing attention to neighbor transformations, and auxiliary category and topic classification tasks to make transformations consistent by sharing underlying semantics among representations. We adapt some typical methods from visual storytelling and dense video captioning tasks, considering their similarity with VTT. Our experimental results show that TTNet achieves better performance on transformation reasoning. In addition, our empirical analysis demonstrates the soundness of each module in TTNet, and provides some insight into transformation reasoning.

Under review as a conference paper at ICLR 2023
VISUAL TRANSFORMATION TELLING
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we propose new visual reasoning task, called Visual Transforma- tion Telling (VTT). Given series of states (i.e. images), machine is required describe what happened (i.e. transformation) between every two adjacent states. Different from most existing visual reasoning tasks, which focus on state reason- ing, VTT concentrates on transformation reasoning. We collect 13,547 samples from two instructional video datasets, i.e. CrossTask and COIN, and extract de- sired states and transformation descriptions to form a suitable VTT benchmark dataset. After that, we introduce an end-to-end learning model for VTT, named TTNet. TTNet consists of three components to mimic human's ' cognition process of reasoning transformation. First, an image encoder, e.g. CLIP, reads content from each image, then a context encoder links the image content together, and at last, a transformation decoder autoregressively generates transformation descrip- tions between every two adjacent images. This basic version of TTNet is difficult to meet the cognitive challenge of VTT. that is to identify abstract transformations from images with small visual differences, and the descriptive challenge, which asks to describe the transformation consistently. In response to these difficulties, we propose three strategies to improve TTNet. Specifically, TTNet leverages dif- ference features to emphasize small visual gaps, masked transformation model stress context by forcing attention neighbor transformations, and auxiliary category and topic classification tasks to make transformations consistent by shar- ing underlying semantics among representations. We adapt some typical methods from visual storytelling and dense video captioning tasks, considering their simi- larity with VTT. Our experimental results show that TTNet achieves better perfor- mance on transformation reasoning. In addition, our empirical analysis demon- strates the soundness of each module in TTNet, and provides some insight into transformation reasoning.
INTRODUCTION
What will come to your mind when you are given a series of images, e.g. Figure T Probably we first notice the content of each image, then we link these images in our mind, and finally conclude series of events from images, i.c. the whole intermediate process of cooking noodles. In fact, this is typical reasoning process from states (i.e. single images) transformation (i.e. changes between images), as deseribed in Piaget's s theory of cognitive development (Bovet) 1976; [Pagget 1977). More specifically, children the preoperational stage (2-7 years old) usually pay their attention mainly to states and ignore the transformations between states, whereas the reverse is true for children at the concrete operational stage (7-12 years old). Interestingly, computer vision developed through a similar evolution pattern. In the last few decades, image understanding, including image classi- fication, detection, captioning, and question answering, mainly focusing on visual states, has been comprehensively studied and achieved satisfying results.
Now time to pay more attention to the visual transformation reasoning tasks. Recently, there have been some preliminary studies (Park et al., 2019) Hong et al. (221)) on transformation. For example, Hong et al. [22II) defines a transformation driven visual reasoning (TVR) task, where both initial and final states are given, and the changes of object properties including color, shape, and position are required to be obtained based on a synthetic dataset. However, the current studies of transformation reasoning remain limited in two aspects. Firstly, the task is defined in artificial environment that is far from reality. Secondly, the definition transformation is limited to pre- defined properties, which cannot be well generalized unseen new environments. As result,

---

Under review as a conference paper at ICLR 2023
0144
So To add seasoning to the boiling water
S
S2
S3 T2 : pour the cooked noodles
Ti pour noodles into the water and stir
Figure/ I: Visual Transformation Telling (VTT): given states represented by images (constructed from videos), the goal is to reason and describe transformations between every two adjacent states.
the existing transformation reasoning task cannot meet the requirement of real-world applications. Furthermore, the lack of strong transformation reasoning ability will hinder some more advanced event-level reasoning tasks, such as visual storytelling(Ting (Ting-Hao et al. 2016) and procedure plan- Chang et al. 2020), since transformation plays important role in these tasks.
To tackle these limitations, we propose a new visual transformation telling (VTT) task in this paper. The main motivation is to provide descriptions for real-world transformations. For example, given two images with dry and wet ground respectively, it should be described it rained, which precisely describes cause-and-effect transformation. Therefore, the formal definition of VTT is to output language sentences to describe the transformation for a given series of states, i.e. images. VTT is different from video description tasks, e.g. dense video captioning Krishna al. (2017), since the complete process of transformations is shown by videos, which reduces the challenge of reasoning.
To facilitate the study of VTT, collect 13,547 samples from two instructional video datasets, including Cross/aakee Tassssaask et al., (019) and COIN (Tang et al., (02T) They are originally used for evaluating step localization, action segmentation, and other video analysis tasks. But we found them suitable to be modified to fit VTT, because the transformations are mainly about daily activities. and more importantly, some main steps to accomplish a certain job have been annotated in their data, including temporal boundaries and text descriptions. Therefore, we extract key im- ages from video as input, and directly use their text labels of the main steps as transformation descriptions. More details can be found in Section 3.2
When designing an effective VTT model, we face two kinds of challenges. The first one related to the cognitive challenge, which is to derive abstract transformation from images with small dif- ferences, c.g. from the difference between wet and dry ground to rained. The second one is the descriptive challenge, that is, the description of transformations should consider the consistency in a series of images to output a reasonable event. If we only consider the description for a single transformation, i.e. between two images, it is easy to output logical errors in the results.
In order to address these challenges, we propose a difference-sensstivv and context-an awwre model, named TTNet (Transformation Telling Net). TTNet consists of three major components, to mimic the human cognition process of transformation reasoning. To be specific, CLIP (Radford et al. 2021] is utilized as the image encoder to read semantic information from images into image vec- tors. Then a transformer-based context encoder interacts image vectors together to capture context information. At last, transformer decoder autoregressively generates descriptions according to context features. However, this basic model is not enough to meet the cognitive and descriptive challenges, so we use three well-designed strategies to improve TTNet. Specifically, the first strat- egy is to compute difference features on image vectors and fed them into the context encoder as well, to emphasize small visual gaps. Then, masked transformation model applied to capture the context-aware information, by randomly masking out the inputs of the context encoder like masked language model (Devlin et al., (2019). Finally, in addition to the general text generation loss, the whole network is also supervised under the auxiliary task of category and topic classification, which is to constrain the transformation representations to share underlying semantics, by mimicking hu- man's 's behavior that forms global event in mind.
Since the task of VTT is new, there is no made baseline model. Considering the similarity of visual storytelling and dense video captioning to VTT, modify typical methods including
2

---

Under review as a conference paper at ICLR 2023
CST Gonzalez- & Fuentes- Pinttes--iiedaaPP/ 2018). GLACNet (Kim et al. 2019), and Densecap Johnson et al. (2016) in these two applications as our baseline methods. Our experimental results show that TTNet significantly outperforms these methods. Additionally, we conduct comprchensive studies to show the importance of contextual information for VTT and the effectiveness of three strategies, including difference features, masked transformation model, and auxiliary learning.
In conclusion, our major contributions include: 1) the proposal of a new task called visual trans- formation telling to emphasize the reasoning of transformation in real world applications: 2) the introduction of TTNet which is a difference-sensstive and context-aware model for transformation reasoning. 3) extensive experiments on our collected data from instructional videos, demonstrating the effectiveness of TTNet and providing many insights for understanding the VTT task.
2 RELATED WORKS
VTT belongs the direction of visual reasoning, so first list some typical visual reasoning tasks and then discuss the relation between VTT and these tasks. CLEVR (Johnson et al., (017)) and GQA (Hudson & Manning 2019) concentrate
relation and logical reasoning
objects.
RAVEN (Zhagg et al. (0019) and V-PROM (Tnnyy et al. 2020) care about the induction and reason- ing of graphic patterns. VCR (Zellers et al. 2019) and Sherlock (Hessel et al., 2022) test whether machines are able to learn commonsense knowledge to answer daily questions. These tasks mainly concentrate on state-level reasoning. Apart from these tasks, there is a series of works related to dynamic reasoning. Physical reasoning (Bakhtin et al. 2019, Yi et al. 2020) Girdhar & Ramanan 2020 Baradel et al. 20200, RRochet et al., 2022) evaluates the ability to learn physical rules from data to answer questions solve puzzles. VisualCOMET (Park et al.. (2020) asks to reason beyond the given state to answer what happened before and what will happen next. Visual storytelling (Park) ct al. 2020) requires completing the missing information between states to describe story logically. Visual reasoning has a tendency to shift from static scenes to dynamic ones. While state and trans- formation are both important for reasoning in dynamic scenes, concentrate transformation reasoning, between state-only scenarios and more complex composite scenarios.
To the best of our knowledge, there are rare studies on designing specific tasks for visual transfor- mation reasoning. The only work is TVR (Hong et al.) (202T). Given the initial and final states, TVR requires to predict a sequence of changes in properties, including size, shape, material, color, and position. However, the synthetic scenario is far from reality and property change is not common fashion to describe transformations in life. more natural way the event-level description. For example, it is more natural to tell it rained when describing what happened between dry and wet ground outside. Visual storytelling (Ting- Maa et al., 2016; Ravi et al.. (20T)) requires event-level description but transformations are mixed in the description, making it difficult to evaluate transfor- mation only. Visual abductive reasoning (Liang et al. 2022) has a similar core idea to us, which aims to find the most likely explanation for incomplete set of observations. The difference is they only require machines to reason one single missing transformation from multiple transformations, while our task aims reason multiple logically related transformations from states. The motiva- tion of procedure planning Chang et al.] (2020) is to complete a job given states, while VTT is to explain transformations between states, which has wider scenarios, e.g. explaining the wet ground with rain. Furthermore, the requirement for natural language generation makes VTT have different evaluations and unique challenges such as generalization on language compositions. Walkthrough planning Chang et al. (2020) has a different target which is to predict intermediate states.
Talking about transformation description, there is another topic related, i.c. visual description. Here we review some typical visual description tasks and discuss their differences. Tasks that describe a single image include image captioning (Farhadi et al., 2010, Kulkarni et al.) (001)) dense image captioning (Johnson et al.) (016). and image paragraphing (Krause et al. 2017p. The difference lies in the level of detail. Similarly, tasks for videos include video description (Venugopalan et al.) 2015), video paragraph description (Yu et al. 2016), grounded video description (Zhou al. 2019), and dense video captioning (Krishna et al. (017)). Different from image captioning tasks that focus only on single state, video description tasks start to describe events. For example, dense video captioning asks to predict temporal boundaries and descriptions of key events in a video. However, they provide the full process of transformation throughout videos, reducing the need for reasoning.
3

---

Under review as a conference paper at ICLR 2023
Drink and Snaet
can
Applanne
#Samoles
Dish
ingredient eereente
add
Water de
halr
2000
Sport
take
Pata and
cover
#Sentoncoa
trarefomations
Godgets
sholl tace
tatses
nack
10000
Nursing and Care
ulter bed
Lesure and Performance Furnlure and Deszonaioo
Science Craht
words
(a)
(b)
(c)
Different distributions of VTT samples. (a) Category. (b) Words. (c) Transformation length (top), and sentence length (bottom).
Figure 2:
VISUAL TRANSFORMATION TELLING
3.1
TASK DEFINITION
Visual transformation telling aims to test the ability of machines to reason and describe transfor- mations from sequence of visual states, i.c. images. Formally, N + 1 images S = {s,, N++1 given, which are logically related and semantically different. Logically related means these images are associated with certain event, e.g. completing job, while semantically difference is to expect some substantial changes that are meaningful to people, i.e. transformation. The target is then to reason N transformations T = {t,, [t,] between every two adjacent images and describe them with natural languages, so that 81 - 1 +
n=1 are
ln - Su+1 logically sound.
3.2 VTT DATASET
To construct a meaningful dataset for VTT, we require the data to cover large scope of real world transformations. Therefore, we choose instructional videos as our basic library, because they con- tain many daily life activities. Specifically, we choose two typical instructional video datasets, i.e. CrossTast (Zhukov et al. 2019) and COIN (Tang et al. 2019: (021), and construct our data. Figure [ iillstrates an instruction video from COIN for cooking noodles and how we transform their
annotation into VTT dataset. We can see that the video is segmented into multiple main steps, and each step annotated with precise temporal boundaries and text labels. We directly use their text labels as transformation descriptions and extract states based on temporal boundaries. Specifically, for the first transformation, the first frame of the corresponding step segment becomes its start state and the last frame becomes its end state. For the remaining transformations, the end state is ex- tracted the same way, while the start state shares the end state of the last transformation. We check the quality of states and find transformations can be reasoned out in almost all samples. In this way, we collected 1 13,547 samples as well as 55,482 transformation descriptions from Cross Task and COIN, forming our new data for VTT. Figure 2 shows the distribution of the sample category, keyword, transformation length, and sentence length. From the category distribution and the word cloud, can that VTT data covers lots of daily activities, like dish, drink and snack, elec- trical application, vehicle, gadgets, leisure and performance, etc. Furthermore, the distribution of transformation length shows its diversity while most of samples contains about 2-5 transformations. Sentence length is around 2-6 on average which means short descriptions make up the majority.
4 METHOD
Problem Formulation. Our main idea for solving visual transformation telling is to find a parame- terized model fo to estimate the conditional probability p[T|S; 0) = ( N+1; 0). where s; E X is a state represented as an image and t, = {.r,, is sentence of length L. The conditional probability can also be written as regr-ssegysyyvvlly generating N sentences:
L III d (t)
p(
(1)
4

---

Under review as a conference paper at ICLR 2023
ispresemattins
masked transformation model
difference features
3. Transformation Decoder
Ltext
N
pour
thc
608
CLIP
AA
A
AA
Ee
Ee
Transfarmer
Transformer
sharing
Transformer
1. Image Encoder
+
auxiliary learning
Lestegory
FC
Ltopie
FC
representations
transtoration
2. Context Encoder
Figure 3: The architecture of TTNet. Images are first encoded into state representations in the im- age encoder, then transformed into transformation representations in the context encoder, and finally decoded into text by the transformation decoder. To be more difference-sensitiv and context-aware, three strategies are considered to enhance TTNet (marked with *). Difference features computed according to state representations are used extra features, state representations and difference fea- tures are randomly masked as zero during training, and auxiliary tasks are used for supervision.
Overview of TTNet. Our TTNet is designed to mimic human's 's cognitive process of transformation. The first step is independent recognition, which means that people may understand each image inde- pendently. Therefore, we introduce an image encoder fimage to represent each image to vector and N+1 obtain a set of image vectors V = [ = [mmage (s;) i=1 N+1 After that, humans will associate these images together, and form an understanding of all images guided by a global event. To reflect this process, we introduce context encoder, e.g. a bi-directional RNN or transformer encoder, de- noted as Scontext, to obtain context-aware image representations C' [c; N+1 { (Jconter (i, V by considering contextual information. The final step is to describe these transformations based on previous understanding. In TTNet, feed the last N context-awarr image representations to a transformation decoder feuptions implemented with an RNN or a transformer decoder, to gen- erate each transformation description T = { = {feaption (C++1) separately and auto- regressively. We empirically found adding the transformation representation to the word embedding in each step is better than using it as the start token.
The model is then trained with ground truth transformations T (4) by minimizing the following negative log-likelihood loss, where t ( in transformation.
is the ground truth description of the
N
L
Liext
log pat
In order to tackle the two unique challenges of VTT, i.c. cognitive challenge and descriptive chal- lenge, propose three specific strategies to enhance the above TTNet, including difference sitive encoding, masked transformation model, and auxiliary learning. To distinguish more clearly, called the model that does not these three strategies TTNet
4.1 DIFFERENCE SENSITIVE ENCODING
In visual transformation telling, the differences between two adjacent states are usually very small. Imagine the scene of cooking noodles, the whole picture does not change much before and after the noodles are added to the pot. This characteristic requires the model not only to understand the con- tent of each image, but also to focus on differences between images to facilitate the understanding of transformations. For this purpose, due to its strong semantic representation ability trained on large scale unsupervised data. We also introduce difference features, by subtracting the current state and the previous state representations
first utilize CLIP (Radford al. 2021 our image encoder,
AV {t. - J where vo UN+11 to emphasize the subtle difference. The above two kinds
5

---

Under review as a conference paper at ICLR 2023
of representations concatenated and fed the context encoder. Furthermore, type embedding added to distinguish these two kinds of features.
Since transformations are not independent, transformation description process, named the descriptive challenge of VTT. For example, the logic of descriptions does not make sense as shown in Figure 4 TTNet recognizes oranges eggs, which is logically unreasonable with the two transformations before and after. In TTNet, we intro- duce masked transformation model in the context encoder and auxiliary learning in the loss function to alleviate this problem.
may meet the logical consistency problem in the
4.2 MASKED TRANSFORMATION MODEL
Masked transformation model (MTM) inspired by masked language model (Devlin al. 2019).
The intuition behind this is that one transforma- tion can be reasoned from nearby transforma- tions. For example, if you are told the previous transformation is washing the watermelon and the next is putting the watermelon into a planet, it obvious that the intermediate transformation should be related to the watermelon. Following this intuition, 15% of the input features, including state representations and difference features, are randomly masked during training. Furthermore, we empirically found that, for each sample, using this strategy with half the probability works better.
1. Cut both ends and remove fruit seeds.
2. Pour the egg into the bowl.
3. Pour the orange juice into the cup.
Figure 4: TTNetaas failes to effectively use contextual information and mistakenly identi- fies the orange as egg.
4.3 AUXILIARY LEARNING
Human usually tries guess the category or topic before describing transformations, e-g. cook- ing noodles. Therefore, this category or topic information may help guide description generation. Inspired by this, we propose to leverage auxiliary tasks, i.e. category and topic classification, to su- pervise the training process. Specifically, we introduce two additional cross entropy losses Leategory and Looic on the global context vector. We expect to make the learned transformation represen- tations share the underlying category and topic information to enhance the learning of consistent representations. So the final training loss becomes a combination Liext. Leategory, and Ltopic:
L Liext + Ceateggry + B Lropic
where a and are two adjustment factors.
EXPERIMENTS
In this section, we first introduce our empirical setups including baseline methods and evaluation metrics. Then we demonstrate the main empirical results on the collected VTT dataset, including both quantitative and qualitative results. After that, we show extensive ablation studies on different strategies used in TTNet.
5.1 EMPIRICAL SETUPS
Visual storytelling and dense video captioning are the two most similar tasks VTT. Visual storytelling requests to generate descriptions from N images. We select two classic methods from the winners of visual storytelling challenge Mitchell et al. (2018), including CST Gonzalez-Rico & Fuentes-Pineda (2018), and GLACNet Kim al. (2019) for comparison. CST contextualizes image features by LSTM and then generates descriptions with separate LSTMs for each image. GLACNet mixtures global LSTM features and local image features into context features and then generates descriptions with a shared LSTM decoder. When generating transfor- mation descriptions, only the last N context features are used. Dense video captioning has a similar target to describe series of events. The difference is the input is video and it additionally requires to predict temporal boundaries for events. We choose DenseCap Johnson et al. (2016) for adapta- tion which proposed in the paper that introduces dense video captioning. DenseCap integrates the
Baseline Models.

---

Under review as a conference paper at ICLR 2023
Table 1: Performance on the test set of VTT dataset. B are short for BLEU @4 / METEOR ROUGE-L/ / CIDEr / SPICE BERT-Score Fluency Relevance / Logical Soundness. * indicates to use CLIP image encoder for a fair comparison. f indicates TTNet signifi- cant (p 0.05) outperforms the corresponding model on this human evaluation metric.
Model
B@4
M
R
C
S
BS
F1
Re
LS
CST
10.09
11.39
25.98
43.22
9.28
16.30
CST
13.96
19.21
38.11
84.60
21.85
25.66
2.04
3.16
2.96
GLACNet
42.77
45.26
52.98
381.48
45.33
60.12
GLACNet*
55.24
59,48
66.25
508.18
60.21
71.13
4.75
3,82
3,78
DenseCap"
48.25
52.00
59.79
439.68
53.73
66.30
4.74
3.67
3.59t
TTNel
55.68
60.47
67.05
515.12
61.45
72.22
4.79
4.04
3,951
TTNet
61.22
66.31
71.84
570.63
66.20
76.25
4.78
4.10
4.11
past and future information into image features capture the context information. There are many advanced methods for dense video captioning but highly rely on fine-grained video features, which are not suitable for our task. All three methods are implemented as closely as possible according to
the original paper and provide a fair comparison by using the same image encoder with TTNet. We describe the implementation details of TTNet as well as baseline models in Section C
Evaluation Metrics. Following previous works visual descriptions (Ting-Hao et al. 2016 Kr-
ishna et al. [iing et al. al.||2022 (2022), automated metrics including BLEU @4 (Papineni et al. 2002).
CIDEr Vedantam et al. 2015), METEOR (Banerjee & Lavie, 2005). ROUGE-L (Lin & Hovy 2002]. SPICE (Anderson et al. 2016), and BERT-Score (Zhang et al. 2020) are selected as auto- matic metrics. Furthermore, descriptions using a Likert scale ranging from 1 to 5, for following criteria: fluency, measuring how well-written the transformation is; relevance, how relevant the transformations toward the image states; logical soundness, how well the overall logic conforms to common sense.
asked 25 human annotators to assess the quality of transformation
5.2 RESULTS ON VTT DATASET
Quantitative Results. Table D summarizes the results of 7 models on the VTT dataset, including TTNet, TTNetaase CST and its CLIP version, GLACNet and its CLIP version, and CLIP version DenseCap. From the results, TTNet surpasses other models on most metrics with large margin, e.g. CIDEr is 11% higher than the second best model, i.e. TTNe This large improvement comes from the three strategies we proposed, which are the only differences between TTNet and TTNetaee- Further comparing human metrics between them, the main strength of TTNet is the much stronger overall logic of the generated descriptions, while the relevance is only slightly better and the flu- ency is about the same. Secrefsec:ablation further shows the advantages of TTNet with detailed ablation studies. It is not difficult find that the performance gap between CST GLACNet*, and Densecap is also very large. While they all use CLIP, the difference lies in the way context decoding and text generation. GLACNet* outperforms DenseCap* mainly because LSTM captures more information than past and future attention features according to the higher scores of relevance and logical soundness. The gap between GLACNet* and CST is caused by the way of text gen- eration. GLACNet uses word embeddings and context features as inputs in each LSTM step, while CST only uses the context as the initial state of LSTM. In our empirical studies, this little difference improves the fluency a lot, and it is the reason that TTNet chooses to add context embedding to word embedding as the inputs of the transformation decoder rather than using the context feature as the start token. The underlying design philosophy between TTNetause and GLACNet* is similar, therefore, the performance is close. However, TTNetaase converges faster than GlAtNNet* during training because the transformer captures the context information more efficiently than LSTM.
Qualitative Results. We show two examples from the VTT test data in Figure a sowing and pasting car sticker. From these two examples, we can first realize that the gap between the states is really small. For example, in the sticker case, only a small area of the sticker is changed, making it difficult to reason certain transformation without considering the overall pasting process. We can see that when the states are confusing, e.g. DenseCap and GLACNet identify the wrong entity in the case, TTNeet is able to reason the correct transformations from the differences and the
7

---

Under review as a conference paper at ICLR 2023
DonseCop:
GLACNet:
TINet:
candle vessel.
Dig
Sow the soil.
up the pumpkin. Dig some
Apply on soil.
Groundtruth: Sow sod. Apply the
DemoeCapp
GLACNet:
TTNet:
Groundrutt::
1. Tear off the from of the sticker.
1. Clean window surface.
Clean the window surface.
Clesn the window sun fufecee
the Tear off the front the sticker
3. Tear front sticker.
3. Tear the front of the sticker.
3. Tear off the
the sticker.
4. Tear off the front of the sticker.
4. Tear off the fromt of the sticker.
4. Presa the sticke.
Pross the slicker.
5. Tear the front of sticker.
5. Teur off the front of the sticker.
5. Tear off the from the sticker.
Tear off the front the sticker.
sticker.
the
the sicker.
Press sticker.
7. Tear the other side of the sticker.
7. Tour off the other side the sticket.
Tear
the sticker.
7. Tear off the other side the sticker.
Figure 5: Qualitative comparison on the VTT test data. Above: sow. Below: paste car sticker.
Table 3: Ablation studies
Table Analysis of difference
Table 2: CIDEr of independent transformation prediction.
the effect of key components.
features and auxiliary tasks.
Diff. MTM Aux.
State
Difference
CIDEr
Model
Original Indep.
515.28
72.22
V
527.62
CST
84.90 49.80
556.85 75.00
early late
559.78
DenseCap*
439.53 295.75
520.04 72.72
570.63
GLACNet*
508.19 268.49
v
521.93 72.97
TTNet w/o diff
527.62
422.04
Category Topic
CIDEr
562.25 75.62
TTNet
570.63
349.96
562.83 75.72
549.44
TTNet (retrain)
459.84
V
527.62 73.54
V
562.96
V
V
V
570.63 76.25
V
570.63
context. Furthermore, when the difference between states becomes rather small and the transforma- tion length becomes large, TTNNet is still able to judge subtle differences between transformations. In contrast, GLACNet indeed understands the topic of pasting the sticker but fails distinguish some transformations. In conclusion, TTNet is able to reason transformations from confusing states and distinguish subtle differences between transformations, making it excel other methods.
5.3 ABLATION STUDIES
In Section 4 introduce three strategies to improve TTNet, including difference sensitive encod- ing, masked transformation model, and auxiliary learning. In this section, we discuss the effective- ness of these three strategies. But before that, we first need answer the question that whether the context information is crucial for VTT, since all three strategies act on the context encoder to enhance the ability capture context information. If the answer yes, then comes to answer how these strategies work and whether there exist alternative choices, e.g. other types of difference features. Experimental analyses are organized into five following topics according to this logic.
Importance A Analysis of Context. To answer the question of whether the context is really important models predict each transformation independently.
for reasoning transformations, We design ie. only from two states before and after. If transformations can be reasoned without considering the context, model performance should remain roughly the same. However, from Table the CIDEr score of all five models drops sharply from the original setting to the independent setting, showing that the context is clearly very important. Without context, reasoning transformations become rather difficult, and retraining the model with independent data does not help either.
Effectiveness of Three Strategies. Next, we move on to analyze the effectiveness of the three strategies and their combinations. The first row in Table the result of TTNetasse and the next
8

---

Under review as a conference paper at ICLR 2023
three rows show the results of using each strategy independently the base model. Among them, the improvement of using difference feature is the most significant, indicating the difference is also crucial for resolving transformation reasoning. The next four rows show the results of combining these strategies and the conclusion is combining all three strategies leads to the best result. The next three topies will go through all these strategies one by one in detail, to see how these strategies work.
Analysis of Difference Sensitive Encoding. We just show difference feature is the most significant strategy for TTNet. However, it not clear how difference features help the TTNet model and if there are alternative choices for difference features, e.g. differences of raw images. To answer the first question, we need to go back to T which contains an interesting result that TTNet without using difference features overtakes the full model in the independent setting. This phenomenon suggests that difference features help to capture contextual information. Contextual information is more important for the original setting, and the model tries to capture it by attention more to the difference features. However, this does not prevail in the independent setting since contextual information is less effective and the model should attention more to the image features. This is why retraining the full model with the independent data works, because the focus of attention is adjusted during retraining. The second question is about the alternative type of difference features. We compare early and late differences. The early difference is pixel-level difference computed on raw images before input to the image encoder, while the late difference is used by TTNet and computed on encoded image vectors to become the semantic difference. In TVR (Hong et al.) (021), early difference is more effective while Table f shows the opposite result. This because TVR requires to predict property changes on synthetic data, which relies more on pixel differences. In contrast, VTT requires event-level descriptions, with more emphasis on semantic distinctions.
Analysis of MTM. We expect MTM to guide the model to reason transformation from nearby transformations. In order to validate this ability, we design to let models predict trans- formations with incomplete states, e.g mask one state of three. Specifically, we test models under two special settings. In the first setting, we randomly mask one state for all test samples. In the second setting, only providing start and end states for each sample. The re- sults shown in Figure [ We see that when there is less and less information, the performance of all models de- creases. However, TTNet has the slowest decline in perfor- Figure 6: Effect of missing states.
500
TTNet
200
give even fewer states on average by
DenseGap
mance, showing its robustness to missing states. By further comparing the results between TTNet and TTNet w/o MTM, we can conclude this robustness is contributed by the MTM strategy.
Analysis of Auxiliary Tasks. Finally, we analyze the effects of different auxiliary tasks and report the results in Table From the table, topic classification is more effective than category classi- fication, since topics are more granular than categories. Supervision with two classification tasks simultaneously improves the overall performance, e.g. 562.25 + 570.63 in terms of CIDEr.
6 CONCLUSION
This paper introduces new visual reasoning task to focus transformation reasoning, i.e. changes between every two states, named visual transformation telling. Given a series of images as states, the description of each transformation is required to represent what happened between every two adjacent images. In this way, the task could be used to test the machines' ability of transforma- tion reasoning, which is an important cognitive skill for humans, as described in Piaget's theory. To the best of our knowledge. this is the first real world application for transformation reasoning by defining transformation deseriptions as output. To facilitate the study on VTT, we build bench- mark data based on 13,547 samples from two instructional video datasets, i.e. CrossTast and COIN. After that, we design a model named TTNet, by applying three well-designed strategies into a ba- sic human-inspired transformation telling model to make it differrence-sensittee and context-aware. From the experiments, we find that the proposed strategies help VTT generate consistent transfor- mation descriptions, and thus obtain better results in terms of natural language generation metries. The empirical studies provide valuable insights for understanding VTT and the proposed model and may help design more complicated transformation reasoning tasks models in the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic Propo- sitional Image Caption Evaluation. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, Lecture Notes in Computer Science, pp. 382-399 2016.
Anton Bakhtin, Laurens der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. PHYRE: A New Benchmark for Physical Reasoning. In Advances in Neural Information Pro- cessing Systems, volume 32, 2019.
Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACI. Workshop on Intrinsic and Extrinscc Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, 2005.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BERT Pre-Training of Image Trans- formers. In International Conference on Learning Representations, 2022.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. CoPhy: Counterfac- tual Learning of Physical Dynamics. In International Conference on Learning Representations, 2020.
Magali Bovet. Piaget's Theory of Cognitive Development and Individual Differences. In Bärbel Inhelder, Harold H. Chipman, and Charles Zwingmann (eds.). Piaget and His School: A Reader in Developmental Psychology, Springer Study Edition, pp. 269-279. 1976.
Chien-Yo Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure Planning in Instructional Videos. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, volume 12356, pp. 334-350. 2020.
Boxing Chen and Colin Cherry. Systematic Comparison of Smoothing Techniques for Sentence- Level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp- 362-367, 2014.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2022.
Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. Every Picture Tells a Story: Generating Sentences from Images. In Kostas Danilidiss, Petros Maragos, and Nikos Paragios (eds.), Computer Vision ECCV 2010, Lecture Notes in Computer Science, pp. 15-29, 2010.
Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for Compositional Actions & ТЕmporal Reasoning. In International Conference on Learning Representations, 2020.
Diana Gonzalez-Rine Rico and Gibran Fuentes-Pineda. Contextualize, Show and Tell: A Neural Visual Storyteller, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The Abduction of Sherlock Holmes: Dataset for Visual Abduc- tive Reasoning, 2022.
10

---

Under review as a conference paper at ICLR 2023
Xin Hong, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. Transformation Driven Visual Reasoning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6899-69085 2021.
Drew A. Hudson and Christopher D. Manning. GQA: A New Dataset for Real- Woald Visual Rea- soning and Compositional Question Answering. In 2019 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 6693-6702, 2019.
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. DenseCap: Fully Convolutional Localization Networks for Dense Captioning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2016.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Feifei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Vi- sual Reasoning. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997, 2017.
Tachyeong Kim, Min-Oh Heo, Seonil Son, Kyoung- Wha Park, and Byoung- Tak Zhang. GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation, 2019.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A Hierarchical Approach for Generating Descriptive Image Paragraphs, 2017.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Centi-naptioning Events in Videos. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 706- 715, 2017.
Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR 2011, pp. 1601-1608, 2011.
Chris der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. Best practices for the human evaluation automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 355-368, 2019.
Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang. Visual Abductive Reasoning. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15544-15554, 2022.
Chin-Yew Lin and Eduard Hovy. Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Sunmarization Volume 4, AS '02, pp. 45-51, 2002.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hicrarchical Vision Transformer using Shifted Windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992-10002, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Con- ference on Learning Representations, 2022.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac Makarand Tapaswi, Ivan Laptev, and Josef Sivie. How To000:00 Learning Text- Video Embedding by Watching Hundred Million Narrated Video Clips. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2630- 2640, 2019.
Margaret Mitchell, Ting-Hao 'Kenneth' Huang, Francis Ferraro, and Ishan Misra (eds.). Proceed- ings of the First Workshop on Storytelling. 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust Change Captioning. In 2019 IEFF/CVF International Conference on Computer Vision (ICCV), pp. 4623 2019.
11

---

Under review as a conference paper at ICLR 2023
Jac Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visual- COMET: Reasoning about the Dynamic Context of a Still Image, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32, 2019.
Jean Piaget. The Role of Action in the Development of Thinking. In Willis F. Overton and Jeanette McCarthy Gallagher (eds.), Knowledge and Development: Volume I Advances in Re- search and Theory, pp. 17-42. 1977.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to- Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. ISSN 1533-7928.
Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, and Mubbasir Kapadia. AESOP: Ab- stract Encoding of Stories, Objects, and Pictures. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2032-2043, 2021.
Ronan Riochet, Mario Ynocente Castro, Mathicu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. IntPhys 2019 Benchmark for Visual Intuitive Physics Under- standing. IEEE Transactions on Pattern Analysis and Machine Intelligence, (409):011-5002) 2022. ISSN 1939-3539.
Tomas Soucek, Jean- nann-Batistte Alayrac, Antoine Miech, Ivan Laptev, and Josef Sivic. Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13936- 13946, 2022.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis.
[cs], 2019.
Yansong Tang, Jiwen Lu, and Jie Zhou. Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 43(9):3138-315
2021. ISSN 0162-8828, 2160-9292, 1939-3539.
Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices. Proceedings of the AAAI Conference Artificial Intelligence, 34407): 2020. ISSN 2374-3468.
Ting- Hao, Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. Visual Storytelling, 2016.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-basee image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566-4575, 2015.
Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. Translating Videos to Natural Language Using Deep Recurrent Neural Networks. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languge Technologies, pp. 1494-1504, 2015.
12

---

Under review as a conference paper at ICLR 2023
Xiang Wang, Shiwei Zhang, Zhiwu Qing. Yuanjie Shao, Changxin Gao, and Nong Sang. Self- Supervised Learning for Semi- Seni-Supervised Temporal Action Proposal. In 2021 IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 1905-1914, 2021.
Kexin Yi, Chuang Gan Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: Collision Events for Video Representation and Reasoning. In Interna- tional Conference on Learning Representations, 2020.
Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks. In 2016 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 4584-4593, 2016.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From Recognition to Cognition: Vi- sual Commonsense Reasoning. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 67 6713-6724, 2019.
Chen-Lin Zhang, Jianxin Wu, and Yin Li. ActionFormer: Localizing Moments of Actions with Transformers. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, volume 13664, pp. 492-510. 2022.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Songchun Zhu. RAVEN: A Dataset for Re- lational and Analogical Visual RiasoNing. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5312-5322, 2019.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinherger, and Yoav Artzi BERTScore: Eval- uating Text Generation with BERT. In ICLR, 2020.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, and Marcus Rohrbach. Grounded Video Description. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6571-6580, 2019.
Dimitri Zhukov, Jean-Baptiss Alayrac. Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivie. Cross-Tade Task Weakly Supervised Learning From Instructional Videos. 2019 IEFF/CV Conference Computer Vision and Pattern Recognition (CVPR), pp. 3532-3540, 2019.
13

---

Under review as a conference paper at ICLR 2023
A VTT DATASET CONSTRUCTION
In order to provide a deeper understanding of the VTT dataset, we describe how we construct VTT in detail. The whole process consists of four major parts, i.c. combine and complete annotations, extract state images, and split the dataset. Additionally, we mention some annotation information from CrossTask and COIN, which is important for understanding our decisions when building VTT, such as how to choose the start and end states.
Combine and complete annotations. Both CrossTask and COIN provide the annotations of step labels and corresponding segments. Step labels are predefined before they started annotating videos. The difference that COIN employed experts to define steps while CrossTask derived them from WikiHow, a website teaching how to do many things. With predefined steps, they find annotators to label the step categories and the corresponding segments for each video. We collect and organize these annotations in uniform format to become the basis of the VTT dataset. Apart from that, category and topic labels are used for auxiliary learning. Both CrossTask and COIN provide topic information, which is the task to solve. However, only COIN provides categories as the domain information and they are missing in the CrossTask. We manually classify all topics from CrossTask into existing categories. The full list of 12 categories and 198 topics is shown in Table
Table 5: The Categories and topics in VTT dataset. Topics marked with are from CrossTast and others belong to COIN.
Category
Topics
Nursing and (14)
Cure
Wash Dog. Use Earplugs, Use Neti Pot, Put On Hair Extensions, Use Epinephrine Auto-injector, Perform CPR, Wear Contact Lenses, Remove Blackheads With Glue, Give An Intramuscular Injection, Shave Beard, Wash Hair, Bandage Dog Paw, Draw Blood, Bandage Head
Pets Fruit (7)
Tree, Transplant, Graft, Cut Grape Fruit, Cut Mango, Cantaloupe, Sow
Furniture and Deco- ration (15)
Install Shower Head, Install Ceramic Tile, Install Air Conditioner. Install Curtain, Lubricate A Lock, Replace Door Knob, Install Wood Flooring, Install Closestool, Assemble Cabinel. Assemble Sofa, Replace Faucet, Replace Toilet Seat, Assemble Bed, Build Simple Flonting Shelves", Assemble Office Chair
Leisure and Porfor- mance (17)
Make Paper Wind Mill, Perform Vanishing Glass Trick, Raise Flng, Play Frishae With A Dog. Make Chinese Lantern. Carve Pumpkin, Change Guitar Strings, Perform Paper To Money Trick. Pitch A Tent. Open Champagne Bottle, Blow Sugar. Make Puper Easter Buskets, Cul And Restore Rupe Trick, Do Lino Printing, Replace Drum- head, Prepure Sumi Ink, Prepare Canvas
Dish (23)
Make Kimchi Fried Ricef, Cook Omelet, Make Sandwich, Grill Steak", Clean Fish, Use Toaster, Clean Shrimp. Make Burger, Make French Tonst", Wmap Zangzi, Make French Simwberry Cake", Make Pickles, Boil Noodles, Make Bread and Butter Pickles", Make Kerala Fish Curry", Make Lamb Kehab, Make French Fries, Use Rice Cooker To Cock Rice, Make Pizza. Make Youtino, Make Salmon, Smash Garlic, Make Pancakes*
Electrical Appliance (20)
Replace Graphies Card, Replace Light Socket, Replace Electrical Ourlet, Replace Memory Chip, Use Soy Milk Maker, Change Toner Carrrigge, Replace Laptog Screen, Replace Refrigerator Water Filter, Use Vending Machine, Replace Filter For Air Purifier, Replace Hard Disk, Replace Blade Of A Saw, Refill Cartridge, Clean Laptop Keyboard, Are Weld, Install Coiling Fan, Replace A Bulb, Pasts Screen Protector On Pad, Assemble Desktop PC, Use Sewing Machine
Science and (15)
Craft
Prepare Standard Solution, Make Flower Press, Use Volumetric Pipetre, Hang Wallpaper, Make Candle, Make Scap. Use Triple Beam Balance, Make Flower Crown, Use Volumetric Flask, Paste Car Sticker, Make Slime With Glue, Make Paper Dice, Wrap Gif Box, Set Up Hamster Cage, Use Analytical Balance
Drink and Snack (20)
Make Meringue Make Salad, Make Lemonade*, Make Taco Salad", Make Tea. Make Chocolate. Make a Latte*, Make Homerndde Ice Crean, Make Jello Shots Make Cuffee, Make Cocktail, Make Cookie, Make Irish Collee', Roast Chestnut, Make Banana Ice Cream", Make Orange Juice, Make Matcha Tea Make Sugar Costed Haws, Make Strawberry Smoothie, Make Hummus
Vehicle (21)
Change Bike Chain. Replace Car Fuse, Replace Rearview Mirror Glass, Tie Boat To Dock, Pump Up Bicycle Tire, Change Car Tire, Use Jack, Remove Seratches From Windshield. Jack Up Carf, Change Bike Tires, Install Install Bicycle Rack, Replace Tyre Valve Stem, Change a
License Plate Frame, Fuel Car, Replace A Wiper Head, Head, Tire", Patch Bike Inner Tube, Polish Car, Replace Car Window, Add Oil to Your Car", Park Parallel
Housework (15)
Put Quilt Caver, Clean Bathuuh Wash Dish, Clean Leather Seat, Pack Sleeping Bag, Wooden Floor, Clean Toilet, Iron Clothes. Drill Hole, Remove Crayon From Walls, Clean Hamster Cage. Make Bed, Unclog Sink With Baking Soda, Clean Rusty Poi, Clean Cement Floor
Sport (10)
Practise Karate, Wear Shin Guards. Practise Triple Jump, Throw Hammer. Play Curling, Practise Skiing Aerials, Practise Pole Vauli, Attend B Skills Challenge, Glue Ping Pong Rubber, Practise Weight Lin
Gadgets (21)
Open A Lock With Paperclips, Replace Mobile Screen Protector, Load Grease Gun, Change Mobile Phone Battery, Replace Sewing Machine Needle, Change Battery of Watch, Replace SIM Card, Resize Watch Band, Replace CD Drive With SSD Refill Mechanical Pencils, Make Wireless Farhuds, Refill Fountain Pen, Refill A Lighter, Rewrap Battery, Replace Battery On Key To Car, Fix Laptop Screen Scratches, Operate Fire Extinguisher, Replace Battery On TV Control, Use Tapping Gun, Retill A Stapler, Make RJ45 Cable
14

---

Under review as a conference paper at ICLR 2023
alorc)
comorree
2
in
wm
Figure 7: The sample distribution of all the topies in VTT.
Table 6: Statistics of the VTT dataset.
Categories
Topics
Samples
States
Transformations
Unique Transformations
CrossTask COIN
4
18
1825
12860
11035
105
180
11722
56169
44447
749
Train
12
198
10759
54716
43957
853
Val
198
1352
6974
5622
812
Test
12
198
1436
7339
5903
806
Total
198
13547
69029
55482
853
Extract State Images. When building the VTT dataset from videos, one important question is how to select images from video frames to construct states. Ideally, the starting state should be an image of the moment just before the transformation starts, and the ending state should be an image of the moment when the transformation ends. Based on CrossTask's 's precise temporal segment annotations and COIN's precise boundaries of segments annotated with three rounds of refinement, we choose to directly use the start end frame of the segments the start or end states. As shown in Figure [T] frames in the boundary of transformations are selected as states. After extracting, we randomly sample 200 examples check the quality of states and only find a few flaws such as black screens or text transitions. This is mainly caused by careless annotators. As a whole, transformations are predictable from extracted states for humans but sometimes can be challenging. For those difficult samples, it is not enough to reason the transformations by only considering the nearby before and after image states. Our motivation for the masked transformation model is from this observation that transformation should be reasoned in the overall context. Another problem someone may care about is whether there exist multiple transformations between two adjacent states. Since the annotation target of CrossTask and COIN is to segment all key steps in video, the probability of multiple transformations between states is small.
Split Dataset. Finally, we split the data randomly into Train / Val / Test sets with samples of 10759 / 1352/ / 1436 in the level of topic. The detailed topic distribution is shown in Figure 7 From the figure, can that about half of the topics have over 100 samples. We also summarize the main statistics of the VTT dataset in T
Discussions. Currently, the size of the VTT dataset is small compared with large video datasets such as H0wTo100M (Miech et al. 2019), which limits the range of transformations covered by the dataset. The biggest restriction is the high cost of labeling steps/transormationss with descrip-
15

---

Under review as a conference paper at ICLR 2023
Table
The VTT human evaluation guidelines.
Metric
Score
Criteria
5
All sentences are fluent.
4
Most sentences fluent with only few flaws.
Fluency
3
About half the sentences are fluent.
2
Most of the sentences are difficult to read, only a few are okay. All sentences are hard to read.
1
5
The descriptions are all related to the corresponding before and after images. A few descriptions are slightly irrelevant, e.g. the description is related to the underlying topic but cannot be clearly inferred from the images. Many descriptions are slightly irrelevant few descriptions irrelevant, e.g. the action or target object mentioned the transformation does not match the images.
4
3
Relevance
2
Many descriptions are irrelevant.
1
Most descriptions irrelevant, vant, e-g. transformation unrelated the underlying topic of the images.
descriptions are completely irrele-
5
The underlying logic of the descriptions is consistent with common sense. The overall logic consistent with common sense, with minor flaws. There are few obvious logical problems between the deseriptions, e.g. un- resonable repeating transformations.
3
Logical Soundness
2
There some obvious logical problems, e.g. the order of transformations obviously not line with common sense.
Logic cannot be judged hecause of the extremely poor fluency or poor rele- vance leading to overall logic inconsistent with the underlying topic.
tions and temporal boundaries. One possible way to reduce this cost is first using pretrained step localization models (Wang et al.] 2021 Zhang et al.) 2022) or action and object state-recognition models (Soucek et al. (2022) to propose coarse steps/transformations and then refine the results with human annotators. With more data to cover large transformations, machines can learn more pow- erful transformation reasoning models, which have potential enhancement value for many tasks, such as procedure planning, visual storytelling, explaining the phenomenon in life with events, etc. Another issue the precision of the boundary of existing step segments in CrossTask and COIN. For future construction of larger datasets, we suggest strategy for possible refinement by applying object state-recognition models (Soucek et al. 2022).
B EVALUATION DETAILS
B. AUTOMATIC EVALUATION
We introduce some details that not included in the main content of the paper when computing automatic metrics. Firstly, we follow the smooth strategy introduced by Chen & Cherry (2014) when computing BLEU@4 provide more accurate results. This is because descriptions in VTT are usually short, the original BLEU@ @4 gives a zero score for short texts. In addition, BERT-Score rescaled with the pre-computed baseline (Zhang et al., (2020) to have more meaningful score range. BLEU@A 4 is computed using NLTK package CIDEr, METEOR, ROUGE, and SPICE are computed with the code from coco-caption BERT-Score is computed by using the official code provided by the authors.
B.2 HUMAN EVALUATION
Automatic evaluation metrics have limitations on reflecting the quality of generated text mainly because they are uninterpretable and do not correlate with human evaluations (van der Lee et al. 2019). In the VTT task, we consider three levels of text quality, evaluated by people. The first
16

---

Under review as a conference paper at ICLR 2023
Human Eveluation
Calogey
Stark
AUDO
Teeeth
AUD
AUTO
D-1: remove the old nerview miror
1-2 nenutal the rearvien minoe
Famm
Teloventt
Lngeod Spentree
Cannet
Saarii
Figure The web interface of human evaluation on VTT.
Table 8: Implementations details of baseline models and TTNet.
Model
Image Encoder Context Encoder
Transformation Decoder
Params
CST
Inception V3
LSTM
LSTM
379M
CST
CLIP (ViT-L/14)
LSTM
LSTM
661M
GLACNet
RcsNet152
bi-LSTM
LSTM
128M
GLACNet
CLIP (ViT-L/14)
bi-LSTM
LSTM
373M
DenseCap*
CLIP (ViT-L/14)
Attention
LSTM
361M
TTNe
CLIP (ViT-L/14)
Transformer
Transformer
368M
TTNet
CLIP (ViT-L/14)
Transformer
Transformer
368M
level considers only the fluency of the text itself. The second level considers the relevance of cach individual transformation description to the topic and to the images before and after. The third level considers the logical consistency between transformation descriptions. The assessment uses the 5-point Likert scale and follows the guidelines below in Table
We asked 25 people to evaluate the major baseline models' results shown in Table [ subset of samples is selected for human evaluation by randomly sampling testing samples, T sample from each topic and 2 extra samples, resulting in 200 samples total. All annotators are asked to read give their scores. During human evaluation, the annotators can see
and follow the guidelines the images in addition to the category and the topic as a reference. The web interface for human evaluation is shown in Figure Each sample result from each model is evaluated by least people. Our code of the evaluation webpage will also be released along with the VTT source code.
C IMPLEMENTATION DETAILS
During training, we apply commonly used image augmentation tricks, including randomly cropping images into 224 x 224 patches. and random flipping. The overall architectures of all baseline mod- els are shown in Table 8 We re-implemented CST and GLACNet following the original paper and their released source code F We didn't find the code of DenseCap and followed their paper to implement the final model. The image encoder of DenseCap is replaced with CLIP since the orig- inal model targets video descriptions and uses video encoders. When implementing TTNet, in the
https://gi http:///iiimmmm///aaaaaeeee ACNet
17

---

that the orig- Under review as a conference paper at ICLR 2023
Table Results of different image encoders.
Image Encoder
Params
Acc
@@4
C
BS
Inception 3 ( Szegedy al.
2016
23M
77.44
44.88
404.85
61.75
ImageNet Pretrained
5esNet152 le
2016
59M
82.82
50.71
464.01
67.40
ViT-L Dosovitskiy
al
2022
304M
85.84
58.26
540.46
73.59
Swin-L Liu ct al. 2021
196M
86.32
57.36
531.51
73.03
BEiT-L
Bao ct al.
2022
306M
87.48
41,57
370.00
58,80
RN50
39M
73.30
53.35
491.80
69.79
Image text Pretrained
RNIO1
57M
75.70
53.78
495.30
70.08
ViT-
88M
76.10
55.21
510.08
71.27
VÝT-B/16
86M
80.20
57.73
534.92
73.37
ViT-L/14
304M
83.90
61.22
570.63
76.25
image encoder part, the default CLIP image encoder is ViT-L/14. Image encoders in all models pretrained and fixed during training. Further details about the image encoder and the comparison can be found in Section D Transformer-based context encoder consists of two transformer en- 6 Simplified relative positional encoding (Raffel et al. 2020 applied in all transformer layers. In the transformation decoder part, we directly borrow CLIP's s tokenizer and their vocabulary list. Each transformation description generated separately with a shared two-layer transformer decoder. The idea of adding transformation representations into word embeddings is inspired by GLACNet (Kim et al. 2019) and we empirically found this way improves lot on language influence compared with using the representation as the start token. Like the context encoder, simplified relative positional encoding also used in the transformation decoder. We sample text with top-k top-p filtering with k = 100 and p 0.9. The dimension of intermediate vectors is uniformed to be 512, including state rep- resentations, transformation representations, and word embeddings. In the training loss part, the adjustment factor a for Ceategory is is Adam W (Loshchilov & Hutter] (022). with the learning rate first warming up to le-4 in the first 2000 steps and then gradually decreasing O. All models implemented with PyTorch (Paszke et al. 2019) and trained on one single Tesla A100 80G GPU card with 50 epochs. The code for training and inference will be released publicly.
coder layers. The implementation of the transformer is based on x-transfomer
be 0.025 and 8 for Lioiie 0.I. The optimizer we used
D MODEL OPTIMIZATION
In this section, we show more detailed information about the selection of the image encoders and the hyperparameters of masked transformation model when optimizing our model.
D.1 SELECTION OF IMAGE ENCODERS
Image encoding quality is the basis for subsequent reasoning and description of the model, and thus greatly affects the overall performance of the model. From Table [ we inal version of CST and GLACNet, with Inception V3 and ResNet as image encoders accordingly perform worse than CST* and GLACNet* indicating the choice of image encoder matters. We conduct a more detailed analysis of the image encoder by testing 10 state-of-the-art image encoders, were pretrained on ImageNet and 5 are CLIP models pretrained on large-scale image-text data. In the table, we show their parameter size, ImageNet top-1 accuracy, and performance on the VTT dataset. We can see that when the parameter sizes are similar, models pre-trained on image and text data perform better than that pre-trained only on image data, e.g. ViT-L/14 vs. VIT-L. This consistent with the existing understanding that CLIP encodes more semantic information. In ad- dition to training data, factors that affect model performance include model size, patch size used
httpps:///githbb.ccm///tttt trant formers
Model weights and top-1 accuracy on ImageNet ImageNet pretrained models are from: https: modela
github. cww/ rwi rww http://iiggnnaaaatroochhhh-c immde---e-im
Prettriieed weights of CLIP models are from https:// /ttgtttttttbbbtt wwm wwwp://na////c/cc//////.oaaaaaaa i/C.IP and top-1 accu-
racy ImageNet from Table 10 of the original paper.
18

---

Under review as a conference paper at ICLR 2023
in vision transformers, and training strategies. For example, CLIP models have parameters performs better. While the parameter size between ViT-B/16 and ViT-B/32 are similar, ViT-B/16 encodes finer image has smaller patch size resulting in better image representation. BEiT-L has the highest accuracy on ImageNet but performs the worst among all models. Our explanation is that BEiT has learned enough image pattern information, but there is defect in the capture of semantic information.
D.2 HYPERPARAMETERS OF MASKED TRANSFORMATION MODEL
There are two hyperparameters masked transformation model, i.e. the mask ratio and the sample ratio. The mask ratio is similar with BERT's 's mask ratio (Devlin et al.] 2019). that is, the percentage of state representations and difference features that are replaced with zero. We compare the mask ratio from 0%-30% and find 15% works the best (Table TO). which is in line with BERT's find- ing. Another hyperparameter is the sample ratio. The motivation is to tackle the inconsistent issue between training and inference, that is, features samples have opportunities to completely bypass the mask strategy during training. The sample ra- tio is the probability that the sample will accept the mask strategy. From Table 11 50% probability is the best, better than the strategy of masking all samples used in BERT.
masked during inference. To fill this gap,
Table 10: Ablation on the mask ratio.
Table 11: Ablation on sample ratio.
Mask Ratio
B@4
C
BS
Sample Ratio
B @@4
C
BS
0%
60.38
562.83
75.72
5%
60.93
567.92
76.11
0%
60.38
562.83
75.72
10%
61.02
568.71
76.13
25%
60.39
562.15
75.63
15%
61.22
570.63
76.25
50%
61.22
570.63
76.25
20%
61.07
568.99
76.21
75%
60.96
567.99
76.00
25%
61.16
570.18
76.35
100%
60.95
568.18
76.10
30%
60.72
565.43
75.94
E GENERALIZATION ANALYSIS
There are two levels of generalization that should considered in visual transformation telling. At the level of one single transformation, the question is whether machines able to generalize to different language compositions, i.c. new action-target combinations that do not exist the training set. At the higher level of multiple transformations, the question becomes to be whether machines can generalize to different transformation combinations, such as different numbers and orders of transformations the problems and see how well our models perform.
topic. In this section, we diseuss these kinds of generalization
Count
40
0
put
Info
remove 6
out
cut
with pour new take cover
in
up
of
di
cleen
head water paper screv
back
apply
wipe
place
Insert
from
open
glue
Ingrediens niss
detergent hair
prepare
press
wall
battery
close
hamster
gun
pull
wash
use
check
Figure 9: Top 50 words (exclude the and and) appear multiple times in all 853 unique descriptions.
E.1 LANGUAGE COMPOSITION
Language composition Johnson al. (217)) has been widely discussed in the community of visual reasoning that is very useful if machines are able to generalize to unseen language combinations with seen words. In the VTT dataset, it common that different transformation descriptions share
19

---

Under review as a conference paper at ICLR 2023
DemoeCap:
GLACNet: 1. Pour espresso.
TINet: 1. Pour alcobol.
Groundrruti:
Pour expresso.
Pour coffee into glas. glass, Pour cream.
2. espresso.
2. Pour whipped
2. Pour whipped cream.
whipped cream.
Figure 10: Models failed to compose unseen transformations with words.
Table 12: Statistics of unique transformation combinations. Val Only and Test Only count combina- tions that do not exist in the train split.
Topics
Train
Val
Val Only
Test
Test Only
All
Dish
922
154
95
158
103
1118
Drink Snack
804
968
Electrical Appliance
472
99
34
100
41
546
Furniture and Decoration
481
83
51
93
45
577
Gadgets
414
82
33
84
40
487
Housework
406
48
69
40
494
Leisure and Performance
408
85
39
85
39
485
Nursing and Care
316
66
29
66
27
371
Pets and Fruit
208
42
20
45
22
249
Science and Craft
395
82
37
87
42
473
Sport
167
38
Vehicle
541
110
56
120
50
643
Total
5534
1054
1091
559
6618
the same words, especially verbs, and nouns. We count the top 50 words, excluding the and and, from all 853 unique transformation descriptions in the VTT dataset and results are shown in Figure
With such a large proportion of shared vocabulary. the natural language generation of the transfor- mation is more valuable than the classification of the transformation, since models would have more chances to learn common patterns from transformations with shared words. To investigate whether models learned the VTT have the generalization ability of language composition, on one manually annotated by us, which comes from a related task in CrossTask, i.c. Make Bicerin. The topic contains transformation descriptions that not included in our training but com- posed without new words. Figure TO show the results three models. Unfortunately, all models failed to generate these new descriptions from the existing words but with descriptions that match the states as closely as possible but exist in the list of transformations from the training set. We believe there
test models
main reasons. Firstly, the small size of unique transformations limits models gain language composition ability from data. Secondly, current models lack effective design for this generalization ability. Therefore, enlarging the dataset to cover more diverse transformations and designing models with stronger generalization ability will be important future directions for visual transformation telling.
E.2 TRANSFORMATION COMBINATION
The states and transformations under the same topic can be very different, one important reason is the different combinations key transformations. For example, add seasoning can the step after the water is boiling, the noodles poured, or both, depending entirely the preference of the person cooking the noodles. This freedom leads to rich transformation combinations even with few key single transformations under each topic. We count the unique combinations in the VTT dataset
20

---

Under review as a conference paper at ICLR 2023
Table 13:
Performance on all test samples and subsets of test samples whose transformation combi- nations are shared and non-shared with the training set.
All
Share
Non-share
Model
Fl
C
Fl
LS
C
Re
LS
CST
0.85
2.04
3.16
2.96
0.99
1.95
3.22
3.00
0.73
2.17
3.08
2.91
GLACNet*
5.08
4.75
3.82
3.78
6.21
4.80
3.90
3.91
4.11
4.69
3.70
3.59
DenseCap
4.40
4.74
3.67
3.59
5.16
4.72
3.66
3.61
3.75
4.76
3.68
3.57
TTNetase
5.15
4.79
4.04
3.95
6.02
4.80
4.08
4.00
4.40
4.77
3.99
3.88
TTNet
5.71
4.78
4.10
4.11
7.01
4.81
4.23
4.29
4.59
4.74
3.93
3.86
that more than half combinations in the test set do not exist in the training set (559 test only v.s. 1091 total), which means models need a strong generalization ability predict these unseen transformation combinations well.
shown in Table 12 From the table,
To investigate how models perform on these non-share combinations, we separately computed the performance of each model on three subsets of test samples, including all test samples, test samples that have shared combinations with the training set, and test samples that have combinations not included in the training set. From Table 13 we can see that models perform much worse on non- share combinations than shared combinations, e-g. the logical soundness of TTNett is decreased from 4.29 to 3.86. This significant performance degradation suggests that the combination gemeralization is challenging for existing models. Furthermore, TTNet performs indeed much better on shared combinations than TTNetaae (logical soundness 4.29 v.s. 4.00), but situation not true for non-shared combinations (logical soundness is 3.86 v.s. 3.88), which means the proposed three strategies contribute on learning existing combinations on the training set but has little or even negative effects model's combination generalization ability.
F MORE QUALITATIVE RESULTS.
We show more results in Figure and Figure 12 Figure 12 shows several hard cases that TTNet fails to reason and describe. We point out three potential directions to improve the TTNet. The first relates the image recognition ability. From the first case, TTNet recognizes the tent platfond which is wrong. This error might respond to the image encoder that fails to distinguish these objects. Therefore, it may lead here is that the model needs to identify subtle differences between states and determine specific transformations based on context. The last case is oft-of-domain cases, that is, the test samples are quite different from the training data.
better result by using a more powerful image encoder. The major point
21

---

Under review as a conference paper at ICLR 2023
Demse Cap:
GLACNet:
TTNet:
Grouddruth
patterns.
puper.
pictures
2. Out the now materialls.
2. Fold bent the puper.
Fold or bent the puper.
bent the paper.
3. Out the packing piperr.
3. Fold or bent the puper.
Cun the raw materials,
Cut the raw miterials.
4. Out the materials.
4. Nail down stick together.
Nail down stick logether.
Nail down sück logether.
together.
logether.
down lugether.
logether.
Close Valve
GLACNet:
DenseCap:
1. Screw off the valve and open the
TTNe::
Groudtruth:
Remove
Screw
open
cup open
Remove
Pump up tire.
valve.
valve:
3. Pamp up the tre.
3. Pump up the tire.
Install the air nozole.
Install the air nosole.
4. Pamp the tire.
4. Tighten the valvee and screw the
Pump the tire.
Pump to
Remove
Remove norzle.
CLACNet
Grouddruuth:
two jamps.
Bogin up
Regin up.
Regin up.
2. Do the first two jampe.
2. Do the third jump.
Do tie fins jumps.
Do the linst two jumps.
3. Do the third
3. Do the third jump.
Do the thind jump.
Do the thind jump.
DenseC Cape
GLACNet:
TTNet:
Grouddruuth:
Dig proper
Dig pit proper size.
pot with groper
with proper size
pit.
2. the tree into the pir.
Fill the tree into the pit.
Fill the tree into the pit.
3. Fill the uree into the pit.
3. Fill the tree into the put.
Hill the pit with soms soil.
Fill the pit with some soil.
4. Fill the tee into
Pour
the
the
Fill
pit
pit.
6. Fill the pit with some woil.
6. Pour water to the aree.
6. Pour water to the tree.
Pour water the tree.
Figure 11: More qualitative results in the VTT test data.
22

---

Under review as a conference paper at ICLR 2023
Dense the
GLACNet:
TTNet:
Gruundruthh:
eventy. 2. up the platfond.
cushion evenly. Set up the platfood.
Lay evenly. Sei up the brackts. Fix the ground mail.
Sot up the platfond.
3. Fix the ground nail.
3. Set up the platfond.
Fix the ground mail.
Caps
GLACNet:
Groundtruth:
Kise the sky..
1. Rise to the sky.
1. Ski down from the hill.
Ski down from the hill.
2. Rise to the sky.
2. Rise to the sky.
2. Ski down from the hill.
Ski up from the hill.
3. Rise
Rise the sky.
the
sky.
Rise
5. Rise the sky.
5. Rise to sky.
down from the
Ski up from the hill.
6. Rise to the sky.
6. Rise to the sky.
Rise to the sky.
Rise the sky.
Kise sky.
7. Rise
down from
Fall ground.
up
9. Mise to the sky.
9. Kise to the sky.
St. Rise to the sky.
Rise the sky.
Dense Cape
GLACNet:
TENe::
Grouddrrcth:
1. Wipe the plue to layer.
1. Wipe up the face.
1. Disinfect the anjecting place.
1. Add some water the vessel. Grind roundly and evenly. Grind roundly and evenly.
2. Wipe the pluc to layer.
2. Wipe the glue to layer.
2. Disunfect the injecting place.
up
neadle
Figure 12: Bad cases on the VTT dataset.
23


Reviews:
[{"review_id": "eim9MXH1Uw", "reviewer": "Reviewer_ipHd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new task as well as a dataset that was collected from instructional video datasets. Apart from this, they also have a model proposed for this task. They analyze the proposed data and achieve good performance with their model.", "summary_of_the_review": "Generating descriptions for video has been a long time, and this paper proposes a new task called Visual Transformation Telling. This task is new and interesting, and the baseline models reported in this paper cover most of the recent frameworks. However, considering the dataset is their major contribution, I still think this paper still has some room to improve.", "strength_and_weaknesses": "The difference in state representations or difference between visual features has been widely used in video understanding tasks. The difference in sensitive encoding in this paper is not new since lots of video captioning also has a close idea.\n\nThe masking mechanism in this paper is not to predict the specific ground truth, such as words or attributes; instead, they mask the input features and predict the representation. However, this paper lacks details, and the authors should describe more about the masking and loss function design.\n\n\nAlso, since the feature keeps changing, masking in the feature space makes the MTM harder. Instead of masking 15%, did the authors try other masking ratios?\n\nIn their setting, they miss the SPICE score, which is also widely used in the image captioning domain. Also, considering the gap between machine evaluation and human judgment, it is necessary to have humans evaluate the predictions captions. Since the visual description in this paper is much longer than image captioning, simply replying to language rules may not well reflect the real performance of predictions.\n\nCLIP model was pretrained on image-text data, while the data in this paper is video data; considering the gap between images and videos, what are the effects of the pretrained model for this setting? Did the authors consider some video-text pretrained model for this task?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X0faCx3Znm0", "reviewer": "Reviewer_tWf2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\n- The paper proposes a new task called Visual Transformation Telling (VTT), which is defined as predicting the intermediate steps ('steps' are referred to as ‘transformations’ in the paper) in forms of natural language given the start and end state of each step. \n- Two existing instructional video understanding datasets, CrossTask and COIN, are combined to form the VTT benchmark. \n- A Transformation Telling Net (TTNet), consisting of the CLIP image encoder, a transformer based context encoder and a transformer based decoder for text generation, is proposed to tackle the VTT benchmark problem. \n- Three strategies are further proposed to address the challenges of VTT and thus allow the TTNet to be difference sensitive, context aware and context consistent. \n- In-depth experiments demonstrate the effectiveness of the proposed model and strategies. \n", "summary_of_the_review": "The paper is really well-written, and the performance of the proposed methods is well-supported by experiments. However, the authors may need to further justify the VTT task and how they construct the VTT benchmark. \n\n*After rebuttal*: I appreciate the new analysis added by the authors which to some extent addressed my major concerns on whether the current VTT benchmark really requires generalizability from models and whether the VTT benchmark was reasonably constructed. Therefore I raised the score from 5 to 6. However, there are indeed several aspects of this manuscript that can be improved, e.g., technical novelty of the proposed model - as the authors admitted, the generalizability of the proposed model is not satisfactory. Overall, I think the idea/direction of the paper is worth pursuing, so I encourage the authors to continue improving the paper no matter what the final outcome is.\n\n", "strength_and_weaknesses": "Strength:\n1. Experiments and analysis are quite thorough with ablation studies clearly demonstrating the impact of each proposed component of the paper.\n2. The proposed model TTNet significantly outperforms the baselines and the proposed strategies are intuitive, reasonable and effective.\n3. Interesting qualitative results were presented. \n4. Failure cases were shown and discussed.\n5. The paper is well-written and easy to follow. \n\nWeakness:\n1. The necessity or importance of the proposed new VTT task is not clear. The concerns are:\n-  There are similar tasks such as Procedure Planning [1], Walkthrough Planning [1], and Assistant Writer [2] which is an abstract visual storytelling task. \n    * Comparison between the VTT task and these tasks should be discussed. \n    * The proposed VTT task seems to be a simplified version of the Procedure Planning task. Procedure Planning requires a model to predict the intermediate steps given only the start and end state of a *video*, whereas the VTT task sort of asks a model to predict the step given the start and end state of each *step*. Therefore, Procedure Planning seems to be more complex than VTT. In addition, the application-wise impact of the Procedure Planning task is quite obvious - an agent that is learned to perform Procedure Planning basically learned how to achieve a certain goal/task through some ordered steps. The application-wise importance of the VTT task is currently not clear to me. \n2. Whether the current way of constructing the VTT benchmark is sensible is not clear. The reasons are:\n    * According to the paper, the first frame of the first transformation (i.e., step) is used as the first state and the last frame of all transformations are used as the remaining states of the video. However, can the start frame and the end frame really display the start state and end state of a step/transformation? Based on my experience with the CrossTask and COIN datasets, very often that the first frame and the end frame of a step cannot truly show the start state and end state of a step. For example, unlike what’s shown in Figure 8, the last frame of the step “Dig a pit with proper size” could be *not* actually displaying the digged pit - the outcome of the transformation, but rather to be a frame just showing the narrator, some tool, etc. There are existing works like \"Look for the Change\" [3] that aims to find out which 3 frames in the video correspond to the time depicting the object initial state, the state-modifying action and the object end state. Using the method in [3] or designing a method to identify the frames that truly showcase the start and end state of a step, in my opinion, is more sensible.  Otherwise, I’m concerned about how clean the start and end states are in the current VTT benchmark, and how this kind of noise would affect the benchmark. \n    * Does the current VTT benchmark really require sophisticated generalizability and reasoning skills from a model? How challenging is the benchmark really? Right now the train/val/test splits of the benchmark were randomly splitted. It is not really clear whether there are any types of biases in the current benchmark, and whether a model can leverage that potential bias to bypass reasoning skills (this is especially important because the authors claimed the proposed benchmark is a reasoning benchmark). Also because the dataset splits were randomly generated, it is not clear to me if the natural language generation of the transformation is necessary, or how important it is for the TTNet to have the natural language generation capability. Since the TTNEt has the natural language generation capability, this means that TTNet should be able to generate unseen verb-noun combinations, unseen order of steps for seen or unseen tasks. After checking the qualitative results, I’m curious about whether the ordered list of transformation descriptions appears in training and whether it is possible that the model simply learns to memorize these descriptions. If the set of transformation descriptions is the same in training and testing, then maybe there is no need to define the VTT task to output the natural language of the transformations; using the step classes like the traditional approaches would be sufficient. \n    * One possible improvement to the current VTT benchmark might be providing annotations for a small fraction of the videos from the “Related Task” of the CrossTask dataset and use these videos as testing videos (I’m assuming only videos from the “Primary Task” of the CrossTask dataset were used to form the VTT benchmark since only those videos were provided temporal step boundary annotations). In this way, the VTT training data will have a step transformation like “pour water”, and a step transformation like “pour coffee into glass” will be absent in training but present in testing. Such a testing set would allow one to demonstrate the power of the natural language generation complement. \n\n[1] Chang, Chien-Yi, et al. \"Procedure planning in instructional videos.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n[2] Ravi, Hareesh, et al. \"AESOP: Abstract Encoding of Stories, Objects, and Pictures.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[3] Souček, Tomáš, et al. \"Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CXqGxmMuLR", "reviewer": "Reviewer_NhJc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new visual reasoning task, Visual Transformation Telling (VTT), to evaluate models' understanding on action-state associations. The authors leveraged instructional videos in existing datasets to form image sequences and ask models to predict the corresponding transformation text in a captioning manner. The authors also proposed an end-to-end transformer-based model for solving this task and show compelling performance results with captioning baselines.", "summary_of_the_review": "This paper proposes an interesting and potentially significant task, visual transformation telling, for vision-language/video-language learning. However, given the current status of the paper, there are several unclear or unjustified aspects of the paper that still need to be clarified by the authors or considered in future revisions (see weakness). The authors might also want to connect with other works (e.g. text-guided video generation, visual dynamics learning) to better address the significance of their model. Therefore, I'm recommending a reject for this paper and hope the authors could improve in future revisions.", "strength_and_weaknesses": "[+] The proposed visual transformation telling task, sharing the same spirit with text-guided visual prediction or dynamics learning, studies important problems including action grounding and world dynamics learning. However, evaluating the description of transformation can be a more direct evaluation compared with image generation metrics since it could be made more structuralized in a quantitative evaluation setting. Therefore I believe, this task could be a good evaluation approach (or checklist) to test models' capabilities on fine-grained world dynamics learning and activity understanding.\n\n[+] The authors provided sufficient ablative studies showing the limitation and strengths of current models and also their proposed TTNet. This could provide new insights into the model design (e.g. masking for autoregressive decoding), incorporating state difference features as context for better transformation learning, etc.\n\n[-] One key concern of this paper also lies in the task design. With the authors evaluating the final transformation as a captioning task, I do not feel that the clarity of text for a direct evaluation has been fully utilized. More specifically, the authors argued that the ultimate goal of the transformation-telling task is to provide a logically consistent explanation for a sequence of observations (i.e. images), however, I do not see any evaluation regarding this part. Can the captioning scores reflect the quality of transformation telling generated by the models? or to what extent should the model elaborate to have a good enough transformation description (e.g., should it be a simple action verb? verb+noun phrase? phrase with adverbs, or what?). Therefore, I do feel that the current task does not suffice as a thorough evaluation benchmark for fine-grained action grounding or action-state association understanding, limiting its significance.\n\n[-] Next, I do have questions regarding the data. The authors select start and end images as states and chain all such states into an image sequence to form the input data for transformation telling given an instructional video. However, I do feel that there are concepts that can not be simply represented by a single image (e.g. reflected by temporal patterns); then did the authors check if there exists (or could be visually distinguished) transformation between each two images so that the transformation could be reasoned out? As I believe the current data sources CrowdTask and COIN contain various instructional activities, I do feel that this could potentially be a problem. Following the same though, several end states could be the result of several previous steps and not only a single step (e.g., soup being red not only because of adding tomato but also adding ketchup), the transformation might not necessarily follow a chain structure, causing ambiguous states for visual transformation telling. These all contribute to the concerns of task quality.\n\n[-] Finally, despite the good performance of the proposed TTNet, I feel that its significance is somewhat limited as there is no specific representation bound to the transformation and therefore incapable of being used as a transformation representation for forward dynamics simulation. The authors could potentially connect with text-guided generation to see if this loop from transformation to text and text to transformation could be closed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject