Hopper: Multi-hop Transformer for Spatiotemporal Reasoning
Authors: Honglu Zhou
Asim Kadav
Farley Lai
Alexandru Niculescu-Mizil
Martin Renqiang Min
Mubbasir Kapadia
Hans Peter Graf
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-hop Reasoning
Object Permanence
Spatiotemporal Understanding
Video Recognition
Transformer

Abstract: This paper considers the problem of spatiotemporal object-centric reasoning in videos. Central to our approach is the notion of object permanence, i.e., the ability to reason about the location of objects as they move through the video while being occluded, contained or carried by other objects. Existing deep learning based approaches often suffer from spatiotemporal biases when applied to video reasoning problems. We propose Hopper, which uses a Multi-hop Transformer for reasoning object permanence in videos. Given a video and a localization query, Hopper reasons over image and object tracks to automatically hop over critical frames in an iterative fashion to predict the final position of the object of interest. We demonstrate the effectiveness of using a contrastive loss to reduce spatiotemporal biases. We evaluate over CATER dataset and find that Hopper achieves 73.2% Top-1 accuracy using just 1 FPS by hopping through just a few critical frames. We also demonstrate Hopper can perform long-term reasoning by building a CATER-h dataset that requires multi-step reasoning to localize objects of interest correctly.

Published as a conference paper at ICLR 2021
HOPPER: MULTI-HOP TRANSFORMER FOR SPATIOTEMPORAL REASONING
Honglu Zhou'; :Asim Kadav?, Farley Lai Alexandru Niculescu-Mizill, Martin Renqiang Min?, Mubbasir Kapadia', Hans Peter Graf Department of Computer Science, Rutgers University, Piscataway, NJ, USA 2 NEC Laboratories America, Inc., San Jose, CA, USA (h2289, mk1 3 (cs. rutgerr.......... edu edu a tasim, im, farley ai, alex, rengiang, hpg @nee lase-labs..ee com
ABSTRACT
This paper considers the problem of spatiotemporal object-centric reasoning videos. Central our approach is the notion of object permanence, i.e., the ability to reason about the location of objects as they move through the video while being occluded, contained carried by other objects. Existing deep learning based approaches often suffer from spatiotemporal biases when applied to video reasoning problems. We propose Hopper, which for reasoning object permanence in videos. Given video and a localization query, Hopper reasons over image and object tracks automatically hop over critical frames in iterative fashion to predict the final position of the object of interest. We demonstrate the effectiveness of using contrastive loss to reduce spatiotemporal biases. We evaluate over CATER dataset and find that Hopper achieves 73.2% Top-1 accuracy using just 1 FPS by hopping through just a few critical frames. We also demonstrate Hopper can perform long-term reasoning by building a CATER-h datase() that requires multi-step reasoning to localize objects of interest correctly.
Multi-hop Transformer
INTRODUCTION
this paper, we address the problem of spatiotemporal object-centric reasoning in videos. Specifically, we focus on the problem of object permanence, which is the ability to represent the existence and the trajectory of hidden moving objects (Baillargeon 1986). Object permanence can be essential understanding videos in the domain of: (1) sports like soccer, where one needs to reason, "which player initiated the pass that resulted in goal?", (2) activities like shopping, one needs infer "what items the shopper should be billed for?", and (3) driving, infer "is there a car next to me in the right lane?". Answering these questions requires the ability to detect and understand the motion of objects the scene. This requires detecting the temporal order of one or more actions of objects. Furthermore, it also requires learning object permanence, since it requires the ability to predict the location of non-visible objects as
PREVIOUS FRAMES ...
snitch
FRAME 11
FRAME 12
+Y
+Y
30
36
+X
+X
6
11
FRAME 13
ILAST FRAME UNDER FPS)
0
2
3
4
Figure 1: Snitch Localization CATER (Girdhar &
Ramanan 2020) is an object permanence task where goal is to classify the final location of the snitch object within 2D grid space.
Work done as a NEC. Labs intern. https: thu.. com nec/ m1 c/cater-h

---

Published as a conference paper at ICLR 2021
they are occluded. contained carried by other objects (Shamsian et al. 2020). Hence, solving this task requires compositional, multi-step spatiotemporal reasoning which has been difficult to achieve using existing deep learning models (Bottou 2014 Lake et al. 2017).
Existing models have been found lacking when applying to video reasoning and object permanence tasks (Girdhar& & Ramanan 2020). Despite rapid progress in video understanding benchmarks such as action recognition over large datasets, deep learning based models often suffer from spatial and temporal biases and are often easily fooled by statistical spurious patterns and undesirable dataset biases (Johnson et al., 2017b). For example, researchers have found that models can recognize the action "swimming" even when the actor is masked out, because the models rely on the swimming pool, the scene bias, instead of the dynamics of the actor (Choi et al.) 2019).
Hence, we propose Hopper to address debiased video reasoning. Hopper uses multi-hop reasoning over videos to reason about object permanence. Humans realize object permanence by identifying key frames where objects become hidden (Bremner et al. 2013) and reason to predict the motion and final location of objects in the video. Given video and localization query, Hopper uses a Multi-hop Transformer (MHT) over image and object tracks to automatically identify and hop over critical frames in an iterative fashion to predict the final position of the object of interest. Additionally, Hopper uses a contrastive debiasing loss that enforces consistency between attended objects and correct predictions. This improves model robustness and generalization. We also build a new dataset, CATER-h, that reduces temporal bias in CATER and requires long-term reasoning.
We demonstrate the effectiveness of Hopper over the recently proposed CATER 'Snitch Localization' task (Girdhar& & Ramanan 2020) (Figure Hopper achieves 73.2% Top-1 accuracy in this task at just 1 FPS. More importantly, Hopper identifies the critical frames where objects become invisible or reappears, providing an interpretable summary of the reasoning performed by the model. To summarize, the contributions of our paper are as follows: First, we introduce Hopper that provides a framework for multi-step compositional reasoning in videos and achieves taateoof-the-at accuracy in CATER object permanence task. Second, we describe how to perform interpretable reasoning in videos by using iterative reasoning over critical frames. Third, we perform extensive studies to understand the effectiveness of multi-step reasoning and debiasing methods that are used by Hopper. Based on our results, we also propose new dataset, CATER-h, that requires longer reasoning hops, and demonstrates the gaps of existing deep learning models.
2 RELATED WORK
Video understanding. Video tasks have matured quickly in recent years (Hara et al. 2018);
approaches have been migrated from 2D or 3D ConvNets (Ji et al.) (012) to two-stream
networks (Simonyan & Zisserman 2014). inflated design (Carreira & Zisserman 2017). models with additional emphasis on capturing the temporal structures (Zhou et al. 2018). 8), and recently models that better capture spatiotemporal interactions (Wang et al. 2018 Girdhar al. 2019).
Despite the progress, these models often suffer undesirable dataset biases, easily confused by backgrounds objects in new environments as well varying temporal scales (Choi et al. 2019) Furthermore, they are unable to capture reasoning-based constructs such as causal relationships (Fire
& Zhu 2017) or long-term video understanding (Girdhar & Ramanan 2020).
Visual and video reasoning. Visual and video reasoning have been well-studied recently, but existing research has largely focused on the task of question answering (Johnson et al. 2017a] Hudson & Manning 2018 2019a; Yi et al. 2020).. CATER, a recently proposed diagnostic video recognition dataset focuses on spatial and temporal reasoning as well as localizing particular object of interest. There also has been significant research in object tracking. often with an emphasis on occlusions with the goal of providing object permanence (Wojke et al. 2017: Wang al. 2019b). Traditional object tracking approaches often require expensive supervision of location of the objects in every frame. In contrast, we address object permanence and video recognition on CATER with a model that performs integrate object-centric reasoning without this strong supervision.
Multi-hop reasoning. Reasoning systems vary in expressive power and predictive abilities, which include symbolic reasoning, probabilistic reasoning, causal reasoning, etc. (Bottou) 2014). Among them, multi-hop reasoning is the ability to reason with information collected from multiple passages to derive the answer (Wang et al., 2019a), and it gives a discrete intermediate output of the reasoning process, which can help gauge model's behavior beyond just the final task accuracy (Chen et al.) 2019). Several multi-hop datasets and models have been proposed for the reading comprehension
2

---

architecture Published as a conference paper at ICLR 2021
DETECTION REPRESENTATION
TRACKTNG
TI QUERT CEPPESSNTTATION RECOERITION
positionl encoded leatuze majs
larget:
Targeti
+
Tracking:
encodding
video query
Source
Hungarian
positional time encoding
positiinnal sporial resclution encoding
Between
DETR:
Frame
051
Transfommer
Matching
Algorithm
feature
Time
object rspersentation
Source Sequence
[Ty
ofjece
CNN
Multi-Hop Transformer
MLP
object
Tree sraeee Franes
dlas label
MLP
1 Classification
Imagee
query
Figure An overview the Hopper framework. Hopper first obtains frame representations from the input video. Object representations and object tracks are then computed to enable tracking-interrates object-centric reasoning for the Multi-hop Transformer (details in Section
task (Welbl et al.
2018 Yang et al. 2018b Dua al. 2019 Dhingra al. 2020 We extend multi-
hop reasoning to the video domain by developing a dataset that explicitly requires aggregating clues from different spatiotemporal parts of the video, as well as multi-hop model that automatically extracts a step-by step-by-step reasoning chain, which improves interpretability and imitates natural way of thinking. We provide an extended discussion of related work in Appendix[]
HOPPER
Hopper (Figure 2} a framework inspired from the observation that humans think terms of entities and relations. Unlike traditional deep visual networks that perform processing over the pixels from which they learn and extract features, object-centric learning-basee explicitly separates information about entities through grouping and abstraction from the lowelevel information (Locatello et al. 2020). Hopper obtains representations of object entities from the low- level pixel information of every frame (Section B.2). Additionally, to maintain object permanence, humans are able to identify key moments when the objects disappear and reappear. To imitate that, Hopper computes object tracks with the goal to have more consistent object representation (Section 3.3) and then achieves multi-step compositional long-term reasoning with the Multi-hop Transformer pinpoint these critical moments. Furthermore, Hopper combines both fine- ane-grained (object) and yoarse-graine (image) information to form a contextual understanding of a video. As shown in Figure 2
Hopper contains 4 components; we describe them below.
3.1 BACKBONE
Starting from the initial RGB-based video representation Vy the number of frames of the video, 3 is for the three color channels, and Ho and Wo denote the original resolution height and width, a conventional CNN backbone would extract the feature map f R x''xxxxx and for every frame t a compact image representation és E R.P. The backbone we use is Xt-111 from Ma et al. (2018). P 2048 and H, W 8, 10. 1x1 convolution (Carion et al.] (220)) then reduces the channel dimension of f from P to a smaller dimension
where T represents
256), and a linear layer is used to turn the dimension of i, from to d.
3.2 OBJECT DETECTION AND REPRESENTATION
We collapse the spatial dimensions into 1 dimension and combine the batch dimension with the temporal dimension for the feature map f. Positional encodings are learned for each time step (T' in total) and each spatial location (H W in total), which further added to the feature map in an element-wise manner. The positional
augddeted feature map is the source input the transformer encoder (Vaswani et al., (017)) of DETR (Carion al. 2020). DETR is object detector for image input; it additionally accepts N
a recently proposed transformer-bass embeddings of object queries for every image (assuming every image at most has N object the transformer decoder. We also combine the batch dimension with temporal dimension for the object queries. Outputs from DETR are transformed object representations that are used as inputs to multilayer perceptron (MLP) to predict the bounding box and class label of every object. For Snitch Localization, DETR is trained on object annotations from LA-CATER (Shamsian et al. 2020).
0, i.c., none object, will he predicted if the number of objects in an image is less than N.
3

---

Published as a conference paper at ICLR 2021
3.3 TRACKING
Tracking produces consistent object representations as links the representations of each object through time. We perform tracking using the unordered object representations, bounding boxes and labels as inputs, and applying our Hungarian-bass based algorithm to match objects between every two consecutive frames. We describe the details follows.
Tracking is essentially an association problem (Bewley et al. 2016). An association between objects respectively from consecutive 2 frames be defined by the object class agreement and the difference of the two bounding boxes. Let us denote by y = [0 the predicted list of objects all frames in video, where ýt {vt J denotes the predicted set of objects at frame Each object is represented as 4-tule of = (ct, b,, (ff(C)ec E C) of) where oj denotes the class label that has the maximum predicted likelihood for object at frame I, by E [o, 1] is a vector that defines the bounding box top left and bottom right coordinates relative to the image size, pi(e) denotes the predicted likelihood for class e (where C = flarge metal green cube, small metal green cube,
Ø )), and of Idd denotes the representation vector of this object at frame t.
In order to obtain the optimal bipartite matching between the set of predicted objects at frame and t ++ 1, we search for permutation of N elements a E GN with the lowest permutation cost:
(o(() Σ Ltrack (Dl. D6+1
arg min G
1=1
where Ltrac is a pair-wise track matching cost between predicted object of (i.e., object i at frame t) and predicted object frame t with index (i) from the permutation a, denoted by 92-1- Following Carion et al. (2020), the optimal assignment is computed efficiently with the Hungarian algorithm. The track matching cost at time i for object i is defined as
(a(()
Lrrack (Dl. b0t1) = (cto)) (0()) (c) + >b (eixe) Soox (61, b0(1)
where I denotes an indicator function such that the equation after the symbol 1 only takes effect when the condition inside the {...} is true, otherwise the term will be O. λc, E R weight each term. Loox is defined as linear combination of the L1 loss and the generalized loU loss (Rezatofighi et al. 2019). When the predicted class label object at frame / is not Ø, we aim to maximize the likelihood of the class label ět for the predicted object o (i) at frame | 1, and minimize the bounding box difference between the two. The total track matching cost of video
the aggregation of Ltrack (1% ooff) from object 1 to N and frame /
T 1.
This Hungarian-basee -base tracking algorithm is used due to its simplicity. A more sophisticated tracking solution (e.g. DeepSORT (Wojke al.) 2017)) could be easily integrated into Hopper, and may improve the accuracy of tracking in complex scenes.
3.4 VIDEO QUERY REPRESENTATION AND RECOGNITION
The N object tracks obtained from the Hungarian algorithm and single track of image features from the backbone are further added with the learned positional time encodings to form the source input to our Multi-hop Transformer (which will be introduced in Section 4). Multi-hop Transformer produces the final latent representation of the video query e E IR... MIP uses the video query representation an input and predicts the grid class for the Snitch Localization task.
4 MULTI-HOP TRANSFORMER
Motivated by how humans reason an object permanence task through identifying critical moments of key objects in the video (Bremner et al. 2015). we propose Multi-hop Transformer (MHT). MHT reasons by hopping over frames and selectively attending to objects in the frames, until it arrives at the correct object that is the most important for the task. MHT operates in an iterative fashion, and each iteration produces one-hop reasoning by selectively attending to objects from collection of frames. Objects in that collection of frames form the candidate pool of that hop. Later iterations are built upon the knowledge collected from the previous iterations, and the size of the candidate pool decreases as iteration runs. We illustrate the MHT in Figure i6 Appendix. The overall module is described in Algorithm MHT accepts a frame track Ty: [er, 82, ..., iT an object track To: [o]. o2. "., o/..". " N.. og. ..... of initial target video query embedding E, the number of objects
4

---

Published as a conference paper at ICLR 2021
N and number of frames T. h denotes the hop index, and è is the frame index that the previous hop (i.e., iteration) mostly attended to, in Algorithm [I
Overview. Multiple iterations are applied over MHT, and each iteration performs one hop of reasoning by attending to certain objects in critical frames. With a total of H hops, MHT produces refined representation of the video query & (E R!xd). As the complexity of video varies, H should also vary across videos. In addition, MHT operates in an autoregressive manner to process the incoming frames. This is achieved by "Masking(' (will be described later). The autoregressive processing in MHT allows hop h+1 to only attend to objects in frames after frame t, if hop l mostly attends an object at frame t. We define the most attended object of a hop as the object that has the highest attention weight (averaged from all heads) from the layer in Transformer, (will be described later). The hopping ends when the most attended object an object in the last frame.
multi-head attention
MHT Architecture. Inside of the MHT, there are 2 encoder-decoder transformer units (Vaswani et al. 2017): Transformer, and Transformer. This architecture is inspired by study in cognitive science that reasoning consists of 2 stages: first, one has to establish the domain about which one reasons and its properties, and only after this initial step can one's reasoning happen (Stenning & Van Lambalgen 2012). We first use Transformer, to adapt the representations of the object entities, which form the main ingredients of the domain under the context of an object-centric video task. Then, Transformer, is used to produce the task-oriented representation to perform reasoning.
We separate 2 types of information from the input: attention candidates and helper information. This separation comes from the intuition that humans sometimes rely on additional information outside of the candidate answers. We call such additional information as helper information H (specifically in our case, could be the coarse- grerneegrained global image context, or information related to the previous reasoning step). We define candidate answers as attention candidates U1 which are representations of object entities (because object permanence is task that requires reasoning relations of objects). For each hop. we first extract attention candidates and helper information from the source sequence, then use Transformer to condense the most useful information by attending to attention candidates via self-attention and helper information via mecode-d attention. After that, we use Transformer, to learn the latent representation of the video query by attentively utilizing the information extracted from Transformer, (via encoder-decoder attention). Thus, MHT decides on which object to mostly attend to, given the current representation of the video query E, by reasoning about the relations between the object entities (L4), and how would each object entity relate to the reasoning performed by the previous hop or global information (H).
Transformery. Transformer, uses helper information H. from the previous hop, to adapt the representations of the object entities U to use in the current reasoning step. Formally, U is the object track sequence To as in line (u E I NTNxx xd NT tokens), whereas H. encompasses different meanings for hop and the rest of the hops. For hop 1, H is the frame track T, (H E Txxd T tokens, line 7). This is because hop 1 is necessary for all videos with the goal to find the first critical object (and frame) from the global information. Incorporating frame representations is also beneficial because it provides complementary information and can mitigate occasional errors from the object detector and tracker. For the rest of the hops. H. the set of representations of all objects in the frame that the previous hop mostly attended to (Hl Nxd xd. N tokens, Extract( in line 5). The idea is that, to select answer from object candidates after frame 4, objects in frame could be the most important helper information. Transformer, produces Uupdnte
Algorithm 1 Multi-hop Transformer module.
Input: Ty E RTxd To E NTxx xd E E R Ixdd N E R,
T
Params: LayerNorm, Transformery, Transformer., Wg. by
1: h < 0- 0
Algorithm [
2: while + (T- 1) do
3: 2--h--. h+1
4: if > then
5:
H. Extract (To,N, T.
6:
else
7:
H. x-11 T-T
8:
end if
9:
u - To
10:
Uugeines +- Transformery (V, H)
11:
Uaqdime Sigmoid (W, y Uhngaae + bg) 2
12:
Umisk + Masking Masdine. 1)
13:
E, + Transformer, (Umusks E)
14:
t + (A)
15:
end while
16: LayerNorm (E)
Return
5

---

Published as a conference paper at ICLR 2021
Mupdate E NNxx NT tokens), updated version of U. by selectively attending to H. Further, MHT conditionally integrates helper-fuse representations and the original representations of U. This conditional integration is achieved by Artentional Feature-based Gating (line 11), with the role to combine the new modified representation with the original representation. This layer, added on top of yransformer,. provides additional new information, because it switches the perspective into learning new representations of object entities by learning a feature mask (values between O and I) to select salient dimensions of U, conditioned on the adapted representations of object entities that are produced by Transformery. Please see details about this layer in Margatina et al. (2019)..
Transformer,. Transformer, is then used to produce the task orientrde video query representation E. As aforementioned, MHT operates in an autoregressive manner to proceed with time. This is achieved by 'Masking(" that turns Mugiate into Umask (umask E RNNTxd) for Transformer, by only retaining the object entities in frames after the frame that the previous hop mostly attended to (for hop 1, Umask Mupstnn)) Masking is commonly used in NLP for the purpose of autoregressive processing (Vaswani et al. 2017). Masked objects will have 0 attention weights. Transformer, learns representation of the video query & by attending to Umask (line 13). It indicates that, unlike Transformer, in which message passing is performed across all connections between tokens in U, between tokens in H. and especially across 2/ and H. (we use 24 for Transformery. instead of Mmasks because potentially,
determine which object model should mostly attend to in frames after objects in and before frame might also be beneficial), message passing in Transformer, is only performed between tokens in E (which has only 1 token for Snitch Localization), between tokens in unmasked tokens in Lupdate. and more importantly, across connections between the video query & and unmasked tokens in Uupiasee The indices of the most attended object and the frame that object is in, are determined by attention weights A from the previous hop with a differentiable 'Softargmax()"
Chapelle & Wu 2010: Honari et al. 2018), defined as, softargmax(x)) Ei
eAng Σ, E, i, where
is an arbitrarily large number. Attention weights A (A E NTX1 is averaged from all heads. E is updated over the hops, serving the information exchange between the hops.
Summary & discussion. H. Umusk and È are updated in every hop. & should be seen as an encoding for a query of the entire video. Even though this dataset, a single token is used for the video query, and the self-attention in the decoder part of Transformer, is thus reduced to a stacking of 2 linear transformations, it is possible that multiple queries would be desirable in other applications. These structural priors that are embedded in (c.g., the iterative hopping mechanism and attention, which could be treated as a soft tree) essentially provide the composition rules that algebraically manipulate the previously acquired knowledge and lead to the higher forms of reasoning (Bottou) 2014). Moreover, MHT could potentially correct errors made by object detector and tracker, but poor performance of them (especially object detector) would also make MHT suffer because (1) inaccurate object representations will confuse MHT in learning, and (2) the heuristic-basee loss for intermediate hops (will be described in Section 5 will be less accurate.
5 TRAINING
We propose the following training methods for the Snitch Localization task and present an ablation study in Appendix[A] We provide the implementation details of our model in Appendix HH
Dynamic hop stride. A basic version of autoregressive MHT is to set the per-hop frame stride to 1 with 'MaskingO' as usually done in NLP. It means that Transformer, will only take in objects in frame {+1 as the source input if the previous hop mostly attended to an object in frame t. However, this could produce an unnecessary long reasoning chain. By using dynamic hop stride, we let the model automatically decide on which upcoming frame to reason by setting 'MaskingO' to give unmasked candidates as objects in frames after the frame that the previous hop mostly attended to.
Minimal hops of reasoning. We empirically set the minimal number of hops that the model has to perform for any video as 5 to encourage multi-hop reasoning with reasonably large number of hops (unless not possible, e.g., if the last visible snitch is in the second last frame, then the model is only required to do 2 hops). This is also achieved by 'MaskingO'. E.g.. if hop I mostly attends to an object in frame 3. 'MaskingO' will not mask objects in frames from frame 4 to frame 10 for hop 2, in order to allow hop 3, 4, 5 to happen (suppose 13 frames per video, and frame 4 is computed from mar(3 + 1, 13 (5 - 2))).
3 + 1, frame 10 is computed
Auxiliary hop 1 object loss. Identifying the correct object to attend to in early hops is critical and for Snitch Localization, the object to attend to in hop 1 should be the last visible snitch (intuitively).

---

Published as a conference paper at ICLR 2021
Hence, we define an auxiliary hop object loss as the cross-entropy of classifying index of the last visible snitch. Inputs to this loss are the computed index of the last visible snitch from To (with the heuristic that approximates it from predicted object bounding boxes and labels), as well as the attention weights A from Transformer, of hop 1, serving as predicted likelihood for each index.
Auxiliary hop 2 object loss. Similarly, we let the second hop to attend to the immediate occluder or container of the last visible snitch. The auxiliary hop 2 object loss is defined as the cross-entropy of classifying index of the immediate occluder or container of the last visible snitch. Inputs this loss are the heuristic CouuristicCooppeu computed index and attention weights A from Transformer, of hop 2.
Auxiliary hop 1 1&2 frame loss. Attending to objects in the correct frames in hop and 2 critical for the later hops A L1 loss term could guide the model to find out the correct frame index.
Teacher forcing is often used as a strategy for training recurrent neural networks that uses the ground truth from prior time step as an input (Williams & Zipser 1989). We use teacher forcing for hop? and by providing the ground truth H and Umask (since we can compute the frame index of the last visible snitch with heuristics as described above).
Contrastive debias loss via masking out. This loss is inspired from the human mask confusion loss in Choi et al.) (2019). It allows penalty for the model if it could make predictions correctly when the most attended object in the last frame is masked out. However, in contrast to human mask, enforee consistency between attended objects and correct predictions, ensuring that the model understands why it is making a correct prediction. The idea here is that the model should not be able to predict the correct location without seeing the correct evidence. Technically, the contrastive debias loss is defined as the entropy function that we hope to maximize, defined as follows.
K
Ldebias E
Σ go (Maegi ... (log go (Mangi ))
(3)
where ge denotes the video query representation and recognition module (Multi-hop Transformer along with MLP) with parameter that produces the likelihood of each grid class. Macg is the source sequence to the Multi-hop Transformer with the most attended object in the last hop being masked out (set to zeros), and K denotes the number of grid classes.
linear combination of hop 1 and hop 2 object & frame loss, contrastive debiasing loss for the last hop, and the final grid classification cross-entropy loss. The object & frame loss for hop and 2 are based on heuristics. The motivation is to provide weak supervision for the early hops to avoid error propagation, as multi-hop model can be difficult to train, without intermediate supervision or when ground truth reasoning chain is not present (as in Hopper) (Dua et al.) 2020 Qi et al. 2019 Ding et al.) 2019 Wang et al. [2099a] Chen & Durrett 2018 Jiang & Bansal 2019). One can use similar ideas as the ones here on other tasks that require multi-hop reasoning (c.g., design self-supervision or task task-specific heuristic-based weak supervision for intermediate hops, as existing literature often does).
Summary & discussion. The total loss of the model
6 EXPERIMENTS
Datasets. Snitch Localization (Figure I is the most challenging task in the CATER dataset and it requires maintaining object permanence to solve the task successfully (Girdhar & Ramanan 2020 However, CATER is highly imbalanced for Snitch Localization task in terms of the temporal cues: snitch is entirely visible the end of the video for 58% samples, and entirely visible the second last frame for 14% samples. As a result, it creates a temporal bias in models to predict based on the last few frames. To address temporal bias in CATER, we create a new dataset, CATER-han (CATER-h for short), with diverse temporal variations. In CATER-h, every frame index roughly shares an equal number of videos, to have the last visible snitch in that frame (Figure 13).
Baselines & metrics. We experiment with TSM (Lin et al., 2019), TPN (Yang et al. 2020), and SINet (Ma et al. 2018). Additionally, Transformer that uses the time-encoded frame track as the source sequence is utilized. Both SINet and Transformer are used to substitute our novel MHT in the Hopper framework in order to demonstrate the effectiveness of the proposed MHT. This means that Hopper-transformee transforrrr and Hopper-sinet use the same representations of image track and object tracks as our Hopper-multihoy Moreover, we report results from Random baseline, a Tracking baseline used by Girdhar & Ramanan (2020) (DaSiamRPN), and Tracking baseline based on our Hungarian algorithm (Section 3.3 in order to have a thorough understanding of how
"The heuristic is La distance to find out that in the immediate frame which object's bounding box bottom midpoint location is closest to that of the last visible snitch.
7

---

Published as a conference paper at ICLR 2021
well our tracking component performs. For CATER. also compare with the results in Girdhar & Ramanan (000)) Details of the baselines are available Appendix H We evaluate models using Top-1 and Top-o accuracy, well mean L, distance of the predicted grid cell from the ground truth following Girdhar & Ramanan 2220). L is cognizant of the grid structure and will penalize confusion between adjacent cells less than those between distant cells.
Methods
FPS
# Frames
Top
Top 5
I
Random
2.8
13.8
DaSiamRPN (Tracking) (hu et al.
2018
33.9
40.8
2.4
Hungarian (Tracking ours)
46.0
52.7
1.9
TSN (RGB) (Wan
2016
14.1
38.5
3.2
ISN (RGB) LSTM Wang
2016
25.6
67.2
TSN (Flow) Wang et al. 2016
9.6
32.3
3.7
TSN (Flow) + LSTM (Wang et al.
2016
3
14.0
43.5
3.2
I3D-50 (Carreira & Zisserman 2017)
5
64
57.4
78.4
1.4
I3D-50 +T.TMM (Carreira Zisserman
2017
5
60.2
81.8
1.2
I3D-50 NI. (Wang et al. 2018)
2.5
26.7
68.9
I3D-50 + NI +LST (Wang et al. 2018
2.5
32
46.2
69.9
1.5
TPN-101 (Yang
2020)
2.5
32
65.3
83.0
1.09
TSM-50 (Lan
2019)
64.0
85.7
0.93
SINet (Ma at.) 2018)
21.1
47.1
3.14
Transformer Vassaani ct al. 2017)
1
13
13.7
39.9
3.53
Hopper-transtormer (last frame)
1
13
61.1
86.6
1.42
troperr-rrnnformmrr
1
64.9
90.1"
1.11
Hopper-sinct
1
69.1
91.8
1.02"
Hopper-mulihop (our proposed method)
1
13
73.2*
93.8"
0.85
Table CATER Snitch Localization results (on the test set). The top performance scores are highlighted as: First" Second, Third". Hopper outperforms existing methods under only 1 FPS.
Methods
FPS
Frames
Top 1
Top 5
L1 +
Random
2.4
13.6
3.9
DaSiamRPN (Tracking) (Zhu et al.
2018
17.1
26.1
2.9
Hungarian (Tracking- ours)
37.2
44.4
2.3
TPN-101 Yang et al. 2020)
2.5
32
50.2
88.3"
1.46
TSM-50 Lin et al 2019
1
13
44.0
75.7
1.54
SINet MM et al. 2018)
13
18.6
44.3
3.24
Transtormer Vaswani
2017)
13
11.6
34.4
3.49
Hopper-transtorme: (last frame)
13
41.8
79.3
2.10
topper-transformer
13
57.6"
90.1
1.39*
Hopper-sinct
13
62.8
91.7"
1.25
Hopper-multihop (our proposed method)
13
68.4"
87.9
1.09
Table 2: CATER-h Snitch Localization results (on the test set). The top 3 performance scores are highlighted
First" Second, Thind". Hopper outperforms existing methods under only 1 IPS.
Results. We present the results on CATER in Table DalleTand and d CATER-h in Table 2 For all methods, we find that the performance on CATER-h lower than that on CATER, which demonstrates the difficulty of CATER-h. Such performance loss is particularly severe for the tracking baseline and the temporal video understanding methods (TPN and TSM). The DaSiamRPN tracking approach only solves about third of the videos on CATER and less than 20% on CATER-h. This is because the tracker is unable to maintain object permanence through occlusions and containments, showcasing the challenging nature of the task. Our Hungarian tracking has 46.0% Top- on CATER and 37.2% Top-1 on CATER-h. On CATER, TPN and TSM, as two stte-offthe-aas art methods focusing on temporal modeling for videos, achieve a higher accuracy than methods Girdhar & Ramanan (2200). However, on CATER-h, TPN only has 50.2% Top-1 accuracy and 44.0% for TSM. SINet performs poorly even though SINet reasons about the higher-order object interactions via multi- headed attention and fusion of both coarse- and fine-grained information (similar to our Hopper). The poor performance of SINet can be attributed to the less accurate object representations and the lack of tracking whereas effective temporal modeling is critical for the task. Without the object-centric modeling, Transformer that uses sequence of frame representations performs poorly. However, for both SINet and Transformer, significant improvement observed after utilizing
8

---

Published as a conference paper at ICLR 2021
BEGINAING
FRAME
FRAME 2
FRAME 3
FRAME 4
FRAME 5
FRAME 6
Meg Rexd
2
FRAME 7
FRAME
FRAME
FRAME 10
FRAME #
FRAME 12
FRAME 13
crack
STREE
Snitch is contained by the purple cone at frame 5 (the last visible snitch is in frane 4). Then, purple cone 15 contained by the blue cone at frame 7, After that the blue cone keeps sliding. The blue cone uncontains the purple cone at frame 13. In the end, the snztch is inside the purple cone.
object attended per hop and per head from Hopper-multihop. The frame border of attended object colored based on the hop index (accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). The hounding hox of the most attended object in each hop shares the same color as the color of the hop index. Please zoom see the details. Best viewed in color.
Figure Qualitative result & interpretability
our model.
highlight
Hopper framework. This shows the benefits of tracking-enabled object-centric learning embedded in Hopper. Since almost 60% of videos in CATER have the last visible snitch in the last frame, we run another experiment by using Transformer in Hopper but only uses the representations of the frame and objects in the last frame. This model variant achieves 61.1% Top-1 CATER, even higher than the best best-performing method in Girdhar & Ramanan (2020). but only 41.8% Top-1 on CATER-h, which proves and reiterates the necessity of CATER-h. The accuracy is relatively high, likely due to other dataset biases. More discussion method, Hopper with the MHT, outperforms all baselines in Top-1 using only 13 frames per video, highlighting the effectiveness and importance of the proposed multi-hop reasoning. (Shamsian et al.) (2020) report slightly better Top-1 (74.8%) and L1 (0.54) on CATER but they impose strong domain knowledge, operate at 24 FPS (300 frames per video) and require the location of the snitch to be labeled every frame, even when contained or occluded, for both object detector and their reasoning module. Labeling these non-visible objects for every frame of a video would be very difficult in real applications. Furthermore, their method only has 51.4% Top-1 and L1 of 1.31 on CATER-h at 24 FPS. Please refer to Appendix where we provide more quantitative results.
available in Appendix G.6 Our
visualize the attended objects in Figure 3 illustrated, the visible snitch is in frame 4 in this video, and at frame 5 snitch is contained by the purple cone. At frame 7, the purple cone is contained by the blue cone, but in the end, the blue cone excludes the purple cone. Hop 1 of hopper-multiho, attends to the last visible snitch at frame 4, and hop 2 attends to snitch's immediate container: the purple cone at frame 5. Hop 3 mostly attends to the purple cone's immediate container: the blue cone frame 7, and secondly attends to the blue cone at frame 11 (by the other head) as the blue cone is just sliding from frame 7 till 12. Hop 4 mostly attends to the blue cone frame 12. Hop 5 mostly attends to the purple cone (who contains the snitch) frame 13, and secondly attends to the blue cone at frame 13. The visualization exhibits that Hopper- Hopper-multihop performs reasoning by hopping over frames and meanwhile selectively attending objects in the frame. It also showcases that MHT provides more transparency to the reasoning process. Morcover, MHT implicitly learns to perform snitch-oriented tracking automatically. More visualizations are available in Appendix.
Interpretability.
CONCLUSION AND FUTURE WORK
This work presents Hopper with novel Multi-hop Transformer to address object permanence in videos. Hopper achieves 73.2% Top-1 accuracy at just FPS on CATER, and demonstrates the benefits of multi-hop reasoning. In addition, the proposed Multi-hop Transformer uses iterative attention mechanism and produces a step-by-step reasoning chain that improves interpretability. Multi-hop models often difficult train without supervision for the middle hops. We propose several training methods that can be applied to other tasks to address the problem of lacking a ground truth reasoning chain. In the future, we plan experiment on real-world video datasets and extend our methods to deal with other complex tasks (such as video QA).
ACKNOWLEDGMENTS
The research supported in part by NSF awards: IIS- 103883,, IIS- 1955404, and IIS- 1955365.

---

Published as a conference paper at ICLR 2021
REFERENCES
Somak Aditya, Yezhou Yang, and Chitta Baral. Explicit reasoning over end-to-end neural architectures for visual question answering. arXiv preprint arXiv:1803.00 08896, 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning compose neural networks for question answering. arXiv preprint 1rXiv:1601.00 1000000000...,. 2016a.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016b.
Renee Baillargeon. Representing the existence and the location of hidden objects: Object permanence in 6-and 8-month-old infants. Cognition, 23(1):21-41, 1986.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Counterfactual learning of physical dynamics. arXiv preprint arXiv: 12000, 2019.
Cophy:
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464-3468. IEEE, 2016.
Léon Bottou. From machine learning to machine reasoning. Machine learning, 94(2):133-149, 2014.
J Gavin Bremner, Alan M Slater, and Scott P Johnson. Perception of object persistence: The origins of object permanence in infancy. Child Development Perspectives, 9(1):-113, 2015.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 1287 2020.
End-to-end object detection with transformers. arXiv preprint
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Olivier Chapelle and Mingrui Wu. Gradient descent optimization of smoothed information retrieval metrics. Information retrieval, 13(3):21-22) 235, 2010.
Jifan Chen and Greg Durrett. How learn (and how not learn) multi-hop reasoning with memory networks. 2018.
Jifan Chen, Shih-ting Lin, and Greg Durrett. Multi-hop question answering via reasoning chains. arXiv preprint (01i::9910026111 2019.
Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why can't dance in the mall? learning to mitigate scene bias in action recognition. In Advances in Neural Information Processing Systems, pp. 853-865, 2019.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li,, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference computer vision, pp. 764-773, 2017.
Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. Chains of reasoning entities, relations, and text using recurrent neural networks. arXiv preprint arXiv:1607.001226 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W. Cohen. Differentiable reasoning over virtual knowledge base. In International Conference on Learning Representations, 2020. URLhttps: http://////ppppeee///// net forrm??
id=S.Jxst 1 HFPH
10

---

Published as a conference paper at ICLR 2021
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang. and Jie Tang. Cognitive graph for multi-hop reading comprchension at scale. arXiv preprint arXiv:1905.05460, 2019.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2625-2634, 2015.
Dheeru Dua, Yizhong Wang. Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint drXiv:1903.00161, 2019.
Dheeru Dua, Sameer Singh, and Matt Gardner. Benefits of intermediate annotations in reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 5627-5634, 2020.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1933-1944 2016.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE international conference on computer vision, pp. 6202- 2211,2019)
Amy Fire and Song-Chun Zhu. Inferring hidden statuses and actions in video by causal reasoning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 48-56, 2017.
Rohit Girdhar and Deva Ramanan. CATER: diagnostic dataset for Compositional Actions and TEmporal Reasoning. In ICLR, 2020.
Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 244-253, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, volume 1, pp. 5, 2017.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546-6555, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher Pal, and Jan Kautz. Improving landmark localization with semi-supervised learning. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1546-1555, 2018.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 804-813, 2017.
Drew Hudson and Christopher D Manning. Learning by abstraction: The neural state machine. In Advances in Neural Information Processing Systems, pp. 5903-5916, 2019a.
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine
reasoning. arXiv preprint 18033 2018.

---

Published as a conference paper at ICLR 2021
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700-6709, 2019b.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2758-2766, 2017.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1)2221-231, 2012.
Yichen Jiang and Mohit Bansal. Self-assembling modular networks for interpretable multi-hop reasoning. arXiv preprint /rXiv: 005800 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2901-2910, 2017a.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2989-2998, 2017b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: wet::44226660 69800 2014.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint vxXiv:/809.01660 2018.
Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8971-8980, 2018.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7083-7093, 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
Chih-Yao Ma, Asim Kadav, lain Melvin, Zsolt Kira, Ghassan AlRegib, and Hans Peter Graf. Attend and interact: Higher-order object interactions for video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6790-6800, 2018
Katerina Margatina, Christos Baziotis, and Alexandros Potamianos. iicettio-bbsed conditioning methods for external knowledge integration. arXiv preprint arXiv:1906.03677 2019.
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4942-4950, 2018.
Jonghwan Mun, Paul Hongsuck Sco, Ilchae Jung, and Bohyung Han. Marioqa: Answering questions by watching gameplay videos. Proceedings of the IEEE International Conference on Computer Vision, pp. 2867-2875, 2017.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. arXiv preprint 1709.077, 2017.
Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning. Answering complex open-domain questions through iterative query generation. arXiv preprint arXiv:1910.0700 2019.
12

---

Published as a conference paper at ICLR 2021
Hamid Rezatofighi, Nathan Tsoi, Jun'oung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 658-666, 2019.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
Aviv Shamsian, Ofri Kleinfeld, Amir Globerson, and Gal Chechik. Learning object permanence from video. arXiv preprint arXiv:2003. 2020.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood homes: Crowdsourcing data collection for activity understanding. European Conference Computer Vision, pp. 510-526. Springer, 2016.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 568-576, 2014.
Keith Stenning and Michiel Van Lambalgen. Human reasoning and cognitive science. MIT Press, 2012.
Joseph Suarez, Justin Johnson, and Fei-Fei Li. Ddrprog: A clevr differentiable dynamic reasoning programmer. arXiv preprint arXiv:1803.11300 2018.
Alon Talmor and Jonathan Berant. Repartitioning of the complexwebquestions dataset. arXiv preprint (107.08807009233 2018.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4631-4640, 2016. 2016.
Graham Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of spatio-temporal features. In European conference computer vision, pp. 140-153, Springer, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and lllia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
Haoyu Wang, Mo Yu, Xiaoxiao Guo, Rajarshi Das, Wenhan Xiong, and Tian Gao. Do multi-hop readers dream of reasoning chains? arXiv preprint 191.::2910.14520, 2019a.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. European conference on computer vision, pp. 20-36. Springer, 2016.
Qiang Wang. Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1328-1338, 2019b.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302, 2018
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with deep association metric. In 2017 IEEE international conference on image processing (ICIP), pp. 3645- 3649. IEEE, 2017.
13

---

Published as a conference paper at ICLR 2021
Saining Xie, Ross Girshick, Piotr Dollär, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 591 591-600, 2020.
Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathon Shlens, and David Sussillo. A dataset and architecture for visual reasoning with a working memory. In European Conference on Computer Vision, pp. 729-745. Springer, 2018a.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: 09900, 2018b.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020. /RI/tttps:///peeeeviee. net forum?
Hd=HHXYZANYDD
Joe Yue-Hei Ng. Matthew Hausknecht, Sudheendra Vjjaanaraaiimann, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4694-4702 4702, 2015.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis Phillipe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8807-8817, 2019.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 803-818, 2018.
Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 101-117, 2018.
14

---

Published as a conference paper at ICLR 2021
A
ABLATION STUDY
Dynamic Stride
Min Hops
Hop Loss
Hop 2 Loss
Frame Loss
Teacher Forcing
Debias Loss
Top 11
Top 5
L1 +
68.41
87.89
1.09
64.65
87.75
1.19
X
X
63.32
86.94
1.23
X
62.88
86.57
1.19
X
X
X
61.92
88.19
1.20
X
X
X
X 59.48
84.58
1.37
X
X
X
X
X
59.11
86.20
1.37
X
X
X
X
57.27
84.43
1.39
Ablation Study of Hopper training methods. We gradually add training methods described in Section [ i.c. dynamic hop stride, minimal hops of reasoning. auxiliary hop object loss, auxiliary hop 2 object loss, auxiliary frame loss, teacher forcing, and contrastive debias loss masking out, the base Hopper-multine model. The results are obtained from the CATER-h test set.
Table 3:
We conduct an ablation study of training methods described in Section 4 in Table 3 As shown, all proposed training methods are beneficial. 'Dynamic Stride' gives the model more flexibility whereas 'Min 5 Hops' constrains the model to perform reasonable number of steps of reasoning. 'Hop Loss', 'Hop Loss", 'Frame Loss" and "Teacher Foreing' stress the importance of the correctness of the first 2 hops to avoid error propagation. "Debias Loss' is the most effective one by contrastively inducing the latent space to capture information that maximally useful to the task at hand.
Object Detector & Tracker
Reasoning Model
Top f
Top 5
I +
DETR Hungarian (ours)
MHT (both Masked)
65.73
88.39
1.13
DETR +
Hungarian (ours)
MHT (no Gating)
66.62
87.43
1.16
DETR (no Tracking)
MHT (no Tracking)
67.51
88.32
1.14
DETR +
Hungarian (ours)
MHT (mask out LAST)
32.49
60.92
2.53
DETR Hungarian (ours)
MHT (mask out ALL)
11.68
28.98
3.60
Table 4: Ablation study & comparative results of analyzing components of our method (on CATER-h test set).
We then study how would different choices of the sub-components of method affect the Snitch Localization performance. In Table
MHT (both Masked)': This refers to using our Hopper-multhoog but replacing the original U input to Transformer, with masked version. In this way, both Transformer, and Transformer, have 'Masking(' applied beforehand.
'MHT (no Gating)': This refers to using our Hopper- mulpeoo but removing the Attentional Feature-based Gating (line 11 in Algorithm I inside of MHT.
'MHT (no Tracking)': This refers to using Hopper- Hopper-multihop but entirely removing the Hungarian tracking module. Thus, MHT directly takes in unordered object representations inputs.
'HT (mask LAST)': This refers taking trained Hopper-multhhp. masking out the representation of the most attended object in the last hop by zeros, and then making predictions. This is verify whether the most attended object in the last hop is important for the final Snitch Localization prediction task.
'HT (mask out ALL)': Similar to the above, 'MHT (mask out ALL)' refers to taking our masking the representations of the most attended objects
trained Hopper-muliie in all hops by zeros, and then making predictions. This is to verify how important are the most attended objects in all hops that identified by Hopper-multhop.
15

---

Published as a conference paper at ICLR 2021
As shown in Table 4 all of these ablations give worse performance, thus, indicating that motivations for these designs are reasonable (see Section 4 Recall that in Table 2 'DETR + Hungarian' (without MHT) has only 37.2% Top-1 accuracy CATER-h (learning perfect object detector or tracker is not the focus of this paper). This highlights the superiority of our MHT as a reasoning model, and suggests that MHT has the potential correct mistakes from the upstream object detector and tracker, by learning more robust object representations during the process of learning the Snitch Localization task. Masking out the most attended object identified by our Hopper- multihhpp in the last hop only has 32.49% Top-1 accuracy. Masking out all of the most attended objects from all hops only has 11.68% Top-1 accuracy, Such results reassure us about the interpretability of our method.
B PARAMETER COMPARISON
Model
Parameters (M)
GFLOPs
Top 1 Acc.
SINet
138.69
7.98
18.6
Transformer
15.01
0.11
11.6
Hopper-transforme (last frame)
15.01
0.09
41.8
hupper-transforme
15.01
1.10
57.6
Hopper-sinet
139.22
8.05
62.8
Hopper-multine (our proposed method)
6.39
1.79
68.4
Table Parameter and FLOPs comparison our Hopper-multhhpp to alternative methods. (M) indicates millions. Results of the methods on CATER-h test set is also listed.
In Table 5 we compare the number of parameters of our Hopper-multhopp with alternative methods. Our proposed method is the most efficient one in terms of the number of parameters. This is because of the iterative design embedded in MHT. Unlike most existing attempts on using Transformer that stack multiple encoder and decoder layers in a traditional way, MHT only has one layer of Transformer, and Transformer.. As multiple iterations are applied to MHT, parameters of Transformer, f and Transformer, from different iterations are shared. This iterative transformer design is inspired by previous work Locatello et al.) (2020). This design saves parameters, accumulates previously learned knowledge, and adapts to varying number of hops, c.g., some require 1 hop and some require more than 1 hop (e.g, hops). Because for these videos that tentatively only require hop, stacking multiple layers of Transformer such as 5 might be wasteful and not necessary, our design of MHT could address such issue, being more parameter-effiiient We also report the GFLOPs comparison. Given that the FLOPs for MHT depends on the number of hops predicted for video, we report the average number of FLOPS for the CATER-h test set.
C DIAGNOSTIC ANALYSIS
C.I DIAGNOSTIC ANALYSIS ON THE HOPPING MECHANISM
# Hops
2
3
2 5
Ground Truth
104
105
111
110
1026
Prediction
109
102
112
108
1025
Jaccard Similarity ?
0.9541
0.9714
0.9561
0.9818
0.9990
Table 6: Diagnostic analysis the Multi-hop Transformer terms of the 'hopping' ability (# hops performed).
We evaluate the 'hopping' ability of the proposed Multi-hop Transformer in Table 6 The prediction made by our Hopper-multhhop that requires at least 5 hops of reasoning unless not possible. For each test video, we compute the ground truth number of hops required by this video and obtain the number of hops that Hopper actually runs. In the table, we provide the ground truth count and predicted count of test videos that require hop, 2 hops, 3 hops, hops, and equal or greater than hops. Since the numbers are close, we further compute Jaccard Similarity (range from 0 to 1 and higher is better) to measure the overlapping between the ground truth set of the test videos and predicted set of the test videos. According to these metrics, our proposed Hopper-multingn functionally performs the correct number of hops for almost all test videos.
16

---

Published as a conference paper at ICLR 2021
Frame Index
3
6
9
10
12
Hop 1
127
136
106
120
104
111
109
107
105
0
Hop? 2
127
136
106
120
104
111
109
107
105
Hop 3
8
2
66
10
13
24
17
107
778
Hop 4
0
0
0
1
0
0
0
0
0
10
1013
Hop 5
0
0
0
0
0
0
0
0
0
0
1016
Hop 6
1
0
0
Table 7: Hop Index Frame Index: the number of times hop h mostly attends to frame t (results are obtained from Hopper-muliite on CATER-h). See details Appendix [C.]
In Table we show the number of times hop h mostly attends to frame t. The results are obtained from Hopper-multhhop on CATER- h. for those 1025 test videos predicted with 2 hops shown in Table 6 In Figure 4 we plot the index of frame of the most attended object identified by each hop (conveying the same meaning as Table 7 The transparency of the dot denotes the normalized frequency of that frame index for that particular hop.
12
Frame Index
11
10
We can observe that: (1) Hop 3 to tend to attend to later frames, and this is due to lacking supervision for the intermediate hops. As discussed in Section [5] multi- hop model is hard to train in general when the ground truth reasoning chain is missing
Hop Index
Figure 4: Hop Index Frame Index: plot the index frame of the most attended object identified by each hop. Each hop has its unique color, and the transparency of the dot denotes the normalized frequency of that frame index for that particular hop.
during training (Dua et al.) 2020 Chen & Durrett 2018 Jiang & Bansal 2019 Wang
et al. 2019a). Researchers tend to use ground truth reasoning chain as supervision when they 2019 Chen et al. 2019). The results reconfirm
train multi-hop model (Qi et al. 2019 Ding et al. that, without supervision for the intermediate steps, is not easy for a model to automatically figure out the ground truth reasoning chain; (2) MHT has learned to predict the next frame of the frame that identified by hop 1, as the frame that hop should attend to; (3) there are only 9 videos predicted with more than 5 hops even though hops (unless not possible). Again, this is because no supervision is provided for the intermediate hops. As the Snitch Localization task itself is largely focused on the last frame of the video, without supervision for the intermediate hops, the model tends to "look at" later frames as soon as possible. These results suggest where we can improve for the current MHT, e.g., one possibility to design reff-supervision for each intermediate hop.
only constrain the model to perform at least 5
C.2 COMPARATIVE DIAGNOSTIC ANALYSIS ACROSS FRAME INDEX
In Figure 5 we present the comparative
Accuracy Ratio of test set
Tracking (CW: 0.55 SD: 0.09)
diagnostic analysis of the performance in terms of when snitch becomes last visible. We bin the test set using the frame index of when snitch becomes last visible in the video. For each, we show the test set distribution with the bar plot, the performance over that bin using the line plot, and performance of that model on the full test set with the dashed line. We find that for Tracking (DaSiamRPN) and TSM, the Snitch Localization performance drops if the snitch becomes not visible earlier in the video. Such phenomenon, though still exists, but is alleviated for Hopper-multihe We compute the standard deviation (SD) and coefficient of
TSM (CV: 0.27 0.12) Hopper (CV: 0.1 SD: 0.07)
Frame index of when snitch became last visible
Figure 5: Diagnostic analysis of the performance in terms of when snitch becomes last visible the video.
17

---

Published as a conference paper at ICLR 2021
variation (CV). Both are measures of relative variability. The higher the value, the greater the level of dispersion around the mean. The values of these metrics as shown in Figure | further reinforce the stability of our model and necessity of CATER-h dataset.
D EXTRA QUALITATIVE RESULTS
In Figure we visualize the attention weights per hop and per head from Transformer, to showcase the hops performed by Hopper-min mullihop for video "0 (the one in Figure 3 in details. Please see Figure 7 8 I0, and [1] for extra qualitative results from Hopper-multhop. We demonstrate the reasoning process for different cases (i.e., 'visible', 'occluded', 'contained', "contained recursively', and 'not visible very early in the video").
(a) Qualitative result via visualizing the most attended object(s) of every hop.
BE.IINNINNG...
F
FRAME 2
FRAME 3
FRAME
FRAME 5
FRAME 6
Hop 1 Head 1,
top 2 Head A,
FRAME 7
FRAME 8
FRAME: 4
FRAMÉ TO
FRAME f
FRAME 12
FRAME 13
Hep Reud sep sead
HOJ Ha35
S 9936
HoRE
Snitch is contained by the purple cone at frane (the last visible snitch is in frane 4). Then, purple cone is contained by the blue cone at frame 7. After that the blue cane keeps sliding The blue cone uncontains the purple cone at frane 13. In the end, the snitch is inside the purple cone.
(b) Attention weights visualization.
20
100
Hop 1t Video CATERA 054110 mostly attends to the 30th etement from the object tracks in the nemory sequence, Lic., track 3 at frane Hop Head CATER 854110 mostly atterds 38th clesent froe the object tracks in nemory sequence, Litr, neeory sequence,
bystly
Video CATERA 054110 mostly attends to the 83th etement from the object tracks in the nemory sequence, Lec., track 7 at frane 5 Hop Head 1: Video CATER 854118 mostly attends to the 83th element froe the object tracks in the aceory sequence, L.b., track at frame Hop Head Video 63411 mostly attends to the B3th element free the object tracks in the nemory sequence, Lob., track frame
Hop 21
Video CATERA 054110 mostly
98th ctement from the
at frane
Hop Head 1: Video CATERA 054110 mostly attends the b0th element froe the object tracks in the eceory sequence, Lib., track 8 at frame Hop Head 2: Video catem_ msstly attends to the 182th eleeet from the object tracks in the netory sequence, 1.6., track at frane 11
netory sequence, i.t., track at frane Hop read 2: Video Catemn 054110 mostly attends to the 183th aleeent from object tracks nenory sequence, track frane
Hop Video CATERA 054110 mestly
elenest free object
i0
lse
120
Hop S: Video CATERA 054110 mostly attends to the 13th element from the object tracks in the nemory sequence, Lit., track at frane 13 Hop Head Video CATERA 054118 mostly attends to the element from object tracks netory sequence, Licr, aceory
frane
054110
13th
Lic.,
Figure 6: Visualization of attention weights & interpretabiliity of our model. In (a), we highlight object(s) attended in every hop from Hopper-multihop (frame border is colored accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). In (b), we visualize the attention weights per hop (the smaller attention weight that an object has, the larger opacity is plotted for that object entity). As shown, Hopper-multihop performs hops of reasoning for the video 'CATERh CATERho054101 05101551111 10'. Our model performs reasoning by hopping over frames and meanwhile selectively attending to objects in the frame. Please zoom in to see the details. Best viewed in color.
E TRACK REPRESENTATION VISUALIZATION
Please see Figure 12 for visualization of the object track representations video 'CATERh "0 (attention weights from Hopper- Hopper-multihop for this video are shown in Figure Hopper utilizes tracking-integratee object representations since tracking can link object representations through time and the resulting representations are more informative and consistent. As shown in the figure, the tracks that are obtained from our custom Hungarian algorithm that are competitive. Our model Hopper multthho takes in the best-effort object track representations (along with the coarse- grained frame track) as the source input to the Multi-hop Transformer, and then further learns the most useful and correct task-oriented track information implicitly (as shown in Figure
18

---

Published as a conference paper at ICLR 2021
(a) Qualitative result via visualizing the most attended object(s) of every hop.
BEGSINNIING
PRAME.1
FRAME 2
FRAME 3
PRAME
FRAME
FRAME 6
M
FRAME
FRAME 8
FRAME q
FRAME 10
FRAME
FRAME 12
FRAME
Traak
(Visibla) In the end, the snitch is visible.
(b) Attention weights visualization.
Hap 11 Video CATERA, 915368 wostly attends to the 184th elesent free the object tracks in the aesory sequence, L.b.. track at frane mostly attends the element object tracks the nenory sequence, Life, frane Video CATERA 915356
Figure 7: We visualize the attention weights per hop and per head from Transformer, Objects attended every hop are highlighted (whose frame border colored accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). Please zoom in to see the details. Best viewed in color.
Hopper-
(a) Qualitative result via visualizing the most attended object(s) of every hop.
BEGINNING
FRAME
FRAME
FRAME 3
FRAME
FRANE
FRAME
FRAME 7
FRAME
FRAME
FRAME IO
FRANE
FRAME 12
FRAME
(Occlusion) In the end, the snitch is accluded by the green cylinder.
(b) Attention weights visualization.
20
do
189
120
Hop Video CATEAN 812713 mostly attends to the 77th etement from the object tracks the nemory sequence, 1ser, track at frame acmory sesuence, i.t.,
eestly to eleeent sbject
caterh
120
Hop Video CATERh, 012713 nostly attends to the 91th etenent from the object tracks the nemory sequence, Lebr, track at frame 13 Hop Head Video CATERA 012713 mostly attends to the 91th element froe the object tracks the acmory sequence, i.t., track frane 13 object tracks memory sequence, Leta, track
Head CA1E11_012713 asatly attends the element
Figure 8: We visualize the attention weights per hop and per head from Transformer, our sropeer-uulihan Objects attended in every hop are highlighted (whose frame border is colored accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). Please zoom in to see the details. Best viewed in color.
FAILURE CASES
We present a sample of the failure cases of Hopper-multhop in Figure 15 Generally, video is more difficult if: (1) there are similar looking objects present simultaneously (especially the object is similar snitch or another cone in the video); (2) wrong hop or 2 identified; (3) critical moments are occluded (e.g. in Figure [5] when the smitch becomes occluded, it is contained by the brown cone); (4) complex object interactions such as recursive containment along with container moving (since such case usually has the last visible snitch very early). Hopper-multihg, fails in the first scenario due to the error made by the object representation and detection module, which can be avoided by using fine-tuned object detector model. Hopper-multihop fails in the second scenario can attribute to the error made by the object detector, tracker, our heuristics, or capability of the saadequately-trinne
Multi-hop Transformer. The third scenario is easy even for humans under FPS. thus increasing FPS with extra care might case the problem. The last scenario requires more sophisticated multi-step reasoning, thus changing the minimal number of hops of the Multi-hop
19

---

Published as a conference paper at ICLR 2021
(a) Qualitative result via visualizing the most attended object(s) of every hop.
BEGINNINNG...
FRAME! !
FRAME
FRAME 3
FRAME
FRAME: 5
FRAME 6
FRAME
FRAME
FRAME
FRAME ID
FRAME
FRAME F2
FRAME 13
top 1 Hose 2
Hop Head 3,
Hop Head
Hop 4 Reed 3,
Wop 5 Hoad I.
(Containment) ) In the end, the snitch is inside of the green cone.
(b) Attention weights visualization.
L.e., track frane Hop Head 1: Video CATEM, 892274 sestly attends the 74th clement from the object tracks the menory sesuence, L.s., track frame Htp Head 2: Video CATERh 832274 sostly attends to the 74th eleeent from the object tracks in the menory sequence, 1.t., track frame? 9
Video GATERN_83223
attends 74th eteeent object tracks
60
300
120
Hop 2:
Video CATER, mastly attends 127th element object tracks the nenory sequence, L.c., track 18 frame Video CATERA, 832274 attends 127th etement object tracks nemory sequence, i.m.,
10
40
100
120
Hop 3: Video CATERA 832274 mastly attends to the 128th elenent from the object tracks in the nenery sequence, tet., track 10 at frame 11 Hop Head 1: Video CATEsh, 832274 mostly attends to the 128th elenent from the abject tracks in the menory sequence, Ieter track 10 frame 11 128th etesent the object tracks the neaory sequence, Iet., track frame
Hop 2: Video GATERn_ 692274 nostly attends
I.e., track 18 trame 12 Hop Head I: Video CATERA 892274 nostly attends the 129th elenent from the object tracks In the accory sequence, Let., track 10 frane 12 mtp Head 2: Video 69227 mostly attends to the 129th clenent from the object tracks in the nceory sequence, i.c., track 10 at frame 12
Hop d: Video CATERA, 832274 mostly attends 129th etesent free object tracks the
100
object Hop Head I: Video CATERA, 832274 attends 130th elenent object tracks
neesry sequence, tum., track 10 frame 13 Hop Head 2: Video CATEM)_892224 692274 mostly attends the 130th etenent from the object tracks in the aceory sequence, Lic., track 18 frome 13
Figure 9; We visualize the attention weights per hop and per head from Transformer, in Hopper-multie Objects attended in every hop are highlighted (whose frame border colored accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). Please zoom in to see the details. Best viewed in color.
Transformer into larger number with self-supervision for the intermediate hops handle the long hops should help in solving this scenario. Overall, an accurate backbone, object detector, tracking method, or the heuristics to determine visibility and the last visible snitch's 's immediate container (or occluder) will help improve the performance of Hopper-multihg enhancing Hopper- for these challenges and verify our hypothesis in our future work.
We would like to focus on
G THE CATER-H DATASET
G.1 BASICS: CATER
provides a diagnostic video dataset that requires spatial and temporal understanding to be solved. It is built against models that take advantage of wrong scene biases. With fully observable and controllable scene bias, the 5, 500 videos in CATER are rendered synthetically at 24 FPS (300-frame 320x240pxx using a library of standard 3D objects: 193 different object classes in total which includes 5 object shapes (cube, sphere, cylinder, cone, snitch) in 3 sizes (small, medium, large), 2 materials (shiny metal and matte rubber) and 8 colors. Every video has small metal snitch (see Figure []] There is large "table" plane on which all objects are placed. At a high level, the dynamics in CATER videos are in analogy to the cup-and-ball magic routine A subset of 4 atomic actions ('rotate', 'pick-place', 'slide" and "contain') is afforded by each object. See Appendix G.2 for defimition of the actions. Note that 'contain" is only afforded by cone and recursive containment possible, i.e., a cone can contain a smaller cone that contains another object. Every video in CATER is split into several time slots, and every object in this video randomly performs an action in the time slot (including 'no action'). Objects and actions vary across videos. The "table" plane is divided into 6 x 6 grids (36 rectangular cells), and the Snitch
CATER GGirdaar& & Ramanan 2020)
https:://en.wikiipeet
ba1
20

---

Published as a conference paper at ICLR 2021
(a) Qualitative result via visualizing the most attended object(s) of every hop.
BEGINAIING...
1
FRAME 2
FRAME 3
FRAME
FRAME 5
FRAME
FRAME 7
FRAME
FRAME q
FRAME 10
FRAME IT
FRAME 12
FRAME 13
Eep 1 fiedd 4.
Hop nead Hop Hand
top A Heed A, 2
Hop Road 4,
( (Recur sive containment) In the end, the snitch is inside of the yellow cone.
(b) Attention weights visualization.
40
100
120
Hop 2: Video CATER 337288 nostly attends to the 61ta alesent from the object tracks in the nemory aequence, L.e., track at frame Hop 1 Heed video caremh, 237288 eastly attends to 61th element from the soject tracks the nemory sequence, L.s., track frame neeory sequence, 1.t., track
Heed Video CATERA, costly attends
object
120
Hop 2: Video CATEM 837280 nostly attends to the 181th alement free the object tracks in the menory sequence, L.c., track frane 10
Hop 2 Heed 1: Video CATERn 237280 nostly attends to the 181th elenent from the object tracks in the nenory sequenee, sun., track at frame 10 ttp 2 Heed 2: Video CATERA 337280 eostly attends to the 181th elenest from the abject tracks in the nenory sequesce, 1ot., track frame
Video CATEM
attends 182th clement
tracks
track frane track frame 11
Heed Video CATERA 337280 costly attends 182th elenent the object tracks the top 3 leed 2: Video CATERA 337288 037280 costly attends to the 182th elenent from the object tracks in the nenory sequesee, tot., track frane 11
100
100
attends 183th elesent object tracks nenory sequence, iet., track frane 4 Head 2: Video CATTER_837220 937200 mostly attends to the 185th slenent frem the object tracks in the sequence, Li.., track frane 12
OATERA
Head CATERA 837288
40
Ude
Lio
Hop St Video CATERA #37288 nostly attends to the 78th etenent from the object tracks in the nenory sequence, I.c., track at frane 13 Head Video
EATER mostly attends 78th clesent from object tracks the nceory sequence, Lib., track frame neeory
Lib.,
Figure 10: We visualize the attention weights per hop and per head from Transformer, in our Hopper- multihop. Objects attended in every hop are highlighted (whose frame border is colored accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). Please zoom in to the details. Best viewed in color.
Localization task is to determine the grid that the snitch is in at the end of the video, as a single-label classification task. The task implicitly requires the understanding of object permanence because objects could be occluded or contained (hidden inside of) by another object.
G.2 DEFINITION OF ACTIONS
We follow the definition of the four atomic actions in Girdhar & Ramanan 2020). Specifically:
1. 'rotate': the 'rotate' action means that the object rotates by 90% about its perpendicular or horizontal axis, and is afforded by cubes, cylinders and the snitch.
2. 'pick-place': The 'pick-place' action means the object is picked up into the air along the perpendicular axis, moved to new position, and placed down. This is afforded by all objects.
3. 'slide': the 'slide" action means the object is moved to a new location by sliding along the bottom surface, and is also afforded by all objects.
4. 'contain': 'contain' is a special operation, only afforded by the cones, in which cone is pick-placed on top of another object, which may be a sphere, a snitch or even a smaller cone. This allows for recursive containment, contains another object. Once a cone 'contains' an object, the "slide" action of the cone effectively slides all objects contained within the cone. This holds until the top-most cone is pick-placed to another location, effectively ending the containment for that top-most cone.
cone can contain a smaller cone that
21

---

Published as a conference paper at ICLR 2021
(a) Qualitative result via visualizing the most attended object(s) of every hop.
BEGTNNING....
FRAVE
FRAME
FRAME
FRAME
FRAME
FRAME
S,
FRAME?
FRAME
FRAME
FRANE 10
FRAME
FRAME 12
FRAME 13
A
Hop Head 1.
Read 5
(Snitch becomes not visible very early) In the end, the snitch inside of the medium red rubber cone.
(b) Attention weights visualization.
ide
Hop 1: Video CATERA 031371 aestly attends to the 27th element from the object tracks in the nemory sequence, L.s., track at frane Hep Head 1: Video CATERA 831371 mostly attends to the 27th element from the object tracks in the nceory sequence, L.b.. track 3 at frame Hop Head 2: Video CATERa031371 eostly attends to the 27th element free the abject tracks in the nemory sequence, 1.b., track at frame
Hop Head Video CATERS 011371 mostly attends the 53th element froe the tracks the Hop Head 2: Video CATERA 031371 aostly attends to the 93th element free the object tracks in the neeory sequence, L.tr, track B frame
frame
20
100
120
Hop Video CATERA, 031171 sestly attends to the 127th elenest free the object tracks mesory sequence, 1.6., track frame netory sequence, 1.6..
mostly
elenent object
Head CATERA 031371 attends
20
i6e
Hep 4: Video CATERA #31371 sestly attends to the 120th elenest free the abject tracks in the menory sequence, 1.e., track 18 at frame 12 Hop Head 1: Video CATERA #31371 mostly attends to the 129th element from abject tracks the netory sequence, I.c.. track 10 at frame CATERO 031377 mostly attends to the etement from object tracks nenory sequence, Lic., track
Hop
Hop S: Video CATERA 031371 mestly attends to the 130th clanest free the object tracks in the nenory sequence, L.te, track 18 frame 13 Hop Head 1: Video CATERA 831371 msstly attends to the 138th element from the abject tracks in the necory sequenee, 1.t., track 10 at frame 13 Hop Head 2: Video CATERN_01131 mestly attends to the 138th element from the object tracks the necory sequence, Sit.. track trame
Figure 11: We visualize the attention weights per hop and head from Transformer, in our Hopper- multihop. Objects attended in every hop are highlighted (whose frame border is colored accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). Please zoom to see the details. Best viewed color.
G.3 DATASET GENERATION PROCESS
The generation of the CATER-h dataset is built upon the CLEVR (Johnson et al.) 2017a) and CATER (Girdhar Ramanan 2020) codebases. Blender used for rendering. The animation setup is the same as the one in CATER. A random number of objects with random parameters are spawned at random locations at the beginning of the video. They exist on a 6 x 6 portion of a 2D) plane with the global origin in the center. Every video has snitch, and every video is split into several time slots. Each action is contained within its time slot. At the beginning of each slot, objects are randomly selected to perform a random action afforded by that object (with no collision ensured). Please refer to Girdhar & Ramanan (2020) for more animation details.
In order to have a video dataset that emphasizes on recognizing the effect of the temporal variations on the state of the world, we set roughly equal number of video samples to have the last visible snitch along the temporal axis. In order to obtain such a dataset, we generated a huge number of videos, computed the frame index of the last visible snitch in every video under 1 FPS (13 frames per video). Then, for every frame index we obtained the set of videos whose last visible snitch at frame index i, and finally, randomly chose 500 and more videos from this set and discarded the rest. Eventually, the total number of videos in CATER-h is 7, 080. We split the data randomly in 70 30 ratio into training and test set, resulting in 5, 624 training samples and 1, 456 testing samples.
G.4 CATER-H V.S. CATER
Figure 13 compares the CATER-h dataset and the CATER dataset.
22

---

Published as a conference paper at ICLR 2021
TRACK1
TRACK 2
TRACK TRACK
TRACK
TRACK
TRACK 7
TRACK TRACK TRACK
FRAME 13 FRAME 12 FRAME fI FRAME 10 FRAME 9 FRAME 8 FRAME 7 FRAME & FRAWE 5 FRAME 4 FRAME 3 FRAME 2 FRAME!
Figure 12: maacking-integraln object representation visualization. Hopper utilizes muckkngginneegren object representations since tracking can link object representations through time and the resulting representations are informative and consistent. We visualize the object track representations of video "CATERh_O5AA 0541000 (attention weights from Hopper-multhhop for this video are shown Figure Here, every column is for a track and every row is for a frame. The bounding box representation are plotted (104 is the object class label for Ø, i.e., none object, and 1410 is the object class label for the snitch). As shown in the figure, the tracks that are obtained from our designed Hungarian algorithm are perfect but acceptable since having perfect tracking here is the goal of this paper. Our model Hopper- multihop takes in the (imperfect) object track representations (along with the coarse-grained frame track) as the source input to the Multi-hop Transformer, and then further learns the most useful and correct taie-oriented track information implicitly (as shown in Figure 6). Hopper-mulihop preforms 5 hops of reasoning for this video; objects attended in every hop are highlighted (whose frame border is colored accordingly: Hopl, Hop2, Hop3, Hop4, and Hop5). Please zoom
object class label computed from object
the details. Best viewed in color.
G.5 TRAIN/TEST DISTRIBUTION
Figure 14 shows the data distribution classes in CATER-h.
G.6 OTHER POTENTIAL DATASET BIAS
doopeer-ranssfrmen (last frame)' still has a relatively high accuracy on CATER-h. We hypothesize that the reason it has 41.8% Top-1 accuracy CATER-h might be due
As shown in Table 2
23

---

Published as a conference paper at ICLR 2021
CATER
58%
CATER-h
Ratio of dataset
Ratio of dataset
7% 9 8967 7
cam-
14%
2%%0%1%9999 %2 %3 2% 4%
6%%6%
1 Frame index of when snitch became last visible in CATER-h
0:0
Frame index of when snitch became last visible in CATER
Figure 13: Histogram of the frame index of the last visible snitch every video. We find that CATER highly imbalanced for Snitch Localization task in terms the temporal cues: e.g., snitch is entirely visible at the end of the video for 58% samples. This temporal bias results in high-accuracy even if it ignores all but the last frame of the video. Our dataset CATER-h addresses this issue with balanced dataset.
to other dataset biases (apart from the snitch grid distribution bias, and temporal bias that CATER has). Upon further investigation, we identify one type of additional bias, the "cone bias", i.e., snitch can only be contained by a cone in the videos of CATER and CATER-h.
In order to verify the existence of the "cone bias", we compute the accuracy if we make a random guess among the grids of cones that not covered by any other cones, for all test videos whose snitch is covered in the end. This gives us 48.26% Top-1 accuracy. This shows that the "cone bias" does exist in the dataset. The so-called "cone bias" comes from the nature of objects used in CATER and the fact that only the cone can carry the snitch (thus, it is closer to a feature of the dataset, rather than being a "bias" per se). Furthermore, because of the animation rules of CATER, there might exist other dataset biases, such as bias in terms of object size and shape, etc., which are hard to discover and address. This highlights the glaring challenge in building fully unbiased (synthetic or real) dataset. CATER-h addresses the temporal bias that CATER has. A model has to perform long-term spatiotemporal reasoning in order to have a high accuracy on CATER-h.
H IMPLEMENTATION DETAILS
H.1 HOPPER
We introduce the implementation of Hopper-multhhpp in this section. For both training and testing, we only used FPS (frames per second) to demonstrate the efficiency of our approach. This means only have 13 frames per video. Note that trained the video query representation and recognition part (Multi-hop Transformer along with the final MLP) end to end.
The CNN backbone we utilized is the pre-trained ResNeXt-101 (Xie et al. (217)) model from Ma et al. 2018). We trained DETR (Carion et al. 2020 on LA-CATER (Shamsian et al. 2020 which dataset with generated videos following the same configuration to the used by CATER, but additional ground-truth object bounding box location and class label annotations are available [Shamsian al. (2020) predicts the bounding box of snitch in the video given the supervision of the bounding box of the snitch in 300 frames). We followed the settings in Carion et al. (2020) to set up and train DETR, c.g. stacking transformer encoder layers and 6 transformer decoder layers, utilizing the object detection set prediction loss and the auxiliary decoding loss per decoder layer. d is 256, N The MLP for recognizing the object class label is one linear layer and for obtaining the object bounding box a MLP with 2 hidden layers with d neurons. After DETR was trained, we tested it on CATER to obtain the object representations, predicted bounding boxes and class labels. For tracking, we set Ae- = 1, Ab = 0 because under a low FPS, using the bounding boxes for tracking counterproductive, and it yielded reasonable results. Then, we trained the video recognition part, i.e., Multi-hop Transformer along with the final MIP, end to end with Adam (Kingma & Ba 2014) optimizer. The final MLP we used is one linear layer that transforms the video query representation e of dimension into the grid class logits. The initial learning rate was set to 10-4 and weight decay to 10-3 The batch size was 16. The number of attention heads for DETR was set to 8 and
and C' 193. The initial N object query embeddings are learned.
24

---

Published as a conference paper at ICLR 2021
for the Multi-hop Transformer set to 2. Transformer dropout rate was set to 0.1. We used multi-stage training with the training methods proposed. Moreover, we found that DETR tends to predict smitch for every cone the "table" plane when there is visible snitch in that frame. To mitigate this particular issue of the DETR object detector trained on Shamsian et al. (2020), we further compute object visibility map V E NTx1, which is binary vector and determined by a heuristic: an object is visible if the bounding box of the object is not completely contained by any bounding box of another object in that frame. The 'Masking(" function uses V by considering only the visible objects.
H.2 DESCRIPTION OF REPORTED CATER BASELINES
For CATER, we additionally compare our results with the ones reported by Girdhar & Ramanan (2020) from TSN (Wang et al. 2016), I3D (Carreira & Zisserman 2017). NL (Wang et al., (018)) as well as their LSTM variants. Specifically, TSN (Temporal Segment Networks) was the top performing method that is based on the idea of long-range temporal structure modeling before TSM and TPN. Two modalities were experimented with TSN, i.c., RGB or Optical Flow (that captures local temporal cues). I3D inflates 2D ConvNet into 3D for efficient spatiotemproal feature learning. NL (Non-Local Networks) proposed a spacetime non-local operation as a generic building block for capturing long-range dependencies for video classification. In order to better capture the temporal information for these methods. Girdhar & Ramanan (2020) further experimented with a 2-layer LSTM aggregation that operates on the last layer features before the logits. Conclusions from Girdhar & Ramanan (2020) are: (1) TSN ends up performing significantly worse than 13D instead of having similar performance which contrasts with standard video datasets; (2) the optical flow modality does not work well as the Snitch Localization task requires recognizing objects which is much harder from the optical flow; (3) more sampling from the video would give higher performance; (4) LSTM for more sophisticated temporal aggregation leads to major improvement in performance.
H.3 BASELINES
We introduce the implementation of the baselines that we experimented with in this section. First, for our Hungarian tracking baseline, for every test video, we obtain the snitch track (based on which track's first object has snitch as its label) produced from our Hungarian algorithm, and project the center point of the bounding box of the last object in that track the 3D plane (and eventually, the grid class label) by using homography transformation between the image and the 3D plane (same method used in Girdhar& & Ramanan (2020). We also try with the majority vote, i.e., obtain the snitch track as the track who has the highest number of frames classified as snitch. We report the majority vote result in Table [ and 2 because is a more robust method. The results of using the first frame of our Hungarian tracking baseline are 31.8% Top-1, 40.2% Top-5, 2.7 L1 on CATER, and 28.2% Top-1, 36.3% Top-5, 2.8 L1 on CATER-h.
We used the public available implementation provided by the authors (Yang et al. 2020) for TSM and TPN. Models were defaultly initialized by pre-trained models on ImageNet (Deng et al., 2009). The original ResNet (He et al. 2016) serves as the 2D backbone, and the inflated ResNet (Feichtenhofer et al. 2019) as the 3D backbone network. We used the default settings of TSM and TPN provided by Yang et al. (2000), TSM-50 (2D ResNet-50 backbone) settings that (Goyal et al. 2017). which are also the the well as TPN-101 (I3D-101 backbone, i.e. 3D ResNet-101)
they used to obtain results on Semething-Somettiee
protocols used in Lin al. (2019), with multi-depth pyramid and the parallel flow that they used to obtain results on Kinetics (their best-performing setting of TPN) (Carreira & Zisserman, 2017). Specifically, the augmentation of random crop. horizontal flip and dropout of 0.5 were adopted to reduce overfitting. BatchNorm (BN) was not frozen. A momentum of 0.9, a weight decay of 0.0001 and a synchronized SGD with the initial learning rate 0.01, which would be reduced by factor of 10 at 75, 125 epochs (150 epochs in total). The weight decay for TSM was set to 0.0005. TPN used auxiliary head, spatial convolutions in semantic modulation, temporal rate modulation and information flow (Yang et al.) 2020). For SINet, we used the implementation provided by Ma et al. (2018). Specifically, image features for SINet were obtained from a pre-trained ResNeXt- 101 (Xie et al. (017)) with standard data augmentation (randomly cropping and horizontally flipping video frames during training). Note that the image features used by SINet are the same as the ones used in our Hopper. The object
25

---

Published as a conference paper at ICLR 2021
features were generated from Deformable RFCN (Dai et al. 2017). The maximum number of objects per frame was set to 10. The number subgroups of higher-order object relationships (K) was set to 3. SGD with Nesterov momentum were used as the optimizer. The initial learning rate was 0.0001 and would drop by 10x when validation loss saturates for 5 epochs. The weight decay was 0.0001 and the momentum Hopper-trane tranforrra,sformeer and Hopper-sinet used the Adam optimizer with total of 150 epochs, initial learning rate of 10-4, a weight decay of 10-3, and a batch size of 16. Same as our model, the learning rate would drop by factor of 10 when there has been no improvement for 10 epochs on the validation set. The number of attention heads for the Transformer (and troper-transfommer) was set to 2, the number of transformer layers was set to 5 to match the hops in our Multi-hop Transformer, and the Transformer dropout rate was set 0.1. For OPNet related experiments, we used the implementation provided from authors (Shamsian et al., 2020). We verified we could reproduce their results under 24 FPS on CATER by using their provided code and trained models.
0.9. The batch size was 16 for these 3 baselines. Transformer,
For the Random baseline, it is computed the average performance of random scores passed into the evaluation functions Girdhar& & Ramanan (2020). For the Tracking baseline, we use the DaSiamRPN Specifically, the ground truth information of the starting position of the snitch was first projected to screen coordinates using the render camera parameters. fixed size box around the snitch is defined to initialize the tracker, and run the tracker until the end of the video. At the last frame, the center point of the tracked box is projected to the 2D plane by using a homography transformation between the image and the 2D plane, and then converted to the class label. With respect to TSN, I3D, NL and their variants, the results were from Girdhar & Ramanan 2020), and we used the same train, val split as theirs when obtaining our results on CATER.
implementation from Girdhar & Ramanan (0020)
0.030
0,025
Rabio of troin Set
0.015
0.010
CATER-h Train Set
0.030
Ratio of test set
0.625
0.020
0.000
0.005
0.000
(b) CATER-h Test Set
Figure 14: Data distribution over classes in CATER-h.
26

---

Published as a conference paper at ICLR 2021
BEGDNNNNG...
FRAMET
FRAME
FRAME 3
FRAME 4
FRAME
FRAME
Hip meod
Read
FRAME
FRAME
FRAME
FRAME 10
FRAME 11
FRAME
FRAME F3
Head
Hend
Rend
(a) contained brown
[the
4),
that, the
the
BEGINNING...
FRAME
FRAME
FRAME 3
FRAME 4
FRAME
FRAME
HAp 3. Iood 3,
PUD Rood
FRAME 7
FRAME
FRAME 9
FRAME 10
FRAME !1 Iep Hese
FRAME
FRAME r3
Hoy Wood
Hog 4
Hoad
Hop Hond 5,
(b) Snitch is contained the contained the yellow twice (frame and frame 9), but the the cyan is visible the snitch inside of the cyan
frame (the last visible snitch is in frame 2). After that,
Figure 15: Failure cases. Hopper produces wrong Top-1, Top-5 prediction terrible L1 results for these failure cases. Similarly, we highlight the attended object per hop and per head (Hopl, Hop2, Hop3, Hop4, and Hop5). Case (a). "CATERh :082295' the occlusion has made the Snitch Localization task extremely difficult since when the snitch got occluded was contained by the brown cone. Mcanwhile, Hopper fails attend to the immediate container of the last visible snitch (should be the hrown cone frame in Hop 2. Case (b). "CATERh 022965': the snitch was not visible very early in video (at frame 3), recursive containment, as well as the presence of two similar looking cones have made the task extremely difficult. Hopper fails to attend to the correct object in Hop (should be the yellow cone).
I RELATED WORK (FULL VERSION)
In this section, we provide detailed discussion of related work. Our work is generally related the following recent research directions.
Video understanding & analysis. With the release of large-scale datasets such Kinetics (Carreira & Zisserman, 2017). Charades (Sigurdsson et al. 2016). and Something something (Goyal al. (017), the development
video representation has matured quickly in recent years. deep visual features from 2D ConvNets with LSTMs for temporal aggregation (Donahue et al. 2013) Yue-Hei Ng et al. 2013). As natural extension to handle the later proposed (Ji al. 2012 Taylor al. 2010) but with the issue of inefficiency and huge increase in parameters. Using both RGB and optical flow modalities, Two- stream networks (Simonyan & Zisserman| 2014 Feichtenhofer al. 2016) and Two-Stream Inflated 3D ConvNets (I3D) (Carreira & Zisserman (0017) were designed. With the emphasis on capturing the temporal structure of video, TSN (Wang al. 2016), TRN (Zhou al. 2018). TSM (Lin et al.) 2019) and TPN (Zhou et al. 2018) were successively proposed and gained considerate improvements. Recently, attention mechanism and Transformer design (Vaswani et al., 2017) have been utilized for more effective and transparent video understanding. Such models include Non- local Neural Networks (NL.) (Wang et al. (008) that capture long-range spacetime dependencies, SINet (Ma et al. 20188) that Iearns higher-order object interactions and Action Transformer (Girdhar (ct al. 2019) that learns to attend to relevant regions the actor and their context. Nevertheless, instead of the reasoning capabilities, existing benchmarks and models for video understanding and analysis mainly have focused on pattern recognition from complex visual and temporal input.
Early approaches
video data, 31D ConvNets
Visual reasoning from images. To expand beyond image recognition and classification, research on visual reasoning has been largely focused on Visual Question Answering (VQA). For example, diagnostic VQA benchmark dataset called CLEVR (Johnson al. 2017a) was built that reduces
27

---

Published as a conference paper at ICLR 2021
spatial biases and tests a range of visual reasoning abilities. There have been few visual reasoning models proposed (Santoro et al., 2017 Perez et al., [017] |Macharka et al. 2018 Suarez et al, 2018 Aditya et al. 2018). For example, inspired by module networks (Andreas et al. 2016b a Johnson et al. (201 7b) propose a compositional model for visual reasoning on CLEVR that consists of a program generator that constructs explicit representation of the reasoning process to be performed, and an execution engine that executes the resulting program to produce an answer; both are implemented by neural networks. Evaluated on the CLEVR dataset, HHu et al. (2017) proposed N2NMNs, i.e., Bnd Module Networks, which learn to reason by directly predicting the network structures while simultaneously learning network parameters. MAC networks (Hudson & Manning, 2018), that approach CLEVR by decomposing the VQA problems into a series of attention-based reasoning steps, were proposed by
by stringing the MAC cells end-to-end and imposing structural constraints to effectively learn to perform iterative reasoning. Further, datasets for real- world visual reasoning and compositional question answering are released such as GQA (Hudson & Manning 2019b). Neural State Machine (Hudson & Manning 2019a) was introduced for real-world VQA that performs sequential reasoning by traversing the nodes over a probabilistic graph which is predicted from the image.
Video reasoning. There has been a notable progress for joint video and language reasoning. For example, in order strengthen the ability to reason about temporal and causal events from videos, the CLEVRER video question answering dataset (Yi et al. 2020) was introduced being a diagnostic dataset generated under the same visual settings as CLEVR but instead for systematic evaluation of video models. Other artificial video question answering datasets include COG (Yang et al., 2018a) and MarioQA (Mun et al. 2017). There have been also numerous datasets that are based on real-world videos and suman-genneatte
questions such as MovieQA (Tapaswi et al. 2016). TGIF- QA (Jang et al., 2017). TVQA (Lei al.] 2018) and Social-IQ (Zadeh et al. 2019). Moving beyond the question and answering task, CoPhy (Baradel et al. 2019) studies physical dynamics prediction in a counterfactual setting and a small-sized causality video dataset (Fire & Zhu 2017) was released to study the causal relationships between human actions and hidden statuses. To date, research on the general video understanding and reasoming is still limited. Focusing on both video reasoning and general video recognition and understanding, we experimented on the recently released CATER dataset (Girdhar & Ramanan 2020), a synthetic video recognition dataset which is also built upon CLEVR focuses on spatial and temporal reasoning well localizing particular object of interest.
There also has been significant research in object tracking, often with an emphasis on occlusions with the goal of providing object permanence (Bewley et al. 2016: Wojke et al., 2017 Li et al. 2018 Zhu et al. 2018 Wang et al., (019b).. Traditional object tracking approaches have focused on the fine-grained temporal and spatial understanding and often require expensive supervision of location of the objects in every frame (Shamsian et al.) (020). We address object permanence and video recognition on CATER with model that performs macking-integrann object-centric reasoning for localizing object of interest.
Multi-hop reasoning. Reasoning systems vary in expressive power and predictive abilities, which include systems focus on symbolic reasoning (e.g., with first order logic), probabilistic reasoning, causal reasoning, etc (Bottou 2014). Among them, multi-hop reasoning is the ability to reason with information collected from multiple passages to derive the answer (Wang et al. 2019a). Because of the desire for chains of reasoning. several multi-hop datasets and models have been proposed for nature language processing tasks (Dhingra et al. 2020 Dua et al. 2019 Welbl al., 2018 Talmor& & Berant 2018 Yang et al.] al]] (0188)). For example, Das et al. (2016) introduced a recurrent neural network model which allows chains of reasoning over entities, relations, and text. Chen al. (2019) proposed a two-stage model that identifies intermediate discrete reasoning chains over the text via an extractor model and then separately determines the answer through a BERT- based answer module. WWang et al. (2019a) investigated that whether providing the full reasoning chain of multiple passages, instead of just one final passage where the answer appears, could improve the performance of the existing models. Their results demonstrate the existence of the potential improvement using explicit multi-hop reasoning- Multi-hop reasoning gives us a discrete intermediate output of the reasoning process, which can help gauge model's 's behavior beyond just final task accuracy (Chen) ct al. 2019). Favoring the benefits that multi-hop reasoning could bring, in this paper, we developed video dataset that explicitly requires aggregating clues from different spatiotemporal parts of the video and a multi-hop model that automatically extracts a step- reasoning chain. Our proposed Multi-hop Transformer improves interpretability and imitates natural way of thinking.
28

---

Published as a conference paper at ICLR 2021
The iterative ceanntion-bssel neural reasoning Locatello al. 2020 Hudson & Manning 2019a) with a contrastive debias loss further offers robustness and generalization.
J MHT ARCHITECTURE
We illustrate the architecture of the proposed Multi-hop Transformer (MHT) in Figure 16
Final Video Query Representation
H: the total number of hops
Layer Norm
Attentional Feature- Gating
Hadamard Product
Transformer
Layer Norm
Linear + Sigmoid
Addition
Transformer
Feed-Forward
Layer Norm
Layer Norm
Addition
Layer Norm
Feed-Forward
Addition
Multi-Head
Layer Norm
Attention
Feed-Foorwad
Repeat
Layer
Addition
for
Addition
Multi-Head
Layer Norm
Layer Norm
H - times
Addition
Feed-Forward
Layer Norm
Layer Norm
Multi-Head Self-Attention
Multi-Head Self-Attention
Addition
Addition
Tgt: Video
Query
Self-Attention
Self-Attentinn
Updated
Sie: Helper Information
Tgt: Attention Candidates
Masking
Helper Information
Softargmax A)
Attention Candidates
Source Sequence [Ty, Tol
Target: Video Query E
Architecture the Multi-hop Transformer (MHT) that learns a comprehensive video query representation and meanwhile encourages multi-step compositional long-term reasoning of a spatiotemporal sequence. As inputs to this module, the 'Source Sequence" is [T,, T.J. where [.] denote E Rlxd 'Final Video Query Representation' is e E Riddd A E N refers to attention weights from the encoder-decoder multi-head attention layer in Transformer, averaged over all heads. To connect this figure with Algorithm [ 'Ohtain Helper Information & Attention Candidates' refers line for 'Helper Information' H. (or line 7 for the first iteration). and line 9 for the 'Attention Candidates" 24 E NTxx Dimensionality 'Helper Information' H x d hop and N x d for the rest of the hops. Transformer/ and Transformer, are using the original Transformer architecture (Vaswani et al.] (017). 'Attentional Feature-based Gating' corresponds to line 11. 'Updated Attention Candidates' is Uugeste E RRNTH NTxd 'Masking" corresponds line 12. Its output, 'Masked Attention Candidates' is Unaa E R2 NTxx (however. certain tokens out of these NT tokens are masked and will have attention weights). The last 'Layer Norm' after the Transformer, corresponds to line 16. H denotes the total number of hops, i.c., total number of iterations, and it varies across videos. Please refer to Section for details.
Figure 16:
concatenation; and the 'Target Video Query'
29


Reviews:
[{"review_id": "xraK-CJP_bZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-hop transformer method for the video-based object permanence task. The proposed method performs multi-hop reasoning via the encoder-decoder architecture of transformers over critical frames in the video. To mitigate the problem of lacking ground truth for the middle hops, the paper proposes some interesting training tricks. Overall, the paper is well organized and easy to follow.\n\nReasons to accept the paper:\n1. The paper extends multi-hop reasoning techniques that are widely used in NLP domain to video domain, which may inspire other researchers working on other video-based tasks that require multi-hop reasoning.\n2. The paper proposes a new benchmark dataset, which requires longer reasoning chains.\n3. Experiments on the CATER dataset achieve state-of-the-art performance on the object permanence task.\n\nReasons to reject the paper:\n1. The paper claims to address the problem of biased video reasoning, however, all the experiments in this paper are done on a synthetic dataset and it's not clear why such dataset can rule out the possibility of having biases.\n2. The proposed method is somewhat dedicated to a proposed object permanence task, which may not be general enough to be extended to other video-based tasks.\n3. The paper motivates the task using real-world examples, such as asking \"which car was responsible for the accident\". However, such question seems much harder to answer even with the proposed technique for object permanence task. Also, the paper only shows experimental results on synthetic dataset, and leaves experiments on real-world video datasets as future work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "q9np37uBpRA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents Hopper, a method that performs multi-hop reasoning to address the problem of object permanence in videos.\n\n### Strengths:\n- Very good results in the Snitch Localization task. Good and sensible baselines.\n- Good ablations, experiments and visualizations. Also good justification of results both for Hopper and for baselines.\n- They introduce a method that is modular, and clearly separates several stages of processing (while still making everything end-to-end trainable). This provides good understanding of the system, and makes ablations and comparisons with baselines easier (for example, baselines can work a the per-module level).\n- The multi-hop transformer reasoning module they introduce is intuitive, and it is very well executed.\n- Overall well written and good positioning among state of the art literature.\n- Contribution of a new CATER-h dataset that corrects some biases of CATER for the specific task the paper is solving. Good explanation of the need for this dataset, both by analyzing the dataset statistics, and by looking at the results of the baselines on the two datasets.\n\n### Weaknesses:\n- A lot of supervision and limited dataset.\n\t- Object annotation class and bounding boxes are available.\n\t- Fixed set of objects and attributes, with no variation across samples (synthetic data).\n- The tracker learns based on the class. If there are two objects belonging to the same class, it will probably not distinguish them (see Figure 15a). Probably humans could watch a video and follow the snitch even if all objects looked exactly the same, just by following their movements, and this method is, _by construction_, not doing this. Therefore, Figure 15a is not just some very hard case, but a case that the system as it stands cannot solve. This implies that the tracking done by the system is a very weak form of tracking, and it is more a relabeling of objects. Actually, to solve the task the system only needs to know 1) where the snitch was before and 2) where did the container move (just reidentify by classification). While this can be improved, the main contribution of the paper is in the step after this, so it is OK to leave it for future work.\n- In some cases the method feels very specific for the presented task and dataset. This is acceptable to some extent because it is a hard problem and it doesn't have to scale right away, but I am concerned it is too specific about this formulation and cannot be used. There are a lot of specific intuitions for this specific problem and setting (not interesting for any other case). See for example page 6. More specifically, an example is the heuristic for computing the occluder, which would fail if the snitch has moved between frames, and it is visible but somewhere else (and this case is not even generalizing to a slightly different task, but a potential situation in the current dataset). While these heuristics are optional and the authors present ablations, the feeling is general for the whole method. \n- Some parts of the method could use more clarity:\n\t- Figure 3 is confusing, as it does not exactly follow Algorithm 1. It would be convenient to use the same names as in the rest of the paper for the layers and the inputs/outputs.\n\t- Why the attentional feature-based gating (line 11 algorithm 1) is necessary? Why doesn't Transformer_f directly output the final U_update?\n\t- Why two transformers are necessary, actually? The masking could be applied at the beginning of the first one and just use one. Note this is not the same as the \"hopper-transformer\" in the baselines.\n\t- t is computed with a softargmax. However, it is used in places that require it being a hard number, for example in line 2 in the algorithm. This implies these places have to detach the gradient from t. Apart from line 2, is there any other place that requires this? What about lines 12 (Masking) and 5 (Extract). My understanding of the paper tells me that at least the Masking() module requires a hard t. In what case is the gradient actually propagated through t?\n\t- Teacher forcing: the paper argues that hop 2 should focus on the first occluder. What about hop 3, what is it supposed to predict?\n\t- For line 14 in the algorithm, are the attentions of the heads averaged? \n- About the 5 steps: Looking at the examples (eg Figure 15b) it looks like there are much more than just 5 key steps to follow. However, Hopper never uses H > 5 in the shown examples (while H=5 is actually the minimum number of hops that are allowed to the system). Why is that? It looks like as soon as it has a chance, the system tries to predict the last frame available, even if it is not the most convenient (it can always get later, to the last frame). This could make sense for CATER, but the shown examples are for CATERh . This is important because the main point of Hopper is that it can select where to attend. I would appreciate some intuitive explanation and statistics of the number of steps Hopper takes.\n- Unclear what the model is _really_ learning. While it is not learning temporal biases, there are potentially a lot of other biases it can be exploiting. It would be very interesting to have analyses to answer this question. Otherwise it is hard to believe the models are really \"solving\" this task.\n\n### Additional comments and questions:\n- When an object is occluded or contained by another one, the goal is to predict the position of the first object, or the position of the occluder?\n- Is the 1 fps chosen for any specific reason? The solution to the problem is actually different depending on the sampling rate. For example, it would be possible that in between frames there is a key move that we do not see.\n- \"Humans realize object permanence by identifying key or critical frames where objects become hidden\", and similar sentences throughout the paper. Citation?\n- I think the third sentence in the introduction refers to the first one. It is confusing because it reads as it is referring to the second one instead. I would move the second one to later when the authors talk about object permanence (line 20).\n- If the key point of the paper is the multi-hop transformer, an interesting comparison would be the same system without DETR, and instead using only the true class label and bounding box (remove steps A and B). At least to see how much these are a bottleneck, or how important they really are.\n- Analyses on learned attributes. The classes are combinations of 4 attributes. Why not learning them separately? Is the model creating different represenations for each specific combination? Would it generalize to a new object or to a new combination of attributes not seen during training?\n- Will the code for the paper and the cater-h dataset be released?\n\n### Final recommendation\nOverall, I believe this paper should be clearly accepted to ICLR, as the strengths outweigh the weaknesses. This paper is not the final solution to the object permanence task, and it has a lot of possible improvements, but it is a good step.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R0YZJu_mC2c", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe authors proposed a multi-hop transformer, which takes information encoded in forms of object track and image track as input, to reason over the critical frame sequence to locate the final location of the object of interest. Although I like the idea presented in the paper, I think there are several aspects that will strengthen the paper.\nPros:\n1.\tThe idea of using transformer in a recurrent manner for reasoning in videos is intuitive and interesting. \n2.\tThe authors show the effectiveness of the proposed model by superior quantitative results and the visualization of the most attended object in the inference process of one video sequence.\n3.\tThorough ablation is provided for the proposed multi-hop transformer.\n4.\tThe paper is written and presented well.\n\nCons:\n1.\tOne baseline comparison is missing. The tracking baseline seems to be from prior works rather than from the tracking results produced in the proposed frameworks. It is important to know how well the tracking component itself performs.\n2.\tMore ablation on the framework should be provided. The authors used the DETR for object detection. How does the final performance benefit from using this transformer-based object detection model? Is the proposed framework only compatible with transformer-based object detection?\n3.\tAlthough I like the visualization results, it is still a question whether the most attended object is the most important one in the final results or not [1]. It will be interesting to see the gradient visualization rather than the attention visualization.\n[1] Sofia Serrano, and Noah A. Smith. 2019. Is Attention Interpretable.arXiv preprint arXiv:1906.03731. \n\n\n\n==============\n\n\nI have read the authors' rebuttal information. The authors have addressed my concerns with additional ablation studies and experiments to verify the effectiveness of the proposed multi-hop transformer. And also some experiments are provided  to illustrate whether the most attended object is the most important one. \n\nTherefore, I am still standing on the previous justification for accepting the submitted manuscript.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2YAWx03TFRk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an architecture (Multi-Hop Transformer) for spatio-temporal reasoning in video, focusing on a localisation task for scenes where the object of interest is often occluded (Snitch Localisation task in CATER). The model extracts objects using an external object-detector and predicts objects’ trajectories using the Hungarian algorithm. The Multi-Hop Transformers learns to hop over unnecessary frames, by focusing only on a set of few critical steps (at each time step, the next critical frame is the one containing the most attended object). To alleviate the problems encountered during the training procedure (such as error propagation), several auxiliary training methods are proposed to guide the first few hops or to ensure contrastive debias. Additionally, a harder version of the existing CATER dataset is created, to alleviate the temporal bias existent in the previous version of the dataset.\n\nPro:\n- The “dynamic stride” proposed in the Multi-Hop Transformer is a novel and interesting idea, showing promising results in tasks requiring object permanence. Moreover, the combination of global features and object-centric representations is sound and suitable.\n- The ablation studies (Table 1-2 but also those from the supplementary material) are carefully designed, showing the importance of each component and offering a fair comparison between the methods. \n- The temporarily unbiased version of CATER dataset, where the last visible snitch is distributed uniformly across the length of the video, is crucial for improving the evaluation of different models in video reasoning tasks. \n\nCons:\nThe method is poorly described in the paper (especially in Section 4): the writing is very hard to follow, without a clear structure of ideas. Several details about the sub-modules are missing and the proposed techniques are not supported by any intuitive motivation.  Some example for this are the followings:\n- In the second layer, the Transformer_f module receives as input both U and H. Does it send messages across all types of edges (between tokens in U, between tokens in H and across them)? Same question for Transformer_s. \n- Since the video query should be a single token (the input has dimension 1xd, thus making me think that it is preserved along the algorithm since additional information about the intermediary shapes are not provided), what is the purpose of the first self-attention layer (column 4 in Fig. 3). What tokens should this layer combine? \nThe meanings of each latent representation are not provided (or presented too vaguely e.g. “by reasoning about the relations between the object entities and how would each object entity relate to the reasoning performed by the previous hop or global information”). From the paper, it seems that the \\epsilon variable starts by representing a query embedding, but becomes a video representation immediately after. Should that token be seen as an encoding for the entire video, or as a representation of the object of interest across the video (the snitch in this case)?\n- Why does the Transformer_f send messages between all time steps and not only the U_masked?\n- Since CLEVR_h has a uniform distribution of the last visible frame, why does the Hopper-transformer (last frame) have such a high performance (in Table 2)? Is it able to learn anything else besides the 7% cases when the snitch becomes last visible in the 13th frame?\n- The purpose of the tracking module (the Hungarian algorithm applied on top of predicted objects) is not clearly mentioned. Is it used only for consistency reasons,  to help the model in preserving the positional index across frames, or do those matching-edges guide the transformers in any way?\n- The proposed architecture is composed of several big components (transformers). Thus a comparison against existing methods in terms of the number of parameters and flops would be useful.\n- Since the additional training techniques bring considerable improvement, it seems to me that the proposed algorithm is not very robust to different kinds of noise encountered during training. This could be a problem when applying to real-world scenarios when probably the object detector will not be as accurate as the one used in this synthetic setup.  Do you think that the general algorithm, but especially all the heuristics proposed for the auxiliary losses, will suffer because of that?\n\nMinor:\n- In G.1 (appendix) there is a typo: “stacking 6 transformer decoder layers and 6 transformer decoder layers”\n- In the “Interpretability” paragraph, the sentence … learns to perform snitch-oriented tracking automatically, without having per-frame supervision“ is confusing. Even if no snitch-level supervision is used to guide the localisation across video, the per-frame supervision is used to learn the detector. I agree that other datasets could be used in real-world scenarios, but in this case, the per-frame annotation is used instead.\n\nI really think that the idea of multi-hop and “selective” pointers in transformers is valuable and useful for video reasoning. Moreover, I appreciate the empirical study presented in this work. However, I have major concerns about the way the algorithm is explained in the paper. Several crucial parts are not clearly explained and the usage of the two transformers inside the multi-hop method are not motivated at all. Taking these into consideration, I don’t think that this work is in a proper form to be published since several sections should be rewritten entirely. In conclusion, I recommend the rejection.\n\n########### UPDATE #########\n\nI thank the authors for clarifying the method description, providing additional experiments and actively engaging in discussion. Since my main concern regarding the writing part was addressed I change my rating and I agree with the acceptance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to ICLR 2021. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes.  It is also very important to think about how to extend this framework to the more challenging CLEVERER dataset (http://clevrer.csail.mit.edu/). 
Decision: Accept (Poster)