Hopper: Multi-hop Transformer for Spatiotemporal Reasoning

1 INTRODUCTION In this paper , we address the problem of spatiotemporal object-centric reasoning in videos . Specifically , we focus on the problem of object permanence , which is the ability to represent the existence and the trajectory of hidden moving objects ( Baillargeon , 1986 ) . Object permanence can be essential in understanding videos in the domain of : ( 1 ) sports like soccer , where one needs to reason , “ which player initiated the pass that resulted in a goal ? ” , ( 2 ) activities like shopping , one needs to infer “ what items the shopper should be billed for ? ” , and ( 3 ) driving , to infer “ is there a car next to me in the right lane ? ” . Answering these questions requires the ability to detect and understand the motion of objects in the scene . This requires detecting the temporal order of one or more actions of objects . Furthermore , it also requires learning object permanence , since it requires the ability to predict the location of non-visible objects as ∗Work done as a NEC Labs intern . 1https : //github.com/necla-ml/cater-h they are occluded , contained or carried by other objects ( Shamsian et al. , 2020 ) . Hence , solving this task requires compositional , multi-step spatiotemporal reasoning which has been difficult to achieve using existing deep learning models ( Bottou , 2014 ; Lake et al. , 2017 ) . Existing models have been found lacking when applying to video reasoning and object permanence tasks ( Girdhar & Ramanan , 2020 ) . Despite rapid progress in video understanding benchmarks such as action recognition over large datasets , deep learning based models often suffer from spatial and temporal biases and are often easily fooled by statistical spurious patterns and undesirable dataset biases ( Johnson et al. , 2017b ) . For example , researchers have found that models can recognize the action “ swimming ” even when the actor is masked out , because the models rely on the swimming pool , the scene bias , instead of the dynamics of the actor ( Choi et al. , 2019 ) . Hence , we propose Hopper to address debiased video reasoning . Hopper uses multi-hop reasoning over videos to reason about object permanence . Humans realize object permanence by identifying key frames where objects become hidden ( Bremner et al. , 2015 ) and reason to predict the motion and final location of objects in the video . Given a video and a localization query , Hopper uses a Multi-hop Transformer ( MHT ) over image and object tracks to automatically identify and hop over critical frames in an iterative fashion to predict the final position of the object of interest . Additionally , Hopper uses a contrastive debiasing loss that enforces consistency between attended objects and correct predictions . This improves model robustness and generalization . We also build a new dataset , CATER-h , that reduces temporal bias in CATER and requires long-term reasoning . We demonstrate the effectiveness of Hopper over the recently proposed CATER ‘ Snitch Localization ’ task ( Girdhar & Ramanan , 2020 ) ( Figure 1 ) . Hopper achieves 73.2 % Top-1 accuracy in this task at just 1 FPS . More importantly , Hopper identifies the critical frames where objects become invisible or reappears , providing an interpretable summary of the reasoning performed by the model . To summarize , the contributions of our paper are as follows : First , we introduce Hopper that provides a framework for multi-step compositional reasoning in videos and achieves state-of-the-art accuracy in CATER object permanence task . Second , we describe how to perform interpretable reasoning in videos by using iterative reasoning over critical frames . Third , we perform extensive studies to understand the effectiveness of multi-step reasoning and debiasing methods that are used by Hopper . Based on our results , we also propose a new dataset , CATER-h , that requires longer reasoning hops , and demonstrates the gaps of existing deep learning models . 2 RELATED WORK . Video understanding . Video tasks have matured quickly in recent years ( Hara et al. , 2018 ) ; approaches have been migrated from 2D or 3D ConvNets ( Ji et al. , 2012 ) to two-stream networks ( Simonyan & Zisserman , 2014 ) , inflated design ( Carreira & Zisserman , 2017 ) , models with additional emphasis on capturing the temporal structures ( Zhou et al. , 2018 ) , and recently models that better capture spatiotemporal interactions ( Wang et al. , 2018 ; Girdhar et al. , 2019 ) . Despite the progress , these models often suffer undesirable dataset biases , easily confused by backgrounds objects in new environments as well as varying temporal scales ( Choi et al. , 2019 ) . Furthermore , they are unable to capture reasoning-based constructs such as causal relationships ( Fire & Zhu , 2017 ) or long-term video understanding ( Girdhar & Ramanan , 2020 ) . Visual and video reasoning . Visual and video reasoning have been well-studied recently , but existing research has largely focused on the task of question answering ( Johnson et al. , 2017a ; Hudson & Manning , 2018 ; 2019a ; Yi et al. , 2020 ) . CATER , a recently proposed diagnostic video recognition dataset focuses on spatial and temporal reasoning as well as localizing particular object of interest . There also has been significant research in object tracking , often with an emphasis on occlusions with the goal of providing object permanence ( Wojke et al. , 2017 ; Wang et al. , 2019b ) . Traditional object tracking approaches often require expensive supervision of location of the objects in every frame . In contrast , we address object permanence and video recognition on CATER with a model that performs tracking-integrated object-centric reasoning without this strong supervision . Multi-hop reasoning . Reasoning systems vary in expressive power and predictive abilities , which include symbolic reasoning , probabilistic reasoning , causal reasoning , etc . ( Bottou , 2014 ) . Among them , multi-hop reasoning is the ability to reason with information collected from multiple passages to derive the answer ( Wang et al. , 2019a ) , and it gives a discrete intermediate output of the reasoning process , which can help gauge model ’ s behavior beyond just the final task accuracy ( Chen et al. , 2019 ) . Several multi-hop datasets and models have been proposed for the reading comprehension Target : video query representation object class label positional spatial resolution encoding Time CNN feature maps DETR : End-to-end Transformer based object detector positional time encoding Tracking : Hungarian Between Frame Matching Algorithm Multi-Hop Transformer MLP image features Source object queries object representation object bounding box MLP ... : the number of frames : the number of objects : batch size ... Track 0 Track 1 Track 2 Track N IMG OBJ OBJ OBJ positional time encoding Target : video query Classification Source Sequence A. backbone Ref . Figure 16 B. obj ect det ect ion and r epr esentat ion C. t r acking D. video Quer y r epr esentat ion and r ecognit ion ... : dimension size positional encoded feature maps Figure 2 : An overview of the Hopper framework . Hopper first obtains frame representations from the input video . Object representations and object tracks are then computed to enable tracking-integrated object-centric reasoning for the Multi-hop Transformer ( details in Section 4 ) . task ( Welbl et al. , 2018 ; Yang et al. , 2018b ; Dua et al. , 2019 ; Dhingra et al. , 2020 ) . We extend multihop reasoning to the video domain by developing a dataset that explicitly requires aggregating clues from different spatiotemporal parts of the video , as well as a multi-hop model that automatically extracts a step-by-step reasoning chain , which improves interpretability and imitates a natural way of thinking . We provide an extended discussion of related work in Appendix I . 3 HOPPER . Hopper ( Figure 2 ) is a framework inspired from the observation that humans think in terms of entities and relations . Unlike traditional deep visual networks that perform processing over the pixels from which they learn and extract features , object-centric learning-based architecture explicitly separates information about entities through grouping and abstraction from the low-level information ( Locatello et al. , 2020 ) . Hopper obtains representations of object entities from the lowlevel pixel information of every frame ( Section 3.2 ) . Additionally , to maintain object permanence , humans are able to identify key moments when the objects disappear and reappear . To imitate that , Hopper computes object tracks with the goal to have a more consistent object representation ( Section 3.3 ) and then achieves multi-step compositional long-term reasoning with the Multi-hop Transformer to pinpoint these critical moments . Furthermore , Hopper combines both fine-grained ( object ) and coarse-grained ( image ) information to form a contextual understanding of a video . As shown in Figure 2 , Hopper contains 4 components ; we describe them below . 3.1 BACKBONE . Starting from the initial RGB-based video representation xv ∈ RT×3×H0×W0 where T represents the number of frames of the video , 3 is for the three color channels , and H0 and W0 denote the original resolution height and width , a conventional CNN backbone would extract the feature map f ∈ RT×P×H×W and for every frame t a compact image representation it ∈ RP . The backbone we use is ResNeXt-101 from Ma et al . ( 2018 ) , P = 2048 and H , W = 8 , 10 . A 1×1 convolution ( Carion et al. , 2020 ) then reduces the channel dimension of f from P to a smaller dimension d ( d = 256 ) , and a linear layer is used to turn the dimension of it from P to d . 3.2 OBJECT DETECTION AND REPRESENTATION . We collapse the spatial dimensions into 1 dimension and combine the batch dimension with the temporal dimension for the feature map f . Positional encodings are learned for each time step ( T in total ) and each spatial location ( H × W in total ) , which are further added to the feature map in an element-wise manner . The positional encoding-augmented feature map is the source input to the transformer encoder ( Vaswani et al. , 2017 ) of DETR ( Carion et al. , 2020 ) . DETR is a recently proposed transformer-based object detector for image input ; it additionally accepts N embeddings of object queries for every image ( assuming every image at most has N objects2 ) to the transformer decoder . We also combine the batch dimension with temporal dimension for the object queries . Outputs from DETR are transformed object representations that are used as inputs to a multilayer perceptron ( MLP ) to predict the bounding box and class label of every object . For Snitch Localization , DETR is trained on object annotations from LA-CATER ( Shamsian et al. , 2020 ) . 2∅ , i.e. , none object , will be predicted if the number of objects in an image is less than N . 3.3 TRACKING . Tracking produces consistent object representations as it links the representations of each object through time . We perform tracking using the unordered object representations , bounding boxes and labels as inputs , and applying our Hungarian-based algorithm to match objects between every two consecutive frames . We describe the details as follows . Tracking is essentially an association problem ( Bewley et al. , 2016 ) . An association between 2 objects respectively from consecutive 2 frames can be defined by the object class agreement and the difference of the two bounding boxes . Let us denote by ŷ = [ ŷt ] Tt=1 the predicted list of objects at all frames in a video , where ŷt = { ŷit } Ni=1 denotes the predicted set of objects at frame t. Each object is represented as a 4-tuple ŷit = ( ĉ i t , b̂ i t , { p̂it ( c ) |c ∈ C } , oit ) where ĉit denotes the class label that has the maximum predicted likelihood for object i at frame t , b̂it ∈ [ 0 , 1 ] 4 is a vector that defines the bounding box top left and bottom right coordinates relative to the image size , p̂it ( c ) denotes the predicted likelihood for class c ( where C = { large metal green cube , small metal green cube , . . . , ∅ } ) , and oit ∈ Rd denotes the representation vector of this object i at frame t. In order to obtain the optimal bipartite matching between the set of predicted objects at frame t and t+ 1 , we search for a permutation of N elements σ ∈ SN with the lowest permutation cost : σ̂ = argmin σ∈SN N∑ i=1 Ltrack ( ŷit , ŷ σ ( i ) t+1 ) ( 1 ) where Ltrack is a pair-wise track matching cost between predicted object ŷit ( i.e. , object i at frame t ) and predicted object at frame t + 1 with index σ ( i ) from the permutation σ , denoted by ŷσ ( i ) t+1 . Following Carion et al . ( 2020 ) , the optimal assignment is computed efficiently with the Hungarian algorithm . The track matching cost at time t for object i is defined as Ltrack ( ŷit , ŷ σ ( i ) t+1 ) = −λc1 { ĉit 6=∅ } p̂ σ ( i ) t+1 ( ĉit ) + λb1 { ĉit 6=∅ } Lbox ( b̂it , b̂ σ ( i ) t+1 ) ( 2 ) where 1 denotes an indicator function such that the equation after the symbol 1 only takes effect when the condition inside the { . . . } is true , otherwise the term will be 0. λc , λb ∈ R weight each term . Lbox is defined as a linear combination of the L1 loss and the generalized IoU loss ( Rezatofighi et al. , 2019 ) . When the predicted class label of object i at frame t is not ∅ , we aim to maximize the likelihood of the class label ĉit for the predicted object σ ( i ) at frame t+ 1 , and minimize the bounding box difference between the two . The total track matching cost of a video is the aggregation of Ltrack ( ŷit , ŷ σ ( i ) t+1 ) from object i = 1 to N and frame t = 1 to T − 1 . This Hungarian-based tracking algorithm is used due to its simplicity . A more sophisticated tracking solution ( e.g . DeepSORT ( Wojke et al. , 2017 ) ) could be easily integrated into Hopper , and may improve the accuracy of tracking in complex scenes .

Summary: This paper introduces an architecture (Multi-Hop Transformer) for spatio-temporal reasoning in video, focusing on a localisation task for scenes where the object of interest is often occluded (Snitch Localisation task in CATER). The model extracts objects using an external object-detector and predicts objects’ trajectories using the Hungarian algorithm. The Multi-Hop Transformers learns to hop over unnecessary frames, by focusing only on a set of few critical steps (at each time step, the next critical frame is the one containing the most attended object). To alleviate the problems encountered during the training procedure (such as error propagation), several auxiliary training methods are proposed to guide the first few hops or to ensure contrastive debias. Additionally, a harder version of the existing CATER dataset is created, to alleviate the temporal bias existent in the previous version of the dataset.