CATER: A diagnostic dataset for Compositional Actions & TEmporal Reasoning

1 INTRODUCTION . While deep features have revolutionized static image analysis , video descriptors have struggled to outperform classic hand-crafted descriptors ( Wang & Schmid , 2013 ) . Though recent works have shown improvements by merging image and video models by inflating 2D models to 3D ( Carreira & Zisserman , 2017 ; Feichtenhofer et al. , 2016 ) , simpler 2D models ( Wang et al. , 2016b ) still routinely appear among top performers in video benchmarks such as the Kinetics Challenge at CVPR ’ 17 . This raises the natural question : are videos trivially understandable by simply averaging the predictions over a sampled set of frames ? At some level , the answer must be no . Reasoning about high-level cognitive concepts such as intentions , goals , and causal relations requires reasoning over long-term temporal structure and order ( Shoham , 1987 ; Bobick , 1997 ) . Consider , for example , the movie clip in Fig . 1 ( a ) , where an actor leaves the table , grabs a firearm from another room , and returns . Even though no gun is visible in the final frames , an observer can easily infer that the actor is surreptitiously carrying the gun . Needless to say , any single frame from the video seems incapable of supporting that inference , and one needs to reason over space and time in order to reach that conclusion . As a simpler instance of the problem , consider the cup-and-balls magic routine1 , or the gamblingbased shell game2 , as shown in Fig . 1 ( b ) . In these games , an operator puts a target object ( ball ) under one of multiple container objects ( cups ) , and moves them about , possibly revealing the target at various times and recursively containing cups within other cups . The task at the end is to tell which of the cups is covering the ball . Even in its simplest instantiation , one can expect any human or computer system that solves this task to require the ability to model state of the world over long temporal horizons , reason about occlusion , understand the spatiotemporal implications of containment , etc . An important aspect of both our motivating examples is the adversarial nature of the task , ∗Now at Facebook AI Research 1https : //en.wikipedia.org/wiki/Cups_and_balls 2https : //en.wikipedia.org/wiki/Shell_game where the operator in control is trying to make the observer fail . Needless to say , a frame by frame prediction model would be incapable of solving such tasks . Given these motivating examples , why don ’ t spatiotemporal models dramatically outperform their static counterparts for video understanding ? We posit that this is due to limitations of existing video benchmarks . Even though video datasets have evolved from the small regime with tens of labels ( Soomro et al. , 2012 ; Kuehne et al. , 2011 ; Schuldt et al. , 2004 ) to large with hundreds of labels ( Sigurdsson et al. , 2016 ; Kay et al. , 2017 ) , tasks have remained highly correlated to the scene and object context . For example , it is trivial to recognize a swimming action given a swimming pool in the background ( He et al. , 2016b ) . This is further reinforced by the fact that state of the art pose-based action recognition models ( Yan et al. , 2018 ) are outperformed by simpler frame-level models ( Wang et al. , 2016b ) on the Kinetics ( Kay et al. , 2017 ) benchmark , with a difference of nearly 45 % in accuracy ! Sigurdsson et al . also found similar results for their Charades ( Sigurdsson et al. , 2016 ) benchmark , where adding ground truth object information gave the largest boosts to action recognition performance ( Sigurdsson et al. , 2017 ) . In this work , we take an alternate approach to developing a video understanding dataset . Inspired by the recent CLEVR dataset ( Johnson et al. , 2017 ) ( that explores spatial reasoning in tabletop scenes ) and inspired by the adversarial parlor games above ( that require temporal reasoning ) , we introduce CATER , a diagnostic dataset for Compositional Actions and TEmporal Reasoning in dynamic tabletop scenes . We define three tasks on the dataset , each with an increasingly higher level of complexity , but set up as classification problems in order to be comparable to existing benchmarks for easy transfer of existing models and approaches . Specifically , we consider primitive action recognition , compositional action recognition , and adversarial target tracking under occlusion and containment . However , note that this does not limit the usability of our dataset to these tasks , and we provide full metadata with the rendered videos that can be used for more complex , structured prediction tasks like detection , tracking , forecasting , and so on . Our dataset does not model an operator ( or hand ) moving the tabletop objects , though this could be simulated as well in future variants , as in ( Rogez et al. , 2015 ) . Being synthetic , CATER can easily be scaled up in size and complexity . It also allows for detailed model diagnostics by controlling various dataset generation parameters . We use CATER to benchmark state-of-the-art video understanding models ( Wang et al. , 2018 ; 2016b ; Hochreiter & Schmidhuber , 1997 ) , and show even the best models struggle on our dataset . We also uncover some insights into the behavior of these models by changing parameters such as the temporal duration of an occlusion , the degree of camera motion , etc. , which are difficult to both tune and label in real-world video data . 2 RELATED WORK . Spatiotemporal networks : Video understanding for action recognition has evolved from iconic hand-designed models ( Wang & Schmid , 2013 ; Laptev , 2005 ; Wang et al. , 2011 ) to sophisticated Dataset Size Len Task # cls TO STR LTR CSB UCF101 ( Soomro et al. , 2012 ) 13K 7s cls 101 7 7 7 7 HMDB51 ( Kuehne et al. , 2011 ) 5K 4s cls 51 7 7 7 7 Kinetics ( Kay et al. , 2017 ) 300K 10s cls 400 7 3 7 7 AVA ( Gu et al. , 2018 ) 430 15m det 80 7 3 7 7 VLOGs ( Fouhey et al. , 2018 ) 114K 10s cls 30 7 3 7 7 DAHLIA ( Vaquette et al. , 2017 ) 51 39m det 7 3 3 3 7 TACoS ( Regneri et al. , 2013 ) 127 6m align - 3 3 3 7 DiDeMo ( Anne Hendricks et al. , 2017 ) 10K 30s align - 3 3 3 7 Charades ( Sigurdsson et al. , 2016 ) 10K 30s det 157 3 3 7 7 Something Something ( Goyal et al. , 2017 ) 108K 4s cls 174 3 3 7 3 Diving48 ( Li et al. , 2018 ) 18K 5s cls 48 3 3 7 3 Cooking ( Rohrbach et al. , 2012a ) 44 3-41m cls 218 3 3 7 3 IKEA ( Toyer et al. , 2017 ) 101 2-4m gen - 3 3 3 3 Composite ( Rohrbach et al. , 2012b ) 212 1-23m cls 44 3 3 3 3 TFGIF-QA ( Jang et al. , 2017 ) 72K 3s qa - 3 3 7 7 MovieQA ( Tapaswi et al. , 2016 ) 400 200s qa - 3 3 3 7 Robot Pushing ( Finn et al. , 2016 ) 57K 1s gen - 3 3 7 3 SVQA ( Song et al. , 2018 ) 12K 4s qa - 3 3 7 3 Moving MNIST ( Srivastava et al. , 2015 ) - 2s gen - 3 3 7 3 Flash MNIST ( Long et al. , 2018 ) 100K 2s cls 1024 7 3 7 3 CATER ( ours ) 5.5K 10s cls 36-301 3 3 3 3 spatiotemporal deep networks ( Carreira & Zisserman , 2017 ; Simonyan & Zisserman , 2014 ; Girdhar et al. , 2017 ; Wang et al. , 2018 ; Xie et al. , 2017 ; Tran et al. , 2018 ; 2015 ) . While similar developments in the image domain have lead to large improvements on tasks like classification ( Szegedy et al. , 2016 ; He et al. , 2016a ; Huang et al. , 2017 ) and localization ( He et al. , 2017 ; Papandreou et al. , 2017 ) , video models have struggled to out-perform previous hand-crafted descriptors ( Wang & Schmid , 2013 ) . Even within the set of deep video architectures , models capable of temporal modeling , such as RNNs ( Karpathy et al. , 2014 ) and 3D convolutions ( Tran et al. , 2015 ; Varol et al. , 2017a ) have not shown significantly better performance than much simpler , per-frame prediction models , such as variants of two-stream architectures ( Wang et al. , 2016b ; Simonyan & Zisserman , 2014 ) . Though some recent works have shown improvements by merging image and video models by inflating 2D models to 3D ( Carreira & Zisserman , 2017 ; Feichtenhofer et al. , 2016 ) , simple 2D models ( Wang et al. , 2016b ) were still among the top performers in the Kinetics Challenge at CVPR ’ 17 . Video action understanding datasets : There has been significant effort put forth to collecting video benchmarks . One line of attack employs human actors to perform scripted actions . This is typically done in controlled environments ( Schuldt et al. , 2004 ; Shahroudy et al. , 2016 ; Ionescu et al. , 2014 ) , but recent work has pursued online crowd sourcing ( Goyal et al. , 2017 ; Sigurdsson et al. , 2016 ) . Another direction collects videos from movies and online sharing platforms . Many popular video benchmarks follow this route for diverse , in-the-wild videos , such as UCF-101 ( Soomro et al. , 2012 ) , HMDB-51 ( Kuehne et al. , 2011 ) and more recently Kinetics ( Kay et al. , 2017 ) and VLOGs ( Fouhey et al. , 2018 ) . As discussed earlier , such datasets struggle with the strong bias of actions with scenes and objects . Our underlying thesis is that the field of video understanding is hampered by such biases because they favor image-based baselines . While some recent work ( Goyal et al. , 2017 ; Li et al. , 2018 ) attempts to control for this bias , it still remains a challenge for long-term reasoning tasks . One might argue that since such biases are common in the visual world , video benchmarks should reflect them . We take the view that a diverse set of benchmarks are needed to enable comprehensive diagnostics and validation of the state-of-affairs in video understanding . Table 1 shows that CATER fills a missing gap in the benchmark landscape , most notably because of its size/video length , label distribution , relative resilience to object and scene bias , and diagnostic abilities . Synthetic data in computer vision : Our work , being synthetically generated , is also closely related to other works in using synthetic data for computer vision applications . There has been a large body of work in this direction , with the major focus on using synthetic training data for real world applications . This includes semantic scene understanding ( Dosovitskiy et al. , 2017 ; Shah et al. , 2018 ; Richter et al. , 2017 ) , 3D scene understanding ( Girdhar et al. , 2016 ; Su et al. , 2015 ; Wu et al. , 2016 ; Song et al. , 2017 ) , human understanding ( Varol et al. , 2017b ; De Souza et al. , 2017 ) , optical flow ( Butler et al. , 2012 ; Mayer et al. , 2016 ) and navigation , RL or embodied learning ( Wu et al. , 2018 ; Kolve et al. , 2017 ; Kempka et al. , 2016 ; Mnih et al. , 2013 ) . Our work , on the other hand , attempts to develop a benchmark for video based action understanding . Similar attempts have been made for scene understanding through abstract scenes ( Zitnick et al. , 2016 ) , with more recently focusing on building a complex reasoning benchmark , CLEVR ( Johnson et al. , 2017 ) . In the video domain , benchmarks such as Flash-MNIST ( Long et al. , 2018 ) , Moving MNIST ( Srivastava et al. , 2015 ) and SVQA ( Song et al. , 2018 ) have been proposed . Concurrent to us , CLEVRER ( Yi et al. , 2020 ) , PHYRE ( Bakhtin et al. , 2019 ) , COPHY ( Baradel et al. , 2020 ) and IntPhys ( Riochet et al. , 2018 ) benchmarks have been proposed with a focus on causal physical reasoning through QA , RL , prediction and ranking interfaces respectively . On the other hand , CATER focuses on spatiotemporal video reasoning tasks building upon CLEVR , with a simple classification interface , making it easily amenable for existing video understanding systems . Object tracking : Detecting and tracking objects has typically been used as an initial representation for long-term video and activity understanding ( Shet et al. , 2005 ; Hongeng et al. , 2004 ; Lavee et al. , 2009 ) . Extensions include adversarial tracking , where the objects are designed to be hidden from plain view . It has typically been used for tasks such as determining if humans are carrying an object ( Dondera et al. , 2013 ; Ferrando et al. , 2006 ) or abandoned / exchanging objects ( Tian et al. , 2011 ; Li et al. , 2006 ) . We embrace this direction of work and include state-of-the-art deep trackers ( Zhu et al. , 2018 ) in our benchmark evaluation .

Summary: This paper introduces a new synthetic video understanding dataset, borrowing many ideas from the visual question answering dataset CLEVR. The new dataset is the first to account for all of the following fundamental aspect of videos: temporal ordering, short- and long term reasoning, and control for scene biases. Due to the inherent biases in available action recognition datasets, models that simply averages video frames do nearly as well as models that take temporal dependencies into account. In contrast, the authors show that with the proposed dataset, models without spatiotemporal reasoning largely fail.