Tracking Without Re-recognition in Humans and Machines

1 Introduction . Lettvin and colleagues [ 1 ] presciently noted , “ The frog does not seem to see or , at any rate , is not concerned with the detail of stationary parts of the world around him . He will starve to death surrounded by food if it is not moving. ” Object tracking is fundamental to survival , and higher biological visual systems have evolved the capacity for two distinct and complementary strategies to do it . Consider Figure 1 : can you track the object labeled by the yellow arrow from left-to-right ? The task is trivial when appearance cues , like color , make it possible to solve the temporal correspondence problem by “ re-recognizing ” the target in each frame ( Fig . 1a ) . However , this strategy is not effective when objects can not be discriminated by their appearance alone ( Fig . 1b ) . In this case integration of object motion over time is necessary for tracking . Humans are capable of tracking objects by their motion when appearance is uninformative [ 2 , 3 ] , but it is unclear if the current generation of neural networks for video analysis and tracking can do the same . To address this question we introduce PathTracker , a synthetic challenge for object tracking without re-recognition ( Fig . 1c ) . * †These authors contributed equally to this work . 1Carney Institute for Brain Science , Brown University , Providence , RI 2Northeastern University , Boston , MA 3DeepMind , London , UK 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Leading models for video analysis rely on object classification pre-training . This gives them access to rich semantic representations that have supported state-of-the-art performance on a host of tasks , from action recognition to object tracking [ 4–6 ] . As object classification models have improved , so too have the video analysis models that depend on them . This trend in model development has made it unclear if video analysis models are effective at learning tasks when appearance cues are uninformative . The importance of diverse visual strategies has been highlighted by synthetic challenges like Pathfinder , ( a ) a visual reasoning task that asks observers to trace long paths embedded in a static cluttered display [ 7 , 8 ] . Pathfinder tests object segmentation when appearance cues like category or shape are missing . While humans can easily solve it [ 8 ] , deep neural networks struggle , including state-of-theart vision transformers [ 7–9 ] . Importantly , models that learn an appropriate visual strategy for Pathfinder also exhibit more efficient learning and improved generalization on object segmentation in natural images [ 10 , 11 ] . Our PathTracker challenge extends this line of work into video by posing an object tracking problem where the target can be tracked by motion and spatiotemporal continuity , not category or appearance . Contributions . Humans effortlessly solve our novel PathTracker challenge . A variety of state-ofthe-art models for object tracking and video analysis do not . • We find that neural architectures including R3D [ 12 ] and state-of-the-art transformer-based TimeS- formers [ 5 ] are strained by long PathTracker videos . Humans , on the other hand , are far more effective at solving these long PathTracker videos . • We describe a solution to PathTracker : a recurrent network inspired by primate neural circuitry involved in object tracking , which renders decisions that are strongly correlated with humans . • These same circuit mechanisms improve object tracking in natural videos through a motion-based strategy that builds tolerance to changes in target object appearance , resulting in the state-of-the-art score on TrackingNet [ 13 ] . • We release all PathTracker data , code , and human psychophysics at http : //bit.ly/InTcircuit to spur interest in the challenge of tracking without re-recognition . 2 Related Work . Models for video analysis A major leap in the performance of models for video analysis came from using networks which are pre-trained for object recognition on large image datasets [ 4 ] . The recently introduced TimeSformer [ 5 ] achieved state-of-the-art performance with weights initialized from an image categorization transformer ( ViT ; [ 14 ] ) that was pre-trained on ImageNet-21K . The modeling trends are similar in object tracking [ 15 ] , where successful models rely on “ backbone ” feature extraction networks trained on ImageNet or Microsoft COCO [ 16 ] for object recognition or segmentation [ 6 , 17 ] . Shortcut learning and synthetic datasets A byproduct of the great power of deep neural network architectures is their vulnerability to learning spurious correlations between inputs and labels . Perhaps because of this tendency , object classification models have trouble generalizing to novel contexts [ 18 , 19 ] , and render idiosyncratic decisions that are inconsistent with humans [ 20–22 ] . Synthetic datasets are effective at probing this vulnerability because they make it possible to control spurious image/label correlations – providing a fairer test of the computational abilities of these models . For example , the Pathfinder challenge was designed to test if neural architectures can trace long curves in clutter – a visual computation associated with the earliest stages of visual processing in primates . That challenge identified diverging visual strategies between humans and transformers that are otherwise state of the art in natural image object recognition [ 9 , 14 ] . Other challenges like Bongard-LOGO [ 23 ] , cABC [ 8 ] , SVRT [ 24 ] , and PSVRT [ 25 ] have highlighted limitations of leading neural network architectures that would have been difficult to identify using natural image benchmarks like ImageNet [ 26 ] . These limitations have inspired algorithmic solutions based on neural circuits discussed in SI §1 . Translating circuits for biological vision into artificial neural networks While the Pathfinder challenge [ 7 ] presents immense challenges for transformers and deep convolutional networks [ 8 ] , it can be solved by a simple model of intrinsic connectivity in visual cortex , with orders-of-magnitude fewer parameters than standard models for image categorization . This model was developed by translating descriptive models of neural mechanisms from Neuroscience into an architecture that can be fit to data using gradient descent [ 7 , 11 ] . Others have found success in modeling object tracking by drawing inspiration from “ dual stream ” theories of appearance and motion processing in visual cortex [ 27–30 ] , or basing the architecture off of a partial connectome of the Drosophila visual system [ 31 ] . We adopt a similar approach in the current work , identifying mechanisms for object tracking without re-recognition in Neuroscience , and developing those into differentiable operations with parameters that can be optimized by gradient descent . This approach has the dual purpose of introducing task-relevant inductive biases into computer vision models , and developing theory on their relative utility for biological vision . Multi-object tracking in computer vision The classic psychological paradigms of multi-object tracking [ 2 ] motivated the application of models , like Kalman filters , which had tolerance to object occlusion when they relied on momentum models [ 32 ] . However , these models are computationally expensive , hand-tuned , and because of this , no longer commonly used in computer vision [ 33 ] . More recent approaches include flow tracking on graphs [ 34 ] and motion tracking models that are relatively computationally efficient [ 35 , 36 ] . However , even current approaches to multi-object tracking are not learned , instead relying on extensive hand tuning [ 37 , 38 ] . In contrast , the point of PathTracker is to understand the extent to which state-of-the-art neural networks are capable of tracking a single object in an array of distractors . 3 The PathTracker Challenge Overview PathTracker asks observers to decide whether or not a target dot reaches a goal location ( Fig . 2 ) . The target dot travels in the midst of a pre-specified number of distractors . All dots are identical , and the task is difficult because of this : ( i ) appearance is not useful for tracking the target , and ( ii ) the paths of the target and distractors can momentarily “ cross ” and occupy the same space , making it impossible to individuate them in that frame and meaning that observers can not solely rely on the target ’ s location to solve the task . This challenge is inspired by object tracking paradigms in cognitive psychology [ 2 , 3 , 39 ] , which suggest that humans might tap into mechanisms for motion perception , attention and working memory to solve a task like PathTracker . The trajectories of target and distractor dots are randomly generated , and the target occasionally crosses distractors ( Fig . 2 ) . These object trajectories are smooth by design , giving the appearance of objects meandering through a scene , and the difference between the coordinates of any dot on successive frames is no more than 2 pixels with less than 20◦ of angular displacement . In other words , dots never turn at acute angles . We develop different versions of PathTracker with varying degrees of complexity based on the number of distractors and/or the length of videos . These variables change the expected number of times that distractors cross the target and the amount of time that observers must track the target ( Fig . 2 ) . To make the task as visually simple as possible and maximize contrast between dots and markers , the dots , start , and goal markers are placed on different channels in 32×32 pixel three-channel images . Markers are stationary throughout each video and placed at random locations . Examples videos can be viewed at http : //bit.ly/InTcircuit . Human benchmark We began by testing if humans can solve PathTracker . We recruited 180 individuals using Amazon Mechanical Turk to participate in this study . Participants viewed PathTracker videos and pressed a button on their keyboard to indicate if the target object or a distractor reached the goal . These videos were played in web browsers at 256×256 pixels using HTML5 , which helped ensure consistent frame rates [ 40 ] . The experiment began with an 8-trial “ training ” stage , which familiarized participants with the goal of PathTracker . Next , participants were tested on 72 videos . The experiment was not paced and lasted approximately 25 minutes , and participants were paid $ 8 for their time . See http : //bit.ly/InTcircuit and SI §2 for an example and more details . Participants were randomly entered into one of two experiments . In the first experiment , they were trained on the 32 frame and 14 distractor PathTracker , and tested on 32 frame versions with 1 , 14 , or 25 distractors . In the second experiment , they were trained on the 64 frame and 14 distractor PathTracker , and tested on 64 frame versions with 1 , 14 , or 25 distractors . All participants viewed unique videos to maximize our sampling over the different versions of PathTracker . Participants were significantly above chance on all tested conditions of PathTracker ( p < 0.001 , test details in SI §2 ) . They also exhibited a significant negative trend in performance on the 64 frame datasets as the number of distractors increased ( t = −2.74 , p < 0.01 ) . There was no such trend on the 32 frame datasets , and average accuracy between the two datasets was not significantly different . These results validate our initial design assumptions : humans can solve PathTracker , and manipulating distractors and video length increases difficulty . 4 Solving the PathTracker challenge Can leading models for video analysis match humans on PathTracker ? To test this question we surveyed a variety of architectures that are the basis for leading approaches to many video analysis tasks , from object tracking to action classification . The models fall into four groups : ( i ) deep ( e ) convolutional networks ( CNNs ) , ( ii ) transformers , ( iii ) recurrent neural networks ( RNNs ) , and ( iv ) Kalman filters . The deep convolutional networks include a 3D ResNet ( R3D [ 12 ] ) , a space/time separated ResNet with “ 2D-spatial + 1D-temporal ” convolutions ( R ( 2+1 ) D [ 12 ] ) , and a ResNet with 3D convolutions in early residual blocks and 2D convolutions in later blocks ( MC3 [ 12 ] ) . We trained versions of these models with random weight initializations and weights pretrained on ImageNet . We included an R3D trained from scratch without any downsampling , in case the small size of PathTracker videos caused learning problems ( see SI §3 for details ) . We also trained a version of the R3D on optic flow encodings of PathTracker ( SI §3 ) . For transformers , we turned to the TimeSformer [ 5 ] . We evaluated two of its instances : ( i ) where attention is jointly computed for all locations across space and time in videos , and ( ii ) where temporal attention is applied before spatial attention , which results in massive computational savings . Both models performed similarly on PathTracker . We report the latter version here as it was marginally better ( see SI §3 for performance of the other , joint space-time attention TimeSformer ) . We include a version of the TimeSformer trained from scratch , and a version pre-trained on ImageNet-20K . Note that state-of-the-art transformers for object tracking in natural videos feature similar deep and multi-headed designs [ 6 ] . For the RNNs , we include a convolutional-gated recurrent unit ( Conv-GRU ) [ 41 ] . Finally , our exemplar Kalman filter is the standard Simple and Online Realtime Tracking ( SORT ) algorithm , which was fed coordinates of the objects extracted from each frame of every video [ 37 ] . Method The visual simplicity of PathTracker cuts two ways : it makes it possible to compare human and model strategies for tracking without re-recognition as long as the task is not too easy . Prior synthetic challenges like Pathfinder constrain sample sizes for training to probe specific computations [ 7–9 ] . We adopted the following strategy to select a training set size that would help us test tracking strategies that do not depend on re-recognition . We took Inception 3D ( I3D ) networks [ 4 ] , which have been a strong baseline architecture in video analysis over the past several years , and tested their ability to learn PathTracker as we adjusted the number of videos for training . As we discuss in SI §1 , when this model was trained with 20K examples of the 32 frame and 14 distractor version of PathTracker it achieved good performance on the task without signs of overfitting . We therefore trained all models in subsequent experiments with 20K examples . By happenstance , this dataset size gives PathTracker a comparable number of frames to large-scale real world tracking challenges like LaSOT [ 42 ] and GOT-10K [ 43 ] . We measure the ability of models to learn PathTracker and systematically generalize to novel versions of the challenge when trained on 20K samples . We trained models using a similar approach as in our human psychophysics . Models were trained on one version of PathTracker , and tested on other Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = g i [ t 1 ] + ( 1 g ) [ z [ t ] ( i [ t ] a [ t ] + ) m [ t ] i [ t 1 ] ] + e [ t ] = h e [ t 1 ] + ( 1 h ) [ i [ t ] + ( ⌫i [ t ] + µ ) n [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E + Ua ⇤ Z ) Me , i = h m ( c ) xy i = We , i ⇤ ( E A ) Ni , e = h n ( c ) xy i = Wi , e ⇤ I G = h g ( c ) xy i = ( Wg ⇤ I + Ug ⇤ Z ) H = h h ( c ) xy i = ( Wh ⇤ E + Uh ⇤ I ) E = h e ( c ) xy i I = h i ( c ) xy i I = h z ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = g i [ t 1 ] + ( 1 g ) [ z [ t ] ( i [ t ] a [ t ] + ) m [ t ] i [ t 1 ] ] + e [ t ] = h e [ t 1 ] + ( 1 h ) [ i [ t ] + ( ⌫i [ t ] + µ ) n [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , i = h m ( c ) xy i = We , i ⇤ ( E A ) Ni , e = h n ( c ) xy i = Wi , e ⇤ I G = h g ( c ) xy i = ( Wg ⇤ I + Ug ⇤ Z ) H = h h ( c ) xy i = ( Wh ⇤ E + Uh ⇤ I ) E = h e ( c ) xy i I = h i ( c ) xy i I = h z ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = i [ t 1 ] g + ( 1 g ) [ z [ t ] ( i [ t ] a [ t ] + ) m [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] h + ( 1 h ) [ i [ t ] + ( ⌫i [ t ] + µ ) n [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , i = h m ( c ) xy i = We , i ⇤ ( E A ) Ni , e = h n ( c ) xy i = Wi , e ⇤ I G = h g ( c ) xy i = ( Wg ⇤ I + Ug ⇤ Z ) H = h h ( c ) xy i = ( Wh ⇤ E + Uh ⇤ I ) E = h e ( c ) y i I = h i ( c ) xy i I = h z ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 Excit.Inhib . Inhib . ConnectivityInT Circuit Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i̇xyc = ixyc + 1 ⌘ ⇥ ztxyc ( exycaxyz + ) mi ! exyc ⇤ + ėxyc = exyc 1 ✏ ⇥ ixyc + ( ⌫ixyc + µ ) m e ! i xyc ⇤ + ( 1 ) where axyc = ( WA ⇤ E ) xyc mi ! exyc = ( W I ⇤ ( E Me ! a ) ) xyc me ! ixyc = ( W E ⇤ I ) xyc exyc ( 2 ) zxyc ( 3 ) ixyc ( 4 ) axyc ( 5 ) ( 6 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 7 ) where meijk = ( W I ⇤ E ) ijk miijk = ( W E ⇤ I ) ijk 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i̇xyc = ixyc + 1 ⌘ ⇥ ztxyc ( exycaxyz + ) mi ! exyc ⇤ + ėxyc = exyc + 1 ✏ ⇥ ixyc + ( ⌫ixyc + µ ) m e ! i xyc ⇤ + ( 1 ) where axyc = ( WA ⇤ E ) xyc mi ! exyc = ( W I ⇤ ( E Me ! a ) ) xyc me ! ixyc = ( W E ⇤ I ) xyc exyc ( 2 ) zxyc ( 3 ) ixyc ( 4 ) axyc ( 5 ) ( 6 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 7 ) where meijk = ( W I ⇤ E ) ijk miijk = ( W E ⇤ I ) ijk 1 Div . ( ÷ ) Sub . ( - ) Mult . ( x ) Add . ( + ) [ [ [ [ Time Preferred feature Spat ial lo catio n Excit . Inhib . Gate TimeP re fe rr ed fe at ur e Spatial location Description Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = i [ t 1 ] + ⌧1 [ z [ t ] ( i [ t ] a [ t ] + ) n [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] + ⌧2 [ i [ t ] + ( ⌫i [ t ] + µ ) m [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , a = h l ( c ) xy i = Wa ⇤ A Mi , e = h m ( c ) xy i = Wi , e ⇤ ( I Me , a ) Me , i = h n ( c ) xy i = We , i ⇤ E E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) where 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in in ut feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , an C , respectively : i [ t ] = i [ t 1 ] + ⌧1 [ z [ t ] ( i [ t ] a [ t ] + ) n [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] + ⌧2 [ i [ t ] + ( ⌫i [ t ] + µ ) m [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , a = h l ( c ) xy i = Wa ⇤ A Mi , e = h m ( c ) xy i = Wi , e ⇤ ( I Me , a ) Me , i = h n ( c ) xy i = We , i ⇤ E E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) where 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = i [ t 1 ] + ⌧1 [ z [ t ] ( i [ t ] a [ t ] + ) n [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] + ⌧2 [ i [ t ] + ( ⌫i [ t ] + µ ) m [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , a = h l ( c ) xy i = Wa ⇤ A Mi , e = h m ( c ) xy i = Wi , e ⇤ ( I Me , a ) Me , i = h n ( c ) xy i = We , i ⇤ E E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) where 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextua illusions I this model , contextual phenomena merge from in eractions between units in input feature maps X 2 RH , W , C , with height , width , and numb of channels denoted by H , W , nd C , respectively : i [ t ] = i [ t 1 ] + ⌧1 [ z [ t ] ( i [ t ] a [ t ] + ) n [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] + ⌧2 [ i [ t ] + ( ⌫i [ t ] + µ ) m [ t ] e [ t 1 ] ] + ( 1 ) , where E h e ( ) I h i ( c ) xy i A h a ( Wa ⇤ E ) Me , a h l ( c ) xy i = Wa ⇤ A Mi , e h m ( c ) xy i i , e ⇤ ( I Me , a ) Me , i = h n ( c ) xy i = We , i ⇤ E E = h e ( c ) xy i I = h i ( c ) xy i I = h z ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 No memory Memory ( a ) ( b ) ( c ) Synapses Units Abstract 1 I troduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input fe ture maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = i [ t 1 ] g + ( 1 g ) [ z [ t ] ( i [ t ] a [ t ] + ) m [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] h + ( 1 h ) [ i [ t ] + ( ⌫i [ t ] + µ ) n [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , i = h m ( c ) xy i = We , i ⇤ ( E A ) Ni , e = h n ( c ) xy i = W , e ⇤ I G = h g ( c ) xy i = ( Wg ⇤ I + Ug ⇤ Z ) H = h h ( c ) xy i = ( Wh ⇤ E + Uh ⇤ I ) E = h e ( c ) xy i I = h i ( c ) xy i I = h z ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 Abstract 1 Int oduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual phenomena emerge from interactions between units in input feature maps X 2 RH , W , C , wi h height , width , and numb r of channels denoted by H , W , and C , respectively : i [ t ] = i [ t 1 ] g + ( 1 g ) [ z [ t ] ( i [ t ] a [ t ] + ) m [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] h + ( 1 h ) [ i [ t ] ( ⌫i [ t ] + µ ) n [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , i = h m ( c ) xy i = We , i ⇤ ( E A ) Ni , e = h n ( c ) xy i = Wi , e ⇤ I G = h g ( c ) xy i = ( Wg ⇤ I + Ug ⇤ Z ) H = h h ( c ) xy i = ( Wh ⇤ E + Uh ⇤ I ) E = h e ( c ) xy i I = h i ( c ) xy i I = h z ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral co textual illusions In this model , contextual phenomena emerge from interactions etween u its in i put feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = g i [ t 1 ] + ( 1 g ) [ z [ t ] ( i [ t ] a [ t ] + ) m [ t ] i [ t 1 ] ] + e [ t ] = h e [ t 1 ] + ( 1 h ) [ i [ t ] + ( ⌫i [ t ] + µ ) n [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , i = h m ( c ) xy i = We , i ⇤ ( E A ) Ni , e = h n ( c ) xy i = i , e ⇤ I G = h g ( c ) xy i = ( Wg ⇤ I + Ug ⇤ Z ) H = h h ( c ) xy i = ( Wh ⇤ E + Uh ⇤ I ) E = h e ) xy i I = h i ( c ) xy i I = h c ) y i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 Abstract 1 Introduction 2 Results [ ? ] Mely et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral contextual illusions In this model , contextual pheno ena emerge from interactions between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i [ t ] = i [ t 1 ] g + ( 1 g ) [ z [ t ] ( i [ t ] a [ t ] + ) m [ t ] i [ t 1 ] ] + e [ t ] = e [ t 1 ] h + ( 1 h ) [ i [ t ] + ( ⌫i [ t ] + µ ) n [ t ] e [ t 1 ] ] + ( 1 ) , where E = h e ( c ) xy i I = h i ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) Me , i = h m ( c ) xy i = We , i ⇤ ( E A ) Ni , e = h n ( c ) xy i = Wi , e ⇤ I G = h g ( c ) xy i = ( Wg ⇤ I + Ug ⇤ Z ) H = h h ( c ) xy i = ( Wh ⇤ E + Uh ⇤ I ) E = h e ( c ) xy i I = h i ( c ) xy i I = h z ( c ) xy i A = h a ( c ) xy i = ( Wa ⇤ E ) ( 2 ) i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 3 ) 1 Abstract 1 Introduction 2 R sults [ ? ] M y et al developed a model of circuits in primate early visual cortex , which explained a host of behavioral co extual illusions In this model , contextual phenomena emerge fro interactio s between units in input feature maps X 2 RH , W , C , with height , width , and number of channels denoted by H , W , and C , respectively : i̇xyc = ixyc + ⇥ ztxyc ( ↵exycme ! aijk + µ ) mi ! exyc ⇤ + ėxyc = exyc + ⇥ ixyc + ( ixyc + ⌫ ) m e ! i xyc ⇤ + ( 1 ) where me ! axyc = ( W A ⇤ E ) xyc mi ! exyc = ( W I ⇤ ( E I ) ) xyc me ! ixyc = ( W E ⇤ I ) xyc i̇xyc = 2 64ztxyc ( ↵exyc + µ ) mexyc| { z } Excitatory drive 3 75 + ėxyv = 2 64ixyc + ( ixyc + ⌫ ) mixyc| { z } Inhibitory drive 3 75 + ( 2 ) w ere meijk = ( W I ⇤ E ) ijk miijk = ( W E ⇤ I ) ijk ⌘İijk + ✏ 2Iijk = ⇥ xtijk ( ↵Eijk + µ ) RExyk ⇤ + ⌧ Ėijk + 2Eijk = ⇥ Iijk + ( Iijk + ⌫ ) R I ijk ⇤ + ( 3 ) where RIijk ( W I ⇤ E ) ijk REijk ( E I ) ijk 1 Input Figure 4 : The Index-and-Track ( InT ) circuit model is inspired by Neuroscience models of motion perception [ 45 ] and e ecutive cognitive functions [ 46 ] . ( a ) The circuit receiv s input encodings from a video ( z ) , which are pro essed by interacting recurrent inhibitory and excitatory units ( i , e ) [ 7 , 10 ] , and a mechanism for selective “ attention ” ( a ) that tracks the target location . ( b ) InT units have spatiotemporal receptive fields . Spatial connections are formed by convolutions with weight kernels ( We , Wi ) . Temporal onnections are controlled by gates ( g , h ) . ( c ) Mod l para eters are fit with gradient descent . Softplus= [ . ] , sigmoid= σ , convolution= ∗ , ele entwise product = . versions with the same number of frames , and the same or different number of distractors . In the first experiment , models were trained on the 32 frame and 14 distractor PathTracker , then t sted on the 32 frame PathTracker datasets with 1 , 14 , or 25 distractors ( Fig . 3a ) . In the second experiment , models were trained on the 64 frame and 14 distractor PathTracker , then tested on t e 64 frame PathTracker datasets with 1 , 14 , or 25 distractors ( Fig . 3a ) . Models were trained to detect if the target dot reached the blue goal marker using binary crossentropy and the Adam optimizer [ 44 ] until performance on a test set of 20K videos with 14 distractors decreased for 200 straight epochs . In each experiment , we selected model weights that performed best on the 14 distractor dataset . Models were retrained three times on learning rates ∈ { 1e-2 , 1e-3 , 1e-4 , 3e-4 , 1e-5 } to optimize performance . The best performing model was then tested on the remaining 1 and 25 distractor datasets in the experiment . We used four NVIDIA GTX GPUs and a batch size 180 for training . Results We treat human performance as the benchmark for models on PathTracker . Nearly all CNNs and the ImageNet-initialized TimeSformer performed well enough to reach the 95 % human confidence interval on the 32 frame and 14 distractor PathTracker . However , all neural network models performed worse when systematically generalizing to PathTracker datasets with a different number of distractors , even when that number decreased ( Fig . 3a , 1 distractor ) . Specifically , model performance on the 32 frame PathTracker datasets was worst when the videos contained 25 distra tors : no CNN or transformer reached the 95 % confidence interval of humans on this version of the dataset ( Fig . 3a ) . The optic flow R3D and the TimeSformer trained from scratch were less successful than the standard CNNs but still above chance , while the Conv-GRU performed at chance . The SORT Kalman filter was extremely sensitive to distractors , performing better than any other model on 1-distractor PathTracker datasets , but dropping well-below the human confidence interval on 14- and 25-distractor PathTracker datasets . The performance of all models plummeted on 64 frame PathTracker datasets . The drop in model performance from 32 to 64 frames reflects a combination of the following features of PathTracker . ( i ) The target becomes more likely to cross a distractor when length and the number of distractors increase ( Fig . 2 ; SI Fig . 2c ) . This makes the task difficult because the target is momentarily impossible to distinguish from a distractor . ( ii ) The target object must be tracked from start-to-end to solve the task , which can incur a memory cost that is monotonic w.r.t . video length . ( iii ) The prior two features interact to non-linearly increase task difficulty ( SI Fig . 2c ) . Neural circuits for tracking without re-recognition PathTracker is inspired by object tracking paradigms from Psychology , which tested theories of working memory and attention in human observers [ 2 , 3 ] . PathTracker may draw upon similar mechanisms of visual cognition in humans . However , the video analysis models that we include in our benchmark ( Fig . 3 ) do not have inductive biases for working memory , and while the TimeSformer uses a form of attention , it is insufficient for learning PathTracker and only reached human performance on one version of the challenge ( Fig . 3 ) . Neural circuits for motion perception , working memory , and attention have been the subject of intense study in Neuroscience for decades . Knowledge synthesized from several computational , electrophysiological and imaging studies point to canonical features and computations that are carried out by these circuits . ( i ) Spatiotemporal feature selectivity emerges from non-linear and time-delayed interactions between countervailing neuronal subpopulations [ 47–49 ] . ( ii ) Recurrently connected neuronal clusters can maintain task information in working memory [ 46,50 ] . ( iii ) Synaptic gating , inhibitory modulation , and disinhibitory circuits are neural substrates of working memory and attention [ 51–56 ] . ( iv ) Mechanisms for gain control may aid motion-based object tracking by building tolerance to visual nuisances , such as illumination [ 57 , 58 ] . We draw from these principles to construct the “ Index-and-Track ” circuit ( InT , Fig . 4 ) . InT circuit description The InT circuit takes an input z at location x , y and feature channel c from video frame t ∈ T ( Fig . 4a ) . This input is passed to an inhibitory unit i , which interacts with an excitatory unit e , both of which have persistent states that store memories with the help of gates g , h. The inhibitory unit is also gated by another inhibitory unit , a , which is a non-linear function of e , and can either decrease or increase ( i.e. , through disinhibition ) the inhibitory drive . In principle , the sigmoidal nonlinearity of a means that it can selectively attend , and hence , we refer to a as “ attention ” . Moreover , since a is a function of e , which lags in time behind z [ t ] , its activity reflects the displacement ( or motion ) of an object in z [ t ] versus the current memory of e. InT units have spatiotemporal receptive fields ( Fig . 4b ) . Interactions between units at different locations are computed by convolution with weight kernels We , i , Wi , e ∈ R5,5 , c , c and attention is computed by Wa ∈ R1,1 , c , c . Gate activities that control InT dynamics and temporal receptive fields are similarly computed by kernels , Wg , Wh , Ug , Uh ∈ R1,1 , c , c . Recurrent units in the InT support non-linear ( gain ) control . Inhibitory units can perform divisive and subtractive operations , controlled by γ , β. Excitatory units can perform multiplicative and additive operations , controlled by ν , µ. Parameters γ , β , ν , µ ∈ Rc . “ SoftPlus ” rectifications denoted by [ . ] + enforce inhibitory and excitatory function and competition ( Fig . 4c ) . The final e state is passed to a readout for PathTracker ( SI §4 ) . InT PathTracker performance We trained the InT on PathTracker following the procedure in §4 . It was the only model that rivaled humans on each version of PathTracker ( Fig . 3 ) . The gap in performance between InT and the field is greatest on the 64 frame version of the challenge . How does the InT solve PathTracker ? There are at least two strategies that it could choose from . One is to maintain a perfect track of the target throughout its trajectory , and extrapolate the momentum of its motion to resolve crossings with distractors . Another is to track all objects that cross the target and check if any of them reach the goal marker by the end of the video . To investigate the type of strategy learned by the InT for PathTracker and to compare this strategy to humans , we ran additional psychophysics with a new group of 90 participants using the same setup detailed in §3 . Participants were trained on 8 videos from the 14 distractor and 64 frame PathTracker and tested on 72 videos from either the ( i ) 14 distractor and 64 frame dataset , ( ii ) 25 distractor and 64 frame dataset , or ( iii ) 14 distractor and 128 frame dataset . Unlike the psychophysics in §3 , all participants viewing a given test set saw the same videos , which made it possible to compare their decision strategies with the InT . InT performance reached the 95 % confidence intervals of humans on each test dataset . The InT also produced errors that were extremely consistent with humans and explained nearly all variance in Pearson ’ s ρ and Cohen ’ s κ on each dataset ( Fig . 5 , middle and right columns ) . This result means that humans and InT rely on similar strategies for solving PathTracker . What is the visual strategy learned by the InT ? We visualized activity of A units in the InT as they processed PathTracker videos and found they had learned a multi-object tracking strategy to solve the task ( Fig . 5 ; see SI §5 for method , and http : //bit.ly/InTcircuit for animations ) . The A units track the target object until it crosses a distractor and ambiguity arises , at which point attention splits and it tracks both objects . This strategy indexes a limited number of objects at once , consistent with studies of object tracking in humans [ 2 ] . Since the InT is not explicitly constrained for this tracking strategy , we next investigated the minimal circuit for learning it and explaining human behavior . We developed versions of the InT with lesions applied to different combinations of its divisive/subtractive and multiplicative/additive operations , a version without attention units A , and a version that does not make a distinction of inhibition vs. excitation ( “ complete + tanh ” ) , in which rectifications were replaced with hyperbolic tangents that squash activities into [ −1 , 1 ] . While some of these models marginally outperformed the Complete InT on the 14 distractor and 64 frame dataset , their performance dropped precipitously on the 25 distractor and 64 frame dataset , and especially the very long 14 distractor and 128 frame dataset ( Fig . 5e ) . Attention units in the complete InT ’ s nearest rival ( complete + tanh ) were non-selective , potentially contributing to its struggles . InT performance also dropped when we forced it to attend to fewer objects ( SI §5 ) .

Summary: The authors note that state-of-the-art computer vision methods for object tracking in natural videos largely rely on _appearanced-based tracking_ -- that is, some kind of matching between the features of a target object and features in subsequent frames of a movie. In contrast, people (and likely non-human animals) need to and are able to track objects in cases where appearance is unhelpful; they probably do this by somehow integrating the motion of an attended object over time.   Motivated by this discrepancy, the authors devise a tracking task in which object appearance is useless and find that people nevertheless perform well. They test a variety of neural network architectures (CNNs, RNNs, Transformers), none of which performs at human level on harder versions of the task. On the other hand, a custom-designed RNN with gating, inhibitory interactions, and "attention" both performs at human level and even makes similar errors to people. Finally, the authors show that they can improve on the state-of-the-art for several large-scale object tracking challenges by introducing their custom RNN as additional modules in the current best neural network.