Tracking Without Re-recognition in Humans and Machines
Authors: Drew Linsley
Girik Malik
Junkyung Kim
Lakshmi Narasimhan Govindarajan
Ennio Mingolla
Thomas Serre
Venue: NeurIPS 2021 Poster (2021)
Keywords: Motion processing
neural circuits
visual perception

Abstract: Imagine trying to track one particular fruitfly in a swarm of hundreds. Higher biological visual systems have evolved to track moving objects by relying on both their appearance and their motion trajectories. We investigate if state-of-the-art spatiotemporal deep neural networks are capable of the same. For this, we introduce PathTracker, a synthetic visual challenge that asks human observers and machines to track a target object in the midst of identical-looking "distractor" objects. While humans effortlessly learn PathTracker and generalize to systematic variations in task design, deep networks struggle. To address this limitation, we identify and model circuit mechanisms in biological brains that are implicated in tracking objects based on motion cues. When instantiated as a recurrent network, our circuit model learns to solve PathTracker with a robust visual strategy that rivals human performance and explains a significant proportion of their decision-making on the challenge. We also show that the success of this circuit model extends to object tracking in natural videos. Adding it to a transformer-based architecture for object tracking builds tolerance to visual nuisances that affect object appearance, establishing the new state of the art on the large-scale TrackingNet challenge. Our work highlights the importance of understanding human vision to improve computer vision.



Tracking Without Re-recognition in Humans and Machines
Drew Linsley*), Girik Malik
JJunkyung Kim
Lakshmi Govindarajan's Ennio Mingolla Thomas Serret drew www._inseee@bbron edu and malik. .iiiii..gee. gnoottheastern edu
Abstract
Imagine trying to track partieular fruitfly in swarm of hundreds. Higher biological visual systems have evolved to track moving objects by relying on both their appearance and their motion trajectories. We investigate if state-of-the- art spatiotemporal deep neural networks are capable of the same. For this, we introduce PathTrucker, synthetic visual challenge that asks human observers and machines to track target object in the midst of identical-looking "distractor" objects. While humans effortlessly learn PathTracker and generalize to systematic variations in task design, deep networks struggle. To address this limitation, we identify and model circuit mechanisms biological brains that are implicated tracking objects based on motion cues. When instantiated as recurrent network, our circuit model learns solve PathTracker with a robust visual strategy that rivals human performance and explains significant proportion of their decision-makinn on the challenge. We also show that the success of this circuit model extends to object tracking in natural videos. Adding it to transformer-bait based architecture for object tracking builds tolerance to visual nuisances that affect object appearance, establishing the new state of the art on the large-scale TrackingNet challenge. Our work highlights the importance of understanding human vision to improve computer vision.
Introduction
Lettvin and colleagues [ presciently noted, "The frog does not seem to see or, at any rate, is not concerned with the detail of stationary parts of the world around him. He will starve to death surrounded by food if is not moving." " Object tracking fundamental to survival, and higher biological visual systems have evolved the capacity for two distinct and complementary strategies to do it. Consider Figure [ can you track the object labeled by the yellow arrow from left-to-right? The task is trivial when appearance cues, like color, make it possible to solve the temporal correspondence problem by "re-recognizing" the target in each frame (Fig. Th.. However, this strategy is not effective when objects cannot be discriminated by their appearance alone (b). In this object motion over time necessary for tracking. Humans are capable of tracking objects by their motion when appearance is uninformative 2 but it is unclear if the current generation of neural networks for video analysis and tracking can do the same. To address this question we introduce PathTracker, synthetic challenge for object tracking without re-recognition (Fig. [e].
integration of
These authors contributed equally to this work. CCamey Institute for Brain Science, Brown University, Providence, RI "Nottheastemn University, Boston, MA 'DeepMind, London, UK
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Leading models for video analysis rely on object classification pre-training. This gives them access to rich semantic representations that have supported soute-of-the-eee performance on host of tasks, from action recognition to object tracking [4 [ As object classification models have improved, so too have the video analysis models that depend on them. This trend in model development has made unclear if video analysis models are effective at learning tasks when appearance cues are uninformative. The importance of diverse visual strategies has been highlighted by synthetic challenges like Pathfinder, visual reasoning task that asks ob- servers to trace long paths embed- ded in a static cluttered display 87 Patlifinder tests object segmentation when appearance cues like category shape are missing. While humans can easily solve it (8). deep neural net- works struggle, including state-of-the- art vision transformers 7-).- Impor-
tantly, models that learn an appropri- visual strategy for Pathfinder also exhibit more efficient learning and im- proved generalization on object seg- mentation in natural images [I] Our PathTracker challenge extends this line of work into video by pos- ing an object tracking problem where the target can be tracked by motion and spatiotemporal continuity, not cat- egory or appearance.
Time
The appearance of objects makes them (a) easy or (b) hard to track. We introduce the PathTracker Challenge (c), which asks observers to track a particular green dot travels from the red-to-blu markers, testing object tracking when re-recognition impossible.
Figure 1:
Contributions. Humans effortlessly solve our novel PuthTracker challenge. A variety of state-of- the-art models for object tracking and video analysis do not.
We find that neural architectures including R3D [2] and state-of-ther tfetthe-rrt transformer-baut TimeS- formers (5) are strained by long PathTracker videos. Humans, on the other hand, are far more effective at solving these long PathTracker videos.
We describe a solution to PuthTracker: a recurrent network inspired by primate neural circuitry involved in object tracking, which renders decisions that strongly correlated with humans.
These same circuit mechanisms improve object tracking in natural videos through motion-based strategy that builds tolerance to changes in target object appearance, resulting in the state-of-the-a score TrackingNet [13).
We release all PathTracker data, code, and human psychophysics at ahttt::/biit.ll/IITarrrr to spur interest in the challenge of tracking without re-recognition.
Related Work
Models for video analysis A major leap in the performance of models for video analysis came from using networks which are pre-trained for object recognition on large image datasets (4). The recently introduced TimeSformer (5) achieved sfate-of-the-art performance with weights initialized from an image categorization transformer (VIT; (14) that was pre-trained on ImageNet-21KK The modeling trends are similar in object tracking [13], where successful models rely on "backbone" feature extraction networks trained on ImageNet or Microsoft COCO (16) for object recognition or
segmentation 17).
Shortcut learning and synthetic datasets A byproduct of the great power of deep neural network architectures is their vulnerability learning spurious correlations between inputs and labels. Perhaps because of this tendency, object classification models have trouble generalizing to novel contexts [18] 19), and render idiosyncratic decisions that are inconsistent with humans (0 Synthetic datasets are effective at probing this vulnerability because they make it possible to control spurious image/label correlations - providing fairer test of the computational abilities of these models. For example, the
Pathfinder challenge was designed to test if neural architectures can trace long curves in clutter

---

visual computation associated with the earliest stages of visual processing in primates. That challenge identified diverging visual strategies between humans and transformers that are otherwise state of the
art in natural image object recognition [9 14). Other challenges like Bongard-L.OGO [23], cABC (8],
SVRT (24), and PSV (25) have highlighted limitations of leading neural network architectures
that would have been difficult to identify using natural image benchmarks like ImageNet (26). These
limitations have inspired algorithmic solutions based on neural circuits discussed in SI $1.
Translating circuits for biological vision into artificial neural networks While the Pathfinder challenge (7) presents immense challenges for transformers and deep convolutional networks (8) can be solved by simple model of intrinsic connectivity in visual cortex, with onders-of-magntuun fewer parameters than standard models for image categorization. This model was developed by translating descriptive models of neural mechanisms from Neuroscience into architecture that can be fit to data using gradient descent [I]. Others have found success in modeling object tracking by drawing inspiration from "dual stream" theories of appearance and motion processing in visual cortex (27-30), or basing the architecture off of a partial connectome of the Drosophila visual system (3I]. We adopt a similar approach in the current work, identifying mechanisms for object tracking without re-recognition in Neuroscience, and developing those into differentialle operations with parameters that can be optimized by gradient descent. This approach has the dual purpose of introducing task-relevant inductive biases into computer vision models, and developing theory on their relative utility for biological vision.
Multi-object tracking in computer vision The classic psychological paradigms of multi-object tracking 2 motivated the application of models, like Kalman filters, which had tolerance to object occlusion when they relied on momentum models (32). However, these models are computationally expensive, hand-tuned, and because of this, no longer commonly used in computer vision (33) More recent approaches include flow tracking on graphs (34) and motion tracking models that are relatively computationally efficient [35|36] . However, even current approaches to multi-object tracking are not learned, instead relying on extensive hand tuning (37 38) In contrast, the point of PathTracker is to understand the extent which state-of-the-ar neural networks are capable of tracking a single object
in an array of distractors.
3 The Path Tracker Challenge
Overview PathTracker asks observers to decide whether or not a target dot reaches goal location (Fig. 2 The target dot travels in the midst of a pre-specified number of distractors. All dots are identical, and the task difficult because of this: (i) appearance not useful for tracking the target, and (ii) the paths of the target and distractors can momentarily "cross" and occupy the same space, making it impossible to individuate them in that frame and meaning that observers cannot solely rely on the target's 's location to solve the task. This challenge is inspired by object tracking paradigms in cognitive psychology (23 39 which suggest that humans might tap into mechanisms for motion perception, attention and working memory to solve a task like PathTracker.
The trajectories of target and distractor dots are randomly generated, and the target occasionally crosses distractors (Fig. 2). These object trajectories are smooth by design, giving the appearance of objects meandering through scene, and the difference between the coordinates of any dot on successive frames is no more than 2 pixels with less than 20 of angular displacement. In other words, dots never turn at acute angles. We develop different versions of PathTracker with varying degrees of complexity based on the number of distractors and/or the length of videos. These variables change the expected number of times that distractors cross the target and the amount of time that observers must track the target (Fig. 2 To make the task as visually simple as possible and maximize contrast between dots and markers, the dots, start, and goal markers are placed on different channels in 32x32 pixel three-channel images. Markers are stationary throughout each video and placed at random locations. Examples videos can be viewed at
Human benchmark We began by testing if humans can solve PathTracker. We recruited 180 indi- viduals using Amazon Mechanical Turk to participate in this study. Participants viewed Prrcerakker videos and pressed button on their keyboard to indicate if the target object or distractor reached the goal. These videos were played in web browsers at 256x256 pixels using HTML55, which helped ensure consistent frame rates [40]. The experiment began with an 8-trial "training" stage, which
3

---

distractor and 32 frames
4
75 32 num.
(a)
(b)
8
24
32
14 distractors and 32 frames
(c)
(d)
16 32
Frame num.
8
16
24
32
14 distractors and 64 frames
8
16
24
32
40
48
56
64
10 40
(e)
Frame
Frame number
Figure 2: PathTracker is a synthetic visual challenge that asks observers to watch a video clip and answer if target dot starting in red marker travels to a blue marker. The target dot surrounded by identical "distractor" dots, each of which travels in a randomly generated and curved path. positive examples, the target dot's path ends in the blue square. In negative examples, "distractor" dot ends in the blue square. The challenge the task is due to the identical appearance of target and distractor dots, which, as will show, makes appearance based tracking strategies ineffective. Moreover, the target dot can momentarily occupy the same location as distractor when they cross each other's paths, making it impossible to individuate them in that frame and compelling strategies like motion trajectory extrapolation or working memory to recover the target track. (b) A 3D visualization of the video in (a) depicts the trajectory of the target dot, traveling from red-to-blue markers. The target and distractor cross approximately half-way through the video. (c,d) We develop versions of PathTracker that observers sensitivity the number distractors and length videos (e.f). The number of distractors and video length interact to make it more likely for the target dot to cross distractor in video (compare the one in b vs. two in d vs. three f; see SI $2 for details).
familiarized participants with the goal of PathTracker. Next, participants were tested on 72 videos. The experiment was paced and lasted approximately 25 minutes, and participants were paid $8 for their time. See http:: http//tttlllll www.IyyyInnuuii... and SI $2 for an example and more details.
Participants were randomly entered into one of two experiments. In the first experiment, they were trained on the 32 frame and 14 distractor PathTracker, and tested on 32 frame versions with 1, 14, or 25 distractors. In the second experiment, they were trained the 64 frame and 14 distractor PathTracker, and tested on 64 frame versions with 1, 14, or 25 distractors. All participants viewed unique videos to maximize were significantly above chance on tested conditions of PathTracker (p < 0.001, test details SI $2). They also exhibited significant negative trend in performance the 64 frame datasets the number of distractors increased (L = -2.74, p < 0.01). There was no such trend on the 32 frame datasets, and average accuracy hetween the datasets not significantly different. These results validate our initial design assumptions: humans can solve PathTracker, and manipulating distractors and video length increases difficulty.
sampling over the different versions of PathTracker. Participants
4 Solving the Path Tracker challenge
Can leading models for video analysis match humans on PathTracker? To test this question surveyed variety of architectures that are the basis for leading approaches to many video analysis tasks, from object tracking to action classification. The models fall into four groups: (i) deep

---

(a)
Train: 14 distractors and 32 frames
Test: distractors and 32 frames
Test: 14 distractors and 32 frames
Test: 25 distractors and 3? frames
100%
Accuracy
Menn butte acs.
80%
60%
40%
(b)
Train: 14 distractors and 64 frames
Test:
distractors and 64 frames
Test: 14 distractors and 64 frames
Test: 25 distractors and 64 frames
100%
Accuracy
80%
80%
Int Circul imsgetet RaD imagehee Stade PotD
40%
imageliat MCg
Time Storner
SORT R3D Flaar
Rile
Time Sherar Cone GHU
Int Clrcuat
imsuttee RD1110 RND
Imagehei MC3
Imsgetter Timestomm
Kaiman Flaar
RaD Cpic Flawr
Fimestorme Storarr
GNU
Inf Cirouil
imagetter RSD slude RaD
imageseet Mes
1D
Imagatie Tima Stmast
Kalnan Pouer
RSD Qutl Ploe
Sloymer
Imagettar
SORT
SORT
Figure 3: Model accuracy on the PathTracker challenge. Video analysis models were trained to solve 32 (a) and 64 frame (b) versions of challenge, which featured the target object and 14 identical distractors. Models were tested on PathTracker datasets with the same number of frames but I, 14, or 25 distractors (leftmiidle/rightt) Colors indicate different instances of the same type of model. Grey hatched boxes denote 95% bootstrapped confidence intervals for humans. Only our InT Circuit rivaled humans each dataset.
convolutional networks (CNNs), (ii) transformers, (iii) recurrent neural networks (RNNs), and (iv)
Kalman filters. The deep convolutional networks include a 3D ResNet (R3D [T2] ). space/time separated ResNet with "2D-spatial + DDtempoorll" convolutions (R(2+1)D 12) and a ResNet with 3D convolutions in early residual blocks and 2D convolutions later blocks (MC3 (12). We trained versions of these models with random weight initializations and weights pretrained on ImageNet. We included an R3D trained from scratch without any downsampling. in case the small size of PathTracker videos caused learning problems (see SI $3 for details). We also trained a version of the R3D on optic flow encodings of PathTracker (SI $3). For transformers, we turned to the TimeSformer (5) We evaluated two of its instances: (i) where attention is jointly computed for all locations across space and time in videos, and (ii) where temporal attention is applied before spatial attention, which results in massive computational savings. Both models performed similarly on PathTracker. We report the latter version here as was marginally better (see SI $3 for performance of the other, joint space-time attention TimeSrrrmer). We include a version of the TimeSformer trained from scratch, and a version pre-trained on ImageNet-20K. Note that statc-of-the-ans transformers for object tracking in natural videos feature similar deep and multi-headed designs (6]. For the RNNs, we include a convolutional-ge gated recurrent unit (Conv-GRU) [41). Finally, our exemplar Kalman filter is the standard Simple and Online Realtime Tracking (SORT) algorithm, which was fed coordinates of the objects extracted from each frame of every video (37].
Method The visual simplicity of ParhTracker cuts two ways: it makes it possible to compare human and model strategies for tracking without re-recognition as long as the task not too easy. Prior synthetic challenges like Pathfinder constrain sample sizes for training to probe specific computations (79.. We adopted the following strategy to select a training set size that would help us test tracking strategies that do not depend on re-recognition. We took Inception 3D (I3D) networks (4) which have been a strong baseline architecture in video analysis over the past several years, and tested their ability to learn PathTracker SI $1, when this model was trained with 20K examples of the 32 frame and 14 distractor version of PathTracker it achieved good performance the task without signs of overfitting. We therefore trained all models in subsequent experiments with 20K examples. By happenstance, this dataset size gives PathTracke
adjusted the number of videos for training. As we discuss in
comparable number of frames to large-scale real world tracking challenges like LaSOT 42 and GOT- [43).
We measure the ability of models to learn PathTrackr and systematically generalize to novel versions of the challenge when trained on 20K samples. We trained models using a similar approach as in our human psychophysics. Models were trained on one version of PathTracker, and tested other
5

---

InT Circuit
Connectivity
Description
Inhib.
Exelt.
Synapses
[=ld (gir)aad) g) d[t] g 1
1]1-
566. Di (+)
Preferred feature
A.fe 1] A) (30) (or(
e) mld
<lc 1[]-
InA
O
w
where
M,
m00
W.,
Codaa
R-
N,
W.../
Spatial location
G
-o[W, + U,+
(8)
Input
Inhib.
NO momory
(b)
(0)
A -oW, + U.
11
So U.
Figure 4: The Index-and- Track (InT) circuit model is inspired by Neuroscience models of motion
perception [45] and executive cognitive functions [46). (a) The circuit receives input encodings from a video (2), which are processed by interacting recurrent inhibitory and excitatory units (i, é) [7 and a mechanism for selective "attention" (a) that tracks the target location. (b) InT units have spatiotemporal receptive fields. Spatial connections are formed by convolutions with weight kernels (We, W,). Temporal connections are controlled by gates (g, h). (c) Model parameters are fit with gradient descent. Softplus= [-], sigmoid= a, convolution *, elementwise product = O
versions with the same number of frames, and the same or different number of distractors. In the first experiment, models were trained on the 32 frame and 14 distractor PathTracker, then tested on the 32 frame PathTracker datasets with I, 14, or 25 distractors (Fig. FA. In the second experiment, models were trained on the 64 frame and 14 distractor PathTracker then tested on the 64 frame PathTracker datasets with 1, 14, or 25 distractors (Fig. 3a). Models were trained to detect if the target dot reached the blue goal marker using binary crossentropy and the Adam optimizer (44) until performance on a test set of 20K videos with 14 distractors decreased for 200 straight epochs. In each experiment, we selected model weights that performed best on the 14 distractor dataset. Models were retrained three times on learning rates E (1e-2, le-3, 1e-4, 3e-4, le-5) to optimize performance. The best performing model was then tested on the remaining and 25 distractor datasets in the experiment. We used four NVIDIA GTX GPUs and batch size 180 for training.
Results We treat human performance as the benchmark for models on PathTracker. Nearly all CNNs and the imageNet-initialzzed TimeSformer performed well enough to reach the 95% human confidence interval on the 32 frame and 14 distractor PathTrucker. However, all neural network models performed worse when systematically generalizing to ParTTacker datasets with a different number of distractors, even when that number decreased (Fig. Ba. 1 distractor). Specifically, model performance on the 32 frame PathTracker datasets was worst when the videos contained 25 distractors: no CNN or transformer reached the 95% confidence interval of humans this version of the dataset (Fig. B4).
The optic flow R3D and the TimeSformer trained from scratch were less successful than the standard CNNs but still above chance, while the Cony-GRU performed at chance. The SORT Kalman filter was extremely sensitive to distractors, performing better than any other model on 1-distractor Path PathTracker datasets, but dropping well-below the human confidence interval on 14- and 25-distractor PathTracke datasets.
The performance of all models plummeted on 64 frame PathTracker datasets. The drop in model performance from 32 to 64 frames reflects a combination of the following features of PathTracker. (i) The target becomes more likely to cross a distractor when length and the number of distractors increase [ SI Fig. 2c). This makes the task difficult because the target is momentarily impossible to distinguish from a distractor. (ii) The target object must be tracked from start-to-end to solve the task, which can incur a memory cost that is monotonic w.r.t. video length. (ii) The prior two features interact to non-linearly increase task difficulty (SI Fig. 2c).
Neural circuits for tracking without re-recognition PathTracker is inspired by object tracking paradigms from Psychology, which tested theories of working memory and attention in human observers PathTracker may draw upon similar mechanisms of visual cognition in humans. However, the video analysis models that we include in our benchmark (Fig. do not have inductive biases for working memory, and while the TimeSformer uses form of attention, it is insufficient for learning PathTracker and only reached human performance on one version of the challenge (Fig. 3)

---

Train (all rows): 14 distractors and 64 frames
Test: 14 distracttors and 64 frames
Complete InT Circuit Altention per-framm
192%
14
1.0
97%
Accuracy
40%
Pearson's 8
Cohen's K
0.-
70%
0.4
60%
0.-
0:0
0.0
32 Frame number
(a) 42%
(b)
Test: distraccors and frames
92%
0.-
Intorsblect
0.8
Accuracy
80%
Pearson's
Cohen's K
06-
70%
0.4
Inteondieed
#0%
0.2
0.2
63%
-0.2
-0.-
Frame number
(c) 47%
(d)
Test: 14 distractors and 128 frames
102%
1.0
95%
Addiive
Accuracy
82%
Pearson's
0:5
Cohen's 9
0.5
-Scoractiv ins belan
Divavee Bon
70%
Intersutjedt
90%
0.2
0.2
90%
64
(e)
-0.-
-0.-
Frame number
complete Tarh
complete
no atin altn
completo completee+ larth
etn
and attn
complnto land
completo
no atin and min
Figure 5: Performance, decision correlations, and error consistency between models and human observers on PathTracker. In a new set of psychophysics experiments, humans and models were trained on 64 frame PathTracker datasets with 14 distractors, and rendered decisions challenging versions. Decision correlations are computed with Pearson's p. and error consistency with Cohen's 's x [59]. Only the Complete InT circuit rivals human performance and explains the majority of their decision and error variance on each test dataset (a,c,e). Visualizing InT attention (a) reveals that it has learned to solve PatheTracker by multi-object tracking (b.d,f; color denotes time).
variety of
Neural circuits for motion perception, working memory, and attention have been the subject of intense study in Neuroscience for decades. Knowledge synthesized from several computational, electrophysiological and imaging studies point canonical features and computations that are carried out by these circuits. (i) Spatiotemporal feature selectivity emerges from non-linear and
time-delayed interactions between countervailing neuronal subpopulations 447 (ii) Recurrently
connected neuronal clusters can maintain task information in working memory [46 50]. (iii) Synaptic
gating, inhibitory modulation, and disinhibitory circuits are neural substrates of working memory
and attention (56).)). (iv) Mechanisms for gain control may aid motion based object tracking by
building tolerance to visual nuisances, such illumination (57,58). We draw from these principles to construct the aeeee--nn-Trraeee circuit (InT, Fig. 4.
InT circuit description The InT circuit takes an input & at location 2, y and feature channel c from video frame E T (Fig. 44). This input is passed to an inhibitory unit i, which interacts with an excitatory unit both of which have persistent states that store memories with the help of gates g, h. The inhibitory unit also gated by another inhibitory unit, a, which a non-linear function of and can either decrease or increase (i.e., through disinhibition) the inhibitory drive. In principle, the sigmoidal nonlinearity of means that it can selectively attend, and hence, we refer to as "attention". Moreover, since is a function of e, which lags in time behind =({), its activity reflects the displacement (or motion) of an object in 2 versus the current memory of e. InT units have spatiotemporal receptive fields (Fig. 4b). Interactions between units at different locations are computed by convolution with weight kernels We,s, Wi.e E _,.,..c. and attention is computed by W, E IR 1.1.c. Gate activities that control InT dynamics and temporal receptive fields are similarly computed by kernels, W Wn, U.. U, E R...I.... Recurrent units in the InT support non-linear (gain) control. Inhibitory units can perform divisive and subtractive operations, controlled by 7, B. Excitatory units can perform multiplicative and additive operations, controlled by v, μ. Parameters
7

---

Int+TransT:
Target object crop on Tab
Tabs
Tar
Object
(8)
Image
Circuit
(8)
TT-TannTT
(c)
Figure 6: Circuit mechanisms for tracking without re re-recognition build tolerance visual nuisances that affect object appearance. (a) The TransT [60] a transformer architecture for object tracking. We develop extension, the InT+TranST, in which our InT circuit model recurrently modulates Trans'T activity. Unlike the TransT, the InT+TranST' trained on sequences to promote tracking strategies that do not rely re-recognition. (b-d) The InT+Trans'] excels when the target object visually similar to other depicted objects, undergoes changes in illumination, or is occluded.
7, u E R "SoftPlus" rectifications denoted by [[]+ enforce inhibitory and excitatory function and competition (Fig. 4.. The final E state passed to a readout for PathTracker (SI $4).
InT Path Tracker performance We trained the InT on PathTracker following the procedure in $4 It was the only model that rivaled humans on each version of PathTracker (Fig. The gap in performance between InT and the field is greatest the 64 frame version of the challenge.
There are at least two strategies that it could choose from. One is to maintain perfect track of the target throughout its trajectory, and extrapolate the momentum of its motion to resolve crossings with distractors. Another is to track all objects that cross the target and check if any of them reach the goal marker by the end of the video. To investigate the type of strategy learned by the InT for PathTracker and to compare this strategy to humans, we ran additional psychophysics with were trained on videos from the 14 distractor and 64 frame PathTracke and tested on 72 videos from either the (7) 14 distractor and 64 frame dataset, (ii) 25 distractor and 64 frame dataset, (iii) 14 distractor and 128 frame dataset. Unlike the psychophysics in $3 all participants viewing a given test the videos, which made possible compare their decision strategies with the InT.
How does the InT solve PathTrach
group of 90 participants using the same setup detailed in $ Participants
InT performance reached 95% confidence intervals of humans on each test dataset. The InT also produced errors that were extremely consistent with humans and explained nearly all variance in Pearson's 's P and Cohen's x on each dataset (Fig. 5 middle and right columns). This result means that humans and InT rely similar strategies for solving PathTracker.
What is the visual strategy learned by the InT? We visualized activity of units in the InT as they processed PathTracker videos and found they had learned multi-object tracking strategy to solve the task (Fig. 5 see SI $5 for method, and http::/biis 1y InTeircuit] for animations). The A units track the target object until it crosses distractor and ambiguity arises, at which point attention splits and it tracks both objects. This strategy indexes a limited number of objects once, consistent with [2. Since the InT is not explicitly constrained for this tracking strategy, we next investigated the minimal circuit for learning it and explaining human behavior.
studies of object tracking in humans
We developed versions of the InT with lesions applied to different combinations of its divi- sive/subtractive and multiplicative/addiiiee operations, a version without attention units A, and a version that does not make a distinction of inhibition vs. excitation ("complete + tanh"), in which rectifications were replaced with hyperbolic tangents that squash activities into [-1, 1). While some of these models marginally outperformed the Complete In' on the 14 distractor and 64 frame dataset, their performance dropped precipitously on the distractor and 64 frame dataset, and especially the very long 14 distractor and 128 frame dataset (Fig. (b).. Attention units in the complete In''s nearest rival (complete tanh) were non-selective, potentially contributing to struggles. InT performance also dropped when forced it to attend to fewer objects (SI $5).

---

5 Appearance-free mechanisms for object tracking in the wild
The InT solves PathTracker by learning to track multiple objects at once, without relying on the re-reconiition strategy that has been central progress in video analysis challenges in computer vision. However, it is not clear if tracking without re-recognition is useful in the natural world. We test this question by turning to object tracking in natural videos. At the time of writing, the state-of-the-art object tracker is the TransT [60]. a deep multihead transformer [6]]. The TransT finds pixels in video frame that match the appearance of image crop depicting target object. During training, the TransT receives a tuple of inputs, consisting of this target object image and a random additional "search frame" from the same video. These images are encoded with modified ResNet50 (2). passed to separate transformers, and finally combined by a "cross-feature" attention" (CFA) module, which compares the two encodings via a transformer key//uery//value computation. The target frame is used for key and value operations, and the search frame is used for the query operation. Through its pure appearance- based approach to tracking, the TransT has achieved top performance on TrackingNet [3]. LaSOT 42). and GOT-10K 43
InT+TransT We tested whether or not the InT circuit can improve TransT performance by learning a complementary object strategy that does not depend on appearance, or re-recognition. We reasoned that this strategy might help TransT tracking in cases where objects are difficult to discern by their appearance, such as when they are subject to changing lighting, color, or occlusion. We thus developed the InT+TransT, which involves the following modifications of the original TransT (Fig. (b). (i) We introduce two InT circuits to form a bottom-up and top-down feedback loop with the TransT (3) J. which in principle will help the model select the appropriate tracking strategy depending
the
video: re-recognition or not. One InT receives ResNet50 search image encodings and modulates the TransT'ss CFA encoding of this search image. The other receives the output of the TransT and uses this information to update memory in the first In'. (ii) The TransT is trained with pairs of target and search video frames, separated in time by up to 100 frames. We introduce the intervening frames to the InT circuits. See SI $6 for extended methods.
Training InT+TransT training and evaluation hews close to the TransT procedure. This includes training on the latest object tracking challenges in computer vision: TrackingNet (13), LaSOT (42) and GOT-10K (43) All three challenges depict diverse classes of objects, moving in natural scenes that range from simplistic and barren to complex and cluttered. TrackingNer (30, 132 train and 511 test videos) and GOT-1OK (10,000 train and 180 test) evaluation is performed on official challenge servers, whereas LaSOT (1,120 train and 280 test) is evaluated with Matlab toolbox. While the TransT is also trained with static images from Microsoft COCO (16), in which the search image is an augmented version of the target, we do not include COCO in TnTnTra since we expect object motion to be an essential feature for our model 60). The InTaTaS is initialized with TransT weights and trained with AdamW (64) and a learning rate of 1c-4 for InT parameters, and le-6 for parameters in the Trans'T readout and CFA module. Other TransT parameters are frozen and not trained. The T is trained with the same objective functions as the TransT for target object bounding box prediction in the search frame, and an additional objective function for bounding box prediction using In'T circuit activity in intervening frames. The complete model was trained with batches of 24 videos on 8 NVIDIA GTX GPUs for 150 epochs (2 days). We selected the weights that performed best on GOT- 10K validation. hyperparameter controls the number of frames between the target and search that are introduced into the InT during training. We relied on coarse sampling (maximum of 8 frames) due to memory issues associated with recurrent network training on long sequences
Results An InT+TransT trained on sequences of 8 frames performed inference around 30FPS on a single NVIDIA GTX and beat the TransT on nearly all benchmarks. It is in first place on the TrackingNet leaderboard /htaa: tracking net org/), ) better than the TransT on LaSOT, and rivals the TransT on the GOT-1OK challenge (Table 5). The InT+TransT performed better when trained with longer sequences (compare T = and T Table 5). Consistent with InT success on ParhTracker, the InT+TranST was qualitatively better than the TransT on challenging videos where the target interacted with other similar looking objects (Fig. 6
http:///bt..lyyIfccirrcco
We also found that the Trans'T excelled in other challenging tracking conditions. The LaSOT challenge provides annotations for challenging video features, which reveal that the InT+TransT is
9

---

Model
TrackingNet [3]
LaSOT 42
GOT
43
Color
rColor
Ocel.
InT+Trans T
87.5
74.0
72.2
43.1
62.5
56.9
TTT+TransTT== 87.3
73.6
70.0
36.2
37.8
25.4
TransT [60]
86.7
73.8
72.3
40.7
57.5
55.2
Table 1: Model performance TrackingNet (P norm), LaSOT norm)) GOT-1OK (AO), and pertur-
bations applied to the GOT-IOK (AO). Best performance is in black, and state of the art is bolded. Perturbations on the GOT-IOK are color inversions every frame (Color) or random frames (rColor). and random occluders created from scrambling image pixels (Occl.). InT+TransTy=s was trained sequences of frames, and InT+TransTy-- was trained on 1 1-frame sequences.
especially effective for tracking objects with "deformable" parts, such moving wings tails (SI $6). We further test if introducing object appearance perturbations to the GOT-1OK might distinguish performance between the Trans'T and InT+TransT. We evaluate these models on the GOT-10K test set with one of three perturbations: inverting the color of all search frames (Color), inverting the color of random search frames (rColor), or introducing random occlusions (Occl.). The InT+TransT outperformed the TransT on each of these tests (Table
6 Discussion
A key inspiration for our study is the centrality of visual motion and tracking across a broad phylogenetic range, via three premises: (i) Object motion integration over time per se is essential for ecological vision and survival [T] (ii) Object motion perception cannot be completely reduced to recognizing similar appearance features two different moments in time. In perceptual phenomena like phi motion, the object that is tracked is described as "formless" with no distinct appearance [65]. (iii) Motion integration over space and time is a basic operation of neural circuits in biological brains, which can be independent of appearance [66]. These three premises form the basis for our work.
We developed ParkTraccke to test whether state-of-the-art models for video analysis solve visual task when object appearance ambiguous. Prior visual reasoning challenges like Pathfinder (7H)). indicate that this is problem for object recognition models, which further serve as a backbone for many video analysis models. While no existing model was able to contend with humans on PathTracker, our InT circuit was. Through lesioning experiments, we discovered that the InT's ability to explain human behavior depends on its full array of inductive biases, helping it learn a visual strategy that indexes and tracks limited number of the objects at once, echoing classic theories on the role of attention and working memory in object tracking [2
We further demonstrate that the capacity for video analysis without relying re-recognition helps in natural scenes. Our Tran'T model is more capable than the Trans'i' at tracking objects when their appearance changes, and is the state of the demonstrate that object appearance is a necessary element for video analysis, but it is not sufficient for modeling biological vision and rivaling human performance.
the TrackingNet challenge. Together, our findings
Acknowledgments and Disclosure of Funding
We are grateful to Daniel Bear and Matthew Ricci for their suggestions to improve this work. We would also like to thank Rajan Girsa for help on Mechanical Turk experiments. GM is affiliated with Labrynthe Pvt. Ltd., New Delhi, India. Funding provided by ONR grant 1N00014-19-1-200 NSF grant (IIS-195 (91-10,,,,,), and the ANR-3IA Artificial and Natural Intelligence Toulouse Institute 1N-R-19-P3AA-00 Computing hardware supported by NIH grant S100D0251 181 and the Center for Computation and Visualization (CCV) at Brown University.
References
[1] Lettvin, J.Y Maturana, H.R., McCulloch, W.S., Pitts, W.H.: What the frog's 's eye tells the frog's brain. Proceedings of the IRE (1) (November 1959) 1940-1951
[2] Pylyshyn. Z. W.W Storm, R.W.: Tracking multiple independent targets: Evidence for parallel tracking mechanism. Spat. Vis. 3(3) (1988) 179 -97
[3 Blaser, E., Pylyshyn, Z.W., Holcombe, A.O.: Tracking an object through feature space. Nature 408(6809) (November 2000) 196-199
10

---

[4] Carreira, J. Zisserman, A.: Quo vadis, action recognition? new model and the kinetics dataset. In: 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), IEEE (July 2017)
[5] Bertasius, G., Wang, H., Torresani, L.: Is Space-Time attention all you need for video understanding?
ICML (2021)
[6] Wang. N., Zhou, W., Wang. J., Li, H.: Transformer meets tracker: Exploiting temporal context for robust visual tracking. (March 2021)
[7] Linsley, D., J.m, Vecrabadran, V., Windolf, C., T: Learning long range spatial dependencies with
horizontal gated recurrent units. Bengio, S., Wallach, H.,, Larochelle, H., Grauman, K., Cesa- ussa-Bianchi, N., Garnett, R., eds.: Advances in Neural Information Processing Systems 31. Curran Associates, Inc. (2018) 152-164
[8] Kim*, J., Linsley*, D.. Thakkar, K., Serre, T.: Disentangling neural mechanisms for perceptual grouping. International Conference on Representation Learning (2020)
[9] Tay. Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J. Yang, L., Ruder, S., Metzler, D.: Long range arena: A benchmark for efficient transformers. ICLR (2021)
[10] Linsley, D., Kim, J., Ashok, A., Serre, T.: Recurrent neural circuits for contour detection. International Conference on Learning Representations (2020)
[11] Linsley, D., Ashok, A., Govindarajan, I.., Liu, R., Serre, T:: Stable and expressive recurrent vision models. NeurIPS (2020)
[12] Tran, D., Wang, H., Torresani. L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: 2018 IEEE/CVF Conference Computer Vision and Pattem Recognition, IEEE (June 2018)
[13] Müller, M., Bibi, A.. Giancola, S., Alsubaihi, S., Ghanem, B.: TrackingNet: A Large-Scale dataset and benchmark for object tracking in the wild. In: Computer Vision ECCV 2018, Springer International Publishing (2018) 310-327
[14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at seale. ICLR (2021)
[15] Fiaz, M., Mahmood, A., Jung, S.K.: Tracking noisy targets: A review of recent object tracking approaches. (February 2018)
[16] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: Computer Vision - ECCV 2014. Lecture notes in computer science. Springer International Publishing, Cham (2014) 740-755
[17] Bertasius, G., Torresani, L.: Classifying, segmenting, and tracking object instances in video with mask propagation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). (June 2020) 9745
[18] Barbu, A., Mayo, D., Alverio, Luo, W., Wang, C., Gutfreund, D., Tenenbaum, Katz, B.: ObjectNet: A large-scale bias-controlled dataset for pushing the limits object recognition models. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alché-Buc, F,, Fox, E., Garnett, R., eds.: Advances in Neural Information Processing Systems 32. Curran Associates, Inc. (2019) 9453-9463
[19] Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11) (November 2020) 665-673
[20] Ullman, S., Assif, L.,, Fetaya, E., Harari, D.: Atoms of recognition human and computer vision. Proc. A. 113(10) (March 2016) 2744-2749
Natl. Acad. Sci.
[21] Linsley, D., Eberhardt, S. Sharma, T., Gupta, P., Serre, T.: What are the visual features underlying
human versus machine vision? In: 2017 IEEE International Conferenee Computer Vision Workshops (ICCWW). (October 2017) 2706-2714
[22] Linsley, D., Shichler, D., Eberhardt, S., Serre, T.: Learning what and where to attend. ICLR (2019)
[23] Nie, W., Yu, Z., Mao, L., Patel, A.B., Zhu, Anandkumar, A.: Bongard-LOGO: A new benchmark for Level-Level concept learning and reasoning. NeurIPS (2020)
[24] Fleuret, F, Li, T., Dubout, C., Wampler, E.K., Yantis, S., Geman, D.: Comparing machines and humans on
visual categorization test. Proc. Natl. Acad. Sci. U. S. A. 108(43) (2011) 17621-17625
[25] Kim, J., Ricei, M., Serre, T.: Not-So-CLEVR, learning same-di diffeeifferent relations strains feedforward neural
networks. Interface Focus 8(4) (August 2018) 20180011
[26] Deng, J., Dong. W., Socher, R., Li, L.J. Li, K., Fei-Fei, L.: ImageNet: large-scale hierarchical image
database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. (June 2009) 248 255
[27] Kosiorek, A.R., Bewley, A., Posner, I.: Hierarchical attentive recurrent tracking. (June 2017)

---

[28] Simonyan. K.. Zisserman, A.: Two-Stream convolutional networks for action recognition in videos.
NeurlPS (2014)
[29]
Gladh, S., Danelljan, M., Khan, F.S., Felsberg, M.: Deep motion features for visual tracking. In: 2016 23rd International Conference on Pattern Recognition (ICPR), IEEE (December 2016)
[30] Teng, Z., Xing, J., Wang, Q. Lang, C., Feng, S., Jin, Y:: Robust object tracking based on temporal and spatial deep networks. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE (October 2017)
[31] Tschopp, F.D., Reiser, M.B., Turaga, S.C.: A connectome based hexagonal lattice convolutional network model of the drosophila visual system. (June 2018)
[32] Fortmann, T.E., Bar-Shalom, Y., Scheffe, M.: Multi-target tracking using joint probabilistic data association. In: 1980 19th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes. (December 1980) 807-812
[33] Milan, A., Taixe, L.. Reid, I., Roth, S., Schindler, K.: MOTI6: A benchmark for Multi Object tracking. (March 2016)
[34] Zhang, L., Li, Y., Nevatia, R.: Global data association for multi-object tracking using network flows, In: 2008 IEEE Conference on Computer Vision and Pattern Recognition. (June 2008) 1-8
[35] Dicle, C., Camps, O.I., Sznaier, M.: The way they move: Tracking multiple targets with similar appearance.
In: 2013 IEEE International Conference on Computer Vision. (December 2013) 2304-2311
[36] Li. S., Yeung. D.: Visual object tracking for unmanned aerial vehicles: models, AAAI (2017)
benchmark and new motion
[37] Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE
International Conference on Image Processing (ICIP), IEEE (September 2016)
[38] Luo, W., Xing. J., Milan, A., Zhang. X., Liu, W., Kim, T.K.: Multiple object tracking: A literature review. Artif. Intell. 293 (April 2021) 103448
[39] Malik, G., Linsley, D.. Serre, T., Mingolla, E.: The challenge of appearance- free object tracking with feedforward neural networks. CVPR Workshop on Dynamic Neural Networks Meet Computer Vision (2021)
[40] Eberhardt, S., Cader, J.G., Serre, T.: How deep is the feature analysis underlying rapid visual categorization? In Lee, D.D., Sugiyama, M.. Luxburg, U.V., Guyon, I., Garnett, R., eds.: Advances in Neural Information Processing Systems 29. Curran Associates, Inc. (2016) 1100-1100 108
[41] Bhat, G., Danelljan, M., Van Gool, L., Timofte, R.: Know your surroundings: Exploiting scene information for object tracking. In: Computer Vision ECCV 2020, Springer International Publishing (2020) 205-221
[42] Fan, H., Ling. H., Lin, L., Yang. F., Chu, P.. Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C.: LaSOT: A high- quality benchmark for large-scale single object tracking. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEFE (June 2019)
[43] Huang. L., Zhao, X., Huang, K.: GOT- A large High-Diversity benchmark for generic object tracking in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 43(5) (May 2021) 1562-1577
[44] Kingma, D.P., Ba, J.: Adam: method for stochastic optimization. (December 2014)
[45] Berzhanskaya, J.. Grossberg, S., Mingolla, E.: Laminar cortical dynamics of visual form and motion
interactions during coherent object motion perception. Spat. Vis. 20(4) (2007) 337-395
[46] Wong, K.F. Wang, X.J.: A recurrent network mechanism of time integration in perceptual decisions. J. Neurosci. 26(4) (January 2006) 1314-1328
[47] Takemura, Y., Bharioke, A., Lu, Z., Nem, A., Vitaladevuni, S., Rivlin, P.K., Katz, W.T., Olbris, D.J., Plaza, S.M., Winston, P., Zhao, T., Horne, J.A., Fetter, R.D., Takemura, S., Blazek, K., Chang, L.A., Ogundeyi, O., Saunders, M.A., Shapiro, V., Sigmund, C., Rubin, G.M., Scheffer, L.K., Meinertzhagen, I.A., Chklovskii, D.B.: A visual motion detection circuit suggested by drosophila connectomics. Nature 500(7461) (August 2013) 175-181
[48] Kim, J.S., Greene, M.J., Zlateski, A., Lee, K., Richardson. M., Turaga, S.C., Purcaro. M., Balkam, M.. Robinson, A., Behabadi, B.F., Campos, M., Denk, W., Seung, H.S., the EyeWirers: Space-time wiring specificity supports direction selectivity in the retina. Nature 509 (May 2014) 331
[49] Adelson, E.H., Bergen, J.R.: Spatiotemporal energy models for the perception of motion. Opt. Soe, Am. A. JOSAA 2(2) (February 1985) 284-299
[50] Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2) (April 1990) 179-211
[51] Hochreiter, s., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8) (November 1997)
1735-1780
12

---

[52] O'Reilly. R.C., Frank, M...: Making working memory work: computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 18(2) (February 2006) 283-328
[53] Badre, D.: Opening the gate to working memory. Proc. Natl. Acad. Sci. U. S. A. 109(49) (December 2012) 19878-19879
[54] D' Ardenne, K., Eshel, N., Luka, J., Lenartowicz, A., Nystrom, L.E., Cohen, J.D.; Role of prefrontal cortex and the midhrain dopamine system in working memory updating. Proc. Natl. Acad. Sci. U. S. A. 109(49) (December 2012) 19900-19909
[55] Mitchell, J.F., Sundberg. K.A., Reynolds, J.H.: Differential attention dreetion-deeendentt response modulation across cell classes in macaque visual area V4, Neuron 55(1) (July 2007) 131-141
[56] Fazl, A., Grossberg, S., Mingolla, E.: View-invariant object calegory learning, recognition, and search: how spatial and object attention are coordinated using surface-based attentional shrouds. Cognitive psychology 58(1) (2009) 1 48
[57] Berzhanskaya, J., Grossherg, S., Mingolla, E.: Laminar cortical dynamics of visual form and motion interactions during coherent object motion perception. Spat. Vis. 20(4) (2007) 337-395
[58] Mély, D.A., Linsley, Serre, T:: Complementary surrounds explain diverse contextual phenomena across visual modalitics. Psychol. Rev. 125(5) (October 2018) 769-784
[59] Geirhos, R., Meding, K., Wichmann, F. Beyond accuracy: quantifying rril-by-triic behaviour of CNNs and humans by measuring error consistency. NeurIPS (2020)
[60] Chen, X., Yan, B., Zhu, J., Wang, D., Yang. X., Lu. H.: Transformer tracking. (March 2021)
[61] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A.N., Kaiser, E.U., Polosukhin, Attention is all you need. In Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garett, R., eds.: Advances in Neural Information Processing Systems. Volume 30., Curran Associates, Inc. (2017) [62] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. (December 2015)
[63] Gilbert, C.D., Li, W.: Top-down influences on visual processing. Nat. Rev. Neurosci. 14(5) (May 2013) 350-363
[64]
Loshchilov, I. Hutter, F: Decoupled weight decay regularization. (November 2017)
[65] Steinman, R.M., Pizlo, Z., Pizlo, FJJ.; Phi is not beta, and why wertheimer's discovery launched the gestalt revolution. Vision Res, 40(17) (August 2000) 2257-2264
[66] Huk, A.C., Shadlen, M.N.; Neural activity in macaque parietal cortex reflects temporal integration of visual
motion signals during perceptual decision making. J. Neurosci. 25(45) (November 2005) 10420-1044 10436
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See SI $1.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See SI $1.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] An example experiment and collected data is hosted at alhttp//bbtt..yy/Iiiiirrttt
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $ and (
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)? [No] We took max performance over random seeds for our Pathracker challenge.
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See $ and $
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See and $
(b) Did you mention the license of the assets? [Yes] See SI $6 for licenses of the tracking datasets we relied on this paper.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We included gifs, links code, and an example experiment at the anonymized website http://bit. 1y MyIInncircuutt
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [Yes] See }
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See 3
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See $2.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] See SI $2.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] See $ and SI $2.
14


Reviews:
[{"review_id": "cHZoEn8g-cn", "reviewer": "Reviewer_utTc", "summary": "The authors note that state-of-the-art computer vision methods for object tracking in natural videos largely rely on _appearanced-based tracking_ -- that is, some kind of matching between the features of a target object and features in subsequent frames of a movie. In contrast, people (and likely non-human animals) need to and are able to track objects in cases where appearance is unhelpful; they probably do this by somehow integrating the motion of an attended object over time. \n\nMotivated by this discrepancy, the authors devise a tracking task in which object appearance is useless and find that people nevertheless perform well. They test a variety of neural network architectures (CNNs, RNNs, Transformers), none of which performs at human level on harder versions of the task. On the other hand, a custom-designed RNN with gating, inhibitory interactions, and \"attention\" both performs at human level and even makes similar errors to people. Finally, the authors show that they can improve on the state-of-the-art for several large-scale object tracking challenges by introducing their custom RNN as additional modules in the current best neural network. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like this paper and this approach, and I hope that comparing models to actual human behavior on challenging tasks becomes more common at venues like NeurIPS. Even if the authors had not included the part at the end on \"object tracking in the wild,\" the fact that standard ANN architectures don't reach human performance or follow the same strategies for object tracking on a simple task is interesting and worth communicating: it means that those standard architectures are not good models of all human visual abilities (though I have some questions about this experiment below.) I take the authors' main claims to be\n\n(1) People can do a tracking task where object appearance cues are useless\n\n(2) This task, in its more challenging variants, strains a number of standard ANN architectures, whether pretrained or not;\n\n(3) A custom RNN architecture (InT) can perform the tracking task as well as people and adopts a similar algorithmic strategy to do so;\n\n(4) Adding this architectural module to state-of-the-art object trackers improves them, suggesting that the computations learned by InT modules are useful \"in the wild,\" not just for the task where object appearance is not informative.\n\nOf these claims, I think the authors provide strong evidence for (1), pretty good evidence for (2) (but I have some concerns), very strong evidence for (3), and weak evidence for (4). So I'll focus my comments on (2) and (4) below.\n\n_Claim 2: Are standard ANN architectures really unable to do the tracking task?_  It looks like many of the non-InT models do as well as humans -- or could do so with slight modification (more channels, more training, etc.) -- on the 32-frame version of the task (Figure 3a.) However, they're nowhere close on the 64-frame version. It's not intuitively obvious why this would be the case: the authors say the task difficulty should increase linearly with the number of frames (and they show that the number of distractor-target crossings increases only a little faster than linearly in the Supplement). This fact makes me a little worried that there's some critical hyperparameter that, if sampled thoroughly, might allow the standard ANNs to do a lot better on the 64-frame version. I have no way of knowing this for sure, but it seems worth trying to understand **why** the best standard ANNs fail in the longer version of the task. Are they learning a completely different algorithmic strategy from what people use, which just breaks down for longer movies? Is their memory capacity limiting? For some of the model architectures it may be hard to figure out exactly what's going on in natural language terms, but for others (like the Transformer architectures with their visualizable attention maps) maybe the authors could get some insight.\n\nRelated to this: even on the variants of the task where the standard ANNs succeed, how do they compare to humans in response/error pattern (i.e. using the metrics in Figure 5?) It would at least be suggestive if it turned out that those models were reaching human performance without making human-like errors, since that could mean that they're biased toward learning a strategy that just doesn't scale to longer movies. \n\n_Claim 3: What makes InT so good?_ The authors' results with InT are very compelling, showing that this network reaches human performance and makes human-like responses. Their ablation study (Figure 5) also suggests that most if not all of the custom circuit components in InT are needed to get these results. That's most important, but I'm frankly a little surprised that the authors were able to engineer a recurrent circuit to do this task \"from principles\" (described in the Supplement) -- designing such highly customized circuits hasn't shown widespread success on most large-scale computer vision tasks. This doesn't mean the authors are wrong, but it does suggest to me that there's some key algorithmic strategy that the InT circuit is well-biased to learn that the other architectures they test aren't. Is this because the other architectures are too large? Missing some components (e.g. local recurrence or sigmoid activations for local \"attention\" or spatial information in the inputs?) Require more training? \n\nIt might be useful to try some \"gain of function\" experiments in which individual InT components are _added_ to the standard ANN architectures to see if this helps on the task. I especially wonder about the Conv-GRU, since it was the only other recurrent circuit tested: is it just incapable of (or biased against) learning the same algorithm as InT? If so, what needs to change?\n\n_Claim 4: Is the InT circuit useful for tracking on large-scale computer vision benchmarks?_ The TransT + InT model does achieve state-of-the-art on 2/3 tracking benchmarks (albeit by small amounts) and seems more robust to the perturbations in the GOT task. \nBut these improvements are small enough that I wonder whether they're really due to the authors' custom architecture or to the changes they made to train it. Those changes are a little hard to follow in the text. For instance, it seems like InT + TransT gets additional frames at training time, maybe additional supervision signals (on the output of the InT module), and maybe additional inputs (the spatial location of the target object in the initial frame) compared to TransT. I also wasn't clear on whether it had approximately the same number of parameters (this isn't the most critical thing IMO, but it would be nice if TransT had a \"control augmentation\" that could be trained instead of the InT modules.) It would be very helpful if the authors could clarify all this in the main text and explain in more detail why they're making a fair comparison.\n\nThe three examples shown in Figure 6 are in good concordance with the authors' claims about the advantages of InT over purely recognition-based tracking (i.e., a number of objects with similar appearance that change positions a lot over the course of the movie and have occlusions.) However, since overall performance is not much better, there must also be some cases where TransT + InT performs worse (or else the ones where it does better are extremely rare.) It would be interesting to see some examples of these cases, or if possible do an analysis just on videos that have full occlusion/multiple objects of the same category. I'm not sure if annotations for these exist in the benchmark datasets, so if not the authors could think about creating small testing datasets to isolate these factors explicitly (not for a revision here, but in the medium-term.)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any specific mention of limitations or societal impact, but I'm not sure the manuscript/topic really calls for it. Short-term object tracking has been studied in people for decades; I don't think society would change in some predictably negative way if we understood human-level object tracking better (unlike, e.g., developing algorithms with super-human ability to recognize faces.) \n\nI think if the authors could address my concerns above, it would adequately cover the \"limitations\" of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No, I don't have ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Uf3lI_5h4TF", "reviewer": "Reviewer_JexJ", "summary": "The paper presents a synthetically designed challenge called Pathtracker, which is to predict if a given green square reaches the desired destination. The idea is to abstract out the appearance and focus on the ability of the video analysis method to track a particular target among other similarly-looking distractors. The challenge is posed as a classification task (which requires tracking abilities beneath to solve it).  \n\n\nThe paper proposes a circuit model in a recurrent architecture to solve the task and demonstrates superior performance to some of the existing SOTA methods for video analysis. Adequate human comparisons are made throughout the paper, in terms of the employed strategy as well as the final performance.  The experiment shows that combining the proposed circuit model with TransT leads to marginal improvements on large-scale object tracking benchmarks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing is clear and the paper is easy to read/comprehend. The Pathtracker challenge design and the following experiments appear novel to me. I liked the human comparisons and the strategy discussion with supporting experiments. The experiments look correct and reproducible. \n\nTo me, the experiments on the larger datasets like LASOT, GOT, etc. are not convincing enough. When the complexity increases, it becomes more and more difficult to judge the reason for the performance increase. Yes on the face, it appears the circuit model might be aiding performance, but possibly there are simpler improvements, which could lead to similar performance gains. I would have preferred the paper to focus on experiments in a different domain, something like cell tracking (medical), where the ideas would have been much more useful. \n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- There are differences in the literature, on how they approach single target vs multi-target tracking. Most multi-target tracking papers use a combination of appearance, motion (Kalman filters), spatial configuration/location, etc. Some of the recent papers suggest, that appearance can be learned unsupervised and may not be the strongest contributing factor in multi-target tracking (Karthik et al. Simple Unsupervised Multi-Object Tracking. https://arxiv.org/abs/2006.02609). A discussion in the paper would be useful. In fact, for me that makes the results in the paper not fully aligned with the motivation.\n\n- A baseline experiment of applying simple Kalman filter-based tracking would be interesting to see. That might actually do well and I  would be keen to look at the results. Tackling tracking as tracking, rather than classification. By saying this, I am not downplaying the contributions. I just think the paper would look more complete with such an experiment. \n\n- Another interesting direction would be to evaluate the method on cell tracking datasets (medical imaging). Such experiments would be better aligned with the primary motivation of the work. \n\n- An experiment on a multi-target tracking dataset, just using Int would be useful (might be difficult to configure in rebuttal period, it is more of a future suggestion)\n\n- Some writing suggestions and typos:\nFigure1 is not clear. Is it tracking the jellyfish or the object below it. \nLine 50 - Pathfinder are also exhibit\nVenues are missing in many references (e.g. [7] is Neurips 2018)\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "8ZgS6awnyD-", "reviewer": "Reviewer_hxQK", "summary": "This paper presents a new tracking challenge specially for testing the motion-based target tracking capability. \"Objects\" in the synthetic data are with the same appearance so as to exclude the appearance information. Human study and baseline methods are proposed to show how challenging this task is.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to emphasize the importance of the motion information of tracking targets and construct a corresponding synthetic dataset. However, the crucial rule of motion has already been noticed and explored in the conventional object tracking tasks, such as [1,2,3], although appearance information is also an important clue in those tasks.\n\nActually, even two objects are visually very similar from we human views (eg., cells), sometimes deep neural networks can still distinguish them. I think the designed task goes a bit to extremes. To avoid this, using real-world tasks would be better, such as tracking cells or fish.\n\nOverall, I do not think this task has much value.\n\n[1] Deep Motion Features for Visual Tracking.\n[2] Visual Object Tracking for Unmanned Aerial Vehicles: A Benchmark and New Motion Models\n[3] Robust Object Tracking based on Temporal and Spatial Deep Networks", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the detailed review above.\n\n------------------Post Rebuttal-------------------------\n\nThe response does not address my concerns. The experiments on Page 8-9 are conducted on existing conventional visual object tracking datasets, where the target object usually has a different appearance from backgrounds/distractors. However, this paper aims to address the tracking problem when appearance has little if not no difference. A convinced evaluation in real-world scenarios should be to track an instance with distractors of the same category, such as a fish/sheep swimming/running surrounded by similar-looking fish/sheep as distractors. The current experimental results on real-world tracking datasets (Table 1) show only marginal improvements (< 1% on two out of three datasets, and -0.1% on the remaining one), and please note that the improvements are achieved under the condition that the tracking target has large appearance difference from distractors. \n\nOverall, the proposed InT only shows effectiveness on toy synthetic data (the dataset scale is far smaller than existing real-world tracking datasets), and there are too many factors that may affect the results on synthetic data, such as the motion pattern of the distractors, the variation in the number of distractors as the video goes. And crutial experiments are missing (as mentioned above but not limitted to). Thus, I render a rejection for this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Having read the paper carefully and looked at the reviews, I am pretty convinced this is a useful contribution and should definitely be accepted.  Two of the reviewers gave the paper high scores -- but not a free pass, having asked detailed questions and gotten apparently satisfying responses from the authors. I see that one of the reviewers is very skeptical, and remains unsatisfied by the authors' responses.  But looking at the author responses, and having tried to probe that reviewer's thoughts myself, it seems to me that the reviewer's skepticism is rather biased (e.g. not responding to logical questions, but just kind of baked in).  So I've decided to largely discount that low reviewer's comments in making my final decision. 
Decision: Accept (Poster)