Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This paper presents a new tracking challenge specially for testing the motion-based target tracking capability. "Objects" in the synthetic data are with the same appearance so as to exclude the appearance information. Human study and baseline methods are proposed to show how challenging this task is.
main_review: This paper aims to emphasize the importance of the motion information of tracking targets and construct a corresponding synthetic dataset. However, the crucial rule of motion has already been noticed and explored in the conventional object tracking tasks, such as [1,2,3], although appearance information is also an important clue in those tasks.

Actually, even two objects are visually very similar from we human views (eg., cells), sometimes deep neural networks can still distinguish them. I think the designed task goes a bit to extremes. To avoid this, using real-world tasks would be better, such as tracking cells or fish.

Overall, I do not think this task has much value.

[1] Deep Motion Features for Visual Tracking.
[2] Visual Object Tracking for Unmanned Aerial Vehicles: A Benchmark and New Motion Models
[3] Robust Object Tracking based on Temporal and Spatial Deep Networks
limitations_and_societal_impact: Please refer to the detailed review above.

------------------Post Rebuttal-------------------------

The response does not address my concerns. The experiments on Page 8-9 are conducted on existing conventional visual object tracking datasets, where the target object usually has a different appearance from backgrounds/distractors. However, this paper aims to address the tracking problem when appearance has little if not no difference. A convinced evaluation in real-world scenarios should be to track an instance with distractors of the same category, such as a fish/sheep swimming/running surrounded by similar-looking fish/sheep as distractors. The current experimental results on real-world tracking datasets (Table 1) show only marginal improvements (< 1% on two out of three datasets, and -0.1% on the remaining one), and please note that the improvements are achieved under the condition that the tracking target has large appearance difference from distractors. 

Overall, the proposed InT only shows effectiveness on toy synthetic data (the dataset scale is far smaller than existing real-world tracking datasets), and there are too many factors that may affect the results on synthetic data, such as the motion pattern of the distractors, the variation in the number of distractors as the video goes. And crutial experiments are missing (as mentioned above but not limitted to). Thus, I render a rejection for this paper.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors note that state-of-the-art computer vision methods for object tracking in natural videos largely rely on _appearanced-based tracking_ -- that is, some kind of matching between the features of a target object and features in subsequent frames of a movie. In contrast, people (and likely non-human animals) need to and are able to track objects in cases where appearance is unhelpful; they probably do this by somehow integrating the motion of an attended object over time. 

Motivated by this discrepancy, the authors devise a tracking task in which object appearance is useless and find that people nevertheless perform well. They test a variety of neural network architectures (CNNs, RNNs, Transformers), none of which performs at human level on harder versions of the task. On the other hand, a custom-designed RNN with gating, inhibitory interactions, and "attention" both performs at human level and even makes similar errors to people. Finally, the authors show that they can improve on the state-of-the-art for several large-scale object tracking challenges by introducing their custom RNN as additional modules in the current best neural network. 
main_review: I like this paper and this approach, and I hope that comparing models to actual human behavior on challenging tasks becomes more common at venues like NeurIPS. Even if the authors had not included the part at the end on "object tracking in the wild," the fact that standard ANN architectures don't reach human performance or follow the same strategies for object tracking on a simple task is interesting and worth communicating: it means that those standard architectures are not good models of all human visual abilities (though I have some questions about this experiment below.) I take the authors' main claims to be

(1) People can do a tracking task where object appearance cues are useless

(2) This task, in its more challenging variants, strains a number of standard ANN architectures, whether pretrained or not;

(3) A custom RNN architecture (InT) can perform the tracking task as well as people and adopts a similar algorithmic strategy to do so;

(4) Adding this architectural module to state-of-the-art object trackers improves them, suggesting that the computations learned by InT modules are useful "in the wild," not just for the task where object appearance is not informative.

Of these claims, I think the authors provide strong evidence for (1), pretty good evidence for (2) (but I have some concerns), very strong evidence for (3), and weak evidence for (4). So I'll focus my comments on (2) and (4) below.

_Claim 2: Are standard ANN architectures really unable to do the tracking task?_  It looks like many of the non-InT models do as well as humans -- or could do so with slight modification (more channels, more training, etc.) -- on the 32-frame version of the task (Figure 3a.) However, they're nowhere close on the 64-frame version. It's not intuitively obvious why this would be the case: the authors say the task difficulty should increase linearly with the number of frames (and they show that the number of distractor-target crossings increases only a little faster than linearly in the Supplement). This fact makes me a little worried that there's some critical hyperparameter that, if sampled thoroughly, might allow the standard ANNs to do a lot better on the 64-frame version. I have no way of knowing this for sure, but it seems worth trying to understand **why** the best standard ANNs fail in the longer version of the task. Are they learning a completely different algorithmic strategy from what people use, which just breaks down for longer movies? Is their memory capacity limiting? For some of the model architectures it may be hard to figure out exactly what's going on in natural language terms, but for others (like the Transformer architectures with their visualizable attention maps) maybe the authors could get some insight.

Related to this: even on the variants of the task where the standard ANNs succeed, how do they compare to humans in response/error pattern (i.e. using the metrics in Figure 5?) It would at least be suggestive if it turned out that those models were reaching human performance without making human-like errors, since that could mean that they're biased toward learning a strategy that just doesn't scale to longer movies. 

_Claim 3: What makes InT so good?_ The authors' results with InT are very compelling, showing that this network reaches human performance and makes human-like responses. Their ablation study (Figure 5) also suggests that most if not all of the custom circuit components in InT are needed to get these results. That's most important, but I'm frankly a little surprised that the authors were able to engineer a recurrent circuit to do this task "from principles" (described in the Supplement) -- designing such highly customized circuits hasn't shown widespread success on most large-scale computer vision tasks. This doesn't mean the authors are wrong, but it does suggest to me that there's some key algorithmic strategy that the InT circuit is well-biased to learn that the other architectures they test aren't. Is this because the other architectures are too large? Missing some components (e.g. local recurrence or sigmoid activations for local "attention" or spatial information in the inputs?) Require more training? 

It might be useful to try some "gain of function" experiments in which individual InT components are _added_ to the standard ANN architectures to see if this helps on the task. I especially wonder about the Conv-GRU, since it was the only other recurrent circuit tested: is it just incapable of (or biased against) learning the same algorithm as InT? If so, what needs to change?

_Claim 4: Is the InT circuit useful for tracking on large-scale computer vision benchmarks?_ The TransT + InT model does achieve state-of-the-art on 2/3 tracking benchmarks (albeit by small amounts) and seems more robust to the perturbations in the GOT task. 
But these improvements are small enough that I wonder whether they're really due to the authors' custom architecture or to the changes they made to train it. Those changes are a little hard to follow in the text. For instance, it seems like InT + TransT gets additional frames at training time, maybe additional supervision signals (on the output of the InT module), and maybe additional inputs (the spatial location of the target object in the initial frame) compared to TransT. I also wasn't clear on whether it had approximately the same number of parameters (this isn't the most critical thing IMO, but it would be nice if TransT had a "control augmentation" that could be trained instead of the InT modules.) It would be very helpful if the authors could clarify all this in the main text and explain in more detail why they're making a fair comparison.

The three examples shown in Figure 6 are in good concordance with the authors' claims about the advantages of InT over purely recognition-based tracking (i.e., a number of objects with similar appearance that change positions a lot over the course of the movie and have occlusions.) However, since overall performance is not much better, there must also be some cases where TransT + InT performs worse (or else the ones where it does better are extremely rare.) It would be interesting to see some examples of these cases, or if possible do an analysis just on videos that have full occlusion/multiple objects of the same category. I'm not sure if annotations for these exist in the benchmark datasets, so if not the authors could think about creating small testing datasets to isolate these factors explicitly (not for a revision here, but in the medium-term.)

limitations_and_societal_impact: I don't see any specific mention of limitations or societal impact, but I'm not sure the manuscript/topic really calls for it. Short-term object tracking has been studied in people for decades; I don't think society would change in some predictably negative way if we understood human-level object tracking better (unlike, e.g., developing algorithms with super-human ability to recognize faces.) 

I think if the authors could address my concerns above, it would adequately cover the "limitations" of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a synthetically designed challenge called Pathtracker, which is to predict if a given green square reaches the desired destination. The idea is to abstract out the appearance and focus on the ability of the video analysis method to track a particular target among other similarly-looking distractors. The challenge is posed as a classification task (which requires tracking abilities beneath to solve it).  


The paper proposes a circuit model in a recurrent architecture to solve the task and demonstrates superior performance to some of the existing SOTA methods for video analysis. Adequate human comparisons are made throughout the paper, in terms of the employed strategy as well as the final performance.  The experiment shows that combining the proposed circuit model with TransT leads to marginal improvements on large-scale object tracking benchmarks. 
main_review: The writing is clear and the paper is easy to read/comprehend. The Pathtracker challenge design and the following experiments appear novel to me. I liked the human comparisons and the strategy discussion with supporting experiments. The experiments look correct and reproducible. 

To me, the experiments on the larger datasets like LASOT, GOT, etc. are not convincing enough. When the complexity increases, it becomes more and more difficult to judge the reason for the performance increase. Yes on the face, it appears the circuit model might be aiding performance, but possibly there are simpler improvements, which could lead to similar performance gains. I would have preferred the paper to focus on experiments in a different domain, something like cell tracking (medical), where the ideas would have been much more useful. 





limitations_and_societal_impact: - There are differences in the literature, on how they approach single target vs multi-target tracking. Most multi-target tracking papers use a combination of appearance, motion (Kalman filters), spatial configuration/location, etc. Some of the recent papers suggest, that appearance can be learned unsupervised and may not be the strongest contributing factor in multi-target tracking (Karthik et al. Simple Unsupervised Multi-Object Tracking. https://arxiv.org/abs/2006.02609). A discussion in the paper would be useful. In fact, for me that makes the results in the paper not fully aligned with the motivation.

- A baseline experiment of applying simple Kalman filter-based tracking would be interesting to see. That might actually do well and I  would be keen to look at the results. Tackling tracking as tracking, rather than classification. By saying this, I am not downplaying the contributions. I just think the paper would look more complete with such an experiment. 

- Another interesting direction would be to evaluate the method on cell tracking datasets (medical imaging). Such experiments would be better aligned with the primary motivation of the work. 

- An experiment on a multi-target tracking dataset, just using Int would be useful (might be difficult to configure in rebuttal period, it is more of a future suggestion)

- Some writing suggestions and typos:
Figure1 is not clear. Is it tracking the jellyfish or the object below it. 
Line 50 - Pathfinder are also exhibit
Venues are missing in many references (e.g. [7] is Neurips 2018)
review_title: null

Meta-review: Having read the paper carefully and looked at the reviews, I am pretty convinced this is a useful contribution and should definitely be accepted.  Two of the reviewers gave the paper high scores -- but not a free pass, having asked detailed questions and gotten apparently satisfying responses from the authors. I see that one of the reviewers is very skeptical, and remains unsatisfied by the authors' responses.  But looking at the author responses, and having tried to probe that reviewer's thoughts myself, it seems to me that the reviewer's skepticism is rather biased (e.g. not responding to logical questions, but just kind of baked in).  So I've decided to largely discount that low reviewer's comments in making my final decision. 
Decision: Accept (Poster)