Beyond Accuracy: Tracking more like Human via Visual Search
Authors: Dailing Zhang
Shiyu Hu
Xiaokun Feng
Xuchen Li
Meiqi Wu
Jing Zhang
Kaiqi Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: visual object tracking
central-peripheral dichotomy
human behaviour
Primary area: machine_vision
TL;DR: Inspired by central-peripheral dichotomy, we developed a tracker that emulates human visual search abilities, validated by its high accuracy and error consistency.

Abstract: Human visual search ability enables efficient and accurate tracking of an arbitrary moving target, which is a significant research interest in cognitive neuroscience. The recently proposed Central-Peripheral Dichotomy (CPD) theory sheds light on how humans effectively process visual information and track moving targets in complex environments. However, existing visual object tracking algorithms still fall short of matching human performance in maintaining tracking over time, particularly in complex scenarios requiring robust visual search skills. These scenarios often involve Spatio-Temporal Discontinuities (i.e., STDChallenge), prevalent in long-term tracking and global instance tracking. To address this issue, we conduct research from a human-like modeling perspective: (1) Inspired by the CPD, we pro- pose a new tracker named CPDTrack to achieve human-like visual search ability. The central vision of CPDTrack leverages the spatio-temporal continuity of videos to introduce priors and enhance localization precision, while the peripheral vision improves global awareness and detects object movements. (2) To further evaluate and analyze STDChallenge, we create the STDChallenge Benchmark. Besides, by incorporating human subjects, we establish a human baseline, creating a high- quality environment specifically designed to assess trackers’ visual search abilities in videos across STDChallenge. (3) Our extensive experiments demonstrate that the proposed CPDTrack not only achieves state-of-the-art (SOTA) performance in this challenge but also narrows the behavioral differences with humans. Additionally, CPDTrack exhibits strong generalizability across various challenging benchmarks. In summary, our research underscores the importance of human-like modeling and offers strategic insights for advancing intelligent visual target tracking. Code and models are available at https://github.com/ZhangDailing8/CPDTrack.

Beyond accuracy: Tracking more like Human via Visual Search
Dailing Zhang Shiyu Hu Xiaokun Feng!.2
Xuchen Lin.2 Meiqi Wu Jing Zhang Kaiqi Huang
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences
Institute of Automation, Chinese Academy of Sciences
SSchool of Computer Science and Technology, University of Chinese Academy of Sciences
4 *Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences
School of Physical and Mathematical Sciences, Nanyang Technological University zhangdailing20230 @ cn. shiyu.hu wachhuunnuueeee wtw..nnuuuuuuu... fengxiaokun2022@@h aari222@iaaaa.... @a..cc.cn wumeiqi 18 @ mails. ucas.ac.en, (lixuchen2224, zhang, kqhuang) @ia.ac.cn
Abstract
Human visual search ability enables efficient and accurate tracking of arbitrary moving target, which is significant research interest in cognitive neuroscience. The recently proposed Peripal-eripphraal Dichotomy (CPD) theory sheds light on how humans effectively process visual information and track moving targets in complex environments. However, existing visual object tracking algorithms still fall short of matching human performance in maintaining tracking over time, par- ticularly in complex scenarios requiring robust visual search skills. These scenarios often involve Pentiol Discontinuities (i.e., STDChallenge), prevalent in long-term tracking and global instance tracking. To address this issue, we conduct research from human-like modeling perspective: (I) Inspired by the CPD, pro- pose new tracker named CPDTrack to achieve human-like visual search ability. The central vision of CPDTrack leverages the spatio-temporal continuity of videos to introduce priors and enhance localization precision, while the peripheral vision improves global awareness and detects object movements. (2) To further evaluate and analyze STDChallenge, we create the STDChallenge Benchmark. Besides, by incorporating human subjects, we establish human baseline, creating a high- quality environment specifically designed to assess trackers' visual search abilities in videos across STDCChallenge. (3) Our extensive experiments demonstrate that the proposed CPDTrack not only achieves state-of-the-an (SOTA) performance in this challenge but also narrows the behavioral differences with humans. Additionally, CPDTrack exhibits strong generalizability across various challenging benchmarks. In summary, our research underscores the importance of human-like modeling and offers strategic insights for advancing intelligent visual target tracking, Code and models are available at hhttttt///// hhttps://github //ggtttttb..//////////// camub.ccom/ae wwt.cooo/meaiiee /tt////aaaaaanng//aaaaaaaa CCtpg//
Introduction
In the real world, humans excel at locating an arbitrary moving target within complex backgrounds and can resume tracking it even after temporary loss. This enduring question in cognitive neuroscience has recently been explained to some extent by the Central-Periphers
Dichotomy (CPD) theory, which suggests that the human eyes processes all visual inputs by sorting them into central and peripheral visions [ 22 Central vision, decoded and understood by higher brain areas, focuses on interpreting details and minimizing distractions; while peripheral vision, processed in the primary visual cortex (V1), swiftly detects dynamic changes.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Visual representations of STDChallenge
Failures except CPDThack
Recovery
STDChelleng
Failures except CPDTrack
Fedlures CPD
shotcat
shaltt
absent
Grosdttrdh
Lititbckk (oun)
Presen
Absent
MaNEE GHANERB
2000
400 5000
1062
2000
2000
Shotout
(b) Status of the target within the sequeence "095"
Figure 1: Illustration of the STDChallenge, depicting the absent of targets and shotcut. STDChallenge quite challenging, but CPDTrack maintain robust tracking performance, demonstrating stronger visual search abilities compared to other trackers. (a) In the first column, most trackers fail; in the second column, they can recover; in the third column, STDChallenge occurs; in the fourth column, all trackers except for CPDTrack fail; and in the fifth column, this remains the case, meaning trackers' limited recovery. (b) shows the status of the target within the sequence of "095" from VideoCube (4] and red dots means shotcut.
Meanwhile. in computer vision research, advancements in single object tracking (SOT) are increas- ingly closing the gap between human dynamic visual ability (DVA) and proxy tasks B
In 2013, 7 g However, the
the short-term tracking (STT) task and related benchmarks were proposed 6 implicit assumption of continuous motion simplifies this task to continuously locating the target in short videos of tens of seconds, which is far from human DVA. Drawing on human visual search skills, the VOT community is broadening its scope by incorporating long-term tracking (LTT) p 10, which includes target absent (i.e., the moving target can disappear and reappear in tracking), and global instance tracking (GIT) [4) which involves shotcut (i.e., the video sequence may include different viewpoints and scenes). We refer to these challenges collectively as spatio-temporal discontinuity STDChallenge), which demands enhanced visual search abilities from trackers, as shown Fig. [
Unfortunately, mainstream machines have not kept pace with the aforementioned expansion of task definitions achieve human performance. Most mainstream trackers [2 I3 14
15
[6).
influenced by the definition of STT, depend on the spatio-temporal continuity of the target. These trackers typically perform local cropping based on the previous frame's 's results within the motion model, module for calculating the search region on the current frame, as shown in Fig. (b-1). Obviously, this modeling mechanism only simulates human central vision, making it far from replicating human visual search ability, especially in scenarios where the STDChallenge increases significantly, as shown Blg.Baa Several studies [7 [8) have also demonstrated that STDChullenge presents a significant challenge for trackers. Thus, natural question is: what causes trackers to overlook the importance of the STDChallenge in their modeling process?
Some possible reasons from the benchmark perspective may answer the above question: Limitation of datasets. Although the definition of proxy tasks has been moving closer to human DVA (STT LTT + GIT), most datasets are still influenced by the initial task characteristics (i.e., single-targer, model- free, causal -rackkes, single-com canee, and short-term) data collection process [9 20, 21 99 as in Fig. 10 resulting in the importance of the STDChallenge being overlooked. Consequently, trackers can easily achieve good performance on most datasets without possessing sufficient visual search ability to handle the STDChullenge. Limitation of evaluation. Researchers contend that integrating insights from human DVA could address STDChallenge and enhance the real-world robustness of machines
22 3 23 However, the substantial differences in methodologies and experimental

---

Staame
strame
Tempoeal
SUDKISS
(b-1) Local
(b-3) local-global
Motian Medil
Tamplate
Extraction
Feature
gratss
Modide
Hled
Htramte
Fa
Atrame t+1
strame
HTrame I+1
Interfomnct
SUCDOSS
success
(b-2) global
(b-D)CPD
(a) Pipeline of popalar trackers
(b) Different motion mode) trackour pipeline
Figure 2: Comparison of tracking pipeline. Our CPDTrack differs from previous trackers in motion
model. (a) The components of mainstream tracking frameworks consist of motion model, feature extraction, and temporal module. (b-1) 1) local tracker, which tracks targets in local areas and has difficulty recovering after failure: (b-2) global tracker, which tracks targets globally in the current frame, is susceptible to background interference and has lower efficiency; (b-3) local-ge local-global tracker, which can switch between the above two, depending on the performance of the local tracker; (b-4) our CPDTrack, which can model both central and peripheral information simultaneously.
approaches between computer science and cognitive science make it difficult to determine if current technological advancements are truly bridging the gap between SOT machines and human DVA. Consequently, there is a notable absence of mechanisms to assess the true visual intelligence of trackers.
In summary, existing trackers lack consideration of human visual search ability in their modeling mechanisms, making it difficult to cope with the STDChallenge. This limitation is not only due to the algorithm design itself but also involves the limitations of datasets and evaluation. Therefore, this work aims to address these aspects (algorithms and benchmark) and verifies the relevant results through experimental analysis.
new algorithm CPDTrack (Section Our analysis of the tracker pipeline and understanding of the CPD suggests that mainstream motion models, which crop local regions from the current frame, as shown in Fig. 2 restrict the tracker's ability to locate moving targets. Drawing on the distinction between central and peripheral vision in CPD, we propose
tracker named CPDTrack divide the current frame accordingly. Central vision captures detailed information, leveraging the video's s apatio-temporal continuity to introduce priors, while peripheral vision manages global information to enhance overall scene understanding, as shown in Fig. 3 CPDTrack also models the information query, achieving top-down control based on cognitive outcomes.
new STDChallenge Benchmark with Visual Turing Test (Section 4). address the shortcomings of existing benchmarks environments, we introduce specialized challenge environment named STDChallenge Benchmark to assess the visual search abilitics of both machines and humans in tracking an arbitrary moving target. This environment comprises sequences that represent the STDChallenge, carefully selected from both LTT and GIT benchmarks. By sampling from a variety of benchmarks, ensure reduction in dataset-specifie biases. Simultaneously,
employ the Visual Turing Test (VTT) [24] 25] on the STDChallenge Benchmark to assess the disparity in intelligence between trackers and human DVA. Specifically, we examine error consistency [26] to compare the performance of humans and machines, and explore behavioral differences among machines with varying architectures and parameters in dynamic visual tasks. Building on these results, we conduct a thorough and detailed analysis to reveal the impact of the STDChallenge on trackers.
Comprehensive and integral experimental analyses (Section 5 Finally, analysis results from extensive experimental settings show that, in the STDChallenge, CPDTrack not only achieved state- of-the-art (SOTA) performance, getting 1.7% superior N-PRE and 1.4% PRE score on STDChallenge Benchmark compared to the second algorithm, as shown in Table but also demonstrated remarkable aligmment with human DVA, as shown in Fig. 4 Furthermore, it performed well across various tracking benchmarks, particularly excelling in difficult benchmarks [4 27 28
Contributions. Our research explores the visual search ability of humans and trackers through detailed analyses based on algorithmic strategies and benchmark evaluations.
3

---

2 Related Work
Tracking Benchmark. In 2013, the VOT competition characterized SOT using five key terms: single-target, model-free, cawsal-trackers, single-camera and short-term. The initial three terms serve to differentiate SOT from other visual tasks [29] 30 31 321. whereas the last two, single-camera
and short-term, were introduced to simplify early-Stage research 6 7 19 21
Since 2018, several with the VOT
researchers have moved beyond the short-term limitation to embrace LTT setting new criterion that tasks permitting complete target disappearance quality as long-term [IO] More recently, the introduction of GIT [4] has eliminated the single-camera restriction, thus offering a more realistic simulation of the world. STDChallenge Benchmark is dedicated to examining the distinctions between LTT and GIT compared to STT, and integrates human DVA to steer algorithmic
advances.
Visual Trackers. Deep learning trackers are typically categorized by motion models into local and global trackers, as shown in Fig, [ Local trackers, which are most popular method, crop search region in current frame M 12
16 36). These trackers continuously improved by advancements in feature extractor and temporal analysis, have reached SOTA performance on various benchmarks
9 19 21) To tackle STDChallenge, it is common to switch to global detector when local trackers fail, though this transition depends on the local tracker's 's performance 40 4T.. Local trackers often struggle with proactive global awareness, reducing their robustness against STDChallenge. Global trackers, conversely, aim to identify targets throughout the entire image, raising computational demands and easy to be confused by similar objects in the background. Due to these limitations, they are less commonly employed [42 45 CPDTrack is designed to model CPD by combining local and global information, thereby enabling precise and efficient global modeling and offering insights for addressing the STDChallenge. For detailed analysis of these trackers, see Appendix D
37 38 39
Visual Turing Test. Traditional evaluation methods, focused on "machine versus machine" com- parisons, struggle comprehensively assess the development of visual intelligence as they rely on predefined metrics in experimental settings. The VTT, at the intersection of turing tests and computer vision, gaining attention for its "human versus machine" approach, which uses human visual capabilities as baseline to fully evaluate machine effectiveness 226 44 45
46 26
25 24) This
paradigm has been applied in SÓT that simulate human DVA: GIT demonstrates that humans can effectively resolve STDChallenge, and PathTracker 122 47 has developed circuit model to explain human decision-makinn processes. We are going to explore the visual search ability of humans and machines when tracking an arbitrary moving target.
3 Solving the STDChallenge
This section presents the proposed CPDTrack method in detail. We introduce CPD motion model, which divides global information into central and peripheral vision to optimize feature extraction. Additionally, we have developed a temporal clues module that serves as the information query component within CPD theory, facilitating top-down control mechanisms.
3.1
CPD Motion Model
A erucial challenge is balancing the maintenance of local resolution and computational efficiency during the extraction of global information. To address this, we incorporate insights from cognitive neuroscience, specifically the fovea concept (48) and CPD, segmenting the current frame into central and peripheral. Their computation is divided into the following steps:
Determine the position. For the central vision, shown in Fig. we the tracking result 4- = -00 1 Wt 1t-1, 11 on the previous frame to determine its position.
Central vision. To simulate the decay in visual sensitivity characteristic of the fovea, we employ gaussian distribution model-a method that has found widespread application in vision science
48 49]. Visual sensitivity is defined as:
(I b5=1)
S(x) = So exp(
(I)
2 o²

---

(a] The encoding suecoding-selection -edeoddn framework of CPD, and human visual model information
Ioput
mainly central
-10'9 bits
Encoding e.g. data sampling
Selection looking starts in V1
D Decoding Seeing
redina
V +higher visual areas
A ttection bottleneck
Macmun
OWibbrn 2014
bits per second
<rep you
(b) A rchitocture of the proposed CPDTrack
Information query
-10'' bits
Temporal
-10'6 bits
Trans- former Encoder
Head Decoder
Central
Perpheral
Peripheral
feature of template
feature of macolar area
fecture of
temporal query
temporal query with curment information
peripheral
Figure 3: The overall architecture of CPDTrack, referencing the latest one-stream trackers, models
CPD. (a) The encoding-seleccion-- selection-decodii framework of CPD. Visual selection is the process of choosing the information to focus on, and visual decoding is the process of deeply understanding the selected features and information make cognitive decisions. On the left is mathematical model of visual selection, where we align human visual information processing with the resolution of devices and machines. (b) Architecture of the proposed CPDTrack: The original frame is treated encoded visual data. We apply the acuity model described in (a) for visual selecting, which can be modulated by information query from temporal. A transformer employed replicate the decoding processes occurring in higher brain regions, facilitating complex cognitive tasks. The grey arrows running through both parts highlight the correspondence between the two parts.
where ((x) ) is sensitivity in x and So is the sensitivity in central. we set the a W based on the overall size (H, W) of the image and the 3-sigma rule. This approach ensures that the cumulative visual sensitivity to current frame exceeds 99.7%, effectively covering the vast majority of the visual field.
&
Sensitivity of central. We determine the sensitivity levels within this region using the values from b.-1. Integrating sensitivity as:
sens
S(x)da,
where b b,- 11_1 is the top-left coordinate of b.-1, and b,3, nottom-right coordinate of b1-1-
65th
We-1 1 is the
Image crop resizing. As the central region is resized to constant size, thus its width wf-1 x seme, and wi_1 x Wt-1. Therefore, w determined to be:
W
Wc
w sens
24
3
where a constant factor that can be adjusted, and + is the cumulative density function of the standard normal distribution. Then, we get central region m. (bg=,, b,",, wf-1: h(-1).
As the lower limit of the bounding box is set, typically at pixels in tracking, equation 3 will not division by zero error in practice.
5

---

We prove the range wi-1 in equation range of values for is (0, W]:
3 First, take the derivative with respect to wf-1(x), The
3m 2
310 x.
w (I)
(4)
(2
1)²
w
where f is the probability density function of the standard normal distribution. We define the numerator as (r) and denominator as du (z)
It is clear that du (x) 2 O when 0; differentiate The (x)): na (z)
18
3z
f(((
(5)
W
W
Clearly, S < 0. Thus, (x) has its minimum value at x = O, nu (a:) 2 0. Therefore, w monotonically increasing function when æ 0. When æ takes the maximum value, W. w(-1(x) & 0.9987SW.
Intuitively, the central area is determined by cropping around the target based the - This differs from traditional motion models, which typically use a static context ratio for cropping. contrast, the CPD model adapts the contextual range dynamically, offering more flexibility and precision, more visualization in Appendix B
For the peripheral area, we adjust the entire image to match the size of the central area. This resizing facilitates the simulation of peripheral vision, aligning the scale both regions.
3.2 Information query
In CPD [2]. the selection process is influenced by cognitive feedback through top-down approach, as illustrated in Fig. 3 Similarly mode that humans can actively focus on areas of interest, CPDTrack incorporates an information query mechanism. Spatially, we determine central vision based on the results b of the previous frame. Semantically, we use the query to map high-level semantic information from the previous frame downwards, achieving top-down control. Specifically, each past frame represented by token while template max- max-pooled into a query as initialization. This query, using aross-attention, extracts refined high-level semantic insights from multiple past frame tokens, then concatenated to images tokens and inputted into the ViT (50). Within the ViT backbone, the information query interacts with the global information of the current frame and the online template, achieving implicit cognitive feedback's 's top-down control. Simultaneously, this query integrates the ppatio-tempora information of the current frame for use in future frame computations.
4 The STDChallenge Benchmark
STDChallenge Benchmark is an integrated task closely associated with various object tracking subtasks 10 We select sequences with STDChallenge from established benchmarks based on specific sequence level attributes. Furthermore, inspired by cognitive psychology [51], we have designed dynamic visual capability assessment framework evaluate and compare the performances of humans and machines when tackling STDChallenge. This evaluation directly informs and guides machine development.
Construct STDChallenge Benchmark. Our objective is not to develop a new dataset to evaluate trackers' visual search ability. Rather, aim to highlight the overlooeer issue of STDChal- lenge, already acknowledged by few recent datasets but underemphasized in algorithms design. We have chosen existing benchmarks our data sources to assess trackers' proficiency with STDChal- lenge, specifically selecting sequences from LaSOT 9. VOTI.T2019 [52] and VideoCube 4 where such challenges existing. Drawing inspiration from cinematography, we introduce the metric
(na + n.,) la
STD
12
to quantify STDChallenge, where nh means the number of absent, n. means the number of shotcul, la means the length of absent and 1 means the length of the sequence. This formula takes into account the frequency and duration of absent, and the frequency of shotcut. A higher STD value indicates

---

greater STDChallenge. From LaSOT 19) VOTLT2019 152,, and VideoCube 4 we have compiled dataset of 252 sequences that constitute the STDChallenge Benchmark, detailed attributes and distribution would typically be visualized in Appendix C
Visual Turing Test. Our investigation begins by assessing whether individuals can effectively handle the STDChallenge. In line with methodologies from the field of static vision [26] (25], we organize the study using small-N design [51 26). enlisting five participants aged 20 to 30. During the experiment, participants watch videos and are instructed to track the target with a mouse cursor. The cursor's 's position is recorded each frame by background program. Sixteen videos (named STDChallenge Turing) sampled from STDChallenge Benchmark are displayed in 15 fps, each in a different order across participants. Prior starting, participants receive detailed training to familiarize themselves with the experimental process and techniques, more details in Appendix
(8) STDChellexg--Turing
Boxplot Human and Machine
consisteccy
humans machines
0.307 cerrnn
0.16
De (75022)
Suntoo 0344
unpasse (2.99)
engrince C.304
RApendinn 0807
Pys 0.00
2.11
cooee 0.00
0053
di-bc track
0.0
desierep COME
0.211
15511
00 02 644
Sanulled
Figure 4: Quantitative indicators show that CPDTrack not only has higher accuracy in the STDChal- lenge but also behaves more similarly to humans. (a) represents the N-PRE score on STDhhallenge- Turing. (b) represents the distribution of N-PRE scores of humans and machines on various sequences of STDChallenge -hiring (c) represents the error consistency between machines and humans, with kappa representing the average error consistency.
BOILLES
KXILES
200544
STOP
Spotcut
absent
431063
shotcut
SegTrack F. V1M220
Figure 5: Human results do not necessarily mean correct, but humans can usually quickly re-locate the target after STDChallenge. In the upper row, humans can recognize environmental factors closely related to the target in the second image. In the lower row, even if the target is absent, humans are not distracted by the background in the second image. Humans robust to occlusions in the fifth image.
5
Experiments
5.1 Visual Turing Test in STDChallenge-Turing
Evaluation metrics. Given that human typically focuses on single point and emphasizes an object's salient features rather than its center, we have adjusted our evaluation metrics and outputs accordingly. For error consistency measurements refer to (26), human is successful if the point fall within the groundtruth while trackers is successful if ToU > 0.5 in frame. For performance measurements, we employ the N-PRE metric (4) which evaluates the precision based on relative position between the center of boundingbox and groundtruth, to assess the performance of both humans and machines.
7

---

Table State-of- oftte-of-the-are comparison on STDChallenge, VideoCube
4 and LaSOT 19] The top three
results are highlight with red, blue and green fonts, respectively.
VideoCube R-OPE
STDChallenge
VideoCube
LaSOT
Motion Model
Method
N-PRE
PRE
SUC
N-PRE
PRE
SUC
N-PRE
SUC
Robust
AUC P
CPD
CPDTTacck
84.2
73.3
65,9
82.9
67.1
70.4
89.5
75.6
75.3
66.1
73.0
Soq Track
81.9
71.9
66.3
76.8
54.0
63.3
88.3
72.3
14.6
69.9
76.3
OSTrack
79.1
68.9
64.6
73.7
50.7
71.3
74.4
69.1
75.2
MixVIT 1
82.5
71.6
66.7
76.9
52.2
63.1
88.5
72.7
74.7
69.6
75.9
STARK [33
80.7
68.2
64.5
76.3
49.4
62.1
86.8
70.4
74.5
67.1
KeepTrack L5
80.4
62.8
73.0
64.4
70.2
Ocean [54
57.1
39.9
40.7
53.9
19.5
34,2
74,8
51.2
73.7
56.0
56.6
Local Crop
SuperDiMP S5
72.6
56.7
56.5
64.6
31.4
47.4
80.1
61.2
74.3
64.1
PrDMMP [55
70.3
51.7
52.7
65.4
28.6
44.5
58.3
74.3
59.8
60.8
DIMP [56
65.9
47.0
48.6
S4.6
18.7
37.1
77.2
$6.0
74.0
56.9
56.7
SiamRPN [35
53.4
35.6
37.3
46.7
15.0
29.0
72.6
50.3
73.6
ATOM [57
57.8
39.8
40.8
43.6
14.0
26.7
75.2
53.1
73.8
51.5
50.5
KYS [58
60.1
42.6
44.5
49.3
17.1
33.7
80.1
59.4
73.3
55.4
SiamPC
33.6
20.6
15.8
3.6
35.6
72.7
33.9
Local Global
SPLT 38
60.9
38.2
40.3
56.5
15.7
33.7
72.4
47.6
13.5
39.9
42.6
DaSiamRPN 39
53.4
35.4
37.1
46.3
14.4
29.1
72.2
50.4
73.6
42.7
44.8
Global
StamRCNN
75.3
62.8
60.7
72.6
47.9
58.8
65.8
GlobalTrack
42
65.5
49.5
49.5
64.3
29.6
46.1
72.7
53.7
74.3
52.1
52.7
Experiments results. In the STDChallenge, humans outperform most trackers, indicating that the DVA of trackers is still far from humans. Furthermore, while the performance of machines tends to vary significantly with STD changing, human consistently exhibit stable tracking ability, as shown in (b) of Fig. | Notably, as research progresses, the scores of SOTA trackers are gradually approaching those of humans, suggesting narrowing gap between machine and human performance. Additionally, the low error consistency between trackers and humans suggest they employ different tracking strategies in past studies. There's significant positive correlation between the N-PRE of trackers S7DChallenge and their error consistency with humans, indicating that trackers are incorporating some implicit human-like strategies to improve performance progressively. Among them, CPDTracker, which explicitly models human strategies, achieved the highest score of 0.167. This method not only effectively tackles the STDChallenge but also aids in guiding machines to exceed the biases of evaluation benchmarks and adapt to real-world scenarios.
Some Reasoning. () Like global trackers shown in Fig. 2 humans process entire image information can handle the STDChallenge more effectively at higher frame rates. This demonstrates the superior efficiency of humans in vidco information processing. (ii) When a target is lost, humans not only quickly become aware of this but also swiftly reacquire the target. This process involves the integrated use of spatial cognition and temporal memory, demonstrating human DVA to understand global information, in contrast to mainstream trackers that rely on local information. (iii) Furthermore, observations reveal that humans tend to focus object's 's salient features over its center when tracking. This significantly leads to current evaluation metrics underestimating human capabilities, while also highlighting the clear target recognition ability of humans.
Moreover, we examined how different modules within the trackers' pipeline influence performance and error consistency, For detailed findings, refer to the Appendix F
5.2 Comparison with SOTA
STDChallenge Benchmark. The STDChallenge Benchmark is a newly proposed benchmark for LTT that focuses on STDChallenge. Designed refer human visual research ability, it tests capability of trackers to reacquire lost targets. CPDTrack shows improvements of 1.7% N-PRE and 1.4% PRE over tracker, as shown in Tab. T Given the long-tail distributions of STDChallenge, we charted performance fluctuations across different STD. As illustrated in Fig. (a). CPDTrack excels robustness and significantly outperforms SOTA trackers, especially sequences with increasing STDChallenge (the last 40%, consisting of 100 sequences). We assessed CPDTrack's ability to search targets during the STDChallenge. A recovery deemed successful if loU between bounding box and groundtruth exceeds 0.5. As illustrated in Fig. 6 (b), CPDTrack reliably maintains targer tracking under STDChallenge. CPDTrack achieves a recovery success rate of 56.91 % within the same time period, significantly outperforming the next best tracker, mixvit.
the
8

---

(3) SUC Rortwating with STD
(h) Recaembiiiity of difforent trolaas
07
8387
sspendirg 1.055
g6oddrcck 0483
84
3228 41.44
Editcct E1.55
Carope 240
0.122
perentt STDChallnns
Figure 6: We further emphasize the visual search capabilities of CPDTrack, especially in challenging STDChallenge scenarios. (a) performance of SUC as the STD changes. We have listed the SUC scores the toughest 100 sequences. (b) represents robustness when facing the S7DChallenge, where success rate refers to the percentage of frames which the tracker successfully tracks the target.
Table 2:
Ablation studies of CPDTrack STDChallenge and VideoCube, We conducted comparisons of central-periphent vision and information query.
STDChallenge Benchmark
VideoCube
Method
consistency
N-PRE
PRE
SUC
N-PRE
PRE
SUC
baseline
0.159
83.8
72.9
65.6
82.7
67.0
70.2
central vision
0.137
81.5
72.2
66.8
77.7
57.7
65.9
peripheral vision
0.156
80.7
69.2
61.7
80.6
60.0
65.7
baseline+query
0.167
84.2
73.3
65.9
82.9
67.1
70.4
local central 0.165
83.0
70.6
64.5
84.3
62.9
69.0
VideoCube. VideoCube is a detailed and challenging LTT benchmark aimed at simulating real- world complexities, including View View-point changes and disappearances. It features two evaluation methods: One-Pass Evaluation (OPE) and Restarting OPE (R-OPE), the latter allows trackers to restart after failure evaluate their robustness. CPDTrack outperforms the second-ranked machine across all performance metrics, with improvements ranging from 1 1.0% to 13.1%. Additionally, there has been notable enhancement in its robustness 75.8, as shown in Tab.
LaSOT. LaSOT a high-quality, large-scale benchmark for LTT featuring a test set of 280 videos with average of 2448 frames each sequence. We attribute underperformance of CPDTrack to the limited STDChallenge scenarios and varying interpretations of groundtruth within these tests as in Fig. 7
5.3 Ablation and Analysis
Through ablation studies, we validate the effectiveness of each module of CPDTrack in STDChallenge.
We conducted ablation studies of CPDTrack in STDChallenge Benchmark and VideoCube, comparing "central-peripheria peripheral" vision dichotomy and the use of information query. Information
the impacts of "central- query was removed in #1 as bascline examine the performance of the CPD Motion Model itself. #2 traditional local crop, which can be considered as only containing central vision. #3 employed only peripheral vision. #4 is the proposed CPDTrack to validate the effectiveness of cognitive feedback control in ANN. Based on #2, peripheral vision and information query was added in #5 to demonstrate the effectiveness of our designed CPD Motion Model.
The combination central and peripheral vision performs better than using either alone. can be seen that performance metrics and error consistency of baseline #1 surpass those of #2 using only central vision and #3 using only peripheral vision, with improvements range 0.7% 9.3% in various metrics S7DChallenge Benchmark and VideoCube and more like human, in Tab. However,

---

#000000
15
S000 S0G
Trakkng resils llion SoqUEIO
(b) Tracking results on "hellcgger" sequenoe
GroadTrrut
CPDTaac (ouns)
OSTrack (PCCVZ)
SegTrack (CVPR22
(PAMEZO
Figure 7: Visualizations of CPDTrack and some local trackers tracking results. This demonstrates that CPDTrack, influenced by the global perspective, tends to frame the entire target, showing similarity to human cognition of objects, such as "a lion's tail" or "the tail rotor of a helicopter." This an error, but rather bias introduced by the dataset's setup.
central vision may be better at precise localization. We believe this demonstrates the complementarity of central and peripheral vision, further validating CPD theory.
Information query is effective. It can be seen in Tab, 2 compared to baseline#1, I, #4 has improved performance by 0.1 100.4% on several metrics in STDChallenge Benchmark and VideoCube, and with 0.8% improvement in error consistency. This indicates that in tracking, utilizing higher-level semantic information for top-down control beneficial for trackers understand sequence information. On the other hand, information query can also be considered as the retrieval of short-term memory, which suggests that combining short-term and long-term memory (online template) in tracking can better address the S7DChallenge.
Modeling central vision with Gaussian acuity effective. Compared the traditional local crop as central vision #5, CPDTrack #4 has surpassed in performance metrics on STDChallenge Benchmark and VideoCube, with improvements ranging from 1.2% to 4.2%, in Tab, 2 The error consistency is also higher than #5. This suggests that considering the bounding box and image size, dynamically extracting context for central vision can better align with peripheral vision.
Conclusion.
Our study is inspired by the differences in dynamic visual abilities between humans and machines, notably challenging tasks like LTT and GIT. We attribute this gap primarily to the machines' deficiency in essential visual search skills. Based on CPD from cognitive neuroscience, which categorizes visual inputs into central and peripheral vision, we understand humans' robust visual search ability under limited computational resources.
We propose CPDTrack, extensively validated for effectiveness and closer behavioral alignment with humans in STDChallenge Benchmark. Ablation studies highlights that its success is due to the efficient integration of central and peripheral vision. Our findings indicate that robust visual search capabilities are crucial for gradually adapting trackers to real-world applications.
Acknowledgments and Disclosure of Funding
This work is jointly supported by the National Science and Technology Major Project (No. 16403), the National Natural Science Foundation of China (No.62176255), and the Strategic Priority Research Program of Chinese Academy of Sciences (DA277220))))))
10

---

References
[1] Li Zhaoping. Evaluating the central central-erriperu dichotomy human visual cortex using anatom- ical and retinotopic data in human connectome project. Journal of Vision, 23(9):5755-5777 5755, 2023.
[2] Li Zhaoping. Peripheral and central sensation: multisensory orienting and recognition across species. Trends in Cognitive Sciences, 2023.
[3] Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thuemmel, and Martin V Butz. Learning what and where: Disentangling location and identity tracking without supervision. The Eleventh International Conference on Learning Representations. 2022.
[4] Shiyu Hu, Xin Zhao, Lianghua Huang, and Kaiqi Huang. Global instance tracking: Locating target more like humans. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):576 -995,, 2023.
[5] Shiyu Hu, Xin Zhao, and Kaiqi Huang. Sotverse: A user-defined task space of single object tracking. International Journal of Computer Vision, 132:872-930, 2024.
[6] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence,
2015.
[7] Matej Kristan, Jiri Matas, Ales Leonardis, Tomá Vojif, Roman Pflugfelder, Gustavo Fernández, Georg Nebehay, Fatih Porikli, and Luka Cchovin. A novel performance evaluation methodology for single-target trackers. IEEE Transactions Pattern Analysis and Machine Intelligence, 38(11):2137-2155 2016.
[8] Xin Zhao, Shiyu Hu, Yipei Wang, Jing Zhang, Yimin Hu, Rongshuai Liu, Haibin Ling, Yin Li, Renshu Li, Kun Liu, and Jiadong Li. Biodrone: A bionic drone-based single object tracking benchmark for robust vision. International Journal of Computer Vision, 132: 1659-1666,,,, 2024.
[9] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5374-5383, 2019.
[10] Alan Lukezič, Luka Čehovin Zaje, Tomáš Vojiř, Jifí Matas, and Matej Kristan. Performance evaluation methodology for long-term single-object tracking, IEEE transactions on cybernetics, 51(12):6305-631 2020.
[11] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF conference compuler vision and pattern recognition, pages 13608-13618 13618, 2022.
[12] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In European conference computer vision, pages 341-357. Springer, 2022.
[13] Xin Chen, Houwen Peng, Dong Wang, Huchuan Lu, and Han Hu. Seqtrack: Sequence to sequence learning for visual object tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14572-14581, 2023.
[14] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 8 8126-8135, 2021.
[15] Christoph Mayer, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool. Learning target candidate association to keep track of what not to track. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13454, 2021.
[16] Miaobin Cen and Cheolkon Jung. Fully convolutional siamese fusion networks for object tracking. 2018 25Th IEFE international conference on image processing (ICIP), pages 3718-3722 IEEE, 2018
11

---

[17] Heng Fan, Fan Yang, Peng Chu, Yuewei Lin, Lin Yuan, and Haibin Ling. Tracklinic: Diagnosis of challenge factors visual tracking. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 970-979, 2021.
[18] Shiyu Hu, Xin Zhao, and Kaiqi Huang. Sotverse: A user-defined task space of single object tracking. International Journal of Computer Vision 132(3):872-930, 2024.
[19] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high high-diversity benchmark for generic object tracking in the wild. IEEE transactions on pattern analysis and machine intelligence, 4365):1562-1577 2019.
[20] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13763- 13773, June 2021.
[21] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Track- ingnet: large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), pages 300-317,201
[22] Drew Linsley, Girik Malik, Junkyung Kim, Lakshmi Narasimhan Govindarajan, Ennio Mingolla, and Thomas Serre. Tracking without re-recognition in humans and machines. Advances in Neural Information Processing Systems, 34:19473-19486, 2021.
[23] Xuchen Li, Shiyu Hu, Xiaokun Feng. Dailing Zhang, Meiqi Wu, Jing Zhang, and Kaiqi Huang. Visual language tracking with multi-modal interaction: A robust benchmark. arXiv preprint arXiv:2409.08885 2024.
[24] Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
[25] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885- 23899, 2021.
[26] Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial- by-trial behaviour of cnns and humans by measuring error consistency. Advances in Neural Information Processing Systems, 33:13890-13902 2020.
[27] Xuchen Li, Shiyu Hu, Xiaokun Feng, Dailing Zhang, Mciqi Wu, Jing Zhang, and Kaiqi Huang. Dtvlt: A multi-modal diverse text benchmark for visual language tracking based on 1lm. arXiv preprint arXiv:2410.02442, 2024.
[28] Xuchen Li, Xiaokun Feng, Shiyu Hu, Meiqi Wu, Dailing Zhang, Jing Zhang, and Kaiqi Huang. Dillm-vit: Diverse text generation for visual language tracking based on 11m. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7283-7292 2024.
[29] Gioele Ciaparrone, Francisco Luque Sánchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, and Francisco Herrera. Deep learning in video multi-object tracking: A survey, Neurocomputing, 381:61-80 88, 2020.
[30] Brian Q. Geuther, Sean Deats, Kai J. Fox, Steve A. Murray, Robert E. Braun, Jacqueline K. White, Elissa J. Chesler, Cathleen M. Lutz, and Vivek Kumar. Robust mouse tracking in complex environments using neural networks. bioRxiv, 2018
[31] Shiyao Wang, Yucong Zhou, Junjie Yan, and Zhidong Deng. Fully motion-aware network for video object detection. Proceedings of the European conference on computer vision (ECCV), pages 542-557, 2018.
12

---

[32] Andre Esteva, Katherine Chou, Serena Yeung, Nikhil Naik, Ali Madani, Ali Mottaghi, Yun Liu, Eric Topol, Jeff Dean, and Richard Socher. Deep learning-enabaee medical computer vision. NPJ digital medicine, 4(1):5, 2021.
[33] Jack Valmadre, Luca Bertinetto, Joao F Henriques, Ran Tao, Andrea Vedaldi, Amold WM Smeulders, Philip HS Torr, and Efstratios Gavves. Long-term tracking in the wild: A benchmark. In Proceedings of the European conference on computer vision (ECCV), pages 670-685, 2018.
[34] Abhinav Moudgil and Vineet Gandhi. Long-term visual object tracking benchmark. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part II 14, pages 629-645. Springer, 2019.
[35] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu, High performance visual tracking with siamese region proposal network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8971-8980, 2018.
(36] Yutong Kou, Jin Gao, Bing Li, Gang Wang, Weiming Hu, Yizheng Wang, and Liang Li. Zoomtrack: Target-aware non-uniform resizing for efficient visual tracking. Advances in Neural Information Processing Systems, 36, 2024.
[37] Yuqing Huang, Xin Li, Zikun Zhou, Yaowei Wang, Zhenyu He, and Ming-Hsuan Yang. Rtracker: Recoverable tracking via pn tree structured memory. arXiv preprint arXiv:2403. 19242, 2024.
[38] Bin Yan, Haojie Zhao, Dong Wang, Huchuan Lu, and Xiaoyun Yang. "skimming- perusal' cruusalttracking. A framework for real-time and robust long-term tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2385-2392 2019.
[39] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. In Proceedings of the European conference on computer vision (ECCV), pages 101-117, 2018.
[40] Haojie Zhao, Bin Yan, Dong Wang, Xuesheng Qian, Xiaoyun Yang, and Huchuan Lu. Effective local and global search for fast long-term tracking. JEEE Transactions on Pattern Analysis and Machine Intelligence, 4561):460-474 2022.
[41] Yunhua Zhang, Lijun Wang, Dong Wang, Jinqing Qi, and Huchuan Lu. Learning regression and verification networks for robust long-ten tracking. International Journal of Computer Vision, 129(9):2536-2544 2021.
[42] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Globaltrack: simple and strong baseline for long-term tracking. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 11037-11044, 2020.
[43] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe. Siam r-cnn: Visual tracking by re-detection. Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pages 6578-6588, 2020.
[44] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266)) 1332-1338, 2015.
[45] Sam Devlin, Raluca Georgescu, Ida Momennejad, Jaroslaw Rzepecki, Evelyn Zuniga, Gavin Costello, Guy Leroy, Ali Shaw, and Katja Hofmann. Navigation turing test (ntt): Learning to evaluate human-like navigation. In International Conference on Machine Learning, pages 2644-2653 PMLR, 2021.
[46] Thomas Langlois, Haicheng Zhao, Erin Grant, Ishita Dasgupta, Tom Griffiths, and Nori Jacoby. Passive attention in artificial neural networks predicts human visual selectivity. Advances in Neural Information Processing Systems, 34:27094-22 106, 2021.
[47] Girik Malik, Drew Linsley, Thomas Serre, and Ennio Mingolla. The challenge of appearance- free object tracking with feedforward neural networks. arXiv preprint arXiv:2 10.02772, 2021.
13

---

[48] Andrew B Watson. A formula for human retinal ganglion cell receptive field density as function of visual field location. Journal of vision, 14(7):15-15, 2014.
[49] Guillaume Lio, Roberta Fadda, Giuseppe Doneddu, Jean-René Duhamel, and Angela Sirigu. Digit-tracking as new tactile interface for visual perception analysis. Nature Communications, 10(1):5392, 2019.
[50] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint urXiv:2010.11929 2020.
[51] Philip I. Smith and Daniel R Little. Small is beautiful: In defense of the small-n design. Psychonomic bulletin & review, 25:2083-2101, 2018.
[52] Matej Kristan, Jiri Matas, Ales Leonardis, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kamarainen, Luka "Cehovin Zaje, Ondrej Drbohlav, Alan Lukezic, Amanda Berg, et al. The seventh visual object tracking vot2019 challenge results. In Proceedings of the IEEE/CVF international conference computer vision workshops, pages 0-0, 2019.
[53] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning pratio-tempord transformer for visual tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10448-104577 2021.
[54] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu. Ocean: Object-aware anchor- free tracking. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 771-787. Springer, 2020.
[55] Martin Danelljan, Lue Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7183-7192, 2020.
[56] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6182-6191, 2019.
[57] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Atom: Accurate tracking by overlap maximization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4660-4669, 2019.
[58] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Know your surroundings: Exploiting scene information for object tracking. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXIII 16, pages 205-2211 Springer, 2020.
[59] Li Zhaoping and Yushi Liu. The central pertral-peripheral dichotomy and metacontrast masking. Perception, (1(5):449-5555 564, 2022.
[60] Junhao Liang, Severin Maher, and Li Zhaoping. Eye movement evidence for v1 saliency hypothesis and central-peripheral dichotomy in an anomalous visual search task. In Systems Vision Science: Summer School & Symposium, 2023.
[61] Yushi Liu and Li Zhaoping. Testing prediction of the -entral-prripheral dichotomy in visual inference: visual backward masking is weaker in the peripheral visual field. In 42nd European Conference on Visual Perception (ECVP 2019), page 188. Pion Ltd., 2019.
[62] Giorgio Roffo, Simone Melzi, et al. The visual object tracking vot2016 challenge results. In Computer Vision-ECCC 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II, pages 777-823. Springer International Publishing, 2016.
[63] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka "Cehovin Zaje, Tomas Vojir, Goutam Bhat, Alan Lukezic, Abdelrahman Eldesokey, et al. The sixth visual object tracking vo12018 challenge results. In Proceedings of the European conference computer vision (ECCV) workshops, pages 0-0, 2018.
14

---

[64] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[65] Matej Kristan, Aleš Leonardis, Jifí Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Martin Danelljan, Luka Cehovin 7aje, Alan Lukežič, Ondrej Drbohlav, et al. The eighth visual object tracking vot2020 challenge results. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 547-601. Springer, 2020.
[66] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2017.
[67] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Eco: Efficient convolution operators for tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6638-6644 2017.
15

---

Appendices
A Extended related work.
The attention bottleneck force animals to only process a subset of sensory input in depth. This motivates unifying central-peripheral dichotomy (CPD), categorizing multisensory processing into central and peripheral senses based on their functions D
48
49
59
60
61].
In the CPD theory, the encoding-selecttin seleccionndecoddeee framework combined with central and peripheral sensations reflects how the brain processes sensory information stages to optimize resource usage and cope with environmental challenges.
The encoding stage involves the reception and preliminary processing all initial sensory inputs entering the sensory system. At this stage, sensory information is converted into signals that the brain can further process.
The selection stage is the phase in CPD where 'central' and 'peripheral" selections are separated. This process is driven only by saliency mechanisms but also influenced by top-down expectations, which are achieved through higher cognitive processes in the brain.
The decoding stage is the active part of central senses' in CPD. After the selection stage, the selected few central sensory inputs are transmitted to higher brain regions for in-depth analysis. This stage requires complex cognitive processing, including integrating information, recognizing patterns, and understanding the deep meaning inputs. We draw from these principles to construct the CPDTrack.
B
More details of central vision modeling
Using W 1024, we sampled one hundred thousand points from 10 to W and visualized the function.
Range of w
1000
800
600
W Wt-1
400
200
y=x
0
W=1024
0
200
400
600
800
1000
W
Figure 8: wi_1 monotonically increases with W;-1- The green line represents wf_1 Wt-1, while the black line represents the image width W.
16

---

C Detailed Components of STDChallenge
Table 3: Representative Benchmarks in STT. LTT. GIT and STDChallenge Benchmark
Min frame
Mean
Max
Total
Subtask
Benchmark
Videos
absent
shotcut
frame
frame
frame
OTB2015
100
71
590
3872
59K
X
VOT2016 62
60
41
357
1500
21K
X
X
STT
VOT2018 63
60
41
356
1500
21K
X
XXX
X
VO12019 52
60
41
332
1500
20K
GOT-10k
10000
29
149
1418
1.45M
VOTI.T2019522
50
1389
4305
29700
215K
X
LIT
LaSOT[9
1400
1000
2502
11397
3.5M
X
GIT
VideoCube
500
4008
14920
29834
7.46M
V
LTT+
STDChallenge Benchamrk
252
1000
5192
29700
1.3M
GIT
(6) Components STOchsllenge
Di of mctt
(C)
WaleCaae
304
aino parte
00.5
and
Figure The STDChallenge Benchmark is composed of high-difficutty sequences extracted from LTT and GIT benchmarks. Its STD still exhibits a long-tail distribution.
The STDChallenge integrates LTT and GIT tasks to assess the visual search capabilities of tracking machines, which addresses the bias of single datasets, shown in Fig. [1] This challenge incorporates sequences drawn from the LaSOT (9) VOTLT2019 [52), and VideoCube 4. thereby inheriting their characteristics. The sequence lengths vary, with the shortest being "yoyo-15" from LaSOT and the longest "liverRun" from VOTLT2019, 9. Notably, the extensive dataset of LaSOT contributes significantly the diversity and complexity of the challenges presented in STDChallenge.
STDChallenge sequence-leved challenge which we analyze the impact of frame-level challenges, shown in Fig. 10 These frame-level challenges structured following SOTVerse Verse (18), employing the same computational methods. The STDChallenge benchmark is noted for its comprehensive inclusion of various challenges, particularly in terms of blur_ _bbx, relative_ ssccle, and relative ratio. Trackers that effectively manage these specific challenges are likely to achieve superior performance within the STDChallenge framework.
can seen that on the STDChallenge Benchmark, the STD still shows a long-tail distribution, where the majority of sequences exhibit fewer challenges, while a minority contain higher number of challenges. There have already been many large-scale benchmarks target tracking, and how to increase the amount of data under specific conditions is promising direction.
In our setup, sequence with shotcut or absent is deemed to include STDChallenge, naturally integrating challenges found in new real-world scenarios. Thus, STDChallenge not only incorporates but also enhances other challenging attributes to some extent, as depicted in Fig. Fig. absent is often associated with occlusions and out-of- scenarios. Formally, STDChallenge does not exclude other tasks (such as pathtracker (22]) but we advocate integrating STDChallenge with real-world scenarios to avoid ill-posed issues (such as shotcul confusing same-looking blocks in pathtracker). Comparison between CPDTrack and pathtracker are Tab.
17

---

100
75
0
4
6
8
10
13
15
17
21
29
32
52
(a) Absent challenge in LaSOT.
125
100
75
50
25
11
45 (b) Absent challenge in LaSOText.
57
74
90
1250
1000
750
500
0
10
16
18
20
24
26
29
34
49
99
(c) Absent challenge TNL.2k
1
12
18
30
41
46
53
113
142
321
(d) Absent challenge in VideoCube.
Figure 10: STDChallenge exhibits long-tail distribution across most datasets. The horizontal axis represents the number of STDChallenge in each sequence, while the vertical axis represents the number of sequences.
Table 4: From the model design perspective, believe TransT+InT aims address Tracker,,e, challenge that relies on trajectory information rather than appearance features, which diverges from the Motivation of STDChallenge We attempted to replicate TransT+InT in VidcoCube and STDChal- lenge; however, we did not achieve optimal performance due to the lack of clear documentation as the issue in the github. Therefore, we believe these can partly explain the suboptimal performance of
TIn on STDChallenge.
STDChallenge
VideoCube PRE
Motion Model
Method
Error Consistency
N-PRE
PRE
SUC
N-PRE
SUC
CPD
CPDTrack
16.7
84.2
73.3
65.9
82.9
67.1
70.4
TransT
13.9
73.9
62.2
59.1
69.2
42.9
55.1
Local Crop
TransT+InT
9.2
70.6
59.0
56.4
61.0
36.7
47.0
18

---

Gi) hreprbon othorett ptotutts
(E Convoxticn senurred krrate
6 Detribtton of Mo
Figure 11: The STDChallenng Benchmark is composed of multiple datasets and has a more rational distribution of challenging attributes. Compared to VideoCube, expands the range of STD and lengths of sequence, mitigating biases in single dataset and thus providing more comprehensive assessment environment. It is recommended to view this in enlarged and color format for better clarity.
D Machines.
D.1
Implementation Details
Model. We adopt ViT-B[50] our encoder architecture for CPDTrack. The encoders are initialized with the MAE[64] pre-trained parameters. The patch size is set to 16 x 16. The decoder consists of 2 transformer blocks. The decoder hidden size is 256, the number of attention heads is 8, and the hidden size of the feed forward network (FFN) is 1024. The number of quantization nbins and the vocabulary size all set 4000. The dimension of word embedding is 256, which is consistent with the decoder hidden size. The embedding-to-oee by a softmax. The hidden dimension of the perceptron is 256. The output dimension is nbins, which is aligned with the number of words in vocabulary The word with the maximum likelihood is sampled as the output word. In addition, we present model parameters, FLOPs, and inference speed in T
sampling uses a 3-layer perceptron followed
Training. Our training data includes the training splits of VideoCube[], LaSOT COT-10K[19) and TrackingNet([1]. Aligned with VOT2020 evaluation 650, we remove the IK forbidden videos in GOT-10k during training. For the evaluation on GOT-10k test set, follow the official requirements and only use the training split of GOT-10k. Brightness jittering are used for data augmentation. We train the model with Adam [66) optimizer and set the learning rate of the encoder to le - 5, the decoder and remaining modules to le 4, and the weight decay to le 4. The model trained for total of 300 epochs with 60k image pairs per epoch. The learning rate decreases by factor of 10 after 240 epochs. The model is trained on server with four A5000 GPUs and is tested on an A5000 GPU. The tracking speed is about 23 FPS
Inference. We use the first template and online template together with the central region and the peripheral region as input of CPDTrack. The online template update interval T is set to 400 by default, while the update threshold 7 is set to 0.7. CPDTrack will attempt to update online template within 10 frames; if none meet the threshold, it will not update.
D.2 Local Crop Trackers
OSTrack [12. A novel single-stream tracking framework has been proposed, which unifies fea- ture learning and relationship modeling. At the same time, an candidate prior elimination mod- ule has been introduced to eliminate background interference while improving computational ef- ficiency.
Table 5: Details of CPDTrack.
Params
FLOPs
Speed
Model
Encoder
(M)
(G)
(fps)
CPDTrack
VIT-B
119
88
23
MixVIT ITI. ViX proposes a Mixed Attention Module (MAM), used for simultaneously extracting features and integrating target information. This synchronous modeling approach can extract distinctive features of specific targets and facilitates extensive communication between the target and the search area. In this version, ViT is used to compute attention.
19

---

SegTrack [13]. The predictive object bounding boxes are generated using an autoregressive method, which simplifies the loss function.
STARK [53].
Using an encoder-decoder for relational modeling, and head that directly predicts the corners of bounding boxes has been proposed.
KeepTrack [15]. KeepTrack combines SuperDiMP with a target candidate association network to achieve robust tracking. The authors retrained the target candidate association network challenging sequences extracted from LaSOT.
Ocean [54]. Ocean also adopts an anchor-free structure. The backbone network uses parameters pre-trained on ImageNet for initialization.
PrDiMP and SuperDiMP accuracy.
55 PrDiMP and SuperDiMP use probabilistic regression to improve
DiMP [56]. Based on the framework proposed by ATOM, DiMP optimizes the loss function to achieve stronger discriminative ability. The backbone network is initialized with ImageNet weights.
SiamRPN 35). SiamRPN introduces a region proposal network achieve precise object regression.
ATOM 57
ATOM attempts to combine CF and SNN and proposes a new framework to take advantage of offline training and online updates.
KYS 58 KYS represents scene information as state vectors and combines them with the appearance model to locate objects.
SiamFC [16]. As the pioneer of SNN-based trackers, SiamFC achieves satisfactory tracking perfor- mance by matching features between the template area and the search area through a simple network structure.
D.3
Local-Global Trackers
SPLT designed a verifier to switch between global search and local search. SPLT uses feature extractor and down-samples the spatial resolution of template features 1X1 through average pooling. In terms of the verification model, SPLT employs ResNet50 as the backbone the verifier.
SPLT 38
Learning regression and verification networks for robust long-term tracking [41] It employed a verification network with local view to identify the target from the detected candidates. If the target disappears, the learning based switching scheme determines whether to trigger the global search mode.
Effective Local and Global Search for Fast Long-Ter -eem- Tracking 40). used the score from target verifier within the local search module to decide whether to switch strategies.
DaSiamRPN [39]. DaSiamRPN uses data augmentation enhance its discriminative ability.
RTracker (37]. RTracker uses a tree-structured memory to dynamically link tracker and detector by controlling the collection of positive and negative target samples, thereby enabling self-recovery capabilities.
D.4 Global Trackers
SiamRCNN 43). SiamRCNN utilizes re-detection mechanism and a small trajectory dynamic programming machine address the issue of object disappearance. SiamRCNN based on the implementation of CNN,erRCNN, using ResNet-101-FPN its backbone.
GlobalTrack 42 GlobalTrack does assume motion consistency and performs full image search to eliminate cumulative errors.
20

---

E
Experiment Organization.
This section provides a comprehensive overview of the experimental setup for the VTT, including composition of test data, selection of equipment, experimental instructions, experimental procedures, survey questionnaire, and grouping information.
Ansent
3150
Present
absele
Labeent
Present
Present
Absent
Figure 12: Representative sequences used in Visual Turing Test: (T) TEST video example, which helps participants familiarize themselves with the operational process. (1) First group example from the STDChallenge Benchmark. Lower difficulty, containing fewer absent, almost shotcut, and no changes in scenery. (2) Second group example from the STDChallenge Benchmark. Medium difficulty, containing
absent, fewer shotcut, and some changes in scenery. (3) Third group example from the STDChallenge Benchmark. Higher difficulty, containing many absent, more shotcut, and significant changes in scenery.
E.1 Experiment Environment
In our dataset preparation, we initially categorize the STDChallenge benchmark into three distinct groups based on the STD as previously described: sequences with STD < 10-4, sequences with 10-4 STD < 10-3 and sequences with STD > 10-3 From each category, five sequences are randomly selected and paired with a designated test sequence to establish the STDChallenge-Turee test environment. This environment composed of sequences that represent a spectrum chal- lenges within STDChallenge Benchmark, providing robust framework for evaluating performance across varying difficulty levels.
E.2 Experiment Statement
We have communicated with our institution's IRB, submitted the necessary materials, and obtained IRB approval. The experimental tasks include watching videos and moving the mouse track target. To avoid the risk of worsening the participants' vision conditions, we allow participants to rest between two videos. Additionally, participants are allowed adjust their chairs and the distance to the screen to achieve the most comfortable conditions. Participants will be compensated 100 RMB per hour. The total amount used 1000 RMB.
Participants were asked to review and sign the following declaration of consent:
Psychophysical research. You will be asked to watch video sequences computer screen and track target using mouse. For completing session, you will be compensated rate of 100 RMB per hour.
21

---

Declaration of consent. agree to participate in behavioral experiment on visual perception. I participate voluntarily in this study. have been informed that can stop the experiment at any time without giving any reason and without any negative consequences. know that I can contact the experimenters at any time to ask questions about the research project.
Declaration of consent for data processing and data publication. consent that the experimental data obtained during the experiment may used for scientific evaluation and publication semi- anonymous form. agree to have my personal data (such as name, phone number, address) stored digitally; except for being used to contact me, may not be used for any other purposes. These personal data will only be retained within the researchers' group and will never be transferred to third parties.
E.3 Statistical Testing
we focused on ensuring the accuracy and validity data collected from a limited number of subjects to accurately reflect human visual capabilities. Consequently, implemented measures to prevent citt----iittibution (OOD) responses, enabling the use of all responses. 1) We did not crowdsourcing platforms; instead, the Visual Turing Test were conducted in laboratory environment under supervision. 2) Before the experiment, we assessed participants cognitive and perceptual levels ensure normalcy, and allowed them to familiarize with the equipment through basic exercises. 3) During the experiment, participants were permitted to pause the experiment certain number of times voluntarily adjust their state. 4) In cases where the mouse cursor moved off-screen, we retained the results from the previous frame until the cursor returned within the screen area.
Referencing [25],
E.4 Questionnaire.
Q1. Your name:
02. Your id:
Q3. Your age:
Q4. Your vision condition:
Q5: Approximately how many times were you unable to locate the target?
Q6: How do you feet about the video playback speed?
07: Do you feel that the videos vary in difficulty, and which ones do you find more challenging?
E.5
Device Selection
Measuring human DVA is a crucial aspect our research. Traditionally, studies have employed eye trackers computer mice assess visual capabilities. Our preference for using mouse over eye tracker is based on several considerations:
(1) Eye trackers, which passively record eye movements, are influenced by multiple factors including device accuracy, user posture, the distance between the eye and the device, and the presence of eyeglasses, leading to potential limitations in data accuracy.
(2) Eye trackers have greater cumulative errors. When errors occur, subjects can only adjust their posture and cannot actively correct the capture results. However, when significant cumulative errors occur with the mouse, we allow subjects actively pause the video and correct the errors.
(3) Some researchers have also pointed out from a theoretical perspective that eye trackers are affected by the tracking rate (i.e., the amount of eye movement data lost). Blinking brief shifts in gaze can result lower tracking rate, which requires subjects to remain highly focused while using the eye tracker. However, the video sequences we test are long, inevitably affecting the performance of the eye tracker due to the subject's 's fatigue.
(4) The use of a mouse to document observation points is well-established within both academic and industrial settings, providing a reliable and practical alternative for tracking visual observation in long-duration tasks.
22

---

E.6 Experiments Process
Process DVA Experiments:
(1) Participants initially inspect the setup equipment and adjust their chair height, seating posture, and their distance from the screen to ensure comfort and optimal viewing conditions.
(2) test video then plays centrally on the screen. the first frame, participants are tasked with observing and memorizing specific features of designated target, subsequently using the mouse to track this target throughout the remaining video frames.
(3) Upon completion of the test, the lead researcher addresses any questions from participants, clarifying any uncertainties.
(4) A series 15 official videos from varied groups are sequentially displayed, with breaks between videos to reduce visual strain. During these sessions, participants required to diligently track the target and accurately maintain the mouse's position.
(5) Finally, participants complete self-assessment questionnaire to evaluate their own performance and experience during the test.
E.7 Group Information
Table 6: Subjects and corresponding videos. Everyone must watch the same videos, but in different orders. One the sequences is the TEST sequence, used help subjects familiarize themselves with the process.
index
sequenee
STD
length
Exp_00
Exp 02
Exp_ 03
Exp_06 04
Exp_0s 05
TEST
cup-1
0.0
1433
TEST
TEST
TEST
TEST
TEST
01
kangaroo-5
4.996
10-5
3119
13
02
14
02
02
lank-9
3.195 x 10-5
1312
05
02
03
15
04
03
racing-15
4.279
10-5
1070
15
03
10
10
08
04
mouse-17
5.105 x 10-5
1608
09
07
05
13
05
sepia-8
3.363 10
3694
02
05
04
13
15
06
362
2.807 x 10
12704
06
01
11
06
07
squirrel-13
2.190 X 10
2000
12
07
09
03
05
08
gametarget-2
2.473 X
1834
10
14
09
436
4.611 X 10-1
11531
01
09
08
07
12
bottle-12
2.361 X 10-4
2400
10
11
015
6.290 X 10-3
7710
03
11
15
02
03
12
373
4.037 10
12972
14
12
07
13
401
4.860 x 10-3
4978
14
13
11
06
09
14
350
1.159
10
6648
14
13
01
11
15
033
4.074 X 10-3
14300
04
15
12
04
01
length
89313
89313
89313
89313
89313
89313
23

---

Performance of Human and Machines
F. Error Consistency of Humans and Machines
1.0
Error consistency of Machine and Human
<pdtreck
seqtrack-globel
Exp90
Human
Exp100
0.8
Exp110
Exp130
estrack
estrack.384
ostrack.git
seqtrack
segtrack b384
0.6
seqtrack-1256
1eqtrack-1384
seqtrack git
mixvit
Local Crop
mixvit-large
mixvit-git
stark-st
siamfc
superdimp
0.4
dimp
prdimp
ocean
keeptrack
Global
siamrcnn
globaltrack
Gobakloral
dasiamrpn
spit
cpdtrack
Exp90
Exp100
Exp110
Exp120
Exp138 pp1
ostrack
ostrack- 384
astrack-gn git
seqtrock
seqtracie b384
seatrack-- 1256
seqtrack- 1384
sentrack-git
mixvit
mixvit-lage
mixvit-git
starkest
siamifc
superdimp
dimp
atom
prdimp
ocean
keeptrack
siampon
kys
siamronn
globeltrack
desiemrrn
spit
seqtrack
The error consistency differs among decision Decision-makers (machines/humans) in STDChallenge. This represents the degree of behavioral similarity between different decision-makers. For each sequence in the STDChallenge-Tiree for different decision-makers, and then average these across sequences to obtain the overall error consistency among the decision-makers.
Figure 13:
Benchmark, we calculate the error consistency each frame
As shown in Fig. 13 CPDTrack exhibits higher degree of consistency with human behavior compared to other machines, yet it remains closely aligned with machines. Notably, it demonstrates a higher level of consistency with trackers that utilize ViT as their backbone. This observation under- scores the significant influence that backbone can have on algorithmic performance. Furthermore, CPDTrack shares more similarities with trackers that implement global strategy, under the same network architecture. We attribute this similarity to the adoption of strategies that more closely mirror human cognitive processes.
As shown Fig. 13 ViT-based trackers display high similarity due to shared components like backbone and motion model. Enhanced parameters, increased resolution, and more training data have improved their alignment with human error consistency. This is believed to result from the labels in supervised learning being annotated by humans, enhancing the algorithms' ability to fit these labels.

---

As shown in Fig. 13 compared to ViT-based trackers, those based on CNN exhibit lower similarity among themselves. Nonetheless, certain algorithms that build on others still achieve high consistency in error consistency.
As shown in Fig. T the motion model plays crucial role in this consistency, particularly affecting how trackers process and interpret video sequences, difference that becomes particularly pronounced high-resolution settings.
F.2
Effect of different modules in pipeline on trackers
Net malized Precision Score
Error Consétency
Nomalized Preckdon Score
18
Tree Convistency
0.8
0.8
14
14
0.7
0.7
0.6
10
0.5
10
0.5
[a] Impact the temporal module.
[b) Impact different architecture.
1owith/wttouut temporal module
0,1-2-3 4:nustomized snn-ef Emor Consisency
hermatized Precivion Score
Brier Consistency
Normatzed Prccsson Score
24
0.4
0.6
10-
0.5
0.5
Impact the motion model.
Impact Different contextual 1122345.((25--1388555--ooooo
0-1-2-3:local crop-local global-giobal-edd
homatized
Errr
Mormalteed Preciiion Soore
Deratalancy
0.550
16
36
0,800
10
0.
0.4
0.675
(e) Impact of different parameters 0-1-2-3 Stalesnat resnet 38-resnet_50 vit Li_bae--vit_lage
Impact different train data
Figure 14: Impact of different settings in tracker pipeline. We highlight the median each setting. In (f), "1" means the training contains GOT-10k, COCO, TrackingNet. and LaSOT, while "2" means the training set contains GOT-IOK, COCO, TrackingNet, LaSOT, and VideoCube, "3" means the training set contains GOT-1Ok. TrackingNet, LaSOT, and VideoCube.
(a) As shown in Fig. 14 a). Using temporal information is more effective and also more human-like like.
(b) As shown in Fig. b). Compared other network architectures, one-stream is more effective and also more human-like. However, since current one-stream trackers are all based on ViT, is unclear whether the effects are due one-stream or VIT.
(c) As shown in Fig. 14(c). Undoubtedly, the CPD motion model has better effects and more human-like behavior. At the same time, compared to the local-based motion model, the global one is more human-like, as it treats the entire image as central vision, improving performance at the cost of computational efficiency.
25

---

(d) As shown in Fig. d)... Increasing the context ratio does not seem to change the performance and error consistency, but after changing the resolution from 4(256) 4(384), there has been an improvement. How to maintain computational efficiency while increasing resolution is promising direction.
(c) As shown in Fig. [4 Obviously, increasing the number of parameters and switching to newer backbone can significantly improve performance and error consistency. We believe this due the increased parameters enhancing the network's 's ability to fit human labels. The error consistency of vit large has decreased, and we believe this might be because the network has learned some strategies that surpass human capabilities.
(f) As shown in Fig. f). Increasing training data does not necessarily mean improvement in performance, but it often enhances error consistency with humans. We believe this is because richer human labels enhance the network's s learning capabilities.
26

---

G Broader Impact and Limitation
Broader Impact. The fundamental objective of our research is to model human visual search capabilities through CPD theory. The STDChallenge Benchmark facilitates our study by aligning human and machine performances in a complex visual task designed to explore tracking strategies under spatio-temporal discontinuity, Our developed CPD Motion Model has not only achieved SOTA results in this task but has also enhanced the behavioral similarities between humans and machines, substantiating CPD theory and illuminating the potent visual search abilities of humans under constraints of limited information. This work underscores the significance of human- -ike modeling in enhancing tracking algorithms.
However, must be cautious about the potential misuse of this technology, particularly in surveil- lance applications, which could have serious ethical implications. Similarly, the development of highly human-like algorithms has already sparked some concerns, similar those associated with large language models (LLMs). Despite these challenges, the technology holds crucial potential for beneficial applications in fields such as sports, autonomous driving, roboties, and broader machine vision contexts in the real world. To promote responsible usage and further research into these positive applications, plan make our code and data publicly available.
Limitation. CPDTrack focuses on the STDChallenge but neglects performance in simpler scenarios, leading to its underperformance on some datasets compared to SOTA trackers, shown in Tab. 7 In the future, we plan to further improve it to adapt to various environments. The human-like modeling approach adds overhead, making CPDTrack less immediately applicable to real-world applications as it currently stands. STDCChallenge in the STDDCallenge Benchmark still shows a long-tail distribution, meaning most sequences contain fewer STDChallenge. This is still far from the real world, and how to increase data under specific challenges remains an open question.
Generalization of CPDTrack. We believe that many simple scenarios in Single Object Tracking (SOT) are based on laboratory settings and even possess characteristics of toy scenarios (for example, tracking smoothly moving targets within very short time) to some extent. In contrast, CPDTrack extends much further into the real world:
Local trackers introduce stronger priors, which enhance their performance in STT scenarios. There- fore, STDChallenge is fatal for them [7] however, CPDTrack effectively addresses this challenge.
Setup of the STDChallenge Benchmark is that any sequence containing at least one absent or shotcut considered include the STDChallenge. is evident that the S7DChallenge Benchmark already includes many relatively simple video environments, as shown in Fig. (1).
"Experiments indicate that humans perform worse than SOTA trackers in simple scenes, likely due to the precise focus of local trackers. While a global perspective, used by humans and CPDTrack, offers comprehensive view, it may not align with the dataset setup (e.g., 'lion's tail' or 'helicopter tail rotor'; see Fig. 7
However, in complex scenarios, human capabilities to track moving targets surpass those of trackers, a trait also inherited to some extent by the human-like modeling of CPDTrack.
Computation overhead of CPDTrack. We acknowledge that compared to some mainstream trackers, CPDTrack's 's human-like modeling causes some additional computational costs. However, in our experimental setup, CPDTrack achieves 20-30 fps, which already meets the requirements for real-time performance.
Moreover, the main contribution of our paper is to validate the effectiveness of numan-like modeling through CPDTrrack, and the experimental conclusions have affirmed the motivation behind our work. While there are computational costs, these do not overshadow the contributions of CPDTrack. We greatly appreciate your questions and will continue explore improvements in future work.
Table 7: Comparison with state-of-the the-art art methods on additional benchmarks, where is AUC scores on TNL2k and LaSOText, and AO scores on GOT- We add a symbol over GOT-10k to indicate that the corresponding models are only trained with the GOT-10k training set.
Fi
ECOW7
O4ean]
ATOM[S7]
DiMPEN0
TransT
02T
Teq
CPDTrack
200
29.5
32.6
38.4
40.1
44.7
30.7
54.1
5499
50.4
LaSOTea
37.6
39,2
47.4
49.5
42.0
GOT-11KK*
34.8
58.6
61.1
71.0
74.7
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Justification: The main claims in the abstract and introduction accurately summarize the key contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed in detail in Sec. G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition machine may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed machines and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

For Gaussian modeling of central vision, we base our correct derivation on results published papers, more details in Sec. 3.1
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a detailed description of the model architecture in Sec. and
Sec. D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new machine, the paper should make clear how reproduce that machine.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets used in this paper are open-source datasets, available through the provided references, We have not yet released the code, but the details of building the model are described in Sec. 3 and Sec. D.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
quuidees/odeSubmissii.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, inediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide a detailed description of the implementation details in Sec. D.I
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We have explained all the statistical results in Sec. F.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in Sec. D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////eerrrpp.cpp////tt/ttttt
Answer: [Yes]
Justification: We carefully review the NeurIPS Code of Ethics and ensure that the research aligns with it in all aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of this paper in Sec. G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic machine for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper focuses on the video tracking task, which does not pose such high
risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used the paper, properly credited and are the license and of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit the creators or original owners of all assets used in the paper, and cite the relevant papers for each asset. Human data were also released with the experimenter's permission.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and of service of that source should be provided.
32

---

If assets are released. the license, copyright information, and of use in the package should be provided. For popular datasets, ppaeessithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help deine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not yet released new assets proposed in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We explained this before we recruited Human Subjects, and more details can found the E.2
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
More details in Sec.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "wHwnFzyZ8f", "reviewer": "Reviewer_x2Xs", "summary": "- The authors generalize the visual tracking problem as human dynamic visual ability task, and propose a new benchmark named STDChallenge to evaluate the visual tracking algorithms. This new benchmark includes challenging scenarios with spatio-temporal discontinuity, where previous short-term tracking oriented algorithms fail to address. They also propose a new tracking algorithm oriented to the new task where it mimics the human visual search behavior, and evalute its effectiveness on multiple benchmarks.", "questions": "Please refer to the questions in the weaknesses section.", "limitations": "The authors included a separate discussions and limitations section in their paper, with adequate explanations and descriptions on the weaknesses and possible future directions.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- A new large-scale benchmark is always a welcome addition to the computer vision and machine learning community, where the new generalized object tracking task proposed by the authors is more challenging in some aspects.\n\n- The authors evaluate multiple state-of-the-art tracking algorithms on their proposed dataset, verifying the need for the proposed dataset and providing some insights for solving the new task.\n\n- A new tracking algorithm CPDTrack oriented for the proposed task is provided, showing some new directions for solving the visual tracking problem under spatio-temporal discontinuity.", "weaknesses": "- Comparision with VideoCube [3] dataset\n\nSince the proposed dataset includes video sequences with spatio-temporal discontinuities, it bears large similarities with the VideoCube dataset proposed in [3], where it includes sequences that require global search and target recovery after disappearances. In what aspects do the proposed dataset differ from VideoCube? (i.e. sequence length, object category, discontinuity etc.)\n\n- Proposed CPDTrack algorithm\n\nThe proposed CPDTrack utilizes global and local search simultaneously, which is implemented by using two search images cropped at different search areas of large and small sizes. Although the authors provide an explanation that the proposed method was inspired by cognitive neuroscience, this approach was widely used in previous visual tracking algorithms.\n\n  - [a] Effective Local and Global Search for Fast Long-term Tracking, TPAMI 2022\n  - [b] Learning Regression and Verification Networks for Robust Long-term Tracking, IJCV 2021\n  - [c] ‘Skimming-Perusal’ Tracking: A Framework for Real-Time and Robust\nLong-term Tracking, ICCV 2019\n\nHow does the proposed method differ from these algorithms? Also, these papers should be included in the references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We will include the new analyses below in the revision. We provide responses to the specific points below:\n\n**Differences with VideoCube:**\nVideoCube is a benchmark specifically designed for the GIT task, consisting of 500 sequences characterized by frequent shotcut and extra-long sequences. Unlike VideoCube, the STDChallenge Benchmark is aimed at integrating LTT and GIT, providing a more comprehensive video environment for the study of spatio-temporal discontinuities. Derived from multiple large datasets, the STDChallenge Benchmark not only addresses the bias of single datasets—a focal point in AI research—but also exhibits the following characteristics compared to VideoCube:\n- **Data volume:** The test set of STDChallenge Benchmark contains 252 sequences, significantly more than the 50 sequences in VideoCube.\n- **More rational division of challenge attributes:** Referencing [d], we have standardized the challenge attributes across sequences from different datasets, marking the sequences in various datasets with the same set of challenge attributes. Furthermore, we have quantified the difficulty of the STDChallenge across sequences.\n- **More diverse dataset distribution:** As shown in Fig.8 in the paper and Fig.R1 in the supplementary pdf, the greater volume of data allows the STDChallenge Benchmark to include a wider variety of sequences with different STD; the length distribution is also broader. This facilitates a more comprehensive evaluation of trackers.\n---\n\n**Differences with local-global trackers:**\nWe discuss these works in the related work section and Fig. 2 of the paper. These algorithms are referred to as local-global trackers in the paper, characterized by a \"switching\" module that decides whether to switch to global re-detection based on the performance of local trackers. We believe that this is fundamentally different from CPDTrack:\nThe key issue in the design of local-global trackers is how to devise a switching strategy between local tracking and global re-detection. In existing algorithms, the decision to switch from local tracking to global re-detection is still entirely determined by the local tracking predictions. This means that when making the switching decision, information outside the local search area is still ignored. If the actual target is not within the local search area, it increases the risk of the algorithm mistakenly identifying a distractor as the target instead of activating the global re-detector. We analyzed the three papers you referenced:\n- [a] uses the score from a target verifier within the local search module to decide whether to switch strategies;\n- [b] employs a verification network with local view to identify the target from the detected candidates. If the target disappears, the learning based switching scheme determines whether to trigger the global search mode.\n- [c] uses a verifier in the 'perusal' module to judge the confidence score of the target's presence, thereby deciding whether to switch strategies.\n\nCPDTrack, on the other hand, possesses both local and global perspectives, mitigating the risk of drifting to distractors in the local area in STDChallenge, and enhancing the tracker's visual search capabilities.\n\nThese three excellent papers are significant in the evolution of visual trackers; therefore, we will include a discussion of these works in the related work section.\n\n---\n[a] Zhao, H., Yan, B., Wang, D., Qian, X., Yang, X., & Lu, H. (2022). Effective local and global search for fast long-term tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 460-474.\n\n[b] Zhang, Y., Wang, L., Wang, D., Qi, J., & Lu, H. (2021). Learning regression and verification networks for robust long-term tracking. International Journal of Computer Vision, 129(9), 2536-2547.\n\n[c] Yan, B., Zhao, H., Wang, D., Lu, H., & Yang, X. (2019). 'skimming-perusal'tracking: A framework for real-time and robust long-term tracking. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 2385-2393).\n\n[d] Hu, S., Zhao, X., & Huang, K. (2024). SOTVerse: A user-defined task space of single object tracking. International Journal of Computer Vision, 132(3), 872-930.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0ecB17ovz", "reviewer": "Reviewer_17n1", "summary": "This paper presents a novel approach to visual object tracking by drawing inspiration from the Central-Peripheral Dichotomy (CPD) theory. The proposed CPDTrack aims to improve tracking performance by emulating human visual search mechanisms, particularly under challenging scenarios involving spatio-temporal discontinuities (STDChallenge). The paper introduces a new benchmark, STDChallenge, and demonstrates the effectiveness of CPDTrack through extensive experiments.", "questions": "See the weaknesses.", "limitations": "This paper does not include a dedicated section discussing the limitations of their approach.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of incorporating human visual search mechanisms into object tracking is both innovative and promising. The use of the CPD theory to separate central and peripheral vision for enhanced tracking performance is well-motivated and addresses a significant gap in existing tracking methodologies.\n2. The creation of the STDChallenge Benchmark is a significant contribution. \n3. The experimental results are impressive, showing that CPDTrack achieves state-of-the-art performance in the STDChallenge Benchmark.", "weaknesses": "1. CPDTrack is designed specifically to address the STDChallenge, and as noted in the paper, it may not perform as well in simpler or different scenarios. Despite the explanations provided, this lack of generality is a significant concern.\n2. The paper states that the tracking speed on an A5000 GPU is 23fps, is this because the introduced modules are particularly time-consuming? This represents a considerable demand for computational resources, which could be a factor to consider in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We will include the new analyses below in the revision. We provide responses to the specific points below:\n\nAs described in the paper, compared to existing algorithms, CPDTrack's advantage lies in its more robust performance in STDChallenge and its behavior more closely resembling that of humans. This also serves to validate the accuracy of the CPD theory to some extent.\n\n---\n\n**Generalization of CPDTrack:**\n\nWe believe that many simple scenarios in Single Object Tracking (SOT) are based on laboratory settings and even possess characteristics of toy scenarios (for example, tracking smoothly moving targets within very short time) to some extent. In contrast, CPDTrack extends much further into the real world:\n- Local trackers introduce stronger priors, which enhance their performance in STT scenarios. Therefore, STDChallenge is fatal for them [a,b]; however, CPDTrack effectively addresses this challenge.\n- Setup of the STDChallenge Benchmark is that any sequence containing at least one \"disappearance-reappearance\" or \"shotcut\" is considered to include the STDChallenge. It is evident that the STDChallenge Benchmark already includes many relatively simple video environments, as shown in Fig. 10 (1) in the paper and Fig. R2 in the supplementary PDF.\n- Additionally, experiments have shown that humans perform worse than state-of-the-art (SOTA) trackers in simple scenes. This is likely because local trackers can more precisely focus on the target due to their local field of view. Moreover, the inclusion of a global perspective allows humans and CPDTrack to have a more comprehensive understanding of the target, which, although not incorrect, differs from the dataset's setup, such as \"lion's tail\" or \"the tail rotor of a helicopter\", refer to Fig. R3 in the supplementary PDF. However, in complex scenarios, human capabilities to track moving targets surpass those of trackers, a trait also inherited to some extent by the human-like modeling of CPDTrack.\n---\n\n**Computation overhead of CPDTrack:**\n\nWe acknowledge that compared to some mainstream trackers, CPDTrack's human-like modeling  causes some additional computational costs. However, in our experimental setup, CPDTrack achieves 20-30 fps, which already meets the requirements for real-time performance.\n\nMoreover, the main contribution of our paper is to validate the effectiveness of human-like modeling through CPDTrack, and the experimental conclusions have affirmed the motivation behind our work. While there are computational costs, these do not overshadow the contributions of CPDTrack. We greatly appreciate your questions and will continue to explore improvements in future work.\n\n---\n\n**Limitations:**\nOverall, our work has the following shortcomings, which we will further address in future research:\n\n1. **Specialization on STDChallenge:** CPDTrack focuses on the STDChallenge, resulting in suboptimal performance in some idealized simple scenarios where it underperforms compared to state-of-the-art (SOTA) trackers on certain datasets.\n2. **Increased Computational Overhead:** The human-like modeling approach adds overhead, making CPDTrack less immediately applicable to real-world applications as it currently stands.\n3. **Long-Tail Distribution in STDChallenge Benchmark:** The STDChallenge within the STDChallenge Benchmark still exhibits a long-tail distribution, which is somewhat distant from real-world conditions. How to increase data under specific challenges remains an open problem that needs addressing to bridge this gap.\n\n---\n\n[a] Hu, S., Zhao, X., & Huang, K. (2024). SOTVerse: A user-defined task space of single object tracking. International Journal of Computer Vision, 132(3), 872-930.\n\n[b] Fan, H., Yang, F., Chu, P., Lin, Y., Yuan, L., & Ling, H. (2021). Tracklinic: Diagnosis of challenge factors in visual tracking. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 970-979).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eprunCyoA6", "reviewer": "Reviewer_16gz", "summary": "The authors touch upon a very important topic in visual tracking. They try to build upon the Central-Peripheral Dichotomy (CPD) theory that talks about how humans use visual information to track targets in complex environments. To this end, they propose a tracker named CPDTrack and STDChallenge Benchmark (for spatio-temporal discontinuities). They claim that their CPDTrack achieves SOTA performance on STDChallenge, aligns with human behavior and generalizes across various benchmarks.", "questions": "There is another similar algorithm called DorsalNet. Have the authors considered the similarity/differences with it?\nFor the citation of PathTracker, the paper cited is InT which is a solution to the PathTracker challenge. For completeness, the challenge was introduced in https://arxiv.org/abs/2110.02772 which the authors should consider citing. \nIn fig 5 the authors say that both CPDTrack and humans are robust to occlusions, environment changes and absence of target. InT has similar claims; have the authors tested their benchmark with the TransT+InT proposed in the paper ``Tracking without re-recognition”?", "limitations": "The gains are small compared to other networks on the STDChallenge benchmark. Have authors made an attempt to check if they are statistically significant?\nAbsence of IRB could raise ethical concerns which is why authors should work to mitigate it. \nIn appendix E4, the authors talk about mouse movements. Are the authors tracking mouse movements from the participants’ computers? If so, that certainly calls for IRB approvals due to additional security and privacy concerns. \nFigure 11 is not clear enough as to where/how the humans fit. The authors should make clarifications about what they mean by the heatmap in terms of errors. \nThe model presented in fig. 3 is rather convoluted. If possible, the authors should clarify it as well. \nThe authors mention releasing code, but i didn’t find anything with the paper. The authors should make sure the code is released. If possible, they should also provide scripts/links to download the public datasets they used for better reproducibility.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The authors clearly highlight the importance of aligning visual tracking in both humans and machines, and to that respect also conduct a human study. \nThey also cover the related works decently well to highlight the previous works in the field. \nThe distinction they made between LTT and GIT, and the attempt at doing a Visual Turing Test makes the paper stand out. \nThe absolute results presented on the STDChallenge also show performance gains compared to the other networks (albeit small).", "weaknesses": "The authors do not have IRB which is one of the basic requirements for working with human subjects. \nI also did not see any strong statistical testing for human vs machine responses. Seems like to authors selected responses from all human participants. If not, i don’t see any methods for dropping the out of distribution responses. \nThe authors also conducted 5 experiments with human subjects, but i didn’t find a clear link between the selected experiments for humans and the STDChallenge benchmark. \nThere is also no justification for why removing information query was treated as baseline in ablation studies. Please justify. \nFor the STDChallenge benchmark, i didn’t find a justification as to why only the said challenges were included in the benchmark. The authors should clarify that if possible. \nHow easy/hard is it to include newer challenges in the benchmark?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We will release our code upon acceptance and include the new analyses below in the revision. Here are our responses to the following points:\n\n**IRB**: Previous studies [a] have demonstrated that such experiments only involve interaction between human subjects and computer systems (screen, mouse), posing no risks throughout the experiment, thus not requiring IRB review. Our experimental paradigm is similar to theirs. Under our consideration, before the experiment, participants were thoroughly briefed and confirmed their full understanding of the process by signing comprehensive instructions, ensuring clarity on the experimental procedures. We appreciate your reminder; we have communicated with our institution's IRB, submitted the necessary materials, and obtained IRB approval.\n\n**Statistical Testing**: Referencing [a], we focused on ensuring the accuracy and validity of data collected from a limited number of subjects to accurately reflect human visual capabilities. Consequently, we implemented measures to prevent out-of-distribution responses, enabling the use of all responses. 1) We did not use crowdsourcing platforms; instead, the Visual Turing Test were conducted in a laboratory environment under supervision. 2) Before the experiment, we assessed participants' cognitive and perceptual levels to ensure normalcy, and allowed them to familiarize with the equipment through basic exercises. 3) During the experiment, participants were permitted to pause the experiment a certain number of times voluntarily to adjust their state. 4) In cases where the mouse cursor moved off-screen, we retained the results from the previous frame until the cursor returned within the screen area.\n\n**Visual Turing Test**: In the paper, we provide a detailed description of the Visual Turing Test process, which aims to study the differential capabilities between algorithms and humans under the STDChallenge. You can refer to sec. 4 and E for more details. We aligned the experimental procedures and metric calculations between humans and algorithms, allowing for comparison within the evaluation pipeline.\n\n**The Setting of the Baseline**: CPDTrack aims to study the effectiveness of CPD theory in the design of artificial neural networks (ANN). Moreover, from a model design perspective, the CPD motion model and information query are decoupled, allowing for stepwise ablation experiments on CPDTrack. 1) We examine the performance of the CPD Motion Model itself. This is achieved by comparing CPD-only central vision-only peripheral vision. 2) We center around the CPD Motion Model and enhance the encoder-selector-decoder framework by incorporating information query, validating the effectiveness of cognitive feedback control in ANN. 3) We adjust the CPD Motion Model part in the ANN and align it with traditional tracking networks to demonstrate the effectiveness of our designed CPDTrack.\n\n**New Challenges**: As described in our Introduction and Related Work, STDChallenge is recognized by many studies for distinguishing STT from more challenging tasks like LTT and GIT, which are considered to represent more realistic video environments. In our setup, a sequence with a shotcut or disappearance-reappearance is deemed to include STDChallenge, naturally integrating challenges found in new real-world scenarios. Thus, STDChallenge not only incorporates but also enhances other challenging attributes to some extent, as depicted in Fig8(c), such as disappearance-reappearance is often associated with occlusions or out-of-view. Formally, STDChallenge does not exclude other tasks (such as pathtracker), but we advocate integrating STDChallenge with real-world scenarios to avoid ill-posed issues (such as shotcut or occlusion confusing same-looking blocks in pathtracker).\n\n**Compare with DorsalNet**: We identified the most related papers using networks named DorsalNet, which we believe fundamentally differ from our CPDTrack [b]: Proposes a new hypothesis on dorsal visual pathway neurons' function during self-motion, using a 3D ResNet model to explain non-human primate neuron activity patterns and their selectivity to motion stimuli, enhancing understanding of animal behavior and self-localization in dynamic environments. Similarities: Both approaches are based on cognitive science theories to construct models, and focus on research involving moving targets. Differences: 1) Our model is based on established cognitive science theory (CPD theory), not assumptions. 2) Instead of studying humans or neurons, our goal is to enhance visual object tracking algorithms using human capabilities and explore differences in behavior (Science4AI). 3) Our research focuses on tracking other moving targets, not self-localization.\n\n**Compare with PathTracker**: Thank you for your reminder. We really learned a lot from pathtracker, and will include them in the revised version. Comparison is in the supplementary pdf Tab. R1.\n\n**Figure 11** is designed to show the error consistency among different decision-makers (machines/humans) in the STDChallenge, calculation methods can be referenced from [a]. This represents the degree of behavioral similarity between different decision-makers. For each sequence, we calculate the error consistency on each frame, and then average these across all sequences to obtain the overall error consistency among the decision-makers.\n\n**Figure 3** is divided into two parts. The upper part shows the relationship between visual acuity and eccentricity and the \"encoder-selector-decoder\" human vision framework found in cognitive neuroscience. The lower part illustrates our proposed CPDTrack model. The grey arrows running through both parts highlight the correspondence between the two parts.\n\n[a] Partial success in closing the gap between human and machine vision. Neurips 2021\n\n[b] Your head is there to move you around: Goal-driven models of the primate dorsal pathway. Neurips 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their efforts and invaluable suggestions. 

This work is inspired by the cognitive science theory of the central-peripheral dichotomy (CPD) and introduces a tracker, CPDTrack, designed to address the STDChallenge. Its effectiveness and similarity to human behavior have been validated through benchmark tests and the Visual Turing Test. We are also very grateful that all reviewers have given our work high praise (accept *1, weak accept *1, borderline accept *1), which encourages us to continue working courageously within the 'Science for AI' research paradigms.

Thank you for the valuable reviews pointing out that 1) we focus on a challenging yet valuable issue (the problem of spatio-temporal discontinuity in videos) and acknowledge the proposed STDChallenge Benchmark (x2Xs, 17n1), 2) it is highly novel to integrating CPD theory into visual object tracking and affirm the performance of CPDTrack (x2Xs, 17n1, 16gz). 3) our evaluation on the STDChallenge Benchmark is comprehensive (x2Xs, 16gz) and note that the Visual Turing Test is particularly noteworthy (16gz). Prompted by the insightful reviews, we mainly present the following additional experimental results and analyses for the common questions:

- Reviewer x2Xs recognizes the performance and value of CPDTrack. Your concerns primarily focus on the novelty aspects of CPDTrack and the STDChallenge Benchmark. Some ambiguous descriptions may have led to confusion between our work and others. We will clarify these distinctions using figures in the paper, supplementary figures, and references, highlighting the fundamental differences between our proposed method and existing approaches.
- Reviewer 17n1 affirmed the novelty of several aspects of our work. Your concerns mainly focus on the generalizability and costs associated with CPDTrack. We will provide a comprehensive analysis of CPDTrack's advantages and potential limitations. Moreover, we believe that CPDTrack's extended applicability in real-world scenarios worth the associated costs.
- Reviewer 16gz affirmed the novelty and performance of our work. Your concerns primarily focus on the evaluation experiments conducted on the STDChallenge Benchmark. We will further explain the scalability of the STDChallenge Benchmark and provide additional details on the setup of the Visual Turing Test.

Thank you for your valuable suggestions. We will address each of your concerns individually in our responses. Additionally, we have included a supplementary 1-page PDF, which presents some attributes of the STDChallenge Benchmark and visualization in graphical form, aiming to provide you with a clear and intuitive understanding of the details of our work. We also look forward to discussing further with the reviewers during discussion. Should you have any more questions at that time, please feel free to engage with us. We hope to improve our work with your assistance.

Meta-review: The paper introduces a novel visual tracking approach inspired by the Central-Peripheral Dichotomy (CPD) theory, along with a new benchmark, STDChallenge, designed to evaluate tracking algorithms under spatio-temporal discontinuities. The reviewers all agree that the introduction of this benchmark as a significant contribution, highlighting its potential to address gaps in existing visual tracking methodologies. The proposed CPDTrack algorithm is praised for aligning with human visual search behavior and achieving promising results on the new benchmark. 

However, concerns were raised about the originality and generalizability of CPDTrack, with some reviewers noting similarities to existing methods and its performance limitations in simpler scenarios. Overall, the paper is considered technically solid and impactful. Please consider incorporating reviewers' suggestions in the revision.
Decision: Accept (poster)