Nearing or Surpassing: Overall Evaluation of Human-Machine Dynamic Vision Ability
Authors: Shiyu Hu
Xin Zhao
Yipei Wang
Yanhu Shan
Kaiqi Huang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Dynamic Visual Ability
Machine Intelligence Evaluation
Single Object Tracking

Abstract: Dynamic visual ability (DVA), a fundamental function of the human visual system, has been successfully modeled by many computer vision tasks in recent decades. However, the prosperity developments mainly concentrate on using deep neural networks (DNN) to simulate the human DVA system, but evaluation systems still simply compare performance between machines, making it tough to determine how far the gap is between humans and machines in dynamic vision tasks. In fact, neglecting this issue not only makes it hard to determine the correctness of current research routes, but also cannot truly measure the DVA intelligence of machines. To answer the question, this work designs a comprehensive evaluation system based on the 3E paradigm -- we carefully pick 87 videos from various dimensions to construct the environment, confirming it can cover both perceptual and cognitive components of DVA; select 20 representative machines and 15 human subjects to form the task executors, ensuring that different model structures can help us observe the effectiveness of research development; and finally quantify their DVA with a strict evaluation process. Based on detailed experimental analyses, we first determine that the current algorithm research route has effectively shortened the gap. Besides, we further summarize the weaknesses of different executors, and design a human-machine cooperation mechanism with superhuman performance. In summary, the contributions include: (1) Quantifying the DVA of humans and machines, (2) proposing a new view to evaluate DVA intelligence based on the human-machine comparison, and (3) providing a possibility of human-machine cooperation. The datasets, toolkits, codes, and evaluation metrics will be open-sourced to help researchers develop intelligent research on dynamic vision tasks.

Under review as a conference paper at ICLR 2023
NEARING OR SURPASSING: OVERALL EVALUATION OF HUMAN-MACHINE DYNAMIC VISION ABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Dynamic visual ability (DVA), fundamental function of the human visual system, has been successfully modeled by many computer vision tasks in recent decades. However, the prosperity developments mainly concentrate on using deep neural networks (DNN) to simulate the human DVA system, but evaluation frameworks still simply compare performance between machines, making tough to determine how far the gap is between humans and machines in dynamic vision tasks. In fact, neglecting this issue not only makes it hard to determine the correctness of current research routes, but also cannot truly measure the DVA intelligence of machines. To answer the question, this work designs a comprehensive evaluation framework based on the 3E paradigm to construct the environment, confirming it can cover both perceptual and cognitive components of DVA; select 20 representative machines and 15 human subjects to form the task executors, ensuring that different model structures can help us observe the effectiveness of research development; and propose multiple evaluation indicators to quantify their DVA. Based on detailed experimental analyses, we first determine that the current algorithm research route has effectively shortened the gap. Besides, we further summarize the weaknesses of different executors, and design a human-maching cooperation mechanism with superhuman performance. In summary, the contributions include: (1) Quantifying the DVA of humans and machines, (2) proposing a new view to evaluate DVA intelligence based on the numan-maching comparison, and (3) providing possibility of cooperation. The 87 sequences with frame-level Human--aching comparison and cooperation results, the toolbox for recording real-time human performance, codes for sustaining various evaluation metrics, and evaluation reports for 20 representative models will be open-sourced help researchers develop intelligent research on dynamic vision tasks.
carefully pick 87 videos from various dimensions
INTRODUCTION
Research on visual abilities can be dated back the last century (Hubel & Wiesel (9559 1962), ).
Neuroscientists divide the human visual system into two categories, namely the static vision ability (SVA) to perceive the details of static objects (Chan & Courtney (1996)), and the dynamic vision ability (DVA) to track moving objects (JW et al. (1962)). These two visual abilities are essential in our daily life (Land & McLeod (000);; Beals al. (197I: Burg 1966); Kohl al. (199T), and have been modeled by series of computer vision tasks. Recently, with the growth of dataset scale and the abundance of computing resources, most data-driven algorithms achieve higher and higher scores in experimental environments, and are widely employed in various scenarios (Dankert et al. (2009);
Weisshrod et al. (2013); Wei & Kording (2018)).
However, some bad cases hidden under the prosperity development challenge the state-of-the-an (SOTA) algorithms. For example, visual models usually decrease their perception when encountering unknown-category targets under special illumination conditions (e.g., vision-based autonomous vehicle crashes into a large truck at night). This shortcoming is far from humans' powerful visual abilities and may cause safety hazards, causing us rethink with the support of massive datasets and powerful computing resources, why can't 't SOTA models achieve a similar visual ability to humans? Do existing evaluation methods actually measure the visual intelligence of machines?

---

Under review as a conference paper at ICLR 2023
bot
dotan
Human
Homan machins
Cooparadoon
bounding Sot
center dittance
Machine
vorioun
EXECUTOR
ENVIRONMENT
EVALUATION
elatanets
(a) Existing research
(b) Framework in our research
Figure 1: Comparison of the evaluation frameworks in dynamic vision abilities. (a) Existing research on DVA evaluation for human and machine is entirely separate. (b) We follow 3E paradigm (Hu et al. (2022a) to design an overall evaluation framework human-maching DVA.
Compared to thriving algorithm research, evaluation methods have received far less attention. Tradi- tional evaluations mainly focus on the performance comparison between machines (Deng et al. (2009); Russakovsky al. 22055), causing the goal become downward (i.e., a method that exceeds all others is considered excellent). In fact, these evaluation mechanisms actually measure the machine's s performance rather than intelligence. When we refer to machine intelligence, natural association is Turing test (Turing] (2009). which indicates that machine intelligence evaluation requires human participation, and has gradually attracted scholars to propose a series of essential works in decision- making problems (e.g., AlphaGo (Silver et al. (2017) and DeepStack (Brown & Sandholm (2018) Using humans as a reference, the goal will become upward exemplary machines must gradually move closer to human capabilities. In other words, exploring the gap between humans and machines is very crucial for machine intelligence evaluation.
Based this idea, recent works have introduced humans to several vision tasks, using human baseline to measure machine intelligence. Some researchers endeavor to analyze the gap between machines and humans in image classification (Geirhos et al. (2018) 2020 (02T). p). others investigate the attention areas in static images to explore visual selectivity (Langlois el al. (2021)). Regrettably, their research scope mainly restricted to static vision tasks, while neuroscience and cognitive psychology studies have shown that the correlation between SVA and DVA is naturally low (Long & Penn (1987). On the other hand, existing research on DVA evaluation for human and machine is entirely separate. As shown in Figure [ (a), neuroscientists use toy environments to measure human DVA (Pylyshyn & Storm 1988), while computer scientists evaluate machines through large-scale datasets (Fan et al. (2020) Huang et al. (202T)- neither the environment nor the evaluation mechanism are compatible, causing the comparison of human-maching machine DVA to be impossible. Therefore, an intuitive question is, how to compare humans and machines in dynamic vision ability? To answer this question, we should first select a suitable computer vision task to represent human DVA, then design a evaluation framework accomplish human- machhne DVA measurement and comparison.
Compared with other tasks like multi-object tracking (MOT) (Caparrone et al. (2019) and video object detection (VOD) (Wang et al. 2018)), ), single object tracking (SOT, i.e., locates user-specified moving target in a video) (Wu et al. (2015) ) is a category-independent task with no constraint on motion continuity, scene change, or object category, and can be regarded as the closest task to represent human DVA (Appendix Ib. Due to the human-like task definition of SOT, excellent task executors should not only keep tracking the moving target (perception) but also re-locate the target when its position is mutated (cognition) (Appendix A.2 From above analyses, we select SOT as the representative task, and follow 3E paradigm (Hu al. (2022a)) to design an overall evaluation framework (Figure T. The technical difficulties and our contributions are as follows.
Experimental environment construction (Section 2.1 The first difficulty in environment con- struction is compatibility. Choosing high-contrast toy environment used in classical neuroscience work too simple to evaluate DVA accurately. On the other hand, human psychophysical experiments expensive and time-consuming, and cannot be assessed on large-scale datasets like machines. Based on this, the second difficulty is representativity. With the limitation of dataset scale, the environment should not only fully represent the characteristics and difficulties of DVA, but also provide a graded experimental setup for subsequent analyses. As shown in Table [ to entirely reflect the task characteristics and thoroughly compare the human-machine DVA, choose 87 videos
2

---

Under review as a conference paper at ICLR 2023
with 244,455 frames to construct the environment. All videos are carefully picked from various dimensions, ensuring the environment can cover the perceptual and cognitive components of DVA.
Experimental executor selection and record (Section 2.2 As an interdisciplinary experiment, the task executors involve humans and machines. The difficulty for human subjects is accurately recording their performance in dynamic vision tasks and ensuring that the recorded results can be quantified to support subsequent evaluations. Through detailed comparison and analysis, we select the mouse-based method and design a toolbox for human subjects to record their frame-level performance in real-time. For machines, since we aim to study the DVA gap between humans and machines, there are two concerns model selection: trends (i.e., selecting various algorithms) and gap (i.e., focusing on the upper bound of algorithmms). Thus, both classic and SOTA methods with different architectures are selected to explore whether the research route has effectively shortened the gap. Consequently, task executors included 20 representative algorithms and 15 subjects (all subjects are computer vision researchers aged 20-30). All experiments are managed in a strict process.
Evaluation metrics design (Section 2.3 Traditional indicators usually select the positional rela- tionship (e.g., intersection over union (IoU) and center distance) between predicted result pt and ground-truth de to accomplish calculation. However, the tracking result of humans is point, which cannot be calculated by IoU (Zheng et al. (2020); Rezatofighi et al. (2019)) Thus, we use two center points (i.e., the target center Cp predicted by executors, and the center point Cg of ground-truth) to design three granularities (frame-level, sequence-eeel and group-level) for evaluation. Especially, consider the influence of sequence length and revise the group-level indicators to generate more appropriate evaluation. Experimental analyses have verified the validity of proposed metrics.
Based on above steps, we quantify the DVA of executors and conduct detailed comparisons (Section 3) By analyzing and summarizing the results, we find that: (1) Synthetically, humans have stronger dynamic vision ability than machines. Experimental results demonstrate that humans outperform machines when both perceptual and cognitive abilities are required. Particularly, when we mainly examine the perceptual ability in dynamic vision tasks, we also notice that the gap of human- machine perceptual ability is closing, (2) Human-machine cooperation is possible and effective in dynamic vision tasks. Based on numan-machin differences in perception and cognition, we design a set of cooperative experiments. Results indicate that the SOTA machines with human- machine cooperation can achieve even better performance than humans in some cases.
In summary, this work starts with designing more intelligent evaluation framework, measures the DVA gap between humans and machines, determines the effectiveness of current algorithm research routes, finds the weaknesses of different exccutors, and notes the possibility of human-machine cooperation. We will open-source the toolbox, code, and metrics used in the experiments to help researchers further develop intelligent research dynamic vision tasks.
2 METHODS
2.1 ENVIRONMENT: DATASETS
For DVA evaluation, researchers initially require subjects to observe moving objects on a high-contrast background. For example, an early work displays some moving plus (+) signs on the screen(Pylyshyn & Storm (1988)). Recently, researchers evaluate the DVA of athletes based on a software named DY (Qevvedo al. (2012). which includes various stimuli with different color and motion trajectories. Although above works have developed from simple designs to computer programs, they all have limited target categories (e.g., specific symbols), simple backgrounds (e.g., colors or some static photographs), and limited motion patterns (e.g., lateral, vertical, and oblique).
Unlike above toy environments that track simple symbols in a high-contrast background, our environ- ment fully models the application scenes and designs various data combinations for ability analyses (e.g., perception, cognition, and robustness). Concretely, the environment should consider the diver- sity of video contents and includes the variation of video length (especially long sequences). Thus, we refer Verse (Hu et al. (2022a)) and select sequences from representative benchmarks (OTB (Wu et al. (2013 2015)). VOT series (Kristan et al. (2013) 2014 2015) 2016 2017 2018 2019)). GOT-1Ok (Huang et al. (202T). LaSOT (Fan et al. (0200). and VideoCube (Hu et al. (202b))) to form the experimental environment (Table and Appendix C.1
3

---

Under review as a conference paper at ICLR 2023
Table 1: Information on environment settings.
Characteristis
Task Settings
Target Abseut
Shot-
Ability
Group
Frames
cut
Short-ferm tracking
500-1,000
(Targel presents from beginning to end)
N
N
Perception
B
1,000-20000
Long-term tracking
Y
N
C
1,000-2,000
(Target may disappear and reappear in single shot)
Perception
5,000- TOUORD
Global instance tracking
and
E
1,000--2,000
Y
cognition
5,000-TDO000
(Target may disappear and reappear in mulltppe shots)
G
15,0000-000000
Challenging
Abnormal rabo
factors
Abtormal scale
Short-term tracking with
single frume
Aonermal Ellumination
Blur bounding-bor box
Perception and robusness
challenging factors (Target presents from beginning to end)
Challenging
factors
Drastic scale variation
ratio variation
N
M
500-1.000
Drustic illumination variation
between
Drastic clarity variation
consecutive
frumes
Low correlation cofficionl
2.2 EXECUTOR: MODELS, HUMAN SUBJECTS
As shown in Table 2 20 represent machines covering both classic and SOTA methods are selected:
2 CF-based trackers (Henriques et al. (2015); Danelljan et al.] (2017)) 10 SNN- methods
(Bertinetto et al. 2016 et (2018b); /hu al. (2018) ; et al. [201 8ay Yan et al. (2019) /Zhang
& Peng (2019) Guo et al. (2020) : Xu et al. [2020] Zhang & Peng (2020) Voigtlaender et al. (2020) ).
5 algorithms that combine CF and SNN (Danelljan et al. (2018); Bhat et al. (2019); Danelljan et al.
22200 Mayer et al. (02T)) ) and custom networks (Huang al. (2019); Bhat et al. (2020): Cui
et al. 2022)). Please refer to Appendix B for technical details.
For humans, start the measurement of human DVA based on the following concerns:
Organization, 87 videos with different durations, object classes, and scene categories are played at 25FPS to 15 subjects (aged between 20-30, all computer vision researchers) under high-quality experimental organization this organization known as small-N design (Smith & Little (2018))
and has been widely used in SVA experiments (Geirhos et al. (2018 2020 2021)).
Approval. We have obtained the approvals of all subjects. They all have signed an experimental statement (the template is provided by our IRB, which includes experiment purpose, procedure, risks and discomforts, costs, and confidentiality (Appendix C.2) before the experiment.
Process. Previous works mainly eye-tracker Hu al. (002bb) Xia et al. (021) mouse Geirhos et al. (2018) 2020: 202I) to record human visual ability. To select better measurement method, all subjects tried both mouse and eye-tracker, and they express that the mouse is better. Besides, we have compared these two devices from practical and theoretical views in Appendix C.3] and finally select the mouse- based method. Experiment includes four steps (Appendix C.4p: (1) Subjects check instruments; then adjust the seat height, sitting posture, and the distance to the screen. (2) 2 TEST videos occur sequentially in the screen center. Subjects observe and remember the target features in the first frame, then move the mouse to track the target in subsequent frames. (3) 17 FORMAL videos belonging to different groups are played sequentially (a rest time occurs between two videos to relieve visual fatigue); subjects should concentrate on the target and maintain the mouse's position. (4) Subjects fill in a questionnaire to self-evaluate their performance (Appendix C.3).
2.3 EVALUATION: METRICS
This work evaluates executors under the one-pass evaluation (OPE) mechanism, which initializes an executor in the first frame and continuously records the tracking results. For the (-th frame F, in sequence s; = {F1, Fg..... .., we support the predicted result is pt, the ground-trui truth is ge, and their center points Cp and eg used design indicators. Note that target absent is regarded as empty set (i.e., 9: = Φ) and excluded by the evaluation process. To accurately evaluate executors' performance, divide the evaluation dimensions into three granularities, as shown in Figure 2
4

---

Under review as a conference paper at ICLR 2023
Table 2: The performance (based on ()())) about human subjects and 20 representative models 1SNN-Simess Neural Network. CF-Correlation Filter. CNN-Convolutiona. Neural Network. Red, magenta and eyan represent the top-3 machines).
Executor
Aritciture
Characteristic
Subject_ Top
The best performance of subjects
0.891
Subject_Mean
The mean performance of subjects
0.853
Subject_Rotuan Bottom
The worst performance of subjects
0.801
MixFormer (Cui (2022)
Custom networks
Transformer based framework
0.766
KYS (Bhat al. (0201)
Custom networks
Scene information
0.528
GlobalTrack (Huang et al. (2019))
Custom networks
Zero cumulative error
0.645
Track Track (Mayer et ul. (221T)
SNN+CF
Target candidate association
0.718
SuPerDiMP [Danelljan
(020
SNN+CF
Probabilistic regression
0,701
PrDiMP Danellyan
2020)
SNN+CF
Probabilistic regression
0.683
DiMP Bhat et
2019
SNN+CF
Better discriminative ability
0.597
ATOM [Danclljan cf 2018)
SNN+CF
Combine SNN with CF
0.506
SiamRCNI (Voigtleender et
(0200)
SNN
Re-detection mechanism
0.748
Ocean (7hang & Peng
2020)
SNN
frce
0.635
SiamFC++ (Xu al. 20201
SNN
Ance
0.512
SiamCAR Guo
SNN
Anchor-fiee
0.480
SiamDW Zhang & Peng
2019
SNN
Deeper and wider backbone
0.558
SPLT Yan el al. 20199)
SNN
Local search and global search
0.610
[L SiamRPN++ al. 2018 8a
SNN
Deeper backbone
0.662
DaSiamRPN Zhu el a. 2018
SNN
Data augmentation
0.528
SiamRPX (Li al. 801
SNN
Region proposal network
0.495
SiamFC (Bertinetto
2016
SNN
Originator of 3NN-based trackers
0.285
ECO (Danelljan et al.
CNN+CF
Combine CNN with
0.377
KCF Henriques et al. 2015)
CF
Representative CF-based method
0.270
L1: Frame-level. Precision score (PRE) (Wu et al. 2015) in frame-level PLi equals the center distance de- Recently, normalized precision exclude the influence of target size and frame resolution. Trackers with a predicted center outside the ground-truth will add a penalty item de" (i.e., the shortest distance between center point Cp and the ground-truth edge). For trackers whose center point falls into the ground-truth, the center distance
(N-PRE) (Hu et al. (202bb) is proposed to
de equals the original precision de (i.e., d.P 0). Besides, the maximum value in frame F, is used to normalize the result and generates the final N-PRE score NP L1. as shown in Equation
PLL = de [lo, call2
NP N(d=))= max((d [i F.)) min([d] (4'" i F.))
d.' min( (d. F.))
de' d''= d
(1)
L2: Sequence-level. Precision score in sequence-level PL2(Os) defined as the proportion of frames whose center distance d, < 0g- To illustrate the performance under different thresholds, previous works (Wu et al.] (2055): Fan et al. (2020)) usually draw the statistical results based on different 0a into curve named precision plot. Since 0₂ 20 is wildly used to rank trackers, we define PL2 P (20) in following experiments. Similarly, draw statistical results based on different 0,' into a curve generates the normalized precision plot. However, we note that directly select 0j to rank executors may introduce human factors, thus we use the proportion of frames whose predicted center Cp successfully fall in the ground-truie -tuun rectangle gi (Flag=Y Y in Figure (L2)) as NPL2. The calculation listed in Equation where |-] is the cardinality.
PL2(As) = [.d]
(Fe : S Salll [ao] [[F, de s 20)]
(P [sc]
aa' 10. 10. NN12 [s.] [(F cp é g:ll
L3: Group-level. For a group of sequences G = {si, $2...., ,,, .t....), existing works usually use the mean value of all sequence-level results the group-level evaluation, illustrated in Equation
5

---

Under review as a conference paper at ICLR 2023
Example
(L1) Frame-level
Example
d:
d
wcrm (PAc]
prodiio (8-255)
srowe
Normalitdd prediton
(L2) Sequence-lever
PAS-O, N-PR6-0 Rager
PR6-2, M: FRE-0.01 fager
PR6-12, PRE-0.06 Rager
FRE-45, PR6-0.88 Tageri
PRE-28, NiPR5-0.14 Flager
PR6-86, PRE-0.17 Flagnk
PR6-36, PRE-0.2 Flagrt
L3 Group-level
Sequeece legth 2003
leaghh 1500
Sequence leagh $3500
Figure 2: Illustration of three granularities evaluiation process.
(Ls(8a)
[s:] (FF s PLs
Σ
[[Fa de S 20H1
[8:]
(Pu)(9)
[ a.' [0.
NP
Σ
[(F) 1. eall
FEE
It is worth noting that longer sequences challenge all executors in dynamic vision tasks humans demand higher concentration of attention, and algorithms should overcome the cumulative However, existing metrics all ignore the influence of the sequence length, causing us to propose new metrics for group-level evaluation, as illustrated in Equation 4
([s(O,) En.co :d. S 0aHl
PT Eo,col KE d. 20)]
Eorco 19:1
Enco [7,
(0,) Σneo (F, :.:.N) S0x')
NP Enee I(Fs c 0c)
(4)
E,,ec
Ex,ep
3 EXPERIMENTS
3.1 A COMPREHENSIVE COMPARISON OF HUMAN-MACHINE DYNAMIC VISION ABILITY
We first interest in the comprehensive performance of human-machine DVA. Subject_Top and Subject Bottom represent the best and worst subjects, and Subject_ denotes the average. Figure (1) shows the distribution of NP in all sequences. As sequence-level evaluation, box width represents stability, box position represents performance. Figure (II) illustrates the normalized precision plot weighted by sequence length and the NPW, gevoup-level evaluation) all executors. Clearly, humans score higher than most machines, indicating that the DVA of machines is still far from humans. Besides, the boxplot distribution shows that machines' performance is more unstable. It is worth noting that the SOTA methods have been comparable to Subject Bottom in most cases, indicating that the research routes do significantly close the gap with humans.
As tasks become more complicated (from short-term tracking to global instance tracking), executors not only need to perceive moving targets, but also should have a good cognitive ability to quickly re-locate the target when its position suddenly changes. Figure 8 illustrates the performance of human machine DVA under various task constraints. Based on experimental results, we find that: (1) A typical short-term tracking task only needs perceptron ability. Thus, both humans and most

---

Under review as a conference paper at ICLR 2023
Normalized precision plats (weighted by length) (tracking result in all sequences)
Suppeciyy
Subtect Prace
Ferter
Sublecl Meare 863)
Sunject Battor: In 601)
Histormen 166)
Normatzdd precssio
Supar sarer [D 2013
0.6
Ocnan
SPET 10-5200
0.4
475
KYS: 528)
DiStar (0.5
0.2
Stavel7le 20000
StamCAn: 3 0-000)
BCD: 3.771
KCh 10.270)
0.0
0.4
Normalzze Ized location error threshold
Figure 3: A comprchensive comparison of human- machine dynamic vision ability in all sequences. (I) Boxplots enumerate the distribution of NPL2 scores. The blue area represents the maximum, mean, and minimum scores of human subjects, the yellow area represents machines. (II) The normalized precision plot weighted by sequence length and the NP scores.
Easy
Hard
(Only Perception, Tay Exnmple)
Difficulty
(Perception Cognition,
The bospret of cvscutors
The boopict esacutore bucking lexe]
The boopler
roAuroo bradery buok)
Sujject Jean
Sutyes, Moan
Sutesl, Vean
Maformr
StavRONN
SlamACNN
RespTrack
KespPeck
Sppertipp
Downn
Ocean
Deccttrr
Ocean
DPP
SanC
D
DPN
JTCM
ALON
SammAA
SamnAA
StereC
SuneCC
0.2
12.4
0.6
0.0
0.7
5.4
2.6
1.0
0.0
0.2
2.4
0.6
0.3
3.0
Figure 4: A comprehensive comparison human- manane dynamic vision ability in short-term
tracking (I), long-term tracking (II), and global instance tracking (III) (based on NPL2 scores).
machines perform well in tracking the moving target: some SOTA machines even outperform humans accuracy and stability. (2) For the long-term tracking task, since the sequence length increases and the target disappearance is allowed during the tracking process, executors should locate the target when it reappears in the frame. Results demonstrate that humans can maintain high tracking precision, but the performance of most algorithms drops significantly. (3) Global instance tracking, the most difficult task that sequence length increases with shot-cuts and scene transitions, challenges almost all algorithms. However, humans still sustain efficient and stable tracking, indicating that the cognitive ability humans much better than machines.
Specifically, the top-3 machines with various model structurescan help to understand the perfor- mance of different modeling processes (Table (.. MixFormer (Cui et al. (2022). a simple end-to-end model based transformer structure, performs well in both short-term and long-time tracking tasks,
7

---

Under review as a conference paper at ICLR 2023
EST
E41
010
referde
1013
3:00
0:00
645
1002
1.64
3
044
8.11
030
0.08
8.2
050
0.00
8.33
Human
061 0.01
3:03
000
0.00
1:00
0.11
3/6
sapalion
Machine
Figure 5: Performance of human and machine (MixFormer) on challenging factors. Each cell represents a mixture of challenging factors, and the blank cells represent no such combination. Cell color represents the executor's 's mean NPL2 score across all videos that satisfy the combination. The purple rectangle represents short-term tracking with challenging factors (i.e., target presents from beginning to end, related to perception, mainly composed in groups H to Q), of which the red and blue rectangles represent static and dynamic challenging factors separately. The green rectangle represents task constraints (i.e., allowing target absence shot switching, related to both perception and cognition, mainly composed in groups C to G). For detailed results, please refer to Appendix D.5
indicating that the Mixed Attention Module (MAM) and a straightforward detection head can provide powerful tracking ability. The SiamRCNN (Voigtlaender et al. (2020)) model combines two-stage scheme with a new trajectory-based dynamic planning algorithm, and uses re-detection to accomplish stable tracking. KeepTrack (Mayer al. (202T) model improved by SuperDiMP (Danelljan et al.) (2020) has an enhanced ability to discriminate interferers and perform well in short-time tracking, but lacking a mechanism for processing target mutation decreases its execution in longer sequences. However, all these three machines have a performance gap with humans in the global tracking task. Since MixFormer performs best in most cases, we select it as a representation to accomplish more detailed analyses in the following sections.
Cognition. We further explore the effect of sequence length and shot-cuts, which requires the executor to have better cognition ability to re-locate the target in complex situations. Figure [3 in Appendix shows the performance variation of human-machine under different videos. Clearly, the SOTA machine's 's performance fluctuates wildly when the shot-cut occurs, while the subject can still maintain a stable tracking ability. In addition, machines may ultimately lose the target after frequent shot-cuts, but humans can quickly relocate the target in new shot. When the sequence length increases, the performance fluctuation of machines will continue to increase, but humans are better than machines in terms of fluctuation amplitude and tracking effect. It reveals that when the target motion or apparent information is mutated, the performance of machines fluctuates sharply, while humans can keep robust and precise target tracking based their high cognition ability.
Perceptron. Given that multiple challenging factors may coexist in video, we show the performance of human and SOTA machine under different factor combinations in Figure 5 Obviously, the purple rectangle is primarily related to the perceptive ability, and blue rectangle reflects the dynamic changes between consecutive frames. We can find that most cells in the blue rectangle are challenging for both algorithms and humans, but algorithms are slightly better than humans in some factors like fast motion and small targets. Conversely, task constraints represented by the green rectangle have less impact on humans, but it powerfully influences machines. This phenomenon is consistent with the above analysis of numan-machinn cognitive abilities.
Further analyses about humans. Former analyses indicate that to err is human. Thus, we provide detailed human performance analysis in Appendix [D6], which indicates that fast-moving targets and small targets are challenging track, but tracking in long sequence is not difficult for humans.
The above results show that current research, which mainly focuses on improving tracking robustness under short-term tracking tasks, has progressed. However, machines still have gap with humans when task constraints are widened to a more general situation. Some bad cases are illustrated in Figure 14 more results based on different indicators are listed in Appendix D.3 to D.4
8

---

Under review as a conference paper at ICLR 2023
Figure 6: The comparison and cooperation of humans and machines, based on NP score. (I-II) Performance of humans (blue bar) and machines (MixFormer, yellow bar) on different sequences. (III-VV) Performance of humans (blue bar) and human- human-machine machine cooperations (MixFormer_ Human, yellow bar) on different sequences. The green bar represents the overlapping part.
3.2 MACHINE PERFORMANCE IMPROVEMENTS WHEN COOPERATING WITH HUMANS
The analysis of Section 3. shows that humans and machines can adapt to different challenging factors, an intuitive idea is what if we allow human-machinn cooperation?
The cooperation mechanism is designed as follows: when machines fail to track, subjects can provide the current target position and restart machines from the failure frame. In this cooperation pattern, machines are responsible for performing relatively simple but repetitive tasks, while humans are responsible for supplementing information at critical moments. Specifical examples and detailed information are listed in Appendix C.7
Figure 6 illustrates the performance of humans, machines, and hucann collaborators. Com- pared with machine itself, human-machine collaborator is greatly improved and even outperforms humans in most cases. Subfigures of Figure 3 and Figure 24 indicate that human-machine collabora- tors have significantly improved their performance based on the assistance of humans. Therefore, humans and machines have superb possibilities for cooperation in dynamic vision tasks, which not only combines strengths and decreases weaknesses, but also saves human resources and improves efficiency. For detailed results based on different indicators, please refer to Appendix E. to
CONCLUSIONS AND FUTURE WORK
In this paper, we aim to answer how far the gap between humans and machines in dynamic vision tasks, and design an overall evaluation framework from three aspects we choose 87 videos to construct the experimental environment, select 20 representative algorithms and 15 human subjects to form the executors, and quantify their DVA with a strict evaluation process. Results indicate that SOTA machines are close to the lower limit of human subjects, meaning the DVA gap between humans and machines has significantly closed. Besides, humans and machines excel at different dynamic vision tasks, as humans can continuously track in general environments and longer sequences based better cognitive ability, while SOTA machines can maintain efficient and stable tracking ability in short-term tracking with excellent perceptual ability. Finally, we also find that the human- machine cooperative executor can perform even better than humans, which provides a possibility of human- manahie cooperation for application scenarios like self-driving systems.
As a preliminary research of the numan-machin DVA. we have obtained interesting conclusions above, but some aspects still deserve further exploration. For example, human DVA is usually decoupled into different tasks - SOT concentrates on locating an arbitrarily moving object, MOT aims to track multiple known objects simultaneously. and VOD focuses on detecting all known objects. Obviously, SOT is the basis of other dynamic vision tasks, thus we use it as representative task for research. In the future, researchers can apply the proposed evaluation framework to multiple dynamic vision tasks and further explore the DVA of humans and machines. Besides, future work can select various subjects (e.g., different age, gender, and occupation (Burg & Hulbert [961]; Melcher & Lund (1992)) to compose a more comprehensive human baseline.

---

Under review as a conference paper at ICLR 2023
REFERENCES
RP Beals, AM Mayyasi, AE Templeton, and WL Johnston. The relationship between basketball shooting performance and certain visual attributes. American Journal of Optometry and Archives of American Academy of Optometry. 48(7):585-555 590, 1971.
Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. Fully- convolutional siamese networks for object tracking. CoRR, abs/1606.09549, 2016. htthtpp
09549
Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. CoRR, abs/1904.07220, 2019. URL htll://ctp//aavv...rrr//////////
07220
Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Know your surroundings: Exploiting scene information for object tracking. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 205-221 Cham, 2020. Springer International Publishing. ISBN 978-3-030-55592-11.
Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418-424, 2018.
Albert Burg. Visual acuity measured by dynamic and static tests: comparative evaluation. Journal of applied psychology, 50(6):460, 1966.
Albert Burg and Slade Hulbert. Dynamic visual acuity as related to age, sex, and static acuity. Journal of Applied Psychology, 45(2):111, 1961.
Alan HS Chan and Alan J Courtney. Foveal acuity, peripheral acuity and search performance: A review. International Journal of Industrial Ergonomics, 18(2-3):113-119, 1996.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.35321 2014.
Gioele Ciaparrone, Francisco Luque Sanchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, and Francisco Herrera. Deep learning in video multi-object tracking: A survey. Neurocomputing, 381: 61-88, 2019.
Yutao Cui, Jiang Cheng, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. 2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp. 886-895 Teee, 2005.
M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg. Eco: In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6931-6933) July 2017. doi: 2t..9///VV..20........
Efficient convolution operators for tracking.
Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. ATOM: accurate tracking by overlap maximization. CoRR, abs/1811.00628,, 2018. URL
abs / /11/1111.110 07628
Martin Danelljan, Lue Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Heiko Dankert, Liming Wang, Eric D Hoopfer, David Anderson, and Pietro Perona. Automated monitoring and analysis of social behavior in drosophila. Nature Methods, 6(4):297-303, 2009.
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
10

---

Under review as a conference paper at ICLR 2023
Andre Esteva, Katherine Chou, Serena Yeung, Nikhil Naik, Ali Madani, Ali Mottaghi, Yun Liu, Eric Topol, Jeff Dean, and Richard Socher. Deep learning-enabled medical computer vision. NPJ digital medicine, 4(1):5. 2021.
Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juchuan Liu, Yong Xu, Chunyuan Liao, Lin Yuan, and Haibin Ling. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 2020.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. Advances in Neural Information Processing Systems, 33: 13890-13902, 2020.
Robert Geirhos, Kantharaju Narayanappa. Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34, 2021.
Brian Q. Geuther, Sean P. Deats, Kai J. Fox, Steve A. Murray, Robert E. Braun, Jacqueline K. White, Elissa J. Chesler, Cathleen M. Lutz, and Vivek Kumar. Robust mouse tracking in complex environments using neural networks. Communications Biology, 2(1):124, 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, and Shengyong Chen. Siamcar: Siamese fully convolutional classification and regression for visual tracking. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6269-6277, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):583-500 -56,, March 2015. doi: 10. 10.1109/TPAM.2201424555
Kenneth Holmqvist, Saga Lee Örbom, Ignace TC Hooge, Diederick C Nichorster, Robert G Alexander, Richard Andersson, Jeroen S Benjamins, Pieter Blignaut, Anne-Mart Brouwer, Lewis L Chuang, et al. Eye tracking: empirical foundations for a minimal reporting guideline. Behavior research methods, pp. 1-53, 2022.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 704.07700.44401111111 2017.
Shiyu Hu, Xin Zhao, and Kaiqi Huang. Sotverse: A user-defined task space of single object tracking. CoRR, abs/2204.07414, 2022a. doi: 10.48550/axiw.2204..07 7/doi .al.oorg/10 48550/arXix 2204.07411
URL https
Shiyu Hu, Xin Zhao, Lianghua Huang, and Kaiqi Huang. Global instance tracking: Locating target more like humans. IEEF Transactions Pattern Analysis and Machine Intelligence, pp. 1-1, 2022b. doi: 10. 109/TPAMM..0222.115333
Lianghua Huang, Xin Zhao, and Kaiqi Huang. Globaltrack: A simple and strong baseline for long-term tracking. arXiv: Computer Vision and Patlern Recognition, 2019.
Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 (5):1562-1577, 2021. doi: 10. 10.1109/10 109/TTAMM..00999....

---

Under review as a conference paper at ICLR 2023
David H. Hubel and Torsten N Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of physiology, 148(3)5574, 1959.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architec- ture in the cat's visual cortex. The Journal of physiology, 160(1):106, 1962.
MILLER JW al. The effect of relative motion on visual acuity. Survey of Ophthalmology, 83-116, 1962.
P Kohl, B Coffey, A Reichow, W Thhompson, and P Willer. A comparative study of visual performance in jet fighter pilots and non-pilots. Journal of Behavioral Optometry, 5(2): 1991.
Matej Kristan, Roman Pflugfelder, Ales Leonardis, Jiri Matas, and et al. Fatih Porikli. The visual object tracking vot2013 challenge results, Dec 2013.
Matej Kristan, Roman Pflugfelder, Ales Leonardis, Jiri Matas, and et al. Čehovin, Luka. The visual object tracking vot2014 challenge results. 09 2014. doi: 10.
Matej Kristan, Roman Pflugfelder, Ales Leonardis, Jiri Matas, and et al. Čehovin, Luka. The visual object tracking vor2015 challenge results. In Visual Object Trucking Workshop 2015 at ICCV2015, Dec 2015.
Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, and et al. Luka Ce- hovin Zaje. The visual object tracking vot2016 challenge results. Springer, Oct 2016. URL http:// www. swr.sgeeeeee com hwwoo//gpbaaee /ttp////ooo//3319948880880
Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder. Luka Čehovin, and et al. Vojir, Tomas. The visual object tracking vot2017 challenge results. pp. 1949-1972, 10 2017. doi: 10.1 00.10//CCCW:2211..333
Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, and al. Roman Pfugfelder. The sixth visual object tracking vo12018 challenge results, 2018.
Matej Kristan, Jiri Matas, Ales Leonardis, Michael Felsberg, and et al. Roman Pflugfelder. The seventh visual object tracking vot2019 challenge results, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Communicutions of the ACM, 60(6):84-90, 2017.
Michael F Land and Peter McLeod. From eye movements to actions: how batsmen hit the ball. Nature neuroscience, (311)):340-114444 2000.
Thomas Langlois, Haicheng Zhao, Erin Grant, Ishita Dasgupta, Tom Griffiths, and Nori Jacoby, Passive attention in artificial neural networks predicts human visual selectivity. Advances in Neural Information Processing Systems, 34, 2021.
Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. CoRR, abs/1812.11703, 2018a. URL http::/
www./rggg//////o/////.../...eerr abs/ 18 018881... 11703
Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Gerald M Long and David I. Penn. Dynamic visual acuity: normative functions and practical implications. Bulletin of the Psychonomic Society, 25(4):253-256 1987.
Christoph Mayer, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool. Learning target candidate association to keep track of what not to track. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 13424-13434, 2021. doi: 10. 101119//CCV4992.201..0011
12

---

Under review as a conference paper at ICLR 2023
MH Melcher and DR Lund. Sports vision and the high school student athlete. Journal of the American Optometric Association, 63(7):466-44) 4744, 1992.
Matthias Müller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Trackingnet:
A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 310-327, 2018.
Siddhartha Sankar Nath, Girish Mishra, Jajnyaseni Kar, Sayan Chakraborty, and Nilanjan Dey. A survey of image classification methods and techniques. In 2014 International conference on control, instrumentation, communication and computational technologies (ICCICCT), pp. 554-557. IEEE, 2014.
Zenon W Pylyshyn and Ron W Storm. Tracking multiple independent targets: Evidence for parallel tracking mechanism. Spatial vision, 3(3):179-197, 1988.
I Lluisa Quevedo, José Antonio Aznar-Casanova, Dolores Merindano- Genís Cardona, and Joan Folé-Fortó. A novel computer software for the evaluation dynamic visual acuity. Journal of Optometry, 133):131-138, 2012.
Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: A large high-precision human-annotated data set for object detection video. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5296-5305, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 358-666, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25) 2015.
David Silver, Julian Schritwwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676):354 359, 2017.
Philip L Smith and Daniel R Little. Small is beautiful: In defense of the small-n design. Psychonomic bulletin & review, 25(6):2083-2) 200-- 2018.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEF/CV international conference on computer vision, pp. 9627-9636, 2019.
Alan M Turing. Computing machinery and intelligence. In Parsing the turing test, pp. 23-65. Springer, 2009,
Joost Van De Weijer, Cordelia Schmid, Jakob Verbeek, and Diane Larlus. Learning color names for real-world applications. IEEE Transactions on Image Processing. 18(7):1512-153 2009.
Paul Voigtlaender, Jonathon Luiten, Philip H.S. Torr, and Bastian Leibe. Siam r-cnn: Visual tracking by re-detection. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6578-6588, 2020.
Shiyao Wang, Yucong Zhou, Junjic Yan, and Zhidong Deng. Fully motionn-aaar network for video object detection. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018, pp. 557-573, Cham, 2018. Springer International Publishing.
Kunlin Wei and Konrad Paul Kording. Behavioral tracking gets real. Nature Neuroscience, 21(9): 1146-1147, 2018.
13

---

Under review as a conference paper at ICLR 2023
Aharon Weissbrod, Alexander Shapiro, Genadiy Vasserman, Liat Edry, Molly Dayan, Assif Yitzhaky, Libi Hertzberg, Ofer Feinerman, and Tali Kimchi. Automated long-term tracking and social behavioural phenotyping of animal colonies within semi-natural environment. Nature Communi- cations, 4(1) 2(1),8018, 2013.
Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Online object tracking: A benchmark. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2411-2418, 2013.
Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):133--184 2015.
Chen Xia, Junwei Han, and Dingwen Zhang. Evaluation of saccadic scanpath prediction: Subjective assessment database and recurrent neural network based metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, (3(12):4377-4333) 2020.
Chen Xia, Junwei Han, and Dingwen Zhang. Evaluation of saccadic scanpath prediction: Subjective assessment database and recurrent neural network based metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12):4378-432) 2021. doi: 10. 00.1109/PAMM.22000300000..
Ning Xu, Linjie Yang. Yuchen Fan, Jianchao Yang. Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Scquence-to-sequenee video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 585-601, 2018.
Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. Siamfc++: Towards robust and accurate visual tracking with target estimation guidelines. Proceedings of the AAAI Conference Artificial Intelligence, 34(7):12549-1))
2020.
Bin Yan, Haojie Zhao, Dong Wang, Huchuan Lu, and Xiaoyun Yang. 'skimming-pente tracking: framework for real-time and robust long-term tracking. 2019 IEFF/CVF International Conference on Computer Vision (ICCV), pp. 2385-2393, 2019.
Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networks for real-time visual tracking. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4591- 4600, 2019.
Zhipeng Zhang and Houwen Peng. Ocean: Object-aware anchor-free tracking. In ECCV (2CV(21), pp. 771-787, 2020.
Zhaohui Zheng, Ping Wang. Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren. Distance-iou loss: Faster and better learning for bounding box regression. Proceedings of the AAAI Conference on Artificial Intelligence, 34(7): 12993-13000, 2020.
Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. CoRR, Abs//1808066/ 06048, 2018. URL htllhtttpp//axxiv.orr abs 1808 06048
14

---

Under review as a conference paper at ICLR 2023
A TASK DESCRIPTION
A.1 VISION TASK
Image
Image
R1
#200
#500
#700
Classifcctton:
Classification:
Classiication:
Image Classification
(II) Video Object Detection (VOD)
Single Scene
Scene
Scene 2
Scene #7200
B30
#1000
$4.500
A9300
User-
selected
Detection
Tracking
Tracking
Tracking
Object
Tracking
Tracking
Tracking
Tracking
(IIl) Multi-object Tracking (MOT)
(IV) Single Object Tracking (SOT)
(a) The execution flow of classical computer vision tasks (Image Classification (I), Video Object Detection (II),
Multi-object Tracking (III), and Single Object Tracking (IV)).
Object
Scenario
Visual Ability
Motion
Arbitrary Instance
Random
Task
Scene- changing
Object Category
Obejct Selection
Consistency Assumption
Scenario
Image
SVA
Limited training
N
Classification
Video Object
Limited training
Detected
Y
N
Detection
Multi-object
Limited, usually
Detected in the
Tracking
pedestrians or vehicles
first frame
Single Object
User-specified
Arbitrary
Y
Y
Tracking
the first frame
(b) The comparison task characteristics.
Figure 7: The execution flow classical computer vision tasks (a) and comparison of task character- istics (b). Obviously, SOT is closer human dynamic vision ability.
Image Classification. As static vision task, image classification (Nath et al. (2014) allows for classifying given image as belonging to one of the labeled and pre-defined categories. An excellent model should understand the target features and have robust classification ability facing challenging factors like target deformation or image blur.
Video Object Detection (VOD). VOD (Esteva al. (2021) an essential task for multiple computer vision applications, desires to accurately determine the category and location of each target in videos. However, the target category is typically limited pre-defined categories in the training dataset.
Multi-object Tracking (MOT). MOT (Geuther et al. (2019) in research usually combines with detection. MOT algorithms detect the object's position in the first frame, then calculate the similarity to determine instances belonging to the same target in consecutive frames. Thus, MOT is model- specific task and mainly focuses on tracking specific categories (e.g., pedestrians or vehicles).
Single Object Tracking (SOT). Unlike the above vision tasks, SOT (Wu et al. (2015)) is category-
independent, which means intends track a moving target without any assumption about target category. This characteristic allows SOT to be suitable for open-set testing with broad prospects. Besides, some recent research proposes a new task named Global Instance Tracking (GIT) (Hu al. (2022b)) to cancel the continuous motion assumption. Thus, the SOT definition is additionally extended, making it further step toward human dynamic vision tasks.
15

---

Under review as a conference paper at ICLR 2023
A.2
SINGLE OBJECT TRACKING TAsk
Credl
ABSCN
ABSENI
ABSEN
Mealler
indresmen
Id Global
RISENI
Figure 8: Representative sequence of single object tracking tasks. (a) For the short- term tracking task, the target appearance does not change significantly, but obstacles may partially obscure the target. (b) For the long-term tracking task, the target constantly moves in the same shot but may include the disappearanee-rappeaaane process. (c) For the global instance tracking task, the target may appear in different shots and scenes, but note that its apparent information does not vary significantly.
Due to the human-like task definition of SOT, excellent task executors should not only keep tracking the moving target (perception in short-term tracking, long-term tracking, and global instance tracking) but also ro-locate the target when its position mutated (cognition long-term tracking, and global instance tracking).
16

---

Under review as a conference paper at ICLR 2023
B DATASETS AND MACHINES
B. DATASETS
Table 3: The URL of open-sourced benchmarks used in this work.
Benchmarks
URL
OTB Wu ct
2013 2015
http://wlabbhanyagg. ac kr/tracker bew.brckoe.eccccccnnnnnndee.d
VOT
Kristan al. 2013:
2014
2015
2016
2016
2017
2018
2019
https://oocchlllnnee.s net/
GOT-TOK (Huang
http://ot- wik.aaiteettiioncco
La8OT Fan ot al. 20201
VidenCuhe (Hu et
2022h
We have correctly cited all benchmarks involved in the experiments and carefully checked the licenses involved in each benehmark (CC BY-NC-SA 4.0.). All datasets used in the experiments are open-sourced, and our usage is under the license scope.
B.2 MACHINES
Our algorithm evaluation experiments are performed on a server with 4 NVIDIA TITAN RTX GPUs and 64 Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz. We use the parameters provided by the original authors.
B.2.1 CORRELATION FILTER BASED TRACKERS
KCE The CF-based trackers represented by KCF (Henriques et al. (2015)) incorporate high speed and tracking accuracy, thus becoming representative tracking framework in the early stage. We the Python 2.7 version based on link
deep networks to improve feature representation. The official version is released in Py PyTracking/ which is a general python framework for SOT based on Py PyTorc. The feature representation of ECO a combination of the first and last convolutional layer in the VGG-m network (Chatheldd et al. (2014)) along with HOG (Dalal Triggs (2005)) and Color Names (CN) (Van De Weijer al. (209)))
ECO. ECO [Danelljan al. (2177) combines CNN with CF, aiming
B.2.2 SIAMESE NEURAL NETWORK BASED TRACKERS
SiamFC. As the originator of SNN-based trackers, SiamFC (Bertinetto al. (0016) achieves satisfactory tracking performance by matching features between the template region and the search region through simple network structure. The backbone SiamFC is AlexNet (Krizhevsky et al.)
(2117) ), which is trained on the ILSVRC15 dataset for object detection in video with 50 epochs.
SiamRPN. SiamRPN (Li et al. (2118)) introduces the region proposal network (Girshick (015 to achieve accurate target regression. This model is trained based on image pairs from ImageNet- VID (Russakovsky et al. (0155 and Youtube-BB (Real et al. (2017) with 50 epochs.
DaSiamRPN, DaSiamRPN (7hu et al. (2018) ) uses data augmentation to enhance the discriminative
ability, which trained based on ImageNet-V VID (Russakovsky et al. (2015)) Youtube-BB (Real et al.)
(2117 ImageNet (Deng et al. (2009h) and COCO (Lin et al. (2014) with 50 epochs.
SPLT. SPLT Yan et al. (2019) designs a verifier switch global search and local search. SPLT uses MobileNet (Howard et al. (2017) for feature extractor and downsamples the spatial resolution of the
"ntpp://imag--net.rg////i
"htpp://githuu.co/// wwwp://wwww..niiiiaaaaaaaaanonio
"https:/github..om.. com/fiolwwoddDDaaimamm
https://github httpp://gituh..om/iiaa... httm//iauuttacke/SSPP/...
17

---

algorithms refer to detection methods to Under review as a conference paper at ICLR 2023
template feature to 1 by average pooling. For the verification model, SPLT adopts ResNet50 the backbone of verifier. The parameters of aforementioned networks are initialized with the ImageNet (Deng et al. (2009) pre-trained models and then fine-tuned the ImageNet-Vil VID (Russakovsky et al. (2015) dataset.
introduces deeper backhone and selects ResNet (He) eet al. (0166) for feature extraction. The backbone network is pre-trained on ImageNet (Deng et al. (2009) ) for image labeling. Authors train the SiamRPN++ on the training sets of Youtube-BB (Real
SiamRPN+++ SiamRPN++ (Li et al. 201 Sa
et al. 2017)) ImageNet-VID (Russakovsky et al. (2015) ) and COCO (Lin et al. (2014) and to learn
a generic notion of how to measure the similarities between general objects for visual tracking.
selects deeper and wider backbones for feature extraction. The backbone network pre- trained on ImageNet (Deng et al. (2009) for image labeling.
SiamDW. SiamDW (Zhang & Peng 2019
Authors train the proposed cropping inside residual (CIR) units with SiamRPN on Youtube-BB (Real
al. (207)) and ImageNet- VID (Russakovsky et al. (2015).
Except for backbone improvements, some improve the network architecture.
SiamFC++, SiamFC++ (Xu et al. (0200) 10
employs an anchor-free structure (Tian et al. (2019) to
eliminate the dependence on anchors. It adopts ImageNet- VII (Russakovsky et al. (2015). ). COCO
(Lin et al. 2014)). Youtube-BB (Real et al. (017)) LaSOT (Fan et al. (2020)) and GOT-TOk (Huang
ct al. (222)) h) as basic training set.
Ocean. Ocean (hang & Peng (020 also employs the anchor-free structure. The backbone
network is initialized with the parameters pretrained on ImageNet (Deng et al. (009)). The Ocean
tracker is trained on the datasets of Youtube-BE BB (Real et al. (2017))
ImageNet- VID (Rusakkossk)
ct al.
(0155)) GOT-1Ok (Huang et al. (02[ and COCO (Lin et al.
(2014)).
SiamCAR, As another anchor-free tracker, SiamCAR (Guo al.
2020 is trained on COCO
[Lin et al. 2014)) ImageNet (Deng et al. (009)), ImageNet- VID
Russakovsky et al. 2015) and
Youtube- BB (Real et al. (2017).
SiamRCNN. SiN /Voiglaennerr et al. (0200) utilizes re-detection mechanism and tracklet
dynamic programming algorithm to process object disappearance. SiamRCNN is built upon the FasterRCNN (Ren et al. (2015)) implementation with a ResNet-
backbone. The backbone
has been pre-trained on COCO (Lin et al. (2014)), and the overall model is trained on ImD
(Russakovsky et al. (2015)). YouTube- VOS (Xu et al. (2018)), GOT- 100-10k (Huang et al. (022 and
LaSOT (Fan al. 20200).
B.2.3 COMBINE CORRELATION FILTER AND SIAMESE NEURAL NETWORK
ATOM. ATOM (Danelljan et al. 2018 14 tries to combine CF and SNN, and proposes a new
framework to use the ad adaatagess of offline training and online updating. The authors use the training splits of LaSOT (Fan et al. (2020) and TrackingNet (Müller et al. (2018)), and augment the training data with synthetic image pairs from COCO (Lin al. (2014))
DiMP. Based on framework proposed by ATOM, DiMP Bhat et al. 2019 optimizes the loss
function for stronger discriminative ability. The backbone network initialized with the ImageNet (Deng et al. (2009) ) weights. DIMP is trained on TrackingNet (Müller et al. (201)). LaSOT (Fan jet al. 20200. ) GOT-1Ok (Huang et al. (202T)) and COCO (Lin et al. (2014)) datasets with 50 epochs
by sampling 20,000 videos per epoch.
Baciiippusss__PS_PPP_S PoToh
httpss//ghhbb.ccmm//secamm////a analyst
12 hhugss//ghhbb.comm/eeaaaamm//nhr
13 hhugss//ihttb.omommmmhhhhhhhhiin
15 http:://gghhb...ommooomoomlnnnnn
18

---

Under review as a conference paper at ICLR 2023
PrDiMP and SuperDiMP. PrDiMP and SuperDiMP Daneeljjn et al. (2020) use probabilistic regression to improve the accuracy. The training splits the LaSOT (Fan et al. (2020)). GOT-1Ok
(Huang al. (02T)), TrackingNet (Müller al. (0118) and COCO (Lin et al. (2014)
used,
running 50 epochs with 1000 iterations each.
KeepTrack. KeepTrrack (Mayer al. 2021 combines the SuperDiMP with target candidate association network to accomplish robust tracking. The authors retrain the target candidate association
network hard sequences mined from LaSOT Fan et al. (2020))
B.2.4 CUSTOM NETWORKS
Some other works design custom networks to solve specific problems like target absence similar in- stance interference, naturally demonstrating a development from perceptual to cognitive intelligence.
GlobalTrack. GlobalTrack (Huang al. (2019) 18 does not assume motion consistency and performs
a full-image search to eliminate cumulative error. The authors use COCO (Lin et al. (2014). ) GOT-10k
(Huang et al. (2021)) and LaSOT (Fan et al. (2020) for model training.
KYS. KYS Bhat et al. 2020) represents scene information as state vectors and combines them
with the appearance model locate the object. The authors the training splits of TrackingNet
(Müller et al. (208)). GOT-10k (Huang et al. (202T)) and LaSOT (Fan et al. (2200) for model
training.
MixFormer. MixFormer (Cui et al. 2022) 20
designs an end-to-end transformer-based framework
simultaneously accomplish feature extraction and target information integration. The authors use COCO (Ln et al. (2044), TrackingNet (Müller et al. (2018), GOT-1Ok (Huang et al. (22T and LaSOT Fan al. (2020)) for model training.
"htts://github... htttgs//ggiuuu.oo/////oom////// comm/iiiionm//yyttaakinnn
15 "http:://itthub... "harps//gttubb.cmmuunngiii http:://gthhbb..ommmuunngggnnnnt
19 wtw.g/hwww..momiiiiioo///rrrrrra
19

---

Under review as a conference paper at ICLR 2023
C EXPERIMENT ORGANIZATION
C.1 EXPERIMENTAL ENVIRONMENT
ABSENT
65
AVB
4,30
#L99
SHOT-aUT
SHOT-CUT
SHOT-CUT
Figure 9: Representative sequences used in experiments. (T) Example of TEST videos, which help the experimenter familiar with the operation. (A) Example of short-term tracking videos, the sequence length of this group is concentrated in 500-1000 frames. (B) Example of short-term tracking videos, 1000-2000 frames.
the sequence length this group is concentrated tracking videos, the sequence length of this group is concentrated in 1000-2000 frames. (D) Example of long-term tracking videos, the sequence length of this group concentrated (E) Example of global instance tracking videos, the sequence length of this group is concentrated in 1000-2000 frames. (F) Example of global instance tracking videos, the sequence length this group concentrated in 5000- 10000 frames. (G) Example of global instance tracking videos, the sequence length of this group is concentrated in 15000-30000 frames.
(C) Example of long-term
5000-10000 frames.
20

---

Under review as a conference paper at ICLR 2023
Fight
JIP
BLY
Representative sequences used in experiments, the sequence length in this part is concentrated in 500-1000 frames. (H) Example of tracking the target with abnormal ratio. (I) Example of tracking the target with abnormal scale. (J) Example of tracking the target with abnormal illumination. (K) Example of tracking the target with blur bounding-bos (L) Example of tracking the target with drastic ratio variation. (M) Example of tracking the target with drastic scale variation.
Figure 10:
(N) Example of tracking the target with drastic illumination variation. (O) Example of tracking the
target with drastic clarity variation. (P) Example of tracking the target with fast motion. (Q) Example of tracking the target with low correlation coefficient.
21

---

Under review as a conference paper at ICLR 2023
C.2 EXPERIMENTAL STATEMENT
We have obtained the approvals of all experiments, the template is provided by our IRB. Every experiment has signed experimental statement before the experiment. Here list the contents:
Purpose You have been asked to participate in a research study that studies the dynamic visual abilities of humans. We would like your permission to enroll you as participant in this research study. The instruments involved in the experiment are a computer screen and a mouse. The experimental task consisted of gazing at the presented image sequence and manipulating the mouse to point as consistently as possible at the center of the moving target. You will be given specific instructions for the task before it begins.
Procedure In this study, you should read the experimental instructions and ensure that you understand the experimental content. The whole experiment process lasts about one hour, and the experiment divided into the following steps:
Read and sign the experimental statement;
Test the experimental instrument, and adjust the seat height, sitting posture, and the distance between your eyes and the screen. Please ensure that you in comfortable sitting position during the experiment;
Two TEST videos will be played. You should comprehend the specific instrument operation rules and be familiar with the experimental process through the test videos;
Start the FORMAL experiment. Please follow the operation to watch the test sequence and complete the relevant operations. Note that after watching each video sequence, you should have rest;
After the experiment, you need to fill in a questionnaire.
Risks and Discomforts The only potential risk factor for this experiment is trace electron radiation from the computer. Relevant studies have shown that radiation from computers and related peripherals will not cause harm to the human body. To rule out the impact of COVID-19 on the experiment, all participants completed nucleic acid tests before the experiment ensure their health. Everyone needs to wear a mask throughout the experiment. addition, the relevant devices are cleaned and disinfected during the experiment.
Costs Each participant who completes the experiment will be paid 200 RMB.
Confidentiality The results of this study may be published in academic journal/book used for teaching purposes. However, your name or other identifiers will not be used in any publication or teaching materials without your specific permission. In addition, if photographs, audio tapes or videotapes were taken during the study that would identify you, then you must give special permission for their use.
I confirm that the purpose of the research, the study procedures and the possible risks and discomforts as well as potential benefits that may experience have been explained to me. All my questions have been satisfactorily answered. I have read this consent form. My signature below indicates my willingness to participate in this study.
22

---

Under review as a conference paper at ICLR 2023
C.3 DEVICE SELECTION
How to measure the DVA of humans is an important section of our work. Previous works mainly eye-tracker (Hu al. (20226); Xia et al. (202T)) or mouse (Geirhos et al. (2018) 2020 2021)) to record human visual ability. To select better measurement method, we divide them into active (mouse) and passive (eye-tracker) methods, then compare them from practical and theoretical views:
From practical view, we have organized all subjects to experience both mouse and eye- tracker, and they all express that the user experience of the mouse is better. possible reason is that the eye-tracker passively records eye movements, causing the effect to be limited by many factors (e.g., the device accuracy, the user's posture, the relative distance between eyes and the eye-tracker, and the user's glasses). Besides, many subjects mention that the eye-tracker can only effectively track eye movements for short period. When the experimental time is prolonged, the recording result easily shifts, while subjects can only adjust their posture rather than actively correct the capture result. Conversely, using mouse can minimize the above problems. Although mouse may also have some limitations, its accuracy is greater than eye-tracker in our experiments. For example, when subjects notice that the mouse position far from the target, they can actively move the mouse and immediately correct the record result.
From a theoretical view, some researchers (Holmqvist et al. (2022) also indicate that the eye-tracker is affected by the tracking ratio (i.e., the amount of eye-tracking data lost). Blinks or brief vision drifts can cause lower tracking ratio, which requires the subject to keep a high degree of concentration when using eye-trackers. To prove this view, we check recent articles and find that experiments based on eye-trackers are either for observing the
images (subjects have sufficient time to adjust the position per image) (Xia et al. (2020)) watching small amount of video (e.g., 6 videos) (Hu et al. (2022b)). Nevertheless, experiment requests each participant to watch 17 videos (including long videos). Thus, while maintaining high concentration of attention, subjects frequently change their sitting position unconsciously, causing the eye-tracker performs poorly in our experiment. What's more, we find that previous works (Geirhos et al. (2018) 2020 2 have allowed subjects to perform image classification tasks with a mouse-based method, and a neuro training company named Reflexion also uses touch screens to provide DVA training for athletes. which means recording the observation position based on hand operation has been applied in academia and industry. Thus, we finally select the mouse-ban based experimental method.
23

---

Under review as a conference paper at ICLR 2023
C.4 EXPERIMENT PROCESS
First Frame
Experiient
Stap1.
Liguttment
Play TEST Vadeo
Following Frame
Figure 11: Process of DVA experiment. (I) 15 subjects participated in the DVA experiment. For each video number, the first letter represents the video group, and the second letter represents the index. Each group contains videos (e.g., A1-A5), and each video is observed by 3 subjects (e.g., A1 observed by subjects 05, 10, 15, respectively). Subjects whose serial numbers differ by 5 should watch the same content but in different order. (II) The experiment includes
steps: observing and remembering the target features in the first frame, then moving the mouse to track the target in subsequent frames. Note that subject has time relieve visual fatigue between two videos.
24

---

Under review as a conference paper at ICLR 2023
C.5 QUESTIONNAIRE
Subjects should fill in a questionnaire to evaluate their performance and make recommendations. Here list the contents:
This questionnaire aims to investigate subjects' opinions and self-evaluation.
QL. Your name:
Q2. Your ID (e.g., subject I):
Q3. Your age:
Q4. Your vision condition (fill in the left and right eyes separately):
Q5. Please rank the following factors in order of perceived difficulty during your experiment (order from difficult to easy):
Video sequence is too long:
Frequent shot changes;
Screen shaking:
Target scale is too small;
Similar objects interference;
Target moves fast;
Target is occluded.
Q6. In addition to the challenging factors in the previous question, what other factors may influence your observation?
Q7. Please provide your self-evaluation score (0 to 100) in short sequences (1-3 minutes):
Q8. Please provide your self-evaluation score (0 to 100) in medium sequences (3-10 minutes):
Q9. Please provide your self-evaluation score (0 100) in long sequences (more than 10 min- utes):
Q10. What the most impressive sequence in the experiment? Why are you impressed by it?
(e.g., the video is too long, the content of the video is exceptional, the video has some challenging factors that lead to poor performance, etc.)
Q11. Do you have any suggestions or comments for this experiment?
25

---

Under review as a conference paper at ICLR 2023
C.6 GROUP INFORMATION
Table Groups of subjects and corresponding videos. Subjects indicated by the same color watch the same content but in different order.
(a) Each human subject will watch 19 videos, including 2 TEST videos and 17 FORMAL experiment vidcos. For each video number, the first letter represents the video group, and the second letter represents the number of videos that group. Except for TEST videos, each group contains five videos (e.g., A A1-A5), and each video observed by three subjects (c.g., Al is observed by subjects 05, 10, and 15, respectively).
Group
II
III
Exp
02
03
05
06
07
09
10
11
12
14
15
T1
T1
T1
T1
T1
T1
T1
T1
T1
T1
T2
T2
12
T2
12
T2
12
T2
T2
T2
T2
12
T2
T2
12
A5
A2
A3
A4
A1
H4
H5
H3
H2
H1
GI
G2
G3
G4
GS
B5
B4
B3
B2
B1
15
II
12
13
14
F4
F5
F3
F2
F1
C5
C3
C2
C4
C1
J5
34
J3
J2
J1
D5
D4
D3
D2
DI
D5
D4
D3
D2
D1
K5
K4
K3
K2
K1
ES
B4
E3
E2
EI
E5
E4
E3
E2
EI
1,5
T.4
L.
1.2
1,3
C5
C3
C2
C4
CI
FS
F3
F1
M5
M4
MI
M3
BS
B4
B2
G2
G3
N3
N2
A5
A2
AI
Video
12
P5
15
QI
Q2
K3
A2
K5
K4
KI
1.4
BS
R4
B3
BI
1.5
TA
1.2
1.3
M4
MI
M2
M3
C5
C3
C2
C4
C1
M5
M4
M2
MB
N4
N3
N5
NI
N2
D5
D4
D3
D2
DI
N4
N3
N5
NI
N2
04
05
01
02
03
E5
E4
E3
E1
04
05
01
02
03
P4
P5
P1
P2
P3
F4
F5
F3
F2
F1
P4
PS
P1
P2
P3
Q1
Q2
Q3
Q4
Q5
G1
62
G3
G4
65
Q1
Q2
Q3
Q4
Q5
(b) Subjects in group watch the video length from short long, then become short again. Subjects in group II watch videos increasing length. Subjects group watch videos decreasing length.
Group Exp
01
02
03
04
05
06
07
08
09
10
1201
120
120
120
120
120
1201
120
1221
120
120
1.50
120
120
120
2000
2000
844
700
708
725
610
631
650
630
581
528
17892
18166
22872
25379
28828
1981
1741
1377
1000
651
570
1961
1735
1568
1865
1266
701
691
681
600
531
8242
7628
7509
5933
5307
8242
7628
7509
5933
5307
806
671
631
631
551
2000
1750
1500
1250
1000
1250
8750
10000
7500
6250
5000
659
601
510
531
550
1981
1741
1500
1377
1000
25379
Frame
631
650
630
581
528
601
671
501
531
581
631
650
630
581
$28
651
570
571
621
645
670
681
601
620
570
571
621
645
701
691
681
600
531
501
501
542
681
600
531
806
671
631
631
551
844
700
708
725
610
806
671
631
631
551
659
601
510
531
550
1961
1735
1568
1865
1266
639
601
510
531
350
601
601
585
556
571
8242
7628
7509
5933
5307
556
571
601
671
501
531
581
2000
1750
1500
1350
1000
531
581
670
681
601
620
680
8750
10000
7500
62.50
5000
670
681
601
620
650
831
542
561
17592
18166
22872
28379
28828
842
561
Total
50052
50165
51008
50710
50970
50052
50165
51038
50710
30970
50052
50165
51038
50710
50970
26

---

Under review as a conference paper at ICLR 2023
C.7 COOPERATION MECHANISM
C.7.1 DETAILED INFORMATION ABOUT COOPERATION MECHANISM
Algorithm Framework of cooperation mechanism
Input: T: tracker, which has two functions (INITIALIZATION and TRACK);
(F
Ft. J: original sequence:
G = [gi: 92. St, ........... J: the set of ground-truth, 9t (digt Mgt: Wgt, hgt) represents the ground-
truth bounding box in Fe;
B = {b). b2. ..... b. .. the set of blur degree, b, represents the blur degree value of F,, higher value means more clarity;
R r2, ........ the set of object relative scale, r; represents the object relative scale value
of Fr. small value means tiny object
Output: P {PI. P ... the set of tracking results, pt (I'pts Upt, pp hpt) represents the predicted bounding box in Fe: I: the set of cooperation frames, [Z] can be regarded number of cooperations
/* Step 1: find suitable gooperate ion frames candidate cooperation frames for + 0 to [S] do
while (b, MEDIAN(B)) (r: > MEDIAN(R)) do Cuui
1* Step 2:
cooperation
set tracking results = set initialization locations I set failure counter a 0 for to |S] do
if = then
P1 T.INTTIAIIZATINN(F....0 p P PUPU
continue
if (a 10) A (i C) then
= IT.NITIALIZATIONNF;; ( P - PUPu I I IUi a a+0
continue
else
P: T.TRACK(FF) P < PUpi
12pp. 9:)
Ps 7.
Ps
if s; 0.5 then + +
12
else
13
14 return P, I
As we have metioned in Section 3.2 the cooperation mechanism is designed follows: when machines fail to track, subjects can provide the current target position and restart machines from the failure frame. It is worth noting that human comannc cooperation is very complex, and our experiments only explore the possibility of cooperation through a simple and intuitive mechanism. There is still much space for further optimization.
Based on Algorithm 1, the numan-maching cooperation divided into the following steps. (1) Find suitable cooperation frames. The quality of the cooperation frame is important for tracking in the following frames. Frames with tiny objects or motion blur may decrease appearance and motion information. Thus, first calculate the blur degree and relative target scale of all frames in S,
27

---

Under review as a conference paper at ICLR 2023
and select frames that include the target with relatively clear and appropriate scale candidate cooperation frames (please refer to Hu et al. (2022a) for more information), and generate the set of candidate cooperation position C. (2) Human-machine cooperation. We use the IoU value s; (Line 10 in Algorithm 1) to determine tracking failure and consider that after ten consecutive frames of tracking failure (failure counter a 2 10), the algorithm will cooperate with the human at the nearest candidate cooperation frame and reinitialize (Line 8 in Algorithm 1, of represents the object information provided by human). Besides, considering that many SOTA machines may re-locate the target in several following frames, we assume that 2 0.5 means successfully re- re-locate the target, and zeroing the failure counter a if the re-location occurs in 10 frames (Line 13 in Algorithm 1).
It is worth noting that our human- machine cooperation experiments are preliminary exploration. and we hope find that cooperation possible for all algorithms. However, for combination of 20 algorithms and 87 videos (1740 sets), it is time-consuming to arrange human subjects to participate in 1740 cooperation experiments. Therefore, we used the following strategy to organize the experiments. (I) We first use the ground-truth as target information provided to the algorithm at cooperation frames (Line in Algorithm I, here we use 9i to re-initialize the algorithm). Since the ground-truth of sequences are provided by professional human annotators, which can be regarded of the highest level of human in static frames. Then we record all the cooperation frames as Ig (g means this set is generated by ground-truth information). (2) We show the frames in Zg human subjects, asking them to find the target in the current frame and annotate it with a bounding box, then we generate the human annotations In (the length of Z, is equal Ig). (3) We evaluate In and Z, based on NPL2 score. Situation 1: If the score is higher than 0.95, we consider the human subject have the same performance as the original data annotator. Thus, we longer organize the cooperation experiment for this algorithm-sequent the score is below 0.95, we consider that the human subject has gap with the ground- arrange for the human subject and the algorithm to track the sequence together from the beginning, ask the human subject to annotate the target position by bounding-box cooperation frames, and then record the collaborator's 's performance (Line 8 in Algorithm 1, here use of to re-initialize the algorithm).
representative
combination. Situation 2: On the contrary, if
We will
We find that most sequences satisfied Situation 1. As the example in Figure 12 the human subject watches the first frame and clearly understands that should locate the blue-clothed player. Then he identify the target at frames #1 125 and #6353, even if he does not watch the middle frames. This phenomenon also demonstrates the solid cognitive and memory abilities humans.
28

---

Under review as a conference paper at ICLR 2023
C.7.2 AN EXAMPLE
INITIALIZE
SHOT-CUT
ABSENT
ABSENT
INITIALIZE
INITIALIZE
INITIALIZE
Figure 12: Example of the cooperation mechanism. The top area illustrates machine tracking results on sequence G4, and the bottom the cooperation results. When machines fail to track (e.g.. #1 125 and #6353, always due to target absent shot-cut), subjects can provide the current target position and re-initialize machines to keep tracking from the failure frame.
To better illustrate the cooperation process, we use video G4 (a long video with 25,397 frames and 450 shot-cuts, describing table tennis match with two players) as an example. The best human subject scores 0.964 (NPL2 score) since it is easy to locate a player consistently. However, when the shot is switched, the position variation of the target player (blue-clothed player) will challenge the SOTA algorithm. Thus, it quickly drifts to the interferer (black-clothed player) and keeps wrong tracking, causing the score to decrease to 0.558. But if we allow algorithms to cooperate with humans, when the human subject finds that the algorithm fails (such as drifting to the distractor), the subject will provide correct target position to the algorithm to avoid persistent mistracking. For our experiment in G4, the human subject provides target location information 23 times while the algorithm tracks the remaining 25,374 frames. Although humans only need to participate 0.1% (23/25,397) in cooperation, the human-machine collaborator MixFormer. Human score 0.977, indicating that cooperation can significantly improve the algorithm's tracking performance. This result is also in line with the original intention: algorithms accomplish many simple tasks, while humans focus on the key points.
29

---

Under review as a conference paper at ICLR 2023
D PERFORMANCE OF HUMANS AND MACHINES
D.1 RESULTS ON DIFFERENT SEQUENCE LENGTHS
Matoorer
2000
4000
$300
D005
LO0CE
33000
14000
30000
5000 10000 35030 20000
Figure 13: Performance of human- machane different sequence lengths (the vertical coordinate indicates NPL1 score per frame). The yellow curve represents the SOTA machine (MixFormer (Cui et al. (2022)) )), the blue curve represents the subject, and the red triangle represents the shot-cut. Subfigures (I-IV) represent four videos with different combinations of lengths and shots. Compared with the SOTA machine, humans still maintain adequate DVA in the face of lengthy sequences and frequent shot switching.
30

---

Under review as a conference paper at ICLR 2023
D.2
EXAMPLES ON DIFFERENT SEQUENCES
D2
Machine
Condition
02
Machine
Condition
Human
Machine
Condition III
M4
Figure 14: Schematic diagram of human- mumanne tracking results partial sequences. Condition I (B2, D2, G4) represents that humans are better than machines, condition (O2, P4, P5) indicates that machines
better than humans, and condition III (E5, GI, M4) indicates that both perform poorly. For each frame, the green rectangle and green point represent the ground-truth the blue point represents the localization result of humans, and the yellow point represents the tracking result of machines.
Obviously, humans outperform machines in longer sequences with shot-cuts and can better distinguish the larget from interfering objects (condition 1). However, when facing challenges such tracking small targets with fast movements (condition II), limited by hand-brain coordination and mouse movement speed, humans cannot precisely locate such targets 25FPS. Note that both humans and machines may fail when multiple challenges superimposed in one sequence (condition III).
31

---

Under review as a conference paper at ICLR 2023
D.3 BOXPLOTS
K
Idade
Figure 15: Boxplots enumerate the distribution of NP scores for TEST videos and Group A to Q. The blue area represents the maximum, mean, and minimum scores of human subjects, and the yellow area represents machines.
32

---

Under review as a conference paper at ICLR 2023
D.4 PRECISION PLOTS
Figure 16: Results for all tasks, short-term tracking task, long-term tracking task, and global instance
tracking task based on various indicators (from left to right: weighted-N-PRE plots (0 ), ranked by : N-PRE plots NPLa(Oa)), ). ranked by NPL3; weighted- plots PT,,(aa,, (0a), ranked by PT:: and PRE plots PLs (fa), ranked by PL3).
33

---

Under review as a conference paper at ICLR 2023
Figure 17: Results for Group A to based on various indicators (from left to right: weighted-N- plots NPW, (Pa') ), ranked by NPW; N-PRE plots NP ranked by NPL3; weighted- -RE plots ('()))) ranked by P and PRE plots (La()a), ranked by PL3).
34

---

Under review as a conference paper at ICLR 2023
Figure 18: Results for Group G to based on various indicators (from left to right: weighted-N-PP plots NPW, (0s') ), ranked by NPW; N-PRE plots NP ) ranked by NPL3; weighted-PRO PRE plots P'(())) ranked by P33:: and PRE plots (0a()a), ranked by PL3).
35

---

Under review as a conference paper at ICLR 2023
Figure 19: Results for Group M to Q and TEST videos based on various indicators (from left to right: weighted-N-PRE plots (P ), ranked by NPW; N-PRE plots NPLs(0g ). ranked by NP weighted- plots (Oa),, ranked by PT;;: and PRE plots PLa(Oa), ranked by Pla)
36

---

Under review as a conference paper at ICLR 2023
D.5 HEATMAPS
Figure 20: Performance subjects (the maximum, mean, and minimum scores of human subjects) on multiple challenging factors based on NP
37

---

Under review as a conference paper at ICLR 2023
Figure 21: Performance of machines (MixFormer Cui et al. (202)) ) SiamRCNN (Voigllander et al. (2020) ). Keep'Track (Mayer al. (202T)), SuperDiMP (Danelljan Ijan et al. (2020). PrDiMP (Danelljan' et al. (2200) SiammRNN+ (Li et al. a (2018a) B GlobalTrack (Huang et al. (2019)) Ocean (Zhang & Peng 2020 SPLL
Yan et (2019) on multiple challenging factors based NPL2
38

---

Under review as a conference paper at ICLR 2023
Figure 22: Performance machines (DiMP (Bhat al. 2019)). SiamDW (Zhang & Peng (2019)).
KYS Bhat et al. DaSiamRN Zhu et al. (2015 8 (011)).SiamFCC. SiamFC++ (Xu et al. (020)) ..ATOM (Danelljan al. (2015) SiamRRN (Li al. (0118b), SiamCAR (Guo el al. (2020)). ECO (Danelljan ett a2.1 S.aam/ (Bertinetto et al. 2016 2016).KCF (Henriques et al. (2015)) multiple challenging factors based on NPL2.
39

---

Under review as a conference paper at ICLR 2023
D.6 ANALYSES OF HUMAN DYNAMIC VISUAL ABILITY
Eeet1
Eaxt2
13
Eipto
Figure 23: Scores subject groups and self self- peff-peeccetions. Subjects with a serial number interval of 5 watches the same videos in different orders. (I-V) demonstrate the performance of subjects (the vertical coordinate indicates NPL2 score per sequence). Gray area represents two TEST videos, and colorful areas 2 represent FORMAL videos. The viewing order of the subjects with blue serial number is 2-3-4, orange is 4-2-3, and green is 3-2-4. (VI) shows their self self-perceptions challenging factors in the questionnaire higher score means greater challenge).
We request three subjects track the same videos in different viewing orders. Figure 23 (I-V)
demonstrates that subjects' performance is independent of viewing order watching long sequences first last has no significant difference. However, we notice that subjects may also make mistakes, like the performance of Exp09 on A4 is significantly lower than others. A4 is a short video about basketball game, while Exp09 misadmits the target player keeps tracking distractor. indicates that humans are not absolutely infallible - they may make low-level mistakes due to carelessness.
Figure 23 (VI) shows the self- elf-perceptions of challenging factors in the questionnaire. Most subjects consider that fast-moving targets and small targets challenging to track. but tracking in a long sequence is not difficult this is consistent with previous experimental conclusions this paper.
40

---

Under review as a conference paper at ICLR 2023
E PERFORMANCE OF HUMAN-MACHING COOPERATIONS
E. BOXPLOTS
The bocalt
Tho loukl of comuters
S.tent Moen
2022
Himumn
Humen
2M0
105 Humen
Oud Mudianat
Nor Howws
2015
00
0.6
0.5 8.8
06. 0.7 0.5
035
Cogettion,
Figure 24: A comprehensive comparison of human-machine collaborators. Boxplots enumerate the distribution of LP scores for all tasks (I), short-term tracking task (II,, long-term tracking task (III), and global instance tracking task (IV). The blue area represents the maximum, mean, and minimum scores of human subjects, and the yellow area represents human- machine collaborators.
41

---

Under review as a conference paper at ICLR 2023
Figure 25: Boxplots enumerate the distribution of NPL2 scores for TEST videos and Group A to Q The blue area represents the maximum, mean, and minimum scores of human subjects, and the yellow area represents human-mas machine cooperation.
42

---

Under review as a conference paper at ICLR 2023
E.2 PRECISION PLOTS
Figure 26: Results for all tasks, short-term tracking task, long-term tracking task, and global instance
tracking task based on various indicators (from left to right: weighted-N-PRE plots (PT) ), ranked by : N-PRE plots NPLa(Oa)), ). ranked by NPL3; weighted- plots PT,,(aa), (0a), ranked by PT3: and PRE plots PLs (fa), ranked by PL3).
43

---

Under review as a conference paper at ICLR 2023
Figure 27: Results for Group A to based on various indicators (from left to right: weighted-N- plots NPW, ), ranked by NPW; N-PRE plots NP ranked by NPL3; weighted- PRE plots ('a)))) ranked by P and PRE plots (La()a), ranked by PL3).
44

---

Under review as a conference paper at ICLR 2023
Figure 28: Results for Group G to based on various indicators (from left to right: weighted-N-Pi plots NPW, (0s') ), ranked by NPW; N-PRE plots NPL:(0d)) ) ranked by NPL3; weighted-PRO PREE plots ("a)))) ranked by P33:: and PRE plots (La()a), ranked by PL3).
45

---

Under review as a conference paper at ICLR 2023
Figure 29: Results for Group M to Q and TEST videos based on various indicators (from left to
right: weighted-N-PRG plots (P ), ranked by NPW; N-PRE plots NPLs(0g ). ranked by NP weighted-PPE plots (Oa),, ranked by PT3:: and PRE plots PLa(Oa), ranked by Pla)
46

---

Under review as a conference paper at ICLR 2023
E.3 HEATMAPS
Figure 30: Performance of human- cooperations (Mimerrmer (Cui (0022) ). SiamRCNN
VVoigllenner et al. (2020)) ), Keep'Track (Mayer el al. (02T)), ), SuperDiMP DDaneljaa et al. (020 )
PrDiMP lDaneljaa
(2020)..izz SiamRPN++ (Li al. (2018a) GlobalTrack (Huang al. (019)).
Ocean (Zhang & Peng (2200),SPPP) (an et al. (2019). DiMP (Bhat et al. (2019)) Siam (Zhang Peng (2019)), KYS (Bhat al. (200 on multiple challenging factors based NP
47

---

Under review as a conference paper at ICLR 2023
Figure 31: Performance of
cooperations (DaSiamRPN (Zhu et al. Siam)
(Xu et al. 2020).ATOM AATM (Danelljan et al. SiimmRNN (Li et al. ). SSiamCAR (Guo et al.
(220 ECO (Danelljan et al. (2017)), SiamFC (Bertinetto et al. 201 (Henriques et al.
(015 ) on multiple challenging factors based on NPL2.
48


Reviews:
[{"review_id": "xDhzxajZZFX", "reviewer": "Reviewer_co2u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The presented work focuses on evaluation of the performance of machines and humans in perceptual and cognitive components of dynamic visual ability and understanding the gap between humans and machines capabilities. In addition, an approach for machine and human collaboration is proposed and tested. \nThe experiments were performed on a wide set of models and input data showing that perceptual ability of both humans and machines are closer than the cognitive ability, in which humans outperform machines. It has been also proved that human machine cooperation is possible and can lead to improved performance. ", "summary_of_the_review": "The work is interesting and claims are supported by extensive experiments. The addressed problem and approach seems novel, even though it's using existing models/metrics. ", "strength_and_weaknesses": "The claims of the paper are supported with experiments performed for multiple models. The topic is introduced well and compared with relevant studies in the same area. The DVA capabilities are examined from static and dynamic perspective. The work flows logically.\n\nOverall the work is interesting, the only concern is comparison of centers of objects. It is not clear how you ensure humans correctly mark centers of objects. It might be difficult for humans to visually find the object center. Have you encountered any issues there? If yes, then comparison of machine's performance might not be fair. Another minor issue is complexity of figures. It's difficult to quickly understand them, and careful analysis is needed. It would be nice to simplify them if possible. It might be also beneficial to compare the proposed human-machine cooperation mechanism with other existing human in the loop approaches. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wlh7RJjJVO", "reviewer": "Reviewer_RdAz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed an overall evaluation system to investigate the gap between humans and machines in dynamic vision tasks. In particular, they chose single object tracking as the representative task, and design an overall evaluation system from three aspects (Experimental environment construction, Experimental executor selection, and Evaluation method design). The results provide insights for fellow researchers in the related fields. ", "summary_of_the_review": "The paper is interesting and has its merits. However, as mentioned above in the weakness part, I feel the title is a bit overclaimed and the actual contribution is not very clear. ", "strength_and_weaknesses": "Strength: \n\nThe paper works on an interesting problem. The presentation of this paper is clear. \n\nThe authors have provided detailed supplementary materials to support the main paper. \n\n\nWeakness:\n\nThe authors used SOT as a single representative task for the evaluation system, but the concept of dynamic vision ability is much broader and includes numerous other tasks. Perhaps the authors can tune down a bit their claims throughout the paper. \n\nCorrespondingly, it is unclear to me how the results could contribute to the real applications, since the evaluation is only on SOT. \n\nAs we all know, human subjects have their biases due to their diverse cognitive background. I wonder if the authors should recruit more diversified human subjects, instead of 15 subjects while all of them are computer vision researchers aged 20-30.  Furthermore, how the machine models are trained also matters a lot. The authors can discuss the above potential issues in order to give a more convincing evaluation. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Upt_7-u0vg", "reviewer": "Reviewer_zwYU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "With the observation that no other work compared humans and machines in dynamic vision ability, the authors designed an overall evaluation system followed by previous work. Specifically, they evaluated humans and machines on the single object tracking task and compared their performance in multiple aspects. The analysis found that humans have stronger DVA, but the gap is not big, especially in the SOTA models. Finally, they show when human-machine cooperate, the performance can be even better than humans.", "summary_of_the_review": "The paper is well written with deep analysis to compare humans and machines in dynamic vision ability. However, the paper lacks novel contributions on the topics of learning representation. ", "strength_and_weaknesses": "**Strengths:**\n\n* The paper is well written, and the reviewer enjoyed reading it. \n\n* The authors provide details about the experimental settings so that the readers can understand the experiments well. \n\n* The authors conducted deep analysis for comparing human and machine DVA. \n\n**Weaknesses:**\n\n* The paper lacks contributions. The authors collect additional human subjects' responses for the SOT task and run 20 models to get machine responses, which is followed by deep analysis to compare the human and machine DVA. However, this reviewer is unclear what are the contributions (especially technical contributions) of this paper. \n\n* Section 2.2 does not contain the related work on the computer vision tasks, but the section rather looks like it introduces the term. Indeed, the tasks were already presented in Section 1.\n\n* In Figure 11: \"the blue sequences\" looked like they indicated the letters with a blue background, i.e. all the sequences. It is recommended either change the text background color or make it transparent. Further, it would be better to call the sequences A, B, C, ... instead of B2, D2, G4, ...  for better readability unless B2, D2, G4, ... has meanings. \n\n* Typos: \n\n\"Aritciture\" in Table 1", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject