Title: ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual tracking

non-uniform resizing

HVS-inspired processing
Abstract: Recently, the transformer has enabled the speed-oriented trackers to approach state-of-the-art (SOTA) performance with high-speed thanks to the smaller input size or the lighter feature extraction backbone, though they still substantially lag behind their corresponding performance-oriented versions. In this paper, we demonstrate that it is possible to narrow or even close this gap while achieving high tracking speed based on the smaller input size. To this end, we non-uniformly resize the cropped image to have a smaller input size while the resolution of the area where the target is more likely to appear is higher and vice versa. This enables us to solve the dilemma of attending to a larger visual field while retaining more raw information for the target despite a smaller input size. Our formulation for the non-uniform resizing can be efficiently solved through quadratic programming (QP) and naturally integrated into most of the crop-based local trackers. Comprehensive experiments on five challenging datasets based on two kinds  of transformer trackers, \ie, OSTrack and TransT, demonstrate consistent improvements over them. In particular, applying our method to the speed-oriented version of OSTrack even outperforms its performance-oriented counterpart by 0.6\% AUC on TNL2K, while running 50\% faster and saving over 55\% MACs. Codes and models are available at https://github.com/Kou-99/ZoomTrack.

ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking

Abstract

Recently, the transformer has enabled the speed-oriented trackers to approach stateof-the-art (SOTA) performance with high-speed thanks to the smaller input size or the lighter feature extraction backbone, though they still substantially lag behind their corresponding performance-oriented versions. In this paper, we demonstrate that it is possible to narrow or even close this gap while achieving high tracking speed based on the smaller input size. To this end, we non-uniformly resize the cropped image to have a smaller input size while the resolution of the area where the target is more likely to appear is higher and vice versa. This enables us to solve the dilemma of attending to a larger visual field while retaining more raw information for the target despite a smaller input size. Our formulation for the nonuniform resizing can be efficiently solved through quadratic programming (QP) and naturally integrated into most of the crop-based local trackers. Comprehensive experiments on five challenging datasets based on two kinds of transformer trackers, i.e. , OSTrack and TransT, demonstrate consistent improvements over them. In particular, applying our method to the speed-oriented version of OSTrack even outperforms its performance-oriented counterpart by 0.6% AUC on TNL2K, while running 50% faster and saving over 55% MACs. Codes and models are available at https://github.com/Kou-99/ZoomTrack .

1 Introduction

In visual tracking, many efforts have been made to improve the discrimination and localization ability of the tracking models, including deeper networks [18, 34], transformer feature fusion [8, 31, 21, 19], joint feature extraction and interaction [33, 7, 9], and so on. However, most of the recent tracking algorithms, including the transformer-based [33, 7, 19, 9], still follow the paradigm proposed by Bertinetto et al. [3], in which a small exemplar image cropped from

the first frame is used to locate the target within a large search image cropped from one of the subsequent frames. The crop size, which determines the visual field size, is derived from a reference bounding box plus a margin for context. Both of the crops are uniformly resized to fixed sizes to facilitate the training and testing of the tracker.

Due to the fact that the complexity of the transformers scales quadratically with input size, many transformer trackers [33, 7, 19, 9] propose both speed-oriented versions with the smaller input size or feature extraction backbone and performance-oriented versions with larger ones. Thanks to the enabled larger visual field size or stronger feature representation, the performance-oriented versions always outperform their speed-oriented counterparts in performance, though the tracking speed is severely reduced. For instance, an increased 1 . 5 × input size for OSTrack [33] will cause doubled Multiply-Accumulate Operations (MACs) and nearly halved tracking speed. Thus, it is natural to pose the question: Is it possible to narrow or even close this performance gap while achieving high tracking speed based on a smaller input size?

Inspired by the human vision system (HVS), we propose to non-uniformly resize the attended visual field to have smaller input size for visual tracking in this paper. Human retina receives about 100 MB of visual input per second, while only 1 MB of visual data is able to be sent to the central brain [36]. This is achieved by best utilizing the limited resource of the finite number of cones in HVS. More specifically, the density of cone photoreceptors is exponentially dropped with eccentricity ( i.e. , the deviation from the center of the retina), causing a more accurate central vision for precise recognition and a less accurate peripheral vision for rough localization [36]. This enables us to solve the dilemma of attending to larger visual field while retaining more raw information for the target area despite the smaller input size.

In our formulation for the non-uniform resizing, the area where the target is more likely to appear is magnified to retain more raw information with high resolution and facilitate precise recognition, whereas the area where the target is less likely to appear is shrunk yet preserved to facilitate rough localization when encountering fast motion or tracking drift. The key to our method is to design an efficient and controllable resizing module based on a small controllable grid. On top of this grid, we define a zoom energy to explicitly control the scale of magnification for the important target area, a rigid energy to avoid extreme deformation and a linear constraint to avoid cropping the source image during resizing. The important area is determined by the previous tracking result as a temporal prior. This formulation can be solved efficiently through

𝔤

𝑑𝑙

𝑟𝑜𝑤

𝑑𝑘 𝑐𝑜

l

𝔤

ₜ

Source Image

I

Target Image

I

'

quadratic programming (QP), whose solution is used to manipulate the controllable grid. Finally, the ideal non-uniform resizing is achieved by sampling according to the controllable grid (See Fig. 2).

Our method can be easily integrated into plenty of tracking algorithms. We select the popular hybrid CNN-Transformer tracker TransT [8] and one-stream Transformer tracker OSTrack [33] to verify the efficiency and effectiveness of our method. Comprehensive experiments are conducted on five large-scale benchmarks, including GOT-10k [16], LaSOT [11], LaSOT ext [12], TNL2k [30], TrackingNet [23]. We observe consistent improvements over the baselines with negligible computational overhead. In particular, we improve the speed-oriented version of OSTrack to achieve 73.5% AO, 50.5% AUC and 56.5% AUC on GOT-10k, LASOT ext and TNL2k respectively, which are on par with the performance-oriented counterpart while saving over 55% MACs and running 50% faster (see Fig. 1). In other words, the performance gap between speed-oriented and performance-oriented trackers can be narrowed or even closed through our method.

In summary, our contributions are as follows: ( i ) We propose ZoomTrack, an efficient non-uniform resizing method for visual tracking that bridge the gap between speed-oriented and performanceoriented trackers with negligible computation; ( ii ) We formulate the non-uniform resizing as an explicitly controlled magnification of important areas with restriction on extreme deformation,

enabling an HVS-inspired data processing with limited resource; ( iii ) Extensive experiments based on two baseline trackers on multiple benchmarks show that our method achieves consistent performance gains across different network architectures.

2 Related Work

Efficient Visual Tracking. A lot of real-world applications require tracking algorithms to have high speed. Consequently, many efforts have been made to improve the efficiency of visual tracking. Yan et al. [32] use NAS to search for a lightweight network architecture for visual tracking. Borsuk et al. [6] design a novel way to incorporate temporal information with only a single learnable parameter which achieves higher accuracy at a higher speed. Blatter et al. [5] design an efficient transformer layer and achieves real-time tracking on CPU. Shen et al. [26] propose a distilled tracking framework to learn small and fast trackers from heavy and slow trackers. Some other works [17, 10] explore the quickly advancing field of vision transformer pre-training [15, 29] and architecture designing [13] to improve both tracking accuracy and efficiency. Previous methods focus on designing either a better network [32, 5, 6, 17, 10] or a new training framework [26] to improve the efficiency of trackers, whereas our work focuses on the non-uniform resizing of the input for the sake of efficiency. Our approach is more general and orthogonal to them.

Non-uniform Resizing for Vision Tasks. Image resizing is a common image processing operation. Yet, the standard uniform resizing is not always satisfactory in many applications. Avidan and Shamir [1] resize an image by repeatedly carving or inserting seams that are determined by the saliency of image regions. Panozzo et al. [24] use axis-aligned deformation generated from an automatic or human-appointed saliency map to achieve content-aware image resizing. Recasens et al. [25] adaptively resize the image based on a saliency map generated by CNN for image recognition. Zheng et al. [37] learn attention maps to guide the sampling of the image to highlight details for image recognition. Thavamani et al. [27] use the sampling function of [25] to re-sample the input image based on temporal and dataset-level prior for autonomous navigation. Bejnordi et al. [2] learn a localization network to magnify salient regions from the unmediated supervision of [37] for video object detection. Thavamani et al. [28] propose an efficient and differentiable warp inversion that allows for mapping labels to the warped image to enable the end-to-end training of dense prediction tasks like semantic segmentation. Existing methods either use time-consuming feature extractors to generate saliency [1, 24, 25, 37] or use heuristic sampling functions that cannot explicitly control the extent of the magnification, which may cause unsatisfactory magnification or extreme deformation [27, 2, 28]. In contrast, our approach directly generates the saliency map or important area using the temporal prior in tracking and proposes a novel sampling method that can achieve controllable magnification without causing extreme deformation.

3 Background and Analysis

3.1 Revisiting Resizing in Deep Tracking

Before introducing our method, we first briefly revisit the resizing operation in the deep tracking pipeline. Given the first frame and its target location, visual trackers aim to locate the target in the subsequent frames using a tracking model ϕ θ ( z , x ) , where z is the template patch from the first frame with the target in the center and x is the search patch from one of the subsequent frames. Both z and x are generated following a crop-then-resize manner. We next detail the cropping and resizing operations.

Image crop generation. Both the template and search images are cropped based on a reference bounding box b , which can be the ground truth annotation for the template image or the previous tracking result for the search image. Denote a reference box as b = ( b cx , b cy , b w , b h ) , then the crop size can be calculated as

where f is the context factor controlling the amount of context to be included in the visual field, and c w and c h denote unit context amount which is solely related to b w and b h . Typical choice of unit context amount is c w = b w , c h = b h [33, 31, 9], or c w = c h = ( b w + b h ) / 2 [8, 19]. A higher context factor means a larger visual field and vice versa. Usually, the search image has a larger

0

10

20

30

40

50

68.2

68.7

69.2

69.7

70.2

256/4

288/4.5

320/5

352/5.5

384/6

MACs (G)

LaSOT AUC (%)

Search Size  / Context Factor

MACs

LaSOT AUC

1.5

2

2.5

3

3.5

66.5

67

67.5

68

68.5

69

69.5

70

4.5

5

5.5

6

Avg  Target Res

(10 3

Pixels)

LaSOT AUC (%)

Context Factor

LaSOT AUC

Avg Target Resolution

context factor for a larger visual field, and the template image has a small context factor providing minimal necessary context. The image crop I is obtained by cropping the area centered at ( b cx , b cy ) with width W and height H . Areas outside the image are padded with a constant value. Finally, box b is transformed to a new reference bounding box r = ( r cx , r cy , r w , r h ) on the image crop I .

Image crop resizing. To facilitate the batched training and later online tracking, the image crop has to be resized to a fixed size. Given an image crop I with width W and height H , a fix-sized image patch I ′ with width w and height h is obtained by resizing I . Specifically, h × w pixels in I ′ ( x ′ , y ′ ) are sampled from I ( x, y ) according to a mapping T : ❘ 2 → ❘ 2 , i.e. ,

Note that T does not necessarily map integer index to integer index. Thus, some pixels in I ′ may be sampled from I according to the non-integer indices. This can be realized by bilinear interpolating from nearby pixels in I . Current methods [33, 9, 19, 7] use uniform mapping T uniform to resize I . T uniform is defined as T uniform ( x ′ , y ′ ) = ( x ′ s x , y ′ s y ) , where s x = w/W, s y = h/H are called resizing factors and indicate the scaling of the image. When applying uniform resizing, the resizing factor is the same wherever on the image crop, which means the same amount of amplification or shrinkage is applied to the different areas of the entire image.

3.2 Solving the Dilemma Between Visual Field and Target Area Resolution

Although the crop-then-resize paradigm is applied in most of the popular deep trackers, the fixed input search size and context factor vary significantly across different algorithms. Generally, the performance can be improved by using a larger input size at the cost of decreasing tracking speed [19, 33]. To understand the key factors behind this, we conduct a series of experiments on LaSOT [11] based on OSTrack [33]. First, we simultaneously increase search size and context factor, which means the target resolution is roughly fixed and the tracker attends to a larger visual field. As shown in the left of Fig. 3, the AUC on LaSOT increases along with the input search size thanks to the larger attended visual field. Note that increasing the search size ( 256 → 384 ) leads to almost doubled computational overhead ( 21 . 5 G MACs → 41 . 5 G MACs). Next, we limit the available resources by fixing the input size while increasing the attended visual field size. As shown in the right of Fig. 3, the AUC on LaSOT first increases and then decreases as the average target resolution 2 keeps decreasing. This result demonstrates that the benefit of an enlarged visual field is gradually wiped out by the decrease in target resolution when the computational cost is fixed.

Inspired by the data processing with limited resources in HVS, we believe the above dilemma of attending to a larger visual field while retaining more raw information for the target can be solved by replacing uniform resizing with non-uniform resizing. In visual tracking, the previous tracking result can serve as a strong temporal prior for the current target location. Based on this prior, we can determine the area where the target is likely to appear and magnify it to retain more raw information with high resolution. On the contrary, areas where the target is less likely to appear is shrunk to avoid decreasing the visual field when the input search size is fixed with limited resources. Notably, the

magnification and shrinkage should not dramatically change the shape and aspect ratio of regions on the image to facilitate the robust learning of the appearance model. Based on the above analysis, we propose some guidelines for our non-uniform resizing, i.e. , G1 : Magnify the area where the target is most likely to appear; G2 : Avoid extreme deformation; G3 : Avoid cropping the original image I so that the original visual field is also retained. We detail how to incorporate these guidelines into our non-uniform resizing in the following section.

4 Methods

As discussed in Sec. 3.1, the resizing process from the source image I ( x, y ) ∈ ❘ H × W × 3 to the target image I ′ ( x ′ , y ′ ) ∈ ❘ h × w × 3 is a sampling operation controlled by a mapping T : ( x ′ , y ′ ) → ( x, y ) . Our aim is to find the mapping T that best follows the guidelines ( G1~G3 ), given a temporal prior box r = ( r cx , r cy , r w , r h ) that indicates the most possible location of the target. We first restrict the domain of the mapping to a few points and acquire a grid representation of the mapping T . Then, we formulate the guidelines as a QP problem based on the grid point intervals. By solving the QP problem via a standard solver [20], we can efficiently manipulate the controllable grid point intervals and achieve the ideal non-uniform resizing by sampling according to the final grid representation.

4.1 Grid Representation for Non-uniform Resizing

Recall the sampling operation in Eq. (2), pixel ( x ′ , y ′ ) on the target image I ′ ∈ ❘ h × w × 3 is sampled from location ( x, y ) on the source image I ∈ ❘ H × W × 3 . Thus, at least h × w pairs of ( x ′ , y ′ ) → ( x, y ) are needed for resizing. We use the exact h × w pairs to represent the mapping T , which can be defined as a grid G ∈ ❘ h × w × 2 , where G [ y ′ ][ x ′ ] = ( x, y ) , x ′ = 1 , ..., w, y ′ = 1 , ..., h , is the location on I that the target image pixel I ′ ( x ′ , y ′ ) should be sampled from.

Controllable grid. The grid G is an extremely dense grid with h × w grid points (equivalent to the resolution of the target image I ′ ), which makes it computationally inefficient to be directly generated. To solve this issue, we define a small-sized controllable grid g ∈ ❘ ( m +1) × ( n +1) × 2 with m +1 rows and n +1 columns ( m<<h,n << w ) to estimate G , where

Intuitively, g [ j ][ i ] is the sample location on the source image for the target image pixel ( w n i, h m j ) in I ′ . Once g is determined, G can be easily obtained through bilinear interpolation on g .

Axis-alignment constraint. To avoid extreme deformation ( G3 ) and reduce computational overhead, we add an axis-aligned constraint [27] to the grid g . Specifically, grid points in the same row j have the same y-axis coordinate y j , and grid points in the same column i have the same x-axis coordinate x i . That is to say g [ j ][ i ] = ( x i , y j ) , i = 0 , ..., n, j = 0 , ..., m . The reason why we use such an axis-aligned constraint is that we need to keep the axis-alignment of the bounding box ( i.e. , the four sides of the bounding box are parallel to the image boundary) after resizing, which is proven to be beneficial to the learning of object localization [27].

Grid representation. The source image is split into small rectangular patches p ( k, l ) , k = 1 , ...n, l = 1 , ..., m by grid points. The top-left corner of p ( k, l ) is g [ l -1][ k -1] and its bottom-right corner is g [ l ][ k ] . The width and height of the patch p ( k, l ) are denoted as the horizontal grid point interval d col k and the vertical grid point interval d row l . d col k and d row l can be calculated as d col k = x k -x k -1 , d row l = y l -y l -1 . The axis-aligned controllable grid g can be generated from grid point intervals d row l , d col k based on

where d row 0 = d col 0 = 0 for the sake of simplicity.

Non-uniform resizing. According to Eq. (3), the resizing process can be visualized using two grids (see Fig. 2): a manipulatable grid on the source image g and a fixed grid on the target image g t [ j ][ i ] = ( w n i, h m j ) . g and g t split the source and target images into rectangular patches p ( k, l ) and p t ( k, l ) respectively. Target image pixel at g t [ j ][ i ] is sampled from g [ j ][ i ] on the source image. This

process can be intuitively understood as resizing p ( k, l ) to p t ( k, l ) . When the grid point intervals of g are manipulated, the grid point intervals of g t remain unchanged. Thus, by reducing/ increasing the grid intervals of g , the corresponding region on the target image is amplified/ shrunk. In this way, a non-uniform resizing operation that scales different image areas with different ratios can be obtained by dynamically manipulating the grid point intervals.

4.2 QP Formulation Based on Grid Representation

Given a reference bounding box r = ( r cx , r cy , r w , r h ) , we aim to dynamically manipulate the grid point intervals to best follow the guidelines ( G1~G3 ) proposed in Sec. 3.2.

Importance score for the target area. According to G1 , areas, where the target is most likely to appear, should be magnified. Following this guideline, we first define the importance score for the target area on the source image. An area has high importance if the target is more likely to appear in this area. As r is a strong temporal prior for the current target location, we use a Gaussian function G ( x, y ) centered at ( r cx , r cy ) as the importance score function, i.e. ,

where µ x = r cx , µ y = r cy , σ x = √ β × r w , σ y = √ β × r h . β is a hyper-parameter controlling the bandwidth of the Gaussian function. Since we denote the rectangular area enclosed by the grid point intervals d row l and d col k as patch p ( k, l ) , its importance score S k,l can be determined by the value of G ( x, y ) at the center of patch p ( k, l ) , i.e. ,

where ϵ is a small constant to prevent extreme deformation and ensure stable computation. Here we use the patch center of a uniformly initialized grid ( d col k = W n and d row l = H m ). By doing so, S k,l is irrelevant of d col k and d row l , which is crucial for the QP formulation in the following.

Zoom energy. Following G1 , we design a quadratic energy E zoom to magnify the area where the target is most likely to appear. To achieve controllable magnification, we define a zoom factor γ controlling the amount of magnification. If we want to magnify some area by γ , the distances between sampling grid points should shrink by 1 γ . E zoom forces d col k and d row l to be close to the grid point intervals under uniform magnification by γ , i.e. , 1 γ H m for d row l and 1 γ W n for d col k . In detail, The zoom energy is defined as

Rigid energy. To avoid extreme deformation ( G2 ), we define a quadratic energy E rigid to restrict the aspect ratio change of patch p ( k, l ) , which has width d col k and height d row l . When being uniformly resized, p ( k, l ) should have width W n and height H m . Thus, the patch p ( k, l ) is horizontally stretched by n W d col k and vertically stretched by m H d row l . To keep the aspect ratio mostly unchanged for the important area, the horizontal stretch should be similar to the vertical stretch. Therefore, the rigid energy can be defined as

Linear constraint to avoid cropping the source image. According to G3 , the resizing should not crop the source image so that the original visual field is also retained. That is to say, the grid g should completely cover the entire source image. This requirement can be interpreted as a simple linear constraint that forces the sum of vertical grid point intervals d row l to be H and the sum of horizontal grid point intervals d col k to be W : ∑ m l =1 d row l = H, ∑ n k =1 d col k = W .

QP formulation. Combining the zoom energy, the rigid energy and the linear constraint, we can formulate the grid manipulation as the following minimization task

where λ is a hyper-parameter to balance the two energies. This convex objective function can be efficiently solved by a standard QP solver [20] (see Appendix A for more detailed derivations).

4.3 Integration with Common Trackers

Reference box generation. Our non-uniform resizing method needs a temporal prior reference box r = ( r cx , r cy , r w , r h ) on the cropped source image I . During testing, r is directly generated from the previous frame tracking result (see Sec. 3.1). During training, we only need to generate r w and r h from the ground truth width g w and height g h as the center of the temporal prior reference box is fixed at r cx = W/ 2 and r cy = H/ 2 . In detail, with probability q we use a smaller jitter j = j s , and with probability 1 -q we use a larger jitter j = j l . r w and r h are calculated as r w = e J w g w , r h = e J h g h , where J w , J h ∼ N (0 , j 2 ) .

Label mapping. The classification loss of the network ϕ θ ( z , x ) is calculated on the target image I ′ . Therefore, the classification labels on I ′ should be generated from the ground truth bounding box on I ′ . The common practice is to use normalized ground truth bounding box for the reason that the normalized coordinates remain the same on I and I ′ under uniform resizing. However, this does not hold true for our non-uniform resizing. Therefore, we need to map the ground truth bounding box from I to I ′ via the grid G . Specifically, we map the top-left corner and bottom-right corner of the bounding box as follows. Given a point on the source image ( x, y ) , we first find two grid points on G that are closest to ( x, y ) . Then, the corresponding point ( x ′ , y ′ ) can be obtained by bilinear interpolation between the indices of these two points.

Reverse box mapping. The bounding box predicted by the network ϕ θ ( z , x ) is on I ′ and needed to be mapped back to I as the final tracking result. We map the top-left corner and bottom-right corner of the predicted bounding box from I ′ to I in a reversed process of label mapping. Specifically, given a point ( x ′ , y ′ ) on I ′ , we first find the two grid points of G whose indices are closest to ( x ′ , y ′ ) , and then the corresponding point ( x, y ) can be obtained by bilinear interpolation between the values of these points. To minimize the impact of the reverse mapping, we directly calculate the regression loss on I instead of on I ′ .

5 Experiments

5.1 Implementation Details

We apply our method to both one-stream tracker OSTrack [33] (denoted as OSTrack-Zoom) and CNN-Transformer tracker TransT [8] (denoted as TransT-Zoom). The proposed non-uniform resizing module is applied in both the training and testing stages on the search image. We do not apply the proposed non-uniform resizing on the template image because it degrades performance (see Sec. 5.3). The search patch size ( 256 × 256 ) is the same as the baseline methods, whereas the context factor is enlarged from 4 to 5. We use a controllable grid g with shape 17 × 17( m = n = 16) . The importance score map is generated by a Gaussian function with bandwidth β = 64 . We set magnification factor γ = 1 . 5 and λ = 1 in Eq. (9). We use smaller jitter j s = 0 . 1 with probability 0 . 8 and larger jitter j l = 0 . 5 with probability 0 . 2 . Notably, we use the same set of parameters for both OSTrack-Zoom and TransT-Zoom to demonstrate the generalization capability of our method. Except for the resizing operation, all the other training and testing protocols follow the corresponding baselines. The trackers are trained on four NVIDIA V100 GPUs. The inference speed and MACs are evaluated on a platform with Intel i7-11700 CPU and NVIDIA V100 GPU.

5.2 State-of-the-art Comparison

We compare our method with the state-of-the-art trackers on five challenging large-scale datasets: GOT-10k [16], LaSOT [11], LaSOT ext [12], TNL2K [30] and TrackingNet [23].

GOT-10k [16] provides training and test splits with zero-overlapped object classes. Trackers are required to be only trained on GOT-10k training split and then evaluated on the test split to examine their generalization capability. We follow this protocol and report our results in Tab. 1. Our OSTrackZoom achieves 73.5% AO which surpasses the baseline OSTrack by 2.5%. And our TransT-Zoom achieves 67.5% AO which surpasses the baseline TransT by 0.4%. Compared with previous speedoriented trackers, our OSTrack-Zoom improves all of the three metrics in GOT-10k by a large margin, proving that our method has good generalization capability.

LaSOT [11] is a large-scale dataset with 280 long-term video sequences. As shown in Tab. 1, our method improves the AUC of OSTrack and TransT by 1.1% and 2.2% respectively. Our method OSTrack-Zoom have the best performance among speed-oriented trackers on LaSOT, indicating that our method is well-suited for complex object motion in long-term visual tracking task.

LaSOT ext [12] is an extension of LaSOT with 150 additional videos containing objects from 15 novel classes outside of LaSOT. In Tab. 1, our method achieves 3.1% and 2.0% relative gains on AUC metric for OSTrack and TransT baselines respectively. Our method OSTrack-Zoom achieves promising 50.5% AUC and 57.4% Precision, outperforming other methods by a large margin.

TNL2K [30] is a newly proposed dataset with new challenging factors like adversarial samples, modality switch, etc . As reported in Tab. 1, our method improves OSTrack and TransT by 2.2% and 3.0% on AUC. Our OSTrack-Zoom sets the new state-of-the-art performance on TNL2K with 56.5% AUC while running at 100 fps.

TrackingNet [23] is a large-scale short-term tracking benchmark. Our method boosts the performance of OSTrack and TransT by 0.1% and 0.4% SUC. The reason for the relatively small improvements is that, contrary to other datasets, TrackingNet has fewer sequences with scenarios that can benefit from an enlarged visual field, e.g. scenarios with drastic movement or significant drift after some period of time. Please refer to Appendix F for a detailed analysis.

Comparison with performance-oriented trackers. We additionally compare our method with the performance-oriented versions of the SOTA trackers. On the one hand, on GOT-10k, LaSOT ext , TNL2K whose test splits object classes have no or few overlap with training data, our OSTrack-Zoom is on-par with the SOTA tracker OSTrack-384, while consuming only 44.5% MACs of the latter and running 50% faster. On the other hand, on LaSOT and TrackingNet whose test splits have many object classes appearing in training data, our OSTrack-Zoom is slightly inferior to some SOTA trackers. A possible reason is that these SOTA trackers with heavier networks fit the training data better, which is

beneficial to track objects with classes that have been seen during training. Nevertheless, the gap between speed-oriented and performance-oriented trackers is narrowed using our method.

5.3 Ablation Experiment

Comparison with other resizing methods. As shown in Tab. 2, we compare our non-uniform resizing method ( ④ ) with uniform resizing with small/large input sizes ( ① / ② ) and FOVEA [27] non-uniform resizing ( ③ ). On both TNL2K and LaSOT, our method outperforms other methods. Notably, while applying FOVEA causes a significantly degraded performance ( ① vs. ③ ), our method can achieve large performance gains ( ① vs. ④ ). To understand the reason behind this, we calculate the average target size (avg) and the standard deviation of the target size (std), which represent the degree of the targets' scale change, on LaSOT. Compared with uniform resizing ( ① vs. ④ ), our method achieves 1 . 34 2 × magnification on average target size, while the standard deviation almost keeps the same. The magnification is close to the desired γ 2 = 1 . 5 2 , showing that our method achieves controllable magnification. In contrast to FOVEA ( ③ vs. ④ ), which severely reduces the tracking performance, our standard deviation is almost unchanged, demonstrating that our method can avoid extreme deformation that severely changes the scale of the target.

Computational overhead analysis. Our resizing module runs purely on CPU. The latency for resizing an image is 1 . 58 ms on an Intel i7-11700 CPU. Compared with uniform resizing ( ① vs. ④ ), our method introduces an additional 1.54 ms latency during resizing, causing a slightly slower speed. However, our performance is better than the uniform resizing counterparts. Especially when compared with uniform resizing with a larger input size ( ② vs. ④ ), our method reaches higher performance while running faster, demonstrating the superiority of using non-uniform resizing in visual tracking. Our method is also faster than FOVEA resizing while having a better performance. Moreover, the MACS for our resizing an image is 1 . 28 M MACS ( 10 6 ), which is negligible compared to the network inference which is usually tens of G MACs ( 10 9 ).

Resizing in different stages. It is worth noting that our method can be directly applied to off-theshell tracking models, i.e. , only applying non-uniform resizing method at testing time. Results on LaSOT [11] and TNL2K [30] are reported in Tab. 4 using the AUC (%) metric. As shown in Tab. 4, directly applying our method to the off-the-shelf trackers only in testing can already improve the tracking accuracy ( ① vs. ② ). Further applying our resizing in training can also boost the performance, owning to the aligned training and testing protocols ( ② vs. ③ ).

Resizing template and search images separately. We also investigate the effects of applying the non-uniform resizing on the template and search images separately. As shown in Tab. 5 either applying non-uniform resizing on the template image alone or on both the template and search images performs worse than applying it only on the search image. We think the reason is that the template

STRADue

image must have a relatively small context factor to contain minimal necessary context, which makes the whole template region important for the tracking network to locate the target. Therefore, the tracker cannot benefit from non-uniform resizing of the template image.

More ablations. First , we study the impact of the bandwidth β used in the importance function. A smaller β means less area is considered to be important and vice versa. As shown in Tab. 3a, the ideal value for β is 64 . Both smaller bandwidth (4) and larger bandwidth (256) cause reduced performance, showing the importance of this factor. Second , we also analyze the impact of the controllable grid. As shown in Tab. 3b, m = n = 16 is the best size for this grid. A smaller grid size cause a drastic drop in AUC, showing that too small grid size will cause inaccurate resizing that damages the tracking performance. A larger grid size also degrades performance, for which we believe the reason is that a large grid size will lead to an over-reliance on the important area generation based on the temporal prior, which is however not always correct. Third , we further look into the impact of different zoom factors γ . As shown in Tab. 3c, 1 . 5 is the best value for γ . A smaller γ will degrade performance due to smaller target resolution. Meanwhile, a larger γ is also harmful to the performance as larger magnification will cause trouble when trying to rediscover the lost target. Finally , we study the impact of E rigid by changing the balance factor λ . As shown in Tab. 3d, setting λ = 1 reaches the best performance. When λ = 0 , the performance drops, demonstrating the effectiveness of the proposed E energy . Alarger λ = 2 also degrades the performance, indicating that too much constraint is harmful to the performance.

Visualization. We visualize our proposed non-uniform resizing, uniform resizing and FOVEA [27] resizing in Fig. 4. Compared with uniform resizing, our method can improve the resolution of the target to retain more raw information. Compared with FOVEA, our method can better preserve the aspect ratio and the appearance of the target.

6 Conclusion

In this paper, we presented a novel non-uniform resizing method for visual tracking, which efficiently improves the tracking performance by simultaneously attending to a larger visual field and retaining more raw information with high resolution. Inspired by the HVS data processing with limited resources, our method formulates the non-uniform resizing as an explicitly controlled magnification of important areas with restriction on extreme deformation. Extensive experiments have demonstrated that our method can bridge the gap between speed-oriented and performance-oriented trackers with negligible computational overhead.

Limitations. One limitation of our method is that we need a fixed context factor to determine the size of the visual field. It would be interesting to develop a method to dynamically adjust the size of the visual field according to the tracking trajectory or appearance of the target in future work.

Acknowledgements. The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. This work was supported in part by the National Key R&D Program of China (Grant No. 2020AAA0106800), the Natural Science Foundation of China (Grant No. U22B2056, 61972394, 62036011, 62192782, 61721004, U2033210), the Beijing Natural Science Foundation (Grant No. L223003, JQ22014, 4234087), the Major Projects of Guangdong Education Department for Foundation Research and Applied Research (Grant No. 2017KZDXM081, 2018KZDXM066), the Guangdong Provincial University Innovation Team Project (Grant No. 2020KCXTD045). Jin Gao and Bing Li were also supported in part by the Youth Innovation Promotion Association, CAS.

References

Shai Avidan and Ariel Shamir. Seam carving for content-aware image resizing. In SIGGRAPH . 2007.

Babak Ehteshami Bejnordi, Amirhossein Habibian, Fatih Porikli, and Amir Ghodrati. SALISA: Saliencybased input sampling for efficient video object detection. In ECCV , 2022.

Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. Fullyconvolutional siamese networks for object tracking. In ECCV Workshops , 2016.

Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In ICCV , 2019.

Philippe Blatter, Menelaos Kanakis, Martin Danelljan, and Luc Van Gool. Efficient visual tracking with exemplar transformers. In WACV , 2023.

Vasyl Borsuk, Roman Vei, Orest Kupyn, Tetiana Martyniuk, Igor Krashenyi, and Jiˇ ri Matas. FEAR: Fast, efficient, accurate and robust visual tracker. In ECCV , 2022.

Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang. Backbone is all your need: A simplified architecture for visual object tracking. In ECCV , 2022.

Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In CVPR , 2021.

Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. MixFormer: End-to-end tracking with iterative mixed attention. In CVPR , 2022.

Yutao Cui, Cheng Jiang, Gangshan Wu, and Limin Wang. MixFormer: End-to-end tracking with iterative mixed attention. arXiv preprint arXiv: 2302.02814 , 2023.

Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. LaSOT: A high-quality benchmark for large-scale single object tracking. In CVPR , 2019.

Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juehuan Liu, Yong Xu, et al. LaSOT: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision , 129:439-461, 2021.

Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. LeViT: A vision transformer in convnet's clothing for faster inference. In ICCV , 2021.

Mingzhe Guo, Zhipeng Zhang, Heng Fan, Liping Jing, Yilin Lyu, Bing Li, and Weiming Hu. Learning target-aware representation for visual tracking via informative interactions. In IJCAI , 2022.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022.

Lianghua Huang, Xin Zhao, and Kaiqi Huang. GOT-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(5): 1562-1577, 2019.

Ben Kang, Xin Chen, Dong Wang, Houwen Peng, and Huchuan Lu. Exploring lightweight hierarchical vision transformers for efficient visual tracking. In ICCV , 2023.

Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. SiamRPN++: Evolution of siamese visual tracking with very deep networks. In CVPR , 2019.

Liting Lin, Heng Fan, Zhipeng Zhang, Yong Xu, and Haibin Ling. SwinTrack: A simple and strong baseline for transformer tracking. In NeurIPS , 2022.

Martin S. Andersen, Joachim Dahl, and Lieven Vandenberghe. CVXOPT: A python package for convex optimization (version: 1.3.0), 2022. URL http://cvxopt.org/index.html .

Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool. Transforming model prediction for tracking. In CVPR , 2022.

Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmark and simulator for UAV tracking. In ECCV , 2016.

Matthias Müller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. TrackingNet: A large-scale dataset and benchmark for object tracking in the wild. In ECCV , 2018.

Daniele Panozzo, Ofir Weber, and Olga Sorkine. Robust image retargeting via axis-aligned deformation. In EUROGRAPHICS , 2012.

Adrià Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, and Antonio Torralba. Learning to zoom: A saliency-based sampling layer for neural networks. In ECCV , 2018.

Jianbing Shen, Yuanpei Liu, Xingping Dong, Xiankai Lu, Fahad Shahbaz Khan, and Steven Hoi. Distilled siamese networks for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44 (12):8896-8909, 2021.

Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan. FOVEA: Foveated image magnification for autonomous navigation. In ICCV , 2021.

Chittesh Thavamani, Mengtian Li, Francesco Ferroni, and Deva Ramanan. Learning to zoom and unzoom. In CVPR , 2023.

Shaoru Wang, Jin Gao, Zeming Li, Xiaoqin Zhang, and Weiming Hu. A closer look at self-supervised lightweight vision transformers. In ICML , 2023.

Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In CVPR , 2021.

Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In ICCV , 2021.

Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Huchuan Lu. LightTrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In CVPR , 2021.

Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In ECCV , 2022.

Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networks for real-time visual tracking. In CVPR , 2019.

Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu. Learn to match: Automatic matching network design for visual tracking. In ICCV , 2021.

Li Zhaoping. A new framework for understanding vision from the perspective of the primary visual cortex. Current opinion in neurobiology , 58:1-10, 2019.

Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition. In CVPR , 2019.

Appendices

A Derivations of the QP-based Minimization Task for Grid Manipulation

The minimization task for grid manipulation in our non-uniform resizing is defined as

where the zoom energy E zoom is defined as

and the rigid energy E rigid is defined as

This minimization task in Eq. (A1) can be efficiently solved using a standard QP solver [20], which requires converting the above minimization task into the general form of QP, i.e. ,

where d = ( d row 1 , ..., d row m , d col 1 , ..., d col n ) ⊤ ∈ ❘ m + n is the unknown grid intervals to be optimized. P ∈ ❘ ( m + n ) × ( m + n ) and q ∈ ❘ m + n can be derived from the energy function E = E zoom + λE rigid . A ∈ ❘ 2 × ( m + n ) and b ∈ ❘ 2 can be derived from the linear constraint in Eq. (A1).

Converting the energy function into the matrix form. Denote S col k = ∑ n l =1 S 2 k,l , and S row l = ∑ m k =1 S 2 k,l , the overall energy E can be expanded as

where C is a constant term that can be ignored during minimization. The first, second and third lines of Eq. (A5) are consisting of squared terms, linear terms and cross terms respectively.

Thus, the overall energy E can be converted into the matrix form described in Eq. (A4) by fitting the above terms into P ∈ ❘ ( m + n ) × ( m + n ) and q ∈ ❘ m + n , i.e. ,

where P row ∈ ❘ n × n and P col ∈ ❘ m × m are diagonal matrices accounting for the squared terms in E . P row and P col can be defined as

P cross ∈ ❘ m × n is a matrix accounting for the cross terms in E , while q row ∈ ❘ n and q col ∈ ❘ m account for the linear terms. P cross , q row and q col can be defined as

Converting the linear constraint into the matrix form. The linear constraint can fit in the matrix form described in Eq. (A1) by defining A ∈ ❘ 2 × ( m + n ) and b ∈ ❘ 2 as

Proof of the convexity. When the matrix P in Eq. (A6) is positive semi-definite, the energy function in Eq. (A1) is convex and a global minimization can be found. Apparently, P is positive semi-definite since, for any d ∈ ❘ m + n ,

B Attribute-based Analysis on LaSOT

We further analyze the AUC gains brought by our method on the two baselines, i.e. , OSTrack and TransT, based on the attributes of the videos on LaSOT [11]. The results are displayed in Fig. A1.

FOC OV

LR

FM VC

IV ARCPOC CM MB SV ROT BC DEF

0

1

2

3

4

5

6

AUC gain (%)

OSTrack

TransT

As shown in Fig. A1, our method significantly improves the performance of both baselines under challenging scenarios with drastic movement (see Fast motion (FM), Viewpoint change (VC), etc ) or

significant drift after some period of time (see Full Occlusion (FOC), Out-of-View (OV), etc ), owning to the enlarged visual field. Performance on videos with low resolution (LR) is also considerably improved thanks to the zoom-in behavior. The deformation caused by non-uniform resizing may cause a little performance degradation when encountering background clutter (BC) or deformation (DEF).

C Comparison Between Non-uniform Resizing and Optimized Uniform Resizing Baseline

To demonstrate the superiority of our method, we additionally explore whether optimizing the crop hyper-parameters in the uniform resizing baseline can have a similar effect to our proposed non-uniform resizing, the results are shown in Tab. A1.

Despite the fact that our limited compute resources make us unable to optimize the above hyperparameters on large tracking datasets in a grid-search manner, our adequately powered experiments clearly show that the above optimizing is indeed hardly able to achieve similar effects to our resizing. Even for the baseline OSTrack-256[33] with the optimal crop hyper-parameters on LaSOT[11] (69.5 AUC), we experimentally find that it can only achieve AUC gains of +0.4 and +0.6 on LaSOT ext [12] and TNL2K[30] respectively while performing worse than the original OSTrack-256 on TrackingNet[23] (83.1 SUC vs 82.1 SUC). In other words, our method still outperforms the above optimal baseline on LaSOT (+0.7 AUC), LaSOT ext (+2.7 AUC), TNL2K (+1.6 AUC), TrackingNet (+1.1 SUC). Moreover, it is experimentally shown that the mismatch of the object size between the search and template images caused by increasing the search-image context factor has a relatively small effect on the final tracking accuracy for modern transformer trackers. That means the low resolution of the object is the major cause for the worse accuracy.

Table A1: Comparison results of our method and the uniform resizing baseline with its crop hyperparameters varied ( i.e. , setting c w and c h using two different choices as in Sec. 3.1, varying the search-image context factor, and varying the template-image size to alleviate the mismatch of the object size between the search and temple images). The experiments are based on OSTrack-256[33] and tested on LaSOT[11]. The context factor and size of search/template images are displayed as context factor / size . The choices of c w , c h : ❶ c w = b w , c h = b h and ❷ c w = c h = ( b w + b h ) / 2

D More Experimental Results on UAV123

In this section, we report the performance comparison on an additional benchmark, i.e. , UAV123 [22]. UAV123 is an aerial tracking benchmark captured from low-altitude UAVs with more small objects and larger variations in bounding box size and aspect ratio. As shown in Tab. A2, our method achieves 1.0% and 0.6% relative gains on AUC for the OSTrack [33] and TransT [8] baselines, demonstrating that our method can improve the tracking performance in a wide range of tracking scenarios.

E More Experimental Results with SwinTrack as the Baseline

We further apply our method to both the performance-oriented version SwinTrack-B (backbone: Swin Transformer-Base) and speed-oriented version SwinTrack-T (backbone: Swin Transformer-Tiny) of SwinTrack [19] to experimentally show the generalization capability of our proposed resizing method for a wide range of visual tracking algorithms. Note that we use the v1 version of SwinTrack (without motion token) as the baseline because only the code of v1 version is publicly available.

As shown in Tab. A3, our method can bring consistent improvement for both SwinTrack-T and SwinTrack-B. Particularly worth mentioning is that our SwinTrack-B-Zoom can even outperforms SwinTrack-B-384 with a much smaller input size.

F Analysis on Small Improvement Over TrackingNet

To find out the reason for the small improvement over TrackingNet [23], we additionally analyze the ratios of challenging scenarios appearing in the different tracking datasets (see Tab. A4), which clearly show that the test split of TrackingNet has a very different property from LaSOT [11], LaSOT ext [12], and TNL2K [30] with respect to the ratios of challenging scenarios. As shown in Fig. A1, our method can achieve significant performance gains under challenging scenarios with drastic movement (see Fast Motion (FM), Viewpoint Change (VC), etc .) or significant drift after some period of time (see Full Occlusion (FOC), Out-of-View (OV), etc .) owning to the enlarged visual field. However, the ratios of the challenging scenarios FOC, OV, FM in TrackingNet (VC is not labeled in TrackingNet) are significantly lower than in LaSOT, LaSOT ext , and TNL2K, which may be the reason for small improvement of our approach on TrackingNet.

G Detailed Latency of Non-uniform Resizing

We provide the latency analysis in terms of time (ms) and MACs for solving QP and resizing images in Tab. A5. It can be seen that resizing images with interpolations in our method costs about half of the total time in spite of its parallel processing, while iteratively solving QP costs the rest of the total time despite its much lower FLOPs.

H Confidence Interval for Ablation Experiments of Hyper-parameters

We calculate the 90% confidence intervals for all the experiments in Tab. 3 using bootstrap sampling. Each of the 90% confidence intervals is obtained using 10000 bootstrap samples, each sample with size 280 (same as the video number of LaSOT-test-split). The results are listed in Tab. A6.

(a) Bandwidth

β

(b) Control grid size

I More Visualization Results

We visualize the results of our method OSTrack-Zoom and its corresponding baseline on videos with challenging attributes to qualitatively show the superiority of our method in Fig. A2. Thanks to the zoom-in behavior and the enlarged visual field, our method can better handle the challenges such as full occlusion, fast motion, etc .

As shown in the left part of Fig. A2, at frame # 150, both our method and the baseline lose track of the target since the target is fully occluded. At frame # 157, our method quickly re-detects the target as the target re-appears. Note the target is significantly larger in the resized input image when using our resizing method, which indicates that the zoom-in behavior allowed by our method helps the re-detection of the target. At frame # 187, our method successfully tracks the target whereas the baseline still drifts to a similar car.

As shown in the right part of Fig. A2, at frame # 15, the target is moving really fast. As a result, in the resized input image generated by the baseline, part of the target is out of view which causes an inaccurate localization. In contrast, the resized input image generated by our method keeps the entire target visible thanks to the enlarged visual field, resulting in a more accurate tracking result. The situation is similar in frame # 27, where the baseline tracks a fully-preserved distractor rather than

the half-cropped target, whereas our method successfully tracks the target since the target is intact. At frame # 32, the baseline completely lose track of the target, while our method successfully tracks it.

15

27

32

150

157

187

SKINLA3FPV

EGY

117

eipes

SKINLA3FPV

Correlos

Sadia

orreios

NISSAN

Sadia

Correios

Ground Truth

OSTrack-Zoom

OSTrack

J Failure Cases

We visualize the failure cases of our method by comparing the tracking results of OSTrack-Zoom (blue) and OSTrack (green) in Fig. A3.

As shown in the left part of Fig. A3, at frame # 1108, the target is out of view. As our method enlarges the visual field, apart from the distractor (orange ball) at the center of the input, an additional distractor (dark green ball at the top-right of the input image) comes into view, which causes the tracker to predict a large box between the two distractors. The localization error quickly accumulates to an overly enlarged visual field at frame # 1117, since the visual field is partly determined by the previous prediction. At this time, a white ball that is very similar to the template is brought into view, causing our method to drift to the far-away white ball. Then, at frame # 1130, when the target re-appears, our method cannot re-detect the target as our method drifts to a far-away distractor, which makes the real target out of the visual field of our tracker.

As shown in the right part of Fig. A3, at frame # 94, both our method and the baseline successfully track the target. However, at frame # 104, the enlarged visual field from our method brought additional contexts (orange robotic arm on the left) into view. The little fraction of orange robotic arm on the left makes the distractor look like a horizontally flipped template which has a small fraction of orange robotic arm on the right. As a result, our method drifts to the distractor. Then, at frame # 114, our

method fails to track back to the squeezed real target since it is at the edge of the input image and partly invisible.

From the visualization of the failure cases, we find that, although an enlarged visual field promotes tracking performance in most cases, sometimes additional distractors or contexts brought by the fixed large visual field will cause drifting. Thus, it would be interesting to develop a method to dynamically adjust the visual field based on the surrounding of the target and its tracking history in the future.

Ground Truth

OSTrack-Zoom

OSTrack

94

104

114

1108

1117

1130

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a very interesting problem in local visual tracking with crop-then-resize operations: if it is possible to reduce the high computational demand of attending to a large visual field while retaining its potential benefit of achieving SOTA tracking accuracy. To this end, the paper takes inspiration from the processing of the human retina and proposes an elegant non-uniform resizing method to enable retaining more raw information for the target area while resizing a cropped image with a large visual field to a smaller size. By enforcing an explicitly controlled magnification of important areas with restriction on extreme deformation on top of a grid representation, this non-uniform resizing can be achieved efficiently based on QP. The authors select two representative local trackers (TransT and OSTrack) to demonstrate the well-balanced tracking accuracy and speed of trackers that are based on the proposed resizing module. Extensive experiments are conducted on five challenging tracking benchmarks, which clearly show the superiority of the proposed approach.\n\nSTRENGTHS:\nIn general, I am positive about this paper. I am glad to see this paper paying some effort to improve the little-explored cropping and resizing strategy in local visual trackers. The strengths of this paper are as follows:\n\n1, Good organization and writing. The paper is well organized and the flow is smooth.\n\n2, Novelty. The proposed non-uniform resizing method is quite interesting and looks solid to me, which enables retaining more raw information for the target area during resizing.\n\n3, Promising performance. This paper enables OSTrack to achieve 70.2% AUC on LaSOT with 100fps, which is impressive. Most existing competitors always rely on heavy trackers to achieve AUC score above 70% on LaSOT. The performance gap between the proposed efficient tracker and the heavy trackers on GOT-10k, LaSOT_ext, and TNL2K is even closed. The experimental analyses are also aligned with the motivation of attending to a large visual field.\n\nWEAKNESSES:\n1, Only two transformer trackers are selected for experimental validations. One is the hybrid CNN-Transformer tracker TransT and the other one is the one-stream Transformer tracker OSTrack. It would be better if one more purely CNN-based tracker or some other top performer is selected to further show the generalization capability. The conclusion would be more convincing.\n\n2, In Table 1, it is observed that the improvement on TrackingNet is very small. The SUC performance gain is only 0.1%. I suppose there are some detailed explanations for the reasons but obviously, there is not.\n\n3, It is not clear to me whether the proposed non-uniform resizing is also applied to the cropped template. A detailed analysis in this regard is also required to make the paper self-contained and fully reproducible.\n\nQUESTIONS:\nThere are several questions.\n\n1, In line 198-199, the authors said they use a (m+1)*(n+1) small-sized controllable grid to approximate the grid G with shape h*w. How can a grid be approximated by another grid with a different size?\n\n2, In Eq. (6) for determining the important score of different areas, a small constant is also used to prevent extreme deformation. What’s the relationship between this part and the specially designed rigid energy in Eq. (8) following G2?\n\n3, In Table 1 and Figure 1, the authors use MACs to indicate the computational overhead, while in line 353 they use FLOPS. Please explain this inconsistency. Moreover, is the computational overhead of your method in Table 1 exactly the same as the corresponding baseline?\n\nOverall, I like this paper a lot, including the novelty with broad interest and its promising performance. However, I rated it as weak accept (6) now due to the above weaknesses and the above-mentioned questions. If my concerns and questions are addressed sufficiently, I will increase my score.\n\nLIMITATIONS:\nYes, the limitations have been briefly addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a non-uniform resizing method for visual tracking, where they aim to boost the performance of transformer-based visual tracking methods at negligible additional computational cost. They achieve this by reducing the input size while attending to the relevant parts of the target, and this approach is applicable to wide range of crop-based tracking algorithms. They show the results of their algorithm on various large-scale datasets.\n\nSTRENGTHS:\n- Authors focus on the easily overlooked aspect of visual tracking algorithms, and their approach seems technically plausible while bringing adequate enlightenment to the field.\n\n- Their approach shows competitive performance on large-scale benchmarks compared to contemporary tracking algorithms.\n\n- They validate their method on multiple baseline algorithms with ablation experiments.\n\nWEAKNESSES:\n- Although the motivation of the proposed method seems plausible, there seems to be a lack of validation when it comes to analyzing the importance of choosing input resizing method for trackers. Are there further analysis on the impact of input resizing for wide range of visual tracking algorithms?\n\n- Since the formulation for the rectangular grid is hand-crafted using multiple controlallable hyperparameters (e.g. bandwitdh of Gaussians $\\beta$, choice of $\\epsilon, \\gamma, \\lambda$, ...), results in Table 3 shows some sensitivity to these hyperparameter choices. Considering these results and the results in Table 1, the performance gains obtained by the proposed strategy seems limited.\n\n- What are the reasons for the design choices of choosing (1) Gaussian importance scores for target area, (2) Using quadratic formulation for zoom and rigid energy? Are there alternative formulations tested?\n\n- Proposed non-uniform grid formulation seems to be target content (e.g. target object class, color, etc.) agnostic and only dependent on the input bounding box position and size. Since a simple learning based model (e.g. lightweight MLP) is also available, would this have been a better approach?\n\nQUESTIONS:\nPlease refer to the weaknesses section for details. I generally concur with the motivations raised by the authors, but their approach to the problem needs more explanation and justifications for their design choices are needed.\n\nLIMITATIONS:\nAuthors provide a limitations section at the end of their paper, and the limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper propose a novel method to achieve high tracking speed with smaller input size. Non-uniformly resizing operation is introduced with respect to the object appearing probability.  This resizing operation is then modeled as QP problem and can be effectively soved. This proposed method can be a flexible plugin for multiple deep trackers. Extensive experiments on various tracking benchmarrk are conducted with TransT and OSTrack to demonstrate the effectiveness of the proposed method.

STRENGTHS:
1. authors identify the conflict of context ratio and search region size in deep trackers, and proposed a novel method to solve it. 

2. the target region resizing is modeled as a QP problem analytically. 

3. Introducing non-uniform resizing for target regions and solve the QP problem efficiently.

WEAKNESSES:
1. The tracking performance of proposed non-uniform resizing is only evaluated on speed-oriented tracker. I recommend the author  better to further validate this method on performance-oriented trackers. 

2. The computation overhead of proposed method is not reported detailly. A more detail evaluation about the latency of solving the QP problem and resizing image regions with larger context regions should be provided, in term of FLOPS.

3. the effect of proposed non-uniform resizing is not abalated on template and search regions separated.  I think it neccessary to find its effects on template and search regions respectively.

QUESTIONS:
plz refer to the weakness

LIMITATIONS:
more ablation studies shoud be conducted.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

comments: SUMMARY:
The paper presents a ZoomTrack, a target-aware non-uniform resizing technique for efficient visual tracking. It narrows the gap between speed-oriented and performance-oriented trackers by non-uniformly resizing the cropped image to have a smaller input size while retaining more raw information for the target. A quadratic programming-based formulation for efficient solving is also designed. The experiments demonstrate the consistent improvements over existing transformer trackers.

STRENGTHS:
It seems the first time to apply the non-uniform resizing to the field of object tracking.

The paper is well written. The authors provide a clear explanation of the proposed method, including the formulation and solution of the resizing problem through quadratic programming.

The results show that it achieves consistent improvements over two Transformer trackers on multiple tracking datasets.

WEAKNESSES:
The idea is similar to "Learning to Zoom and Unzoom" presented at CVPR 2023.

There are many hyperparameters in the proposed method, and how to set or tune their values?

Although the consistent improvements over two Transformer trackers on multiple tracking datasets are achieved, the improvements on LaSOT and TrackingNet are small.

QUESTIONS:
See above weaknesses.

LIMITATIONS:
See above weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple but clever method to improve the accuracy-speed trade-off of state-of-the-art methods for single object tracking. Existing methods can achieve greater accuracy by using higher-resolution images, however this comes at the cost of inference speed. This paper proposes to adopt a non-uniform resizing of the image which preserves a large search region without sacrificing the resolution of the image at the location where the object is expected to appear. Solving for the control points of an axis-aligned grid is formulated as a QP to preserve the bounds of the region while minimizing distortion and approximating a desired zoom in the region where the object is likely to appear. Results on several large datasets with two different trackers show that this simple modification makes a large step towards closing the gap between fast and accurate trackers. The small set of hyper-parameters is fixed for all datasets and both trackers.\n\nSTRENGTHS:\n1. Paper identifies a practical dilemma and proposes a simple but clever solution.\n1. Proposed method can be applied with many different trackers.\n1. The method significantly advances the Pareto front of the AUC vs MACs trade-off, even dominating the best trackers on the TNL2K dataset (Figure 1).\n1. Solving for a non-uniform grid is formulated as a QP with efficient solution.\n1. Small (coordinate-wise) sweep conducted for the key hyper-parameters on large dataset.\n1. Same hyper-parameters used for multiple datasets and multiple methods.\n1. The visualizations show that the target is magnified without being distorted.\n1. The statistics demonstrate that, under the proposed non-uniform resizing, the mean size of the target is increased while its variance is unaffected.\n1. Inclusion of FOVEA as baseline.\n1. Increase in time required by non-uniform resize operation was accounted for.\n\nWEAKNESSES:\n1. My main concern is that the proposed non-uniform resizing may be similar in effect to varying the crop hyper-parameters (method for setting $c_x$ and $c_y$, search-image context factor, template-image size) and there is not a rigorous comparison. Showing that the proposed method (optimized for $\\beta$, $\\gamma$, $\\lambda$) is better than optimizing these hyper-parameters would be much more convincing.\n1. No confidence intervals (error bars). This would be particularly useful for Table 3.\n1. Figure 3 shows that increasing the search-image context factor results in worse accuracy due to the low resolution of the object. However, the drop in accuracy may also be caused by a mismatch in the size of the object in the search and template images. The same applies for decreasing the search-image context factor. I think this study should also consider the template-image size.\n1. The paper did not explain how the non-uniform resizing was applied to the search and template images (question below).\n1. It is concerning that a higher-resolution control grid leads to worse results. This suggests that the QP objective does not capture all necessary factors and instead relies on a coarse grid to avoid over-optimizing the flawed objective.\n1. It seems unnatural to take the square root of the box width in $\\sigma_x = \\sqrt{\\beta \\cdot r^w}$. This would result in $\\sigma$ depending non-linearly on the box size. Is there a reason for this?\n1. Code for experiments not provided.\n\nQUESTIONS:\nPlease provide counter-arguments to (or correct misunderstandings in) the weaknesses above.\n\n1. How is the non-uniform resizing applied to the search and template images? Was it applied to the template image? If so, is the equation solved again with different W and H? Could this result in the object having a different size in the search and template image?\n1. Does FOVEA have a hyper-parameter that would improve its results (or could it be trivially modified to include such a hyper-parameter)? Would the proposed warping still outperform it?\n1. In the experiments, it is stated that the context factor for the search image was increased to 5. This was shown to be optimal for the baseline OSTrack on LaSOT in Figure 3 (~69.5 AUC compared to ~69.0 AUC at 4.5). Table 1 shows OSTrack-256 to achieve 69.1 AUC on LaSOT. This suggests that the optimal context factor was not used for the baseline. Is this because a context factor of 5 has a negative effect on the baseline on other datasets? (OSTrack-Zoom achieved 70.2 AUC, which is less impressive compared to 69.5 than compared to 69.0.)\n\nLess important:\n1. What is the effect of applying ZoomTrack to the performance-oriented trackers like OSTrack-384?\n1. I'm curious about how important it is to re-train the tracker. What is the effect of applying the non-uniform resizing _without_ re-training the tracker?\n\nSuggestion:\n1. If the object was treated as a square, then the QP could be solved once offline for all objects. Would this have a significant negative impact on the method?\n1. Is the solution of the QP always symmetric in x and y?\n\nEdits:\n1. The term \"performance\" is multivalent (may mean speed or accuracy); I suggest \"accuracy\" or \"quality\" instead.\n1. \"throw the question\" -> \"pose the question\"\n1. \"with possibility\" -> \"with probability\"\n1. Citation required for 100MB / 1MB in human vision system.\n1. It is stated that the context factor for the template image is left unchanged while that for the search image is increased to 5. Please state the original values so that the reader does not need to go and find them.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the work. I do not foresee any negative impacts stemming from this specific work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}