SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yutong Kou $^{1,2}$ , Jin Gao $^{1,2*}$ , Bing Li $^{1,5}$ , Gang Wang $^{4*}$ , Weiming Hu $^{1,2,3}$ , Yizheng Wang $^{4}$ , Liang Li $^{4*}$
$^{1}$ State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA  $^{2}$ School of Artificial Intelligence, University of Chinese Academy of Sciences  $^{3}$ School of Information Science and Technology, ShanghaiTech University  $^{4}$ Beijing Institute of Basic Medical Sciences  $^{5}$ People AI, Inc kouyutong2021@ia.ac.cn, {jin.gao,bli,wmhu}@nlpr.ia.ac.cn, liang.li.brain@aliyun.com, g_wang@foxmail.com, yzwang57@sina.com
Recently, the transformer has enabled the speed-oriented trackers to approach state-of-the-art (SOTA) performance with high-speed thanks to the smaller input size or the lighter feature extraction backbone, though they still substantially lag behind their corresponding performance-oriented versions. In this paper, we demonstrate that it is possible to narrow or even close this gap while achieving high tracking speed based on the smaller input size. To this end, we non-uniformly resize the cropped image to have a smaller input size while the resolution of the area where the target is more likely to appear is higher and vice versa. This enables us to solve the dilemma of attending to a larger visual field while retaining more raw information for the target despite a smaller input size. Our formulation for the nonuniform resizing can be efficiently solved through quadratic programming (QP) and naturally integrated into most of the crop-based local trackers. Comprehensive experiments on five challenging datasets based on two kinds of transformer trackers, i.e., OSTrack and TransT, demonstrate consistent improvements over them. In particular, applying our method to the speed-oriented version of OSTrack even outperforms its performance-oriented counterpart by  $0.6\%$  AUC on TNL2K, while running  $50\%$  faster and saving over  $55\%$  MACs. Codes and models are available at https://github.com/Kou-99/ZoomTrack.
In visual tracking, many efforts have been made to improve the discrimination and localization ability of the tracking models, including deeper networks [18, 34], transformer feature fusion [8, 31, 21, 19], joint feature extraction and interaction [33, 7, 9], and so on. However, most of the recent tracking algorithms, including the transformer-based [33, 7, 19, 9], still follow the paradigm proposed by Bertinetto et al. [3], in which a small exemplar image cropped from
Comparison of trackers on TNL2K
Figure 1: Our method consistently improves the OSTrack and TransT baselines with negligible computation.
the first frame is used to locate the target within a large search image cropped from one of the subsequent frames. The crop size, which determines the visual field size, is derived from a reference bounding box plus a margin for context. Both of the crops are uniformly resized to fixed sizes to facilitate the training and testing of the tracker.
Due to the fact that the complexity of the transformers scales quadratically with input size, many transformer trackers [33, 7, 19, 9] propose both speed-oriented versions with the smaller input size or feature extraction backbone and performance-oriented versions with larger ones. Thanks to the enabled larger visual field size or stronger feature representation, the performance-oriented versions always outperform their speed-oriented counterparts in performance, though the tracking speed is severely reduced. For instance, an increased  $1.5 \times$  input size for OSTrack [33] will cause doubled Multiply-Accumulate Operations (MACs) and nearly halved tracking speed. Thus, it is natural to pose the question: Is it possible to narrow or even close this performance gap while achieving high tracking speed based on a smaller input size?
Inspired by the human vision system (HVS), we propose to non-uniformly resize the attended visual field to have smaller input size for visual tracking in this paper. Human retina receives about  $100\mathrm{MB}$  of visual input per second, while only 1 MB of visual data is able to be sent to the central brain [36]. This is achieved by best utilizing the limited resource of the finite number of cones in HVS. More specifically, the density of cone photoreceptors is exponentially dropped with eccentricity (i.e., the deviation from the center of the retina), causing a more accurate central vision for precise recognition and a less accurate peripheral vision for rough localization [36]. This enables us to solve the dilemma of attending to larger visual field while retaining more raw information for the target area despite the smaller input size.
In our formulation for the non-uniform resizing, the area where the target is more likely to appear is magnified to retain more raw information with high resolution and facilitate precise recognition, whereas the area where the target is less likely to appear is shrunk yet preserved to facilitate rough localization when encountering fast motion or tracking drift. The key to our method is to design an efficient and controllable resizing module based on a small controllable grid. On top of this grid, we define a zoom energy to explicitly control the scale of magnification for the important target area, a rigid energy to avoid extreme deformation and a linear constraint to avoid cropping the source image during resizing. The important area is determined by the previous tracking result as a temporal prior. This formulation can be solved efficiently through
Figure 2: We achieve non-uniform resizing by sampling according to a manipulable control grid  $\mathfrak{g}$ , which is generated by solving for best  $d_k^{col}$  and  $d_l^{row}$  that minimize two energies that lead to controllable magnification and less deformation.
quadratic programming (QP), whose solution is used to manipulate the controllable grid. Finally, the ideal non-uniform resizing is achieved by sampling according to the controllable grid (See Fig. 2).
Our method can be easily integrated into plenty of tracking algorithms. We select the popular hybrid CNN-Transformer tracker TransT [8] and one-stream Transformer tracker OSTrack [33] to verify the efficiency and effectiveness of our method. Comprehensive experiments are conducted on five large-scale benchmarks, including GOT-10k [16], LaSOT [11],  $\mathrm{LaSOT}_{ext}$  [12], TNL2k [30], TrackingNet [23]. We observe consistent improvements over the baselines with negligible computational overhead. In particular, we improve the speed-oriented version of OSTrack to achieve  $73.5\%$  AO,  $50.5\%$  AUC and  $56.5\%$  AUC on GOT-10k,  $\mathrm{LASOT}_{ext}$  and TNL2k respectively, which are on par with the performance-oriented counterpart while saving over  $55\%$  MACs and running  $50\%$  faster (see Fig. 1). In other words, the performance gap between speed-oriented and performance-oriented trackers can be narrowed or even closed through our method.
In summary, our contributions are as follows: (i) We propose ZoomTrack, an efficient non-uniform resizing method for visual tracking that bridge the gap between speed-oriented and performance-oriented trackers with negligible computation; (ii) We formulate the non-uniform resizing as an explicitly controlled magnification of important areas with restriction on extreme deformation,
enabling an HVS-inspired data processing with limited resource; (iii) Extensive experiments based on two baseline trackers on multiple benchmarks show that our method achieves consistent performance gains across different network architectures.
Efficient Visual Tracking. A lot of real-world applications require tracking algorithms to have high speed. Consequently, many efforts have been made to improve the efficiency of visual tracking. Yan et al. [32] use NAS to search for a lightweight network architecture for visual tracking. Borsuk et al. [6] design a novel way to incorporate temporal information with only a single learnable parameter which achieves higher accuracy at a higher speed. Blatter et al. [5] design an efficient transformer layer and achieves real-time tracking on CPU. Shen et al. [26] propose a distilled tracking framework to learn small and fast trackers from heavy and slow trackers. Some other works [17, 10] explore the quickly advancing field of vision transformer pre-training [15, 29] and architecture designing [13] to improve both tracking accuracy and efficiency. Previous methods focus on designing either a better network [32, 5, 6, 17, 10] or a new training framework [26] to improve the efficiency of trackers, whereas our work focuses on the non-uniform resizing of the input for the sake of efficiency. Our approach is more general and orthogonal to them.
Non-uniform Resizing for Vision Tasks. Image resizing is a common image processing operation. Yet, the standard uniform resizing is not always satisfactory in many applications. Avidan and Shamir [1] resize an image by repeatedly carving or inserting seams that are determined by the saliency of image regions. Panozzo et al. [24] use axis-aligned deformation generated from an automatic or human-appointed saliency map to achieve content-aware image resizing. Recasens et al. [25] adaptively resize the image based on a saliency map generated by CNN for image recognition. Zheng et al. [37] learn attention maps to guide the sampling of the image to highlight details for image recognition. Thavamani et al. [27] use the sampling function of [25] to re-sample the input image based on temporal and dataset-level prior for autonomous navigation. Bejnordi et al. [2] learn a localization network to magnify salient regions from the unmediated supervision of [37] for video object detection. Thavamani et al. [28] propose an efficient and differentiable warp inversion that allows for mapping labels to the warped image to enable the end-to-end training of dense prediction tasks like semantic segmentation. Existing methods either use time-consuming feature extractors to generate saliency [1, 24, 25, 37] or use heuristic sampling functions that cannot explicitly control the extent of the magnification, which may cause unsatisfactory magnification or extreme deformation [27, 2, 28]. In contrast, our approach directly generates the saliency map or important area using the temporal prior in tracking and proposes a novel sampling method that can achieve controllable magnification without causing extreme deformation.
3 Background and Analysis
3.1 Revisiting Resizing in Deep Tracking
Before introducing our method, we first briefly revisit the resizing operation in the deep tracking pipeline. Given the first frame and its target location, visual trackers aim to locate the target in the subsequent frames using a tracking model  $\phi_{\theta}(\mathbf{z},\mathbf{x})$ , where  $\mathbf{z}$  is the template patch from the first frame with the target in the center and  $\mathbf{x}$  is the search patch from one of the subsequent frames. Both  $\mathbf{z}$  and  $\mathbf{x}$  are generated following a crop-then-resize manner. We next detail the cropping and resizing operations.
Image crop generation. Both the template and search images are cropped based on a reference bounding box  $b$ , which can be the ground truth annotation for the template image or the previous tracking result for the search image. Denote a reference box as  $b = (b^{cx}, b^{cy}, b^{w}, b^{h})$ , then the crop size can be calculated as
$$
W = H = \sqrt {\left(b ^ {w} + (f - 1) \times c ^ {w}\right) \left(b ^ {h} + (f - 1) \times c ^ {h}\right)}, \tag {1}
$$
where  $f$  is the context factor controlling the amount of context to be included in the visual field, and  $c^w$  and  $c^h$  denote unit context amount which is solely related to  $b^w$  and  $b^h$ . Typical choice of unit context amount is  $c^w = b^w$ ,  $c^h = b^h$  [33, 31, 9], or  $c^w = c^h = (b^w + b^h) / 2$  [8, 19]. A higher context factor means a larger visual field and vice versa. Usually, the search image has a larger
Figure 3: Experiments on LaSOT [11] based on OSTrack [33] show that: (left) Thanks to the larger attended visual field, simultaneously increasing search size and context factor leads to consistent performance improvement at the cost of heavy computational burden; (right) Increasing the attended visual field by enlarging context factor while fixing the input size to limit the available resources leads to degraded performance due to the decreased target area resolution.
context factor for a larger visual field, and the template image has a small context factor providing minimal necessary context. The image crop  $\mathcal{I}$  is obtained by cropping the area centered at  $(b^{cx}, b^{cy})$  with width  $W$  and height  $H$ . Areas outside the image are padded with a constant value. Finally, box  $b$  is transformed to a new reference bounding box  $r = (r^{cx}, r^{cy}, r^{w}, r^{h})$  on the image crop  $\mathcal{I}$ .
Image crop resizing. To facilitate the batched training and later online tracking, the image crop has to be resized to a fixed size. Given an image crop  $\mathcal{I}$  with width  $W$  and height  $H$ , a fix-sized image patch  $\mathcal{I}'$  with width  $w$  and height  $h$  is obtained by resizing  $\mathcal{I}$ . Specifically,  $h \times w$  pixels in  $\mathcal{I}'(x', y')$  are sampled from  $\mathcal{I}(x, y)$  according to a mapping  $\mathcal{T}: \mathbb{R}^2 \to \mathbb{R}^2$ , i.e.,
$$
\mathcal {I} ^ {\prime} \left(x ^ {\prime}, y ^ {\prime}\right) = \mathcal {I} \left(\mathcal {T} \left(x ^ {\prime}, y ^ {\prime}\right)\right). \tag {2}
$$
Note that  $\mathcal{T}$  does not necessarily map integer index to integer index. Thus, some pixels in  $\mathcal{I}'$  may be sampled from  $\mathcal{I}$  according to the non-integer indices. This can be realized by bilinear interpolating from nearby pixels in  $\mathcal{I}$ . Current methods [33, 9, 19, 7] use uniform mapping  $\mathcal{T}_{\text{uniform}}$  to resize  $\mathcal{I}$ .  $\mathcal{T}_{\text{uniform}}$  is defined as  $\mathcal{T}_{\text{uniform}}(x', y') = \left(\frac{x'}{s^x}, \frac{y'}{s^y}\right)$ , where  $s^x = w / W$ ,  $s^y = h / H$  are called resizing factors and indicate the scaling of the image. When applying uniform resizing, the resizing factor is the same wherever on the image crop, which means the same amount of amplification or shrinkage is applied to the different areas of the entire image.
3.2 Solving the Dilemma Between Visual Field and Target Area Resolution
Although the crop-then-resize paradigm is applied in most of the popular deep trackers, the fixed input search size and context factor vary significantly across different algorithms. Generally, the performance can be improved by using a larger input size at the cost of decreasing tracking speed [19, 33]. To understand the key factors behind this, we conduct a series of experiments on LaSOT [11] based on OSTrack [33]. First, we simultaneously increase search size and context factor, which means the target resolution is roughly fixed and the tracker attends to a larger visual field. As shown in the left of Fig. 3, the AUC on LaSOT increases along with the input search size thanks to the larger attended visual field. Note that increasing the search size  $(256\rightarrow 384)$  leads to almost doubled computational overhead (21.5G MACs  $\rightarrow$  41.5G MACs). Next, we limit the available resources by fixing the input size while increasing the attended visual field size. As shown in the right of Fig. 3, the AUC on LaSOT first increases and then decreases as the average target resolution<sup>2</sup> keeps decreasing. This result demonstrates that the benefit of an enlarged visual field is gradually wiped out by the decrease in target resolution when the computational cost is fixed.
Inspired by the data processing with limited resources in HVS, we believe the above dilemma of attending to a larger visual field while retaining more raw information for the target can be solved by replacing uniform resizing with non-uniform resizing. In visual tracking, the previous tracking result can serve as a strong temporal prior for the current target location. Based on this prior, we can determine the area where the target is likely to appear and magnify it to retain more raw information with high resolution. On the contrary, areas where the target is less likely to appear is shrunk to avoid decreasing the visual field when the input search size is fixed with limited resources. Notably, the
2The average resolution is calculated by averaging the sizes of the ground truth bounding boxes on the crop-then-resized search patches at frames that do not have full-occlusion or out-of-view attributes.
magnification and shrinkage should not dramatically change the shape and aspect ratio of regions on the image to facilitate the robust learning of the appearance model. Based on the above analysis, we propose some guidelines for our non-uniform resizing, i.e., G1: Magnify the area where the target is most likely to appear; G2: Avoid extreme deformation; G3: Avoid cropping the original image  $\mathcal{I}$  so that the original visual field is also retained. We detail how to incorporate these guidelines into our non-uniform resizing in the following section.
As discussed in Sec. 3.1, the resizing process from the source image  $\mathcal{I}(x,y) \in \mathbb{R}^{H \times W \times 3}$  to the target image  $\mathcal{I}'(x',y') \in \mathbb{R}^{h \times w \times 3}$  is a sampling operation controlled by a mapping  $\mathcal{T}: (x',y') \to (x,y)$ . Our aim is to find the mapping  $\mathcal{T}$  that best follows the guidelines (G1~G3), given a temporal prior box  $r = (r^{cx}, r^{cy}, r^{w}, r^{h})$  that indicates the most possible location of the target. We first restrict the domain of the mapping to a few points and acquire a grid representation of the mapping  $\mathcal{T}$ . Then, we formulate the guidelines as a QP problem based on the grid point intervals. By solving the QP problem via a standard solver [20], we can efficiently manipulate the controllable grid point intervals and achieve the ideal non-uniform resizing by sampling according to the final grid representation.
4.1 Grid Representation for Non-uniform Resizing
Recall the sampling operation in Eq. (2), pixel  $(x', y')$  on the target image  $\mathcal{I}' \in \mathbb{R}^{h \times w \times 3}$  is sampled from location  $(x, y)$  on the source image  $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$ . Thus, at least  $h \times w$  pairs of  $(x', y') \to (x, y)$  are needed for resizing. We use the exact  $h \times w$  pairs to represent the mapping  $\mathcal{T}$ , which can be defined as a grid  $\mathcal{G} \in \mathbb{R}^{h \times w \times 2}$ , where  $\mathcal{G}[y'][x'] = (x, y)$ ,  $x' = 1, \dots, w$ ,  $y' = 1, \dots, h$ , is the location on  $\mathcal{I}$  that the target image pixel  $\mathcal{I}'(x', y')$  should be sampled from.
Controllable grid. The grid  $\mathcal{G}$  is an extremely dense grid with  $h\times w$  grid points (equivalent to the resolution of the target image  $\mathcal{T}'$ ), which makes it computationally inefficient to be directly generated. To solve this issue, we define a small-sized controllable grid  $\mathfrak{g}\in \mathbb{R}^{(m + 1)\times (n + 1)\times 2}$  with  $m + 1$  rows and  $n + 1$  columns ( $m < h, n < w$ ) to estimate  $\mathcal{G}$ , where
$$
\mathfrak {g} [ j ] [ i ] = \mathcal {G} \left[ \frac {h}{m} j \right] \left[ \frac {w}{n} i \right], i = 0, \dots , n, j = 0, \dots , m \tag {3}
$$
Intuitively,  $\mathfrak{g}[j][i]$  is the sample location on the source image for the target image pixel  $\left(\frac{w}{n} i, \frac{h}{m} j\right)$  in  $\mathcal{I}'$ . Once  $\mathfrak{g}$  is determined,  $\mathcal{G}$  can be easily obtained through bilinear interpolation on  $\mathfrak{g}$ .
Axis-alignment constraint. To avoid extreme deformation (G3) and reduce computational overhead, we add an axis-aligned constraint [27] to the grid  $\mathfrak{g}$ . Specifically, grid points in the same row  $j$  have the same y-axis coordinate  $y_{j}$ , and grid points in the same column  $i$  have the same x-axis coordinate  $x_{i}$ . That is to say  $\mathfrak{g}[j][i] = (x_i,y_j)$ ,  $i = 0,\dots,n$ ,  $j = 0,\dots,m$ . The reason why we use such an axis-aligned constraint is that we need to keep the axis-alignment of the bounding box (i.e., the four sides of the bounding box are parallel to the image boundary) after resizing, which is proven to be beneficial to the learning of object localization [27].
Grid representation. The source image is split into small rectangular patches  $p(k,l), k = 1, \ldots, n, l = 1, \ldots, m$  by grid points. The top-left corner of  $p(k,l)$  is  $\mathfrak{g}[l - 1][k - 1]$  and its bottom-right corner is  $\mathfrak{g}[l][k]$ . The width and height of the patch  $p(k,l)$  are denoted as the horizontal grid point interval  $d_k^{col}$  and the vertical grid point interval  $d_l^{row}$ .  $d_k^{col}$  and  $d_l^{row}$  can be calculated as  $d_k^{col} = x_k - x_{k-1}$ ,  $d_l^{row} = y_l - y_{l-1}$ . The axis-aligned controllable grid  $\mathfrak{g}$  can be generated from grid point intervals  $d_l^{row}$ ,  $d_k^{col}$  based on
$$
\mathfrak {g} [ j ] [ i ] = \left(\sum_ {k = 0} ^ {i} d _ {k} ^ {\text {c o l}}, \sum_ {l = 0} ^ {j} d _ {l} ^ {\text {r o w}}\right), \tag {4}
$$
where  $d_0^{row} = d_0^{col} = 0$  for the sake of simplicity.
Non-uniform resizing. According to Eq. (3), the resizing process can be visualized using two grids (see Fig. 2): a manipulatable grid on the source image  $\mathfrak{g}$  and a fixed grid on the target image  $\mathfrak{g}_t[j][i] = \left(\frac{w}{n} i, \frac{h}{m} j\right)$ .  $\mathfrak{g}$  and  $\mathfrak{g}_t$  split the source and target images into rectangular patches  $p(k,l)$  and  $p_t(k,l)$  respectively. Target image pixel at  $\mathfrak{g}_t[j][i]$  is sampled from  $\mathfrak{g}[j][i]$  on the source image. This
process can be intuitively understood as resizing  $p(k, l)$  to  $p_t(k, l)$ . When the grid point intervals of  $\mathfrak{g}$  are manipulated, the grid point intervals of  $\mathfrak{g}_t$  remain unchanged. Thus, by reducing/ increasing the grid intervals of  $\mathfrak{g}$ , the corresponding region on the target image is amplified/ shrunk. In this way, a non-uniform resizing operation that scales different image areas with different ratios can be obtained by dynamically manipulating the grid point intervals.
4.2 QP Formulation Based on Grid Representation
Given a reference bounding box  $r = (r^{cx}, r^{cy}, r^w, r^h)$ , we aim to dynamically manipulate the grid point intervals to best follow the guidelines (G1~G3) proposed in Sec. 3.2.
Importance score for the target area. According to G1, areas, where the target is most likely to appear, should be magnified. Following this guideline, we first define the importance score for the target area on the source image. An area has high importance if the target is more likely to appear in this area. As  $r$  is a strong temporal prior for the current target location, we use a Gaussian function  $G(x,y)$  centered at  $(r^{cx},r^{cy})$  as the importance score function, i.e.,
$$
G (x, y) = \exp \left(- \frac {1}{2} \left(\frac {(x - \mu_ {x}) ^ {2}}{\sigma_ {x} ^ {2}} + \frac {(y - \mu_ {y}) ^ {2}}{\sigma_ {y} ^ {2}}\right)\right), \tag {5}
$$
where  $\mu_x = r^{cx}, \mu_y = r^{cy}, \sigma_x = \sqrt{\beta \times r^w}, \sigma_y = \sqrt{\beta \times r^h}$ .  $\beta$  is a hyper-parameter controlling the bandwidth of the Gaussian function. Since we denote the rectangular area enclosed by the grid point intervals  $d_l^{row}$  and  $d_k^{col}$  as patch  $p(k, l)$ , its importance score  $S_{k,l}$  can be determined by the value of  $G(x, y)$  at the center of patch  $p(k, l)$ , i.e.,
$$
S _ {k, l} = G \left(\left(k + \frac {1}{2}\right) \times \frac {W}{n}, \left(l + \frac {1}{2}\right) \times \frac {H}{m}\right) + \epsilon , \tag {6}
$$
where  $\epsilon$  is a small constant to prevent extreme deformation and ensure stable computation. Here we use the patch center of a uniformly initialized grid ( $d_k^{col} = \frac{W}{n}$  and  $d_l^{row} = \frac{H}{m}$ ). By doing so,  $S_{k,l}$  is irrelevant of  $d_k^{col}$  and  $d_l^{row}$ , which is crucial for the QP formulation in the following.