SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zoom energy. Following G1, we design a quadratic energy  $E_{zoom}$  to magnify the area where the target is most likely to appear. To achieve controllable magnification, we define a zoom factor  $\gamma$  controlling the amount of magnification. If we want to magnify some area by  $\gamma$ , the distances between sampling grid points should shrink by  $\frac{1}{\gamma}$ .  $E_{zoom}$  forces  $d_k^{col}$  and  $d_l^{row}$  to be close to the grid point intervals under uniform magnification by  $\gamma$ , i.e.,  $\frac{1}{\gamma}\frac{H}{m}$  for  $d_l^{row}$  and  $\frac{1}{\gamma}\frac{W}{n}$  for  $d_k^{col}$ . In detail, The zoom energy is defined as
$$
E _ {z o o m} = \sum_ {l = 1} ^ {m} \sum_ {k = 1} ^ {n} S _ {k, l} ^ {2} \left(\left(d _ {l} ^ {r o w} - \frac {1}{\gamma} \frac {H}{m}\right) ^ {2} + \left(d _ {k} ^ {c o l} - \frac {1}{\gamma} \frac {W}{n}\right) ^ {2}\right). \tag {7}
$$
Rigid energy. To avoid extreme deformation (G2), we define a quadratic energy  $E_{\text{rigid}}$  to restrict the aspect ratio change of patch  $p(k,l)$ , which has width  $d_k^{col}$  and height  $d_l^{row}$ . When being uniformly resized,  $p(k,l)$  should have width  $\frac{W}{n}$  and height  $\frac{H}{m}$ . Thus, the patch  $p(k,l)$  is horizontally stretched by  $\frac{n}{W} d_k^{col}$  and vertically stretched by  $\frac{m}{H} d_l^{row}$ . To keep the aspect ratio mostly unchanged for the important area, the horizontal stretch should be similar to the vertical stretch. Therefore, the rigid energy can be defined as
$$
E _ {r i g i d} = \sum_ {l = 1} ^ {m} \sum_ {k = 1} ^ {n} S _ {k, l} ^ {2} \left(\frac {m}{H} d _ {l} ^ {r o w} - \frac {n}{W} d _ {k} ^ {c o l}\right) ^ {2}. \tag {8}
$$
Linear constraint to avoid cropping the source image. According to G3, the resizing should not crop the source image so that the original visual field is also retained. That is to say, the grid  $\mathfrak{g}$  should completely cover the entire source image. This requirement can be interpreted as a simple linear constraint that forces the sum of vertical grid point intervals  $d_l^{row}$  to be  $H$  and the sum of horizontal grid point intervals  $d_k^{col}$  to be  $W$ :  $\sum_{l=1}^{m} d_l^{row} = H$ ,  $\sum_{k=1}^{n} d_k^{col} = W$ .
QP formulation. Combining the zoom energy, the rigid energy and the linear constraint, we can formulate the grid manipulation as the following minimization task
$$
\underset {d _ {l} ^ {r o w}, d _ {k} ^ {c o l}} {\text {m i n i m i z e}} E = E _ {z o o m} + \lambda E _ {r i g i d}
$$
$$
\text {s u b j e c t} \quad \sum_ {l = 1} ^ {m} d _ {l} ^ {\text {r o w}} = H, \sum_ {k = 1} ^ {n} d _ {k} ^ {\text {c o l}} = W \tag {9}
$$
where  $\lambda$  is a hyper-parameter to balance the two energies. This convex objective function can be efficiently solved by a standard QP solver [20] (see Appendix A for more detailed derivations).
4.3 Integration with Common Trackers
Reference box generation. Our non-uniform resizing method needs a temporal prior reference box  $r = (r^{cx}, r^{cy}, r^w, r^h)$  on the cropped source image  $\mathcal{I}$ . During testing,  $r$  is directly generated from the previous frame tracking result (see Sec. 3.1). During training, we only need to generate  $r^w$  and  $r^h$  from the ground truth width  $g^w$  and height  $g^h$  as the center of the temporal prior reference box is fixed at  $r^{cx} = W / 2$  and  $r^{cy} = H / 2$ . In detail, with probability  $q$  we use a smaller jitter  $j = j_s$ , and with probability  $1 - q$  we use a larger jitter  $j = j_l$ .  $r^w$  and  $r^h$  are calculated as  $r^w = e^{J^w}g^w$ ,  $r^h = e^{J^h}g^h$ , where  $J^w$ ,  $J^h \sim \mathcal{N}(0, j^2)$ .
Label mapping. The classification loss of the network  $\phi_{\theta}(\mathbf{z},\mathbf{x})$  is calculated on the target image  $\mathcal{I}'$ . Therefore, the classification labels on  $\mathcal{I}'$  should be generated from the ground truth bounding box on  $\mathcal{I}'$ . The common practice is to use normalized ground truth bounding box for the reason that the normalized coordinates remain the same on  $\mathcal{I}$  and  $\mathcal{I}'$  under uniform resizing. However, this does not hold true for our non-uniform resizing. Therefore, we need to map the ground truth bounding box from  $\mathcal{I}$  to  $\mathcal{I}'$  via the grid  $\mathcal{G}$ . Specifically, we map the top-left corner and bottom-right corner of the bounding box as follows. Given a point on the source image  $(x,y)$ , we first find two grid points on  $\mathcal{G}$  that are closest to  $(x,y)$ . Then, the corresponding point  $(x',y')$  can be obtained by bilinear interpolation between the indices of these two points.
Reverse box mapping. The bounding box predicted by the network  $\phi_{\theta}(\mathbf{z},\mathbf{x})$  is on  $\mathcal{I}'$  and needed to be mapped back to  $\mathcal{I}$  as the final tracking result. We map the top-left corner and bottom-right corner of the predicted bounding box from  $\mathcal{I}'$  to  $\mathcal{I}$  in a reversed process of label mapping. Specifically, given a point  $(x',y')$  on  $\mathcal{I}'$ , we first find the two grid points of  $\mathcal{G}$  whose indices are closest to  $(x',y')$ , and then the corresponding point  $(x,y)$  can be obtained by bilinear interpolation between the values of these points. To minimize the impact of the reverse mapping, we directly calculate the regression loss on  $\mathcal{I}$  instead of on  $\mathcal{I}'$ .
We apply our method to both one-stream tracker OSTrack [33] (denoted as OSTrack-Zoom) and CNN-Transformer tracker TransT [8] (denoted as TransT-Zoom). The proposed non-uniform resizing module is applied in both the training and testing stages on the search image. We do not apply the proposed non-uniform resizing on the template image because it degrades performance (see Sec. 5.3). The search patch size  $(256 \times 256)$  is the same as the baseline methods, whereas the context factor is enlarged from 4 to 5. We use a controllable grid  $\mathfrak{g}$  with shape  $17 \times 17 (m = n = 16)$ . The importance score map is generated by a Gaussian function with bandwidth  $\beta = 64$ . We set magnification factor  $\gamma = 1.5$  and  $\lambda = 1$  in Eq. (9). We use smaller jitter  $j_{s} = 0.1$  with probability 0.8 and larger jitter  $j_{l} = 0.5$  with probability 0.2. Notably, we use the same set of parameters for both OSTrack-Zoom and TransT-Zoom to demonstrate the generalization capability of our method. Except for the resizing operation, all the other training and testing protocols follow the corresponding baselines. The trackers are trained on four NVIDIA V100 GPUs. The inference speed and MACs are evaluated on a platform with Intel i7-11700 CPU and NVIDIA V100 GPU.
5.2 State-of-the-art Comparison
We compare our method with the state-of-the-art trackers on five challenging large-scale datasets: GOT-10k [16], LaSOT [11], LaSOText [12], TNL2K [30] and TrackingNet [23].
Table 1: State-of-the-art Comparison on GOT-10k, LaSOT,  $\mathrm{LaSOT}_{ext}$ , TNL2K and TrackingNet.
Tracker	Size	GOT-10k	LaSOT	\( LaSOT_{ext} \)	TNL2K	TrackingNet	MACs (G)	FPS
AO	\( SR_{0.5} \)	\( SR_{0.75} \)	AUC	P	AUC	P	AUC	P	SUC	P
Baseline &Ours	OSTrack-Zoom	256	73.5	83.6	70.0	70.2	76.2	50.5	57.4	56.5	57.3	83.2	82.2	21.5	100
OSTrack-256[33]	256	71.0	80.4	68.2	69.1	75.2	47.4	53.3	54.3	-	83.1	82.0	21.5	119
TransT-Zoom	255	67.5	77.6	61.3	67.1	71.6	46.8	52.9	53.7	62.3	81.8	80.2	19.2	45
TransT[8]	255	67.1	76.8	60.9	64.9	69.0	44.8	52.5	50.7	51.7	81.4	80.3	19.2	48
Speed-oriented	SwinTrack-T[19]	224	71.3	81.9	64.5	67.2	70.8	47.6	53.9	53.0	53.2	81.1	78.4	6.4	98
SimTrack-B[7]	224	68.6	78.9	62.4	69.3	-	-	-	54.8	53.8	82.3	-	25.0	40
MixFormer-22k[9]	320	70.7	80.0	67.8	69.2	74.7	-	-	-	-	83.1	81.6	23.0	25
ToMP-101[21]	288	-	-	-	68.5	73.5	45.9	-	-	-	81.5	78.9	-	20
TransInMo*[14]	255	-	-	-	65.7	70.7	-	-	52.0	52.7	81.7	-	16.9	34
Stark-ST101[31]	320	68.8	78.1	64.1	67.1	-	-	-	-	-	82.0	-	28.0	32
AutoMatch[35]	255	65.2	76.6	54.3	58.3	59.9	37.6	43.0	47.2	43.5	76.0	72.6	-	50
DiMP[4]	288	61.1	71.7	49.2	56.9	56.7	39.2	45.1	44.7	43.4	74.0	68.7	5.4	40
SiamRPN++[18]	255	51.7	61.6	32.5	49.6	49.1	34.0	39.6	41.3	41.2	73.3	69.4	7.8	35
Performance-oriented	OSTrack-384[33]	384	73.7	83.2	70.8	71.1	77.6	50.5	57.6	55.9	-	83.9	83.2	48.3	61
SwinTrack-B[19]	384	72.4	80.5	67.8	71.3	76.5	49.1	55.6	55.9	57.1	84.0	82.8	69.7	45
SimTrack-L[7]	224	69.8	78.8	66.0	70.5	-	-	-	55.6	55.7	83.4	-	95.4	-
MixFormer-L[9]	320	-	-	-	70.1	76.3	-	-	-	-	83.9	83.1	127.8	18
GOT-10k [16] provides training and test splits with zero-overlapped object classes. Trackers are required to be only trained on GOT-10k training split and then evaluated on the test split to examine their generalization capability. We follow this protocol and report our results in Tab. 1. Our OSTrack-Zoom achieves  $73.5\%$  AO which surpasses the baseline OSTrack by  $2.5\%$ . And our TransT-Zoom achieves  $67.5\%$  AO which surpasses the baseline TransT by  $0.4\%$ . Compared with previous speed-oriented trackers, our OSTrack-Zoom improves all of the three metrics in GOT-10k by a large margin, proving that our method has good generalization capability.
LaSOT [11] is a large-scale dataset with 280 long-term video sequences. As shown in Tab. 1, our method improves the AUC of OSTrack and TransT by  $1.1\%$  and  $2.2\%$  respectively. Our method OSTrack-Zoom have the best performance among speed-oriented trackers on LaSOT, indicating that our method is well-suited for complex object motion in long-term visual tracking task.
$\mathbf{LaSOT}_{ext}$  [12] is an extension of LaSOT with 150 additional videos containing objects from 15 novel classes outside of LaSOT. In Tab. 1, our method achieves  $3.1\%$  and  $2.0\%$  relative gains on AUC metric for OSTrack and TransT baselines respectively. Our method OSTrack-Zoom achieves promising  $50.5\%$  AUC and  $57.4\%$  Precision, outperforming other methods by a large margin.
TNL2K [30] is a newly proposed dataset with new challenging factors like adversarial samples, modality switch, etc. As reported in Tab. 1, our method improves OSTrack and TransT by  $2.2\%$  and  $3.0\%$  on AUC. Our OSTrack-Zoom sets the new state-of-the-art performance on TNL2K with  $56.5\%$  AUC while running at 100 fps.
TrackingNet [23] is a large-scale short-term tracking benchmark. Our method boosts the performance of OSTrack and TransT by  $0.1\%$  and  $0.4\%$  SUC. The reason for the relatively small improvements is that, contrary to other datasets, TrackingNet has fewer sequences with scenarios that can benefit from an enlarged visual field, e.g.scenarios with drastic movement or significant drift after some period of time. Please refer to Appendix F for a detailed analysis.
Comparison with performance-oriented trackers. We additionally compare our method with the performance-oriented versions of the SOTA trackers. On the one hand, on GOT-10k,  $\mathrm{LaSOT}_{ext}$ , TNL2K whose test splits object classes have no or few overlap with training data, our OSTrack-Zoom is on-par with the SOTA tracker OSTrack-384, while consuming only  $44.5\%$  MACs of the latter and running  $50\%$  faster. On the other hand, on LaSOT and TrackingNet whose test splits have many object classes appearing in training data, our OSTrack-Zoom is slightly inferior to some SOTA trackers. A possible reason is that these SOTA trackers with heavier networks fit the training data better, which is
Table 2: Comparison of accuracy, latency and target size using different resizing methods with OSTrack [33].
#	Method	Size	TNL2K [30]	LaSOT [11]	Target Size (103)	Latency (ms)	FPS
AUC	P	AUC	P	avg↑	std↓	resize	net
①	Uniform	256	55.5	55.4	69.5	75.2	2.7	24.8	0.04	8.40	119
②	Uniform	320	56.1	56.8	69.9	76.2	4.2	38.8	0.04	10.92	92
③	FOVEA	256	54.2	54.5	67.6	73.3	4.4	40.8	3.37	8.40	85
④	Ours	256	56.5	57.3	70.2	76.2	4.9	25.0	1.58	8.40	100
Table 3: Ablation of hyper-parameters on LaSOT [11] with OSTrack [33].
(a) Bandwidth β
β	4	64	256
AUC	69.1	70.2	69.6
(c) Zoom factor γ
γ	1.25	1.5	1.75
AUC	69.7	70.2	69.4
(b) Control grid size
Size	8	16	32
AUC	66.4	70.2	69.5
(d) Balance factor λ
λ	0	1	2
AUC	69.8	70.2	69.6
Table 4: Ablation on the effect of applying non-uniform resizing in training and testing stages. Results are reported in AUC (%) metric.
#	Non-uniform resizing	OSTrack[33]	TransT[8]
Testing	Training	LaSOT[11]	TNL2K[30]	LaSOT[11]	TNL2K[30]
①	69.1	54.3	64.9	50.7
②	✓	69.1	55.9	66.3	53.6
③	✓	✓	70.2	56.5	67.1	53.7
Table 5: Ablation on the effect of applying non-uniform resizing on the template and search images with OSTrack [33].
#	Non-uniform Resizing	LaSOT[11]
Template	Search	AUC	P
①	✓	64.8	71.0
②	✓	70.2	76.2
③	✓	✓	69.6	75.5
beneficial to track objects with classes that have been seen during training. Nevertheless, the gap between speed-oriented and performance-oriented trackers is narrowed using our method.
5.3 Ablation Experiment
Comparison with other resizing methods. As shown in Tab. 2, we compare our non-uniform resizing method (4) with uniform resizing with small/large input sizes  $(1/2)$  and FOVEA [27] non-uniform resizing (3). On both TNL2K and LaSOT, our method outperforms other methods. Notably, while applying FOVEA causes a significantly degraded performance  $(1vs.3)$ , our method can achieve large performance gains  $(1vs.4)$ . To understand the reason behind this, we calculate the average target size (avg) and the standard deviation of the target size (std), which represent the degree of the targets' scale change, on LaSOT. Compared with uniform resizing  $(1vs.4)$ , our method achieves  $1.34^2 \times$  magnification on average target size, while the standard deviation almost keeps the same. The magnification is close to the desired  $\gamma^2 = 1.5^2$ , showing that our method achieves controllable magnification. In contrast to FOVEA  $(3vs.4)$ , which severely reduces the tracking performance, our standard deviation is almost unchanged, demonstrating that our method can avoid extreme deformation that severely changes the scale of the target.
Computational overhead analysis. Our resizing module runs purely on CPU. The latency for resizing an image is  $1.58\mathrm{ms}$  on an Intel i7-11700 CPU. Compared with uniform resizing (①vs.④), our method introduces an additional  $1.54\mathrm{ms}$  latency during resizing, causing a slightly slower speed. However, our performance is better than the uniform resizing counterparts. Especially when compared with uniform resizing with a larger input size (②vs.④), our method reaches higher performance while running faster, demonstrating the superiority of using non-uniform resizing in visual tracking. Our method is also faster than FOVEA resizing while having a better performance. Moreover, the MACS for our resizing an image is  $1.28\mathrm{M}$  MACS  $(10^{6})$ , which is negligible compared to the network inference which is usually tens of G MACs  $(10^{9})$ .
Resizing in different stages. It is worth noting that our method can be directly applied to off-the-shell tracking models, i.e., only applying non-uniform resizing method at testing time. Results on LaSOT [11] and TNL2K [30] are reported in Tab. 4 using the AUC (\%) metric. As shown in Tab. 4, directly applying our method to the off-the-shelf trackers only in testing can already improve the tracking accuracy (①vs.②). Further applying our resizing in training can also boost the performance, owning to the aligned training and testing protocols (②vs.③).
Resizing template and search images separately. We also investigate the effects of applying the non-uniform resizing on the template and search images separately. As shown in Tab. 5 either applying non-uniform resizing on the template image alone or on both the template and search images performs worse than applying it only on the search image. We think the reason is that the template
Figure 4: Visualization of different resizing. For each set of images, from left to right: ours, uniform resizing, FOVEA [27] resizing. The targets are marked in red box. (left) Our method can magnify the target without drastically changing the appearance of the target. (right) Our method can magnify the target without significantly changing the aspect ratio of the target.
image must have a relatively small context factor to contain minimal necessary context, which makes the whole template region important for the tracking network to locate the target. Therefore, the tracker cannot benefit from non-uniform resizing of the template image.
More ablations. First, we study the impact of the bandwidth  $\beta$  used in the importance function. A smaller  $\beta$  means less area is considered to be important and vice versa. As shown in Tab. 3a, the ideal value for  $\beta$  is 64. Both smaller bandwidth (4) and larger bandwidth (256) cause reduced performance, showing the importance of this factor. Second, we also analyze the impact of the controllable grid. As shown in Tab. 3b,  $m = n = 16$  is the best size for this grid. A smaller grid size causes a drastic drop in AUC, showing that too small grid size will cause inaccurate resizing that damages the tracking performance. A larger grid size also degrades performance, for which we believe the reason is that a large grid size will lead to an over-reliance on the important area generation based on the temporal prior, which is however not always correct. Third, we further look into the impact of different zoom factors  $\gamma$ . As shown in Tab. 3c, 1.5 is the best value for  $\gamma$ . A smaller  $\gamma$  will degrade performance due to smaller target resolution. Meanwhile, a larger  $\gamma$  is also harmful to the performance as larger magnification will cause trouble when trying to rediscover the lost target. Finally, we study the impact of  $E_{rigid}$  by changing the balance factor  $\lambda$ . As shown in Tab. 3d, setting  $\lambda = 1$  reaches the best performance. When  $\lambda = 0$ , the performance drops, demonstrating the effectiveness of the proposed  $E_{energy}$ . A larger  $\lambda = 2$  also degrades the performance, indicating that too much constraint is harmful to the performance.
Visualization. We visualize our proposed non-uniform resizing, uniform resizing and FOVEA [27] resizing in Fig. 4. Compared with uniform resizing, our method can improve the resolution of the target to retain more raw information. Compared with FOVEA, our method can better preserve the aspect ratio and the appearance of the target.
In this paper, we presented a novel non-uniform resizing method for visual tracking, which efficiently improves the tracking performance by simultaneously attending to a larger visual field and retaining more raw information with high resolution. Inspired by the HVS data processing with limited resources, our method formulates the non-uniform resizing as an explicitly controlled magnification of important areas with restriction on extreme deformation. Extensive experiments have demonstrated that our method can bridge the gap between speed-oriented and performance-oriented trackers with negligible computational overhead.
Limitations. One limitation of our method is that we need a fixed context factor to determine the size of the visual field. It would be interesting to develop a method to dynamically adjust the size of the visual field according to the tracking trajectory or appearance of the target in future work.
Acknowledgements. The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. This work was supported in part by the National Key R&D Program of China (Grant No. 2020AAA0106800), the Natural Science Foundation of China (Grant No. U22B2056, 61972394, 62036011, 62192782, 61721004, U2033210), the Beijing Natural Science Foundation (Grant No. L223003, JQ22014, 4234087), the Major Projects of Guangdong Education Department for Foundation Research and Applied Research (Grant No. 2017KZDXM081, 2018KZDXM066), the Guangdong Provincial University Innovation Team Project (Grant No. 2020KCXTD045). Jin Gao and Bing Li were also supported in part by the Youth Innovation Promotion Association, CAS.
[23] Matthias Müller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. TrackingNet: A large-scale dataset and benchmark for object tracking in the wild. In ECCV, 2018.
[24] Daniele Panozzo, Ofir Weber, and Olga Sorkine. Robust image retargeting via axis-aligned deformation. In EUROGRAPHICS, 2012.
[25] Adrià Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, and Antonio Torralba. Learning to zoom: A saliency-based sampling layer for neural networks. In ECCV, 2018.
[26] Jianbing Shen, Yuanpei Liu, Xingping Dong, Xiankai Lu, Fahad Shahbaz Khan, and Steven Hoi. Distilled siamese networks for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (12):8896-8909, 2021.
[27] Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan. FOVEA: Foveated image magnification for autonomous navigation. In ICCV, 2021.
[28] Chittesh Thavamani, Mengtian Li, Francesco Ferroni, and Deva Ramanan. Learning to zoom and unzoom. In CVPR, 2023.
[29] Shaoru Wang, Jin Gao, Zeming Li, Xiaogin Zhang, and Weiming Hu. A closer look at self-supervised lightweight vision transformers. In ICML, 2023.
[30] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In CVPR, 2021.
[31] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In ICCV, 2021.
[32] Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Hutchuan Lu. LightTrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In CVPR, 2021.
[33] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In ECCV, 2022.
[34] Zhipeng Zhang and Houwen Peng. Deeper and wider siamese networks for real-time visual tracking. In CVPR, 2019.
[35] Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu. Learn to match: Automatic matching network design for visual tracking. In ICCV, 2021.
[36] Li Zhaoping. A new framework for understanding vision from the perspective of the primary visual cortex. Current opinion in neurobiology, 58:1-10, 2019.
[37] Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition. In CVPR, 2019.
A Derivations of the QP-based Minimization Task for Grid Manipulation
The minimization task for grid manipulation in our non-uniform resizing is defined as
$$
\underset {d _ {l} ^ {r o w}, d _ {k} ^ {c o l}} {\text {m i n i m i z e}} \quad E = E _ {z o o m} + \lambda E _ {r i g i d}
$$
$$
\text {s u b j e c t} \quad \sum_ {l = 1} ^ {m} d _ {l} ^ {\text {r o w}} = H, \sum_ {k = 1} ^ {n} d _ {k} ^ {\text {c o l}} = W \tag {A1}
$$
where the zoom energy  $E_{zoom}$  is defined as
$$
E _ {z o o m} = \sum_ {l = 1} ^ {m} \sum_ {k = 1} ^ {n} S _ {k, l} ^ {2} \left(\left(d _ {l} ^ {r o w} - \frac {1}{\gamma} \frac {H}{m}\right) ^ {2} + \left(d _ {k} ^ {c o l} - \frac {1}{\gamma} \frac {W}{n}\right) ^ {2}\right), \tag {A2}
$$
and the rigid energy  $E_{\mathrm{rigid}}$  is defined as
$$
E _ {r i g i d} = \sum_ {l = 1} ^ {m} \sum_ {k = 1} ^ {n} S _ {k, l} ^ {2} \left(\frac {m}{H} d _ {l} ^ {r o w} - \frac {n}{W} d _ {k} ^ {c o l}\right) ^ {2}. \tag {A3}
$$
This minimization task in Eq. (A1) can be efficiently solved using a standard QP solver [20], which requires converting the above minimization task into the general form of QP, i.e.,
$$
\underset {d} {\text {m i n i m i z e}} \quad E = \frac {1}{2} d ^ {\top} P d + q ^ {\top} d \tag {A4}
$$
$$
\text {s u b j e c t} \quad A d = b
$$
where  $d = (d_1^{row},\dots,d_m^{row},d_1^{col},\dots,d_n^{col})^\top \in \mathbb{R}^{m + n}$  is the unknown grid intervals to be optimized.  $P\in \mathbb{R}^{(m + n)\times (m + n)}$  and  $q\in \mathbb{R}^{m + n}$  can be derived from the energy function  $E = E_{zoom} + \lambda E_{rigid}$ .  $A\in \mathbb{R}^{2\times (m + n)}$  and  $b\in \mathbb{R}^2$  can be derived from the linear constraint in Eq. (A1).