SparseFormer: Detecting Objects in HRW Shots via Sparse Vision Transformer

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-09 06:48:31

Abstract: Recent years have seen an increase in the use of gigapixel-level image and video capture systems and benchmarks with high-resolution wide (HRW) shots. However, unlike close-up shots in the MS COCO dataset, the higher resolution and wider field of view raise unique challenges, such as extreme sparsity and huge scale changes, causing existing close-up detectors inaccuracy and inefficiency. In this paper, we present a novel model-agnostic sparse vision transformer, dubbed SparseFormer, to bridge the gap of object detection between close-up and HRW shots. The proposed SparseFormer selectively uses attentive tokens to scrutinize the sparsely distributed windows that may contain objects. In this way, it can jointly explore global and local attention by fusing coarse- and fine-grained features to handle huge scale changes. SparseFormer also benefits from a novel Cross-slice non-maximum suppression (C-NMS) algorithm to precisely localize objects from noisy windows and a simple yet effective multi-scale strategy to improve accuracy. Extensive experiments on two HRW benchmarks, PANDA and DOTA-v1.0, demonstrate that the proposed SparseFormer significantly improves detection accuracy (up to 5.8%) and speed (up to 3x) over the state-of-the-art approaches.

Main paper: 
 related work: 
 Close-up shot detection models. The majority of common object detection datasets, such as PASCAL VOC [9] and MS COCO [24] , collect high-resolution images with close-up shots, which has greatly contributed to the development of object detection. Based on the detection head, the literature can be broadly categorized into two types: one-stage detectors and two-stage detectors. The primary objective of the two-stage object detection is accuracy, and it frames the detection as a "coarse-to-fine" process [3, 12, 13, 18, 34] . On the other hand, one-stage detectors have an edge in terms of speed, such as YOLO [32] . Subsequent works have attempted to make improvements such as more anchors, better architecture, and richer training techniques [11, 25, 33] . To sum up, the current detectors exhibit great speed and accuracy in close-up shots.High-resolution wide shot detection models. The introduction of imaging systems led to the development of a new benchmark for gigapixel-level detection with HRW shots called PANDA [43] . This benchmark has recently gained a lot of attention. Previous works on gigapixel-level detection focus on achieving lower latency through patch selection or arrangement [5, 10, 29] . However, they are unable to solve the unique challenges faced in HRW shots. Some works use sparse policies on patches [31] , self-attention heads [28] , and transformer blocks [28] for image classification. PnP-DETR [41] exploits a poll and pool sampler to extract image features from the backbone and feed the sparse tokens to the attention encoder. This approach shows to be effective for object detection, panoptic segmentation, and image recognition. However, the sparse sampling on the backbone has not been adequately studied yet. DGE [37] is a plugin for vision transformers, but it is not flexible enough to be extended to ConvNet-based models or use arbitrary-size images as input. Therefore, how to design a flexible and model-agnostic architecture for object detection in HRW shots remains underexplored.Transformer backbones. Transformers have been successful in natural language processing (NLP), and their potential for vision tasks has gained considerable attention. One such example is the Vision Transformer (ViT) [8] , which uses a pure Transformer model for image classification and has shown promising results. However, ViT's computational costs for processing high-resolution images are impractical. Several methods have been attempted to reduce ViT model costs, including window-based attention [26] , downsampling in self-attention [42, 44] , and low-rank projection attention [46] . Other works use sparse policies on patches [31] , self-attention heads [28] , and transformer blocks [28] for image classification.Unfortunately, these methods suffer from significant accuracy drops when detecting objects in high-resolution wide shots. 
proposed method: 
 We address the unique challenges of HRW detection by proposing the Sparse Vision Transformer. This model efficiently extracts valuable features from sparse information, while enlarging the receptive field to handle huge scale changes. To tackle the problem of incomplete large objects on intersecting sliced areas, we modify vanilla NMS. Additionally, we introduce our HRW-based augmentation for both training and inference to enhance the detection accuracy for both large and small objects. The pipeline for inference using our proposed modules is shown in Figure 3 .An ideal vision model should be able to extract meaningful information from sparse data using limited calculations, just like our human eyes tend to focus on valuable areas over unimportant background information. To achieve this, we design a novel Sparse First, we perform multi-scale slicing on a gigapixel image. Then, we apply patch partitioning to each slice, and group neighboring patches into windows. Global Attention utilizes aggregated features to quickly obtain coarse-grained information. Local Attention selects important windows to extract fine-grained information.Vision Transformer called SparseFormer. It dynamically selects key regions and enables dynamic receptive fields to cover objects with various scales. The overall framework of SparseFormer is illustrated in Figure 4 . Inspired by Swin Transformer [26] , we split the input image into non-overlapping patches to generate tokens. Sparse-Former consists of four stages that work together to produce an adaptive representation. Each stage begins with a patch merging layer that concatenates the features of each group of 2 × 2 neighboring patches. The concatenated features are then projected to half of their dimension using a linear layer.Each stage of SparseFormer is centered around attention blocks that are designed to capture long-range and short-range interactions at different scales. To achieve this, we take both the advantages of the vanilla self-attention Transformer block and the Swin Transformer block. In this way, we develop two distinct types of sparse-style blocks. One is used to capture long-range interactions at a coarse grain, while the other focuses on short-range interactions at a finer scale. To facilitate this approach, we introduce the concept of a Window which divides each feature map into equally spaced windows. Operations within each window are considered "local", while operations that encompass all windows are designated as "global. "We outline the global and local attention blocks in more detail. We construct the global block using the standard multi-head selfattention (MSA) [38] and MLP module with aggregated features, or only convolution layers, as detailed in Section 3.2. We construct the local block by adding a sparsification step and an inverse sparsification step before and after the Swin Transformer [26] block, as described in Section 3.3. Unlike previous work [41, 49] , we do not build separate branches for global and local attention. Instead, the local attention is positioned after the global one to obtain more details, rather than different features. When a stage has multiple blocks, the ordering of global attention blocks (G) and local attention blocks (L) follows a pattern of 'GGLL'.Feature aggregation. Global attention aims to capture coarsegrained features based on long-range interaction. As such, we generate low-resolution information by sparsifying the feature in each window. As shown in Figure 4 , we begin each stage with the global attention block. The primary function of this block is to aggregate the features of each window. To achieve this, we take the input feature map 𝑧 and divide it into windows of size 𝑀, ensuring they do not overlap. The left-top location of each window is given by (𝑥, 𝑦), and each token within the window has a relative location (Δ𝑥, Δ𝑦). We then calculate the aggregated features using the following formula:EQUATIONHere, 𝑥 ′ = 𝑥/𝑀 and 𝑦 ′ = 𝑦/𝑀. 𝛼 Δ𝑥,Δ𝑦 is the weight of each token.In this paper, we assign equal weights to all tokens by setting 𝛼 Δ𝑥,Δ𝑦 = 1. After aggregating the features using the above formula, we obtain the aggregated feature z which can be further used for attention.Window-level global attention. Feature aggregation is a technique that reduces the number of tokens by a multiple of 𝑀 2 , which is equivalent to a 𝑀× downsampling of resolution. This reduction in tokens allows us to use global attention interaction without expensive computation. With the aggregated features, consecutive global blocks are computed as follows:EQUATIONwhere z𝑙 refers to the output features of the 𝑙-th global block.Inverse aggregated features. The aggregated features contain abstract information that facilitates global content-dependent interactions among different image regions. However, their resolution differs from the input feature map. Consequently, we convert the window-level features back to the token-level using an inverse function of equation Equation ( 1), as follows:EQUATIONHere, 𝑥 = 𝑀 • 𝑥 ′ and 𝑦 = 𝑀 • 𝑦 ′ , where (𝑥 ′ , 𝑦 ′ ) and (𝑥, 𝑦) represent the location on the input and output feature maps, respectively. Additionally, (Δ𝑥, Δ𝑦) represents the relative location with respect to (𝑥, 𝑦). We consider (𝑥, 𝑦) as the top-left of each window on the output feature map, where the windows are partitioned in the same manner as in the feature aggregation process. This step extracts the output feature map from the successive global block (Equation ( 2)). Then, we invert it using Equation (3) and denote the resulting feature map by 𝑧 global . It is worth noting that the final global feature 𝑧 global has the same resolution as the input feature map 𝑧. Even though aggregated features have a lower resolution, the global attention operation can provide more nonlocal information with little extra computation.Variance-based scoring. Note that coarse-grained features for each window can achieve high efficiency. However, we still need fine-grained features that can extract object details to accurately 𝑧 updated by Eq. ( 8) detect objects. As such, we drop certain windows based on their low information content to reduce computation. Our goal is to identify windows that require further local attention because their windowlevel feature cannot represent their inner token-level features.We begin with an initial feature map 𝑧, which is before global and local attention, and has dimensions of 𝐻 , 𝑊 , and 𝐶. We then get the aggregated feature z from 𝑧 using Equation (1) and apply the inverse sparsification function via Equation (3), which produces an intermediate feature map ẑ with the same resolution as 𝑧. Next, we calculate the residual 𝑟 between ẑ and 𝑧 and concatenate the features of each window to obtain the tokens of 𝑀 × 𝑀 × 𝐶 dimensions that are 𝐻 𝑀 × 𝑊 𝑀 in size. We can easily calculate the variance of each window from these tokens. We construct a ScoreNet using MLP to add learnable weights to each residual. The score of each window is given by:EQUATIONwhere the MLP projects (𝑀 × 𝑀 × 𝐶)-dimensional features for each window to 1 dimension, and the SoftMax operation calculates the score for each window. A higher score indicates greater variance, meaning high-variance windows require fine-grained attention. In other words, we discard windows with lower scores during local attention. Once we have ranked the windows, we can selectively choose a part of them to generate finer-level features. Before doing so, we update the feature map 𝑧 with global feature 𝑧 global using:EQUATION)Windows sparsification. We start by analyzing the global attention and variance-based scoring to obtain the initial feature 𝑧 and scores for each window. Next, we partition 𝑧 into windows of size 𝐻 𝑀 × 𝑊 𝑀 , in the same way as the ScoreNet. We represent these windows as a matrix 𝑍 ∈ R 𝑁 ×𝐷 , where 𝑁 is the total number of windows, i.e., 𝑁 = 𝐻 𝑀 × 𝑊 𝑀 and 𝐷 = 𝑀 × 𝑀 × 𝐶. To determine which windows to keep, we define a hyperparameter 𝑘 to represent the keeping ratio. We maintain a binary decision mask vector 𝐴 ∈ {0, 1} 𝑁 to indicate whether to drop or keep each window based on 𝑘 and scores. The value of 𝑘 would depend on the specific task at hand, and can be adjusted as required. The sparse matrix 𝑆 ∈ R 𝐾 ×𝑁 collects the one-hot encoding of vector 𝐴, where 𝐾 is the number of keeping windows, i.e., 𝐾 = 𝑘 • 𝑁 . Using this sparse matrix, we compute the features of the sparse windows as follows:EQUATIONThe output feature Ẑ𝑠 ∈ R 𝐾 ×𝐷 is then used as input for the local attention.Shifted window-based attention. We utilize the Shift Windowbased Attention module which was first introduced in Swin Transformer [26] . The consecutive local blocks can be represented as:EQUATIONwhere 𝑧 𝑙 denotes the output features of the local block 𝑙. The layer can be either a self-attention or convolution module. To fuse the output and input features of local attention, we use the following equation:EQUATIONHere, Ẑ𝑠 is updated by local attention, while Ẑ is the output of each stage of SparseFormer. Finally, we revert Ẑ back into the original dimensional space of 𝐻 × 𝑊 × 𝐶 to obtain the final feature map, denoted as 𝑧. The window-based attention, based on variance-based scoring, can extract more local information in a lightweight form, thus improving the detection performance of small objects while saving computation for the background. AREA(•) calculates the area of a box; End-to-end Optimization. It is challenging to optimize the ScoreNet because we only use the output to sort the windows, and the gradient cannot be back-propagated. To overcome this issue, we implement the Gumbel-Softmax trick [27] to relax the sampling process, making it differentiable. This trick provides a bridge for gradient back-propagation between soft values and binarized values through re-parameterization. Hence, we re-write Equation ( 5) as:1 B ′ 1 ← NMS(B 1 ); B ′ 2 ← NMS(B 2 ); 2 B ← B ′ 1 ∪ B ′ 2 ; B ′ ← ∅; 3 while B ≠ ∅ doEQUATIONHere, 𝑠 represents the output of the SoftMax function, which indicates the scores of windows.In HRW shot processing, the slicing strategy generates box candidates for each slice, which must then be merged into a mutually non-conflicting box set. However, using Non-Maximum Suppression (NMS) to select the highest-scoring boxes may lead to incomplete boxes when objects are on the edge regions of multiple slices (For a more detailed explanation and visual representation, refer to Figure 5 ). To address this, we propose a Cross-slice Non-Maximum Suppression (C-NMS) strategy, as shown in Algorithm 1, that prioritizes boxes with the maximum area across multiple slices, rather than just the highest scores. The C-NMS algorithm consists of two stages: a local suppression stage and a cross-slice suppression stage.Due to memory limitations, it is not possible to train and test super high-resolution datasets at their original size. Therefore, we use a slicing strategy in both the training and testing phases. To make better use of the multi-scale information, we use high-resolution images and divide them into slices of varying sizes using the slicing strategy. All slices are scaled to the same size, enabling effective training and inference for the object detector. We divide the image into grids of 16×16, 8×8, 4×4, and 2×2 grids, respectively, and remove the slices with no objects. This approach allows us to analyze and understand the complex features of these images, ultimately improving the overall accuracy and effectiveness of the detector.During the inference phase, we use slicing windows of two sizes: the original one and one quarter of both height and width. Instead of simply combining the two windows, we set different receptive fields for the two types of windows with a threshold 𝑇 𝑎 . Based on the first window, we remove the prediction boxes larger than 𝑇 𝑎 . We only keep the boxes larger than 𝑇 𝑎 for the second window. This follows the idea of scale-specific design [35, 36] , where we should arrange each window to cover the appropriate scale to improve performance. With this technique, we can quickly and accurately process high-resolution images. 
experiment 4.1 effectiveness evaluation: 
 Datasets. Our evaluation is on two public benchmarks with HRW shots, PANDA [43] and DOTA-v1.0 [45] . PANDA is the first humancentric gigapixel-level dataset. It contains 18 scenes with over 15,974.6k bounding boxes annotated. Specifically, there are 13 scenes for training and 5 scenes for testing. DOTA is a large-scale dataset to evaluate the oriented object detection in aerial images with sizes up to 4000×4000. It contains 2,806 images and 188,282 instances with oriented bounding box annotations, covered by 15 object classes.Evaluation metrics. We report the FLOPs and standard COCO metrics including AP 𝑡𝑜𝑡𝑎𝑙 , AP 𝑆 (< 96×96), AP 𝑀 (96×96-288×288) and AP 𝐿 (> 288×288). For quantitative efficiency evaluation, we use the average FLOPs of each detector to process a 1280 × 800 window in the datasets. Further, we calculate FLOPs in foreground and background, respectively, to show the efficiency of our SparseFormer to reduce the computation on backgrounds.Implementation details. We implement the detectors using MMDetection [4] , incorporating both the officially provided DINO and Dynamic-Head, and integrating the open-source code for the DEG backbone. To ensure a fair comparison, we evaluate these two detectors across four different backbones, including Swin, DEG, and our own proprietary design, all configured with identical numbers of hyperparameters (e.g., depths, embedding dimension, number Table 1 : Comparison with the state-of-the-arts on PANDA. "F" and "B" denote foreground and background, respectively (A=F+B). "*" denotes the re-implementation in [10] . The GFLOPs on the two-stage detector do not include the detection heads due to the dynamic cost.Backbone GFLOPs-F GFLOPs-B GFLOPs-A AP 𝑇 𝑜𝑡𝑎𝑙 AP 𝑆 AP 𝑀 AP 𝐿 FasterRCNN [34] ResNet of multi-heads). All models are trained from scratch for 36 epochs, in line with the observations in [17] . This also ensures a fair comparison with Swin and DEG, affirming that the performance improvement stems from our novel design rather than from better pre-trained weights.Results on PANDA. We compare our model with a different keeping ratio 𝑘 to current state-of-the-art methods on the first gigapixellevel dataset PANDA, which not only has the challenge of wide FoV, but also super high resolution. The results are presented in Table 1 .We first produce two baselines, one based on ATSS framework [55] with dynamic head block [6] which achieves GFLOPs of 114.80, the other based on DINO [54] and GFLOPs of 132.84. Then, the backbone is modified to SparseFormer for further experiments. Note that the keeping ratio means the ratio of keeping tokens based on the previous stage, so the ratio of each stage based on the full number of tokens is [𝑘, 𝑘 2 , 𝑘 3 , 𝑘 4 ]. We can observe our method achieves more than 5% increase in AP over SotAs, with only 75.71 GFLOPs (43% reduction from Swin-T, 63% reduction from PAN [10] ). The most notable thing is that the reduced FLOPs are mainly from the background region, which is why we can significantly reduce the amount of computation but maintain high performance. An additional note is that GigaDet and PAN is to accelerate the detector by optimizing the process. Unlike these approaches, our work does not prescribe a specific pipeline. Instead, we propose a model-agnostic strategy that could be seamlessly integrated into existing pipelines.Results on DOTA. We choose aerial images that also have HRW shots to verify the generalization. The compared methods include: Faster RCNN-O [34] , ICN [2] , RoI-Transformer [7] , CADNet [53] , DRN [30] , CenterMap [39] , SCRDet [52] , R 3 Det [51], S 2 A-Net [15] , CFA [14] , CSL [50] , ReDet [16] , Or-RepPoints [22] . RoI-Transformer is used as the baseline detector for comparison and we set 𝑘 = 0.7. In Table 3 , SparseFormer improves mAP from 69.56% to 77.94% and reduces 296.74 GFLOPs to 174.31 GFLOPs. Compared to SotA Transformer-based method Or-RepPoints, we achieve 0.3% AP improvement and reduce 21% GFLOPs. Compared to the method with similar computation S 2 A-Net, we surpass its AP by 3.82%. This indicates a significant enhancement in terms of accuracy and efficiency.The DOTA [45] dataset presents a formidable challenge, yet our approach achieves the precision of the current SotAs with significantly fewer FLOPs. This not only validates the design intention behind SparseFormer to reduce computational demands but also demonstrates its generalizability across various tasks and domains.Component effectiveness. We investigate the effectiveness of global block, C-NMS, multi-scale training (MS Train) and inference (MS Inference). Evaluation is conducted on the PANDA with 𝑘 = 0.7. As shown in Table 2 , all components can significantly improve performance with a little extra cost which also shows that our strategies are useful for object detection in HRW shots. Keeping ratio. Our strategy involves discarding the grids that are deemed unimportant. We study the impact of the grid-keeping ratio on the final performance. Table 4 presents our findings, where the keeping ratio 𝑘 is represented as [𝑘, 𝑘 2 , 𝑘 3 , 𝑘 4 ] for each stage. As the features become sparser, we observe a significant reduction in FLOPs, but the decrease in accuracy is insignificant.Effect on ScoreNet. We study the effect of different post-processing of residual values which feed into ScoreNet (introduced in Section 3.3). 𝑧 and ẑ denote the original features and aggregated features, respectively, which are the same in Equation ( 4). As can be seen from the last three lines, several variants based on residuals have a performance within the error range, so we consider using less computation and do not perform any redundant processing on them. Compared with 𝑍 , which directly uses all the features in the window, the average feature ẑ can achieve better results. We think this is because the ScoreNet is a simple MLP, it cannot take advantage of complex features 𝑧 well, while the ẑ is easier to be classified based on color (blue for the sky and green for glasses) and other patterns.The HRW shots are usually captured by edge devices like UAVs.UAV detectors typically cannot run on large computing devices, but instead run on low-power edge devices. Because it is usually 6 . Notably, our method can largely reduce inference time compared to previous methods. Our method is 3× faster than PAN and has 5.8% increase of AP. We can see because of the complex head structure, the inference speed of dynamichead [6] is not ideal. On the opposite, DINO [54] shows promising FPS than the previous work and the improvement of the speed is clearer. Compared to the competitive approach DEG [37] , our method could achieve much better performance with faster speed.It is noteworthy that our strategy is model-agnostic, enabling seamless integration with either ConvNet or Transformer architectures. This flexibility leads to the creation of SparseNet and SparseFormer. Building upon the previously mentioned SparseFormer, we have innovated by substituting every self-attention module with convolution layers. As illustrated in Table 1 and Table 3 , SparseNet demonstrates performance that is not only comparable but also competitive with the renowned ResNet [19] . Especially noteworthy is that SparseNet reduces the GFLOPs up to 56% while increasing accuracy compared to CSL [50] and it achieves the lowest GFLOPs on the DOTA dataset [45] , underscoring its efficiency and effectiveness in complex computational tasks.In order to better understand how window sparsification works, we visualize the selected windows from each stage in Figure 7 .The red patches represent regions with higher scores, while the blue patches indicate lower scores. SparseFormer will perform finegrained feature extraction on the regions with higher scores. This illustration highlights the advantages of computation reduction on background areas and low-entropy foregrounds. Additionally, the results validate the effectiveness of our SparseFormer approach.The PANDA [43] and DOTA [45] datasets focus on different target objects, they share the common characteristic of containing large-scale background areas, making the sparsification approach particularly relevant. We believe that this methodology will not only benefit object detection in HRW shots but also various other vision tasks. 
conclusion: 
 We introduced SparseFormer, a sparse Vision Transformer-based detector designed for HRW shots. It uses selective token utilization to extract fine-grained features and aggregate features across windows to extract coarse-grained features. The combination of fine and coarse granularity effectively leverages the sparsity of HRW shots, facilitating handling extreme scale variations. Our Crossslice NMS scheme and multi-scale strategy help detect oversized and diminutive objects. Experiments on PANDA and DOTA-v1.0 benchmarks show significant improvement over existing methods, advancing state-of-the-art performance in HRW shot object detection.  
: 
 Figure 1 : Performance comparison in terms of object size on the PANDA dataset [43] . The horizontal axis indicates object sizes (the area of bounding boxes) on a logarithmic scale. The vertical axis shows detection accuracy per size. Both YOLOv8 [20] and DINO [54] underperform in handling extreme scale variations, especially for small and large objects. The proposed method performs well, achieving new state-of-the-art detection accuracy.objects in high-resolution wide (HRW) shots with square-kilometer scenes and gigapixel-level resolutions have drawn increasing attention [5, 10, 15, 22, 29, 30, 52, 53] . Detecting objects in HRW shots using close-up detectors is not effective due to several unique characteristics of HRW shots, as found in PANDA [43] and DOTA [45] , compared to close-up shots like MS COCO. The most significant challenge is the sparse information in HRW shots, where objects often cover less than 5% of the image. This makes it difficult for detectors to extract key features from a sea of background noise, resulting in false positives within the background and false negatives within the object areas during training and testing. The second challenge is the varying scales of objects in HRW shots, with changes up to 100 times. Detectors relying on fixed settings of the receptive field and anchors cannot adapt to these extreme scales, as shown in Figure 1 . For instance, YOLOv8 [20] underperforms in detecting small objects. While DINO [54] shows marginal improvement, it still falls short in adapting to such exaggerated scale changes, resulting in subpar detection of larger objects (Figure 2 ). Additionally, the typical twostage downsampling schemes [5, 10, 21, 29] miss more small objects. The slicing strategy [1] can result in incomplete boxes when using NMS to merge prediction boxes, as shown in Figure 5 . Therefore, it is imperative to bridge the gap between object detection in close-up and HRW shots.Motivated by recent advanced techniques [28, 31, 37, 40, 41, 47] to enhance object detection accuracy, we present a novel detector for HRW shots that employs a sparse Vision Transformer, called SparseFormer. SparseFormer uses attentive tokens selectively to concentrate on regions of an image where objects are sparsely distributed, facilitating the extraction of fine-grained features. To achieve this, it learns a ScoreNet to assess the importance of regions. By examining the variance of importance scores of all regions, our SparseFormer prioritizes regions that capture rich fine-grained details. In this way, it can focus on complex image regions rather than less significant ones (e.g., smooth content from the background). Concurrently, it divides each HRW shot into non-overlapping windows to extract coarse-grained features. Sharing a similar spirit with the receptive field strategy of the original Vision Transformer [8] , our proposed SparseFormer combines coarse and fine-grained features, achieving much higher efficiency than Swin Transformer [26] . This greatly helps to handle large scale variations and detect both large and small objects accurately.We further present two innovative techniques to improve detection accuracy against huge scale changes. First, we observe that conventional NMS refers to confidence scores only to merge detection results, leading to incomplete bounding boxes on oversized objects. To address this, we propose a novel Cross-slice NMS scheme (C-NMS) that favors large bounding boxes with high confidence scores. The proposed C-NMS scheme greatly improves the detection accuracy of oversized objects. Second, we use a multi-scale strategy to extract coarse-grained and fine-grained features. The multi-scale strategy enlarges the receptive field, enhancing the detection accuracy on both large and small objects. In summary, the main contributions of this work are as follows:• We propose a novel sparse Vision Transformer based detector to handle huge scale changes in HRW images. • We further use cross-window NMS and multi-scale schemes to improve detection on large and small objects. • We extensively validate our method on two large-scale HRW-shot benchmarks, PANDA and DOTA-v1.0. Our method advances state-of-the-art performance by large margins. 
 
 Appendix:  


Decision: This paper presents a novel model-agnostic SparseFormer approach to bridge the gap of object detection between close-up and HRW shot. This authors addressed the concerns of most reviewers well and received all positive ratings after the rebuttal; thus, I recommend acceptance for this submission.

Reviews: review: paper_summary: paper_summary: The paper introduces SparseFormer, a model-agnostic sparse vision transformer for high-resolution wide (HRW) shots. SparseFormer uses attentive tokens to focus on sparsely distributed windows and combines global and local attention by fusing coarse- and fine-grained features. It also features a Cross-slice non-maximum suppression (C-NMS) algorithm and a multi-scale strategy to enhance accuracy. Experiments on HRW benchmarks PANDA and DOTA-v1.0 show significant improvements in detection accuracy (up to 5.8%) and speed (up to 3×) over current methods.
strength_weakness: strength_weakness: 1. This paper is well-written and easy to follow. The motivation is clear and the explanation to sparseformer is concise and easy to understand.
2. Extensive experiments denonstrate the effectiveness  and efficiency of sparseformer, which is exciting and promising.
3. Further analyses towards sparseformer, including ablation studies and visualizations, seems convincing and inspiring.
4. From my perspective, sparseformer will be agreat contribution to HRW research community.
limitations: limitations: None in particular
paper_summary: paper_summary: The paper addresses the challenges of object detection in high-resolution wide-view (HRW) images, such as extreme sparsity and large scale variations. Authors proposes SparseFormer, a sparse vision Transformer that uses selective attention tokens to focus on potential object regions in HRW images. Introduces a ScoreNet for importance evaluation, Cross-window Non-Maximum Suppression (C-NMS) for accurate large object detection, and a multi-scale strategy for enhanced detection. Extensive experiments on PANDA and DOTA-v1.0 datasets were conducted.
strength_weakness: strength_weakness: 1. The proposed method is relatively novel with sound and meaningful motivation. The proposed idea of global-local sparse attention is particularly well-suited for addressing the problem.
2. The paper is well-organized and easy to follow.
3. Extensive experiments were conducted.
limitations: limitations: 1. The comparison in the paper only includes methods up to 2022. Please include the latest work from the past two years.
2. ScoreNet is an existing technique, as proposed in DynamicViT (NeurIPS 2021).
paper_summary: paper_summary: The paper introduces SparseFormer, a novel model-agnostic sparse vision transformer designed to address the challenges of object detection in high-resolution wide (HRW) shots. Unlike traditional close-up object detection methods, SparseFormer selectively uses attentive tokens to focus on sparsely distributed regions that may contain objects, enabling it to handle extreme scale variations and sparsity. The model employs a Cross-slice non-maximum suppression (C-NMS) algorithm for precise object localization and a multi-scale strategy to enhance accuracy. Experiments on two HRW benchmarks, PANDA and DOTA-v1.0, demonstrate that SparseFormer significantly improves detection accuracy and speed over state-of-the-art approaches.
strength_weakness: strength_weakness: + The paper is well-structured.

+ The paper offers extensive experimental validation on two large-scale HRW-shot benchmarks, PANDA and DOTA-v1.0.

+ The inclusion of visualizations, such as the window scores from SparseFormer, helps in understanding how the model prioritizes regions for fine-grained feature extraction.
limitations: limitations: - The latex format is not up to date.

- No loss function is given for the overall network.

- In the experiments in Table 2, the APs for Local Block is 0.319, why does the performance drop with the addition of Global Block (from 0.319 to  0.238)?

- The paper needs to be checked for some errors, such as line 512, where equation 9 should end with a period.