Parameter-Inverted Image Pyramid Networks
Authors: Xizhou Zhu
Xue Yang
Zhaokai Wang
Hao Li
Wenhan Dou
Junqi Ge
Lewei Lu
Yu Qiao
Jifeng Dai
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Vision Foundation Models; Object Detection
Primary area: machine_vision
TL;DR: We propose the Parameter-Inverted Image Pyramid Networks to address the computational challenges of traditional image pyramids.

Abstract: Image pyramids are commonly used in modern computer vision tasks to obtain multi-scale features for precise understanding of images. However, image pyramids process multiple resolutions of images using the same large-scale model, which requires significant computational cost. To overcome this issue, we propose a novel network architecture known as the Parameter-Inverted Image Pyramid Networks (PIIP). Our core idea is to use models with different parameter sizes to process different resolution levels of the image pyramid, thereby balancing computational efficiency and performance. Specifically, the input to PIIP is a set of multi-scale images, where higher resolution images are processed by smaller networks. We further propose a feature interaction mechanism to allow features of different resolutions to complement each other and effectively integrate information from different spatial scales. Extensive experiments demonstrate that the PIIP achieves superior performance in tasks such as object detection, segmentation, and image classification, compared to traditional image pyramid methods and single-branch networks, while reducing computational cost. Notably, when applying our method on a large-scale vision foundation model InternViT-6B, we improve its performance by 1\%-2\% on detection and segmentation with only 40\%-60\% of the original computation. These results validate the effectiveness of the PIIP approach and provide a new technical direction for future vision computing tasks.

Parameter-Inverted Image Pyramid Networks
Xizhou Zhu Xue Yang",, Zhaokai Wangkl, Hao Li41 Wenhan Dou Junqi Ge Lewei Lu,, Yu Qiao', Jifeng Dai2.f
'OpenGVLab, Shanghai AI Laboratory
"Tsinghua University
SShanghai Jiao Tong University *The Chinese University of Hong Kong
TSenseTime Research
Abstract
Image pyramids are commonly used in modern computer vision tasks to obtain multi-scale features for precise understanding of images. However, image pyra- mids process multiple resolutions of images using the same large-scale model, which requires significant computational cost. To overcome this issue, we pro- pose a novel network architecture known as the grrameter-Inverter Image Pyra- mid Networks (PIIP). Our core idea is to use models with different parameter sizes to process different resolution levels of the image pyramid, thereby balancing com- putational efficiency and performance. Specifically, the input to PIIP is set of multi-scale images, where higher resolution images are processed by smaller net- works. We further propose a feature interaction mechanism allow features of different resolutions to complement each other and effectively integrate informa- tion from different spatial scales. Extensive experiments demonstrate that the PIIP achieves superior performance in tasks such as object detection, segmentation, and image classification, compared to traditional image pyramid methods and single- branch networks, while reducing computational cost. Notably, when applying our method on a large-scale vision foundation model Intern VTT-6B, we improve its performance by 1%-2% on detection and segmentation with only 40%-60% of the original computation. These results validate the effectiveness of the PIIP approach and provide new technical direction for future vision computing tasks.
Introduction
In modern computer vision, sereg--erforancce image perception systems increasingly rely on large- scale pre-trained models. These models typically consume tens of thousands to millions of GPU hours during pre-training [43, 44, 41]. To adapt these expensively pre-trained models for fine- grained image perception tasks (e.g., detection [4, 63, 57, 56] and segmentation [19, 50J), researchers usually combine them with image pyramids [40, 37] feature pyramids [27, 42, 34). This combi- nation is crucial for constructing multi-scale features essential for image understanding.
However, integrating these pre-trained models with image pyramids results in significant compu- tational overhead. Image pyramids process the same image at multiple resolutions with the same large-scale model, causing the computational demands to increase quadratically with the image olutions across all scales. Although feature pyramids [27, 16, 42] aim to reduce this overhead, in MS COCO challenges [28], most top-performing models [48, 14, 64, 7] still rely image pyramids due to their superior performance. Therefore, it is necessary to reduce the computing resources for building image pyramids while maintaining high performance.
Equual contribution. Corresponding author: Jifeng Dai <daijifeng @ertinngtttiiiienn tsinghua.edu.cn>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Image
Large Model
Medium Model
Small Model
(a) Plain
(b) Image Pyramid
(c) Buramneler-Eqiivllee Image Pyramid
(d) Parameler-DDreec Image Pyramid
(e) Parameler-Invertee Image Pyramid
Intrractoon
(Multi (Mutti-scale scale Training)
Figure 1: Different parameter-resolution designs of image pyramid networks. (a) Plain network which lacks multi-scale features. (b)(e) Inefficient image pyramid networks (shared weights / separate weights with interactions) using equivalently large networks for all scales. (d) Parameter-direct image pyramid network which processes high-resolution images with large models, leading to high computational cost. (e) Our efficient parameter-invers image pyramid network (PIIP). which pairs models of increasing parameter sizes inversely with images of decreasing resolution. It delivers better performance than those of (b)(c)(d with much lower computational cost.
To address this, our key idea is that is unnecessary to employ vision models of equivalent size for feature extraction at all resolutions (Fig. ((b-)) or adopt a parameter-direet design (Fig. (dd). Fea- tures at different resolutions can complement each other through adequate feature fusion, thereby enhancing computational efficiency and avoiding redundant modeling of similar information. Specif- ically, for lower-resolution pyramid levels, the smaller images allow the efficient use of larger mod- els to extract rich contextual and semantic features. The high-resolutie branches need only provide the detail information missing from the lower-resolution features, instead of re-modeling existing semantic information. Thus, high-resolution features focus smaller receptive fields with less semantic information, making it possible to use smaller models to save computational resources.
Building this strategy, a low-cost and high-performance image pyramid network can be con- structed using series of models with increasing parameter size, paired inversely with images of decreasing resolution, as shown in Fig. ((e). Each resolution level should be able to directly lever- age existing pre-trained vision foundation models for feature extraction, avoiding the large compu- tational costs for training multi-scale image pyramid networks from scratch. In addition, sufficient feature interactions between different levels are also required to ensure the complementarity of fea- tures at different scales and avoid redundant feature extractions.
To this end, we propose Parameter- Inverted Image Pyramid Networks (PIIP) based on the comple- mentarity of image features at different resolutions. Specifically, the network takes images at mul- tiple scales as inputs, where higher resolution features are extracted through networks with fewer parameters for local detail perception, and lower resolution features are extracted with more param- eters for global information extraction. Additionally, we introduce feature interaction module that allows features between different resolutions to complement each other. This structure reduces the number of parameters high-resolutin branches and effectively integrates information from dif- ferent receptive fields, significantly reducing computational costs without sacrificing performance.
We conduct experiments on object detection, instance segmentation, semantic segmentation and image classification. Our method achieves better performance while reducing computational costs, compared traditional image pyramids and single-branch networks. These results validate the effectiveness of our multi-resolution feature interaction strategy and parameter-inverted paradigm and provide a new direction for future visual computing. Our contributions are as follows:
1) We propose a novel architecture named Parameter-Inverted Image Pyramid (PIIP) that enhances the multi-scale representational capability of vision backbones with high computation efficiency. The proposed architecture is capable of effectively and flexibly utilizing strong pre-trained vision foundation models without the need for extensive training from scratch.
2) We evaluate our method on classic vision tasks of object detection, instance segmentation, seman- tic segmentation, and image classification. Through combination of existing pre-trained models, method surpasses Linglebrrace models and other image pyramid methods with higher performance and lower computation cost.
3) validate the generalizability of PIP on large-scale vision foundation models, we apply PIIP to Intern ViT-6B [8], improving its performance object detection and semantic segmentation by 1.9%

---

(55.7 AP") and 1.3% (59.7 mloU) while reducing 43% and 58% of computational costs, respectively. We also provide extensive analysis and valuable insights on ablation and design guidelines for PIIP that may benefit future research.
2 Related Work
Image Pyramids and Feature Pyramids. Image pyramids and feature pyramids are two widely used techniques to enhance the multi-scale perceptive ability for downstream dense prediction tasks. Image pyramids [60, 39, 40, 37] resize the original image and extract features of different resolutions separately. allowing models accurately detect objects of various scales. However, this technique significantly increases computational costs. Feature pyramids [27, 16, 42, 61, 34] represent another method for constructing multi-scale feature representations by merging low-resolution, semantically strong features with high-resolution, semantically weak features. Although significantly reducing computational costs, they cannot fully replace image pyramids when detecting very small or large objects [39]. Our proposed architecture integrates both image and feature pyramids and introduces the parameter-inverteed paradigm achieve efficient computation.
Multi-branch Architectures. Multi- buali-branchhh architectures have been widely adopted to combine fea- tures from different resolutions in various computer vision tasks, including image classification [5], object detection [46, 25, 7, 52], semantic segmentation [58, 17] and multimodal dialogues [33, 21]. Cross ViT [5] adopts a two-branch structure with different patch sizes to obtain inputs of various scales and different model sizes to balance the computational load. HRNet series [46, 58, 17] adopt four-branch architecture, where the number of branches gradually increases the layers deepen. However, they do not adopt the parameter inversion paradigm and cannot utilize existing pre-trained models. In contrast, we propose a general model architecture that supports the of pre-trained models with different parameters to build efficient image pyramids.
Redundancy Reduction for Visual Models. Extensive studies focus reducing computational re- dundancy for acceleration. Some work exploits the sparsity of images to accelerate model inference by reducing the number of visual tokens. Dynamic ViT [38] and ViT [35] design lightweight prediction modules to predict and prune less informative tokens. EVIT [26] and Evo-ViT ViT [55] com- pute attention scores for each token from class token identify less informative tokens and adopt accelerated processing strategies for them. Other approaches focus on improving the model struc- ture for efficient computation, such as attention mechanisms [47, 17, 3] or gradually reducing the spatial resolution as the number of layers increases [30, 49, 20]. Orthogonal to the above studies, we propose to parameter-inverted design to avoid using large models to process high-resolution
images, greatly reducing the computation redundancy.
3 Parameter-Inverter Image Pyramid Networks
To construct efficient image pyramid networks, we employ a multi-branch structure handle im- ages of different resolutions with different sizes of models. As shown in Fig. 2, our architecture con- sists of three parts: multi-resolution branches, cross-branch interactions, and branch merging. Each branch uses an off-the-shelf pre-trained model to process images of different resolutions, where larger resolutions are processed by branches with fewer parameters. Cross-branch interactions are added every few blocks to fuse features across different feature scales. Branch merging combines the outputs from all branches to form a final output. We use the existing pre-trained ViTs [43, 44, 41] to initialize the branches, and initialize the interactions and branch merging from scratch.
3.1 Multi-Resolltion Branches
The multi-resolution branches serve to extract representations from different image scales and se- mantic levels. The input image is first resized to different resolutions through bilinear interpolation, and then fed into corresponding branches to extract features at different scales. All the branches have the same number of blocks N, where each block contains one or multiple ViT [13] layers. Typically, blocks from different branches have different feature dimensions due to the pre-trained models, e.g. ViT-T, ViT-S and ViT-B. Branches with larger image sizes have smaller number of parameters. For clarity, we refer to the branch with the largest number of parameters (with the smallest image size) as Branch 1, the second largest as Branch 2, and so on. The output of the i-th block of Branch is
3

---

Branch (e.g. VIT-B)
Nx
Patch Embed
Block
Proj
Det
Interaction
Branch 2 (e.g. VIT-S)
Patchh Embeed
Block
Proj
Seg
Interaction
Brench (e.g. VIT-T)
Unit
Patch Embed
Block
Proj
CLS
Pan
Element-vise Addition
Positin Embecding
Upsampling
Figure 2: Overall architecture of our method. We use rustiuresolttion branches to process images of different resolutions, where larger images handled by smaller models. Interaction Units build connections between branches. Branch merging combines the features of all branches to form the final output. Our architecture leverage pre-trained models with different model sizes build efficient image pyramids.
denoted as Fj E H,,,//3xD, where H,, Wj. Pj. D, are the image height, image width, patch size, and feature dimension of Branch j. respectively.
3.2 Cross-branch Interactions
Branches of different resolutions focus on different spatial scales and semantic levels. To enhance the features different scales. we propose the cross-branch interactions. Each cross-branch interac- tion consists of several interaction units, where each unit builds connections between outputs from two feature-scale adjacent branches. The structure of the interaction unit shown in Fig. 3.
Specifically, for the outputs of the i-th block of Branch and 2, denoted as Fi "A and F3 E IR H2 D2, we perform two deformable cross-attention [63] between the two features, denoted as Attention(-). Each cross attention is preceded by a linear layer FC(.) to project the feature dimension of key and value into that of the query, i.e. from Dj to D2 or vice versa. A feed- forward network (FN is added after each cross attention provide channel-wise feature fusion. The hidden dimension ratio of FFN is set to 0.25 to save computational overhead.
For the first aross-attention in the interaction unit, the interaction process can be formulated as: = 7f Anttenttio(norm(FF)) norm(FC((F))) = (FFF((norm(FF)
(2)
where norm(.) is LayerNorm [1], of and of are learnable parameters, and Fi is the interaction output. of and 71 are initialized with O to ensure that the feature extraction of the original blocks (i.e. distribution of Fi) will not be modified drastically due to the interactions, better utilizing the pre-trained weights.
Similarly, the second cross-attention is performed by switching the query and key/value obtain Az. The outputs F and Fi are used for subsequent feature extractions. We only construct interaction units between each pair of feature-scale adjacent branches, such as Branch & Branch 2 and Branch 2 & Branch
3.3 Branch Merging
The final feature maps of all branches J:N have different spatial shapes and feature dimensions, where spatially larger feature maps have fewer feature dimensions. A single feature map fails to pro- vide multi-scale semantic features, of all branches into a single feature map.
employ the branch merging module to merge the outputs
As shown in Fig. 2, all branch outputs are first projected to the feature dimension of Branch 1 (the largest feature dimension) with Proj(.). Then, all branch outputs are upsampled by bilinear interpolation Upsample( +) into the feature map size of the last branch (the largest feature map size).

---

Branch Block
F2
P2
F2
Branch 2 +
Deform Attn
FFN
key / valuue
Deform Attn
value
FC
Branch Block
.F
query
Branch 1 Block +
Figure 3: Structure an interaction unit.
Finally, these outputs, with the same spatial shape and feature dimension, are added together with learnable scalar weights wy to form the final output. This process be formulated as:
Fr
Upsample(Proj(E?)
M your Σ j=1
fout
where M is the number of branches. Fout is the final feature map, which has the largest feature resolution and also the largest feature dimension across all branches.
For object detection and semantic segmentation, Proj(-) is a wwo-convolution layer with Group- Norm [51], and the final output Fout is used for feature pyramid network [27] similar ViTDet [23].
For image classification, we do not use the branch merging module, but instead append the original classification heads of the pre-trained models after each branch. The final classification score the average of the output logits of all branches. We observe that using the pre-trained heads can speed up convergence compared to using randomly initialized head after branch merging module.
4 Experiments
4.1 Implementation Details
For comparison with Base-size models, use pre-trained ViT-T/S/B the branches to construct three-branch PIIP network, namely PIIP-TSB. Similarly, ViT-S/B/L. are used to construct PIIP-BIL to match the computation of Large-size models. We also construct four-branch PII--TBLL with ViT-T/S/B/L. We set the number interactions (each with interaction units as shown Fig. 2) N to 12, i.e. after every layer for VIT-T/S/B after every two layers for VIT-L. We construct multiple variants of three-branch and four-branch models with different resolution configurations. For combinations with inconsistent number of layers, will the backbone with fewer layers. For example, for ViT-S/B (12 layers) and ViT-L (24 layers), the learning rate decay for ViT-S/B is set to be twice that of ViT-L (24/12=2).
larger learning rate decay for
For object detection III [44]. VIT-T from DeiT [43]. ViT-H from MAE [18] and Intern scale experiments. For all PIIP-BL models, we use the ImageNet-2IK 384-resolution pre-trained weights to compare with previous approaches. We adopt AdamW [32] optimizer with layer- layer-wise learning rate decay [2] to train the model on NVIDÍA A800 GPUs. For image classification, in Base-size experiments experiments, since DeiT' does not provide ViT-L models, we use ImageNet-21K pre-trained ViT. S/B/L weights from [41].
segmentation, we use ViT-S/B/L pre-trained on ImageNet [11] from DeiT ViT-6B [8] are used for 6B-
use pre-trained VIT-T/S/B weights from DeïT [43]. In Large-size
use the FLOPs calculation script from MMDetection [6], with our modifications to accurately calculate FL.OPs of modules like self-attention and deformable attention. The script is released along with the training code. We have also manually verified the calculations using formulas, and the results consistent with those produced by the script.

---

Table 1: Comparison with baseline COCO val2017. We report the number of parameters and FLOPs of backbone. Underline indicates FLOPs or metrics on par with baseline. Ap and APm represent box AP and mask AP, respectively.
Mask R-CNN 1x schedule
Model
Resolution
#Param
#FL.PPs
APh
AP
AP%
Apm
Al'o
AP76
ViTDer-B [23]
1024
90M
463G
43.8
67.6
47.7
39.9
63.6
42.2
1120/896/48
146M
243G
43.9
65.7
47.5
38.6
61.8
40.6
PUP-TSB (ours)
1568/896/448
147M
287G
45.0
67.0
48.7
40.2
63.8
42.6
1568/1 120/672
149M
453G
46.6
68.4
51.1
41.4
65.2
44.3
ViIDet-L [23]
308M
1542G
46.8
70.8
51.4
42.5
67.3
45.3
1120/672/448
493M
727G
46.7
69.0
50.6
40.8
65.2
42.8
PIIP-SBL (ours)
1344/896/448
495M
1002G
48.2
71.0
52.8
42.5
67.3
45.4
1568/896/672
497M
1464G
49.4
71.9
53.9
43.7
68.4
46.6
1344/896/672/448
506M
755G
46.9
69.9
50.6
41.6
65.9
44.1
PIP-TSBL (ours)
1568/1 120/622/488
507M
861G
48.2
70.5
52.7
42.8
66.9
45.6
1792/1568/1 120448
512M
1535G
49.6
72.4
54.2
44.2
47.5
50
44
48
42
AP box
46
AP mask
VIT-B
VIT-B
44
VIT-L
38
VTT-L
PSB
PIIP
42
PIIP-TSI
PIIP-SBL
36
PIIP-TSAI
PIIP-SBL
200
400
800
1600
200
400
800
1600
GFLOPs
GFLOPs
(a) Object detection
(b) Instance segmentation
Figure 4: Performance of different PIIP variants by adjusting input resolutions. Detailed resolution con-
liguration and results provided in the appendix.
4.2 Object Detection and Instance Segmentation
Settings. The MS COCO [28] dataset is used to evaluate the performance on object detection and in- stance segmentation. We use three detectors, including Mask R-CNN [19], Cascade R-CNN [4] and DINO [59], based on MMDetection [6]. Following common practices [7]. we adopt x (12 epochs) or 3x (36 epochs) training schedules and use window attention [23] save time and memory. The total batch size is 16, and the initial learning rate and weight decay are le-4 and 0.05.
Effectiveness of Parameter-Invertee putational advantages of the Parameter-Inverted Image Pyramid (PIP) Networks, we perform val- idation on two baseline models ViTDet-B and ViTDet-L. [23] in Tab. 1. Taainn the three-branch structure as an example, while maintaining similar performance with ViTDet-B, our PIP--SB re- duces the computational cost by 47.5% (243G vs. 463G) and 38.0% (287G vs. 463G) in object detection and instance segmentation tasks respectively. Similarly, compared with ViTDet-L, our PIIP-SBL reduces the computational cost by about 52.9% (727G vs. 1,542G) and 35.0% (1,002G vs. 1,542G) in the above two tasks respectively. On the other hand, with similar computational cost as the baseline, PIIP-SBB and PIIP-SBL improve the object detection performance by 2.8% and 2.6%, respectively, and instance segmentation by 1.5% and 1.2%, compared to ViTDet-B and ViTDet-L. better illustrate the above conclusion, we depict trend between the computational cost and performance of different PIIP model combinations by adjusting the input resolution, as shown in Fig. 4. Furthermore, when we use the four-branch structures, the curve in the figure slightly better than that of the branch structure.
Image Pyramid. To demonstrate the performance and

---

Table 2: Object detection and instance segmentation performance COCO val2017. 'MS' means using AutoAugment [10] for multi-scale training. Large-size models use ViT weights trained on ImageNet-21K, The ViTDet-B and ViTDet-L. results (and other entries) are cited from ViT-Adapter [7]. PIIP-BL with Mask R- CNN uses higher resolutions than those in Tab. 1, as reported in Tab. 12. For PIIP-TSB with Mask R R-CNN, higher resolutions (1568/896/672 1792/1344/672) and larger window size (14 > 28) used, compared with the results in Tab. 1.
Method
Ap
AP
AP%
APm
AP
AP%
Method
AP
AP
AP
AP"
AP
AP%
Musk R-CNN
1x schedule
Cascade R-CNN Ix schedule
PVTv2-B5 (49)
47.4
68.6
$1.9
42.5
65.7
46.0
Swin-1. [50]
51.8
56.2
44.9
68.4
VIT-R [24]
42.9
65.7
39.4
62.6
Con-NoXt-L [3]]
53.5
72,8
58.3
46.4
70.2
50.2
WiDDet-B [23]
43.2
65.8
39.2
62.7
41.4
PUP-SRL (ours)
53.6
73.3
57.9
Swin-B B [30]
46,9
42.3
Cascade R-CNN
MS schedule
VTT WT--Adppeee-B [7]
47.0
68.2
51.4
41.8
65.1
44.9
Swin-B [30]
$1.9
70.9
57.0
PIPP TSB (ours)
47.9
70.2
52.5
42.6
67.2
45.5
Shuifle-B [23]
$7.0
VTTBB [24]
50.1
69.3
54.3
[IT-L[224]
45.7
68.9
41.5
65.6
VIT-Adapter-B [7]
52.1
70.6
56.5
WiTDerLL [23]
4622
69.2
41.4
65.8
44.1
PMP PUB (ours)
53.1
72.3
57.4
46.5
70.1
51.1
VIT VIT-Adapae-L [7]
48.7
70.1
53.2
43.3
67.0
46.9
PHP--SHI (ours)
49.9
72.8
54.7
44.6
69.3
47.9
Swin-L. [30)
53.9
72.4
58.8
46.7
70.1
50.8
RepLKNet:31L111
72.5
46.5
DINO + MS 3x schedule
ComNNettL [31]
54.8
73.8
59.8
47.6
71.3
51.7
PHI--SBI-33x (owns)
76.9
PIP--BL SBL (ouns)
54.5
73.8
59.1
47.7
71.6
52.1
Table 3: Experiments on the large-scale vision foundation model Vntern
Mask R-CNN x schedule
UperNet 160k
Model
#Param
#FLOPs
Resolution
AP
Ap
Crop Size
#FLOPs
mloU
Intern ViT-6B [8]
5919M
24418G
1024
53.8
48.1
512"
6105G
58.36
7269M
5643G
1280/1024/256
53.5
47.5
640/5112/192
1903G
57.82
PHIP-LH6B (ours)
7271M
10368C
1280/1024/512
47.8
640/5122/256
2592G
7273M
1391 IG
1280/1024/640
55.7
49.0
640/5122/384
4560G
59.65
Results with Base-size and Large-size models. As shown in Tab. 2, combined with Mask R-CNN. PIIP achieves higher performance than ViT-Adapter by considerable margin, about 0.9% and 1.2% on APh With more powerful detector Cascade R-CNN and stronger training schedule (3x + MS), PUP--SB and PIIP-SBL achieve competitive performance of 53.1 1% and 54.5% APb, respec- tively. Finally, we achieve 57.9 APb with the DINO [59] detector. These results demonstrate the scalability of PIIP.
Results with Intern VIT-6B. We further examine PIIP on an extremely large vision foundation model ViT-6B [8]. As can be seen from Tab. 3, PIIP-LH6B finally achieves 55.7% Apb when using Mask R- R-CNN training schedule. In addition, our PIIP can save nearly 43% of the computation and achieve better performance than the single-branch InternViT-66 by 1.9% on APb and 0.9% on Apm.
4.3 Semantic Segmentation
Settings. We use UperNet [54] as the basic framework to train on the ADE20K [62] dataset based on MMSegmentation [9]. We follow the settings of [30] to train the model for 160k iterations. The batch size, initial learning rate and weight decay are 16, 4e-5 and 0.05.
Results with Base-size and Large-size models. In Tab. 5, PIIP achieve better performance with fewer computations compared with single-branch baselines. In Tab, 4, we compare PIIP with state-of-the-at segmentation backbones. PIIP-TSB attains 51.6% mloU with UperNet, exceeding InternImage-B B [48] by 1.4%. Similarly, PHIP-BL yields 54.3% mIoU, which is outstanding com- pared to counterparts like ConvNeXt-XL [31] and Internlmage-L [48].
Results with ViT-6B. As shown in Tab. 3, similar to the conclusions obtained the object detection experiment, our method achieves better performance than the Intern Intern baseline with less computation. PHIP-I.H6B finally achieves 59.655 59.65% mIoU without using additional optimization techniques.
7

---

Table 4: Semantic segmentation performance ADE20K using UperNet.
Table 5: Comparison with baseline ADE20K
using UperNet.
Method
Crop Size
mloU
Method
Crop Size
#FLOPS
mloU
VIT-B
640"
159G
51.0
Swin-B [30]
5122
48.1
PUP-TSB (ours)
896/448//33
118G
51.6
ConvNeXt-B [31]
5122
49.1
VIT-L
640"
545G
53.6
RepLKNNet-11 [12]
5122
49.9
PIP PHPP-SEL (ours)
1120/448"3336
456G
54.3
SLaK-B [29]
5122
Intemlmage-B [48]
5122
Table Image classification performance ImageNet. Underline indicates FLOPs or metrics on par with the baseline.
PHIP-TSB (ours)
/96/4489/336
51.6
Swin-L [30]
640
52.1
RepL.KNet331L [12]
640²
52.4
Model
Resolation
#FLOPs
Top-1 Ace
DeTT-B [43]
224
17.2G
81.8
ConvNeXt-L [31]
640
PUP-TSB (ours)
368/192/128
17.40
ConvNeXt-XL [31]
640
53.6
VITL
840
InternImage-1. [48]
640
53.9
WTEL [41] four impl.)
224
61.66
85.2
PIP-SBB (ours)
320/160/96
39.00
85.2
PIIP-SBL (ours)
/130/4489/336
54.3
PIP SBL-SBL.C (ours)
384/192/128
61.2G
85.9
Table 7: Ablation image pyramid and parameter-inverted design. 'Pr, 'IP' and 'Inter'' represent parameter- invertedd,, image pyramid and interactions. 'MS' means multi-scale training. following [10].
Mask R-CNN schedule
Figure
Branches
IP Inter.
Resolution
#Param
#FLOPs
Aph
A
AP
Apm
AP
AP
Fig. I(a)
B
1024
90M
463G
43.8
67.6
47.7
39.9
63.6
42.2
Fig. ((b)
B
MS
90M
463G
44.8
69.2
49.1
41.0
65.8
43.9
BBB
896/4-18224
262M
369G
43.3
65.8
46.6
37.9
61.5
39.6
BBB
896/67/2/244
263M
457G
43.8
66.3
47.3
38.2
62.2
39.7
Fig. Ite)
BBB
896/4418/224
341M
466G
44.5
66.5
48.2
38.7
62.6
40.6
TSB
896/896/896
148M
468G
44.6
66.4
48.3
39.0
62.7
41.4
Fig. ((d)
TSB
448/672/896
147M
452G
42.6
64.2
45.6
36.5
59.5
38.0
Fig. ((e
TSB
1568/1 21V672
149M
453G
46.6
68.4
51.1
41.4
65.2
44.3
Fig. ((a)
L
1024
308M
1542G
46.8
70.8
51.4
42.5
67.3
45.3
Fig.
LLL
896/448/224
1053M
1458G
69.7
51.2
40.8
65.3
43.3
SBL
848/848/848
495M
1539G
69.4
$1.0
65.4
43.7
Fig.
SBL
1568/896/672
71.9
53.9
43.7
46.6
4.4 Image Classification
Settings. We load the pre-trained models for each branch and train the model for 20 epochs ImageNet-IK [11]. The batch size, initial learning rate and weight decay are 1024, 3e-5 and 0.1. The learning rate for the random initialized interactions is 10 times the base learning rate, i.e. 3e-4. The other settings mainly follow the fine-tuning recipe of [44] and are provided in the appendix.
Results. As shown in Tab. 6, when compared with the DeiT baseline, our PIIP-SBL reduces the computational cost by 36.7% (39.0G vs. 61.6G) while maintaining the performance. When using similar computational cost the baseline models, PIIP-TSB and PIIP-SBL improve the top-1 accuracy by 0.3% and 0.7%, respectively.
4.5 Ablation Study
Superiority of parameter-inverted image networks. We evaluate the effectiveness of the image pyramid and parameter-inverte design by comparing our method with other methods, e.g. designs in Fig. 1. First of all, a single-brand branch with multi-scal training is the simplest image pyramid practice, as shown in Tab. (44.8% vs. 33.8%). Secondly, we conduct experiments by controlling the scale of the branch model and the input resolution while ensuring that the total computational cost close. Specifically, when using the same input image resolution, the combination of models of different sizes does not bring
Compared with the baseline model, its performance improvement is limited

---

47
47
VIT-B
Inter12
Inter6
1344/8984445
15600966/44
1201606/672
(B44/89616772
45
Interl
AP box
12208961448
48 box
44
Inter2
Interl
InterO
42
VIT-B
PIIP-TBL PIIP-TBB
40
200
250
300
350
400
450
500
550
100
200
300
400
500
GFLOPs
LOL
(a) Variants with different resolutions
(b) Number of interactions
Figure 5: Ablation on model variants and number of interactions.
Table 8:
Ablation Branch
Table 9: Ablation attention type and number of interactions with
Merging on COCO val2017.
PIIP- TSB 1120/896/448.
We use PHPP-TSB 1568/896/672.
Regular Attention
Deformable Attention
AInter.
Out Branch
APb
Apm
#FLOPs
AP3
API
AP
APb
#FLOPs
AP3
AP
AP
AP
B
43.1
37.0
176G
41.3
59.0
44.6
22.5
176G
41.3
59.0
44.6
22.5
211G
41.1
59.1
44.9
22.6
182G
41.9
59.8
45.5
22.4
T
45.6
40.6
245G
41.7
59.5
45.2
22.7
1876
42.5
60.5
46.4
23.1
B+T
46.3
41.1
315G
41.6
59.2
45.3
22.8
198G
43.0
61.0
47.3
23.3
S+T
46.2
40,9
6
384G
42.1
59.7
45.8
23.2
210G
43.3
61.8
46.9
23.6
B+S+T
46.6
41.4
12
592G
42.0
60.0
45.9
23.1
243G
43.9
62.4
47.9
24.4
significant improvements to detection performance. Correspondingly, when the three branches the same model (e.g. BBB), the input image resolution adjusted to the pyramid structure. The performance of the final model is slightly improved APb (44.5% 43.8%), but the Apm drops significantly (38.7% vs 39.9%) due to the reduction of the maximum resolution. The former demonstrates the importance of the image pyramid, and the latter further demonstrates the need for the image pyramid to maintain a larger image scale range, which is especially essential for instance segmentation. Drawing experience, parameter-inverted image networks design method that can meet the above requirements, especially when compared to its opposite configuration parameter-direct image pyramid, i.e. TSB with 448/672/896 resolution (46.6% vs. 42.6%). As shown in Tab. 7, with less computation than the baseline, the model can support image inputs in the maximum range from 672 to 1,568, and the performance is significantly improved.
efficient
Design guidelines for parameter-inverted image networks. Through extensive practice, there empirical design guidelines when scaling up the model: 1) Prioritize increasing the image resolution of the largest image branch: as shown the blue dashed circle in Fig. 5(a), the input resolution of the largest image branch is greatly increased without causing sharp increase in the total computational cost. 2) The largest model does not need to exceed the compared baseline model: the introduction of larger models will limit the resolution range of the image pyramid, e.g. TSB is more cost-effeecivee than TBI. according to Fig. 5(a).
Branch merging. Experiments in Tab. 8 prove that branch merging of all branches yields the best performance by providing multi-scale semantically rich features, compared to only using feature maps from single or partial branches.
Attention type. The core of information interaction between branches cross-attention. We adopt PIIP-TSB with resolution 120/896/448 as the basic model and investigate two different attention mechanisms. As shown Tab. 9, deformable attention [53] with linear complexity can significantly improve the performance of the model without substantially increasing the computational cost. We end up using deformable attention as the default configuration. Notably, it can be replaced by other more advanced attention mechanisms in the future further boost performance.

---

Table 10: Ablation interaction directions with PIIP-TSB under resolution 1120/896/448.
Type
#FLOPs
210G
230G
230G
243G
283G
APh
43.5
43.2
43.6
43.9
44.0
Apm
38.7
38.3
38.6
38.6
38.7
VIT-B
Type1
46
Type2
Type3
bbbox
Type4
Type5
200
250
300
350
400
450
GFLOPs
Figure 6: Performance of different interaction directions.
Number of interactions. As shown in Tab. 9, matter which attention mechanism is used, the increase in the number of interactions will improve the performance of the model to varying degrees. Since it also increases the computational cost, further explore the cost-effectiveness of different numbers of interactions. We conduct experiments with different resolution combinations on models with different numbers of interactions, and the scatter plot of all results is shown in Fig. 5(b). can be seen that when the number of interactions is small (less than 2). the growth trend of model performance with the increase computational cost is relatively slow. We attribute this too few interactions and insufficient information complementation between branches. Therefore, we use 12 interactions by default. Note that as the model size increases (e.g. more layers), the number of interactions can also increase accordingly.
Interaction direction between branches. We compare five different interaction directions in Tab. 10. Considering both the computational cost and performance, finally choose the fourth method, í.e. bidirectional connections of adjacent branches, as the default choice. As can be seen from Fig. 6, all the interaction directions achieve satisfactory performance-computation balance, validating their ability to improve communication between branches.
Conclusion
This paper introduces the Srnameter-Inverter Image Pyramid Networks (PIIP) address the com- putational challenges of traditional image pyramids. With the purameter-inverted design and fea- ture interaction mechanism, PIIP effectively balances computational efficiency and performance. Extensive experiments on detection, segmentation and classification tasks demonstrate that PIIP outperforms traditional methods and single-branch networks while reducing computational costs, providing an efficient and effective framework of multi-scale feature integration for future research.
Limitations. While our method manages to save computation, its memory consumption is higher than single-branch models due to the increase of parameter count. Our current method only focuses on ViT-based models. PIP with hierarchical networks (e.g. CNN) or heterogeneous structures (e.g. CNN for some branches and ViT for other branches) remain unexplored for future work.
Acknowledgement
This work is supported by the National Key R&D Program of China (NO. 20227D0161300, NO. 60000,. by the National Natural Science Foundation of China (62376134).
10

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv:1607.0644 2016.
[2] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
[3] Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi scale attention for high-resolution dense prediction. In Proceedings the IEEE/CVV International Conference Computer Vision, pages 17302 17313, 2023.
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 6154 6162, 2018.
[5] Chun-Fu Richard Chen, Quanfu Fan, Rameswar Panda. Crossvit: Cross-attention multi-scale vi- sion transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 357-366, 2021.
[6] Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benehmark. arXiv preprint arXiv:/9066 wet.:/9000.55550 07155, 2019.
[7] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In International Conference Learning Representations, 2022.
[8] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li. Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisee tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024.
[9] MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. pettp::/giithn cww/openn mml mmmlaa/mmmeementaioon 2020.
[10] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 1rXiv:1805.09500 2018.
[11] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In Proceddings of the IEEF/CVF conference on computer vision and pattern recognition, pages 248-255. 2009.
[12] Xiaohan Ding, Xiangyu Zhang. Jungong Han, and Guiguang Ding. Scaling up your kemels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 1 11963-11975, 2022.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
[14] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wa, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358- 19369, 2023.
[15] Inc. Flickr. Flickr terms & conditions of use. https: //wp:://wwww.. flickr wwt.//oloe/eeeeeeeeeeeeeeeeeeeee
[16] Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V Le. Nas-Ipa: Learning scalable feature pyramid architecture for object detection. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 7036-7045, 2019.
[17] Jiaqi Gu, Hyoukjun Kwon, Dilin Wang. Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan. Hrvit: Multi-scale high-resolution vision transformer. arXiv preprint arXiv:2111.01236. 2021.
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.

---

[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2961-2969, 2017.
[20] Byeongho Heo. Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Re- thinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 19936-11945, 2021.
[21] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.0999 2023.
[22] Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint 1exii:2/06600666 2021.
[23] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference Computer Vision, pages 280-296, Springer, 2022.
[24] Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021.
[25] Tingting Liang, Xiaojie Chu, Yudong Liu, Yongtao Wang. Zhi Tang, Wei Chu, Jingdong Chen, and Haibin Ling. Cbnet: A composite backbone network architecture for object detection. IEEE Transactions on Image Processing, 31:6893 09006,696622222 2022.
[26] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xic. Not all patches are what you need: Expediting vision transformers via token reorganizations. In International Conference on Learning Reprexentations, 2022.
[27] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pages 21 17-2125, 2017.
[28] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. Proceedings of the European conference on computer vision, pages 1400-55,222 2014.
[29] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäänen, Mykola Pechenizkiy, Decchal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprini arXiv:2207.0300 2022.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[31] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 1976-11986. 2022.
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
[33] Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: ofssuuiooo adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024.
[34] Yihao Luo, Xiang Cao, Juntao Zhang, Jingjuan Guo, Haibo Shen, Tianjiang Wang, and Qi Feng. Ce-fpn: enhancing channel information for object detcction. Multimedia Tools and Applications, (166)):30855- 30704, 2022.
[35] Lingchen Meng, Hengduo Li, Bor- Chun Chen, Shiyi Lan, Zuxuan Wu, Yu- Yu-Gang Jiang, and Ser-Nan Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309 12309-12318, 2022.
[36] MIT. Ade20k terms & conditions of use. httpp:://gropp csail. wetaail..it.eiieeee.ee saai.eett/risionnattastss ADE20K/ /eeo0O//eemm....
[37] Mahyar Najibi, Bharat Singh, and Larry S Davis. Autofocus: Efficient multi-scale inference. In Proceed- ings of the IEFF/CVF international conference computer vision, pages 9745-9755, 2019.
12

---

[38] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu. Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Effi- cient vision transformers with dynamic token sparsification. Advances in nearal information processing systems, 34:13937-13949 2021.
[39] Bharat Singh and Larry S Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3578-3587, 2018.
[40] Bharat Singh, Mahyar Najibi, and Lary Davis. Sniper: Efficient multi-scale training. Advances in neural information processing systems, 31, 2018.
[41] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv: 106. 10270, 2021.
[42] Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pages 10781- 10790, 2020.
[43] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Con- ference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[44] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European conference on computer vision, pages 516-533. Springer, 2022.
[45] Princeton University and Stanford University. Imagenet terms & conditions of use. org/ dag/ddwwload.
[46] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEFE transactions on pattern analysis and machine intelligence, 43 (3(10):3339-3366 2020.
[47] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint urXiv:2006.04777 2020.
[48] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with de- formable convolutions. In Proceedings of the IEEFSCVF Conference on Computer Vision and Pottern Recognition, pages 14408- 14419, 2023.
[49] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEFF/CVF international conference on computer vision, pages 568-578, 2021.
[50] Qinrou Wen, Jirui Yang. Xue Yang, and Kewei Liang. Patchdet: Patch refinement for high quality instance segmentation. In The Eleventh International Conference on Learning Representations, 2022.
[51] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on com- puter vision, pages 19, 2018
[52] Chunlong Xia, Xinliang Wang, Feng Lv, Xin Hao, and Yifeng Shi. Vit-comer: Vision transformer with convolutional multi-scale feature interaction for dense predictions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2024.
[53] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4794-4803, 2022.
[54] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision, pages 418 434, 2018
[55] Yifan Xu, Zhijic Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Chang- sheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2964-2972, 2022.
[56] Xue Yang, Junchi Yan, Ziming Feng. and Tao He. R3det: Refined single-stage detector with feature refinement for rotating object. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 3163-311 3171, 2021.
13

---

[57] Xue Yang. Jirui Yang. Junchi Yan, Yue Zhang, Tengfei Zhang, Zhi Guo, Xian Sun, and Kun Fu. Scrdet: Towards more robust detection for small, cluttered and rotated objects. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9232-8241, 2019.
[58] Yuhui Yuan, Rao Fu, Lang Huang. Weihong Lin, Chao Zhang. Xilin Chen, and Jingdong Wang. Hrformer: High-resolution vision transformer for dense prediction. Advances Neural Information Processing Systems, 34,20212 2021.
[59] Hao Zhang, Feng Shilong Liu, Lei Zhang, Hang Su, Zhu, Lionel M Ni, Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations, 2022.
[60] Kaipeng Zhang. Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23( 23(10):1499 1900:199995,,,, 2016.
[61] Tianwen Zhang, Xiaoling Zhang, and Xiao Ke, Quad-fpn: A novel quad feature pyramid network for sar ship detection. Remote Sensing, 13(14):2771, 2021.
[62] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 633 631, 2017.
[63] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference Learning Representations, 2020.
[64] Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In Proceedings of IEEE/CVF international conference on computer vision, pages 6748 6758, 2023.
14

---

A Appendix
A.1 Detailed Training Settings for Image Classification
Detailed training settings for image classification are provided in Table 11.
A.2 Full Detection Results
Full results of Fig. 4 are provided in Table 12.
Table 11: Detailed training setting for image classification.
batch size
1024
epochs
20
optimizer
AdamW
weight decay
0.1
learning rate scheduler
cosine
initial learning rate
3e-5
warmup epochs
5
mixup
0.8
cutmix random crasing
1.0
0
auto augment color jitter
/
0.3
label smoothing dropout
0.1
X
0.4 (ViT-I.) / 0.2 (ViT-B)/ /
drop path rate
0.05 (ViT-s, ViT-T)
repeated aug gradient clip loss
x
x
cross entropy
A.3 Broader Impacts
Our method helps to save computational overheads of large-scale vision foundation models such as Intern ViT-6B, therefore reducing energy consumption. This may bring positive impacts on carbon emissions reduction and contribute to environmental sustainability. However, energy consumption of large models still needs be treated with caution.
15

---

Table 12: Full results of PII variants under different resolution configurations.
Mask R-CNN
schedule
Model
Resolution
#FLOPs
AP
AP
AP
AP
APm
API
AP
AP.
896/672/448
176G
42.1
62.2
46.8
20.8
36.9
60.9
40.2
13.7
1120/672/448
42.7
22.7
37.9
61.2
15.4
1344/672/488
212G
43.5
62.1
47.2
23.5
38.9
60.9
41.7
16.2
1120/896/448
43.9
62.4
47.9
38.6
16.6
1344/896/448
263G
44.5
62.1
48.3
24.9
39.5
61.1
42.6
17.5
PIIP-TSB
1568/896/448
287G
45.0
62.0
48.4
26.2
40.2
61.4
43.3
19.0
1568/896/672
387G
45.8
62.9
49.9
27.2
40.7
62.3
44.1
19.5
1568/1 120/672
453G
46.6
63.1
50.9
28.5
41.4
62.3
45.0
20.6
1792/1 120/672
480G
46.7
63.0
50.6
29.0
41.7
1792/1344/672
561G
46.8
62.5
50.8
30.1
42.0
62.5
45.1
21.8
672/448/224
245G
41.1
63.6
45.8
18.4
35.3
61.5
38.4
10.7
896/448/224
298G
43.5
63.9
47.8
37.7
14.3
1120/448/224
367G
45.2
63.7
49.4
25.2
39.6
62.9
42.9
16.7
1120/672/224
45.8
64.7
26.1
40.3
63.3
43.8
120/672/448
727G
46.7
63.0
50.6
29.0
40.8
44.1
PUP-SBL
1344/672/448
811G
47.5
65.8
51.7
27.6
42.0
64.7
45.7
19.5
1344/896/448
1002G
48.2
66.2
28.8
42.5
46.2
1568/896/672
1464G
49.4
66.5
53.9
30.6
43.7
64.9
47.5
22.0
1568/1 120/672
49.9
66.9
31.7
44.3
1792/1 120/672
1824G
49.9
65.9
54.3
32.0
44.6
65.4
48.3
23.1
1344/896/77/448
755G
46.9
65.5
50.4
27.8
41.6
64.4
44.7
19.5
1568/1 120/672/448
861G
48.2
66.1
52.0
29.4
42.8
64.7
46.0
21.0
PIPP-TBL
1568/1 120/896/448
1052G
48.7
66.4
52.4
30.2
43.4
65.2
46.7
21.4
1792/1 344/896/448
1180G
49.0
65.9
52.7
30.5
43.7
65.0
47.0
22.4
1792/1568/1 120/448
1535G
49.6
65.7
53.1
32.1
44.2
65.2
47.5
22.9
A.4 Licenses of Datasets
ImageNet-Ik [11] is subject the ImageNet terms of [45].
COCO [28] is subject to the Flickr terms of use [15].
ADE20K [62] is subject to the ADE20K terms of use [36].
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions are stated in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The details are provided the experiment section and the appendix. Our code has been released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The data is publicly available datasets. Our code has been released. Details to reproduce the experiments is described in the experiments section and the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
for more details.
While we encourage the release of code and data, understand that this might be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: gwwp//u////////eg/// /CodeSumm ssi inloiicy) for more details.
/
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Training details in the experiment section and appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Calculating error bars would be too computational expensive, given the size of the model and the dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Details are in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / //trr//eersp/.ee/ee http//uulic/tthicosiiaeee
Answer: [Yes]
Justification: the paper conforms the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Datasets are referenced. License are provided in the appendix.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode com/ datassts has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
Documentation is be provided along with the code and models.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "qsCnyiOCEf", "reviewer": "Reviewer_H3y6", "summary": "The authors propose a novel vision architecture, Parameter-Inverted Image Pyramid Networks (PIIP), which can be applied to different tasks, including classification, object detection, and instance or semantic segmentation. The authors aim to take advantage of the multi-scale information of image pyramids, without the usual excessive computational demands of processing an image in multiple different resolutions. The core idea is to use a family of networks of different computational requirements, e.g., ViT-S/B/L, and apply the lightest model (least parameters) to the image of the highest resolution in the pyramid, apply the second lighter model to the next bigger image, and so on, ending up to apply the heaviest model (most parameters) to the image of the smallest resolution. The intuition is that stronger models with more parameters can be used to extract semantically rich contextual features from images of coarser resolution, while lighter models can be efficiently applied to extract low-level features from high-resolution images. This way, the computational requirements of the different models are balanced, meaningful features from all scales are extracted, and images can be processed in higher resolution because the lighter and less expensive models are processing them.\n\nThe models are applied to the different images in the pyramid in parallel, and as features are extracted, the models communicate with each other through a proposed Interaction Unit. This way, features of different semantic meaning extracted from different models complement and inform each other. After the features from each individual model are extracted, they are merged to be used for the task at hand.\n\nThe authors perform extensive evaluation of PIIP on 4 different tasks, object detection, instance segmentation, semantic segmentation, and image classification. The authors initialize the PIIP backbones with pre-trained ViT models, and for each task they fine-tune appropriate heads. PIIP is compared to baselines of different scale, and in all experiments it demonstrates superior or comparable performance. \n\nIn addition, the authors perform multiple ablations, offering insights about the behavior of PIIP, and the importance of different design choices.", "questions": "1. In Section 4.1 the authors provide the pre-trained architectures they used for different tasks, how did they decide which models to use?\n\n2. What is the formula the authors use to calculate the FLOPs of the models?\n\n3. Why there are no comparisons with state-of-the-art models on image classification?\n\n4. In Table 8, what is the range of resolutions that the model in the second row is trained on?", "limitations": "The authors offer a short discussion of the limitations in Section 5, which I think should be expanded to address the memory requirements of PIIP.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n\n1. The core idea of using a parameter-inverted pyramid seemed to me counter-intuitive at first, since I would expect heavier models to be applied to the images of higher resolution, which contain more information. However, the authors make a good argument about why the parameter-inverted pyramid design is sensible, and I think it is a novel idea, that in its simplicity offers a valuable contribution to the community.\n\nQuality:\n\n1. The authors provide extensive experiments on multiple tasks, and in their experiments they control for the computational requirements of the models, offering meaningful comparisons.\n\n2. The experimental evaluation includes multiple ablations, which shed light on different aspects of the proposed architecture.\n\nClarity:\n\n1. The manuscript is well written, and easy to follow. The authors explain the intuition and the specifics of their method in detail, accompanying the text with clear visualizations.\n\nSignificance:\n\n1. In addition to the novelty of the proposed idea of the parameter-inverted pyramid, I think a useful conclusion of the paper is the importance of using higher resolution inputs, even if they are processed with models of smaller size. To my understanding, this is the main reason PIIP outperforms the baselines, and is highlighted by the authors in Section 4.5 (ln 260 - 266). The importance of the input resolution is not something new, but the authors offer more evidence about its impact, and importantly, they offer a new paradigm to take advantage of it.", "weaknesses": "Quality:\n\n1. My main concern about this work is that the authors don’t provide actual timings and memory measurements. Computational requirements are quantified through FLOPs, however, theoretical gains in FLOPs don’t always translate to benefits in practice when the compared architectures have considerably different designs. For example, the PIIP models may be applied in parallel, but the interactions between the features may force the models to wait for each other, adding a sequential element that may add to the latency. I want to clarify that I am not claiming that this is the case, but that actual timings beyond theoretical FLOPs are needed to offer conclusive evidence, especially when efficiency is one of the main claims of this work.\n\n2. About memory requirements, it seems to me that using a whole family of models may be prohibitive in many settings, so, memory should be explicitly measured. In addition, the authors control in their experiments for FLOPs, e.g., in Table 1 they show how PIIP models perform compared to baselines with similar FLOPs. I think it would be useful to control for memory too, e.g., in Table 1 the best PIIP models have more than 50% higher number of parameters compared to the baselines. Based on this, I think a question that naturally arises is what if the baseline was a larger model with equal parameters? If we have an extra memory budget, is PIIP the best way to use it? I would like to clarify that I don’t think the authors should necessarily have additional experiments controlling for memory, but if it is a disadvantage of PIIP, as it seems to be due to the parameter count, I think they should discuss it in the limitations.\n\nClarity:\n\n1. In Table 2, the results for ViTDet-B and ViTDet-L are different compared to Table 1, why is that? Similarly, in Table 2, the results of PIIP-TSB and PIIP-SBL are different compared to Table 1. For the PIIP-SBL model, I see that the scores in Table 2 match the scores of the best model reported in Table 12, where models use higher resolutions, is this the reason for the discrepancy? However, the reported performance of the PIIP-TSB in Table 2 is higher than any performance reported for PIIP-TSB in Table 12, so, how the performance reported in Table 2 is achieved? I think the authors should clarify the configurations of the models they use in their experiments.\n\n2. Some of the Table captions are not informative enough. For example, it is not clear why crop size is underlined in Tables 3, 6, 7. On a similar note, in the ablation about branch merging (ln 267 - 269, and Table 5), the authors don’t mention the dataset they use. I guess it is MS COCO, but I think it should be explicitly mentioned.\n\n3. In the caption of Table 8, it is mentioned “‘PI’ and ‘IP’, ‘Inter.’ represent parameter-inverted, image pyramid and interactions”, I think it should be “‘PI’, ‘IP’, and ‘Inter.’”.\n\n4. In Ln 262 the authors mention “green dashed circle in Fig. 5(a)”, but I think it should be “blue dashed circle”.\n\nSignificance:\n\n1. PIIP requires a family of backbones for its processing, and the authors use pre-trained models in their experiments, while they also include some experiments on training from scratch in the Appendix. I think the need to train multiple backbones, or the need to find multiple pre-train models, may be an undesirable overhead for the adoption of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for providing detailed comments and highlighting our strengths. We hope our response will address the reviewer's concerns.\n\n### **W1: Actual timing not reported; Gains in FLOPs don’t always translate to benefits.**\nWe acknowledge that the reduction of FLOPs does not guarantee the improvement of throughput, which is related to engineering techniques and specific hardware. When using PIIP-LH6B with resolution 1280/1024/640, PIIP can reduce the training time of InternViT-6B from 91 hours to 62 hours while increasing the box AP from 53.8 to 55.7, though the actual increase of speed (\\~32%) is less than the reduction of FLOPs (\\~43%). Achieving consistent improvements across all cases can be challenging and may require additional engineering optimizations beyond the scope of this research. Previous works such as MobileNets highlighted the reduction in FLOPs as a contribution. However, the initial implementations didn't match the theoretical speed improvements. After hardware-related optimizations, current implementations have become much more efficient. We hope future work will address this challenge of PIIP.\n\n### **W2: Control for memory; Larger baseline model with equal parameters; Discuss memory limitations.**\nTraining a baseline with an equal number of parameters may be difficult. For example, there is no foundation ViT model with a parameter count equivalent to PIIP-TSB (~150M). However, we can derive insights from the 4th and 5th rows in Tab.1. PIIP-TSB obtains a similar box AP as ViTDet-L with ~50% number of parameters. Their memory consumption during training is 9.6GB and 7.9GB, respectively. We appreciate the reviewer for identifying the memory requirements as a limitation of our work and will include a discussion in the limitation section.\n\n### **W3: Different results of ViTDet-B, ViTDet-L, PIIP-TSB and PIIP-SBL in Tables 1 and 2.**\nWe thank the reviewer for highlighting the clarity issue. The ViTDet-B and ViTDet-L results (and other entries) in Table 2 are cited from the paper of ViT-Adapter, while the results in Table 1 are reproduced by ourselves. \n\nThe discrepancy between PIIP-SBL results in Tables 1 and 2 is indeed from using higher resolutions, as reported in Table 12. For PIIP-TSB in Table 2, higher resolutions (1568/896/672 -> 1792/1344/672) and a larger window size (14 -> 28) are used, compared with the result in Table 1. We will include these explanations in the captions of Tables 1 and 2.\n\n### **W4: Crop size in Tables 3, 6, 7; Dataset used in branch merging ablations.**\nWe underline the crop size to ensure comparability with the baselines. The preprocessing process of semantic segmentation contains cropping from the original image, which is different from object detection that uses the whole image. In our method, the input image of Branch 2 is the cropped image, and the inputs of Branch 1 and Branch 3 are resized from the cropped image. Different initial crop sizes can lead to inconsistencies in the training data, so we annotate this to maintain consistency with the baseline settings.\nThe dataset used in Table 5 and all other ablations are MS COCO. We will revise the captions for improved clarity.\n\n### **W5, W6: Typos.**\nWe shall correct them in the final version.\n\n### **W7: Overhead of finding a family of backbones is undesirable.**\nThis overhead is negligible in most cases, as the open-source community offers pre-trained models of various model sizes and families. If models of different sizes within the same family are not readily available, it is feasible to adopt models from different sources, as we demonstrated in the paper. We also conducted several additional experiments using various pre-training combinations from different sources, mainly because DINOv2 and BEiTv2 do not provide ViT-S, as shown in the table below.\n#### **Table: PIIP-SBL 1568/1120/672 using Mask R-CNN 1x schedule with different pre-trained models.**\n| Pretrain    | Box mAP | Mask mAP |\n| ------ | ----- | ----- |\n| AugReg   | 48.3    | 42.6     |\n| DeiT III   | 50.0    | 44.4     |\n| DeiT III (S) + DINOv2 (BL) | 51.0    | 44.7     |\n| DeiT III (S) + BEiTv2 (BL) | 51.8    | 45.4     |\n\n### **Q1: How to decide the selection of different pre-trained models for different tasks.**\nIn practice, we do not have strict preferences for selecting pre-trained models for different tasks. We prioritize models from the same family, and if a specific model size is unavailable, we substitute with models from other families. For instance, we use ViT-S/B/L from DeiT III, and since DeiT III does not have Tiny size model, we use ViT-T from DeiT. In the above table, we also use the AugReg pre-trained model for detection as it is used for classification in the paper.\n\n###  **Q2: Formula for FLOPs calculation.**\nWe use the FLOPs calculation script from MMDetection, with our modifications to accurately calculate FLOPs of modules like self-attention and deformable attention. We will release this script alongside the training code. We have also manually verified the calculations using formulas, and the results are consistent with those produced by the script.\n\n###  **Q3: Comparison with SoTA on image classification is missing.**\nDue to limited time and resources, this paper does not focus on enhancing the classification performance or comparing it with SoTA. Our primary focus is on detection and segmentation tasks, which benefit more from higher input resolution and are the main target tasks of image pyramids. Besides, to maintain comparable FLOPs with the baseline, the selection space of input resolutions in classification is narrower compared to detection.\n\n###  **Q4: Range of resolutions of \"MS\" in Table 8.**\nWe use AutoAugment [1] for multi-scale training. Please refer to the *attached PDF* in the global response for detailed implementation.\n\n[1] Cubuk, Ekin D., et al. \"Autoaugment: Learning augmentation policies from data.\" arXiv preprint arXiv:1805.09501 (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NFELlPg9ex", "reviewer": "Reviewer_Lww4", "summary": "This paper proposes two techniques: 1) Models with different parameter sizes to process different resolution levels of the image pyramid. 2) A feature interaction mechanism to integrate information from different spatial scales. Extensive experiment results are used to support the claims.", "questions": "See above", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Both techniques are sound. Experiments are well documented and extensive.", "weaknesses": "It is unclear to me if the better accuracy in table 1, 2, 3 and 4 are entirely due to the proposed techniques or it could also be partially explained by the fact that the highest resolution in the pyramid is larger. Take table 1 as an example, the input resolution for baseline is 1024. But the highest resolution for PIIP is 1792. What if you use 1792 for baseline? It is perfectly fine to first hold FLOPS constant and check accuracy, as shown in table-1. But you should also hold input constant and check accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's comments and provide additional experimental results to address these concerns.\n### **Q1: Baseline with higher resolution is needed.**\nTo evaluate the impact of using higher resolutions, we add another baseline, ViTDet-L with 1792 resolution (matching the largest resolution of PIIP-TSBL 1792/1568/1120/448), as shown in the second row of the table below. The first and third rows are from Table 1. We observed that while ViTDet-L with 1792 resolution achieves better performance compared to the 1024 resolution, its FLOPs are approximately 4 times larger. Compared with PIIP-TSBL, ViTDet-L 1792 has a lower box AP (-1.3%) and 4 times larger FLOPs. This experiment well explains that the performance improvement does not entirely come from larger image resolution. The structure of PIIP also plays an important contribution in the calculation amount and performance improvement.\n\n| Model     | Resolution         | #Param | #FLOPs | box AP |\n| --------- | ------------------ | ------ | ------ | ------ |\n| ViTDet-L  | 1024               | 308M   | 1542G  | 46.8   |\n| ViTDet-L  | 1792               | 308M   | 6458G  | 48.3   |\n| PIIP-TSBL | 1792/1568/1120/448 | 512M   | 1535G  | 49.6   |\n\nWe hope that this experimental conclusion can allay your concerns and welcome further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VlHF17OyLB", "reviewer": "Reviewer_hVKM", "summary": "This paper presents a novel method named parameter-inverted image pyramid to handle the issues of high computation overhead of image pyramids. It uses different model sizes to process different resolutions. The method achieves significant results on tasks of object detection, segmentation, and classification by reducing overhead and improving performance.", "questions": "1. I am just a little curious, for PIIP-TSB， why ViT-T from DeiT but ViT-S/B from DeiT-III? Won't different pre-training methods have some impact?  According to Table 5, ViT-T with large resolution input plays the most important role, so the weaker pre-training ViT-T should limit the performance of the framework?\n2. (Unimportant) In Figure 4, why are the improvements in detection better than those on instance segmentation?", "limitations": "NA", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-organized and effectively presents its content, making for a clear and coherent read.\n2. The idea of Parameter-Inverted, e.g., larger models for small images and smaller models for large images, sounds interesting.\n3. The method is novel and simple without complicated handcraft structures. The direct use of existing ViT models is especially interesting and can be easily extended to other tasks.\n4. The experiments are abundant and convincing. The authors provide sufficient experimental results with tables and scatter plots to verify that the proposed framework is solid and effective. The performance on 6B models is impressive.", "weaknesses": "1. The authors did not specify whether the FLOPs and Param in Table 1,3 include a branch merging module.\n2. The discussion with other multi-resolution networks is not sufficient, e.g. HRNet [16,45,57]\n3. The proposed framework can be verified on stronger pre-training, e.g. ViTDet-L (BEiTv2) vs. PIIP-SBL (BEiTv2), to further enhance the effectiveness of the method.\n4. In Lines 180-181, the authors used layer-wise learning rate decay, but the authors do not indicate how to deal with ViT combinations with different numbers of layers, such as PIIP-SBL, ViT-S/B has 12 layers, and ViT-L has 24 layers.\n5. The text in Figure 1 could be slightly larger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for detailed comments and suggestions and provide our response below.\n### **W1: Whether the #FLOPs and #Param in Tables 1,3 include a branch merging module.**\nTo clarify, the #FLOPs and #Param in Tables 1 and 3 contain the branch merging module, which constitutes only a small proportion (\\~1%) of the entire model, similar to the #FLOPs and #Param in Table 13(a).\n\n### **W2: Discussion with HRNet series is insufficient.**\nThe HRNet series [16,45,57] employs a four-branch architecture for pose estimation, semantic segmentation, and object detection.  They also use different resolutions for each branch and add fusion layers every few blocks. However, in their architecture, the number of branches gradually increases as the layers deepen. As a result, they cannot utilize pre-trained models for different branches and must train the whole model from scratch. In contrast, our model uses a symmetric architecture for each branch, allowing for the use of pre-trained backbones. Besides, they do not adopt the parameter-inverted design that uses more parameters for processing smaller images. Lastly, the feature fusion in the HRNet series relies on convolutions and up/downsampling, which is less effective than our deformable cross-attention design.\n\n### **W3: Verifying the method with stronger pre-training.**\nWe verify our method using stronger backbones (e.g., DINOv2, BEiTv2) while maintaining the same FLOPs, and the results are consistent with those reported in the paper.\n| Backbone | Detector   | Pretrain                   | Resolution   | #FLOPs | Schedule | Box mAP | Mask mAP |\n| -------- | ---------- | -------------------------- | ------------ | ------ | ---- | ------- | -------- |\n| ViTDet-L | Mask R-CNN | BEiTv2 (L)                 | 1024         | 1542G  | 1x   | 49.3    | 44.1     |\n| PIIP-SBL | Mask R-CNN | DeiT III (S) + BEiTv2 (BL) | 1568/896/672 | 1464G  | 1x   | 51.6    | 45.4     |\n| ViTDet-L | Mask R-CNN | DINOv2 (L)                 | 1024         | 1542G  | 1x   | 46.3    | 41.7     |\n| PIIP-SBL | Mask R-CNN | DeiT III (S) + DINOv2 (BL) | 1568/896/672 | 1464G  | 1x   | 50.3    | 44.3     |\n\n\n### **W4: Layerwise decay for different numbers of layers.**\nFor combinations with an inconsistent number of layers, we will use a larger learning rate decay for the backbone with fewer layers. For example, for ViT-S/B (12 layers) and ViT-L (24 layers), the learning rate decay for ViT-S/B is set to be twice that of ViT-L (24/12=2).\n\n### **W5: Text in Figure 1 can be larger.**\nThank you for the suggestion. We will revise it.\n\n### **Q1: Weaker ViT-T Pre-training limits performance.**\nWe agree that initializing the small model with stronger pre-trained weights would yield better results. As we prioritize using backbones from the same family in our experiments, we sometimes face situations where a specific model size is unavailable, forcing us to use a backbone from a different family. For instance, we use ViT-S/B/L from DeiT III and ViT-T from DeiT instead because the stronger pre-trained backbone families (e.g. BEiTv2, DINOv2, DeiT III) do not provide smaller weights (e.g. ViT-T).\n\n### **Q2: Detection is better than instance segmentation.**\nThis appears to be a common phenomenon, as observed in ViT-Adapter-B/L and ViT-CoMer-B/L. We speculate that this is because the instance segmentation task is more complex than the detection task, making performance improvements more challenging, especially on a relatively high benchmark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time and efforts. Please refer to rebuttals under each review of detailed responses.

Meta-review: The paper initially received 2 Accepts and 1 Very Strong Accept ratings. The rebuttal helped clarify remaining concerns and all reviewers maintained their ratings, leading to a unanimous acceptance decision. We trust that the clarifications and additional information provided in the rebuttal will be incorporated in the camera-ready version.
Decision: Accept (spotlight)