Parameter-Inverted Image Pyramid Networks

Authors: xizhou  zhu
xue  yang
zhaokai  wang
hao  li
wenhan  dou
junqi  ge
lewei  lu
yu  qiao
jifeng  dai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 22:13:25

Abstract: Image pyramids are commonly used in modern computer vision tasks to obtain multi-scale features for precise understanding of images. However, image pyramids process multiple resolutions of images using the same large-scale model, which requires significant computational cost. To overcome this issue, we propose a novel network architecture known as the Parameter-Inverted Image Pyramid Networks (PIIP). Our core idea is to use models with different parameter sizes to process different resolution levels of the image pyramid, thereby balancing computational efficiency and performance. Specifically, the input to PIIP is a set of multi-scale images, where higher resolution images are processed by smaller networks. We further propose a feature interaction mechanism to allow features of different resolutions to complement each other and effectively integrate information from different spatial scales. Extensive experiments demonstrate that the PIIP achieves superior performance in tasks such as object detection, segmentation, and image classification, compared to traditional image pyramid methods and single-branch networks, while reducing computational cost. Notably, when applying our method on a large-scale vision foundation model InternViT-6B, we improve its performance by 1\%-2\% on detection and segmentation with only 40\%-60\% of the original computation. These results validate the effectiveness of the PIIP approach and provide a new technical direction for future vision computing tasks.

Main paper: 
 introduction: 
 In modern computer vision, high-performance image perception systems increasingly rely on largescale pre-trained models. These models typically consume tens of thousands to millions of GPU hours during pre-training [43, 44, 41] . To adapt these expensively pre-trained models for finegrained image perception tasks (e.g., detection [4, 63, 57, 56] and segmentation [19, 50] ), researchers usually combine them with image pyramids [40, 37] or feature pyramids [27, 42, 34] . This combination is crucial for constructing multi-scale features essential for image understanding.However, integrating these pre-trained models with image pyramids results in significant computational overhead. Image pyramids process the same image at multiple resolutions with the same large-scale model, causing the computational demands to increase quadratically with the image resolutions across all scales. Although feature pyramids [27, 16, 42] aim to reduce this overhead, in MS COCO challenges [28] , most top-performing models [48, 14, 64, 7] still rely on image pyramids due to their superior performance. Therefore, it is necessary to reduce the computing resources for building image pyramids while maintaining high performance. To address this, our key idea is that it is unnecessary to employ vision models of equivalent size for feature extraction at all resolutions (Fig. 1(b-c )) or adopt a parameter-direct design (Fig. 1(d) ). Features at different resolutions can complement each other through adequate feature fusion, thereby enhancing computational efficiency and avoiding redundant modeling of similar information. Specifically, for lower-resolution pyramid levels, the smaller images allow the efficient use of larger models to extract rich contextual and semantic features. The high-resolution branches need only provide the detail information missing from the lower-resolution features, instead of re-modeling existing semantic information. Thus, high-resolution features can focus on smaller receptive fields with less semantic information, making it possible to use smaller models to save computational resources.Building on this strategy, a low-cost and high-performance image pyramid network can be constructed using a series of models with increasing parameter size, paired inversely with images of decreasing resolution, as shown in Fig. 1(e ). Each resolution level should be able to directly leverage existing pre-trained vision foundation models for feature extraction, avoiding the large computational costs for training multi-scale image pyramid networks from scratch. In addition, sufficient feature interactions between different levels are also required to ensure the complementarity of features at different scales and avoid redundant feature extractions.To this end, we propose Parameter-Inverted Image Pyramid Networks (PIIP) based on the complementarity of image features at different resolutions. Specifically, the network takes images at multiple scales as inputs, where higher resolution features are extracted through networks with fewer parameters for local detail perception, and lower resolution features are extracted with more parameters for global information extraction. Additionally, we introduce a feature interaction module that allows features between different resolutions to complement each other. This structure reduces the number of parameters of high-resolution branches and effectively integrates information from different receptive fields, significantly reducing computational costs without sacrificing performance.We conduct experiments on object detection, instance segmentation, semantic segmentation and image classification. Our method achieves better performance while reducing computational costs, compared to traditional image pyramids and single-branch networks. These results validate the effectiveness of our multi-resolution feature interaction strategy and parameter-inverted paradigm and provide a new direction for future visual computing. Our contributions are as follows:1) We propose a novel architecture named Parameter-Inverted Image Pyramid (PIIP) that enhances the multi-scale representational capability of vision backbones with high computation efficiency. The proposed architecture is capable of effectively and flexibly utilizing strong pre-trained vision foundation models without the need for extensive training from scratch.2) We evaluate our method on classic vision tasks of object detection, instance segmentation, semantic segmentation, and image classification. Through combination of existing pre-trained models, our method surpasses single-branch models and other image pyramid methods with higher performance and lower computation cost.3) To validate the generalizability of PIIP on large-scale vision foundation models, we apply PIIP to InternViT-6B [8] , improving its performance on object detection and semantic segmentation by 1.9% (55.7 AP b ) and 1.3% (59.7 mIoU) while reducing 43% and 58% of computational costs, respectively. We also provide extensive analysis and valuable insights on ablation and design guidelines for PIIP that may benefit future research.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The contributions are stated in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Image Pyramids and Feature Pyramids. Image pyramids and feature pyramids are two widely used techniques to enhance the multi-scale perceptive ability for downstream dense prediction tasks.Image pyramids [60, 39, 40, 37] resize the original image and extract features of different resolutions separately, allowing models to accurately detect objects of various scales. However, this technique significantly increases computational costs. Feature pyramids [27, 16, 42, 61, 34] represent another method for constructing multi-scale feature representations by merging low-resolution, semantically strong features with high-resolution, semantically weak features. Although significantly reducing computational costs, they cannot fully replace image pyramids when detecting very small or large objects [39] . Our proposed architecture integrates both image and feature pyramids and introduces the parameter-inverted paradigm to achieve efficient computation.Multi-branch Architectures. Multi-branch architectures have been widely adopted to combine features from different resolutions in various computer vision tasks, including image classification [5] , object detection [46, 25, 7, 52] , semantic segmentation [58, 17] and multimodal dialogues [33, 21] .CrossViT [5] adopts a two-branch structure with different patch sizes to obtain inputs of various scales and different model sizes to balance the computational load. HRNet series [46, 58, 17] adopt a four-branch architecture, where the number of branches gradually increases as the layers deepen. However, they do not adopt the parameter inversion paradigm and cannot utilize existing pre-trained models. In contrast, we propose a general model architecture that supports the use of pre-trained models with different parameters to build efficient image pyramids.Redundancy Reduction for Visual Models. Extensive studies focus on reducing computational redundancy for acceleration. Some work exploits the sparsity of images to accelerate model inference by reducing the number of visual tokens. Dynamic ViT [38] and AdaViT [35] design lightweight prediction modules to predict and prune less informative tokens. EViT [26] and Evo-ViT [55] compute attention scores for each token from class token to identify less informative tokens and adopt accelerated processing strategies for them. Other approaches focus on improving the model structure for efficient computation, such as attention mechanisms [47, 17, 3] or gradually reducing the spatial resolution as the number of layers increases [30, 49, 20] . Orthogonal to the above studies, we propose to use a parameter-inverted design to avoid using large models to process high-resolution images, greatly reducing the computation redundancy.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations are discussed in the conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
parameter-inverted image pyramid networks: 
 To construct efficient image pyramid networks, we employ a multi-branch structure to handle images of different resolutions with different sizes of models. As shown in Fig. 2 , our architecture consists of three parts: multi-resolution branches, cross-branch interactions, and branch merging. Each branch uses an off-the-shelf pre-trained model to process images of different resolutions, where larger resolutions are processed by branches with fewer parameters. Cross-branch interactions are added every few blocks to fuse features across different feature scales. Branch merging combines the outputs from all branches to form a final output. We use the existing pre-trained ViTs [43, 44, 41] to initialize the branches, and initialize the interactions and branch merging from scratch.The multi-resolution branches serve to extract representations from different image scales and semantic levels. The input image is first resized to different resolutions through bilinear interpolation, and then fed into corresponding branches to extract features at different scales. All the branches have the same number of blocks N , where each block contains one or multiple ViT [13] denoted as F i j ∈ R Hj Wj /P 2 j ×Dj , where H j , W j , P j , D j are the image height, image width, patch size, and feature dimension of Branch j, respectively.Branches of different resolutions focus on different spatial scales and semantic levels. To enhance the features of different scales, we propose the cross-branch interactions. Each cross-branch interaction consists of several interaction units, where each unit builds connections between outputs from two feature-scale adjacent branches. The structure of the interaction unit is shown in Fig. 3 . Specifically, for the outputs of the i-th block of Branch 1 and 2, denoted asF i 1 ∈ R H1W1/P 2 1 ×D1 and F i 2 ∈ R H2W2/P 2 2 ×D2, we perform two deformable cross-attention [63] between the two features, denoted as Attention(•). Each cross attention is preceded by a linear layer FC(•) to project the feature dimension of key and value into that of the query, i.e. from D 1 to D 2 or vice versa. A feedforward network FFN(•) is added after each cross attention to provide channel-wise feature fusion. The hidden dimension ratio of FFN is set to 0.25 to save computational overhead.For the first cross-attention in the interaction unit, the interaction process can be formulated as:EQUATIONEQUATIONwhere norm(•) is LayerNorm [1] , τ i 1 and γ i 1 are learnable parameters, and Fi 1 is the interaction output. τ i 1 and γ i 1 are initialized with 0 to ensure that the feature extraction of the original blocks (i.e. distribution of F i 1 ) will not be modified drastically due to the interactions, better utilizing the pre-trained weights.Similarly, the second cross-attention is performed by switching the query and key/value to obtain Fi 2 . The outputs Fi 1 and Fi 2 are used for subsequent feature extractions. We only construct interaction units between each pair of feature-scale adjacent branches, such as Branch 1 & Branch 2 and Branch 2 & Branch 3.The final feature maps of all branches FN j have different spatial shapes and feature dimensions, where spatially larger feature maps have fewer feature dimensions. A single feature map fails to provide multi-scale semantic features, so we employ the branch merging module to merge the outputs of all branches into a single feature map.As shown in Fig. 2 , all branch outputs are first projected to the feature dimension of Branch 1 (the largest feature dimension) with Proj(•). Then, all branch outputs are upsampled by bilinear interpolation Upsample(•) into the feature map size of the last branch (the largest feature map size). Finally, these outputs, with the same spatial shape and feature dimension, are added together with learnable scalar weights w j to form the final output. This process can be formulated as:Branch 2 Block ! Branch 1 Block ! Deform Attn FC FFN FC FFN Deform Attn Branch 2 Block ! + 1 Branch 1 Block ! + 1 ℱ ! $ ℱ # $ query key / value " ℱ # $ " ℱ ! $ # ℱ # $ # ℱ ! $ query key / valueEQUATIONEQUATION)where M is the number of branches. F out is the final feature map, which has the largest feature resolution and also the largest feature dimension across all branches.For object detection and semantic segmentation, Proj(•) is a two-convolution layer with Group-Norm [51] , and the final output F out is used for feature pyramid network [27] similar to ViTDet [23] .For image classification, we do not use the branch merging module, but instead append the original classification heads of the pre-trained models after each branch. The final classification score is the average of the output logits of all branches. We observe that using the pre-trained heads can speed up convergence compared to using a randomly initialized head after a branch merging module.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
implementation details: 
 For comparison with Base-size models, we use pre-trained ViT-T/S/B as the branches to construct three-branch PIIP network, namely PIIP-TSB. Similarly, ViT-S/B/L are used to construct PIIP-SBL to match the computation of Large-size models. We also construct four-branch PIIP-TSBL with ViT-T/S/B/L. We set the number of interactions (each with 2 interaction units as shown in Fig. 2 ) N to 12, i.e. after every layer for ViT-T/S/B or after every two layers for ViT-L. We construct multiple variants of three-branch and four-branch models with different resolution configurations.For combinations with an inconsistent number of layers, we will use a larger learning rate decay for the backbone with fewer layers. For example, for ViT-S/B (12 layers) and ViT-L (24 layers), the learning rate decay for ViT-S/B is set to be twice that of ViT-L (24/12=2).For object detection and segmentation, we use ViT-S/B/L pre-trained on ImageNet [11] from DeiT III [44] , ViT-T from DeiT [43] . ViT-H from MAE [18] and InternViT-6B [8] are used for 6Bscale experiments. For all PIIP-SBL models, we use the ImageNet-21K 384-resolution pre-trained weights to compare with previous approaches. We adopt AdamW [32] optimizer with layer-wise learning rate decay [2] to train the model on 8 NVIDIA A800 GPUs. For image classification, in Base-size experiments we use pre-trained ViT-T/S/B weights from DeiT [43] . In Large-size experiments, since DeiT does not provide ViT-L models, we use ImageNet-21K pre-trained ViT-S/B/L weights from [41] .We use the FLOPs calculation script from MMDetection [6] , with our modifications to accurately calculate FLOPs of modules like self-attention and deformable attention. The script is released along with the training code. We have also manually verified the calculations using formulas, and the results are consistent with those produced by the script. Settings. The MS COCO [28] dataset is used to evaluate the performance on object detection and instance segmentation. We use three detectors, including Mask R-CNN [19] , Cascade R-CNN [4] and DINO [59] , based on MMDetection [6] . Following common practices [7] , we adopt 1× (12 epochs) or 3× (36 epochs) training schedules and use window attention [23] to save time and memory. The total batch size is 16, and the initial learning rate and weight decay are 1e-4 and 0.05.Effectiveness of Parameter-Inverted Image Pyramid. To demonstrate the performance and computational advantages of the Parameter-Inverted Image Pyramid (PIIP) Networks, we perform validation on two baseline models ViTDet-B and ViTDet-L [23] in Tab. 1. Taking the three-branch structure as an example, while maintaining similar performance with ViTDet-B, our PIIP-TSB reduces the computational cost by 47.5% (243G vs. 463G) and 38.0% (287G vs. 463G) in object detection and instance segmentation tasks respectively. Similarly, compared with ViTDet-L, our PIIP-SBL reduces the computational cost by about 52.9% (727G vs. 1,542G) and 35.0% (1,002G vs. 1,542G) in the above two tasks respectively. On the other hand, with similar computational cost as the baseline, PIIP-TSB and PIIP-SBL improve the object detection performance by 2.8% and 2.6%, respectively, and instance segmentation by 1.5% and 1.2%, compared to ViTDet-B and ViTDet-L. To better illustrate the above conclusion, we depict the trend between the computational cost and performance of different PIIP model combinations by adjusting the input resolution, as shown in Fig. 4 . Furthermore, when we use the four-branch structures, the curve in the figure is slightly better than that of the three-branch structure. Results with InternViT-6B. We further examine PIIP on an extremely large vision foundation model InternViT-6B [8] . As can be seen from Tab. 3, PIIP-LH6B finally achieves 55.7% AP b when using Mask R-CNN 1× training schedule. In addition, our PIIP can save nearly 43% of the computation and achieve better performance than the single-branch InternViT-6B by 1.9% on AP b and 0.9% on AP m .Settings. We use UperNet [54] as the basic framework to train on the ADE20K [62] dataset based on MMSegmentation [9] . We follow the settings of [30] to train the model for 160k iterations. The batch size, initial learning rate and weight decay are 16, 4e-5 and 0.05.Results with Base-size and Large-size models. In Tab. 5, PIIP can achieve better performance with fewer computations compared with single-branch baselines. In Tab. 4, we compare PIIP with state-of-the-art segmentation backbones. PIIP-TSB attains 51.6% mIoU with UperNet, exceeding InternImage-B [48] by 1.4%. Similarly, PIIP-SBL yields 54.3% mIoU, which is outstanding compared to counterparts like ConvNeXt-XL [31] and InternImage-L [48] .Results with InternViT-6B. As shown in Tab. 3, similar to the conclusions obtained in the object detection experiment, our method achieves better performance than the InternViT-6B baseline with less computation. PIIP-LH6B finally achieves 59.65% mIoU without using additional optimization techniques. Settings. We load the pre-trained models for each branch and train the model for 20 epochs on ImageNet-1K [11] . The batch size, initial learning rate and weight decay are 1024, 3e-5 and 0.1. The learning rate for the random initialized interactions is 10 times the base learning rate, i.e. 3e-4.The other settings mainly follow the fine-tuning recipe of [44] and are provided in the appendix.Results. As shown in Tab. 6, when compared with the DeiT baseline, our PIIP-SBL reduces the computational cost by 36.7% (39.0G vs. 61.6G) while maintaining the performance. When using a similar computational cost as the baseline models, PIIP-TSB and PIIP-SBL improve the top-1 accuracy by 0.3% and 0.7%, respectively.Superiority of parameter-inverted image networks. We evaluate the effectiveness of the image pyramid and parameter-inverted design by comparing our method with other methods, e.g. designs in Fig. 1 . First of all, a single-branch with multi-scale training is the simplest image pyramid practice, as shown in Tab. 7. Compared with the baseline model, its performance improvement is limited (44.8% vs. 43.8%). Secondly, we conduct experiments by controlling the scale of the branch model and the input resolution while ensuring that the total computational cost is close. Specifically, when using the same input image resolution, the combination of models of different sizes does not bring significant improvements to detection performance. Correspondingly, when the three branches use the same model (e.g. BBB), the input image resolution is adjusted to the pyramid structure. The performance of the final model is slightly improved on AP b (44.5% vs. 43.8%), but the AP m drops significantly (38.7% vs 39.9%) due to the reduction of the maximum resolution. The former demonstrates the importance of the image pyramid, and the latter further demonstrates the need for the image pyramid to maintain a larger image scale range, which is especially essential for instance segmentation. Drawing on experience, parameter-inverted image networks are an efficient design method that can meet the above requirements, especially when compared to its opposite configuration parameter-direct image pyramid, i.e. TSB with 448/672/896 resolution (46.6% vs. 42.6%). As shown in Tab. 7, with less computation than the baseline, the model can support image inputs in the maximum range from 672 to 1,568, and the performance is significantly improved.Design guidelines for parameter-inverted image networks. Through extensive practice, there are two empirical design guidelines when scaling up the model: 1) Prioritize increasing the image resolution of the largest image branch: as shown in the blue dashed circle in Fig. 5 (a), the input resolution of the largest image branch is greatly increased without causing a sharp increase in the total computational cost.2) The largest model does not need to exceed the compared baseline model: the introduction of larger models will limit the resolution range of the image pyramid, e.g. TSB is more cost-effective than TBL according to Fig. 5(a) .Branch merging. Experiments in Tab. 8 prove that branch merging of all branches yields the best performance by providing multi-scale semantically rich features, compared to only using feature maps from single or partial branches.Attention type. The core of information interaction between branches is cross-attention. We adopt PIIP-TSB with resolution 1120/896/448 as the basic model and investigate two different attention mechanisms. As shown in Tab. 9, deformable attention [53] with linear complexity can significantly improve the performance of the model without substantially increasing the computational cost. We end up using deformable attention as the default configuration. Notably, it can be replaced by other more advanced attention mechanisms in the future to further boost performance. As shown in Tab. 9, no matter which attention mechanism is used, the increase in the number of interactions will improve the performance of the model to varying degrees. Since it also increases the computational cost, we further explore the cost-effectiveness of different numbers of interactions. We conduct experiments with different resolution combinations on models with different numbers of interactions, and the scatter plot of all results is shown in Fig. 5 (b). It can be seen that when the number of interactions is small (less than 2), the growth trend of model performance with the increase in computational cost is relatively slow. We attribute this to too few interactions and insufficient information complementation between branches. Therefore, we use 12 interactions by default. Note that as the model size increases (e.g. more layers), the number of interactions can also increase accordingly.Interaction direction between branches. We compare five different interaction directions in Tab. 10. Considering both the computational cost and performance, we finally choose the fourth method, i.e. bidirectional connections of adjacent branches, as the default choice. As can be seen from Fig. 6 , all the interaction directions achieve a satisfactory performance-computation balance, validating their ability to improve communication between branches.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The details are provided in the experiment section and the appendix. Our code has been released. 
conclusion: 
 This paper introduces the Parameter-Inverted Image Pyramid Networks (PIIP) to address the computational challenges of traditional image pyramids. With the parameter-inverted design and feature interaction mechanism, PIIP effectively balances computational efficiency and performance. Extensive experiments on detection, segmentation and classification tasks demonstrate that PIIP outperforms traditional methods and single-branch networks while reducing computational costs, providing an efficient and effective framework of multi-scale feature integration for future research.Limitations. While our method manages to save computation, its memory consumption is higher than single-branch models due to the increase of parameter count. Our current method only focuses on ViT-based models. PIIP with hierarchical networks (e.g. CNN) or heterogeneous structures (e.g. CNN for some branches and ViT for other branches) remain unexplored for future work. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The data is publicly available datasets. Our code has been released. Details to reproduce the experiments is described in the experiments section and the appendix. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Datasets are referenced. License are provided in the appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: ImageNet-1k [11] is subject to the ImageNet terms of use [45] .COCO [28] is subject to the Flickr terms of use [15] .ADE20K [62] is subject to the ADE20K terms of use [36] .NeurIPS Paper Checklist• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.* Equal contribution. † Corresponding author: Jifeng Dai <daijifeng@tsinghua.edu.cn>. 


Decision: The paper initially received 2 Accepts and 1 Very Strong Accept ratings. The rebuttal helped clarify remaining concerns and all reviewers maintained their ratings, leading to a unanimous acceptance decision. We trust that the clarifications and additional information provided in the rebuttal will be incorporated in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The authors propose a novel vision architecture, Parameter-Inverted Image Pyramid Networks (PIIP), which can be applied to different tasks, including classification, object detection, and instance or semantic segmentation. The authors aim to take advantage of the multi-scale information of image pyramids, without the usual excessive computational demands of processing an image in multiple different resolutions. The core idea is to use a family of networks of different computational requirements, e.g., ViT-S/B/L, and apply the lightest model (least parameters) to the image of the highest resolution in the pyramid, apply the second lighter model to the next bigger image, and so on, ending up to apply the heaviest model (most parameters) to the image of the smallest resolution. The intuition is that stronger models with more parameters can be used to extract semantically rich contextual features from images of coarser resolution, while lighter models can be efficiently applied to extract low-level features from high-resolution images. This way, the computational requirements of the different models are balanced, meaningful features from all scales are extracted, and images can be processed in higher resolution because the lighter and less expensive models are processing them.

The models are applied to the different images in the pyramid in parallel, and as features are extracted, the models communicate with each other through a proposed Interaction Unit. This way, features of different semantic meaning extracted from different models complement and inform each other. After the features from each individual model are extracted, they are merged to be used for the task at hand.

The authors perform extensive evaluation of PIIP on 4 different tasks, object detection, instance segmentation, semantic segmentation, and image classification. The authors initialize the PIIP backbones with pre-trained ViT models, and for each task they fine-tune appropriate heads. PIIP is compared to baselines of different scale, and in all experiments it demonstrates superior or comparable performance. 

In addition, the authors perform multiple ablations, offering insights about the behavior of PIIP, and the importance of different design choices.
strength_weakness: strength_weakness: Quality:

1. My main concern about this work is that the authors don’t provide actual timings and memory measurements. Computational requirements are quantified through FLOPs, however, theoretical gains in FLOPs don’t always translate to benefits in practice when the compared architectures have considerably different designs. For example, the PIIP models may be applied in parallel, but the interactions between the features may force the models to wait for each other, adding a sequential element that may add to the latency. I want to clarify that I am not claiming that this is the case, but that actual timings beyond theoretical FLOPs are needed to offer conclusive evidence, especially when efficiency is one of the main claims of this work.

2. About memory requirements, it seems to me that using a whole family of models may be prohibitive in many settings, so, memory should be explicitly measured. In addition, the authors control in their experiments for FLOPs, e.g., in Table 1 they show how PIIP models perform compared to baselines with similar FLOPs. I think it would be useful to control for memory too, e.g., in Table 1 the best PIIP models have more than 50% higher number of parameters compared to the baselines. Based on this, I think a question that naturally arises is what if the baseline was a larger model with equal parameters? If we have an extra memory budget, is PIIP the best way to use it? I would like to clarify that I don’t think the authors should necessarily have additional experiments controlling for memory, but if it is a disadvantage of PIIP, as it seems to be due to the parameter count, I think they should discuss it in the limitations.

Clarity:

1. In Table 2, the results for ViTDet-B and ViTDet-L are different compared to Table 1, why is that? Similarly, in Table 2, the results of PIIP-TSB and PIIP-SBL are different compared to Table 1. For the PIIP-SBL model, I see that the scores in Table 2 match the scores of the best model reported in Table 12, where models use higher resolutions, is this the reason for the discrepancy? However, the reported performance of the PIIP-TSB in Table 2 is higher than any performance reported for PIIP-TSB in Table 12, so, how the performance reported in Table 2 is achieved? I think the authors should clarify the configurations of the models they use in their experiments.

2. Some of the Table captions are not informative enough. For example, it is not clear why crop size is underlined in Tables 3, 6, 7. On a similar note, in the ablation about branch merging (ln 267 - 269, and Table 5), the authors don’t mention the dataset they use. I guess it is MS COCO, but I think it should be explicitly mentioned.

3. In the caption of Table 8, it is mentioned “‘PI’ and ‘IP’, ‘Inter.’ represent parameter-inverted, image pyramid and interactions”, I think it should be “‘PI’, ‘IP’, and ‘Inter.’”.

4. In Ln 262 the authors mention “green dashed circle in Fig. 5(a)”, but I think it should be “blue dashed circle”.

Significance:

1. PIIP requires a family of backbones for its processing, and the authors use pre-trained models in their experiments, while they also include some experiments on training from scratch in the Appendix. I think the need to train multiple backbones, or the need to find multiple pre-train models, may be an undesirable overhead for the adoption of the method.
questions: questions: 1. In Section 4.1 the authors provide the pre-trained architectures they used for different tasks, how did they decide which models to use?

2. What is the formula the authors use to calculate the FLOPs of the models?

3. Why there are no comparisons with state-of-the-art models on image classification?

4. In Table 8, what is the range of resolutions that the model in the second row is trained on?
limitations: limitations: The authors offer a short discussion of the limitations in Section 5, which I think should be expanded to address the memory requirements of PIIP.
paper_summary: paper_summary: This paper proposes two techniques: 1) Models with different parameter sizes to process different resolution levels of the image pyramid. 2) A feature interaction mechanism to integrate information from different spatial scales. Extensive experiment results are used to support the claims.
strength_weakness: strength_weakness: It is unclear to me if the better accuracy in table 1, 2, 3 and 4 are entirely due to the proposed techniques or it could also be partially explained by the fact that the highest resolution in the pyramid is larger. Take table 1 as an example, the input resolution for baseline is 1024. But the highest resolution for PIIP is 1792. What if you use 1792 for baseline? It is perfectly fine to first hold FLOPS constant and check accuracy, as shown in table-1. But you should also hold input constant and check accuracy.
questions: questions: See above
limitations: limitations: yes
paper_summary: paper_summary: This paper presents a novel method named parameter-inverted image pyramid to handle the issues of high computation overhead of image pyramids. It uses different model sizes to process different resolutions. The method achieves significant results on tasks of object detection, segmentation, and classification by reducing overhead and improving performance.
strength_weakness: strength_weakness: 1. The authors did not specify whether the FLOPs and Param in Table 1,3 include a branch merging module.
2. The discussion with other multi-resolution networks is not sufficient, e.g. HRNet [16,45,57]
3. The proposed framework can be verified on stronger pre-training, e.g. ViTDet-L (BEiTv2) vs. PIIP-SBL (BEiTv2), to further enhance the effectiveness of the method.
4. In Lines 180-181, the authors used layer-wise learning rate decay, but the authors do not indicate how to deal with ViT combinations with different numbers of layers, such as PIIP-SBL, ViT-S/B has 12 layers, and ViT-L has 24 layers.
5. The text in Figure 1 could be slightly larger.
questions: questions: 1. I am just a little curious, for PIIP-TSB， why ViT-T from DeiT but ViT-S/B from DeiT-III? Won't different pre-training methods have some impact?  According to Table 5, ViT-T with large resolution input plays the most important role, so the weaker pre-training ViT-T should limit the performance of the framework?
2. (Unimportant) In Figure 4, why are the improvements in detection better than those on instance segmentation?
limitations: limitations: NA

Comments: title: Thank you for your reply
comment: I agree with reviewer H3y6 that the parameter-inverted design is initially counter-intuitive but impressive in the end, given the experimental results. Since my concerns and questions have been addressed by the authors, and considering the high quality of this research has been recognized by all the reviewers, I keep my positive rating.
Thank you for your valuable discussion and positive decision. We will make corresponding revisions in the final manuscript.
Thank you for your valuable discussion and positive decision. We will make corresponding revisions in the final manuscript.
Thanks for the new results. I will keep my positive rating.
I would like to thank the authors for their detailed response. As I mentioned in my review, my main concerns were about actual timings and memory. About timings, the numbers the authors provide are encouraging, and I agree that total agreement between FLOPs and actual runtime is not necessary from the very beginning, when a method is not yet fully optimized. I would suggest the authors include this information in Future Work or Limitations.
About memory, I agree that memory is not easy to control with pre-trained models, and some of the existing results allude to favorable comparisons of PIIP with baselines of similar memory. So, I find the authors' response sensible and encouraging, but in the absence of more detailed resource comparisons in the current manuscript, I will maintain my initial score.