Could Giant Pre-trained Image Models Extract Universal Representations?
Authors: Yutong Lin
Ze Liu
Zheng Zhang
Han Hu
Nanning Zheng
Stephen Lin
Yue Cao
Venue: NeurIPS 2022 Accept (2022)

Abstract: Frozen pretrained models have become a viable alternative to the pretraining-then-finetuning paradigm for transfer learning. However, with frozen models there are relatively few parameters available for adapting to downstream tasks, which is problematic in computer vision where tasks vary significantly in input/output format and the type of information that is of value. In this paper, we present a study of frozen pretrained models when applied to diverse and representative computer vision tasks, including object detection, semantic segmentation and video action recognition. From this empirical analysis, our work answers the questions of what pretraining task fits best with this frozen setting, how to make the frozen setting more flexible to various downstream tasks, and the effect of larger model sizes. We additionally examine the upper bound of performance using a giant frozen pretrained model with 3 billion parameters (SwinV2-G) and find that it reaches competitive performance on a varied set of major benchmarks with only one shared frozen base network: 60.0 box mAP and 52.2 mask mAP on COCO object detection test-dev, 57.6 val mIoU on ADE20K semantic segmentation, and 81.7 top-1 accuracy on Kinetics-400 action recognition. With this work, we hope to bring greater attention to this promising path of freezing pretrained image models.

Could Giant Pretrained Image Models Extract Universal Representations?
Yutong Lin Ze Liu 23, Zheng Zhang Han Hu Nanning Zheng', Stephen Lin Yue Cao Xii an Jiaotong University 2 University of Science and Technology of China
Microsoft Research Asia
{t (t-yutonglin, t-liuze, zhez, hanhu, stevelin) @mi comoootto..ccom com unzheng@mail x
tu. jut.. edu. en, caoyue 00gmaal.....mmm.mm com
Abstract
Frozen pretrained models have become a viable alternative to the pretraining- then- finetuning paradigm for transfer learning. However, with frozen models there are relatively few parameters available for adapting to downstream tasks, which is problematic in computer vision where tasks vary significantly in input/output format and the type of information that of value. In this paper, we present study of frozen pretrained models when applied diverse and representative computer vision tasks, including object detection, semantic segmentation and video action recognition. From this empirical analysis, our work answers the questions of what pretraining task fits best with this frozen setting, how to make the frozen setting more flexible to various downstream tasks, and the effect of larger model sizes. We additionally examine the upper bound of performance using giant frozen pretrained model with billion parameters (SwinV2-G) and find that it reaches competitive performance on a varied set of major benchmarks with only one shared frozen base network: 60.0 box mAp and 52.2 mask mAP on COCÓ object detection test-dev, 57.6 val mloU on ADE20K semantic segmentation, and 81.7 top-1 accuracy on Kinetics-400 action recognition. With this work, we hope to bring greater attention to this promising path of freezing pretrained image models.
Introduction
Transfer learning via the pretraining-thes filen-finetunin paradigm is the cornerstone in the success of
deep neural networks. In computer vision, finetuning backbone networks (29] 21 13 38) pretrained supervised classification leads to top performance for wide range of visual recognition tasks, even on tasks where the input or output format differs from that of pretraining, such as video action recognition [25] object detection 34, and semantic segmentation (58). In natural language processing (NLP), finetuned models that are pretrained on masked language modeling (MLM) also excel at large variety of NLP tasks, e.g., eleven tasks in the case of BERT (11.
Though highly effective, the changes to many network parameters make finetuning parameter- inefficient. An entirely different model is created for each downstream task, problem that is magnified with the rapid increase in model sizes. In NLP, a solution to this problem that has gained momentum is to freeze pretrained language models when transferring them to downstream tasks. A small number of task-specific parameters, e.g., task-specific heads [22] or prompts B 30 32 36), introduced, and only these newly added parameters trained the downstream task while the
Equual contribution. The work is done when Yutong Lin and Ze Liu are interns at Microsoft Research Asia. Correspondence to: Yue Cao (caoyue 10@ (@gmmll.mmmm))...
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

T-Ie-Frozen
B-1kFrroeen
8-22kFrozen #
L-22K-Frozen
*
G-Frozen
T-1K-Ft A B-1KFFt
8-22KFFt 8 L-22K-FL
58
80
COCO mApaox
56
ADE mlou
K400 acce
54
8
78
54
76
52
92
74
50
72
50
70
40
80
120
160
200
40
80
120
160
200
100 150 200 250 300
Number of Trainable Parameters (M)
Number Trainable Parameters
Number of Trainable Parameters (M)
Figure 1: Performance with respect to the number of trainable parameters with different base networks (SwinV2-T/B with supervised training on ImageNet-11 KK Swin V2-B/L with supervised training on ImageNet-22k, and SwinV2-G) under frozen and full finetuning settings on COCO object detection (left), ADE20K semantic segmentation (middle), and Kinetics-400 video action recognition (right). Single-scale box mAP for COCO, multi-scale mloU for ADE20K and top-1 accuracy for Kinetics-400 reported.
rest of the model is fixed. In this way. the computation and memory costs are reduced considerably, with high degree of parameter sharing among models finetuned for different tasks.
This parameter-efficient transfer, however, has yet to gain traction in computer vision. The difficulties in adopting this approach lie in the differences between the two modalities. In NLP, the input and output formats of different tasks are similar, where nearly all of them can be defined as a sequence of tokens. Moreover, the dominant pretraining task of self-supervised masked language modeling learns information that is broadly valuable in NI.P applications. By contrast, such uniformity does not exist in computer vision. The input and output formats in vision vary greatly, such as image/video and high/low resolution for inputs, and image-level categories, target coordinates and so on for outputs. The information that is valuable varies as well among downstream tasks, making the choice of pretraining task unclear. Due to these challenges, previous investigations have focused only on
downstream tasks similar to the pretraining task, image classification 2 42 43 57 56 [59.
In this paper, we study parameter-efficiae transfer learning that generalizes to a diverse set of computer vision tasks. Under this setting, our work seeks answer the following questions:
1. Which pretraining task fits best with this frozen setting?
2. What is the key to making the frozen setting work well when the downstream tasks are significantly different from the pretraining task?
3. How well can a frozen setting perform with a giant pretrained model, such as S2inV2-G with 3B parameters?
Our investigation begins with the first question, where consider the four most widely- pretraining tasks: supervised classification, contrastive learning, masked image modeling and image- text alignment. With S2inV2-B (38] 37 as the backbone, the pretraining tasks are evaluated on popular vision frameworks covering a range of downstream tasks, including Mask R-CNN [20 with FPN [33 for COCO object detection, Mask [9] for ADE20K semantic segmentation, and Video Swin Transformer [39] for Kinetics-400 (K400) video action recognition. We find that supervised pretraining and image-text alignment work best under the frozen setting and the model
with ImageNet-22K pretraining performs better than the ImageNet- 1 counterpart. However, the frozen settings still underperform full finetuning by substantial margins.
Towards improving performance over disparate tasks, we next consider how to extend the frozen setting for better task adaptation. We examine the effects of adding tunable parameters to task- specific heads and task- architectural elements such as FPN for object detection and decoders for semantic segmentation. From performance analysis and an examination of feature similarity across different layers by Centered Kernel Alignment (CKA) (7). we make a number of observations for different tasks on what can help bridge the performance gap between frozen settings and full finetuning. Overall, we find that the number of well well-placed tunable parameters is the key to make frozen settings work well.
Finally, we look at the impact model size on the frozen setting. With its richer content, larger pretrained models have been found in NLP to require fewer tunable layers. Despite a large difference

---

between pretraining and finetuning tasks as well as relatively small change in model size, we have observed similar trends for computer vision between SwinV2-L-22K and SwinV2-T-IK in object detection. We additionally explore the upper bound of performance under the frozen setting, using frozen supervised pretrained model with 3 billion parameters trained on 70M images (SwinV2-G), With this giant model, highly competitive performance is achieved on major benchmarks: 60.0 box AP on COCO object detection test-dev, 57.6 mloU on ADE20K semantic segmentation, and 81.7 top-1 accuracy on Kinetics-400 action recognition.
Thanks to the efficiency of this frozen setting. competitive performance are achieved on different model sizes with much less trainable parameters, as indicated in Figure | We hope our work will inspire further research in this promising direction of freezing pretrained image models. The proposed approach can serve as a simple baseline and guide the evaluation of future work.
Related Work
Throughout the deep learning era, supervised learn- ing, especially by image classification on ImageNet [10), has been prevalent in vision pretraining. Leveraging large amount of data, a supervised pretrained model can efficiently transfer to various downstream tasks [12 28 26
Representative Visual Pretraining Tasks
48 o by finetuning. However, supervised pre- training suffers from severe data-hunger, while the cost of acquiring labeled data is expensive. To address this issue, several recent studies in self-supervised pretraining have demonstrated promising results, where self-supervised pretraining achieves finetuning performance on par with the supervised counterparts several representative downstream tasks [19 8]. Among them, contrastive learn- ing [14] 19 8 5 [7] and masked image modeling 7 18
13
46
16
38
40
54 have been particularly successful. Contrastive learning compares two image views, maximizing the similarity of positive pairs while minimizing the similarity of negative pairs. For linear probing, the state-of of-the-art contrastive model achieves results comparable to supervised models. Differently, masked image modeling learns by randomly masking some input tokens and reconstructing the raw signals. With masked image mod- eling. large-scale model can be trained more effectively and yields better finetuning performance. Another branches of work utilize image-text pairs and leverage natural language as supervision for visual-linguistic learning. Among them, image-text alignment pre-training [4] 24 with web-crawled data have shown great potential for visual recognition tasks, especially under the zero-shot setting. iCar (53) further bridges image-text alignment and image classification and successfully keep the characters of both methods.
These pretraining methods are so successful that almost all the top models of various vision tasks finetuned from them. However, in the frozen backbone setting, the performance of these pretraining tasks for various downstream tasks are still unknown. Based on this motivation, we carried out a comparison study under the frozen setting with the four most widely-used pretraining tasks
supervised classification (3 37,, contrastive learning [31], masked image modeling (54) and image- 53 using Swin Transformer as the backbone.
text alignment
Frozen Language Models
With the large scale of today's 's language models, the setting of frozen
language models has become important in NLP and receives much attention (35). Adding external adapters [22) is a direct solution in this direction and was first introduced in this field. Subsequently, various solutions specific to NLP have been proposed, including prompt tuning [30] prefix tuning (32). and low-rank adaptation 23
Frozen Setting in Vision In computer vision, additive models are the most studied direction for the frozen setting, where the pretrained weights are frozen and a small number of new parameters is added for each task. Some of these works [12 47 directly add the new layers upon off-the- shelf features, while others introduce new network with independent access to the input 44 [7)
to address the information loss of the pretrained model, or add task-specifie components into the such as batch normalization, residual adapter and so on. However, all use the additive models only for tasks similar to the image classification pretraining task. concurrent work [52], image classification features are reused only for object detection and instance segmentation. Our paper, for the first time, studies the frozen setting under different pretrained image models on three representative but diverse vision tasks, namely object detection, semantic segmentation and video action recognition. Furthermore, we demonstrate the
pretrained model 42
2
43
previous works 2 43 57
3

---

Action
Object
Semantic Segmentation
Recognition
Detection
Head
Head
Head
Base Network
Base
Network
i. Pretraining of Base Network
ii. Finctuning the Head Networks
Pipeline of the frozen setting, consisting (i) pretraining the base network, and (ii) finetuning of head networks with the frozen base network.
Figure 2:
effectiveness of our approach via a giant frozen supervised pretrained model (Swin V2-G), with highly competitive performance on major benchmarks for the three tasks. Another path is to directly adapt the prompt learning approach in NLP to computer vision [59 56 but this is limited to very specific tasks or scenarios, e.g., requiring text encoder [59].
3 Methodology
Under the frozen setting, we take the approach of freezing the pretrained base network B()) and training a specially designed task-specific head network H(-) for each downstream task.
3.1
Architectural Elements
Base Network We denote deep neural networks designed for image classification, e.g., VGG [49], ResNet [21] or VIT [3], the base network, B(-). The base network is also referred to as the backbone network in some scenarios, and it is intended to provide some core cognitive or perceptual information about the input. In the frozen setting, after the base network B(-) pretrained, do not update it when it is transferred to downstream tasks, that is, the base network has no tunable parameters for training downstream tasks.
For the base network, we adopt general-purye purppss backbone architecture, e-g., Swin Transformer BS for most experiments, that is compatible with broad range of vision tasks, without components specific to any downstream tasks. We note that Swin Transformer is highly scalable, with a giant version composed of 3 billion parameters 37
Head Network We denote the task-specific network that newly added and tuned during transfer to downstream tasks as the head network, H(.). Note that the input and output formats in vision vary greatly, so a crucial component our approach to effectively adapt the head network to the properties of the downstream task. With a suitable head design, we find this simple framework to generally perform well over the sampled downstream tasks under frozen pretrained image model setting.
3.2 Pretraining
We consider the most widely-used pretraining tasks supervised classification 38 37
contrastive learning, masked image modeling and visual-linguistic learning with Swin Transformer as the base network. Specifically, we utilize Swin Transformer variants specially designed for each of these pretraining tasks, e.g., vanilla Swin Transformer [8 37 for supervised classification, VVVIT [1 for contrastive learning, SimMIM learning, which are all sperce For fair comparison, the same base network capacity is main- tained across the pretraining tasks, such as using Swin Transformer with x24 input and a window size of 7.
(54) for masked image modeling and iCar[53) for visual-linguistic

---

3.3 Frozen Setting on Downstream Tasks
major challenge of the frozen setting the varied input and output formats of different vision tasks, such as object detection with high-resolution images input and target coordinates and the corresponding categories as output, semantic segmentation with high-resolution images for input and pixel-level categories output, and video action recognition with low-resolution videos for imput and video-Ievel categories as output. Therefore, adapting a frozen base network to a new task requires careful model design, including a suitable framework and head network, to bridge the gap between pretraining and finetuning.
Object Detection To adapt a frozen pretrained model to the object detection task, we present a few adjustments. First, object detection generally requires high-resolution images input and large window size in the backbone architecture, while the pretraining task takes low-resolution images and small window size. We find that directly changing the size of input images from small to large but keeping the window size unchanged works relatively well. Second, maintaining the aspect ratio of the input image is preferable in object detection and thus by default arbitrary input resolution is allowed for each image, with undesirable padding on the border of feature maps that change the distributions of features. We deal with this issue by adopting multi-scale augmentation similar to that of image classification. It randomly resizes the original image, then randomly crops a square part of the resized image [13] Third, object detection typically utilizes multi-resolution feature maps as the input of the head network, while most pretraining tasks only take the final output of the base network. Using Swin Transformer as the base network enables us to directly take the output features of different stages as the input to the head network, and we find that this choice empirically works well. For evaluation on this task, we adopt COCO 2017 34,, the most widely- used benchmark for object detection and instance segmentation, which contains 18K training, 5K validation and 20K test-dev images.
Semantic Segmentation Semantic segmentation aims to perform pixel-level classification on high resolution images and has many characteristics similar with object detection. For instance, semantic segmentation also requires a larger input resolution and window size, and multi-resolution feature maps for visual recognition at finer granularity. On these two points, we obtain similar observations for semantic segmentation as with object detection: larger input with unchanged window size achieves relatively good performance, and directly using the output features from different stages of Swin Transformer as the input of the head network also works well for semantic segmentation. For this task, we adopt the most widely-used benchmark, ADE20K [58), for evaluation. ADE20K covers a broad range of 150 semantic categories. It has 25K images in total, with 20K for training, 2K for validation, and another 3K for testing.
Video Action Recognition The main challenge of adapting the frozen pretrained image model to video action recognition lies in the different input formats. Video action recognition aims to recognize the action types of each input video, which consists of sequence of video frames (e.g., 16 frames per clip). Therefore, it essential in video action recognition to capture both spatial and temporal relationships. Previous works based on full finetuning mainly focus on exploring the simultaneous modeling of spatial and temporal relationships in the base network. For the frozen setting, however, the pretraining model is capable only of spatial reasoning, so the head network must compensate for the lack of temporal modeling. For the task of human action recognition, we adopt the widely-used Kinetics-400 (25 dataset, consisting of 240k training videos and 20k validation videos over 400 human action categories.
4 Which Pretraining Task Fits Best with the Frozen Setting?
In this section, we evaluate four prevalent pretraining tasks under the frozen setting, namely supervised pretraining, contrastive learning, masked image modeling, and visual- -inguisin learning. Specifically, we use six pretrained Swin Transformer models, which include supervised pretraining on ImageNet- IK (SUP-IK), supervised pretraining on ImageNet-22K (SUP-22K), contrastive learning of Es VIT on ImageNet- K (EsViT- masked image modeling of SimMIM on ImageNet-1K (SimMIM-1K) image-text alignment of iCar on Laion (iCar- Laion), and jointly training of image-text alignment and image classification for iCar on Laion and ImageNet-22K (CCar-Laio-22 22K). For the downstream
5

---

tasks, we adopt SwinV2-B [37 as the base network and three widely-used frameworks as the head networks: Mask R-CNN 20 with FPN [3] for COCO object detection, Mask2Former [9] with a one-block pixel decoder for ADE20K semantic segmentation, and spatial-only Video- Swin-Transformer (39 variant, where a temporal window size of used, with linear head for Kinetics-400 action recognition.
COCO
ADE20K
Kinetics-400
Approach
Frozen
Full ft.
Frozen
Full ft.
Frozen
Full ft.
Aph
Apmusk
Apbox
Apmask
mloU
mloU
acc@)
acc
SUP-IK
42.4
38.7
50.5
44.5
49.8
52.3
60.4
77.0
SUP-22K
45.0
41.1
51.9
45.7
51.9
55.3
70.3
79.7
EsViT-
42.0
38.5
51.5
45.6
49.7
52.1
62.0
76.5
SimMIM-1K
34.1
32.4
52.9
46.7
42.4
51.7
14.2
75.9
ICAR-Laion
43.3
39.5
51.7
45.5
51.0
55.3
65.1
79.5
iCAR-Laion-228
44.9
41.2
52.3
46.1
51.1
55.4
69.4
80.2
Table I: Comparison of different pretraining tasks on frozen and full finetuning settings. Results of box mAP and mask mAP for COCO object detection, mloU for ADE20K semantic segmentation, and top-1 accuracy for Kinetics-400 action recognition are reported. SUP denotes supervised classification as pretraining.
Results are shown in Table I For the models pretrained on the ImageNet-1K dataset, we can observe that SUP-1K and EsViT-IK perform similarly on almost all the benchmarks, including both the full finetuning and frozen settings. This phenomenon is understandable, because the linear evaluation result of ViT is relatively high, indicating that the features extracted by EsViT- are similar to the SUP-1K counterpart. The SimMIM pretrained model performs comperitively high the full finetuning setting. For example, it achieves much better performance on COCÓ, comparable performance on ADE, and slightly worse performance on K400, in comparison to the SUP-1K counterpart. However, the SimMIM-1K model performs poorly on all benchmarks in the frozen setting. This could be foreseen from its poor linear evaluation performance on ImageNet- IK, indicating that its output features do not capture high-level semantics. The iCAR-Laion model outperforms SUP-1 K and ViT- on almost all the benchmarks, indicating the effectiveness of learning from large scale image-text datasets. In addition, with jointly training on Laion and ImageNet-22K,, the
L-RR-LLinn-200 model achieves similar results with the SUP-22K model. We can also observe that the SUP-22K model outperforms SUP-1K significantly in all settings, reflecting the great benefit that data scaling brings to supervised pretraining. In general, we find that supervised pretraining works best under the frozen setting, and we adopt supervised pretrained models by default in the following experiments. However, in all three downstream tasks, the frozen settings still underperform full finetuning by substantial margins.
5
What is the Key to Making the Frozen Setting Work?
Towards improving performance over disparate tasks, in this section we consider how to extend the frozen setting for better task adaptation. For the following investigation, we utilize the best- pest-performing pretraining task, supervised classification on ImageNet-22K (SUP-22K).
5.1 Adding More Tunable Parameters at Head Networks
To some extent, the poor performance of the frozen setting is understandable. Despite the significant difference between the pretraining and downstream task, all parameters in the base networks locked in the frozen setting, and only a few newly- parameters are available for task adaptation. Thus, direct solution is to add more tunable parameters to the task-specific head networks. As the head network varies among different downstream tasks, we present task-specific strategies on how to add parameters to the head.
Object Detection For object detection, we adopt the framework of Mask R-CNN 20 with FPN 33 as the head network. Previous work on Mask R-CNN mainly explored improvements over FPN and the box/mask head, so we also examine common alternatives to these two components, e.g., replacing

---

Frozen
Full ft.
Head Network
Apbox
Apmask
Apbox
Apmusk
FPN
45.0
41.1
51.9
45.7
FPN 4x Residual Blocks inside
49.5
44.0
52.1
45.6
BiFPN
51.9
46.0
52.3
45.7
FPN w. Cascade Head
49.0
43.0
54.5
46.9
BiFPN w. Cascade Head
53.8
46.7
54.3
46.9
Table 2: Results for different head networks on COCO object detection and instance segmentation, including FPN with vanilla box/mask head, FPN with four more residual blocks inside, BiFPN, FPN with cascade head, and BiFPN with cascade head.
FPN with BiFPN [51 adding more residual blocks in FPN, and changing the vanilla box/mask We can observe that adding
head to cascaded box/mask head 44. Results are shown Table 2 parameters to FPN. e.g., replacing FPN with BiFPN adding more residual blocks, both significantly bridge the gap between the frozen setting and full finetuning, from -6.9 to -2.6 box mAP and -6.9 to -0.4 box mAP, respectively. As BiFPN is carefully designed with many interactions between multi-resolution feature maps, replacing FPN with BiFPN outperforms the model of adding more residual blocks in FPN. However, replacing the box/mask head with cascade head [4] improves the performance of both the frozen and full finetuning settings, from 45.0 to 49.0 box mAP and 51.9 to 54.5 box mAP, but does not bridge the gap between them. Adding more tunable parameters to FPN is thus found to be more effective than adding to the box/mask head.
Px
F1
F2
F3
F4
p=0
F
F2
psl
p52
FS
F2
P2
P5D
P3
P4
Pool
F22
P6
(a) CKA Similarity of Bi-FPN
(b) CKA Similarity of Cascade Head
Cascade head
Output 0
Ro!
BLFPN CLFPNBBack Block
BLFPN Block
Bi-FPN Block
Bi-FPN Block
F3
F4
Input
Cascade head
Output
Output
P3
P2
Cascade head
Output 2
Input
Pio
(c) Bi-FPN
(d) Cascade Head
Figure 3: The CKA heatmap on the features across layers in BiFPN (a) and Cascade Head (b). The detailed architecture of BiFPN (c) and Cascade Head (d).
To understand this difference in behavior, we conduct performance analysis and an examination of feature similarity across different layers in BiFPN and Cascade Head by Centered Kernel Alignment (CKA) [27]. As shown in Figure 3 (a), for each stage (with different resolutions of feature map) of BiFPN, we plot the CKA similarity between the input features (Px of stage z) and the output of each block (Fz of block æ). For the cascade head, as shown in Figure (b), we plot the CKA similarity between the input features each stage (F in stage i) and the hidden features inside each stage (F output j's block stage i). From this figure, we can observe that the features across layers and even stages in cascade heads are almost the same except for the last output, but in BiFPN, features across different layers are different (with lower CKA similarity). In other words, adding parameters in BiFPN provide more useful capacity for transformation, but adding parameters

---

in the cascade head hardly provides useful computation. This difference can be explained by the architecture of BiFPN (igureB[c) and cascade head (Figure d)) BiFPN follows sequential structure, where the input of each block is the output of the previous block. The cascade head follows a parallel structure, and all stages extract Rol features from the original multi-resolution feature map, making the input of each stage have extremely high CKA similarity. Also, the transformations made by each stage are similar. The last output of the cascade head behaves differently because is followed by a pooling layer, which is proven to have great impact CKA similarity. Consequently, the number of well-placed tunable parameters is the key to make frozen settings work well.
Frozen mloU
Full Ft. mloU
Head Network
1 Pixel Dec. w. (3+1)x )x Transformer Dec.
51.0
55.4
6x Pixel Dec. w. (3+1)x Transformer Dec.
52.6
55.3
1x Pixel Dec. w. (9+1)x Transformer Dec.
51.9
55.3
6x Pixel Dec. w. (9+1)x Transformer Dec.
53.2
55.5
Table Results from using different head networks on ADE20K semantic segmentation, with either one six pixel decoder blocks, and Transformer decoder with four or ten blocks.
Semantic Segmentation For semantic segmentation, we adopt Mask2former 9 with a one- block pixel decoder and four-block transformer decoder as the head network. The framework of Mask2former is similar Mask R- R-CNN in object detection, where the pixel decoder corresponds to the FPN, and the transformer head corresponds to the box/mask head. Thus, we employ a setting similar that used for object detection, by increasing the number of parameters in these two major components, e.g., changing the pixel decoder to six blocks and the transformer decoder to ten blocks. As shown in Table 3, similar results can observed where enlarging the pixel decoder helps more in bridging the gap than enlarging the transformer head. We also perform a CKA A analysis on this framework and observe behaviors like that in object detection, as shown in the Appendix.
Frozen
Full Ft.
Head Network
top-1 I accuracy
top-1 accuracy
Linear
70.3
79.7
4x Temporal Blocks
74.7
79.5
4x Global Blocks
77.5
79.8
Table 4: Results with different head networks on Kinetics-400 video action recognition: linear
classifier, four temporal Transformer blocks, and four global (spatiotemporal) Transformer blocks.
Video Action Recognition For video action recognition, it is crucial to capture both spatial and temporal relationships, but the pretraining model is only capable of spatial reasoning. Under the frozen setting, the lack of temporal modeling thus needs to be compensated for in the head network. In our experiments, we adopt spatial- spatial-then- then-temporal is enhanced with additional Transformer blocks, either temporal-only or global (spatiotemporal), In Table 4. we can observe that under the full finetuning setting, adding global blocks and adding temporal blocks work similarly with a linear classifier. On the other hand, adding temporal blocks in the frozen setting leads to significantly better performance than the linear classifier setting, which verifies the need for temporal reasoning. Also, adding global blocks in the frozen setting improves substantially over adding temporal-only blocks. Although the base network pretrained for spatial reasoning and global blocks are added, there is still a non-trivial gap between the frozen and full finetuning settings. This may be due to some missing information from the original features, and the remaining gap may be reduced by further introducing network with independent access to the input, which is beyond the scope of this paper and left for future work.
(39) framework that
5.2 Does Larger Pretrained Base Network Need Smaller Head Network?
When the pretrained model becomes larger, it generally contains richer and more valuable information, so for downstream tasks, we generally expect that a larger pretrained model does not require as large of a head. In NLP, it is verified that only few layers or few prompts are needed when the
8

---

0000 AAP
ADR Amlol)
K500 Anceiig
-1 -100
-1.5
Swin-T 1K
Sein-
-5
Swin-B
1K
-2
Swin- 1K
Swin- B 22K
-12
Swin-B 222
Swin-B 1K
Swin: L. 22K
Swin- L 22K
Swin-l. 22K
14
3
Number of BiiFPN Layers.
Number of Pizel Decoder Layes.
Number of additionnl global blocks.
(a) COCO
(b) ADE20K
(c) Kinctics-400
Figure 4: Performance improvements on COCO object detection, ADE semantic segmentation, and Kinetics-400 action recognition with increases in the number of trainable parameters.
pretrained model is extremely large, e.g., 173B, For computer vision, we seek an answer to whether the current best pretrained models have similar properties. We perform the empirical study with different sizes of pretrained models, SwinV2-T// with ImageNet- -K pretraining and Swin V2-B/22BB/ with ImageNet-22KK pretraining, the three representative downstream tasks. From Figure (a), on COCO object detection, we can surprisingly observe a clear trend, that SwinV2-L-22K requires fewer parameters converge (a 5-layer BiFPN), while Swin V requires to converge (a 9-layer BiFPN). On ADE20K semantic segmentation that SwinV2-T--1 2-1K requires more parameters to converge (a 6-layer pixel decoder) than other larger base networks, On Kinetics-400 video action recognition in Figure + (c), different models appear to require a similar number of parameters converge. This observation matches the previous one, that after careful tuning, there is still a non-trivial gap between the frozen and full finetuning settings on video action recognition (as shown Table 4). indicating a larger difference between pretraining and video recognition tasks than others.
parameters Figure 4 (b), we also find
6 Results with Different Sizes of Base Networks
COCO
ADE20K
Kinetics-400
Base Network
Head
Frozen
Full ft.
Frozen mloU
Full ft.
Frozen
Full ft.
Network
Aphox
Apmask
Aphox
Apmask
mIoU
acc
acc @
SwinV2-1-IK
Mask.
49.6
44.0
49.8
43.8
48.9
50.3
70.6
75.7
SwinV2-B-IK
Mask.
50.3
44.5
51.1
44.8
51.5
52.7
73.3
77.9
SwinV2-B-22K
Mask.
52.5
46.6
52.3
45.7
53.2
55.5
77.7
79.8
SwinV2-L-22K
Mask.
53.0
46.8
53.0
46.5
54.4
56.5
78.7
79.8
SwinV2-B-22K
Cascade.
53.8
46.7
54.3
46.9
HTC
57.9
50.4
SwinV2-G-ex122K
57.6
81.7
HTCI
59.3
51.6
Table 5: Comparison of different-sized base networks with carefully designed head networks frozen and full finetuning settings, including SwinV2-T/B pretrained on ImageNet-1K Swin pretrained on ImageNet-22K,, and SwinV2-GG pretrained ext22K. Results of box mAP and mask mAP for COCO object detection validation set, mloU for ADE20K semantic segmentation, and top-1 accuracy for Kinetics-400 action recognition reported. Mask. denotes Mask R-CNN is used the head network, and Cascade. refers to Cascade Mask R-CNN. denotes multi-scale testing on COCO.
After careful design of the head networks for each downstream task, we compare different sizes of base network with the best setup of frozen and full finetuning settings, including SwinV2-T/B pretrained on ImageNet-1K: and Swin V2- B/L pretrained on ImageNet-225 222K, shown in Table On COCO object detection, the performance gap is well bridged, where Swin V2-B//222 even achieve on-par performance under the frozen and full finetuning settings with two different head networks. But on ADE20K, there is still significant performance gap. For example, using Swin as the base network, the frozen setting still underperforms the full finetuning setting by mloU. A large performance gap between the frozen and full finetuning settings also be observed on Kinetics-400

---

video action recognition. Interestingly, the performance gaps on ImageNet- IK pretrained models are significantly larger than that for models pretrained on ImageNeet22K,, indicating that enlarging the pretraining dataset from ImageNet-I I K to ImageNet-22K helps to bridge the task gap of pretraining and finetuning on action recognition.
We additionally explore the upper bound of performance under the frozen setting, using frozen supervised pretrained model with 3 billion parameters trained on 70M images (SwinV2-G), shown in Table 5 With this giant model, highly competitive performance is achieved on major benchmarks: 59.3/60.0 box mAP and 51.6/52.2 mask mAP on COCO validatiion/tstt-ee sets, 57.6 mloU on ADE20K semantic segmentation, and 81.7 top-1 accuracy on Kinetics-400 action recognition.
7
When Does Frozen Setting Outperform Full Finetuning?
To further explore the potential of the frozen setting, we compare with full finetuning in the low-data scenario. More specifically, we test these two settings with small portion of the COCO training dataset. Following the standard practice semi-supervised learning [50 [55,, we construct two datasets with only 1% and 10% (about 12K and 120K training images, respectively) of COCO.
1% data
10% data
Head Network
Frozen
Full ft.
Frozen
Full ft.
A Abox
A pmask
A Apbox
Apmask
Apbox
A pmask
Apbox
A ppmsk
FPN
27.5
26.4
27.5
25.8
39.1
36.2
42.2
37.5
1x BiFPN Layer
29.8
27.6
26.3
24.7
42.8
38.5
42.0
37.3
BiFPN Layer
31.4
28.4
26.3
24.6
44.2
39.4
42.1
37.3
3x BiFPN Layer
32.1
28.9
25.9
24.3
44.6
39.7
42.3
37.6
4x BiFPN Layer
31.9
28.6
26.0
24.1
45.0
40.0
42.1
37.4
Table 6: Results for different head networks COCO with 1% and 10% labeled data. Head networks
include FPN and BiFPN.
Results are shown in Table For training with 1 % labeled data, when adopting FPN as head, the frozen setting achieves performance competitive with the full finetuning one. With an increasing number of BiFPN layers, we can observe stable performance improvements for the frozen setting and inferior performance for full finetuning, which indicates an overfitting issue for the full finetuning setting. For training with 10% data, there is performance gap of -3.1 box mAP between the frozen setting and full finetuning with FPN as the head network. Compared with training on the full COCO training dataset, this gap is reduced from -6.9 box mAP to -3.1 box mAP. When replacing FPN with BiFPN, the frozen setting surpasses full finetuning with a clear margin of 2.9 box AP and 2.6 mask AP.
8 Conclusion
In this paper, we presented a detailed empirical study of frozen pretrained models for diverse computer vision tasks. Based on observations, proposed strategies for extending the frozen setting to work effectively for various downstream tasks, and also found that a universal representation with solid performance highlights the strong potential of this transfer learning approach, and we hope it will kindle further interest in this research direction.
assorted benchmarks can be learned for giant frozen model. Our study
10

---

References
[1] Bao, H., Dong, L., and Wei, (2021). Beit: Bert pre-training of image transformers. arXiv preprint 21X.::2106.08254.
[2] Bilen, H. and Vedaldi, A. (2017). Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.02275
[3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
[4] Cai, Z. and Vasconcelos, N. (2018). Cascade r-cnm:
Delving into high quality object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6154-6162.
[5] Cao, Y., Xie, Z., Liu, B., Lin, Y., Zhang, Z.,, and Hu, H. (2020). Parametric instance classification for unsupervised visual feature learning, Advances in Neural Information Processing Systems, 33.
[6] Carreira, and Zisserman, A. (2017). Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308.
[7] Chen, M., Radford, A., Child, R., Wu, and Jun, H. (2020a). Generative pretraining from pixels. Advances in Neural Information Processing Systems.
[8] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020b). A simple framework for contrastive learning of visual representations. JCML.
[9] Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R. (2021). Masked-attention mask transformer for universal image segmentation. arXiv preprint arXiv:21
[10] Deng. J., Dong, W., Socher, R., Li. L.J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. leee.
[11] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 110.:8880004055.
[12] Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, F., and Darrell, T. (2014). Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 647-655. PMLR.
[13] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations.
[14] Dosovitskiy, A., Springenberg. J. T., Riedmiller, M., and Brox, T. (2014), Discriminative unsu- pervised feature learning with convolutional neural networks. Advances in neural information processing systems, pages 766-774.
[15] Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.-Y,, Cubuk, E. D., L. V., and Zoph, B. (2021). Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2918-2928.
[16] Girshick, R., Donahue, J., Darrell, T., and Malik, (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587.
[17] Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. (2020). Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33.
[18] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. (2021). Masked autoencoders are scalable vision learners. arXiv preprint 211.0:2111.06377.

---

[19] He, K., Fan, H.. Wu, Y., Xie, S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. CVPR.
[20] He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask r-cnn. In ICCV, pages 2961-2969.
[21] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In CVPR, pages 770-778.
[22] Houlsby, N., Giurgiu. A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q.. Gesmundo, A., Attariyan, M., and Gelly. S. (2019), Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799, PMIR.
[23] Hu, J., Shen, Y. Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0668.
[24] Jia, C., Yang, Y., Xia, Y., Chen, Y-TT., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 9904-4916 PMLR.
[25] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F,, Green, T., Back, T., Natsev, P., al. (2017). The kinetics human action video dataset. arXiv preprint arXiv: 170.006950.
[26] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. (2019). Big transfer (bit): General visual representation learning.
[27] Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019a). Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR.
[28] Kornblith, S., Shlens, J., and Le, Q. V. (2019b). Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661-2671.
[29] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105.
[30] Lester, B., AI-Rfou, R., and Constant, (2021). The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691
[31] Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., and Gao, J. (2021). Efficient self-supervised vision transformers for representation learning. arXiv preprint arXiv:2106.09785
[32] Li, L. and Liang, (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190
[33] Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017). Feature pyramid networks for object detection. ICCV, pages 2117-2125.
[34] Lin. T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In European conference computer vision, pages 740-755. Springer.
[35] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2021a). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.1353
[36] Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, (2021b). Gpt understands, too. arXiv preprint 2rXiv:2103.10385
[37] Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, I., et al. (2021c). Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.0988
12

---

[38] Liu, Z., Lin, Y. Cao, Y., Hu, H., Wei, Y. Zhang, Z., Lin, S., and Guo, B. (2021d). Swin trans- former: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030
[39] Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., and Hu, H. (2021e). Video swin transformer.
[40] Long, J., Shelhamer, E., and Darrell, (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440.
[41] Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.. et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
[42] Rebuffi, S.-A., Bilen, H., and Vedaldi, A. (2017). Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30.
[43] Rebuffi, S.-A., Bilen, H., and Vedaldi, A. (2018). Efficient parametrization of multi-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 81 8119-8121
[44] Rusu, A. A., Rabinowitz, N. C.,, Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K.. Pas- canu, R., and Hadsell, R. (2016). Progressive neural networks. arXiv preprint arXiv: 1606. 04671.
[45] Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. (2021). Laion-400m: Open dataset of elip-filtere 400 million image-text pairs. arXiv preprint arXiv:2111.02114.
[46] Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:
[47] Sharif Razavian, A., Azizpour, H., Sullivan, J., and Carlsson, S. (2014). Can features off-the- shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806-813.
[48] Simonyan, K. and Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pages 568-576.
[49] Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations.
[50] Sohn, K., Zhang, Z., Li, C.-L., Zhang, H., Lee, C.-Y., and Pfister, T. (2020). A simple semi-supervised learning framework for object detection. In a7X/::2005.04757.
[51] Tan, M., Pang, R., and Le, Q. V. (2020). Efficientdet: Scalable and efficient object detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781-10790.
[52] Vasconcelos, C., Birodkar, V., and Dumoulin, V. (2022). Proper reuse of image classification features improves object detection. arXiv preprint arXiv:2204.0044
[53] Wei, Y., Cao, Y., Zhang, Z., Yao, Z., Xie, Z., Hu, H., and Guo, B. (2022). icar: Bridging image classification and image-text alignment for visual recognition.
[54] Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu, H. (2021). Simmim: A simple framework for masked image modeling. arXiv preprint 21X0v:2/11.0986.
[55] Xu, M., Zhang, Z., Hu, H., Wang. J.. Wang, L.. Wei, F., Bai, X., and Liu, (2021), End-to-end semi-supervised object detection with soft teacher. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
[56) Yao, Y., Zhang, A., Zhang, 7., Liu, Z., Chua, T.-S., and Sun, M. (2021). Cpt: Colorful prompt tuning for pre- pre-trained langunelanguage models. arXiv preprint
13

---

[57] Zhang, J. O., Sax, A., Zamir, A., Guibas, L., and Malik, J. (2020). Side-tuning: a baseline for network adaptation via additive side networks. In European Conference on Computer Vision, pages 698-714. Springer.
[58] Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. (2018). Semantic understanding scenes through the ade20k dataset. International Journal on Computer Vision
[59] Zhou, K., Yang, J., Loy, C. C., and Liu, Z. (2021). Learning to prompt for vision-language models. arXiv preprint 200::2209.00134.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Section 6 L309-IL315.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [No] We use public datasets and our code will be released.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 4 5 Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [No] Datasets are all public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "Zn5LitvYKpc", "reviewer": "Reviewer_ynJ6", "summary": "The paper presents a comprehensive analysis of the use of pretrained image models for some major vision tasks. The paper shows the importance of well-placed tunable parameters to bridge the gap between frozen and finetuning settings, and presents an analysis of feature activation to further showcase this.", "questions": "See weakness.\n", "limitations": "Not adequately. There's fairness risks around using frozen pretrained models depending on the datasets the frozen models are trained on.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* The paper presents an insightful analysis of the properties of frozen pretrained models for downstream tasks\n* On detection tasks, the gap between frozen model and finetuning is negligible, while the gap is still there for semantic segmentation and video action recognition.\n* The authors also analyze different sizes of base network and different pre-training strategy and report their properties on various downstream tasks. It’s interesting to larger models require less tunable parameter and achieve better performance without increasing trainable parameters.\n\nWeakness:\n* The results in Table 5 suggest that for some tasks frozen models are good enough (e.g. detection), but for others fine tuning still provides clear benefits (e.g. semantic segmentation, action recognition). It would be interesting to delve deeper into what causes the gap, e.g.  pretraining methods, model design, task itself, or specific dataset. For example, it’s interesting that the AP mask on COCO shows no gap between frozen vs full ft, but there’s a 2 point gap on ADE20K mIoU. I would expect the trend between AP mask and mIoU to be similar as both involve pixelwise localization. For video action recognition, I’m wondering if using a frozen model pretrained on videos (rather than images) would help bridge the gap with full ft.\n* It might be helpful to show the benefits of frozen models by comparing training time or resources used.\n* In Table 1, it may be interesting to add MAE (masked autoencoder) style self-supervised learning and see if that shows any difference between frozen and full ft.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A1sBSDPtWVw", "reviewer": "Reviewer_RULo", "summary": "This paper studies way to use large, pretrained image models for many different tasks. The approach pre-trained a swin transformer model using either supervised learning, contrastive learning or masked image modeling. This model is then frozen, and new layers are added and trained for the specific task. This paper finds that supervised pretraining transfers best and adding more layers for the finetuning is generally beneficial.", "questions": "Please address the weaknesses above. Specifically,\n\n(1) Will these results change with different pretraining data scales?\n(2) Are these methods really the key to using frozen image models? Are there other methods that could be better?\n(3) If this paper is trying to find the key to using frozen models, why are the results lower than finetuned? Is this an inherent limitation of using frozen models? Or has the method to properly utilize pretrained models not been found?", "limitations": "There was no impact section and the only limitations mentioned were that the approach does not perform as well as full finetuning.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, clear and easy to follow. The topic is quite relevant as training large models is becoming common, and finding the right ways to use the pretrained models is quite important.\n\nHowever, there are a few weaknesses that should be addressed:\n\n(1)  Table 1 results are only on small-ish pretraining datasets of ImageNet-1k and -21k. This could be limiting, as a main benefit of the contrastive training, e.g., CLIP and ALIGN has come from the massive dataset sizes (e.g., hundreds of millions to billions). It has also been shown that large image transformer models (e.g., ViT) benefit more from large supervised pretraining (e.g., billions of samples). These observations make it hard to know if the results in table 1, Fig 4, table 5 will generalize, or if they are specific to this setting. For example, adding experiments using the CLIP pretrained model, to see how that scale of pretraining data effects the results.\n\n(2) The paper doesn't really propose anything new. It is a useful study, however, it is limited by the few settings compared. E.g., adding more layers and BiFPN for object detection, adding more layers for segmentation/action classification. These aren't especially interesting insights. One of the questions this paper sets out to address is:\n\"What is the key to making the frozen setting work well when the downstream tasks are significantly different from the pretraining task?\" \nAnd I don't think the current experiments really answer that question. Unless the key is simply adding more tuneable layers. Related to that, it would be interesting to compare adding trainable layers vs. making parts of the pre-trained model trainable. E.g., instead of adding 10 new layers, only add 1, but make the last 9 of the pre-trained model trainable.\n\nThere are many other options not explored in this work, which limits the insight gained from this work.\n\nOverall, the paper has limited originality and significance due to those weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5pJM3jGEC_b", "reviewer": "Reviewer_Fp53", "summary": "Authors present a relatively comprehensive study of how far frozen pre-trained models can get us for multiple downstream computer vision tasks: object detection, semantic segmentation and action classification. They specifically explore 1) which pretraining task leads to most useful representations (out of supervised, self-supervised and masked image modeling, supervised on In22K is best); 2) How is the performance affected by adding more tunable parameters (it helps significantly, but less so for larger frozen models -- similar to trends observed for large language models) and 3) What is the best results one can achieve with a very large frozen model (Swin-V2-G, 3B parameters). They obtain some good results on all the 3 tasks considered.", "questions": "I would be interested in seeing more analysis on computational savings (i.e. training time and GPU memory) of adding additional tunable parameters on frozen backbone VS finetuning the frozen backbone. If that improves accessibility of SOTA models to users with limited resources, that would increase the significance of this work.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. [Significance] Impressive results with a frozen backbone: The results reported without finetuning the backbone are quite impressive and to my knowledge not previously reported. For instance 79.6% on Kinetics-400 with a frame-level feature extractor + few temporal/global aggregation blocks is comparable to recent work like TimeSformer, which is finetuned end-to-end. However it is obtained by adding additional tunable parameters on top of the frozen model, so it's not clear how favorably it compares in training compute cost to full finetuning.\n\n2. [Clarity] The paper is well written and easy to follow. The sections are organized well and flow naturally, making the paper an easy and pleasant read.\n\n## Weaknesses\n\n1. [Novelty] The results aren't particularly novel: While they haven't been reported before to my knowledge and hence are interesting for the community to know, they aren't surprising or unexpected. ImageNet-22K models have been well known to generalize better to downstream task; and masked-image-modeling based models have not generalized well without full finetuning. Adding more tunable parameters is expected to improve performance. \n\n2. [Quality] Analysis of training speed/memory usage: What does training with frozen backbone enable? One important gain would be making state of the art models accessible to users with limited resources. However the paper doesn't show explicit comparison on this axis. How does the training time of frozen + tunable parameter models compare to finetuning the frozen model without tunable parameters? Does it lead to significant training time or GPU memory savings that enable SOTA models to be trained on very small GPUs? Or do the additional trainable parameters somewhat nullify the gains by freezing the backbone?\n\n3. [Clarity] Some parts of the model enhancements are not clear. What is the precise design of the \"global\" and \"temporal\" blocks added in Table 4? Also, how is the aggregation over the full video done?\n\n4. [Quality] Additional analysis: While the paper has decent amount of analysis, it is limited to Swin Transformer based architectures; similar analysis on ViTs would make the paper stronger. Also, the set of tasks is somewhat limited -- object detection and semantic segmentation are quite similar (and authors also observe similar trends for both). Paper's impact can be increased by adding few more computer vision tasks, such as vision-language tasks (VQA, captioning etc), 3D/RGBD recognition tasks etc.\n\n## Overall\nThe paper is generally well written and motivated. The results are interesting however not totally novel or surprising. More analysis of the computational savings would make future versions of the paper stronger. Hence, I am borderline, however leaning towards acceptance given the range of results shown across object detection, segmentation and video classification, with different sized Swin models etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept