Masked Distillation Advances Self-Supervised Transformer Architecture Search
Authors: Caixia Yan
Xiaojun Chang
Zhihui Li
Lina Yao
Minnan Luo
Qinghua Zheng
Venue: ICLR 2024 poster (2024)
Keywords: Knowledge Distillation
Self-supervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A masked image modelling (MIM) based self-supervised neural architecture search method specifically designed for vision transformers.

Abstract: Transformer architecture search (TAS) has achieved remarkable progress in automating the neural architecture design process of vision transformers. Recent TAS advancements have discovered outstanding transformer architectures while saving tremendous labor from human experts. However, it is still cumbersome to deploy these methods in real-world applications due to the expensive costs of data labeling under the supervised learning paradigm. To this end, this paper proposes a masked image modelling (MIM) based self-supervised neural architecture search method specifically designed for vision transformers, termed as MaskTAS, which completely avoids the expensive costs of data labeling inherited from supervised learning. Based on the one-shot NAS framework, MaskTAS requires to train various weight-sharing subnets, which can easily diverged without strong supervision in MIM-based self-supervised learning. For this issue, we design the search space of MaskTAS as a siamesed teacher-student architecture to distill knowledge from pre-trained networks, allowing for efficient training of the transformer supernet. To achieve self-supervised transformer architecture search, we further design a novel unsupervised evaluation metric for the evolutionary search algorithm, where each candidate of the student branch is rated by measuring its consistency with the larger teacher network. Extensive experiments demonstrate that the searched architectures can achieve state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets even without using manual labels. Moreover, the proposed MaskTAS can generalize well to various data domains and tasks by searching specialized transformer architectures in self-supervised manner.

Published as a conference paper at ICLR 2024
MASKED DISTILLATION ADVANCES SELF-SUPERVISED TRANSFORMER ARCHITECTURE SEARCH
Caixia Yan
Xiaojun Chang
School Computer Science and Technology MOEKLINNS Laboratory, Xi'an Jiaotong University yancaixia@xjtul wew.tu..edde...
University of Science and Technology of China Mohamed bin Zayed University of Artificial Intelligence cx cxj273e gma com
Zhihui Li
Lina Yao
School Information Science and Technology University of Science and Technology of China zhihul hhihuill ics00mm @ma
CSIRO's Data61
University of New South Wales
com
1ina waw.@aaaaaaaaa. cs cairo.att
Minnan Luo
Qinghua Zheng
School of Computer Science and Technology MOEKLINNS Laboratory, Xi an Jiaotong University minnluoee uunluooxxtu juu edu.cn
School of Computer Science and Technology
MOEKLINNS Laboratory, Xi' an Jiaotong University qhzheng@ma il xjju. edu. cn
ABSTRACT
Transformer architecture search (TAS) has achieved remarkable progress in au- tomating the neural architecture design process of vision transformers. Recent TAS advancements have discovered outstanding transformer architectures while saving tremendous labor from human experts. However, it is still cumbersome deploy these methods real-world applications due to the expensive costs data labeling under the supervised learning paradigm. To this end, this paper proposes masked image modelling (MIM) based self-supervise neural architecture search method specifically designed for vision transformers, termed as MaskTAS, which completely avoids the expensive costs of data labeling inherited from supervised learning. Based on the one-shot NAS framework, MaskTAS requires train vari- ous weight-sharing subnets, which can easily diverged without strong supervision in MIM-based self-supervised learning. For this issue, we design the search space of MaskTAS siamesed teacher-student architecture distill knowledge from pre-trained networks, allowing for efficient training of the transformer supernet. To achieve self-supervised transformer architecture search, we further design a novel unsupervised evaluation metric for the evolutionary search algorithm, where each candidate of the student branch is rated measuring its consistency with the larger teacher network. Extensive experiments demonstrate that the searched architectures can achieve state-of-the-art accuracy on CIFAR-10. CIFAR-100, and ImageNet datasets even without using manual labels. Moreover, the proposed MaskTAS can generalize well various data domains and tasks by searching specialized transformer architectures in self-supervised manner.
INTRODUCTION
Vision transformer (VIT), self-attention characterized neural network, has recently emerged an alternative to convolutional neural networks (CNNs) in computer vision community. In an attempt progress this field, surge of advanced ViT networks have been developed
a wide range
of vision tasks, such as image recognition Dosovitskiy et al. (2020); Han et al. (2021); Liu et al. (2021b), object detection Liu et (2021b); Zhu al. (2020) and semantic segmentation [Strude] et al. (2021) Xie et al. (2021). However, despite the empirical success of these approaches, they all
"Corresponding Author

---

Published as a conference paper at ICLR 2024
Subeo!
Tocke
Sc/Sppeniall Sepenst Trissg
Sdbas]
Sanoad Stake-Stukktt Sqgae.
Clack
Dynan HLF
Dynnnic MLF
Toeler
Stadee
Renals
Toctation
Search
Supenst
Scadleed Archbedar
Figure 1: The pipeline of the proposed self-supervised transformer architecture search framework, including a) self- self-supervised supernet training, b) self-supervised architecture search, and c) supervised re-training of searched architecture.
rely heavily on hand-crafted ViT architectures designed by human experts, meaning that laborious trial-and-error testing is required. In particular, a large number of design choices needs to be analyzed and determined in ViT architecture, such as patch size of input, number of self-attention heads, query/key/value dimension, MLP ratio and network depth, significantly increasing the difficulty of architecture design. Moreover, these manually designed architectures are fixed once obtained, which cannot necessarily ensure the optimality for diversified data domains and task scenarios.
Neural Architecture Search (NAS) Ren et al. (2021) Liu al. (2021a): Elsken al.) (2019) Yan
al. (2021) has become an effective way for automating the design of neural networks. Inspired by its remarkable success in CNNs, NAS techniques have also been investigated for searching standard transformer architectures recently. Existing studies Transformer Architecture Search (TAS) mainly focus on refining search space (e.g.. S3 [hen et al. (202lc). Ghen et al. (2021a) and AutoFormer Chen et al. (2021b)) and/or improving search algorithms (e.g.. AutoFormer Chen et al. (202lb) and VITAS Su et al. (202T)). Benefiting from the automated network design process, these methods can discover customized architectures that achieve comparable or better performance, compared with manually designed vision transformers. However, despite their empirical success, the training and architecture search stage of these approaches are both performed in fully-supervised paradigm, meaning that the optimal transformer architectures are typically searched based on both images and their associated labels. This dependency would inevitably lead to expensive costs of data labelling, limiting the scalability and transferability of the transformer architecture search approaches. As viable alternative, self-supervised learning obtains supervisory signals from the data itself and has recently been shown to address the appetite for data successfully CChnn et al. (2020); Grill et al. (2020); Wang et al. (202I). A few recent works have explored how to discover desirable network architectures based on sell self-supervised search strategy. However, they are concentrated on convolutional modules in CNNs, which are quite different from vision transformers.
In this paper, we firstly explore self-supervised neural architecture search specifically for vision transformers, dedicated to search promising vision transformer architectures in a self-supervised manner. Concretely, we take advantage of the efficacy of Masked Image Modeling (MIM) Wei et al.) (2022); He et al.] (2022): Feichtenhofer et al. (2022); Assran et al. (2022) in developing scalable ViT models, and propose MIM based self-supervised ViT architecture search method, termed as MaskTAS. Following the standard one-shot NAS pipeline, the overall framework of MaskTAS can be decomposed into two stages, i.e., supernet training and subnet search. The overall pipeline of the proposed MaskTAS has been demonstrated in Figure]! [
To enable self supervieed supernet training, the proposed MaskTAS adapts the standard one-shot NAS framework to the regime of MIM based self-supervised learning. During supernet training, millions of weight-sharing subnets the supernet need to be trained by masking and recovering random image patches. However, due to the lack of strong and stable supervision, the co-training of various subnets prone to diverging, leading to suboptimal results. To this end, we build the MaskTAS supernet as reacher-student based siamese architecture, where the pre-trained larger teacher network can provide strong supervision to facilitate the efficient training of student branch. During supernet training stage, each sampled architecture of the student network can be updated by 1) reconstructing the original image from the masked signals, and 2) predicting the feature of the teacher branch, in self-supervised manner.
2

---

Published as a conference paper at ICLR 2024
After the supernet training is complete, the architecture search stage be performed by rating the candidates sampled from the well-optimized supernet. To avoid the dependency on image labels, we design unsupervised evaluation metric for the evolutionary search algorithm, where the performance of each candidate architecture the student measured by computing its feature similarity with the teacher network. In this way, the teacher with stabilized behavior can act the evaluation target for the sampled architectures of the student branch. Once the optimal architecture has been found, it can be retrained on the target dataset after inheriting the weights from the well- optimized supernet. The contributions of the present work can be summarized as follows:
We establish a MIM-based self-supevised learning framework, called MaskTAS, to pursue scalable and transferable VIT architecture search. To the best of our knowledge, this is the earliest effort to develop self-supervised architecture search paradigm for ViTs.
We reveal the key challenge self-supe supernet training, i.e., the diverging of diverse subnets co-training. A siamesed teacher-studen supemet is designed to address this challenge by introducing extra supervision from pre-trained teacher network.
To allow for self-supervised architecture search, we further design a self-supervised evalua- tion metric based evolutionary search algorithm, where each candidate of the student branch is rated by measuring its consistency with the teacher network.
2 THE PROPOSED METHODOLOGY
In this section, we first briefly review existing one-shot NAS on vision Transformers as preliminary and further point out the dilemma of those methods on producing scalable transformer architectures. Then present the proposed MaskTAS to address this issue, along with its two key components: i) self-supervised supernet training with masked feature distillation, and ii) evolutionary architecture search based on unsupervised metric.
2.1 REVISITING ONE-SHOT NAS TOWARDS TRANSFORMERS
Given a pre-defined transformer search space S, one-shot NAS typically encodes S into weight- sharing supernet (S, Ws) with weights Ws. The supernet includes various weight-sharing subnet architectures candidates, denoted as {a.} where N refers to the total number of candidates with as being the i-th candidate. Without loss of generality, the search optimal architecture a² can be formulated two-stage optimization problem. The first stage weights Ws by solving the following problem:
aims to optimize the supernet
W3 arg min Luzin (N Ws), W Durin);
Ws
where L-ran(.) refers the loss the training dataset D Dirain W represents the optimal weights after optimization. Following the weight-sharing strategy Li et al. (2021): I): Chen et al. (2021b), existing one-shot NAS methods sample and train different subnet architecture paths in each training iteration, so as to avoid exhausted memory usage.
In the second stage, the supernet (SS, Ws) with learned optimal weights W is directly utilized for subnet architecture search. The optimal architecture a can be searched by ranking the performance of subnets with the weights inherited from W i.e.,
a arg max Ace (N W:), Dva), aeS s.t. g(N(a, W:)) SC,
where (-cea(-) denotes the t accuracy of the subnet a over the validation dataset Dval: W refers the weights of inherited from NSS, Ws). The function g(-) calculates the resource consumption of each model architecture with C being the given resource constraint.
few recent works have explored the one-shot NAS framework for vision transformer architecture search, including AutoFormer Cheen et al. (2021b), VITAS Suu et al. (2022) and NASformer Ni et al. (2022). Despite their empirical success, these methods are limited to perform training and searching in the regime of supervised learning, which require tremendous labeled images for training to guarantee the searching of desirable architectures. Moreover, they rely heavily on the domain and quality of data annotation, resulting in insufficient generalization ability of the searched architectures.
3

---

Published as a conference paper at ICLR 2024
loss
Stage training
Stage II: Student Training
Stop Gradient
MIP
Stuani Encoder
Studest Decoder
Figure 2:
The self-supervised supernet training of the proposed MaskTAS, which performs: (a) teacher pre-training with masked pixel regression and (b) student training with masked feature distillation, in a self-supervised manner.
2.2 OVERVIEW OF THE PROPOSED MASKTAS
In order to avoid the dependency manual annotation, we develop self-supervised neural archi- tecture search method for transformers by replacing the supervised signal with a self-supervised counterpart. The overview of MaskTAS is presented in Figure
Algorithm Self-supervised Supernet Training in MaskTAS.
Require: Search space A; Teacher network T; Student network S; Training set Diri Loop number in an epoch Nster: Loss Livain and Crain max epochs Ne.
1: Initialize student N(S, Os) with weights Os;
2: Initialize teacher (T, 2t Pre-Train the teacher network with Lrrvin to get optimal weights :
3: for epoch = to Ne do
4: for 1, , do
5: Sample mini-batch of image data from Drr:
6: Random masking I' get the masked data X
7: Sample subnet architecture A with weights 4 from the student branch of supernet;
8: Feed X into the teacher-student supernet and compute gradients by Ve, ((x()) Liraini
9: Update subnet weights θ A,a while freezing the rest parameters of supernet: A < Adam( Adam(Ve,,) Ve,t. firain θ A(t):
end for
11: end for
Ensure: Supernet architecture with optimal weights A/S, 03) and WTT, (77).
Model Overview The proposed MaskTAS designed siamesed network architecture, which consists of two neural networks that interact with each other, i.e., teacher and student network. As transformer encoder
shown in Figure the teacher network parameterized by Ω is implemented Ea and decoder Ra, while the student network equipped with parameters θ includes a transformer encoder Eg and a decoder Rg. Besides, a feature predictor M further introduced align the output dimension of the teacher and student network.
To explore the possibility of optimal ViT in architecture-level, the architecture of student encoder Eo is designed as a transformer supernet N (A, A The search space A contains all possible architectures that can searched for transformers. For the purpose of flexible architecture search, we incorporate all the essential factors in ViTs to formulate the model search space, including patch embedding dimension, number of heads, MLP ratio and depth of architecture. The supernet contains various weight seiggh- subnet architectures, which can be formulated as
W(A, (A(A O AG) )
4

---

Published as a conference paper at ICLR 2024
where O refers to the weights of the i-th subnet A K denotes the total number of subnets. We introduce three types of supernets for the student branch, i.e., supernet-tiny supernet-small and supemet-base,, each of which corresponds to different range of parameters to satisfy different resource constraints.
Based the siamesed supernet architecture defined above, the proposed MaskTAS performs architec- ture search through a two-step pipeline, including self-supervised supernet training and architecture search.
2.3 SELF-SUPERVISED SUPERNET TRAINING
In this section, we formulate the training of supernet (MA, θA) process of masked distillation based self-supervised learning. As illustrated in Figure our supernet training process is accom- plished by two stages, i.e., teacher pre-training and student training. Specifically, we present the overall procedure for self-supervised supernet training in Algorithm
Teacher Pre-training Driven by the success of the masked auto-encoder in self-supervised pre- training. we perform masked image modelling to optimize the teacher network parameters @ under- pinned by pixel reconstruction loss. The objective of pixel reconstruction is to recover the original image from the masked one with an encoder-decoder scheme.
Given an input image E HxxC we first divide into N non-overlapping patches of size P2C pixels, where H, W, C. refer to the height, width, channel of the image respectively. Thus, each image can be represented by a set of flatted patch vectors = (2,, ... EN J. Following MAE He et al. 2022), results in two non-overlapping patch sets X, = {0f, xz,... ... v,,) and Xm = {xT ....... .. }. where X, and X,, contain N. visible and N,, masked patches respectively. The masked signal X, fed into the encoder of teacher network to generate the latent representation:
uniformly sample a subset of patches from X and mask the remaining ones. This
Z, = En X 9T.),
where Zu refers to the latent representation of the masked image and Te denotes the architecture of teacher encoder with parameters 52,,. Then, Z, is integrated with the masked tokens to generate full set of signals z which is further fed into the teacher decoder, ie.,
P= Ra IT,, Sta),
(5)
where Ta and ly correspond to the architecture and parameters of the decoder. Finally, the teacher network can be optimized by minimizing the pixel-level MSE loss between the predicted targets P and the original image I over the masked patches.
Student Training Under the one-shot NAS framework, the training of student supernet involves the co-training of diverse subnets, which trains one subnet architecture A4t) from A in each training iteration. Each subnet (A is uniformly sampled from K candidates, ie..
{
A E θ Aca E (m) gem..... . A A(K))
(k)
A(K))
(6)
where A and A denote the architecture and weights of the i-th candidate respectively. The architecture of the encoder dynamically changing during the self-supervised training process. However, without strong supervision signals, the joint training of various weight-sharing networks is prone to diverging, leading to thousands of epochs for pre-training.
To enable efficient training of diverse subnets, MaskTAS further employs knowledge distillation technology Ren et al. (2023); Hinton al. (2015) to transfer the knowledge of larger MIM pre-trained teacher models to smaller students. First, the masked signal X,, is fed into the encoder of both the teacher and student branch to output the latent representation:
{
Z, = (x,, li.):
Z, = (X, (
where 27 is the pre-trained weights of teacher encoder; Z, and Z, refer to the latent representation generated by the teacher and student network respectively. The objective is to train the randomly
5

---

Published as a conference paper at ICLR 2024
sampled student subnets efficiently by mimicking the target produced by the teacher in a knowledge distillation manner.
In general, the feature dimensions of the teacher network and the student network mismatched. To tackle this problem, we adopt an extra projection network M on the output the student network to match the feature dimension of the teacher's s target. Thus, the latent representation produced by the student encoder is forwarded to predict: 1) the high-level feature of the teacher branch, and 2) the original image in pixel-level:
7 M W,): P Ro Os,),
(8)
where Sy denotes the student decoder; a is the architecture of projection network M with weights Wo; Z" denotes the full signal derived from Z,; Z. and / refer to the predicted feature-level and pixel-level information respectively. During training, the architecture of the student branch varies by uniformly sampling, while the teacher branch is fixed to the pre-trained weights for stabilized behavior. After pre-training, the well-optimized supernet can be transferred various downstream tasks.
Self-Supervised Training Objective The supernet is trained a self-supervised manner by 1) reconstructing the original image after the masking operation, and 2) predicting the high-level feature produced by the teacher network. Given an image I as input, the overall objective function for supernet training can be derived
{ Cirain = (..I Lirain = Lreg (P, (P,
) + B (Zx, 7,),
where Cirain and Lirain correspond to the loss for teacher pre-training and student training respec- tively; Lreg and Cpre denote the loss for pixel regression and feature prediction respectively with trade-off parameter 3. More specifically, Lreg computes the basedd reconstruction loss between the original image and the reconstructed image over masked patches, i.e.,
1 Σ N PC
(P,
Xm)
(10)
where z refers to the normalized patch of xx Xmm based on the mean and standard deviation; Pk is the k-thh element of P; Nm stands for the number of masked patches. Besides, Cpre is the feature prediction loss used for knowledge distillation, which can be formulated as:
Cpre (Ze, Z,) = Smooth,, (LN (Z). LN.))
(11)
where (.) refers the layer normalization operation and Smoothe, (-) denotes the smooth E loss.
2.4 SELF-SUPERVISED ARCHITECTURE SEARCH
Once the supernet is trained to converge, we can perform architecture search to find the optimal subnets from the well-optimized student encoder N O%)) To allow for self-supervised architecture search, we design a teacher-student consistency based evaluation metric, thus adapting the standard unsupervised paradigm.
evolutionary search algorithm Guo al. (2020)
Evolutionary Search Without loss of generality, the evolutionary search process begins with the generation of N, random architectures initialized population. In each iteration, the architectures with top-k performance are selected as parents in each generation, which are further utilized to generate the next generation through crossover and mutation. For a crossover, two randomly selected candidates are picked and crossed to produce a new one during each generation. For mutation, candidate mutates depth with probability Pa first. Then it mutates each block with probability of Pm to produce new architecture. The optimal subnet architecture can be discovered after sufficient number iterations:
a" arg max (c,(N((a, 07), Dear), s.t. (aN θ2)) SC a
(12)
where Diar stands for the target dataset; is the sampled subnet architecture from A: + refers to the weights of subnet a inherited from O*; Acc(-) is the performance evaluation metric for candidate architectures.

---

Published as a conference paper at ICLR 2024
Self-supervised Architecture Evaluation Driven by the universally acknowledged principle that the larger the better works for deep neural networks, we believe the large teacher network with pre-trained weights can provide fair rating for each sampled architecture of the student branch. Each candidate architecture of the student-branch can be rated by computing its feature similarity with the bacher-branch, i.e.,
[Daar > im
Aco(NV(a, (a), Dtar)
H(Z; 2,),
(13)
where
{ 2 Eq. (e: Z = Eg- (X,, lax, O.),
(14)
where H(() is the similarity function; X refers the visible patch set after random masking of the i-th image; Z and Z refer to the output feature of the teacher and student encoder respectively.
The similarity of two feature maps, i.e., Zj and Zz. cannot be directly measured, since the output dimensions are different for the teacher and student encoder. Thus, we propose to evaluate the similarity of two feature maps by computing the relative relation of features, L...
No N.
H(Z; 22) =
Wi log Nikks
(15)
where
expl Z 5.5 2, /T)
hy = log
Σ Non exp( 22 Z /7)
(16)
hig - log
expl 23: 7.,
Σ exp( "pp 7, /T)
where 2 and Z are the j-th row of Zj and Z respectively; his and his denote the relative relation between the j-th and k-th row of Zi and Z respectively.
3 EXPERIMENTS
In this section, we first provide the implementation details of the proposed MaskTAS. Then present the performance of MaskTAS with comparisons to other state-of-the models designed manually or automatically. Finally, we conduct ablation studies to further evaluate the effectiveness of MaskTAS.
3.1 EXPERIMENTAL SETUP
Dataset Description. We evaluate the effectiveness of the proposed method over the large-scale
ImageNet Russakovsky et al. (2015) dataset, which contains 1.28 million images in 1000 categories collected for the image classification task. We also experiment on other classification tasks, including
CIFAR- 10 Krizhevsky al. (2009), CIFAR-100 Krizhevsky al. (2009), PETS Parkhi et al. (2012) and Flowers Nillsbck & Zisserman (2008), to evaluate the transferability of the self-supervised architecture search method. More specifically, the CIFAR-10 dataset consists of 60000 images in 10 classes, with 6000 images per class. The CIFAR-100 dataset has 100 classes containing 600 images each. PETS is a 37 category pet dataset with roughly 200 images for each class. The Flowers dataset consists of 102 flower categories, where each class consists of between 40 and 258 images. Moreover, the transferability of the searched architectures is verified by ADE20K Zhou al. (2019) semantic segmentation task. The ADE20K dataset contains more than 20K scene-centric images annotated with pixel-level objects and object parts labels. The transferability experiment results are presented in Appendix A
Implementation Details. We experiment with the standard ViT supernet architecture in Tiny, Small and Base setting, i.e., supernet-tiny, supernet-small and supernet-base. The decoders of the teacher and student are both implemented as a lightweight network with transformer blocks. The projection network is implemented as two-layer MLP network. The decoder networks are only used during supernet training process. After training, only the encoder is used to generate the image representation for architecture search. During supernet training, each supernet is pre-trained under
7

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparison between MaskTAS and other state-of-the-art methods ImageNet dataset, which includes three groups of models with respect to different parameter sizes, i.e., tiny, small and base. Our results are highlighted in bold.
Models
Tup- Acc.
Top-5 Acc.
Parrameters
FLOPs
Resolution
Model Type
Design Type
Vo Howard al. 2019)
75.2%
5.1
0.22 G
221*
CNN
Auto
EfficieNNat-BR RO Tan Te 2919)
77.1%
93.3%
5.4 M
0.39
224
CNN
Touuvro
72.2%
5.7
1.2G
224
Transformer
Manual
AutoFormer- tiy
2021b
74.7%
92.6%
5.7
L.3G
224"
Transformer
Auto
ViTAS-Twins-T
S0 ct
79.4%
94.8%
1
14G
224"
Transformer
Auto
MaskTAS-diny
75.6%
98.1%
5.8
1.3G
224²
Transformer
Auto
ResNet50(He at. 2016
79.1%
M
4.1
CNN
Manual
RogNerY-KGF Kadosavovic ot 2020
80.0%
21:4M
4.0G
224
CNN
Auto
ElficietNet- B4 Jan Le
2019)
82.9%
95.7%
19.3 M
4.2G
380
CNN
Auto
BoTNettSSI-55 Srimisas al.
81.7%
95.8%
33.5 M
7.3G
224
CNN Trans
Manual
12T-VIT-14 Yuan al
2021
81.7%
21.5 M
6.1
221
Transformer
Manual
Deil-S Touvron et 22021
79.9%
95.0%
22.1 M
4.7G
224
Transformer
Manual
VIT-S/16 (TTS//6/Dosorie Dosovitskly et
2020
78.8%
22. M
4.7G
3842
Transformer
Manual
AutoFomer--mal
C Chen
202
81.7%
96.7%
22.9 M
5.1.G
224
Transformer
Auto
ViTAS-Twins Twins
Su
2002
82.0%
95.7%
30.5 M
3.0
224
Transformer
Auto
MaskTAS-small
82.5%
95.9%
22.1
4.9.G
221
Transformer
Auto
ResNet152 He all (2016)
80.8%
60 M
11G
224
CNN
Manual
EfficietNet B7 Tan & Le
2019
84.3%
97.0%
66 M
37 G
600
CNN
Auto
VIT-B/16 Dosovitskyy şt
2020
79.7%
86 M
18G
3812
Transformer
Manual
DeiT.B Touvron
81.8%
95.6%
224
Transformer
Manual
AutoFormer hase
202
82.4%
95.7%
224
Transformer
Auto
ViLAS-TWins-B
53.5%
96.5%
124.8 M
16.16
224
Transformer
Auto
MusKTAS-buss
83.8%
96.4%
53.7M M
11G
224"
Transformer
Auto
100-epoch schedule on ImageNet-IK training set. To avoid the high overhead of teacher pre-training, we directly employ the MIM pre-trained models released from the official MAE implementations as our teacher model. We adopt a cosine decay schedule with a warm-up for 20 epochs. We adopt Adam optimizer with weight decay of 0.05. The size of input image is set to 224x 224 and the masking ratio is set to 90% by default. During architecture search, we employ the ImageNet validation set for model testing. We perform evolutionary search for 20 epochs to get the optimal architecture, where the population size Np is set to 50. For model fine-tuning, we also use Adam optimizer with weight decay of 0.05. We fine-tune the searched architecture for 100 epochs with batch size of 2048, learning rate of 5e-3, and a drop path rate of 0.1. For ADE20K semantic segmentation, we follow the same settings MAE and adopt UperNet as our framework.
3.2 MAIN RESULTS ON IMAGENET
We first conduct experiments on the widely used ImageNet- 1K dataset. The self-supervised ar- chitecture search process is performed over each supernet configuration. including supernet-tiny. supernet-small and supernet-base. For each configuration, we compare MaskTAS with both super- vised counterparts and handcrafted architectures. As shown in Table D MaskTAS model families can achieve 75.6%, 82.5% and 83.8% top-1 accuracy, significantly outperforming the handcrafted CNN and transformer architectures, e.g., ResNet, ViT and DeiT. This phenomenon indicates the effectiveness of automatic neural architecture search in designing superior architectures. In addition observe from Table T MaskTAS can consistently out-
to the handcrafted architectures, perform the supervised transformer architecture search counterpart AutoFormer. This indicates the effectiveness of masked image modeling in supe-suee architecture search. When pre-trained for 100 epochs, and 0.7% on top-1 and t accuracy respectively. When compared with another TAS method VITAS-Twing, our method can achieve comparable or better results without the need for data labelling in most cases. Moreover, both the parameter size and FLOPs MaskTAS are much less than ViTAS- Twins. The efficiency and effectiveness of MaskTAS mainly lie in two facts: 1) the MIM-based pixel reconstruction objective helps each subnet architecture to learn better local image features; 2) the distillation objective facilitates the efficient training of diverse subnet architectures.
outperforms AutoFormer- base pre-trained for 800 epochs by 1.4%
8

---

Published as a conference paper at ICLR 2024
3.3 ABLATION STUDIES
In this subsection, we conduct ablation studies to explore the effect of masking ratio and compare the training efficiency between MaskTAS and AutoFormer.
Effect of Masking Ratio. The architecture training and searching are both performed in MIM-based self- supervised paradigm, which learns to reason about the missing patches masked by specific ratio. In order to study the effect of masking ratio on performance, we present the top-1 accuracy of MaskTAS small with spect to different masking ratios in Figure 3 Recall that the best masking ratio of the original MAE method can be surprisingly high as 75%. From Figure 3 we can observe that the proposed MaskTAS can maintain stable perfor- mance over wide range of masking ratios, thus allowing for much higher ratio than MAE. In particular, compared with the 75% masking ratio of MAE, it's interesting to note that MaskTAS can raise the masking ratio to 90% without performance drop. We believe this is because the integration of knowledge distillation strategy into the masked image modelling process. With the assistance of distilled knowledge from the teacher model, the student model can make better of visible patches, even very limited amount.
Figure 3: Top-1 accuracy performance MaskTAS-smal with respect to differ- ent masking ratios.
Comparison Supernet Training. To demonstrate the training efficiency of the proposed method, we compare the supernet training process between AutoFormer and MaskTAS in Figure As shown in Figure (a) even with the supervision of labelled data, the supernet training loss of AutoFormer converges slowly, which can't 't reach convergence
after 500 epochs of training. This is because the supernet contains various candidates that need to be fully optimized. In contrast, MaskTAS performs self-supervised architecture search through Masked image modeling (MIM). MIM has shown great promise for self-supervised learning yet been criticized for learning inefficiency, which further increases the difficulty of supernet training. Even with this challenge. the proposed MaskTAS can still converge to a stable value in only 100 epochs as indicated in 4b) which significantly exceeds AutoFormer in training efficiency. We believe this is benefiting from the knowledge distillation strategy, which can provide strong supervision to allow for efficient supernet training.
(a) AutoFormer
(b) MaskTAS
Figure 4: Comparison of supernet training between AutoFormer and MaskTAS.
4 CONCLUSION
In conclusion, this paper has proposed a novel self-supervised neural architecture search method, MaskTAS, that uses masked image modelling search for efficient transformer architectures. By eliminating the need for data labeling, MaskTAS greatly reduces the expensive costs of supervised learning and enables efficient training transformer supernet. The siamesed teacher-student archi- tecture and the unsupervised evaluation metric based evolutionary search algorithm further enhance the learning efficiency and accuracy the search process. The experimental results on CIFAR- 10, CIFAR-100, and ImageNet datasets demonstrate that MaskTAS can achieve state-of-the-art accuracy without using manual labels. Furthermore, MaskTAS can generalize well to various data domains by searching specialized transformer architectures in sulf-vupee manner. The proposed method has great potential in automating the neural architecture design process for real-world applications, saving tremendous labor from human experts, and reducing the costs of data labeling.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Science and Technology Major Project (No. 20222D0117103,, National Nature Science Foundation of China (No. 62302384, No. 62172326, No. 62192781, No. 62272374, and No. 62137002), the MOE Innovation Research Team No. IRT17R86, China University Innovation Fund No. 2021FNA04003, China Postdoctoral Science Foundation under Grant 2023M742790, Fundamental Research Funds for the Central Universities under Grant xpt0 2023022, and the Project of China Knowledge Centre for Engineering Science and Technology.
REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pp. 456-473 Springer, 2022.
Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12-21, 2021a.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference computer vision, pp. 12270-12280, 2021b.
Minghao Chen, Kan Wu, Bolin Ni. Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, and Haibin Ling. Searching the search space of vision transformer. Advances in Neural Information Processing Systems, 34:8714-8726 2021c.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355-9366 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research,
2019.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena
Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 1777712111 1-22284, 2020.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 544-560. Springer, 2020.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34:15908-15919, 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253) 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang. Xiaodan Liang, and Xiaojun Chang. Bossnas: Exploring hybrid enn- canttransformer with block-wisely self-supe neural architecture search. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 12281-12291, 2021.
Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary Yen, and Kay Chen Tan. survey on evolutionary neural architecture search. IEEE transactions on neural networks and learning systems, 20
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021b.
Bolin Ni, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Nasformer: Neural architecture search for vision transformer. In Pattern Recognition; 6th Asian Conference, ACPR 2021, Jeju Island, South Korea, November 9-12, 2021, Revised Selected Papers, Part 1, pp. 47-61. Springer, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3498-3505, 2012.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pp. 10428-10436, 2020.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang. Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 14(4):1-34, 2021.
Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. Tinymim: An empirical study of distilling mim pre-trained models. arXiv preprint arXiv:2301.01296, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-25 1-222, 2015.
Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16519-16529, 2021.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7262-7272, 2021.
Xiu Su, Shan You, Jiyang Xie. Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xi- aogang Wang, and Chang Xu. Vitas: Vision transformer architecture search. arXiv preprint arXiv:2106. 13700, 2021.
11

---

Published as a conference paper at ICLR 2024
Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vitas: Vision transformer architecture search. In Computer Vision-Eo ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXI, pp. 139-157. Springer, 2022.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pp. 6105 61 14. PMLR, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supe-suee visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033, 2021.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for superviid visual pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14668-14678 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 12077-12090, 2021.
Caixia Yan, Xiaojun Chang, Zhihui Li, Weili Guan, Zongyuan Ge, Lei Zhu, and Qinghua Zheng. Zeronas: Differentiable generative adversarial networks search for zero-shot learning. IEEE transactions on pattern analysis and machine intelligence, 44(12):973399740 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference computer vision, pp. 558-567, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprini arXiv:2010.04159, 2020.
A TRANSFER LEARNING EXPERIMENTS
To further evaluate the searched transformer architectures of our method, we present transfer learning experiments on other popular downstream tasks.
Transfer to Other Classification Tasks In addition to ImageNet- 1k, we also perform architecture search on other classification tasks, including CIFAR-10 Krizhevsky et al. (2009), CIFAR-100 100 Krizhevsky et (2009), PETS Parkhi al. (2012) and Flowers Nilsback & Zisserman (008). Based on the MaskTAS supernet pre-trained on ImageNet, we perform architecture search on each data domain obtain the optimal architecture respectively. Then we inherit the parameter weights of each architecture from the well-optimized supernet, and fine-tune them on the corresponding dataset. The classification results of MaskTAS-small are reported in Table 2 For the purpose of fair comparison, we also present the results achieved by ViT-B and AutoFormer-S pre-trained on ImageNet-Ik simultaneously. can be easily observed from Table 2 gets consistent improvement the four datasets compared with the other two methods. We believe this improvement benefits from the self-supervised architecture search framework. Compared with the supervised counterparts, the searched architectures of MaskTAS not limited by the range and quality of data annotation. Thus, the proposed MaskTAS is capable of producing transformer architectures that better transfer to other data domains.
that the proposed MaskTAS
Results on Semantic Segmentation Task We also evaluate the generalization ability of MaskTAS by transferring the searched architectures to semantic segmentation task. Each searched architecture
12

---

Published as a conference paper at ICLR 2024
Table 2: Transferability comparison different data domains.
CIFAR-10
CIFAR-IU
PETS
Flowers
VIT-B
98.1%
87.1%
93.8%
89.5%
AutoForme-S
98.3%
88.1%
94.1%
97.8%
MaskTAS-small
99.2%
90.3%
94.8%
98. 8%
Table 3: Performance comparison (mloU) on ADE20K semantic segmentation.
Method
Pre-train data
Small
Basc
Naive supervised
INIK w/ labels
47.4%
Twins: PCPVT
INIK w/ lahels
46.2%
47.1%
Twin-SVT
INIK w/ labels
45.9%
47.7%
Twins
labels
MaskTAS
INIK w/o labels
48.3%
51.6%
inherits the weights from the ImageNet--re-trrineed supermet, and then fine-tuned on ADE20K for semantic segmentation. The experimental results of different transformer models in small and base size are reported in Table More specifically, we compare five methods in Table Naive supervised. Twins-PCPVT Chu et al. (202T). Twin-SVT Chu et al. (202T). ViTAS-Twins Su et al. (2022) and the proposed MaskTAS. Naive supervised indicates the supervised pre-training method done from scratch, in which we directly use the reported mloU from He et al. (2022). Wrins-PCPVTTChiee et al. (202T). Twin-SVT Chu et al. (202T) and ViTAS-Twins Su et al. (2022) are the other three supervised counterparts. Without the access of manual labels, the proposed MaskTAS is at disadvantage in classification than the supervised counterparts. Even so, it is noteworthy that MaskTAS can still achieve better performance than the other methods. In particular, the small and base model of MaskTAS can outperform the second best model by 0.4% and 1.4% respectively, showing that the self-supervised pre-training can benefit dense downstream tasks well.
13


Reviews:
[{"review_id": "Sil6n4CZdv", "reviewer": "Reviewer_zgPr", "summary": "This paper incorporates the MIM based self-supervised learning with the NAS framework for vision transformers. It also introduces a siamesed teacher-student architecture for the NAS search space to distill knowledge from pre-trained networks, which shows better convergence. \nIt also develops a novel unsupervised evaluation metric for the evolutionary search algorithm, where teacher networks guide the student branch search. \nExperimental results show competitive results to STOA accuracy supervised and unsupervised model results.", "questions": "1. Could you clarify how does MaskTAS \"which completely avoids the expensive costs of data labeling inherited from supervised learning\"? How does MaskTAS perform classification tasks? Will labels be need for fine-tuning the classification heads?\n2. Is there any evidence to support MaskTAS can \"generalize well to various data domains and tasks by searching specialized transformer architectures in self-supervised manner\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces the self-supervised learning framework into the NAS framework for vision transformers. \n2. This paper also introduces a siamesed teacher-student architecture to facilitate the supernet training convergence. \n2. Experimental results show competitive results to STOA accuracy supervised and unsupervised model results.", "weaknesses": "1. There are some statements in the paper which needs further clarification (see details in the Question section)\n2. It's not clear the impact of the self-supervised learning and teach-student distillation framework individually. Is it possible to add some ablation studies?\n3. It's not clear on the overhead and tradeoff of training another teacher encoder-decoder network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pslrWjeFSZ", "reviewer": "Reviewer_YXCu", "summary": "This paper proposes to combine masked image model with transformer architecture search in a self-supervised setting. The author designed a new metric to measure the searched results from the supernet. The proposed method achieves competitive results on ImageNet.", "questions": "I will raise my rating if the results using only masked image modeling can outperform SOTA methods.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments are comprehensive, including imagenet/cifar/ade20k. \nThe writing is good, the motivation is clear, and the method is sound.", "weaknesses": "The only concern is the unfair comparison in the experiments. It is unclear whether the performance gain is coming from knowledge distillation or training with the reconstruction objective. \n\nTherefore, I suggest the author report the experimental results using only masked image modeling on ImageNet & Cifar & ADE20K, since the baseline that MaskTAS compares did not use KD in their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "35ulULtvML", "reviewer": "Reviewer_42Bt", "summary": "This paper presents a self-supervised NAS method for vision transformers (ViT). Specifically, the proposed method applied masked image modeling as the self-supervised method. The transformer supernet is composed of two models, a fixed-arch teacher model and a student super-network which were optimized simultaneously. The knowledge distillation strategy was used to allow for efficient self-supervised training of the supernet, and an unsupervised evolutionary search algorithm was designed to achieve self-supervised transformer architecture search. The paper is well written and easy to understand. The authors have conducted quite extensive experiments which show that the proposed method achieves good results in general. In particular, the searched architectures can achieve state-of-the-art accuracy on benchmark dataset without using manual labels.", "questions": "Please refer to above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The proposed method designs the first self-supervised NAS technique for vision transformers (ViT). The shining point is that It proposes the first unsupervised evolutionary search algorithm, in order to achieve self-supervised transformer architecture search.\n+ The experiments are quite solid and demonstrate SOTA performance without using manual labels.\n+ The paper is well written. The motivation and problem statement are clear and easy to understand.", "weaknesses": "- I was not able to find the architecture of the final searched model in the paper. I would like to see the entire architecture of searched models, rather than a single number that indicating the final performance. This is particularly important in the NAS literature.\n- Typically, in a NAS paper, the final output is the architecture, which should be trained again from scratch to test the \"true\" performance. In this work, the weights that are distilled during training are preserved and used for fine-tuning in the downstream tasks. It would be better if the authors could provide an explanation on this. \n- Some of the experimental details are missing, for instance, where do all the numbers in Table 1 come from?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6XIA58use8", "reviewer": "Reviewer_nfTy", "summary": "In this paper, the authors propose a masked image modelling (MIM) based self-supervised neural architecture search method specifically designed for vision transformers. The paper avoids the expensive costs of data labeling inherited from supervised learning. The paper can generalize well to various data domains and tasks by searching specialized transformer architectures in self-supervised manner.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is written in a clear and accessible manner, making it easy to comprehend.\n2. The utilization of mask image modeling for the search of efficient Transformer structures significantly mitigates the high costs associated with supervised learning\n3. This paper formulates the configuration of MaskTAS's search space as resembling a teacher-student framework, and the notion of extracting knowledge from pre-trained networks is indeed a noteworthy proposition.", "weaknesses": "1. The authors mentioned that experiments were conducted on CIFAR-10 and CIFAR-100 for assessing the transferability of self-supervised architecture search methods. However, it should be noted that the paper does not provide the corresponding experimental results.\n2. In Table 1, it is evident that the approach introduced in this paper underperforms when compared to the use of a CNN model, across multiple dimensions, including performance, parameter size, and FLOPs. What is the rationale behind the author's decision to employ the Transformer architecture?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)