\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage{cite}
\usepackage{hyperref}

\usepackage{tikz}
\usepackage{comment}
\usepackage{booktabs}
\usepackage{amsmath,amssymb} %
\usepackage{color}
\usepackage{pifont}
\usepackage{subcaption}

\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{makecell}
\usepackage[accsupp]{axessibility}  %
\newcommand{\bfit}[1]{\textbf{\textit{#1}}}
\newcommand{\floor}[1]{\left \lfloor #1 \right \rfloor} 

\newcommand{\xmark}{\ding{55}}%
\newcommand{\cmark}{\ding{51}}%
\newcommand{\samelineand}{\qquad}
\newcommand*\samethanks[1][\value{footnote}]{\footnotemark[#1]}

\makeatletter
\def\@fnsymbol#1{\ensuremath{\ifcase#1\or *\or \dagger\or \ddagger\or
   \mathsection\or \mathparagraph\or \|\or **\or \dagger\dagger
   \or \ddagger\ddagger \else\@ctrerr\fi}}
\makeatother
\begin{document}
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{11}  %

\title{Towards Flexible Inductive Bias via \\ Progressive Reparameterization Scheduling}
\titlerunning{Towards Flexible Inductive Bias via P.R.S.}
\authorrunning{Y. Lee et al.}
\author{Yunsung Lee$^{1}$\thanks{indicates equal contributions} \and
Gyuseong Lee$^{2}$\samethanks \and
Kwangrok Ryoo$^{2}$\samethanks \and \\
Hyojun Go$^{1}$\samethanks \and
Jihye Park$^{2}$\samethanks \and
Seungryong Kim$^{2}$\thanks{indicates corresponding author.}
}

\institute{
$^{1}$Riiid AI Research \qquad \qquad
$^{2}$Korea University
}
\maketitle
\begin{abstract}
There are two \textit{de facto} standard architectures in recent computer vision: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs).
Strong inductive biases of convolutions help the model learn sample effectively, but such strong biases also limit the upper bound of CNNs when sufficient data are available.
On the contrary, ViT is inferior to CNNs for small data but superior for sufficient data.
Recent approaches attempt to combine the strengths of these two architectures. 
However, we show these approaches overlook that the optimal inductive bias also changes according to the target data scale changes by comparing various models' accuracy on subsets of sampled ImageNet at different ratios.
In addition, through Fourier analysis of feature maps, the model's response patterns according to signal frequency changes, we observe which inductive bias is advantageous for each data scale.
The more convolution-like inductive bias is included in the model, the smaller the data scale is required where the ViT-like model outperforms the ResNet performance.
To obtain a model with flexible inductive bias on the data scale, we show reparameterization can interpolate inductive bias between convolution and self-attention.
By adjusting the number of epochs the model stays in the convolution, we show that reparameterization from convolution to self-attention interpolates the Fourier analysis pattern between CNNs and ViTs.
Adapting these findings, we propose Progressive Reparameterization Scheduling (PRS), in which reparameterization adjusts the required amount of convolution-like or self-attention-like inductive bias per layer. 
For small-scale datasets, our PRS performs reparameterization from convolution to self-attention linearly faster at the late stage layer.
PRS outperformed previous studies on the small-scale dataset, e.g., CIFAR-100.

\keywords{Flexible Architecture, Vision Transformer, Convolution, Self-attention, Inductive Bias}
\end{abstract}


\section{Introduction}
\newcommand{\etal}{\textit{et al.}}
Architecture advances have enhanced the performance of various tasks in computer vision by improving backbone networks~\cite{he2016deep, carion2020end, tian2020fcos, he2017mask,tian2020conditional}.
From the success of Transformers in natural language processing~\cite{vaswani2017attention,devlin2019bert,brown2020language}, Vision Transformers (ViTs) show that it can outperform Convolutional Neural Networks (CNNs) and its variants have led to architectural advances~\cite{liu2021swin,touvron2021going,zhou2021deepvit}.
ViTs lack inductive bias such as translation equivariance and locality compared to CNNs.
Therefore, ViTs with sufficient training data can outperform CNNs, but ViTs with small data perform worse than CNNs.
To deal with the data-hungry problem, several works try to inject convolution-like inductive bias into ViTs.
The straightforward approaches use convolutions to aid tokenization of an input image~\cite{xiao2021early,yuan2021incorporating,wu2021cvt, hassani2021escaping} or design the modules~\cite{li2021localvit,zhang2021rest,dai2021coatnet,d2021convit} for improving ViTs with the inductive bias of CNNs.
Other approaches use the local attention mechanisms for introducing locality to ViTs~\cite{liu2021swin,han2021transformer}, which attend to the neighbor elements and improve the local extraction ability of global attention mechanisms.
These approaches can design architectures that leverage the strength of CNNs and ViTs and can alleviate the data-hungry problem at some data scale that their work target. 
However, we show these approaches overlook that the optimal inductive bias also changes according to the target data scale by comparing various models’ accuracy on subsets of sampled ImageNet at different ratios.
If trained on the excessively tiny dataset, recent ViT variants still show lower accuracy than ResNet, and on the full ImageNet scale, all ViT variants outperform ResNet.
Inspired by Park~\etal~\cite{park2022vision}, we perform Fourier analysis on these models to further analyze inductive biases in the architecture.
We observe that ViTs injected convolution-like inductive bias show frequency characteristics between it of ResNet and ViT.
In this experiment, the more convolution-like inductive bias is included, the smaller the data scale is required where the model outperforms the ResNet performance.
Specifically, their frequency characteristics tend to serve as the high-pass filter in early layers and as more low-pass filter closer to the last layer.
Nevertheless, such a fixed architecture in previous approaches has a fixed inductive bias between CNNs and ViTs, making it difficult to design an architecture that performs well on various data scales.
Therefore, each time a new target dataset is given, the optimal inductive bias required changes, so each time the model's architectural design needs to be renewed.
For example, a CNN-like architecture should be used for small-scale dataset such as CIFAR~\cite{krizhevsky2009learning}, and a ViT-like architecture should be designed for large-scale datasets such as JFT~\cite{sun2017revisiting}.
Also, this design process requires multiple training for tuning the inductive bias of the model, which is time-consuming.
In this paper, we confirm the possibility of reparameterization technique~\cite{cordonnier2019relationship,li2021can} from convolution to self-attention towards flexible inductive bias between convolution and self-attention during a single training trial.
The reparameterization technique can change the learned convolution layer to self-attention, which identically operates like learned convolution.
Performing Fourier analysis, we show that reparameterization can interpolate the inductive biases between convolution and self-attention by adjusting the moment of reparameterization during training.
We observe that more training with convolutions than with self-attention makes the model have a similar frequency characteristic to CNN and vice versa. 
This observation shows that adjusting the schedule of reparameterization can interpolate between the inductive bias of CNNs and ViTs.
From these observations, we propose the Progressive Reparameterization Scheduling (PRS). 
PRS is to sequentially reparameterize from the last layer to the first layer. 
Layers closer to the last layers are more trained with self-attention than convolution, making them closer to self-attention.
Therefore, we can make the model have a suitable inductive bias for small-scale data with our schedule.
We validate the effectiveness of PRS with experiments on the CIFAR-100 dataset.

\vspace{5pt}

Our contributions are summarized as follows:

\vspace{-5pt}
\begin{itemize}
  \item We observe that architecture with a more convolutional inductive bias in the early stage layers is advantageous on a small data scale. However, if the data scale is large, it is advantageous to have a self-attentional inductive bias.
  \item We show that adjusting the remaining period as convolution before reparameterization can interpolate the inductive bias between convolution and self-attention.
  \item Based on observations of favorable conditions in small-scale datasets, we propose the Progressive Reparameterization Scheduling (PRS) which sequentially changes convolution to self-attention from the last layer to the first layer.  PRS outperformed previous approaches on the small-scale dataset, e.g., CIFAR-100.
\end{itemize}
\section{Related Work}
\input{table/related_table}
\subsection{Convolution Neural Networks}
CNNs, the most representative models in computer vision, have evolved over decades from LeNeT~\cite{lecun1998gradient} to ResNet~\cite{he2016deep} in a way that is faster and more accurate.
CNNs can effectively capture low-level features of images through inductive biases which are locality and translation invariance. However, CNNs have a weakness in capturing global information due to their limited receptive field.

\subsection{Vision Transformers}
Despite the great success of vision transformer~\cite{dosovitskiy2020image} in computer vision, ViT has several fatal limitations that it requires high cost and is difficult to extract the low-level features which contain fundamental structures, so that it shows inferior performance than CNNs in small data scales.
There are several attempts to overcome the limitations of ViT and improve its performance by injecting a convolution inductive bias into the Transformer.

DeiT~\cite{touvron2021training} allows ViT to take the knowledge of convolution through distillation token.
They can converge a model, which fails in ViT. 
On the other hand, The straightforward approaches~\cite{yuan2021incorporating,li2021localvit,chu2021conditional,zhang2021rest} employ inductive bias to augment ViT by adding depthwise convolution to the FFN of the Transformer. 
ConViT~\cite{d2021convit} presents a new form of self-attention(SA) called Gated positional self-attention (GPSA) that can be initialized as a convolution layer.
After being initialized as convolution only at the start of learning, ConViT learns only in the form of self-attention. Thus, it does not give sufficient inductive bias on small resources. 
Swin Transformer~\cite{liu2021swin} imposes a bias for the locality to ViT in a way that limits the receptive field by local attention mechanisms. A brief comparison of these methods is shown in Table~\ref{table:method-comparison}. 

\subsection{Vision Transformers and Convolutions}
 There have been several studies analyzing the difference between CNNs and ViTs~\cite{park2022vision,raghu2021vision}. 
Park~\etal~\cite{park2022vision} and Raghu~\etal~\cite{raghu2021vision} prove that CNN and Transformer extract entirely different visual representations. In particular, Park~\etal~\cite{park2022vision}  present the several analysis of self-attention and  convolution that self-attention acts as a low-pass filter while convolution acts as a high pass filter.
Furthermore, several approaches~\cite{cordonnier2019relationship,d2021transformed,li2021can} have reparameterized convolution to self-attention by proving that their operations can be substituted for each other. Cordonnier~\etal ~\cite{cordonnier2019relationship} 
demonstrates that self-attention and convolution can have the same operation when relative positional encoding and the particular settings are applied.
T-CNN~\cite{d2021transformed} presents the model using GPSA proposed by ConViT, which reparameterizes convolution layer as GPSA layers. C-MHSA~\cite{li2021can} prove that reparameterization between two models is also possible even when the input was patch unit, and propose a two-phase training model, which initializes ViT from a well-trained CNN utilizing the construction in above theoretical proof.

\section{Preliminaries}
 

Here, we recall the mathematical definitions of multi-head self-attention and convolution to help understand the next section.
Then, we briefly introduce the background of reparameterization from convolution layer to self-attention layer. We follow the notation in~\cite{cordonnier2019relationship}. 
\subsubsection{convolution layer}

The convolution layer has locality and translation equivariance characteristics, which are useful inductive biases in many vision tasks. Those inductive biases are encoded in the model through parameter sharing and local information aggregation. Thanks to the inductive biases, better performance can be obtained with a low data regime compared to a transformer that has a global receptive field. The output of the convolution layer can be roughly formulated as follows:
\begin{equation}\label{eq:conv}
\mathrm{Conv}(\bfit{X}) = \sum_{\Delta}\bfit{X}\bfit{W}^C,
\end{equation}
where $\bfit{X}\in\mathbb{R}^{H\times W \times C}$ is an image tensor, $H$,$W$,$C$ is the image height, width and channel, $\bfit{W}^C$ is convolution filter weight and the set 
\begin{equation}
\Delta = \bigg[-\floor{\frac{K}{2}},\cdot\cdot\cdot\;,\floor{\frac{K}{2}}\bigg] \times \bigg[-\floor{\frac{K}{2}},\cdot\cdot\cdot\;,\floor{\frac{K}{2}}\bigg]
\end{equation}
is the receptive field with $K\times K$ kernel.

\subsubsection{Multi-head Self-Attention Mechanism}

Multi-head self-attention(MHSA) mechanism~\cite{vaswani2017attention} trains the model to find semantic meaning by finding associations among a total of $N$ elements using query $\bfit{Q}\in\mathbb{R}^{N\times d_{H}}$, key $\bfit{K}\in\mathbb{R}^{N\times d_{H}}$, and value $\bfit{V}\in\mathbb{R}^{N\times d_{H}}$, where $d_{H}$ is the size of each head. After embedding the sequence $\textbf{\textit{X}} \in \mathbb{R}^{N \times d}$ as a query and key using $\bfit{W}^Q\in\mathbb{R}^{d\times d_H}$ and $\bfit{W}^K\in\mathbb{R}^{d\times d_H}$, an attention score $\textbf{\textit{A}}\in\mathbb{R}^{N\times N}$ can be obtained by applying softmax to the value obtained by inner producting $\textit{\textbf{Q}}$ and $\textit{\textbf{K}}$, where $d$ is the size of an input token. Self-attention(SA) is obtained through matrix multiplication of $\bfit{V}$ embedded by $\bfit{W}^V\in\mathbb{R}^{N\times d_{H}}$ and $\bfit{A}$:

\begin{equation}\label{eq:SA}
\begin{split}
    \mathrm{SA}(\bfit{X}) = \bfit{A}(\bfit{XW}^Q,\bfit{XW}^K)\bfit{XW}^V,\\
    \textbf{\textit{A}}(\textbf{Q},\textbf{K}) = \mathrm{softmax} \left( \frac{\bfit{QK}^\top}{\sqrt{d}}+\textbf{\textit{B}} \right),
\end{split}
\end{equation}
where \textit{\textbf{B}} is a relative position suggested in~\cite{dai2019transformer}. By properly setting the relative positional embedding $\bfit{B}$, we can force the query pixel to focus on only one key pixel.
MHSA allows the model to attend information from different representation subspaces by performing an attention function in parallel using multiple heads. MHSA with a total of $N_H$ heads can be formulated as follows:

\begin{equation}\label{eq:mhsa}
\mathrm{MHSA}(\bfit{X})=\sum_{k=1}^{N_{H}}{\mathrm{SA}}_k(\bfit{X})\bfit{W}^O_k,
\end{equation}
where $\bfit{W}^O$ is learnable projection and $k$ is the index of the head.
\subsubsection{Reparameterizing MHSA into Convolution Layer}
~\cite{li2021can} showed that $K\times K$ kernels can be performed through $K^2$ heads, where $K$ is the size of the kernel. Since the convolution layer is agnostic to the context of the input, it is necessary to set $\bfit{W}^Q$ and $\bfit{W}^K$ as $\textbf{0}$ to convert the convolution to MHSA. Using equations~(\ref{eq:SA}) and~(\ref{eq:mhsa}) together, MHSA can be formulated as follows:
\begin{equation}
\mathrm{MHSA}(\bfit{X}) = \sum_{k=1}^{N_{H}} \bfit{A}_k\bfit{X}\bfit{W}^V
_k\bfit{W}^O_k.
\end{equation}
As $\bfit{A}_k\bfit{X}$ is used to select the desired pixel, the knowledge of the convolution layer can be completely transferred to the MHSA by setting $\bfit{W}^V$ to $\bfit{I}$ and initializing $\bfit{W}^O$ to $\bfit{W}^C$.
\section{Inductive Bias Analysis of Various Architectures}\label{sec:FourierMain}
\input{table/main_imagenet_subset}
In this section, we analyze various architectures through Fourier analysis and accuracy tendency according to data scale.
Previous works designing the modules by mixing convolution-like inductive bias to ViTs overlook that a fixed architecture has a fixed inductive bias and optimal inductive bias can change according to data scale.
To confirm it, we conduct experiments that measure the accuracy of various architectures by changing the data scale of ImageNet~\cite{deng2009imagenet}.
In these experiments, we observe that the required data scale for outperforming ResNet is different for each architecture.
Then, we link frequency characteristics of the recent ViT variants and the tendency of their accuracy with data scale by expanding observations of Park~\etal~\cite{park2022vision}.
In \cite{park2022blurs,park2022vision}, they analyze feature maps in
Fourier space and demonstrate that self-attention is a low-pass filter, and convolution is a high-pass filter.
This phenomenon of filtering noise of different frequencies is caused by different inductive biases of self-attention and convolution.
With Fourier analysis of Park~\etal~\cite{park2022vision}, we observe that architecture having more CNN-like frequency characteristics shows CNN-like efficiency and accuracy tendency in the small-scale datasets.
Park~\etal~\cite{park2022vision} conducted Fourier analysis only for ViT and ResNet, but we analyzed several models with various attempts to inject convolutional induction biases into ViT architecture.
In this section, the Fourier characteristics vary for each injected inductive bias, and we can see which model among the ViT variables was more convolution-like or self-attention-like.
Section \ref{sec:Reparam} will show that we can interpolate from these convolution-like Fourier features to self-attention-like Fourier features with reparameterization.
\begin{figure*}
\centering
    \begin{subfigure}{0.49\linewidth}
        \includegraphics[width=\linewidth]{figure/Subset_Tiny.pdf}
        \vspace{-5pt}
        \caption{}
        \label{fig:2a}
    \end{subfigure}
    \begin{subfigure}{0.49\linewidth}
        \includegraphics[width=\linewidth]{figure/Subset_Small.pdf}
        \vspace{-5pt}
        \caption{}
        \label{fig:2b}
    \end{subfigure}
\vspace{-5pt}
\caption{\textbf{Comparisons of accuracy between ResNet and various ViT-like architectures.} Each model is trained on the subsets of imagenet, specifically 1\%, 5\%, 10\%, 50\%, and 100\%. We plot the accuracy difference between ResNet and other architectures with the increasing subset ratio. The numbers in parentheses mean the number of parameters of each model.}\vspace{-20pt}
\label{fig:imagenet_subset}
\end{figure*}
\subsection{Our Hypothesis}
We hypothesize that 1) the more convolution-like inductive bias is included, the smaller the data scale is required where the ViT-like model outperforms CNNs, and 2) frequency characteristics can explain whether the inductive bias of model is closer to CNNs or ViTs.
Specifically, the incapacity to which the layer amplifies the high-frequency signal tends to dramatically increase from the first layer to the last layer in CNN, whereas ViT does not increase well.
ViTs injected with the inductive bias of convolutions tend to increase it, but not as drastic as CNN.
Here, we observe that ViTs increasing this incapacity more dramatically perform well on smaller scale data like CNNs.
\subsection{Data Scale Experiment}\label{sec:data_exp}
CNNs have inductive biases such as locality and translation invariance and ViTs do not.
Because of the difference in inductive bias that architecture has, the data scale determines their superiority. 
In small-scale data, CNNs outperform ViTs, and at some point, ViTs outperform CNNs as the data scale grows.
ViT variants injected with the convolution-like inductive bias have stronger inductive bias compared to na\"ive ViT, and the amount of data required to outperform ResNet will be less than it.
In this subsection, we identify accuracy trends and the amount of data required to outperform ResNet for various architectures by changing the data scale.
As shown in Table~\ref{tab:main_imagenet} and Figure~\ref{fig:imagenet_subset}, we make subsets with the ratio of 0.01, 0.05, 0.1, and 0.5 respectively in ImageNet for experiments in various settings with the same data distribution and different data scales. 
By utilizing the taxonomy of vision transformer proposed in ~\cite{liu2021survey}, We choose the representatives in each category as ViT variants to compare together. ResT~\cite{zhang2021rest} injects inductive bias directly by adding convolution layers, whereas Swin~\cite{liu2021swin} and ConViT~\cite{d2021convit} add locality in a new way. Swin uses a method that constrains global attention, while ConViT proposes a new self-attention layer that can act as a convolution layer in the initial stage of training.
Therefore, we select ResNet-18 and ResNet-50 as the basic architecture of CNN,  DeiT-Ti as Vanilla ViT and ResT-Light, ConViT-Ti, and Swin-T as the variations of the ViT to be tested. Since the number of parameters also significantly affects the performance, we compare the tiny version of Swin (Swin-T)~\cite{liu2021swin} with ResNet-50~\cite{he2016deep} and the remaining ViT variants with ResNet-18~\cite{he2016deep}. Swin-T has more parameters than other models since the dimension is doubled every time it passes through one layer. 

At 0.01, the smallest data scale, the ResNet series consisting of only CNNs shows better performance, and between them, ResNet-18 with smaller parameters has the highest accuracy. However, as the data scale increase, the accuracy of other ViT models increase more rapidly than ResNet. In particular, ResTv1-Light~\cite{zhang2021rest} and Swin-T~\cite{liu2021swin}, which have hierarchical structures, show superior performance among ViT variants and ResTv1-Light even records the highest accuracy of all models when the data scale is 0.05 or more.

As illustrated in Figure~\ref{fig:imagenet_subset}, DeiT-Ti~\cite{touvron2021training} shows better performance than ResNet when the data scale is close to 1, while ConViT-Ti~\cite{d2021convit} and Swin-T~\cite{liu2021swin} outperform it at 0.5 or more. meanwhile, the accuracy of ResT is higher than ResNet-18 from quite a small data scale of 0.05. Therefore, we argue that the inductive bias is strong in the order of ResTv1-Light, Swin-T, ConViT-Ti, and DeiT-Ti. Through these experiments, we can prove that inductive bias and hierarchical structure have a great influence on accuracy improvement.
\begin{figure*}
\centering
    \includegraphics[width=\linewidth]{figure/Figure_fourier.pdf}
    \caption{\textbf{Frequency characteristics of ViTs and ResNet.} In ResNet-50, ResTv1-Lite, and Swin-T, the difference in log amplitude sharply increases as the normalized depth increase. On the other side, DeiT and ConViT which softly inject inductive biases into models do not have this tendency.} \vspace{-20pt}
    \label{fig:fourer_analysis}
\end{figure*}

\subsection{Fourier Analysis}\label{sec:fourier}

As shown in Section~\ref{sec:data_exp}, the required data scale for outperforming ResNet is different for each architecture.
Inspired by the analysis of Park~\etal~\cite{park2022vision}, we show that the architectures with frequency characteristics more similar to ResNet tend to outperform ResNet at smaller data scales through Fourier analysis.
As in~\cite{park2022vision, park2022blurs}, the feature maps of each layer can be converted to a two-dimensional frequency domain with Fourier transform.
Transformed feature maps can be represented on normalized frequency, which frequency is normalized to $[-\pi,\pi]$.
The high-frequency components are represented at $-\pi$ and $\pi$ and the lowest frequency components are represented at $0$.
Then, we use the difference in log amplitude to report the amplitude ratio of high-frequency to low-frequency components.
For better visualization, differences in log amplitude between $0$ and $1/3\pi$, $0$ and $2/3\pi$, and $0$ and $\pi$ are used to capture the overall frequency characteristics well.
Figure~\ref{fig:fourer_analysis} shows frequency characteristics through Fourier analysis.
In the ResNet results, the difference in log amplitude sharply increases as the normalized depth increases.
This shows that early layers tend to amplify the high-frequency signal, and the tendency to amplify the high-frequency signal decreases sharply as closer to the last layer.
However, DeiT and ConViT which softly inject inductive biases into models do not have this tendency and their frequency characteristics are similar through the layers.
The results of Swin and ResT that strongly inject inductive biases into models with the local attention mechanism or convolution illustrate that the increase of the difference in log amplitude shows an intermediate level between it ResNet and DeiT.
By combining the results of Figure~\ref{fig:fourer_analysis} and Table~\ref{tab:main_imagenet}, we can see that the model performs well for small-scale data if the increase in the difference in log amplitude through layers is sharp.
It becomes smoother in the order of ResNet, ResT, Swin, ConViT, and DeiT, the accuracy is higher in the low-data regime in this order.
These results are consistent with the observations of previous work that the inductive bias of CNNs helps the model to learn on small-scale data.
From these, we address that the difference in log amplitude through the layers can measure the CNN-like inductive bias of the model.
If it increases sharply similar to CNNs, the model has strong inductive biases and performs well in a low-data regime.


\section{Reparameterization Can Interpolate Inductive Biases}\label{sec:Reparam}

As shown on Section~\ref{sec:FourierMain}, a fixed architecture does not have flexible inductive bias, causing them to have be tuned for each data.
Since modifying the architecture to have a suitable inductive bias for each data is too time-consuming, the method which can flexibly adjust the inductive bias during the training process is needed.
We observe that the model trained more with CNN than self-attention have more CNN-like frequency characteristics through reparameterization.
With these results, we show that reparameterization can interpolate the inductive bias between CNNs and ViT by adjusting the moment of reparameterization during training.
\subsection{Experimental Settings}
Because reparameterization can change convolution to self-attention, we can adjust the ratio of epochs that each layer is trained with convolution and self-attention.
In a $10\%$ subset of the ImageNet data, we adjust this ratio by four settings: model trained with 1) convolution for 300 epochs and self-attention for 0 epochs, 2) convolution for 250 epochs and self-attention for 50 epochs 3) convolution for 150 epochs and self-attention for 150 epochs and 4) convolution for 50 epochs and self-attention for 250 epochs.
We note that the model is more trained with convolution from 1) to 4).
We follow the setting for reparameterization as in CMHSA-3~\cite{li2021can} and Fourier analysis as in Section~\ref{sec:fourier}.
\begin{figure*}[t]
\centering
        \includegraphics[width=\linewidth]{figure/Figure_interpolation.pdf}
\vspace{-5pt}
\caption{\textbf{Visualization of Interpolation.} As the ratio trained with self-attention increases, the difference in log amplitude of early stage layers tends to increase, and the difference in log amplitude of late stage layers tends to decrease. Conv $x$, SA $y$ denotes that the model is trained with convolution for $x$ epochs and self-attention for $y$ epochs.}
\label{fig:interpolation}
\end{figure*}

\subsection{Interpolation of Convolutional Inductive Bias}\label{sec:intconvind}
Figure~\ref{fig:interpolation} shows the results of Fourier analysis according to the ratio of trained epoch with convolution and self-attention.
When comparing 1) to 4), we can see that the degree of increase become smaller from 1) to 4).
As the ratio trained with self-attention increases, the difference in log amplitude of early stage layers tends to increase, and the difference in log amplitude of late stage layers tends to decrease.
These results show that the more training with convolution make the degree of increase sharper.
As we observed in the Section~\ref{sec:fourier}, the more sharply increasing the difference of log amplitude through normalized depth represents that the model have more CNN-like inductive biases.
By combining the results of Figure~\ref{fig:interpolation} and this observation, we can see that the more trained with convolution make the model have more CNN-like inductive biases.

\section{Progressive Reparameterization Scheduling}
\begin{figure}[t]
\centering
\includegraphics[width=0.8\linewidth]{figure/Ours/ConvAttn4.pdf}\\
\vspace{-5pt}
\caption{\textbf{Illustration of PRS.} Conv. is a block with a convolutional layer, and Self Attn. is a block with a self-attention layer. Each block is progressively transformed from a convolution block to a self-attention block as the training progresses.}
\label{fig:main-network}\vspace{-10pt}
\end{figure}
We now propose Progressive Reparameterization Scheduling (PRS) which adjusts the inductive bias of ViT for learning on small-scale data. 
PRS is based on our findings as:
\begin{itemize}
    \item As shown in Section~\ref{sec:FourierMain}, the more convolution-like inductive bias is included, the smaller the data scale is required where the ViT-like model outperforms CNNs. In more detail, we can see that the model performs well for small-scale data if the increase in the difference of log amplitude through layers is sharp.
    \item Furthermore, in the interpolation experiment in Section~\ref{sec:Reparam}, if the layer is trained in a convolution state for longer epochs, the layer has more convolution-like characteristics. If the layer is trained in a self-attention state for longer epochs, the layer has more self-attention-like characteristics. That is, by adjusting the schedule, it is possible to interpolate how much inductive bias the model will have between self-attention and convolution.
\end{itemize}
From these findings, PRS makes the early layer have a small difference in log amplitude as a high-pass filter and the last layer has a large difference in log amplitude as a low-pass filter.
Because convolution and self-attention serve as high-pass filter and low-pass filter respectively as in~Park~\etal~\cite{park2022vision}, PRS wants the rear layer to play the role of self-attention and the front layer to play the role of convolution.
In order to force the rear layers to focus more on the role of self-attention than the front layers, PRS reparameterizes according to linear time scheduling from convolution to self-attention, starting from the rear part. PRS is depicted in Figure~\ref{fig:main-network} and can be expressed as a formula as follows:
\begin{align}
    &\bfit{z}_0 = \mathrm{PE}(\bfit{X}), \\
    &\begin{aligned}
    {\bfit{z}^{'}_{l}} =
        \begin{cases}
        \mathrm{Conv}(\mathrm{LN}(\bfit{z}_{l-1}))+\bfit{z}_{l-1},  & (t \leq T\cdot (1 - \frac{l}{L+1}))\\
        \mathrm{MHSA}(\mathrm{LN}(\bfit{z}_{l-1}))+\bfit{z}_{l-1},  & (t > T\cdot (1 - \frac{l}{L+1}))
        \end{cases}
    \end{aligned} \\
    &\bfit{z}_{l} = \mathrm{MLP}(\mathrm{LN}(\bfit{z}^{'}_{l})) + \bfit{z}^{'}_{l},\\
    &\textbf{y}_{\ } = \mathrm{Linear}(\mathrm{GAP}(\bfit{z}_{L})),
\end{align}
where $\mathrm{PE}(\cdot)$ is the patch embedding function that follows~\cite{li2021can}, $\mathrm{LN}(\cdot)$ is LayerNorm~\cite{ba2016layer}, $\mathrm{GAP}(\cdot)$ is global average pooling layer, $\mathrm{Linear}(\cdot)$ is linear layer, $t$ denotes current epoch at training, $L$ denotes the total number of layers, $l = 1, 2, \cdots, L$ denotes the layer index and $T$ denotes the total number of training epochs, $\textbf{y}$ denotes the output of the model.

\input{table/cifar100}
Table~\ref{tab:cifar100} shows the effectiveness of PRS in CIFAR-100 dataset.
PRS outperforms the baseline with a top-1 accuracy score of +2.37p on the CIFAR-100 dataset, showing that the performance can be boosted by a simple scheduling.
We note that our PRS achieves better 
1023
performance than the previous two-stage reparameterization strategy~\cite{li2021can}.
These results show that PRS can dynamically apply an appropriate inductive bias for each layer. 
Through the successful result of PRS, we conjecture that flexibly inducing inductive bias with reparameterization has the potential for designing the model on various scale data.

 \section{Conclusion}
From the analysis of existing ViT-variant models, we have the following conclusion: the more convolution-like inductive bias is included in the model, the smaller the data scale is required where the ViT-like model outperforms CNNs. Furthermore, we empirically show that reparameterization can interpolate inductive biases between convolution and self-attention by adjusting the moment of reparameterization during training. Through this empirical observation, we propose PRS, Progressive Reparameterization Scheduling, a flexible method that embeds the required amount of inductive bias for each layer. PRS outperforms existing approaches on the small-scale dataset, e.g., CIFAR-100. \vspace{-10pt}
\subsubsection{Limitations and Future Works}
Although linear scheduling is performed in this paper, there is no guarantee that linear scheduling is optimal. Therefore, through subsequent experiments on scheduling, PRS can be improved by changing it to learnable rather than linearly. 
In this paper, we only covered datasets with scales below ImageNet, but we will also proceed with an analysis of larger data scales than ImageNet.
We also find that the hierarchical architectures tend to have more CNNs-like characteristics than the non-hierarchical architectures.
This finding about hierarchy can further improve our inductive bias analysis and PRS.


\bibliographystyle{splncs04}
\bibliography{egbib}
\end{document}


Peer review:
Good paper that shows scientific significance, accept
**Summary** \
This paper investigates the inductive bias in ViT and CNN models. The authors claim that the previous design of injecting convolution-like inductive bias into CNN models ignore that the optimal inductive bias depends on the data scale and fixed inductive bias may not be optimal. Experiments on different data scales of ImageNet illustrate that smaller data scale is needed for ViT to outperform CNN if more convolution-like inductive biases is included. The paper also proves that frequency characteristics can explain whether the inductive bias is closer to convolution or self-attention by conducting Fourier analysis. Then the authors show that the interpolation of inductive bias between CNN and ViT can be realized by adjusting the moment of reparameterization during training. Based on the above findings, a progressively reparameterization scheduling is proposed to make the front layers to act like convolution and the rear layer to act like self-attention. Experiments on CIFAR-100 show the effectiveness of PRS.



**Strengths**
- Investigating the inductive bias injected in CNN and ViT models is crucial and this paper presents a new idea of making the inductive bias flexible, which provides a new research direction.
- The order of the paper and the logic in conducting this research is clear. The authors try to first understand the inductive bias by conducting the 'different data ratio' experiments and the Fourier analysis and then propose the scheduling strategy of reparameterization based on the previous findings.
- The tables, equations and figures used to demonstrate the PRS is clear.
- The paper is generally well-written. 


**Weaknesses**
- The experiments used to verify the effectiveness of PRS is only conducted on CIFAR-100. It would be good if more datasets can be included.
- The legend in Figure 3 is misleadings. The orange line should be Conv50, SA250?

**Overall rating (1-10)** \
7

**Justification of rating** \
The scientific significance of this paper.
Rating: 7: Good paper, accept
Confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct