LipsFormer: Introducing Lipschitz Continuity to Vision Transformers
Authors: Xianbiao Qi
Jianan Wang
Yihao Chen
Yukai Shi
Lei Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Lipschitz Continuity
Vision Transformer
Transformer

Abstract: We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models. In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation, and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability. In LipsFormer, we replace unstable Transformer component modules with Lipschitz continuous counterparts:  CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot-product attention, and weighted residual shortcut. We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer. Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization. As a result,  on the ImageNet 1K dataset, LipsFormer-Tiny training for 100 epochs without learning rate warmup attains a top-1 accuracy of 81.6\% which is higher than Swin Transformer-Tiny training for 300 epochs with warmup. Moreover, LipsFormer-Tiny training for 300 epochs achieves a top-1 accuracy of 83.5\% with 4.7G FLOPs and 24M parameters. 

Published as a conference paper at ICLR 2023
LIPSFORMER: INTRODUCING LIPSCHITZ CONTINUITY TO VISION TRANSFORMERS
Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi & Lei Zhang" International Digital Economy Academy (IDEA), Shenzhen, Guangdong, China. (qi xixxabbb ao, wangj a anan, cheny hao, shiyukai, lei zhang) 8 didea. edu. cn
ABSTRACT
We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models. In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation, and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability. In LipsFormer, we replace unstable Transformer component modules with Lipschitz continuous counterparts: CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot- dot-product attention, and weighted residual shortcut. We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer. Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization. As a result, on the ImageNet IK dataset, Lips Tiny based Swin Transformer training for 300 epochs obtain 82.7% without any learning rate warmup. Moreover, LipsFormer-CSS -eerr--SSinniiiii based on CSwin, training for 300 epochs achieves a top- accuracy of 83.5% with 4.7G FLOPs and 24M parameters.
INTRODUCTION
Transformer 49
has been widely adopted in natural language processing (NLP) 6 40 for its great capability of capturing long-range dependencies with self-attentio. Motivated by its success in NLP, Dosovitskiy el al. [17] introduced Vision Transformer (ViT) general backbone
for computer vision tasks such as image classification 35
53
16,
object detection 9 59]. and
segmentation 112 Nowadays, Transformer [49] remains the dominant architecture for NLP 5
computer vision 58 35
53
[6] and many other AI applications 42
31..
Despite its success, training Transformer remains challenging [33 for practitioners: the training process can be prohibitively unstable, especially at the beginning of training. To address the root cause for training instability, we resort to examining Lipschitz continuity of Transformer components. Intuitively, Lipschitz continuous network is finite in the rate of change and its Lipschitz constant is an useful indicator for training stability. As shown in X 44 Lipschitz properties reveal intriguing behaviours of neural networks, such as robustness and generalization. In this work, we focus on the trainability issue of Transformer architectures by explicitly enforcing Lipschitz continuity at network initialization.
Previous works for overcoming Transformer training instability usually focus on one or a combination of its components which can be divided into four categories: (1) improving normalization [54] 33 51
Xiong et al. [54] has shown that, for a Transformer architecture, Pre-LayerNorm (Pre-LN) is more stable than Post-LayerNNrm (Post-LN). Liu al. (33) identified that Post-LN negatively in influences training stability by amplifying parameter perturbations. They introduced adaptive model initialization (A (Admin) mitigate the amplification effect. Likewise, Wang et al. (51) introduced DeepNorm and depth-specific initialization to stabilize Post-LN, However, even with normalization improvements
"Corresponding author.

---

Published as a conference paper at ICLR 2023
such Admin and DeepNorm, learning rate warmup [20] is still necessity to stabilize training. (2) more stable attention [28] [3|; Kim et al. [28) proved that the standard dot- dot-product attention is not Lipschitz continuous and introduced an alternative L2 attention. (3) re-weighted residual shorteut; Bachlechner et al. K showed that a simple architecture change of gating each residual shortcut with learnable zero-initialized parameter substantially stabilizes training. With ReZero, they were able to train extremely deep Transformers of 120 layers. (4) careful weight initialization; To avoid gradient exploding vanishing at the beginning of training, Zhang et al. [60] proposed fixed-update initialization (Fixup) by rescaling a standard initialization. They also proved that Fixup could enable stable training of residual networks without normalization.
In this paper, we conduct thorough analysis of Transformer architectures and propose a Lipschitz continuous Transformer called LipsFormer. In contrast to previous practical tricks that address training instability, we show that Lipschitz continuity is more essential property to ensure training stability. We focus our investigation on the following Transformer components: LayerNorm, dot- product self-attention, residual shortcut, and weight initialization. For each analyzed module, propose a Lipschitz continuous variant as a new building block for LipsFormer. The final LipsFormer network has upper bound Lipschitz constant initial stages of training. Such Lipschitz guarantee has two implications: 1) we can train LipsFormer without using the common trick of learning rate warmup, yielding a faster convergence and better generalization; Transformer is more unstable at the beginning of training. By ensuring initial network stability, we drastically increases the trainability of Transformer. Note that we could also enforce Lipschitz continuity during the whole training process by simply constraining updates on certain scaling parameters.
Our main contributions can be summarized as follows:
We give a thorough analysis of key Transformer components: LayerNorm, self-attention, residual shortcut, and weight initialization. More importantly, we identify potential instability problems each module brings the training difficulty and propose their Lipschitz continuous counterparts: CenterNorm, scaled cosine similarity attention, scaled residual shortcut, and spectral-based ini- tialization. The proposed Lipschitz continuous modules can serve as drop-in replacements for standard Transformer, such as Swin Transformer (35) and CSwin [16].
We propose a Lipschitz continuous Transformer (LipsFormer) that can be stably trained without the need of carefully tuning the learning rate schedule. We derive theoretical Lipschitz constant upper bounds for both scaled cosine similarity attention and LipsFormer. The derivation pro- vides principled guidance for designing LipsFormer networks. We build LipsPormer-S and HipsFormer-CSwii based on Swin Transformer and CSwin individually.
We validate the efficacy of the LipsFormer on ImageNet classification. We show empirically that LipsFormer can be trained smoothly without learning rate warmup. As a result, on the ImageNet- IK dataset, LipsFormer-Swin Swin-Tinyy training for 300 epochs can obtain top-1 accuracy of 82.7 without any learning rate warmup. Moreover, LipsFormer- Tiny training for 300 epochs achieves top-1 accuracy of 83.5% with 4.7G FLOPs and 24M parameters.
PRELIMINARIES
In this section, we first define Lipschitz continuity and Lipschitz constant and then discuss several Lipschitz properties of a neural network. We use the denominator-layout notation throughout this paper. sequence of N elements denoted as X [x1:.... N T E R NxD, where each vector 2; E IR' D i E {1. N).. Function transformation is parameterized by an associated weight matrix W and an affine transformation denoted as (()) W x. where W E RDxM.
Definition 1. function f(x, W) RR. R M is Lipschitz continuous (L-Lipschitz) under choice of p-norm || lpp in the variable æ if there exists a constant L such that for all (æ), W) and (Iz, W) the domain of
[m W) fa2, W)lp s L| x22 2211p-
where the smallest value of L that satisfies the inequality is called the Lipschitz constant of To emphasize that the Lipschitz constant with respect to æ depends on W and the choice of p. we denote L as (Jp, (W)). A function is generally referred as expansive, non-expansive, and contractive in the variable for (fp
1, Lip,, (f) < and (f=,)((())) < 1, respectively,
2

---

Published as a conference paper at ICLR 2023
exhibiting characteristic differences in the change rate of its output. Contemporary neural networks are rarely Lipschitz continuous under the influence of any constituent non-Lipschitz module. Even network is Lipschitz continuous, calculating its Lipschitz constant exactly is a challenging task [50].
According to Definition I the Lipschitz constant of f(x, W) with respect to æ can be expressed as,
( W)
f W
(fp)(fu(W))
sup
31 ER
x1
22 1lp
Exact computation of the above equation an NP-hard problem. For subsequent analyses, We use y = 2 by default unless specified and suppress to reduce clutter, but our conclusion can be easily extended to other choices p.
Lemma 1. Given W, let [(x,, W) RD R be continuously differentiable function and ) be its Lipschitz constant with respect to æ. According to the mean value theorem, we have the following inequality.
L (W))
[V, will S Lip(fa (W)), Vo E
where [ar, WIll the gradient norm of f(x W) with respect
From Lemmal] T we can see that practical method to compute the Lipschitz constant of continuously differentiable function is compute its maximum gradient norm. To prove a function is Lipschitz, it is sufficient to show that its gradient norm is not bounded. For example. fxx = and f(x) = x² are Lipschitz continuous for x (0, oo), because their gradient can be arbitrarily large as approaches and co, respectively.
Definition 2. Let F(x. I1.,..,... L)) composite function with L transformation functions:
RD + R. be an L-layer neural network defined as
F(x, (W) 1,.... = (J"m) (., w?) W
where (W _ 1,..., L} is the parameter and f is the transformation function of the 1-th layer.
For an affine transformation f (x, W) = W æ. its Lipschitz constant is,
max (W), W x+|p =
if p if p = 0c
Lfp sup
max; Σ, Way
(1)
||
where Fmax (W) is the largest eigenvalue of W.
Many common activation functions such as Sigmoid, Tanh, ReLU, and GELU are I-Lipschitz. See Appendix A.I for simple illustration.
Lemma Given the Lipschitz constant of each transformation function in a network F, the following inequality holds
(ip(Fa((W"1== LJ)) = II Lip(S2
From Lemma 2 the Lipschitz constant of a network is upper bounded by the product of each layer's Lipschitz constant. This multiplicative nature gives insight into understanding why deeper networks usually suffer more severe training instability: if a network's constituent layers are expansive, the upper bound of its Lipschitz constant increases monotonically with its network depth. We refer the interested readers to [18
30 for estimating tighter bounds of deep neural networks.
3 AN ASSUMPTION FOR TRAINING STABILITY
Our design philosophy for LipsFormer is based on the following assumption. Assumption 1. network should satisfy the following Lipsehitz conditions for training stability,
1. [ W) - -f W < Lip(fa æzll.
2. [x Wi) fsx, w2)l < Tip Wi Wall.

---

Published as a conference paper at ICLR 2023
The first inequality focuses the forward process and assumees that stable network should satisfy Lipschitz continuity with respect to its imput æ: a small perturbation of its input should not lead to drastic change of its output. Guaranteeing smoothness is vital for guarding network's generalization ability.
For the second inequality, recall that the forward process of typical neural network propagates computation as a/ (Wi+1) æ', where x' and W are the input and weight matrix of Layer 1+ 1. Since common non-linearities are 1-Lipschitz, drop non-linear activations here for simplicity. To backpropagate the network loss L, we have
ac ax
dc
OC OW
OC T Ox +T
W
x²(
Gradient descent updates network weights according W w W--1 8C As demonstrated ac and
WW ax
above, any value explosion will propagate with the chain derivation: if ac is unbounded, aæ ac
ac aw1+r will consequently be unbounded. Meanwhile, if aw1+1 is not bound, it will largely influence the back-propagation chain the next iteration. This justifies the second inequality for the purpose of training stability.
Intuitively, guaranteeing that network's output does not change too much under small perturbations of either its input or network weights induces more stable training process. In this work, we focus on satisfying the first inequality Assumption T for Transformer architectures.
4 LIPSFORMER
A Lipschitz continuous Transformer (LipsFormer) requires all of its constituent modules to be Lipschitz continuous according to Lemma In this section, we analyze key Transformer components and introduce their Lipschitz continuous counterparts when any Lipschitz continuity is violated.
4.1 LIPSCHITZ CONTINUOUS MODULES
4.1.1 CENTERNORM INSTEAD OF LAYERNORM
LayerNorm 2 is the most widely used normalization method in Transformer. is defined as
1 1 1111) x, 1
(N(x) = 7 ① z + +3 where z Std(y)
and
where æ, y R Std(y) is the standard deviation of the mean-subtracted input y. and @ is an element-wise product. y and /3 are initialized to and 0 respectively. For simplicity, we drop y and from analysis because they can be explicitly constrained within any pre-defined range.
By taking partial derivatives, the Jacobian matrix of z with respect to æ is,
0z Oz By J=(=) Ox
-1-
yy
By Ox
Std(y)
ly
The equation above shows that LayerNorm not Lipschitz continuous because when Std(y) ap- proaches 0, the values in the Jacobian matrix will approach co, causing severe training instability. On the other end, when (y) is very large, training will be hindered by LayerNorm as gradients become extremely small. Also note that backpropagating through LayerNorm is slow due to poor uy M
parallelization when computing the Jacobian matrix, especially for the term /
In practice, we notice that single LayerNorm operation could cause severe training instability without learning rate warmup. The underlying reason that LayerNorm is not Lipschitz continuous and some ill ill-dffied input with zero variance will lead to Jacobian matrix filled with infinity. To stabilize training by enforcing Lipschitz continuity, we introduce CenterNorm as,
CN(x) = σ D-1
D T
x + B
(2)
4

---

Published as a conference paper at ICLR 2023
where D is the dimension of æ. The Jacobian matrix (CN() contains term D-1 (I 111 where D D-1 is a heuristic to avoid the eigenvalue contraction from - 11 is easy to verify that,
D
8x
|| (N - C22)l - Lip(CNa)) - x221,
where Li, D-I for 7 1 and 3 0. As most deep neural networks are dealing with high dimensional data with D > 1, we make simplificatinn that Lip(CNx) is 1-Lipschitz for later discussions. CenterNorm is by design Lipschitz continuous at initialization. To guarantee its Lipschitz continuity through training could simply constraint r and (3 to pre-defined range.
4.1.2 SCALED COSINE SIMILARITY ATTENTION
Self-attention 49 is a key component of Transformer, helping capture long-range relationships within data. In practice, people use multi-head attention to effectively capture such relationships under different contexts. Since multi-head attention is linear combination of multiple single-head attention outputs, for simplicity, we focus our analysis on single-head attention, which is defined as,
Xw (XW*)
XX, Wº, W W softmax
XWV.
VDD
where WA, WKK WV are the projection matrices to transform X into query, key, and value matrices, respectively. Intuitively, every token aggregates information from all the visible tokens by computing weighted sum of the values of the visible tokens according to the similarity between its query and each visible token's 's key. The similarity between the i-th query qe and j-th key ky is denoted as WO (W zz--
Pij X 0;
In 28 Kim al. proved that the standard dot-product self eef-aateetion is not Lipschitz continuous and introduced an alternative L2 self-attention that is Lipschitz continuous. Here we use a scaled cosine similarity attention, which is defined as,
SCSA(X, Wº W WV v. r) = vPV, where P softmax (TQK) )
q1
kj
V1
Q
K
V
AN
ky
VN
where and J predefined or learnable scalars;
Q, K, V are l2 row-normalized:
qi, k,, V; (x. W
(x; W (x. WK)
||x WRI Vix V TW Vilx, I' ;c is smoothing factor guarantee
the validity of cosine similarity computation even when ||x; W = O. For arbitrary pair of rows of Q and K denoted as q, and k,, the cosine similarity on their -normalized vectors proportional to their L2 dot product. The upper bound of SCSA's Lipschitz constant with respect to |l2 and ||. oo the following,
Theorem 1. Single-head scaled cosine similarity attention is Lipschitz continuous, its Lip and Lip, are upper bounded by the following inequalities,
Lip(SCSA)> & / |||WK||
loss Lip(SCSA), & 2N - 1)uTe W ll + 2(N - 1)ure Q 2N 3 W
al + N W | 2N WV
l1a.
Proof of Theorem T can be found in Appendix H For multi-head attention, we heuristically scale 1 where K the number of heads. Please refer to Appendix A.2 for head feature concatenation by K more details.
4.1.3 WEIGHTED RESIDUAL SHORTCUT
Residual block PA is a common component of contemporary neural networks 35 been proven effective in avoiding gradient vanishing, especially when training deep networks. A standard residual shortcut block is defined as,
53 49
It has
RS(æ, W) + f(x, W).
5

---

Published as a conference paper at ICLR 2023
The Lipschitz constant of residual shorteut block with respect to is Lip L = Lip(fe (W)). . For any non-degenerate Lipschitz continuous function f(x. W), its Lipschitz constant is greater than 0, hence residual block is strictly expansive. According to Lemma 2 stacking I. identical residual blocks alone will grow the upper bound of a network's s Lipschitz constant exponentially to Tip (Rp(RR2, (W)) causing substantial vulnerability forward value explosion. One way to mitigate such an instability is to constraint the Lipschitz constant of the residual path to be much smaller than especially at the beginning of training when the network is undergoing fast changes via learning. In this paper, we explicitly multiply the residual path with a scale factor initialized a small value such as 0.1 and 0.2. We define the weighted residual shorteut as,
WRS( W) = x + a @ f(x, W),
where a is a learnable parameter vector with the same dimension as the channel size of æ. It is easy to verify that
WRS( W) WRS( will [ip(WRS, x221:
where Lip(WRS,
++ max( when Lip(f= = 1.
As training progresses, a changes as part of the learning process. We could easily constrain to a pre-defined range to ensure the Lipschitz continuity of a network during the whole training process. Note that re- reigetgh shorteut and residual path has been explored before: in [51 33 the authors redefine a residual block as a @ x f(x,, W) to alleviate the LayerNorm instability; ReZero []] uses similar formulation as Equation to speed up convergence, where is scalar instead of vector. Our formulation is motivated by decreasing the Lipschitz constant of a network, instead of being a practical trick. It provides more principled guidance to network design. For example, when training a very deep network, a smaller a would be justified for the purpose of training stabilization.
4.1.4 SPECTRAL INITIALIZATION FOR CONVOLUTION AND FEED-FORWARD CONNECTION
Both convolution and feed-forward connection compositions of affine transformations. As shown in Equation D affine transformation is Lipschitz continuous, hence by Lemma both convolution and feed-forward connection are Lipschitz continuous.
Note that a careful initialization is important for successfully training a neural network. Many initialization methods have been proposed before such Xavier [19] and Kaiming (23) initialization. Inspired by spectral norm regularization [56]. we introduce a I-Lipschitz initialization called spectral initialization.
W
W
Tmax (W)'
where W Xavier-norm initialized and man (W) is its largest eigenvalue. For affine transformation f(x W..) = Wsi x, its Lipschitz constant satisfies the following inequality,
W x1 W., x21 Lip(fa
-xzll,
where Lip(fa 1 at initialization. We use spectral initialization on all convolutions and feed-forward connections.
4.2 LIPSFORMER
4.2.1
LIPSFORMER BLOCK
We start by introducing the main building block for LipsFormer. shown in Figure each LipsFormer block (LipsBlock) is composed of three sub-modules: convolution blocks (lightweight depth-wise and element-wise convolution), scaled cosine similarity attention, and feed-forward con- nection. CenterNorm operator is optional after each sub-module. In this work we apply CenterNorm after scaled cosine similarity attention and feed-forward connection. Within each residual block, residual path is re-weighted with a learnable and randomly dropped with probability p during training as indicated by dashed lines. For the convolution blocks, we use a 7 x depth-wise and a 1 x element- tosse convolution. Ablation study each component can be found in Sec. 5.3

---

Published as a conference paper at ICLR 2023
Convilett
Depth Conv
LayerNorm
Fesd-Furward
Transfomer
Sall-Attertion
Layerlorm
Feed-Forward
Layerliorm
Swin Transtrrafrrrr
LayerNorm
Self-AAttntion
LayerHoom
Feed- Forward
LipsFormer
Cosine
FIGURE 1: Comparison of a LipsFormer block with ConvNeXt, Transformer and Twin-Transformet Blocks. We use different colors to mark our Lipschitz improvements.
Post-Norm
1i+1 LayerNorm(z; + ((a;))
Pre-Norm
1++1 = æ, + 1 [(LayeNNorm (2:;)
LipsFormer 3 CenterNorm x. + DropPath, (a, (xx)))
TABLE 1: Various forms of residual blocks for Transformer architectures. As illustrated in Figure f represents transformation function E {self-attention, wred-forward). For LipsFormer, / (scaled cosine similarity attention, feed-forward, convolution blocks).
In Table I we compare the LipsFormer residual block with commonly used Post-Norm and Pre-Norm residual blocks. CenterNorm and scaled cosine similarity attention are Lipschitz continuous counter- parts for LayerNorm and dot-product attention. Weighted residual connection and DropPath (29 are used to constrain the Lipschitz constant of deep LipsFormer network.
4.2.2 OVERALL ARCHITECTURE OF LIPSFORMER
In general, LipsFormer follows the architecture Swin Transformer v1. We start by processing an imput image with non-overlapped convolutional token embedding (4 4 convolution with stride 4) to obtain feature representation with resolution H W. Then the main computation passes four stages where each stage consists of a pre-defined number of LipsFormer blocks as shown in Figure Between consecutive stages, we reduce the output resolution by 2 and double the size of output channels by 2 non-overlapped convolution with stride
2 4
We build three variants of LipsFormer in correspondence with CSwin Transformer [16 as detailed in Appendix Table Sa.Forrrr-CSinn- [LipsFormer-CSwin-5 and LipsFormer-CSww
L.ssinn-),..ssFormeeeeee Lipssoomer--CSS CSiiner---Sii---allall- WwwFooome--SSiin----. The number of Lips- Former blocks within the four computation stages are [1, 2, 21, I] for LipsFormer-CSwin-TT [2, 4, 32, The overall architecture of LipsFormer is
CorsFoorme-CSSin--
2] for LipsFormer- S and SipssForme--Swinnn illustrated in 3 of Appendix B We can also build LipsFormer on Swin Transformer, more experiments about Fopssrreee--wn can be found in Appendix.
4.2.3 LIPSCHITZ CONSTANT OF LIPSFORMER
As illustrated in Figure LipsFormer includes four computation stages, each starting with patch merging followed by pre-defined number of LipsFormer blocks. Feed-forward connection, convolu- tion, and patch merging are Lipschitz continuous operators. With spectral initialization these affine transformations are I-Lipschitz at the beginning of training, hence are dropped from analysis. For the Lipschitz constant of the LipsFormer,
have the following theorem.
Theorem 2. For LipsFormer with S stages where the s-th stage has M, residual blocks, when set to De M," the Lipschitz constant of the LipsFormer is upper bounded by exp(R)) where
max (((((()()))) 1,.-..... I...., E. Ms)).
Proof of Theorem 2 is in Appendix Theorem suggests that 1) Deeper networks with more residual blocks should initialize with smaller to avoid exponential growth of its Lipschitz constant; 2) To control the Lipsehitz constant of Lipsformer we should focus on constraining the Lipschitz constant Tip(ft) of each constituent layer, especially the one with the largest Lipschitz constant.
7

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
5.1 DATASET AND TRAINING SETUP
We evaluate LipsFormer-CSwin
the standard ImageNet-1K [3] dataset, which consists of 1.28M images and 1. 1,000 classes. We adopt a similar training strategy as in CSwin Transformer [16] for a fair comparison. Specifically, we use the AdamW (38] optimizer with weight decay 0.05 for LipsFormer- CSwin-T/S and 0.1 for See-ooorm---Siinge with an imput image size of 224 x 224. For LipsFormer-
By default, all our models are trained for 300 epochs
the training batch size 2048 and
the initial learning rate is 0.002 with standard cosine learning rate decay [37] without learning rate warmup [37] We apply stochastic depth (26) for Lips Siieer--Siin-TTT TT Lipssoomer-CSSin-- and LipsFoorme--CSinne with a maximum PropPath rate of 0.2, 0.4, and 0.5, respectively. For ablation study, we train each model for 100 epochs for efficiency. See Appendix C for more details.
5.2 COMPARISON WITH STATE-OF-THE-ART MODELS
Table repoots the MeeFFormrrCSSinn results compared with state-of of-the-art CNN and Transformer models. We evaluate all three variants of Lipssormer-CSSii against state-of of-the-art models of similar sizes: Tiny (< 32M parameters), Small (31- (31-64M parameters), and Base (56-96M parameters).
Method
Param.
HLOPs
Top-1
Method
Param.
FLOPs
Top-1
Method
Param.
PLOPS
Top-1
RogNetY-44
RogNetY
39M
8.0
81.7
RogNerY-16G
BMN
19M
4.2G
K2.9
BB
30M
9.9G
83.6
EffNat-B7
66M
37.0G
84.3
Tom:NeXi-1
30
29M
4.5G
K2.1
CowNNeXI-S
50M
8.7G
83.1
ConvNeXi-B
30
89M
13.4G
83.8
SP-CoTNeD-500
23M
4.10
81.6
SF-CATNNDD-101
22
41M
8.5G
83.2
SF-CATTNeD--12
56M
17.0G
84.0
Dal.:
46
22M
2.43
7538
DE11-
87M
17.30
81.8
T2T-14
530
24M
5.3G
81.5
T2T-19
39M
8.9G
81.9
T2T-24
64M
14.1G
82.3
TNT-T
24M
5.2G
81.3
66M
34.1G
82.8
DecpViT
DeapViT
Swin
C-T-11
20M
R1.6
CVT-21 (30
37M
7.1G
82.5
NesT.T [1]
17M
5.3G
$1.5
NesT.8 61
28M
10.4G
53.3
NesT-B 61
68M
17.9G
83.8
XCII-812 []
26M
4.501
82.0
XCTT-824
48M
9:1G
82.6
XCTI-M24
1
84M
16.2G
82.7
Cross VIT-15 (1]
27M
5.8G
81.5
Crus/VT-188 18 [1]
44M
9.0G
$2.8
RegionViI-I 10
14.3M
2.76
81.5
RegionViT-S 10
JIAM
5.3G
82.6
RegionViT-B
10
72.7M
13.0G
83.2
Focal-T [33
29M
4.00
82.2
Focal-5
13
51M
91G
83.5
Focal-R 33
90M
16.00
83.8
CSwin T
23M
4.3G
82.7
CSwin-S
35M
6.9G
$3.6
CSwin
76M
15.0G
84.2
CrosFormer-S
CrossFormer-B
Croeseemer
WTS DD8T
4.66
81.4
VIE.B (DeiT 110
87M
17.5G
83.8
NAT-T
4.30
NAT-S
51M
7
83.7
NAT-R
90M
13.70
Lipttormer CSwin (ouP)
21M
4.7G
K33
LipeTormer CSwin's (GuES)
38M
7.8G
53.8
Leperorer CSwin 1
83M
16.3G
33.5
TABLE 2: Comparison of different models with input resolution 224 on ImageNet- 1K classification. Red indicates the best result and blue indicates the second best result.
Compared with previous state-of- -h--re-rrttttttt Vision Transformer models, LipsFormer--Cwin attains higher classification accuracy on all its model variants. For instance, LipsFormer-CSwin. obtains 83.5% Top-1 accuracy that outperforms CSwin-T by 0.8%, ViT-S by 2.1 % and NAT-T by 0.3%. LipsFormer-CSwin-TT also outperforms recently improved CNN architectures, such as ConvNeXi-T and Bf by 1.4% and 0.6%, respectively. LipsFormer-CSwin T has fewer parameters than NAT-T and Con-NeXt-T. LipsFormer-CSwin. also outperforms its counterparts, including Swin-B, B, CrossFormer-L, ConvNeXt-B, DeiT-B, ViT-B, and NAT-B with fewer parameters. Also note that all the other Transformer models use learning rate warmup, but LipsFormer-CSSww does not.
5.3 ABLATION STUDY
We conduct extensive ablation study on each key component of LipsFormer-CSwin as shown in Table We use ConsFooree--Swinn for ablation study and all results in this comparison are trained for 100 epochs without learning rate warmup, except for ablation on warmup.
do not learning rate warmup when training LipsFormer- CSwin. Theoretically, warmup is not needed given LipsFormer's is appealing stabilization guarantee. According to the results in Table 3 5 epochs of warmup does not bring in further improvement.
Warmup. In previous experiments
We compare CenterNorm against no-Norm (as in ReZero B and the standard LayerNorm. Results show that: 1) Lipsformer-CS with LayerNorm becomes unstable and does not converge, but with CenterNorm and no-Norm, MarFooorr-CSiiii can successfully converge; 2) Using CenterNorm significantly outperforms no-Norm by 1.3%.
CenterNorm.
8

---

Published as a conference paper at ICLR 2023
Warmup
Tmy
Normalization LayerNorm
Tiny Not converge
Iniiaalizttinn
Tiny 81.3
Atieation
Tmy Divenged
No
81.6
Truncaled Normal
Dot Product
Yes
81.2
No Norm
80.3
Xavier
81.6
1.2 Distance Attn
81.3
CenterNorn
81.6
Spectral
81.6
SCSA
81.6
Alpha
Tiny
Blocks
Tin
Drop Rabo
Tiny
= 0.1
81.4
no Conv
80.7
0.0
81.3
= 0.2
81.6
Conf.
p
0.3
81.3
Conf. C
81.6
0.2
81.6
0.4
Diverged
Conf.
TABLE 3: Ablation study on key components of LipsFormer. "Not converge" means training loss oscillates without converging, and "Diverged" means the loss explodes because of "NaN" or "Inf".
Spectral Initialization. We compare LipsFormer-CSS CSwin results with spectral initialization against truncated normal and Xavier initialization. We find that LipsFormer-CSwin with any of the three ini- tializations converges. Spectral initialization and Xavier initialization slightly outperforms truncated normal initialization, but spectral initialization has a better Lipschitz interpretability than Xavier initialization.
Scaled Cosine Similarity Attention. To validate the effectiveness of scaled cosine similarity attention, compare it with the standard dot-product attention and the L2 distance attention (28). We find that the standard dot- dot-product self- elf-attention leads to forward value explosion, but the scaled cosine similarity attention works well under Lipschitz guarantee. Meanwhile, SCSA works better than the L2 distance attention.
Impact of the Residual Weight As detailed in the weight of residual path a has I Lipschitz constant. We evaluate different
substantial influence on the upper bound of LipsFormer's choices of a, and find that with large initialization value, network either does not converge diverges quickly. This validates that deeper networks need a smaller a.
Convolution Blocks. In LipsFormer-CSwin. we use depth-wise convolutions (dwc) and one point- wise convolution (pwc). We evaluate four different convolution configurations: A) no convolution; B) one dwe; C) dwe + pwe; and D) dwc + pwc dwc. Table 3 shows that one dwe increases LipsFormer's s accuracy by 0.5%, one dwc + one pwc further improves its performance by 0.4%, adding more convolutions saturates performance gains.
DropPath Ratio. In Appendix we show that DropPath effectively decreases the upper bound of network's 's Lipschitz constant, making training process more stable. The results Table S show that reasonable DropPath can effectively improve training performance.
To summarize, CenterNorm, scaled cosine similarity attention, and convolution blocks all contribute positively to LipsFormer-CSSin 's superior performance. Weighted residual shortcut with small a, reasonable DropPath ratio p and spectral initialization are effective stabilizing LipsFormer-CSSw by constraining its Lipschitz constant.
CONCLUSION
In this paper, we present Lipschitz continuous Transformer, called LipsFormer, pursue more stable training process by enforcing the Lipschitz continuity of the whole network. We analyze key components of Transformer and replace the ones violating Lipschitz continuity by introducing CenterNorm, scaled cosine similarity attention, and spectral initialization. LipsFormer also uses weighted residual shortcut and DropPath to further decrease the upper bound its Lipschitz constant. Finally, we derive an upper bound of the Lipschitz constant of LipsFormer network architecture. We empirically validate the effectiveness of LipsFormer-SSwi and SipsFormer--Swie Transformer and CSwin individually, ImageNet IK classification with state-of-the- -he---rrtt performance for model variants of different parameter sizes. The analysis of the Lipschitz continuity of a network is general. We look forward to extending it to broader class of models and application areas, including multi-m modal model and natural language processing. We also hope future works will discuss the Lipschitz continuity of LipsFormer in the backward process in depth.
based on Swin

---

Published as a conference paper at ICLR 2023
REFERENCES
[1] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross- covariance image transformers. Advances in neural information processing systems, 34, 2021.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607:06000004500 2016.
[3] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pages 1352 1361. PMLR, 2021.
[4] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159 2020.
[5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg. Antoine Bossclut, Emma Brunskill, et al. On the opportunitics and risks of foundation models. arXiv preprint arXiv:2108. 07255, 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:18771190,, 2020.
[7] Sébastien Bubeck, Yuanzhi Li, and Dheeraj M Nagaraj. A law of robustness for two-layers neural networks. In Conference Learning Theory, pages 804-820. PMLR, 2021.
[8] Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Advances Neural Information Processing Systems, 34, 2021.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-ena object detection with transformers. In European conference on computer vision, pages 219-229. Springer, 2020.
[10] Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. In International Conference on Learning Representations, 2021.
[11] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi- scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 357-366, 2021.
[12] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing. Mask2former for video instance segmentation. arXiv preprinl arXiv:21 2021.
[13] George Dasoulas, Kevin Scaman, and Aladin Virmaux. Lipschitz normalization for self-attention layers with application to graph neural networks. In International Conference on Machine Learning, pages 2456-2466. PMLR, 2021.
[14] Jared Q Davis, Albert Gu, Krzysztof Choromanski, Tri Dao, Christopher Re, Chelsea Finn, and Percy Liang. Catformer: Designing stable transformers via sensitivity analysis. In International Conference on Machine Learning, pages 2489-2499. PMLR, 2021.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[16] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer. A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2 2021.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2023
[18] Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 2019.
[19] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256 JMLR Workshop and Conference Proceedings, 2010.
[20] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint 17Xx:1700602667, 2017.
[21] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34, 2021.
[22] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. arXiv preprint arXiiv2204.071433 2022.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[26] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7700-4708, 2017.
[27] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[28] Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562-5571 5571. PMLR, 2021.
[29] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint 1rXiv:1605.07666 2016.
[30] Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arXiv preprint arXiv:2004. 00688, 2020.
[31] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. arXiv e-prints, pages arXiv-21 12, 2021.
[32] Yehao Li, Ting Yao, Yingwei Pan, and Tao Mei. Contextual transformer networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[33] Liyun Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747-5763, 2020.
[34] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 12009-12019, 2022.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
11

---

Published as a conference paper at ICLR 2023
[36] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.03555, 2022.
[37] Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[38] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv11711.55001, 2017.
[39] Ilia Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436, 2020.
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 19100:9910.1068 2019.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/22 2022.
[42] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. shro-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831 PMLR, 2021.
[43] Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[44] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
[45] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61
661 14. PMLR, 2019.
[46] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[47] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. arXiv preprint arXiv:2204.071118 2022.
[48] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. Advances neural information processing systems, 32, 2019.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[50] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
[51] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.0055: 22000005555 2022.
[52] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Cross- former: A versatile vision transformer hinging cross-scale attention. arXiv preprint arXiv:2108.000 00154, 2021.
[53] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 22-31, 2021.
12

---

Published as a conference paper at ICLR 2023
[54] Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang. and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference Machine Learning, pages 10524-10533. PMLR, 2020.
[55] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
[56] Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generaliz- ability of deep learning. arXiv preprint arXiv: 705.10941, 2017.
[57] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-toker vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 558-567, 2021.
[58] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for 2021.
computer vision. arXiv preprint
[59] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.0300 03600, 2022.
[60] Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint 1rXiv:1901.09321, 2019.
[61] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Sercan Arik, and Tomas Pfister. Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding. arXiv preprint arXiv:2 105.. /2723, 2022.
[62] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 2021.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 LIPSCHITZ CONSTANT OF COMMON ACTIVATION FUNCTIONS
In Figure plot common non-lineat activation functions in neural networks: Sigmoid, Tanh, ReL.U and GELU. According to [25), GeLU can be approximated by GeLU(() R siigmoid According Lemma the Lipschitz constants of Sigmoid, Tanh, ReLU and GELU are 1, 1.0998 respectively.
3.0
Sigmoid
2.5
ReLU
2.0
1.5
1.0
0.5
0.0
-0.5
FIGURE 2: Sigmoid, Tanh, ReLU and GELU activation function.
A.2 MULTI-HEAD ATTENTION
For K-head attention, have the i-th attention, i E (1,..., K} defined as,
h, W;) Attn, X, W.O. W W.),, where W; the set of projection weight matrices (w; W.K. W.Y) Multi-head attention simply concatenates different attention results, hæx W) (2:, Wi); h2(æ W2); ... hx(x, WK]].
According to the Lipschitz definition, have. [ W) - h((2,, W)ll < (Lip(hi(Wi)) + Lip(ha(Ws)) + + Lip(hx (Wx)))] x2
B NETWORK ARCHITECTURE AND CONFIGURATIONS
The overall architecture of LipsFormer-CSS CSiin shown in Figure For patch embedding and patch merging, non-overlapped convolution in Swin Transformer. Following CSwin Transformer, we use the same cross-shaped window when computing attention results and also the same Locally enhanced Positional Encoding (LePE).
The configurations of Lipsformer- CSwin are based on CSwin Transformer and Table summarizes CerFormer-CSiin-.. and LipsFormer-Co CSinnr--SSii-SSS only varies in
three variants of Meaffoome--SSwin the number of LipsFormer- LipsFormer-CSwin blocks. CysFFomerr-Cwinn. share the same depth configuration but varies in hidden layer channel size.
Similar to LipsFormer-CSWwe CSwin, we also build LipsFormer based on Swin Transformer [35 Here, term it as LipsFormer-Swinn We create five versions of LipsFormer-Swin, and the detailed configurations are shown in Table 5
14

---

Published as a conference paper at ICLR 2023
NWW
Merging
Foreooreer
x Patch
2
Formerr
2 2x22Patch Patcch
Formorrer Block
Marging
LipeFFormt
Pooling & FC
Ice bear
Block
Parch
xtV
X.4
XXN
X.V4
Stage
Stage
Stage
Stage
FIGURE 3: Illustration of the overall LipsFormer architecture.
TABLE 4: Details of HopsFormer-CSwinn model variants.
Model
Channel
Number of Blocks
Num. of Params.
LipsFormer-
64
[1. 2, 21, 1] [2, 4, 32,
24M
LipsFormer-CSwinn.
64
38M
LipsFormer-CSwin-2
96
[2, 4, 32, 2]
83M
C TRAINING DETAILS
In Table f we provide the ImageNet 1K training details used for producing the main results in Table 2 All LipsFormer variants use the same training hyperparameters, except for DropPath ratio, weight decay, learning rate and EMA. All the models are implemented with PyTorch, and trained on NVIDIA Tesla A100 GPUs. We do not use learning rate warmup in all experiments.
D EXPERIMENTS OF LIPSFORMER-SWIN
We evaluate the Tiny, Small, Base and Large versions of LipsFormer-SSin the ImageNet-IKK and compare our results with their corresponding counterpart Swin Transformer. The results are shown in Table
We have the following two findings from Table
the proposed LipsFormer-Swin consistently outperforms its counterpart Swin Transformer. Specifi- cally, LipsFormer-SSin-TT improves Swin-T by 1.5%.
Lipssormer--Singg shows obvious overfitting on ImageNet-IK, and performs Sorror-mme-iinn According to our observation in the training process, the training loss (around 2.2) LipsFormer-Swiin Leenmer-Swii-iiii is much smaller than that (around 2.5) of SopsFoomer--win- but the test accuracy is lower.) We also observe that in some github discussion issues, some people! also find that the original Swin-I. cannot outperform Swin-B if only training on ImageNet-IK.
than
Since LipsFormer-Swiin -win-L of LipsFormer-SSiin-L.. on the table. In the future, we will train it on a larger scale of data to test its fitting ability. On single A100-40GB GPU, with batch size fixed 256 and mixed precision,
has shown overfitting on ImageNet- IK, we do not report the performance
TABLE Details of LipsFormer- model variants.
Model
Channel
Num. of Blocks
Num. of Params.
LipsFormer- Swin-T T
96
[2, 6, 2]
31
LipsFormer-
[2. 18, 2]
54
LipsFormer-Swin-
128
[2, 18,
96
LipsFormer-Swin.
192
12 18,
214
LipsFormer-S Srormer-SSwinL+++
288
2, 18,
526
15

---

Published as a conference paper at ICLR 2023
TABLE 6: Hyperparameters for the models. Le, Las Lo, Le are the number of residual blocks in tiny,
small, base and large.
Hyperparameters
Tiny
Small
Base
Large
Warmup steps
0
Optimizer
AdamW 0.2
AdamW
Adam
AdamW
DropPath ratio
0.4
0.5
0.5
residual as in Eq.
J.
1
L
in cosine attention as Eq. 4.1.2 Learning rate
12
12
12
12
2e-3
2e-3
le-3
le-3
Learning scheduler
cosine
cosine
cosine
cosine
Adam e
1e-8
1e-8
1e-8
1e-8
Adam /
(0.9, 0.999)
(0.9, 0.999)
(0.9, 0.999)
(0.9, 0.99)
Label smoothing
0.1
0.1
0.1
0.1
Randengment
(9, 0.5)
(9, 0.5)
(9, 0.5)
(9, 0.5)
Mixup
0.8
0.8
0.8
0.8
Cutmix
1.0
1.0
1.0
1.0
Training epochs
300
300
300
300
Gradient clipping
0.0
0.0
0.0
0.0
Weight decay
0.05
0.05
0.1
0.1
EMA
0.99984
0.99984
0.99992
0.99992
TABLE 7: Comparison of our LipsFormer-Swin with its corresponding counterpart Swin Transformer with input resolution 224 ImageNet-IK classification.
Method
Param. (M)
FLOPPs (G)
Acc.
Method
Param. (M)
FL.OPs (G)
Acc.
Swin-T
29
4.5
81.3
LipsFormer-Swin-"
31
5.0
82.7
Swin-S
50
8.7
83.0
LipsFormer-Swin-S
54
9.7
83.5
Swin-B
88
15.4
83.5
LipsFormer-Swin-
96
17.0
84.0
Swin-L
190.7
34.5
N/A
LipsFormer-Swin
214
37.7
83.5
the throughput and peak memory are 963 im/s and 5483 MB for LipsFormer-SSin- and 433 im/s and 7415 MB for LipsFormer-Swin.
E OVERFITTING EVALUATION
Following DeiT III [7). also evaluate method on ImageNet-v2 43 and ImageNet-real 4 data sets. As pointed out by [48), to test how the method performs in a nearby setting without any finetuning good way assess overfitting. We directly apply the obtained models trained on the original ImageNet data set onto these two data sets. The results are shown in Table
We can see that from Table our method that works well on the original ImageNet data set consistently performs well on the ImageNet-v2 and ImageNet-real data sets. This observation fully validates the generalization ability of the proposed method.
F TRAINING CURVES
In Figure we show the training curves of the training losses and the classification accuracies of HipsFormer--Sin. and LipsFormer-Swin- We find LipsFormer-Swin. can fit the training data better than LipsFormer- LipsFormer-Swin: Swin-T because the loss of LipsFormer-Swwe Swin-B much lower than that of HipsFormer-Swin-11
16

---

Published as a conference paper at ICLR 2023
TABLE 8: Details of LipsFormer-CSwin model variants, All results except our LipsFormer-CSwin are taken from DeiT III 47
Model
Params (M)
Flops (G)
val
real
V2
ViT-S
22.0
1.6
80.4
86.1
69.7
PIT-S
23.5
2.9
80.4
86.1
69.2
TNT-S
23.8
5.2
87.2
70.6
ConViT-S
27.8
5.8
81.3
87.0
70.3
Swin-S
49.6
8.7
82.1
86.9
70.7
LipsFormer-CSwin-T
24
4.7
83.5
88.0
73.2
VIT-B
86.6
17.6
83.1
87.7
72.6
PÎT-B
73.8
12.5
82.4
86.8
72.0
TNT-B
65.6
14.1
82.9
87.6
72.2
ConViT-B
86.5
17.5
82.0
86.7
71.3
Swin-B
87.8
15.4
82.2
86.7
70.7
CaiT-B12
100.0
87.7
73.3
LipsFormer-CCwin
83
16.3
84.6
88.6
74.5
Training Loss Curve
ImageNet Accuracy by Epoch
6.0
5.5
5.0
60
4.0
3.5
30
3.0
Upsoormer Sw
0K
238
50K
75%
100K
121K
150C
175K
50
100
150
200
253
300
Step
Spoch
FIGURE 4: Training curves of LipsFormer-SSin-TT and LipsFormer-Swine Left: training loss along with epochs. Right: classification accuracy along with epochs.
G PARAMETER VARIATIONS ALONG WITH TRAINING EPOCHS
In Figure 5 show the variations of the along with the training epochs. Our statistic is based on Lips the a. We select model. We statistic the mean and standard variance the absolute value of of from each stage. We find that from Figure the value of the absolute value of the a first grows and then tend to stabilize at value.
17

---

Published as a conference paper at ICLR 2023
Alpha
0.6
StageD
Stagel
Stage2
0.5
Stage3
0.4
Value
0.2
0.1
0.0
Epoch
FIGURE 5: Variation curve of a along with training epochs. Each line denotes one set of a in one stage. We show the mean and standard variance of the absolute value of the a.
H PROOF OF THEOREM
In this subsection, we derive the Lipschitz constant upper bound for the scaled cosine similarity attention (SCSA).
First, we list some useful notations and identities for deriving the Jacobians of attention computation.
X
E NXD.
x N
For column vectors u, E R the chain rule has:
Oz
du
z =
+
0x
8x
0x
The standard dot-produc, attention is defined as,
XWQ (XW*)* XW
DP(X, W W WV) = softmax
VD/H
PXWV
In (28), Kim et proved that the standard dot-product attention is not Lipschitz continuous, and proposed L2-distance attention which is Lipschitz continuous conditioning on W W But enforcing the equality of W and W limits the expressiveness of the Transformer and degrades training performance empirically.
Our scaled cosine similarity attention is defined as,
SCSA(X, wº wK K WV .4.T= =VPV
(6)
where P = softmax (TQKT),
and are predefined or learnable scalars.
The definitions of Q, K, are as follows,
q
kj
v
Q
E ERR NxD
==
ERD ==
E ER NxD
kj
V
18

---

Published as a conference paper at ICLR 2023
For each input x,, the projected qı, k;, are defined as,
x
(x) Wk) Vix; W 1 +
WV) (x;
q
Xx; w kj
V
+
where e is small smoothing factor guarantee that the definition of cosine similarity is valid everywhere.
By taking partial derivatives, we have the following Jacobian matrices,
0q.
WO X,X; T WQ WWQ
Q
(7)
Ox;
V |xx w012 +
X WQ 1- +e
0k,
WK X,X; WK
WKT
K 0xj
(I
(8)
Vixx, T T +c
||x T WK +
WVT xjX, T W
ov Vi 0x,
WWV
(I
(9)
|xx, WVV
||x, 1 +
The attention matrix P is defined as,
Tqi ki
TqI 12
rqi Tkn
Tq2 ki
Tq2
k2
Tq2 kn
P = softmax
Tqa ki
7qn
k2
Tqm
kn
We can rewrite our SCSA attention in Eq. ass
fi(X) :
f(X) =PV softmax (TQKT) VV=
NND
(10)
[x(X)
For simplification focus derivations for single-head attention, mutli-head attention requires only minor modifications for concatenating attention results for each head as discussed in A.2 Jacobian matrix for SCSA can be written as,
The
Jn
JIN
J,
E NDNND
JNL
JNN
(
where Jij
E R DXD 0x,
By taking partial derivatives we can show that,
JU V p() ES + +
1 HuP.V,
(11)
where Eig E NNN is binary matrix with zeros everywhere except the (i. j)-th entry, 8,3 is the Kronecker delta, and the Jacobian of the sofumax is well-known below,
19

---

Published as a conference paper at ICLR 2023
Pa (1-11P (1-Pa)
P11 P12 P2))
PA PaPiN
P2PP
p() diag (P+) P.. Pa:
(12)
Pu
P22
PiN (1 Pin)
When =j. we have,
- =T= p())
EAQK, + KQQ T
vvP.V.
(13)
When + j, we have,
- + +uP, V.
(14)
Lemma The scaled cosine similarity attention (SCSA) is Lipschitz continuous if and only if W WK WV have bounded norm.
Sketch Proof. Our key observation is that most the terms in Jy and Jij have bounded norm: v and scalars; Q, K, V are normalized so all elements are less than equal to 1; Eij has zeros everywhere except the (i.j)-th entry; P is an attention matrix with all elements within [0, 1] so all elements in p(ù) are bounded by [-0.25, 0.25]. Taking a closer look at Qu K, V, as shown in Eq. 7 Eq. 88 and Eq. they are bounded long Wº WK WV are bounded. Consequently the final product of Jú: and Jij have bounded norm if W WK K WV have bounded norm.
H.1 UPPER BOUND ON LIPoe FOR SCSA
Let us review some basic definitions for matrix norm. Suppose we have matrices A E RND and B e NXD, Then, we have:
Alloo = max Σ
Asy.
ISiSN
J=1
| V Amax AA max (A).
We also have the following inequalities,
AB || S || A
B B 1111 + B S ||||+
B and ||,,,...., ANll S
1
N
min(N.D) Σ
|All2 T (A) < |A|F=
ΣΣ A ?
(15)
where || ||r is the Frobenius norm. Equality holds and only if matrix A is rank-one matrix or matrix.
According to the above inequalities, we have
I[. Jln]
< Jailo + +zf#1 |is loc
S 11 [101 |1
E Q
K; 1l + |||
[lo] + vll a Villo
T
ET |1 1
E
Q
K,
11.0 +1llPalla
(16)
20

---

Published as a conference paper at ICLR 2023
We can compute the L2 norm Lipschitz constant by replacing the Lo norm in the above equation with Ly norm.
With simple derivations we list ||
1l for each term in 16
= maxisiso Σj=1
2j=1 syl N
[P 1l.0 maxisieN Σラ= all = max 2(P, - -P.)) s
Es =1
QQ| maxisisN Σラ |1 < VD
[K, 1lo Sc <|W 1oo x
(17)
Proof. for Equation 17
WRT x,X, WK
1
WWKT
[K 20 =11
V|xx; TWK| 112 +
|x] WW 11 +e
8
WK X,X, W
<c x <||
1/x 11(T
|||
||x; TW 12 +E m
<2x W 1xc
K 1l = [K, a < || 11 x2
K = VD
QQ T
1 <2 x - | || ao similar to Equation 17
|Pllo = |P/llo0 =1
| =1
1 22x x x ||WV Il similar to Equation 17
According to 16 the Lipoo constant of the scaled cosine similarity attention (SCSA) is:
Lip(SCSA) SUXTX N x x 1xvDxe x x2 I + xxxxtx2x2 x +
bxlxe x2x W 1bao
(N (N-1) Xxxxxxxxxx x x x x W tuxlx
After merging and rearranging the terms,
Lip(SCSA) DTNVDe 1 | + 1 [1o] + 2ve 1 WV bot
(N 11) VTN b + 2vc W 1lx)
= = | + NN W 2N
| l
21

---

Published as a conference paper at ICLR 2023
H.2 UPPER BOUND ON LIP₂ FOR SCSA
Correspondingly. we list || lll for each term in 16
Ei Vis 1ssl²² = (Σ Y = -VN
||V - E
N -
| 12 -
(18)
N
Proof of Equation
18
According to Eq (() a semi-definite matrix, thus its ordered eigenvalues \1 2 A2 2,...
AN 2 0, and - X = tr(P(t) = Σ p N
N-1
N
According to 15
11 = ( ~7)²
s < Ao)
A₂) 1x -
N-1
N
Bull2 =1
10112 5 VN
K 112 S 2 - x |
|K], VV
T T
QQ
12 2 x |
|Palll2 =1
[Ejill2 =1
[V - 2 c x |WV l12
Substituting the above results into Eq.
16 and changing Loe norm to Lg norm, we have
Lip(SCSA); VTVNNNN 2e [| K 11 + || + 2ve $ 1 112
V
N
(N 1) Nx UT
N 2 - | + 2ue
WWV VVVV l1
-N
W 1|2 2(N 1)UTE | 112 + 2Nve
W ll2.
From the upper bound above, we highlight the following observations: 1) is to guarantee validity of cosine similarity computation when any participating vector is equal to zero; 2) In the scale factor for the first term is 2N(N 11)TTe 1, which multiplies with an extra N when compared to the other terms, meaning that IW K || plays a more significant role in the Lipschitz constant of Lip(SCSA),
Different from the 1.2 distance attention 28 to promise the module is Lipschitz continuous, the scaled cosine similarity attention has no requirement for the weight matrices, but the L2 distance attention detailed in 28 requires that Q and W should be the same.
H.3 COMPARISON WITH DOT-PRODUCT ATTENTION
49 AND L2- 42-ATTENTION 28
As proved in 28), the standard dot-product attention not Lipschitz continuous. The proposed
L2-attention is also not Lipschitz continuous for general W and W but only Lipschitz continuous when Wº W However, enforcing W = W degrades model performance as shown in (28). As proved above, the scaled cosine similarity attention (SCSA) is in general Lipschitz continuous, only requiring that Wº, WK WV have bounded norm and that the computation of cosine similarity is valid. We can easily guarantee that our computation of similarity is valid by introducing small smoothing factor e.
22

---

Published as a conference paper at ICLR 2023
1 PROOF OF THEOREM 2
In this section, we give the upper bound on LipsFormer's Lipschitz constant.
For LipsFormer with S stages where the s-th stage has M, residual blocks, its Lipschitz constant is upper bounded by the inequality below.
Lip(F)
II II (1+ asm Lip(fs,m))
(19)
m=1
Here, we define k max((Lips(f))) :1= E M,}). When is set to inequality can be rewritten as,
the above
E. M,
M
8 M
Lip(F) -
Lip(fs,m)
(1+
R)
S s== Mg
E M,
a= 1 m=1
==1 mlm1
(20)
Disr M,
(1
exp((k)
Σ. Mg
DROPPATH IS AN EFFICIENT WAY TO CONSTRAINT THE LIPSCHITZ CONSTANT
DropPath 29 is another effective technique for training deep transformers, where
x x + a f(x)
if residual path dropped otherwise
y
When using DropPath with drop probability p within each residual block, the Lipschitz constant of LipsFormer refined as,
S II II (1 droppath(as.m Lip(fa.m)))pp)
Lip(F)
(21)
s=1 m=1
0. a'am Lip(Ssmm)
with probability p with probability 1 -
where droppath(as.m:
We can that DropPath effectively decreases the upper bound of network's 's Lipschitz constant by randomly dropping the contributions of residual paths.
K COMPARISON WITH EXISTING WORKS
In this section, clarify our contribution more clearly, we provide a detailed comparison of our
method with existing works, including Admin [33] ReZero 3. Swin-V2 [34), DeepNorm [51), L2 self-attention (28) and Spectral Normalization [56].
Admin 33 identifies that within residual block, the residual branch amplifies network output and the amplification effect makes training unstable. They propose to initialize the weight contributions of residual branch according the variance of its previous layer.
ReZero introduces an effective strategy to improve training stability. They notice that initializing the residual branch with 0 satisfies initial dynamical isometry, thus stabilizes model training. ReZero demonstrates that they can train very deep transformer without warmup but requires removing Layer Normalization. According Equation 19] initializing the residual contribution to 0 trivially constraints network Lipschitz constant. However, with Layer Normalization back into the network, ReZero is likely to encounter training instability again.
23

---

Published as a conference paper at ICLR 2023
DeepNorm [51] shares similar motivation with Admin [3] and analyzes the influence of the residual block and initialization. They introduce new normalization function to modify the residual connec- tion in Transformer and propose new initialization method. However, observe that learning rate warmup is still necessary in DeepNorm [51].
Training of Admin 33 and DeepNorm 51 requires learning rate warmup. ReZero
3 could train
without learning rate warmup but requires that LayerNorm is not present the network. The analyses of Admin, ReZero, and DeepNorm are not from the perspective of Lipschitz continuity.
In [56], Yuichi et inlintrduuce a simple and effective spectral norm regularization, which penalizes high spectral norm of weight matrices in neural networks. This work focuses regularization without considering residual block and self-attention block.
In [28], Kim al.prove that the standard dot-product self-attention is Lipschitz continuous. They introduce an alternative L2 self-attention that is Lipschitz continuous under the constraint that e W Such constraint limits expressiveness of the attention block and empirically degrades training performance. Also, L2 self-attention focuses only on the Lipschitz continuity of self-attention block.
Sw 34 introduces two strategies to improve training stability of transformer model, including replacing post-norm with pre-norm and scaled cosine attention replacing the original dot product attention. The introduced scale cosine attention is defined as,
Sim (q: kj) cos (qi, kj) / + Bij-
should be noted that there's difference between cos (qi, kj) / and T cos (q;, kj). the former is Lipschitz continuous function with respect to variable but the latter is. According to our derivation, self-attention based on the scaled cosine attention defined as in Swin-V2 is not Lipschitz continuous V is not normalized.
The above-mentioned works only deal with one or several standard neural computation modules. Our LipsFormer gives a holistic Lipschitz analysis of typical transformer network instead of focusing exclusively on single or few constituent modules. To derive the Lipschitz constant of LipsFormer, we analyze each constituent module of a standard transformer, including convolutions, fully-connected layer, self-attention, normalization and residual block. In this work we propose Lipschitz continuous self-attention and construct a Lipschitz continuous transformer network by bounding each constituent computation layer. The resultant LipsFormer induces stable training and does not require learning rate warmup.
We summarize contributions from both theoretical and empirical perspectives follows, Theoretically,
We derive theoretical Lipschitz constant upper bound for scaled cosine similarity attention. Meanwhile, we give thorough analysis of key Transformer components: LayerNorm, self- attention, residual shortcut, and weight initialization.
We propose a Lipschitz continuous Transformer (LipsFormer), and derive theoretical Lipschitz constant upper bound for LipsFormer. The derivation provides a principled guidance for designing LipsFormer networks.
Empirically.
We make an assumption about the Lipschitz continuity of the network, and experimentally validate this assumption.
We build LipsFormer on CSwin and Transfrranformerr. We validate the efficacy of the different versions (Tiny, Small, Base, Large and Large++) of LipsFormer on ImageNet, ImageNet-v2 and ImageNet-Real data sets.
24


Reviews:
[{"review_id": "x_BwbcxPhFH", "reviewer": "Reviewer_C9zf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Lipschitz continuous Transformer (LipsFormer), that pursues training stability both theoretically and empirically for vision transformer. LipsFormer replaces unstable Transformer component modules with a series of Lipschitz continuous counterparts. Experiments are performed to validate the proposals.", "summary_of_the_review": "In general, I lean to positive and like this new idea. However, I have several concerns about the insufficient experiments. I hope this issue can be addressed in the next round.", "strength_and_weaknesses": "Strengthen:\n\n++ Writtings are clear in general.\n\n++ The main idea of introducing Lipschitz continuous into vision transformer is novel and interesting.\n\n++ The theoretical analysis indeed strengthens this work.\n\nWeaknesses:\n\n-- Only the performances on ImageNet 1K are reported. It is necessary to report the performances on more downstream tasks like object detection or semantic segmentation.\n\n-- Several recent works of vision transformer are missing in Table 2 for performance comparison:\n\n[A] Crossvit: Cross-attention multi-scale vision transformer for image classification, ICCV. 2021.\n[B] Contextual transformer networks for visual recognition, IEEE TPAMI, 2022.\n[C] Towards robust vision transformer, CVPR. 2022.\n[D] Regionvit: Regional-to-local attention for vision transformers, ICLR. 2022\n[E] Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning. ECCV, 2022.\n\n-- Only Params and FLOPs are reported. However, it is important to add more metrics of computation cost (like memory consumption and latency)  to better understand the trade-offs.\n\n-- Following DeiT, it is necessary to perform evaluation on ImageNet-v2 [F] to measure the level of overfitting.\n\n[F] Do ImageNet Classifiers Generalize to ImageNet? ICML, 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "u8dlScX5Ne-", "reviewer": "Reviewer_H1uW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Lipschitz continuous transformer, namely LipsFormer, to stabilize the training of vision transformers. The authors study the key components of a transformer layer, including the LayerNorm, self-attention, residual shortcut, and weight initialization. In LipsFormer, the authors derive the Lipschitz continuous counterparts of the aforementioned components, such as the CenterNorm, scaled cosine similarity function, scaled residual shortcut, and spectral-based initialization. A theoretical Lipschitz constant upper bound for LipsFormer is provided in the paper. The authors justify the advantage of the proposed method on the ImageNet classification task.\n", "summary_of_the_review": "Overall, I vote for rejecting. I have two major concerns. First, the proposed Lipschitz continuous components are not novel. Second, there is a significant lack of experimental results to validate the advantage of LipsFormer.\n\n*********** After Discussion ***************\n\nThe authors' response largely addresses my concerns. I have increased my original score to 6.\n\n\n", "strength_and_weaknesses": "**Strong points:**\n\n1. The paper addresses an important issue in designing and training transformers, which is how to guarantee that the model is Lipschitz continuous.\n\n2. The paper provides an interesting Lipschitz constant upper bound for the proposed LipsFormer.\n\n3. The paper is well-written with illustrative figures.\n\n**Weak points:**\n\n1. Experiments on more tasks and different data modalities are needed. Also, results for applying LipsFormer on other baseline vision transformers are missing.\n\n2. There is no result showing that the training is stabilized. The authors should show the training curves in the paper.\n\n3. The authors say, “we show that Lipschitz continuity is a more essential property to ensure training stability,” but there is no comparison to other methods that stabilize the training of vision transformers in the paper. The authors seem to try to provide this comparison in Table 3, but this table is just an ablation study of LipsFormer and does not show that “Lipschitz continuity is a more essential property”.\n\n4. The authors do not empirically compare their method with other related works that introduce Lipschitz continuity to transformers, such as [1]. How is the upper bound of the Lipschitz constant of LipsFormer compared to the bounds from these works?\n\n5. In both Scaled Cosine Similarity Attention and Weighted Residual Shortcut, the predefined or learnable parameters can be absorbed into the weights, W^{Q}, W^{K}, W^{V} or W, to recover the standard dot-product attention and residual connection. Also, the Weighted Residual Shortcut has been explored in [2]. Thus, these two components are not novel.\n\n**Additional Concerns and Questions for the Authors:**\n\n1. Compared to the baseline CSwin Transformer, the LipsFormer has more parameters and more FLOPs. I am not sure if the advantage of LipsFormer over the baseline CSwin Transformer is from the Lipschitz continuity or just because the LipsFormer has a larger size and does more computation. I mention CSwin Transformer here because the authors adopt a similar training strategy as in CSwin Transformer [3].\n\n2. The authors say, “The derivation provides a principled guidance for designing LipsFormer networks (e.g. choosing the weight of a residual shortcut according to the network depth)”. Is there any result to support that the value of the weight of a residual shortcut decided by the network depth is better than other values?\n\n**Minor Comments that did not Impact the Score:**\n\n1. What is 11?\n\n**References:**\n\n[1] Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562–5571. PMLR, 2021.\n\n[2] Falong Shen, Rui Gan, and Gang Zeng. Weighted residuals for very deep networks. In 2016 3rd International Conference on Systems and Informatics (ICSAI), pp. 936-941. IEEE, 2016.\n\n[3] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2107.00652, 2021.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GsI-1xPsdP", "reviewer": "Reviewer_HiVm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new ViT architecture called LipsFormer, which replaces non-Lipschitz or unstable modules with Lipschitz continuous counterparts. Experiments show that LipsFormer obtains better Top-1 accuracy on Imagenet-1k dataset compared with previous works. ", "summary_of_the_review": "This paper proposed LipsFormer, which enforces Lipschitz continuity on the submodules of ViT. The proposed techniques show good empirical performances and seem to be effective In stabilizing the training of ViT.", "strength_and_weaknesses": "Strength:\n\nThis paper analyzes the Lipschitz bound of each component module of ViT comprehensively, and proposed Lipschitz counterparts for the non-Lipschitz modules. \nExtensive show superior performance compared with previous works, and ablation studies show the importance of each proposed module.\n\nWeaknesses:\n1) This paper replies on the assumption that training stability relies on Lipschitz continuity. My main question is: whether introducing Lipschitz continuity will cause other issues for training stability, since training stability also depends on other aspects other than Lipschitz continuity. For instance, for instance, BatchNorm and LayerNorm are proposed to relieve the \"covariate shift\" problem. The proposed CenterNorm throw out std(y) in the denominator, will this cause other issues due to \"covariate shift\" of neurons from different layers? It may be good to check how does std(y) change across layers with different normalizations.\n\n2) How does scaled cosine similarity perform compared with L2 self-attention proposed in Kim 2021, given these two operations are both Lipschitz continuous? Either theoretical or empirical justification would be fine.\n\n3) Since some of the proposed modules are initialization techniques, I think it'd be interesting to see how the learnable parameters change during training. For instance, does the weight of skipping connection $\\alpha$ stays on a similar order of magnitude as one initializes it?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ebvenmkxxu", "reviewer": "Reviewer_CmFG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces several modules to stabilize and accelerate the training of the vision transformers. In particular, the CenterNorm, scaled cosine similarity attention, and spectral initialization are proposed, which are the counterpart of the LayerNorm, standard self-attention, and  Xavier/Kaiming initialization respectively. The underlying philosophy is that model's Lipschitz constant plays a critical role in stabilizing the training procedure. The smaller Lipschitz constant enables us to the deeper networks and it is essential to design the structure/modules to control the changes in the Lipschitz constant. The authors verify the efficiency of the proposed modules in the ImageNet-1K classification task and in the ablation study the effect of each individual module is discussed.", "summary_of_the_review": "This paper provides several modules to control the Lipschitz constant of the Transformer models. The modules are theoretically justified and show promising numerical performance in ImageNet-K classification. It would be better if we could see the experiments on the large model (e.g., >300M parameters).\n\nMinor issue:\n\n1. Please add the definition to the $\\odot$ operator in sec. 4.1.1.\n2. Please add a reference for Droppath in the main paper.\n\n\nBased on the current presentation, I tend to accept it and I'm willing to change my evaluation after the rebuttal.", "strength_and_weaknesses": "Strength: \nSimple but effective structures are proposed which are also well-motivated theoretically.\n\nWeaknesses:\nIn the numerical tests, the model size considered is relatively small (e.g., < 100 M). Usually, in highly overparameterization regimes (e.g., > 300M) the landscape or geometry in terms of optimization is quite different from those in the small model regimes. Many training stabling/accelerating tricks that work well with small models may fail to be effective. It is worth evaluating whether the proposed modules can also improve the training for the large enough model (e.g., >300M) and push a higher SOTA accuracy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster