Rethinking skip connection model as a learnable Markov chain
Authors: Dengsheng Chen
Jie Hu
Wenwen Qiang
Xiaoming Wei
Enhua Wu
Venue: ICLR 2023 poster (2023)
Keywords: Language translation
image classification
transformer

Abstract: Over the past few years afterward the birth of ResNet, skip connection has become the defacto standard for the design of modern architectures due to its widespread adoption, easy optimization, and proven performance.
Prior work has explained the effectiveness of the skip connection mechanism from different perspectives.
In this work, we deep dive into the model's behaviors with skip connections which can be formulated as a learnable Markov chain.
An efficient Markov chain is preferred as it always maps the input data to the target domain in a better way.
However, while a model is explained as a Markov chain, it is not guaranteed to be optimized following an efficient Markov chain by existing SGD-based optimizers prone to getting trapped in local optimal points.
In order to move towards a more efficient Markov chain, we propose a simple routine of penal connection to make any residual-like model become a learnable Markov chain.
Aside from that, the penal connection can also be viewed as a particular model regularization and can be easily implemented with one line of code in the most popular deep learning frameworks. 
The encouraging experimental results in multi-modal translation and image recognition empirically confirm our conjecture of the learnable Markov chain view and demonstrate the superiority of the proposed penal connection.

Published as a conference paper at ICLR 2023
RETHINKING SKIP CONNECTION MODEL AS A LEARN- ABLE MARKOV CHAIN
Dengsheng Chen"
Jie Hu*
Meituan chendenggheengeeeituuu
State Key Laboratory of Computer Science, ISCAS University of Chinese Academy of Sciences huj iel ac..ac.co
Wenwen Qiang
Xiaoming Wei
University of Chinese Academy of Sciences Institute of Software Chinese Academy of Sciences wenwen20188m
Meituan we i www.xiimmnnnmmmetteeu com
ac. en
Enhua Wu
State Key Laboratory of Computer Science, ISCAS University of Chinese Academy of Sciences University of Macau ehwu@um, edu, mo
ABSTRACT
Over the past few years afterward the birth of ResNet, skip connection has become the defacto standard for the design of modern architectures due to its widespread adoption, easy optimization, and proven performance. Prior work has explained the effectivenesss of the skip connection mechanism from different perspectives. In this work, we deep dive into the model's behaviors with skip connections which be formulated preferred as it always maps the input data to the target domain a better way. However, while model is explained optimized following an efficient Markov chain by existing SGD-based optimizers prone getting trapped in local optimal points. In order to move towards more efficient Markov chain, we propose simple routine of penal connection to make any residual-like model become learnable Markov chain. Aside from that, the penal connection can also be viewed particular model regularization and can be easily implemented with one line of code the most popular deep learning frameworks. The encouraging experimental results in multi-modal translation and image recognition empirically confirm our conjecture of the learnable Markov chain view and demonstrate the superiority of the proposed penal connection.
learnable Markov chain. An efficient Markov chain is
Markov chain, it is not guaranteed be
INTRODUCTION
Over the last decade, deep learning has been dominant in many tasks, including image recogni- tion (Voulodimos et al. 2018), machine translation (Singh et al. 2017). speech recognition (Zhang et al. 2018), etc. Many SGD-based methods and excellent network structures come to the fore (Alom et al..ll20111)) (019). Among them, skip connection (He et al.) 2016) is widely-used technique im- prove the performance and the convergence of deep neural networks. Aided by the skip connection, models with very deep layers can be easily optimized by SGD-based methods (Amari 1993), e.g.. vanilla SGD (Cherry et al. 1998), Momentum SGD (Sutskever el al.) 2013), Adagrad (Lydia and
Francis 2019) Adam (Kingma and Ba 2014).
Recently, many theoretical explanations of how works have been largely underexplored and Yuan 2017 Allen-Zhu al., 2019). In this work, continued to explore the behaviors of the model with skip connection and view as a learnable Markov chain (short for Markov
"Equal contribution.
Corresponning author. The work is supported in part by NSFC Grants (62072449).

---

Published as a conference paper at ICLR 2023
2017). To our best knowledge, it is the first time to analyze from this perspective. In the conception of the Markov chain, the output of a residual block is noted as the predicted di- rection with respect to the input. For better elaboration, The ideal direction always points to a more accurate direction than the predicted direction, which translate an input to the target domain in more efficient way. Then, define an indicator to reflect how efficient a learned Markov chain is, based on the angle between the predicted and ideal direction. In contrast to the original predicted direction, direction is preferred since it always maps the input to the target domain in better way. However, we are aware that existing SGD-based optimizers are quite lazy to update the model following an efficient Markov chain, which hinders the upper bound performance.
chain) (Gagniue
introduce another term ideal direction.
efficient Markov chain with an ideal
To train more efficient Markov chain, we propose very simple routine of penal connection to convert residual-like model to a Markov chain by just adding one line of code in existing deep learning frameworks. On the one hand, the penal connection capable of enforcing the optimizer to update the model following the rules of the efficient Markov chain. On the other hand, it can be viewed as a type of additional model regularization, which alleviates over-fitting and enhances generalization. Compared with the original residual-like model, the Markov chain also has more benefits deeper networks that suffer from performance degradation corresponding to more learn- able parameters. The experimental results in multi-modal translation and image recognition only demonstrate the feasible analysis of regarding a residual-like model as a Markov chain but also examine the superiority of the proposed penal connection throughout the optimization process.
Our main contributions can be summarized in two folds. First, we present a new perspective to understand the skip connection model learnable Markov chain and carry out exhaustive theory analysis and experimental verification. Second, we propose the penal connection, a simple method enable a network to be optimized within a more efficient Markov chain, which can substantially improve performances both in the fields of NLP and CV.
Target domafn B
Source domain A
d, |Ko
Lit
X4
dylx
Silee (n
X
(a)
(b)
Figure 1: A model M with I. skip connections can be recognized as Markov chain C consisting of L. nodes. The forward pass is corresponding connection along with residual-like block fe, (-) builds up Markov chain node ny (the gray dash box in middle). The input of ny is T1-1, i.e. the output of the previous Markov node. The output of n; be formulated by z Z1-1 21 21-1 where 2; -1 fe. (21-1) is the predieted direction by residual-like block. As shown in Fig. I(b) guided by 21,i11.--. L, the input data 2g E A can gradually shift to the target label y E B along the learned Markov chain. The red dashed arrow de | is the ideal direction with respect to .1-1 and 21.
Markov process. As shown in Fig. [[a] skip
2 METHOD
In this section, we first reformulate the residual-like model as a Markov chain and introduce to reflect the efficiency of a learned Markov chain. Then, also define 8-convex chain and make the convergence proof based on it. Before exploring the optimization algorithm, the dilemma between the behavior of efficient Markov chain and existing backward backwarrd propagation algorithms is thoroughly discussed. Lastly, we propose a penal connection mechanism to boost the performance of the Markov chain.
2

---

Published as a conference paper at ICLR 2023
2.1 THE LEARNABLE MARKOV CHAIN
Similar to the definition of traditional Markov chain in Gagniuc 2017). learnable Markov chain can be defined as:
Definition
(The learnable Markov chain: Cr.). learnable Markov chain Cr. is stochastic
model with learnable parameters 10..... 0L describing a sequence I. possible events in which the state of the current event only depends on the state attained in the previous event.
V- tepeem
XXX
I
T
Inefficient but easy to learn Efficiest but hard to Learn
-
(a) The influence efficient/ineffccient Markov chain. (b) A toy model demonstrate the importance of
suitable s.
Figure 2: A simple task to show the performance difference between plain net, skip net, and our proposed Markov net. this task, we build a model which consists three fully connected layers with 28 learnable parameters to shift coordinate (x, y) to target distribution, which can be formu- lated as x² y². More details are listed in supplementary materials. We update the model with SGD optimizer for 10000 steps. As shown in Fig 2(b) Markov net can better complete this task.
As shown in Fig. I(a) My indicates model with L residual-like blocks, and can also be considered as a learnable Markov chain Cz with I, nodes since the output state (i.e. output feature map of a residual block) xι of node nı only depends
the state 31-1 from the previous node 7y 74-1- As
result, a forward pass through Mr can be viewed as a Markov process, as shown in Fig. I(b) In bit more detail, the corresponding Markov chain with respect the input data zo A is formulated as:
+ Imo
+
+ JL
Cr + xx) == x0
01
T441
#L-1
where fo, is feature transformation function by i-th residual-like block in ML, and is also the i-th chain node in Cz with learnable parameters 9,, 2|/21-1 = fo, (#1-1) 1) is the predicted direction for previous state Fı-1 in node n4. Correspondingly, the ideal direction di(z1-1 with respect to æ7-1 and zι can be defined as:
Definition
(The ideal direction: de(21-11. Assume the function & measures the distance between swo variables, and if ((a, 2 E(b, c), then E(a,e) E(pa k)), c) 2 l(b,c), where μ [0, 1). de [1 is an ideal direction with respect to F1-1 and zı as long as:
(x1-1+ + (1111111111) S ((1-111111111 + 17(0)11919)
where 1i is small step size.
Obviously, given I1-1, and E, the ideal direction di is still not unique because anyone who can outperform the predicted direction 2 under the measurement is qualified, as Eq 2 holds. We collect all the ideal directions for Iı-1, under as 1
Lemma di E 1 then di = pdd + (1 - 1) E De where μ 2 0.
3

---

Published as a conference paper at ICLR 2023
Proof 2.1 Since di is ideal direction, then:
f(x1-1 + ndd, v) _ ((p(m--1 + rjdi) (1 1--1 (1) + (2).1))
1(111111-- + +n(du + v.)
df
S f(11-1+ 112, y):
thus, di = pdg + - (1=
i is an ideal direction with respect to z and 11-1-
Different function fo. takes discrepant effects on the set D Dezal 1' fo. is used to be a sequence of convolutions (e.g. residual block ResNet (He et al. 2016)or a popular Transformer intro- duced in Vaswani al. (2017). long as it can make Cr. shift input A to the target domain correctly. Importantly, we next devise an intuitive indicator to reflect how the efficiency of Cy(20 + ZL):
Definition (The efficiency of Markov chain: C₁ Cr - x1)). The efficiency indicator & measures an average cosine similarity between zi 21| and di -11=
L
COS 04 = (2)
(4)
1=1
where i is the normalized tensor of u. defined as == w/llul12. Las is the angle between 2 2111-1 and di 1-
From the definition, a larger e indicates a smaller Laz between the predicted direction 2 and the ideal direction d, mirroring more efficient Markov chain and vice versa. More specifically, if a always has a positive fraction on di for all nodes, we call this is a convex chain which formally defined as follows:
Definition 4 (é-convex chain). If > O, Vny E CL - æL). (21, > 8 5 then C(ro + L is dubbed as a 8-convex chain.
Lemma Cf (2r - x.) is 8-convex, >
Proof 2.2 Since ( + L)) is 5-convex, then for all I: (z,, d)) = 111=1122,, [|deladdj) = [1 (2, d) > 811411 > 0
(5)
It is noteworthy that 12/12//c4/1e 0, thus Vn,, (7, di) O. which yields 1 Et= di)
0.
Lemma 2 tells that (= + z1) 8-convex, it must be an efficient Markov chain, while the reverse is not necessarily true. As shown in Fig. [(b] the plotted chain is an efficient Markov chain, but it not 8-convex chain since there does not exist non-negative 8 making Zaj satisfy Definition
Cr + xr.) is 8-convex, then in every node ni, positive fraction of the predicted direction 21, i.e., 21 cos Lon, is pointing the same direction d,. Given an appropriate step size 7, the input could eventually arrive at the target domain, despite along with a winding path. Fig. 2(a) also illustrates simple model to verify it. The source input gets closer to target domain B in every step while moving along a 8-convex Markov chain (visualization in brown).
Formally, have concluded the following lemma for ensuring convergence.
Lemma For each chain node ny E (ro + IT), consider the the forward process It = Tt-1 + Eb where E[zi] = d;, +1 < 2 Suppose for all nodes, xı is always inside the 8-convex region with diameter D, i.e., | S D. Then, for any > and any L such that L" log 2 (1+0)77
D²
have E|[l=L v (1+a) log Lz²
The proof Lemma apppears Appendix B Notably, Lemma does not imply that xy equals to y. It only describes that æ is sufficiently close to y if can predict the correct direction for Tı-1- For a longer chain (i.e. deeper network with larger L). the Iy will be restricted closer to target with relatively less error. Taken together, Lemma. guarantees that Markov chain C₁ shift source input Zo E A to target domain B through L nodes in an efficient way if C, is 8-convex. Until now, can trustingly settle down to optimize Mr. from the conception of Ct.-
4

---

Published as a conference paper at ICLR 2023
2.2 MARKOV CHAIN OPTIMIZATION
Before exploring the optimization method of a Markov chain, it is necessary to have a careful dis- cussion about the efficiency of Markov chain taking effect the optimization process. According to aforesaid Lemma. 3 if all parameters 6₁ are optimized in way that makes Cp (2o + σL) al- ways be a ò-convex chain for any input Zo, the convergence speed for optimization appears to be substantially improved, and the final performance
well. However, we caution that a 8-convex chain not guaranteed by existing SGDD-based optimizers, such as SGD and Adam. Actually, the existing optimizers prefer towards to efficient even inefficient Markov chain, where the land scope the loss is more smooth, instead d-convex one. After discovery and practice, we find (GD-based optimizers have the potential to bridge the gap which will be discussed later. The diagrammatic sketch of efficient and inefficient Markov chain is illustrated in Fig. 2(a) which can better help to understand.
Intuitively, while & + 1, an efficient Markov chain acts more like a Ó-convex chain and can obtain its good properties. However, convergence. Hence, the main idea to optimize a Markov chain to train a "reasonable" efficient Markov chain, i.e. let ||=| no larger than a given threshold p. In this way, the objective function can be formulated as:
large is disaster for existing optimizers, resulting in hard
L= L- (LL, + Al|el12.
6
The first item of Eq. 6 is the original loss function of the model Mr and the second item is the additional penalization with to hold ||=||2 S p. set = in Eq. 6 the objective function degenerates into plain mode without any remedy toward forming a more efficient Markov chain. Instead, set À > O, the penalization will take effect, suggesting enable to obtain a more efficient Markov model. The toy example in Fig. 2(b) visualizes the significant effeet of this penalization term over the plain net (MLPs) and skip net (MLPs with skip connection).
a.C, (x
xi Xi 119x1
dx,
az,
dj 19x1
+ 21
1x
21 = (
X1-1
8x
x1
+
fa, (8
[xl
g
x1 X +
Xt-1
xx-1 = 9x + g
(a) The proposed ideal direction.
(b) The backward propagation of gradient within a Markov chain node.
((a) is worth noting that zi can be viewed that we update zi by IR, with small learning rate 17. The backward propagation of gradients within a Markov chain node is plotted in Fig. 3(b) Compared with the residual-like model, we only add an additional gradient Oz, while computing the gradient with respect so.-
Figure 3: A visualization of ideal direction computed based on 9m, in Fig.
In order to solve Eq. 6 the correct ideal direction d, with respect r4-1 is required to figure out. As discussed previously, di is not unique, thus the different definition of l leads to a different set of di- Actually, since the feature space of 2y always lies very high dimension space, it would a great challenge to find suitable é to define an ideal direction di.
As for a chain node n4, we reuse the target loss function LM,, build valid & function:
((r.y) = L.2 (Cr(Ii + x1),,).
(7)
Iy is forward along the rest chain nodes 1141, ny,, and the final output vy is taken to compute the loss between y by LMg, This way, the gradient with respect to x is:
5

---

Published as a conference paper at ICLR 2023
al(z1, y)
0LMz (
Oz
dmi
Hence, Ie, can be obtained while the backward propagation during the training process and an ideal direction di based on 9 is 1992, where is a small step size, as shown in Fig. 3(a)
Proof 2.3 Since 2ι 21-1 + and f(a 112. v) < ((x11)) v) always holds, set d' == - 1122 then
f(x1-1 + di v) = E(x1-1 + 19 v) - f
v) < ((xi, v) = f( 1 + 21.
(9)
Thus, d' = 21 - 199m is an ideal direction for 21-1-
From Lemma 1
we known that di = =udl + (1 - p)zz z - 1199, is also an ideal direction,
i.e., 21 7/ Di
Assume 1|9=; ll2 2 1, then we can set μ 1l0,, E [0, 1], yields
d: n9m. Since is small, di can be approximately expressed as Σ 19m. and & can be reformulated as:
L
Σ (2) di)
1
Σ
(2,
79m.)
1
(2) Ei) + (24, -19m.)
L
L l=1
1=1
1.
1 Σ
= , Σ -ū,
(
- )
(10)
T
1=1
where E' will be used in the following experiments instead of as it indicates the same efficiency properties the Markov chain but is casier to understand (a larger e' indicates a more efficient Ct))
Then, the gradient to 2 can be computed as:
MLM, (CI,, 4) Oxs
@
9=1
Om
O2
0 21)
0
21 L.
1==1
+ 17/2,,
021
Ōz1
θ E + 11(221, -1r))13
~ 92, +
(11)
021
9m. + 0|1 + +,1c22, To
8z1
+ 2X( -1122, )1 + (c21, -1)
9nm + 22(-2113) + 2Xce1²
2\m
(1 [9 1|2
)gm: + 22c7722 21
9Z T
(12)
where 7 a hyper-parameter and |= in Eq. 11 can be regarded as a constant value to simplify the gradient derivation process. The analysis of this estimation can be found in Appendix C Despite lots of hyper-parameter introduced single syper-parammter in the final formulation, which relieves the heavy burden from hyper- parameter sweep. We dubbed this special optimization method as penal connection, which seems to add simple penalization on the norm of 2 type of model regularization. The compute graph has been plotted Fig. (b)] and it can be easily implemented based on the PyTorch framework by adding one line of code (see Algo.
facilitating derivation, e.g., X, JJ, c, P. we only need to specify
TTips: zeregitter hook(lamhda Iz.i Iz, T21) is allowed which could lead memory leak.

---

Published as a conference paper at ICLR 2023
Algorithm Pseudo code of penal connection in a PyTorch-like style.
21 fe, (x1-1)
Only following line is added to register a hook
21 irvegister_hooollanne Oay desgg,,etacchh.coonec) fr. + T22)
31 = 21-1 27
Lastly, due to the various choice of L, the gradient to is correspondingly discrepant. Further community exploration about more reasonable and effective ways to compute the ideal direction of great value so continually push forward the performance of the Markov chain.
3 EXPERIMENTS
In this section, we conduct intensive experiments to demonstrate the superiority of the Markov chain in the field of Natural Language Processing and Computer Vision. Unless otherwise specified, all the residual-like blocks M are converted Markov chain node n, C.
Specifically, in transformer block (Vaswani et al. (017)) which consists of multi multi-head self-attention module (MSA) and feed-forward network (FFN), we convert it as two chain nodes as both MSA and FFN employ skip connection respectively.
For simplicity, we set the same for all nodes in a Markov chain. All experiments are carried out by publicly available projects implemented by PyTorch (Paszke et al. (017)) device equipped with NVIDIA-AION GPUs. More experimental details can be found in the supplementary materials.
3.1 MULTI-MODAL TRANSLATION
Multi-modal translation requires model to be capable of translating the text information from the source language A to the target language B. How the efficient/ineffciient Markov chain performs during the training process can be clearly observed in this experiment.
German Tandationn (250 Specto
Testing Acouacy
035
.5.5
3.5
25-
3.0
-4.5
2.5
24
-0.9
6.2-
WMT16 English Germany Translation (Setting
WDITL Engi in Turnlos Spuchs)
04
3.0
43
2004
18:00
1000
225
25ic
530
2006
1200
2000
(b) WMT16 English to Germany Translation (Setting
Figure 4: The testing curve across different epochs on WMT16 English to Germany translation tasks (Germany translation curve can be found in supplementary materials).
Dataset. WMT16 (Bojar et al. 2016 a widely used translation dataset based on the data from statmt.org. which contains various interesting translation tasks specified domains. Here, focusing on the news translation tasks between English and Germany. The text for all the test sets is drawn from news articles.
7

---

Published as a conference paper at ICLR 2023
Settings
PPL
Accuracy (%)
Task
Wa
Tottl
Transs
Markov.
Improv.
Trans.
Markov.
Improv.
WM DE/EN (Q)
35
200
33.3
7.9
+ 25,4
36.4
70.0
33.6
WMT-16 DE/EN (S)
350
2000
17.3
11.3
6.0
66.4
70.4
4.0
WMT-16 EN/DE (Q)
200
31.8
6.3
+ 25.5
43.1
71.8
28.7
WM EN/DE (S)
350
2000
13.4
8,5
1 4.9
68.7
72.8
1 4.1
Table Testing PPL (the lower the better) and accuracy (the higher the better) of Trans- former (Trans.) and counterpart Markov chain (Markov.).
Model
Baseline
3 x 10-9
3 x 10
3 x 10-7
3 x 10
Improv.
ResNet50
79.2
79.0
79.4
79.2
79.3
+0.2
ViT-S/16-224
77.8
78.0
78.1
78.0
77.8
+0.3
DeIT-S/16-224
79.9
79.9
80.1
80.0
79.8
+0.2
Swin-S/4-7-224
83.3
83.2
83.5
83.3
83.1
+0.2
Table 2: The top-1 accuracy on ImageNet- Ik on different architectures with different T.
Implementation details. We adopt the most widely used benchmark Transformer (Vawwani er al. 2017) our strong baseline. The embedding size demb is 512, and the source and target embed- ding layers are shared. We empirically set T to 3 x 10-4, which generalizes well to all translation tasks. Here, we opt for the mutual translation tasks between English and Germany for validation. All the models are trained using Adam optimizer with 31 = 0.9, 32 = 0.98. We use a batch size of 1024 and weight decay of 0.05 and other recipes for training are identical to the original imple- mentation (Vaswani et al. 2017). We set up two regular training settings, Q and S separately (see Table 1.
Result analysis. From Fig. and Tab. 1 a few patterns can be observed. One is that the Markov chain with penal connection converges faster than the residual counterpart. Surprisingly, the Markov chain training with 200 epochs can outperform the baseline model well-training with 2000 epochs (10x training schedule), demonstrating the good merits of the proposed method. It is worth noting that the transformer model does not saturate without adequate training schedule length. The E' value plotted in Fig. may illustrate this phenomenon. From the curve, we find that the transformer model is more likely to be inefficient Markov chain whose e' is always negative. Aided by the proposed penal connection, it becomes an efficient Markov chain whose e' is mostly positive. Furthermore, we observed that at the early training stage, Cr. an inefficient chain, which is even worse than the baseline. After several epochs, it conversely turns to become a 6-convex chain (with a large s'). less efficient Markov chain with a small positive e'. This phenomenon also confirms our analysis in the method section: Moving along with ó-convex chain can help converge to the target in the most efficient way. However, it may not be the best way under the existing optimizer. In order to achieve higher performance, the model will be pushed to be less efficient chain. As for the baseline model, we find that it always acts as an inefficient Markov chain which more easily gets trapped in local minima.
After that, Cg decays
3.2 IMAGE CLASSIFICATION
Different from translation tasks, the gradient for image classification is quite sparse and the re- dundant parameters often guarantee the optimizer towards to an effective Markov chain, even to a 8-convex chain, suggesting that even without the penal connection, the model can also be optimized efficiently. Despite that, we still observe slight 7 (smaller than 10- can further improve the final accuracy by non-trivial margin. This benefit is at least partly due to the model regularization effect of r which alleviates over-fitting
Dataset. We conduct a series of experiments on the task of image classification using the ImageNet- IK dataset (Deng et al. 2009). which consists of 1.28 million training images across 1000 classes and 50k images for validation.
Implementation details. There are many variants of residual- models used in the image classifi- cation task. We conduct experiments on three representative types models, i.e. ResNet (He et al. 2016), ViT (Dosovitskiy et al.) 2020) (and its variants, e.g., DelT (Touvron el al.) (02T), Swin (Liu el al. 202T)) During experiments, we find that all these models can learn an efficient Markov chain or even d-convex chain, that only very small (less than 10 10-6) is applied. All the models are trained for a 10-epoch linear warmup and a cosine decaying schedule afterward for 290 epochs. More details can be found in the supplementary materials.
Result analysis. As listed in Tab. 2 despite that the baseline models have already achieved a satu- rated accuracy, a suitable with penal connection routine can further push forward this performance
8

---

Published as a conference paper at ICLR 2023
# layers
18
20
32
34
44
50
56
101
152
CIFARIO
ResNet Markov.
92.1
92.6
93.2
92.2
92.3
93.2
93.4
93.4
CIFARI00
ResNet
76.3
77.6
78.5
Markov.
76.7
78.0
79.1
ImageNetIk
ResNet
80.7
81.1
Markov.
81.1
81.4
Improv.
+0.4
+0.2
+0.6
+0.4
+0.2
+0.6
+1.2
+0.4
+0.3
Table 3: The accuracy of ResNet over different depths on CIFARIO, CIFARI00 and ImageNetIK.
by non-trivial margin. Across experiments, large usually does hurt the mode performance. This observation reflects another effect of penal connection, that is r not only engages the model to learn efficient Markov chain, but it also limits the Markov chain to be too efficient, keeping the Markov chain away from a 8-convex chain. We think is also reasonable because a 5-convex chain may lead to hard optimization, and slight 7 can take effect in this situation. In other words, the pe- nal connection can also be viewed as model regularization that can improve the final performance via alleviating over-fitting from new standpoint.
3.3 MODEL DEGRADATION
One of the main reasons that residual-like models become widely used across various tasks is their capacity to counter the problem of model degradation in deeper networks. Model degradation refers to the phenomenon that with the increasing depth of model MM, the performance will no longer be improved and even worse. We find that by converting residual-like model Mr, Markov chain C1, the model degradation problem can be solved better. With the aid of the penal connection routine, a deep model can arrive at higher performance than the original counterpart.
Dataset. CIFAR 1 consists of 60k 32x32 color images in 10 classes. There are 50k training images and 10k test images. The CIFAR 100 dataset is identical to CIFAR-10, except the number of classes is 100.
Implementation details. We take ResNet He et al. (2016) for comparison. In order to investigate the performance trend of models over different depths, we build seven models with different depths, ie., L E {18, 20, 32, 34, 44, 50, 56]. We used a momentum SGD optimizer with a batch size of 128 and weight decay of 0.0001 for CIFARIO and 0.0005 for CIFAR100. The is 3 x 10 for all experiments. We trained all the models for 200 epochs from an initial learning rate of 0.1. The learning rate decayed by factor of 10 at epochs 100, and 150 for CIFAR10 10, and at epochs 60, 120, and 160 for CIFAR100.
Result analysis. The results listed in Tab. All the Markov chain models significantly advance baseline residual models. In particular, as the model goes deeper, the performance of baseline mod- els saturates first and decay afterward. On the contrary, the Markov chain with penal connection consistently achieves stable gains. This evidence confirms that our proposed method can further alleviate the model degradation problem, motivating future scaling efforts in depth.
4 CONCLUSIONS
In this work, we introduce the conception of a learnable Markov chain for the residual-like model, and propose simple routine of penal connection to boost the model performance and alleviate the model degradation in deep depth as well. Adequate theoretical analysis and comprehensive exper- iments on the different types of architectures across spectrum of tasks jointly demonstrate the rationality and effectiveness of the learnable Markov chain. While these initial results are encourag- ing, many challenges remain. For example, better way to compute the ideal direction of Markov chain would likely lead to further improved performance. We expect more research would pay more attention to this new perspective which will inspire future work.
httpp::// httpss:/www.cc. wtw.stttooooooooooiiiuoooo

---

Published as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
Md Zahangir Alom, Tarek M Taha, Chris Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Mahmudul Hasan, Brian C Van Essen, Abdul AS Awwal, and Vijayan K Asari. A state-of-the-art survey on deep learning theory and architectures. Electronics, 8(3):292, 2019.
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputting, 5(4- 5):185-19,, 1993.
Ondfej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. Results of the wmtl6 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, -Shared Task Papers, pages 199-2312 231, 2016.
J Michael Cherry, Caroline Adler, Catherine Ball, Stephen Chervitz, Selina S Dwight, Erich Hester, Yankai Jia, Gail Juvik, Tai Yun Roe, Mark Schroeder, et al. Sgd: Saccharomyces genome database. Nucleic acids research, 26(1):73-79, 1998.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19929, 2020.
Paul A Gagniuc. Markov chains: from theory to implementation and experimentation. John Wiley & Sons, 2017.
Dani Gamerman and Hedibert F Lopes. Markov chain Monte Carlo: stochastic simulation for Bayesian inference. CRC press, 2006.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Samuel Karlin. A first course in stochastic processes. Academic press, 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. Advances in neural information processing systems, 30, 2017.
Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, and Yuexian Zou. Rethinking skip connec- tion with layer normalization. In Proceedings of the 28th International Conference on Computa- tional Linguistics, pages 3586-3598, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IFFF/CVF International Conference Computer Vision, pages 10012-10022, 2021.
10

---

Published as a conference paper at ICLR 2023
Agnes Lydia and Sagayaraj Francis. Adagrad an optimizer for stochastic gradient descent. Int. J. Inf. Comput. Sci, 6(5):566- 568, 2019.
Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé. Vampnets for deep learning of molecular kinetics. Nature communications, 911:1-111, 2018.
Sean P Meyn and Richard L Tweedie. Markov chains and stochastic stability, cambridge university press, 2009.
Martijn van Otterlo and Marco Wiering. Reinforcement learning and markov decision processes. In Reinforcement learning, pages 3-42. Springer, 2012.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Ravid Shwartz-Ziv and Naftali Tishby, Opening the black box of deep neural networks via informa- tion. arXiv preprint 10X3v:1703300811111 2017.
Shashi Pal Singh, Ajai Kumar, Hemant Darbari, Lenali Singh, Anshika Rastogi, and Shikha Jain. Machine translation using deep learning: An overview. In 2017 international conference computer, communications and electronics (comptelix), pages 162-167. IEEE, 2017.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning, pages 1139-11 1147. PMLR, 2013.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018, 2018
Zixing Zhang, Jürgen Geiger, Jouni Pohjalainen, Amr El-Desoky Mousa, Wenyu Jin, and Björn Schuller. Deep learning for environmentally robust speech recognition: An overview of recent developments. ACM Transactions on Intelligent Systems and Technology (TIST), 9(5):1228, 2018.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnı, and Jürgen Schmidhuber. Recurrent highway networks. In International conference machine learning, pages 4189-4198. PMLR, 2017.
A RELATED WORKS
The model with skip connection. In recent years, a large number of skip connection mechanisms have been used for neural network design (Liu al. 2020). Highway network (Zilly et al.) 2017) built skip connection with learnable gating mechanism from the input to the output. He ei af. (2016) proposed the identity skip connections ResNet, which have already become an indis- pensable building module in modern architecture design. This type of model with skip connections can be summarized as a residual-like model. In the field of natural language processing. Trans- former (Vaswani et al.) 2017) made use of skip connection to build the multi-head self-altention module and the feed-forward network. Recently, Transformer has attracted extensive attention in computer vision and various variants (Han et al. [202)) have been proposed which achieve superior performance against traditional CNNs. In this work, we aim to improve the performance of most existing residual-like models from the perspective of learnable Markov chain.
11

---

Published as a conference paper at ICLR 2023
Markov chain. A Markov chain Markov process is stochastic model describing sequence of possible events in which the probability of each event depends only on the state attained in the pre- vious event (Gagniue 2017). Informally, it can be thought of as "What happens next depends only on the state of affairs now.". Markov chains have many applications for statistical models (Karling 2014), such as studying cruise control systems in motor vehicles, queues or lines of customers riving at an airport, currency exchange rates, and animal population dynamics (Meyn and Tweedie 2009). Markov processes are the basis for general stochastic simulation methods known as Markov chain Monte Carlo, which are used for simulating sampling from complex probability distributions, and have found application in Bayesian statistics, thermodynamics, statistical mechanics, physics, chemistry, economics, finance, signal processing, information theory and speech processing (Gamer- man and Lopes) 2006). Recently, the notation of the Markov chain also plays an important role in reinforcement learning (Oiterlo and Wiering 2012), image generation (Ho et al.) 2020) and other deep learning-relie related fields (Mardt et al. 2018). Shwartz- Ziv and Tishby (2017) regards the feed- forward process of neural network Markov chain and explains the behavior of the neural network in the optimization process from the perspective of Shannon Information Theory.
B PROOF OF LEMMA. 3
Proof B.1 According to the forward pass, have
B B|| - y112.1 = E[[ - y + 21+1 21+1
(13)
|8| - ylli-1 + 2(x1 - y, 1) + || [[[ - ull2]+ + 2(xr - y, 1)) + z -² +
(1 26) y11?21
(14)
Now [E[[[=] v1 2 we know the B[||a - y11211 will decrease by a factor of (1 + 8) for every chain node. Otherwise, although it could increase, we know
Z
Blll= - y11 < 5
(15)
We know after L nodes, either B[ll=1 - y11 is already smaller than 2²
(1+a) log L2
or is
8
8
decreasing by factor of (1 + +) 5) for every node, which means
E[[ =
Fl[120 y11 +8)
12 S D2 5L (2t(l+a) log
L
(1 + a) log I.72 82 L
(16)
The last inequality holds since
D28²
La log L
(17)
a)
Thus, 11[1=1 yll?.] will be smaller than (1+a) log 1.22
L
C ANALYSIS OF ESTIMATION 9z
Here, we give a detailed analysis of the estimation error of 9z, in Eq. 12
Firstly, we strictly follow the chain rule derive the accurate formula of 9z,
In order to simplify the derivation process, use the F-norm function to measure the distance hetween each state na and the target y without losing the generality.
12

---

Published as a conference paper at ICLR 2023
M,,, (LL.
0z1
a
I2
+
Ox
21
dz
21)
Ø11Σ==
01
+
m, Fn,Y11
9m
02
dzi
@(11 + (5, -
-9o + X
Ō21
0(11 + 77( azllla Im.) 1
9m + X
Zz
0
7
92 + 2X(1 +
-g ))
||
1
(18)
|
where
0 21
79
A ==
dz1
(2t ---++17
||
n9 1122112 2 + | 211, -
mm
|
cngu; + (2 -1gm.)
(19)
is defined in Eq.
11
Then substitute Eq. 19 into Eq. 18
||l||
9z, == 9z, + 2A(1 + 1| -19,,,) ( engar + + (21, - -)))))
Jm. + 2A(1 (cz4, 70m.) ( cn + c2 )
= Jx. + 2X(1 cntt( - c² mt22)
(20)
where / = (21, T..) Eq. 20 can be further reformulated as:
9z1 == 921 + 2A(1 - cott)(-cnā.. e² nttz)
cspl
= (1 2m7 1921112 )gn, + 2 1)21
(21)
Hence, the estimated eтror term can be calculated by subtracting Eq. 12 from Eq. 21 which is:
= 2en ent
119,, 1l2 Ia. + (2n)Ac - 1) -)=
(22)
Since À and !) are very close to zero, the influence of the first term in right be ignored, so the second term mainly contributes to the estimation error. When we choose a suitable hyper-parameter that makes 7 2ndc@((cn) - 1) hold, the estimation error could be abysmally close zero. Empirically, the value of 7 varies significantly across different tasks.
D THE QUALITY OF THE PROPOSED IDEAL DIRECTION
There are many ways to define an ideal direction. As shown Fig. our proposed ideal direction calculation approach will satisfy the definition in most situations.
13

---

Published as a conference paper at ICLR 2023
f(x) n9x. y) -x(x y)
0.1
0.0
0.2
0.3
2000
4000
6000
8000
10000
12000
14000 16000
Figure 5: The difference between l(m Maa v) f(2,, y) during training in the toy model. Under more than 70.0% of simutions, {( 119a,, <
( strictly holds.
Original point
Lu
Xt-1
Target point
Xi for Res Net
X1 for VGG
Figure 6: Advantage of ResNet over VGG under Markov chain's guide. As long Cr + xr) a 8-convex chain, the chain formed by ResNet more efficient than the one formed by VGG.
E ADVANTAGE OF RESNET OVER VGG UNDER MARKOV CHAIN'S GUIDE
Lemma Cr + FT) is 8-convex, the chain C' formed by ResNet is more efficient than the chain C" formed by VGG, formally:
(23)
where e' is the efficiency of ResNet chain and e" is the efficiency of VGG chain.
Proof E.1 Without losing generality, will discuss only node here. As shown in Fig. 6 since 8-convex, Fi will always get closer to the target compared with Z1-1 and the original point. Hence,
(1-1, di) 20
(24)
holds.
Suppose the parameters 02 are the same for ResNet and VGG, then:
1 = 21 X1-1
(25)
x'l Zis
(26)
where 21 = fo, (2101). x1 and x² are the next chain node for ResNet and VGG respectively.
As for chain node xı-1, the estimated direction for ResNet is ž and the estimated direction for VGG is z 71-1. Therefore,
c' = (21, dz)
(27)
c" (21 II-
(28)
14

---

Published as a conference paper at ICLR 2023
According Eq. 24 have:
(01-1, di) 2 0
(2t, di) (2t, di) + (01-1, dj)
(21, di) (21 I1-1; d1) 2 0
(21, di) 2 (2) -1-1, di)
2 e"
(29)
Proofed.
Lemma. indicates that ResNet can form more efficient Markov chain compared with VGG, and leads to better performance.
F MORE DISCUSSION ON MODEL DEGRADATION
As shown Fig. 7 the randomly initialized Markov chain is prone to move in zigzags and even turn back as the chain goes longer (i.e. the model going deeper), which hinders the model from fitting target distribution efficiently, probably resulting in worse performance. When we apply penal connection enforce the network to be efficient Markov chain, the turn-back chain nodes do exist so each node takes at least non-newe non-negative effect whatever how long the chain is. As result, it could alleviate the model degradation problem.
(a) Folding Markov chain.
(b) Unfolding Markov chain.
Figure 7: Structural comparison between a folding Markov chain (a) and an unfolding Markov chain (b).
15


Reviews:
[{"review_id": "PqcRFXQVM4", "reviewer": "Reviewer_Z5HW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper reformed skip connections into a learnable Markov chain and proposed a simple routine of penal connection to make any residual-like model become a learnable Markov chain. The ablation analysis and the compared experiments showed that the proposed method achieves more competitive results than the SOTA methods. Overall, this paper has a solid theoretical foundation and novel method.", "summary_of_the_review": "This paper reformed skip connections into a learnable Markov chain and proposed a simple routine of penal connection to make any residual-like model become a learnable Markov chain. The ablation analysis and the compared experiments showed that the proposed method achieves more competitive results than the SOTA methods. Overall, this paper has a solid theoretical foundation and novel method. I think it can be accepted.", "strength_and_weaknesses": "Strengths:\nThis paper has a solid theoretical foundation and novel method. \nWeaknesses:\n1、\tThe main contribution of this paper has not been summarized.\n2、\tIn Fig.1, f_{\\theta_{l}} is a residual-like block. Can you explain the difference between f_{\\theta_{l} and residual block?\n3、\tHow to understander the definition of ideal direction? The relationship between ℓ(a, c) ≥ ℓ(µa+ (1−µ)b, c) ≥ ℓ(b, c) and Eq.(1).\n4、\tIn Fig.4, can you plot the curves about ℓ(xl − ηgxl , y) and  ℓ(xl , y)  to illustrate ℓ(xl − ηgxl , y) < ℓ(xl , y) always holds?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M1-Lqz9bJ60", "reviewer": "Reviewer_CKAs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper rethinks the skip connection model as a learnable Markov chain and proposes a penal connection to convert a residual-like model to a Markov chain for more efficient training. Experiments on MLP and CV demonstrate the superiority of this plug-in method.", "summary_of_the_review": "- This paper missing the comparison with the previous close-related methods.\n- This paper needs to compare their performance on a better-implied baseline.", "strength_and_weaknesses": "Pros\n\n- This paper formulated models with skip connections as a learnable Markov chain. And introduced the ideal direction for improving the efficiency of the Markov chain.\n- The proposed penal connection is a plug-in operation and the experiment evaluation the efficiency of this method.\n\nCons\n\n- Previous works [1,2] have already formulated the VGG-like models with the Markov chain, can you explain the advantage of ResNet over VGG under the Markov chain’s guide?\n- The compared models’ performance in Table 2 is lower than their original paper, can you explain why, and can the proposed method promote their SOTA implementation?\n\n[1] Opening the black box of Deep Neural Networks via Information \n\n[2] Markov Chain Neural Networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4J6TubtkV-", "reviewer": "Reviewer_Pv9F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper indicates that a model with residual-like bocks can also be considered as a learnable Markov chain.\nThey prove that a Markov chain can shift source input to the target domain through L nodes in an efficient way if the Markov chain is $\\delta$-convex.\nBy solving a Markov chain optimization problem they introduce the panel connection method, which is acting like a model regularization.\nFinally, they conduct experiments to demonstrate the superiority of the Markov chain.", "summary_of_the_review": "This paper rethinks the skip connection model as a learnable Markov chain and proposes a novel penal connection for better optimization.\nMost of the theorem and experiments are satisfied. However, there are still some concerns as written in \"Strength And Weaknesses\".", "strength_and_weaknesses": "Strength:\n\n[1] Interpret the skip connection with the help of the Markov chain is reasonable and innovative.\n\n[2] When exploring the optimization of the Markov chain, the gradient to $z_l$ seems to add a simple penalization on the $z_l$, which is interesting and easy to use.\n\n[3] This paper evaluated their method on both Natural Language Processing and Computer Vision. Results show that their method improves the performance and the convergence of deep neural networks.\n\nWeaknesses:\n\n[1] Can authors analyze why the proposed method can solve the model degradation problem? Will the penal connection make the convolutional layer more sparsity, and can other regularization methods (i.e. $L_1$ Norm) solve this problem?\n\n[2] Experiments on more deeper networks (i.e. ResNet101, ResNet152, ResNet110, ResNet1202) are needed to evaluate for demonstrating that the Markov chain can better solve the model degradation problem than the residual-like model.\n\n[3] I have noticed ReLU is put before Linearly Layer in Figure 1 in the supplementary materials, so the model used in the main paper is whether ResNets or PreActResNets?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4HSDyC6XfN", "reviewer": "Reviewer_Geje", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper formulates skip connection based models a learnable Markov chain. The paper then introduces the content of an efficient Markov chain which maps data from the input to the target domain efficeintly. A simple regularisation scheme is introduce to enable the learning of efficient Markov chains. Evaluation is performed on Transformer and ResNet type models. ", "summary_of_the_review": "The paper is interesting, however it is not clear if higher \\epsilon always leads to better test accuracy and whether having a higher \\epsilon is actually necessary. Moreover, the quality of the approximation of \\epsilon considered in the paper should be quantised with an upper bound of the error.", "strength_and_weaknesses": "Strengths:\n* The Markov Chain perspective on skip connection type models is novel and interesting.\n* The paper includes useful illustrations e.g. Fig 2 and Fig 3, which make the paper clearer.\n* The paper includes encouraging results on top-1 accuracy on ImageNet-1k across several state of the art models.\n\nWeaknesses:\n* Central claims of the paper are not clear: Consider the “inefficient” Markov chain in Fig 2 (a). Both the inefficient and efficient Markov chains successfully convert data from the source to the target domain. In fact, for a classification type model there should be no difference in the finally test accuracies of both efficient and inefficient models. If learning an efficient Markov chain is hard, it is not clear what is the advantage of an efficient Markov chain. Perhaps the paper wants to claim that efficient Markov chains leads to better fit to the target distribution? \n* In Proof 2.3, the paper derives only an c value for \\epsilon. What is the quality of the approximation? Are there any upper bounds to the absolute error? This is important because the approximation has a direct impact on the quality of the proposed regularizer. (The steps in Proof 2.3 should be numbered).\n* It is not clear if a higher value of \\epsilon leads to a better model? The paper should show better evidence for this. In Fig 4 (b) the difference in \\epsilon between the two models is minimal. In fact, the value of \\epsilon peaks early in training, but is not accompanied by peak accuracy?\n* The paper should also consider including additional state of the art baselines for the machine translation task, as currently only the plain Transformer is considered.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster