ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees

1 Introduction . Constructing deep neural network ( DNN ) models by stacking layers unlocks the field of deep learning , leading to the early success in computer vision , such as AlexNet [ Krizhevsky et al. , 2012 ] , ZFNet [ Zeiler and Fergus , 2014 ] , and VGG [ Simonyan and Zisserman , 2015 ] . However , stacking more and more layers can suffer from worse performance [ He and Sun , 2015 , Srivastava et al. , 2015 , He et al. , 2016a ] ; thus , it is no longer a valid option to further improve DNN models . In fact , such a degradation problem is not caused by overfitting , but worse training performance [ He et al. , 2016a ] . When neural networks become sufficiently deep , optimization landscapes quickly transition from being nearly convex to being highly chaotic [ Li et al. , 2018 ] . As a result , stacking more and more layers in DNN models can easily converge to poor local minima ( see Figure 1 in [ He et al. , 2016a ] ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address the issue above , the modern deep learning paradigm has shifted to designing DNN models based on blocks or modules of the same kind in cascade . A block or module comprises specific operations on a stack of layers to avoid the degradation problem and learn better representations . For example , Inception modules in the GoogLeNet [ Szegedy et al. , 2015 ] , residual blocks in the ResNet [ He et al. , 2016a , b , Zagoruyko and Komodakis , 2016 , Kim et al. , 2016 , Xie et al. , 2017 , Xiong et al. , 2018 ] , dense blocks in the DenseNet [ Huang et al. , 2017 ] , attention modules in the Transformer [ Vaswani et al. , 2017 ] , Squeeze-and-Excitation ( SE ) blocks in the SE network ( SENet ) [ Hu et al. , 2018 ] , and residual U-blocks [ Qin et al. , 2020 ] in U2-Net . Among the above examples , the most popular block design is the residual block which merely adds a skip connection ( or a residual connection ) between the input and output of a stack of layers . This modification has led to a huge success in deep learning . Many modern DNN models in different applications also adopt residual blocks in their architectures , e.g. , V-Net in medical image segmentation [ Milletari et al. , 2016 ] , Transformer in machine translation [ Vaswani et al. , 2017 ] , and residual LSTM in speech recognition [ Kim et al. , 2017 ] . Empirical results have shown that ResNets can be even scaled up to 1001 layers or 333 bottleneck residual blocks , and still improve performance [ He et al. , 2016b ] . Despite the huge success , our understanding of ResNets is very limited . To the best of our knowledge , no theoretical results have addressed the following question : Is learning better ResNets as easy as stacking more blocks ? The most recognized intuitive answer for the above question is that a particular stack of layers can focus on fitting the residual between the target and the representation generated in the previous residual block ; thus , adding more blocks always leads to no worse training performance . Such an intuition is indeed true for a constructively blockwise training procedure ; but not clear when the weights in a ResNet are optimized as a whole . Perhaps the theoretical works in the literature closest to the above question are recent results in an albeit modified and constrained ResNet model that every local minimum is less than or equal to the empirical risk provided by the best linear predictor [ Shamir , 2018 , Kawaguchi and Bengio , 2019 , Yun et al. , 2019 ] . Although the aims of these works are different from our question , they actually prove a special case under these simplified models in which the final residual representation is better than the input representation for linear prediction . We notice that the models considered in these works are very different from standard ResNets using pre-activation residual blocks [ He et al. , 2016b ] due to the absence of the nonlinearities at the final residual representation that feeds into the final affine layer . Other noticeable simplifications include scalar-valued output [ Shamir , 2018 , Yun et al. , 2019 ] and single residual block [ Shamir , 2018 , Kawaguchi and Bengio , 2019 ] . In particular , Yun et al . [ 2019 ] additionally showed that residual representations do not necessarily improve monotonically over subsequent blocks , which highlights a fundamental difficulty in analyzing their simplified ResNet models . In this paper , we take a step towards answering the above-mentioned question by constructing practical and analyzable block-based DNN models . Main contributions of our paper are as follows : Improved representation guarantees for wide ResNEsts with bottleneck residual blocks . We define a ResNEst as a standard single-stage ResNet that simply drops the nonlinearities at the last residual representation ( see Figure 2 ) . We prove that sufficiently wide ResNEsts with bottleneck residual blocks under practical assumptions can always guarantee a desirable training property that ResNets with bottleneck residual blocks empirically achieve ( but theoretically difficult to prove ) , i.e. , adding more blocks does not decrease performance given the same arbitrarily selected basis . To be more specific , any local minimum obtained from ResNEsts has an improved representation guarantee under practical assumptions ( see Remark 2 ( a ) and Corollary 1 ) . Our results apply to loss functions that are differentiable and convex ; and do not rely on any assumptions regarding datasets , or convexity/differentiability of the residual functions . Basic vs. bottleneck . In the original ResNet paper , He et al . [ 2016a ] empirically pointed out that ResNets with basic residual blocks indeed gain accuracy from increased depth , but are not as economical as the ResNets with bottleneck residual blocks ( see Figure 1 in [ Zagoruyko and Komodakis , 2016 ] for different block types ) . Our Theorem 1 supports such empirical findings . Generalized and analyzable DNN models . ResNEsts are more general than the models considered in [ Hardt and Ma , 2017 , Shamir , 2018 , Kawaguchi and Bengio , 2019 , Yun et al. , 2019 ] due to the removal of their simplified ResNet settings . In addition , the ResNEst modifies the input by an expansion layer that expands the input space . Such an expansion turns out to be crucial in deriving theoretical guarantees for improved residual representations . We find that the importance on expanding the input space in standard ResNets with bottleneck residual blocks has not been well recognized in existing theoretical results in the literature . Restricted basis function models . We reveal a linear relationship between the output of the ResNEst and the input feature as well as the feature vector going into the last affine layer in each of residual functions . By treating each of feature vectors as a basis element , we find that ResNEsts are basis function models handicapped by a coupling problem in basis learning and linear prediction that can limit performance . Augmented ResNEsts . As shown in Figure 1 , we present a special architecture called augmented ResNEst or A-ResNEst that introduces a new weight matrix on each of feature vectors to solve the coupling problem that exists in ResNEsts . Due to such a decoupling , every local minimum obtained from an A-ResNEst bounds the empirical risk of the associated ResNEst from below . A-ResNEsts also directly enable us to see how features are supposed to be learned . It is necessary for features to be linearly unpredictable if residual representations are strictly improved over blocks . Wide ResNEsts with bottleneck residual blocks do not suffer from saddle points . At every saddle point obtained from a ResNEst , we show that there exists at least one direction with strictly negative curvature , under the same assumptions used in the improved representation guarantee , along with the specification of a squared loss and suitable assumptions on the last feature and dataset . Improved representation guarantees for DenseNEsts . Although DenseNets [ Huang et al. , 2017 ] have shown better empirical performance than ResNets , we are not aware of any theoretical support for DenseNets . We define a DenseNEst ( see Figure 4 ) as a simplified DenseNet model that only utilizes the dense connectivity of the DenseNet model , i.e. , direct connections from every stack of layers to all subsequent stacks of layers . We show that any DenseNEst can be represented as a wide ResNEst with bottleneck residual blocks equipped with orthogonalities . Unlike ResNEsts , any DenseNEst exhibits the desirable property , i.e. , adding more dense blocks does not decrease performance , without any special architectural re-design . Compared to A-ResNEsts , the way the features are generated in DenseNEsts makes linear predictability even more unlikely , suggesting better feature construction . 2 ResNEsts and augmented ResNEsts . In this section , we describe the proposed DNN models . These models and their new insights are preliminaries to our main results in Section 3 . Section 2.1 recognizes the importance of the expansion layer and defines the ResNEst model . Section 2.2 points out the basis function modeling interpretation and the coupling problem in ResNEsts , and shows that the optimization on the set of prediction weights is non-convex . Section 2.3 proposes the A-ResNEst to avoid the coupling problem and shows that the minimum empirical risk obtained from a ResNEst is bounded from below by the corresponding A-ResNEst . Section 2.4 shows that linearly unpredictable features are necessary for strictly improved residual representations in A-ResNEsts . 2.1 Dropping nonlinearities in the final representation and expanding the input space . The importance on expanding the input space via W0 ( see Figure 2 ) in standard ResNets has not been well recognized in recent theoretical results ( [ Shamir , 2018 , Kawaguchi and Bengio , 2019 , Yun et al. , 2019 ] ) although standard ResNets always have an expansion implemented by the first layer before the first residual block . Empirical results have even shown that a standard 16-layer wide ResNet outperforms a standard 1001-layer ResNet [ Zagoruyko and Komodakis , 2016 ] , which implies the importance of a wide expansion of the input space . We consider the proposed ResNEst model shown in Figure 2 whose i-th residual block employs the following input-output relationship : xi = xi−1 +WiGi ( xi−1 ; θi ) ( 1 ) for i = 1 , 2 , · · · , L. The term excluded the first term xi−1 on the right-hand side is a composition of a nonlinear function Gi and a linear transformation,1 which is generally known as a residual function . Wi ∈ RM×Ki forms a linear transformation and we consider Gi ( xi−1 ; θi ) : RM 7→ RKi as a function implemented by a neural network with parameters θi for all i ∈ { 1 , 2 , · · · , L } . We define the expansion x0 = W0x for the input x ∈ RNin to the ResNEst using a linear transformation with a weight matrix W0 ∈ RM×K0 . The output ŷResNEst ∈ RNo ( or ŷL-ResNEst to indicate L blocks ) of the ResNEst is defined as ŷL-ResNEst ( x ) = WL+1xL where WL+1 ∈ RNo×M . M is the expansion factor and No is the output dimension of the network . The number of blocks L is a nonnegative integer . When L = 0 , the ResNEst is a two-layer linear network ŷ0-ResNEst ( x ) = W1W0x . Notice that the ResNEst we consider in this paper ( Figure 2 ) is more general than the models in [ Hardt and Ma , 2017 , Shamir , 2018 , Kawaguchi and Bengio , 2019 , Yun et al. , 2019 ] because our residual space RM ( the space where the addition is performed at the end of each residual block ) is not constrained by the input dimension due to the expansion we define . Intuitively , a wider expansion ( larger M ) is required for a ResNEst that has more residual blocks . This is because the information collected in the residual representation grows after each block , and the fixed dimension M of the residual representation must be sufficiently large to avoid any loss of information . It turns out a wider expansion in a ResNEst is crucial in deriving performance guarantees because it assures the quality of local minima and saddle points ( see Theorem 1 and 2 ) .

Summary: This work studies the representational power of Residual Networks and try to explain theoretically a desirable property of using ResNets, namely: does a residual network guarantee that stacking more residual blocks does not lead to a degradation in performance of the network. The main approach of the authors is to view the output of a ResNet as an ensemble of several shallow networks as opposed to a non-ResNet which is simply a very deep composition of non-linear functions. Given this view point, the authors identify some simplifying assumptions which make the associated Empirical Risk Minimization problem tractable: 1) introducing a sequence of "de-coupling" weights (H) which allow the weights (W) in the ensemble to be independent of each other 2) removing the non-linearity at the end of the residual block and before the final affine transformation and 3) a few assumptions on the loss function (differentiability and convexity). With these assumptions, the authors prove a few desirable properties of ResNet architectures. Firstly, the authors show that stacking more residual blocks cannot hurt in the representational sense, and can improve if the new feature is linearly independent of the other features in the ensemble basis. Secondly, the authors show that "wide ResNets" always attain the empirical risk minimization lower bound, which has an implication on the quality of local optima attained by the ResNets. The authors also prove analogous results for a related class of architectures (DenseNets), in particular showing that DenseNets may be approximated by wide ResNests.