Double Continuum Limit of Deep Neural Networks

The continuum limit is an effective method for modeling complex discrete structures such as deep neural networks to facilitate their interpretability. The continuum limits of deep networks are investigated with respect to two directions: width and depth. The width continuum limit is a limit of the linear combination of functions, or a continuous model of the shallow structure. We can understand that what shallow networks do is the ridgelet expansion of their approximating functions. The depth continuum limit is a limit of the composition of functions, or a continuous model of the deep structure. We can understand that what deep networks do is to transport mass to decrease a certain potential functional F of the data distribution. A discretization method can potentially replace the backpropagation. Specifically, we can synthesize a deep neural network from broken line approximation and numerical integration of a double continuum model, without backpropagation. In this study-in-progress, we have developed the ridgelet transform for potential field, and synthesized an autoencoder without backpropagation. In this paper, we review recent developments of the width and depth continuum limits, introduce our results, and present future challenges. 1. Depth Continuum Limit The continuum limit with respect to depth is a recently developed technique. It is a continuum analogy of the deep neural network, formulated as a formal limit of the composition of functions as below: φL ◦ · · · ◦ φ1(x) → φt=T (x), (1) where φ` : H` → H`+1 (` = 1, . . . , L) with feature vector space H` is a feature map defined by the `-th hidden layer, Equal contribution Waseda University, Tokyo, Japan. Correspondence to: Sho Sonoda <sho.sonoda@aoni.waseda. jp>. Presented at the ICML 2017 Workshop on Principled Approaches to Deep Learning, Sydney, Australia, 2017. Copyright 2017 by the author(s). and φt : R → R (t ∈ [0, T ]) is a continuous analog of φ`’s. In the following subsections, we identify φt as a transport map with depth as time variable t. We then obtain a finite deep neural network as a broken line approximation of trajectory t 7→ φt(x). The dynamical system viewpoint is not new to recurrent neural networks (Seung, 1998). These days, distributionbased formulations of deep neural networks are successful. For example, they are generative density estimators (Bengio et al., 2013), variational autoencoder (Kingma & Welling, 2014), reverse diffusion process (Sohl-Dickstein et al., 2015) and adversarial generative networks (Goodfellow et al., 2014). In shrinkage statistics, the expression of “transport map” x+ f(x) is known as Brown’s representation of posterior (George et al., 2006). Liu & Wang (2016) analyzed it from a Bayesian viewpoint, apart from deep learning, and proposed kernel Stein discrepancy. Recently, we have observed that some kinds of convolutional networks can also be regarded as transport maps. Specifically, the skip connection structure x+f(x) used in highway networks (Srivastava et al., 2015) and ResNet (He et al., 2016) is formally understood as a transport map. 1.1. Wasserstein Gradient Flow of Deep Network Consider a transport map defined by a velocity vector field ∂tφt(x) = ∇Vt(φt(x)), x ∈ R (2) with time-dependent potential function Vt : R → R, where ∇ denotes the gradient operator on R. When t is small, it has an “explicit” expression φt(x) = x+ t∇Vt(x) + o(t) as t→ 0. (3) Generally, (2) is rewritten as an integral equation