Title: DEEP LIPSCHITZ NETWORKS AND DUDLEY

Abstract: Generative adversarial networks (GANs) have enjoyed great success, however often suffer instability during training which motivates many attempts to resolve this issue. Theoretical explanation for the cause of instability is provided in Wasserstein GAN (WGAN), and wasserstein distance is proposed to stablize the training. Though WGAN is indeed more stable than previous GANs, it takes more iterations and time to train. This is because the ways to ensure Lipschitz condition in WGAN (such as weight-clipping) significantly limit the capacity of the network. In this paper, we argue that it is beneficial to ensure Lipschitz condition as well as maintain sufficient capacity and expressiveness of the network. To facilitate this, we develop both theoretical and practical building blocks, using which one can construct different neural networks using a large range of metrics, as well as ensure Lipschitz condition and sufficient capacity of the networks. Using the proposed building blocks, and a special choice of a metric called Dudley metric, we propose Dudley GAN that outperforms the state of the arts in both convergence and sample quality. We discover a natural link between Dudley GAN (and its extension) and empirical risk minimization, which gives rise to generalization analysis.

1. INTRODUCTION:

Generative adversarial networks (GANs) (Goodfellow et al., 2014) are recently proposed powerful generative models that use two neural networks: (1) a generator network that produces fake samples that tries to look as real as possible; and (2) a discriminator network that seeks to distinguish fake and real samples.

The discriminator typically utilizes a divergence measure between the distributions of the reals and fakes. Early works of GANs are shown to correspond to Jensen-Shannon divergence (Goodfellow et al., 2014) , and later other divergence measures are introduced to GANs such as f-divergences (Nowozin et al., 2016) . These divergence based GANs often suffer from training instability, and several heuristic tricks have been used to tackle this issue such as batch normalization and careful network initialization. Arjovsky et al. (2017) point out at theoretical level that the use of divergence has several flaws such as infinite divergence and vanishing gradient when the two distributions have no common supports (intersection measure zero). They propose to use earth mover's distance which does not need common support. Directly minimizing the earth mover's distance can be expensive, thus they resort to solve its dual wasserstein measure instead. They use several tricks such as weight clipping (Arjovsky et al., 2017) and constrain the norm of the gradient (Gulrajani et al., 2017) to ensure Lipschitz condition in neural nets. However, these significantly limit the capacity of the network and reduce the convergence speed.

Ensuring Lipschitz condition in neural nets are the true cornerstone of stablizing GANs. GANs are sensible to Lipschitz constant: a small value leads to a simpler function with low capacity however a larger value can be unstable to train. This is why bounded Lipschitz functions are desirable. Neural networks use complex functions composed of layered nonlinearities, thus analysis of the Lipschitz condition is highly nontrivial. This is why we develop building blocks to facilitate easy construction of new GANs that always ensure Lipschitz condition.

With the proposed building blocks, and Dudley metric, we propose a new Dudley GAN which outperforms the state-of-the-arts due to ensuring both Lipschitz condition and sufficient network capacity. We further show that it can be extended to a class of metrics called integral probability metric (IPM) (Müller, 1997; Sriperumbudur et al., 2012) , of which wasserstein measure (used in WGAN (Arjovsky et al., 2017; Gulrajani et al., 2017) ), maximum mean discrepancy (used in various GANs (Mroueh et al., 2017; Karolina Dziugaite et al., 2015; Sutherland et al., 2016; Lim & Ye, 2017) ) and Dudley metric are special cases. Viewing GANs in IPM, we can view our approach as minimizing a "function norm" subject to best discrimination.

Our contributions in this paper are as follows: (1) we provide sufficient conditions and building blocks for a Lipschitz neural net (Section 2); (2) we introduce Dudley GAN and show utilizing minimum bounded Lipschitz functions with appropriate regularization for GANs are effective in producing good quality images with stable training and faster convergence (Section 3); (3) we show that our Dudley GAN (and its extension) has a natural link to regularized empirical risk minimization, and provide a generalisation bound on error of the discriminator (Section 4) using Rademacher complexity.

2. DEEP LIPSCHITZ NETWORKS:

Lipschitz condition is crucial for GANs, and for neural networks in general, here we provide sufficient conditions for a deep neural network to be Lipschitz.

2.1. THEORETICAL ASPECT:

A function f is Lipschitz (or Lipschitz continuous) on a space X if there exists a constant

K ≥ 0 such that f (x 1 ) -f (x 2 ) ≤ K x 1 -x 2 for all x 1 , x 2 ∈ X . K is known as a Lipschitz constant.

Similar conditions can be found in other communities, e.g. control theory (Fromion, 2000) . Definition 1 (Lipschitz semi-norm). Lipschitz semi-norm of a function f on a space X is

f L := sup |f (x1) -f (x2)| |x1 -x2| : x1 = x2∀x1, x2 ∈ X .

f L is in fact the smallest Lipschitz constant of f (also known as the Lipschitz constant for short). This means that every differentiable function with bounded first derivative is Lipschitz with Lipschitz constant f L = sup |f (x)|, and a function is Lipschitz when its Lipschitz semi-norm is defined everywhere. Definition 2. [Lipschitz Neural Network] A neural network represents a function f w : X → Y that maps from input space X to output space Y parameterized by a set of parameters w. We say the neural network is a Lipschitz Neural Network if f w L is defined everywhere in its input space.

By Definition 2, we know that a single activation function such as Perceptron is a Lipschitz neural network if that activation function is Lipschitz. Proposition 1. For two Lipschitz neural nets f w1 and g w2 parameterized by w 1 , w 2 with Lipschitz constant L f and L g respectively, we have: (i) Convolution is an operation that satisfies Lipschitz condition on a compact space with Lipschitz constant K when |w| = K for d-dimensional1 weight w or Kd where w = K.

(i) linear combination h w (x) = f w1 (x) + g w2 (x)

(ii) A linear function with weight vector w where w = K satisfies Lipschitz condition with Lipschitz constant K.

(iii) Following nonlinear activation functions: Sigmoid, tanh and Rectified Linear Unit (ReLU), exponential linear unit (ELU), sinusoid (sin), softplus defined as 1 β log(1 + exp(βx)).

(iv) log(x) where x ∈ [1, ∞).

There are other operations in neural networks such as batch and weight normalization (as linear transformations from a compact space to another) and dropout that are widely used and preserve Lipschitz.

2.2. IMPLEMENTING DEEP LIPSCHITZ NEURAL NETWORKS:

Above properties provide guidance for designing Lipschitz Neural Networks, and can be implemented in practice. Here we discuss a few practical tricks can be used or have been used, and relate them to the properties.

Weight Normalization (Salimans & Kingma, 2016a ): A simple reparameterization trick for linear and convolutional layers, shown to be extremely effective in practice. It is an efficient approach for ensuring the weights are normalized and the network is Lipschitz. It should be noted that the learnable scale parameter (equaling the norm of parameter vector w ), is not guaranteed to be constant throughout training. As such, the Lipschitz constant of the network is learned.

L 2 -Regularization: Another approach is to penalize the weight values with large L 2 norm. It provides a soft constraint on the norm of the weight vector.

Batch normalization (Salimans & Kingma, 2016b) : Another approach for normalizing the inputs so that they are all normally distributed with mean zero. The mean and standard deviation is optimized during training and is shown to perform well as a regularizer. For Lipschitz neural networks, it ensures the output of the batch normalized layer is bounded which effectively constrains the weights and gradients. In practice, Xiang & Li (2017) showed weight normalization is more effective for GANs than batch normalization.

Gradient norm regularization and clipping: Since Lipschitz condition limits the rate of change in functions corresponding to the gradient in continuous functions, it is reasonable to consider regularizing the gradient. In general we don't need any gradient regularization as the Lipschitz neural nets ensure bounded gradients. However, in practice this value may be large and incur sudden changes in the network that can lead to divergence. Clipping the gradient is simple however might lead to divergence because it changes the direction in which the parameters of the network are updated. A better choice is to scale the gradients by their norm when the gradients are too large. We found this scaling can be effective in practice when the data is too complex and batches in Stochastic Gradient Descent (SGD) optimization vary which cause sudden changes in the gradient.

Weight normalization in combination with L 2 regularization enable parameters to be updated smoothly when used with variations of SGD in back-propagation. Additional L 2 regularization ensures new parameters of weight normalization are also regularized. van Laarhoven (2017) provides an extensive analysis of the combination of these two.

3. DUDLEY GANS:

With the guidance of the properties presented and a choice of a special metric called Dudley metric, we develop a new GAN called Dudley GAN, which outperforms the state of the arts in both convergence and sample quality, due to ensured Lipschitz and sufficient network capacity. We then show that Dudley metric belongs to a broader class of metrics that can be integrated nicely with GANs. This opens a door to create new GANs with desirable theoretical and practical properties.

3.1. BOUNDED DEEP LIPSCHITZ NEURAL NETWORKS FOR GANS:

Let p be the true data distribution where the real samples are drawn from, and q θ be the fake data distribution represented by the generator network parametrised by θ. We only have access to the real samples but not p. Generating a fake sample x ∼ q θ is typically done by drawing a random vector z from a simple and known distribution p z first, and then feed z into the generator network which outputs the fake same, that is x = g θ (z). Here g θ is a deterministic function that maps the generator network's input to its output. In the ideal case, q θ and p are "equal" with respect to the divergence measure of choice (that is the distance becomes zero). The discriminator utilizes the divergence measure and provides feedback to the generator.

Let f w be the function that maps the input of the discriminator to its output with parameter w. We propose to use integral probability measure (Müller, 1997) below for the generator:

EQUATION

where F is a class of real-valued measurable bounded functions on X . Setting F = f w :

f w BL ≤ 1 in Equation 1 where . BL is defined as f w BL = f w ∞ + f w L with f w ∞ = sup |f w (x)| :

x ∈ X produces a metric called Dudley metric (Müller, 1997) . Effectively, this metric constrains the function to be both bounded and Lipschitz. It is worth noting that Dudley metric is a weaker convergence measure compared to the Wasserstein distance. However, for GANs weaker measure is more desirable as it deters the discriminator from saturation (Liu et al., 2017) . We consider the following cases of bounded Lipschitz functions detailed below.

Constrained Linear: We set the function f w to be both Lipschitz and bounded in the output. As such we have the following objective:

max Ex∼p[fw(x)] -E x ∼q θ [fw(x )], subject to -γ ≤ fw(x) ≤ γ ∀x, 0 < γ < 1

Alternatively, this problem can be written as the following unconstrained objective

ρL 2 (p, q θ ) = max min λ Ex∼p[fw(x)] -E x ∼q θ [fw(x )] -λR(w), R(w) = ( fw -γ) 2 , (2)

where we have a soft constraint with λ as the Lagrangian multiplier. Here, f is the vector constructed from the samples of p and q θ . In fact we approximate the . ∞ constraint for the functional norm with the L 2 norm of the function values for samples from both distributions. At each step of the optimization, in addition to the parameters of the discriminator and generator, we need to update the Lagrangian multiplier.

Using tanh: Another alternative is to use a function with limited output while the network is Lipschitz. We use tanh as a popular bounded activation function, i.e.

ρ tanh (p, q θ ) = max Ex∼p[fw(x)] -E x ∼q θ [fw(x )], fw(x) := tanh(hw(x))

where h w is a Lipschitz neural network mapping the input x to a real number. With tanh as the output of the network, we might suffer from the vanishing gradient. Considering the derivative of tanh, that is 1 -tanh(x) 2 for a real value x, the gradient is at its peak at tanh(x) = 0. When the output is close to ±1 for either real or fake samples, the gradient approaches zero and as such there would be no more improvement on the discriminator or more importantly on the generator. As such, we directly constrain the value of the function to be close to zero. In particular, we use

EQUATION

where f tanh is the vector of the function values for the samples from the real/fake distribution.

There are other cases that we could consider, however, we observed in practice constraining the rate of improvement with Lipschitz condition as well as bounding the output provides the most stable algorithm for GANs. For instance, using tanh as the output of WGAN has been tested when the network is not conditioned to be Lipschitz (Gulrajani et al. (2017) ) and it is reported that the algorithm diverged (as we further observed in practice).

The optimal generator is obtained when we have:

∇ θ ρ(p, q θ ) = ∇ θ (Ex∼p[fw(x)] -Ez∼p z [fw(g θ (z))]) = -Ez∼p z [∇ θ fw(g θ (z))] = 0

Putting all together, we have our algorithm detailed in Algorithm 1. In this algorithm we use Adam (Kingma & Ba (2014)) that generally is considered a fast and stable optimization.

3.2. INTEGRAL PROBABILITY METRICS:

Integral probability metrics (IPM) (Müller, 1997; Sriperumbudur et al., 2012) 

ρ w = ∇ w 1 n i f w (x i ) -1 n i f w (g θ (z i )) {f w is Linear or tanh} 5:

R w = ∇ w R(w) {Equation 2 or 3} 6:

w = w + η D .Adam(w, ρ w -λR w ) 7:

Sample z 1 , . . . , z n ∼ p z 8:

ρ θ = -∇ θ 1 n i f w (g θ (z i )) 9: θ = θ -η G .Adam(θ, ρ θ ) 10:

λ = λ -η λ R w 11: end while Algorithm 1: The algorithm for the Dudley GAN. In this algorithm, are the learning rates of discriminator, generator and Lagrangian multiplier.

Wasserstein distance: Setting F = f : f L ≤ 1 in Equation 1. The Kantorovich-Rubinstein theorem states that for separable space X , this metric in Equation 1 with Lipschitz condition is a dual for L 1 -Wasserstein distance. This measure is used in WGAN.

Total Variation: Setting F = f : f ∞ ≤ 1 yields popular total variation distance. For example, EBGAN (Zhao et al. ( 2016)) uses a similar condition. It should also be noted that when using Lipschitz functions on a compact spaces (such as images) with both positive and negative values, we always have a bounded function, i.e.

f ∞ = sup x |f (x)| ≤ sup x,x |f (x) -f (x )| ≤ diam(X ) f L,

where diam(X ) is the maximum distance between points in the input. Therefore, while bounding the output does not significantly change the GAN framework, it does provide a practical advantage as well as theoretical (as will be further discussed in Section 4).

We concentrate on bounded Lipschitz neural networks because only using bounded functions (not Lipschitz) for GANs might suffer from the vanishing gradient problem and limit the capacity of the output. Secondly, using MMD typically requires a choice of kernel.

Moreover, it is interesting to note that in an IPM when f (x) = φ(x), w for some function φ(x) and w ≤ 1, we recover MMD. Therefore, for a neural network with linear output where φ(x) represents one layer to the last and is weight-normalized (no need for the previous layers to be Lipschitz) another alternative for training a GAN is obtained.

4. GENERALIZATION BOUNDS:

Another benefit of introducing IPM to GANs, is that it provides a natural link between GANs and empirical risk minimization (ERM). If we assign a label y = +1 for real samples and y = -1 for fake ones, it is easy to see the relation between IPM and ERM as a binary classification when, without loss of generality, we have p(y = +1) = p(y = -1), then For (y, f w (x)) = -yf (x), p(x) = µ(x|y = +1), q θ (x ) = µ(x |y = -1) we recover the objective in Equation 1. Hence, it is easy to see the discriminator's objective is a regularized ERM: min w E (x,y)∼µ(x,y) [ (y, f w (x))] + R(w). Although in practice, the complete minimization is not performed in each iteration of GAN.

It can be seen (Sriperumbudur et al., 2012) that the dual of this problem minimizes the norm (depending on the choice of the norm in IPM) of the function at hand, in our case f BL . In other words, we learn the smoothest function (lower f L ) with minimum highest value ( f ∞ ). We need more complex functions (higher Lipschitz constant) when the divergence distance is small (i.e. distinguishing closer distributions are more difficult). Since in GANs we learn the fake distribution q θ it is important to have more complex models for f w , an indication of mixture of fake and real samples. Thus, constraining the Lipschitz constant to be very small is in fact a deterring factor in estimating the fake distributions from the generalization perspective.

Finally we add that learning the function f w is relevant to the Rademacher complexity where we sample from the real (σ = 1) and fake (σ = -1) distributions with the same probability ( 12 ) and evaluate the empirical Rademacher complexity (Shawe-Taylor & Cristianini, 2004) ,

R = Eσ sup f ∈F 1 m m i=1 σif (xi) , xi ∈ X ∪ X , X = {g(z1) . . . , g(zn)} , zi ∼ p(z)

that measures the random labeling of the data. This simple measure allows us to determine how easy it is for our model to distinguish real from fake samples when we randomly label them. Moreover since our function is bounded, with probability at least 1 -δ for δ ∈ (0, 1) over m samples from X ∪ X , we have a bound on the probability of misclassification of fake/real samples,

p [y = sgn(f (x))] ≤ 1 mγ m i=1 (γ -yf (x))+ + 4 diam(X ) mγ 2 + 3 log(2/δ) 2m (4)

As observed, the maximum confusion in the discriminator that is desirable for GANs is inversely related to the value of γ. The proof is provided in the supplements. Examining generative models are generally challenging. In this section, we examine the quality of the generated images using Dudley GAN. To that end, we construct a Lipschitz neural network with the bounded output detailed in Section 2. We build upon the architecture of DCGAN (Radford et al. (2015) ) with an additional linear layer on top and weight-normalization in each layer of the discriminator. The network architecture is detailed in Appendix B. We apply batchnormalization in the generator that stabilizes and improves convergence speed. We set the learning rate to 0.0001, weight decay to 0.001, batch size 64 and 100-dimensional noise samples from normal distribution. We, furthermore, compare with gradient clipping where we clip gradient values exceeding value 2 and gradient decay where we normalize the gradients so that its norm is less than 10. For WGAN we set the weight clipping range to [-0.01, 0.01] as is done in the original paper. We will use the same experimental setup throughout the evaluation with various datasets.

5. EXPERIMENTS:

For the original GAN and its variants where JS-divergence is used, batch normalization is essential for providing common support for the real and fake datasets. Otherwise, it takes a long time to converge. Batch-norm provides a simple practical trick to encourage high correlation between fake and real samples. When we use WGAN, batch normalization helps to induce a compact space after each layer. In Dudley GAN, batch-norm is not required.

Algorithm LL (nats) WGAN (Arjovsky et al., 2017) MNIST Dataset: This dataset contains 60, 000 hand-written digits formatted as binary 28 × 28 images. We use Algorithm 1 to train a generator. Samples from the generator is shown in Figure 1 (a) for linear and 1(b) for tanh at iteration 5000. Moreover, we use kernel density estimation (KDE) to estimate the log-likelihood of the generated samples at iteration 10, 000 similar to Nowozin et al. (2016) . As is common practice, we have taken 16K samples from the trained generator and report the mean log-likelihood in Table 5 . Even though KDE is not an excellent measure and has high variance, we compared the log-likelihoods of variations approaches. As observed, tanh with Lipschitz network outperforms tanh with no weight normalization (corresponding to the total variation). The performance of the bounded linear function with regularization that is proposed in the paper has the best log-likelihood CIFAR-10 Dataset: The CIFAR-10 dataset Krizhevsky & Hinton (2009) is composed of 50, 000 RGB images of natural scenes with size 32 × 32. We train our GAN similar to MNIST except that here we are using three channels. The progress of the generator is shown in Figure 2 . It is interesting to observe that weight clipping exhibits relatively fast initial improvement however our approach catches on quickly. This is because as the time passes the capacity of the discriminator is exploited and the frequency of the weights being clipped increases. As such, the rate of improvement in the network significantly decreases. In addition, when we use tanh for the output of a Lipschitz network, initially the improvement of the generator is slow however we observe a steady improvement and the end result exhibit better diversity and sharper quality compared to WGAN. In Figure 4 we show metrics from training of the CIFAR-10 dataset. In particular, in Figure 4 (a) the discriminator loss without the regularization is shown. As observed, tanh and bounded linear function show a stable value for the difference in Equation 1(empirical expected functions). However, when we use other regularizations, namely gradient clipping or gradient normalization we observe the discriminator loss is not stable. In addition, in Figure 4 (b) the norm of the gradient of the loss is plotted and as is observed generally gradient of the tanh is smaller than the bounded linear. This is because as the output of tanh increases the value of its gradient decreases, which is not the case with the linear function. One interesting case is when we use gradient clipping, the algorithm diverges which is because the crude change of the gradient alters discriminator in a wrong way. In Figure 4 (c), we use Inception score (Salimans et al., 2016) to evaluate the visual features of the generated images. Even though this is not a comprehensive measure, we observe our approach performs better than WGAN (and WGAN-GP). Furthermore, Rademacher complexity is also shown in Figure 4 (d). The functional value of the tanh is smaller and thus its Rademacher value, however the final value for the bounded linear is smaller.

In Figure 3 we show the distribution of the weights in the generator. As observed, when using weight clipping in the discriminator, the output of the generator is also very restricted. As such, the range 5 . Interestingly, our algorithm converges and produces good quality images at 40, 000th iteration. WGAN produces, with arguably worst quality, at iteration 60, 000. We observe, with batch-normalization linear bounded model is not as sharp as its alternative. Moreover, tanh does not show as much diversity in the generated images which we believe is due to the restricted nature of the output function.

CelebA Dataset: For the last experiment, we use the CelebA dataset Liu et al. (2015) . We generate 64 × 64 images as shown in Figure 6 . We use Softplus as another measure based on our framework where the divergence is interpreted as the difference of the log-likelihood of the observations from real and fake data (details are provided in Appendix C).

We observe even though the generated samples have good quality, Softplus converges slower than two other alternatives (takes almost twice the number of epochs).

6. CONCLUSION:

In this paper we develop the building blocks of Lipschitz neural networks using which and Dudley metric, we propose a new GAN called Dudley GAN. Dudley GAN's discriminator is guaranteed to be a bounded Lipschitz neural network, thus produces state of the art result in both converge and sample quality. In addition, we show Dudley GAN and its extension is directly related to empirical risk minimization. This allows us to bridge the gap between GANs and learning complexity and generalization. We provide an upper bound on probability of confusion between real and fake images. We believe this view opens new avenues in GAN research for better understanding and quantifying the ability of the algorithm to capture the distribution of real samples. A PROOFS

proof of 1 (ii). For two Lipschitz continuous functions f and g with L f and L g constants we have:

f (g(x 1 )) -f (g(x 2 )) ≤ L f g(x 1 ) -g(x 2 ) ≤ L f × L g x 1 -x 2

proof of 2(i). Let * denote convolutional operation between the input x and weight vector w. Using Young's convolution inequality we have, proof of 2 (ii). We have w, x 1 -w, x 2 ≤ w x 1 -x 2 and w = C.

proof of 2(iii). Sigmoid and tanh are continuously differentiable and their derivative is globally bounded, hence they are Lipschitz continuous. Considering all possible cases of the input to the ReLU function, it is easy to see ReLU(x 1 ) -ReLU(x 2 ) ≤ x 1 -x 2 .

proof of 2(iv). log is a differentiable function with bounded derivative in the specified domain.

Proof of Equation 4. Let's denote by H the heaviside function and A : R → [0, 1] as

A(a) =      1 if a > 0 1 + a γ if 0 ≤ a ≤ γ 0 otherwise (5)

From Theorem 4.9 in (Shawe-Taylor & Cristianini, 2004) and since H -1 ≤ A -1 we have,

EQUATION

and since A(-y i f (x i )) ≤ (yf (x)-γ)+ γ for i = 1, . . . , m, we have

EQUATION

and since the support of the distribution is in the ball centered on the origin, we have the inequality proven. 

:

In practice, d = 1 + (w -F )/s where s is the number of strides, w is the input volume and F is the receptive field (kernel size) of the input.

Figures and Tables:

Kernel Distance: Setting F = f : f H ≤ 1 yields Maximum Mean Discrepancy (MMD) distance. MMD has been a popular measure in GANs (Karolina Dziugaite et al. (2015); Sutherland et al. (2016); Lim & Ye (2017)).

y)∼µ(x,y) [ (y, fw(x))] = inf f ∈F (+1, fw(x))µ(x|y = +1) + (-1, fw(x))µ(x|y = -1)

Figure 1: Generated samples from MNIST dataset.

Figure 3: Weight distributions of the last layer of the generator when using weight clipping compare to other cases.As observed, the variance of the weights in the linear and is larger. This implies more diversity in the images produced. In the gradient clipping case where the weight has a very small variance our approach diverges.

Figure 4: CIFAR-10 training metrics albeit with larger variance. Lower variance in WGAN is due to the extremely restrictive nature of weight clipping.

Figure 2: Generated images at various steps of training. As observed weight clipping saturates very fast.

Figure 5: Images generated from the LSUN dataset.

Figure7: Various outputs for the neural net and their derivatives. For the original WGAN the derivative is constant, while the tanh has a gradient that decreases as the absolute value of the input increases.

* x 1 -w * x 2 ≤ |w| x 1 -x 2 , To establish the bound for the d-dimensional weight on |w| = i |w 1 | + . . . , |w d |, we consider two cases (1) |w i | ≤ 1 where the elements are upper bounded by d; (2) |w i | > 1 we know w 2 1 + w 2 2 + . . . , w 2 n = C 2 , then w 2 i ≤ C 2 and |w i | ≤ C. Thus, |w| ≤ Cd.

(a) linear (b) linear weight-clip (c) decayed linear (d) grad decayed linear (e) tanh (f) tanh grad clip

Figure 8: Figure

Figure 8: Figure

is a Lipschitz neural network with constant L f + L g and a new set of parameters denoted by w;

Table content: <table><tr><td>(ii) the composition function h w (x) = f w1 (g w2 (x)) is Lipschitz neural network with constant</td></tr><tr><td>L f × L g with a new set of parameters denoted by w.</td></tr><tr><td>Proposition 1 shows that a deep neural network defined by a composition of functions is a Lipschitz</td></tr><tr><td>neural network if the functions in all layers are Lipschitz. And the Lipschitz constant of the deep</td></tr><tr><td>neural network is the product of the Lipschitz constants of the functions in all layers.</td></tr><tr><td>Proposition 2. Applying following operations in a Lipschitz neural network preserves Lipschitz:</td></tr></table>

Peer review (iclr2018):
 It is clear that the problem studied in this paper is interesting. However, after reading through the manuscript, it is not clear to me what are the real contributions made in this paper. I also failed to find any rigorous results on generalization bounds. In this case, I cannot recommend the acceptance of this paper.