SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {l = 1} ^ {L} \lambda \left(D f _ {l} \left(\theta_ {l}, f _ {<   l} (\vec {\theta}, X)\right)\right) \prod_ {j = l + 1} ^ {L} \lambda \left(J f _ {j} \left(\theta_ {j}, f _ {<   j} (\vec {\theta}, X)\right)\right) \tag {12}
$$
is a lower bound for  $\lambda \bigl (DF(\vec{\theta})\bigr)$
Proposition 4.6 tells us that to guarantee good regularity, it suffices to guarantee good regularity of the constituent layers. In fact, since Equation (12) is a sum of non-negative terms, it suffices merely to guarantee good regularity of the parameter-derivative of the first layer<sup>4</sup>, and of the input-output Jacobians of every subsequent layer. Our next theorem says that residual networks with appropriately normalised branches suffice for this.
Theorem 4.7. Let  $\{g_l : \mathbb{R}^{p_l} \times \mathbb{R}^{d_{l-1} \times N} \to \mathbb{R}^{d_l \times N}\}_{i=1}^L$  be a MPS and  $X \in \mathbb{R}^{d_0 \times N}$  a data matrix for which the following hold:
<sup>3</sup>The theorem also holds with normalised biases. We make the bias-free assumption purely out of notational convenience.
4The parameter derivatives of higher layers are more difficult to analyse, due to nonlinearities.
1.  $d_{l - 1}\geq d_l$  , and  $\| Jg_l(\theta_l,Z)\| _2 <   1$  for all  $\theta_l\in \mathbb{R}^{p_l}$ $Z\in \mathbb{R}^{d_{l - 1}\times N}$  and  $l\geq 2$
2.  $N \leq d_0$ ,  $X$  is full rank,  $p_1 \geq d_1d_0$  and  $f_1: \mathbb{R}^{p_1} \times \mathbb{R}^{d_0 \times N} \to \mathbb{R}^{d_1 \times N}$  is a  $P$ -parameterised affine layer, for which  $DP(\theta_1)$  is full rank for all  $\theta_1 \in \mathbb{R}^{p_1}$ .
For any sequence  $\{I_l : \mathbb{R}^{d_{l-1} \times N} \to \mathbb{R}^{d_l \times N}\}_{l=2}^L$  of linear maps whose singular values are all equal to  $I$ , define a new MPS  $\{f_l\}_{l=1}^L$  by  $f_1 := g_1$ , and  $f_l(\theta_l, X) := I_lX + g_l(\theta_l, X)$ . Let  $F : \mathbb{R}^p \to \mathbb{R}^{d_L \times N}$  be the parameter-function map associated to  $\{f_l\}_{l=1}^L$  and  $X$ . Then  $\lambda(DF(\vec{\theta})) > 0$  (but not uniformly so) for all  $\vec{\theta} \in \mathbb{R}^p$ .
In the next and final subsection we will synthesise Theorem 4.5 and Theorem 4.7 into our main result: a global convergence theorem for gradient descent on appropriately normalised residual networks.
5 Main theorem
We will consider normalised residual networks, which are MPS of the following form. The first layer is an  $\epsilon$ -entry-normalised, bias-free affine layer  $f_{1} = \mathrm{aff}_{\mathrm{en}}: \mathbb{R}^{d_{1} \times d_{0}} \times \mathbb{R}^{d_{0} \times N} \to \mathbb{R}^{d_{1} \times N}$  (cf. Example A.1). Every subsequent layer is a residual block
$$
f _ {l} \left(\theta_ {l}, X\right) = I _ {l} X + g \left(\theta_ {l}, X\right). \tag {13}
$$
Here, for all  $l \geq 2$ , we demand that  $d_{l-1} \geq d_l$ ,  $I_i: \mathbb{R}^{d_{i-1} \times N} \to \mathbb{R}^{d_i \times N}$  is some linear map with all singular values equal to 1, and the residual branch  $g(\theta_i, X)$  is a composite of weight- or entry-normalised, bias-free affine layers  $\mathrm{aff}_P$  (cf. Example A.1), rescaled so that  $\| P(w) \|_2 < 1$  uniformly for all parameters  $w$ , and elementwise nonlinearities  $\Phi$  for which  $\| D\phi \| \leq 1$  everywhere (Example A.2). These hypotheses ensure that Theorem 4.7 holds for  $\{f_i\}_{i=1}^L$  and  $X$ : see the Appendix for a full proof.
We emphasise again that our main theorem below does not follow from the usual argumentation using smoothness and the PL inequality, due to the lack of a uniform PL bound. Due to the novelty of our technique, which we believe may be of wider utility where uniform regularity bounds are unavailable, we include an idea of the proof below.
Theorem 5.1. Let  $\{f_i\}_{i=1}^L$  be a normalised residual network;  $X$  a data matrix of linearly independent data, with labels  $Y$ ; and  $c$  any continuously differentiable, convex cost function. Then there exists a learning rate  $\eta > 0$  such that gradient descent on the associated loss function converges from any initialisation to a global minimum.
**Idea of proof.** One begins by showing that Theorems 4.5 and 4.7 apply to give globally  $\beta$ -Lipschitz gradients and a positive smallest eigenvalue of the tangent kernel at all points in parameter space. Thus for learning rate  $\eta < 2\beta^{-1}$ , there exists a positive sequence  $(\mu_t = \mu \lambda (DF(\theta_t)))_{t\in \mathbb{N}}$  (see Theorem 2.4) such that  $\| \nabla \ell_t\|^2\geq \mu_t\ell_t$ , for which the loss iterates  $\ell_t$  therefore obey
$$
\ell_ {t} - \ell^ {*} \leq \prod_ {i = 0} ^ {t} (1 - \mu_ {i} \alpha) (\ell_ {0} - \ell^ {*}),
$$
where  $\alpha = \eta (1 - 2\beta^{-1}\eta) > 0$ . To show global convergence one must show that  $\prod_{t = 0}^{\infty}(1 - \mu_t\alpha) = 0$ .
If  $\mu_t$  can be uniformly lower bounded (e.g. for the square cost) then Theorem 2.3 applies to give convergence as in all previous works. However,  $\mu_t$  cannot be uniformly lower bounded in general (e.g. for the cross-entropy cost). We attain the general result by showing that, despite admitting no non-trivial lower bound in general,  $\mu_t$  can always be guaranteed to vanish sufficiently slowly that global convergence is assured.
We conclude this section by noting that practical deep learning problems typically do not satisfy the hypotheses of Theorem 5.1: frequently there are more training data than input dimensions (such as for MNIST and CIFAR), and many layers are not normalised or skip-connected. Moreover our Lipschitz bounds are worst-case, and will generally lead to learning rates much smaller than are used in practice. Our strong hypotheses are what enable a convergence guarantee from any initialisation, whereas in practical settings initialisation is of key importance. Despite the impracticality of Theorem 5.1, in the next section we show that the ideas that enable its proof nonetheless have practical implications.
5The theorem also holds with normalised biases.
6 Practical implications
Our main theorem is difficult to test directly, as it concerns only worst-case behaviour which is typically avoided in practical networks which do not satisfy its hypotheses. However, our framework more broadly nonetheless yields practical insights. Informally, Theorem 4.7 gives conditions under which:
skip connections aid optimisation by improving loss regularity.
In this section, we conduct an empirical analysis to demonstrate that this insight holds true even in practical settings, and thereby obtain a novel, causal understanding of the benefits of skip connections in practice. With this causal mechanism in hand, we recommend simple architecture changes to practical ResNets that consistently (albeit modestly) improve convergence speed as predicted by theory. All code is available at https://github.com/lemacdonald/skip-connections-normalisation/.
6.1 Singular value distributions
Let us again recall the setting  $\ell = \gamma \circ F$  of Theorem 2.4. In the overparameterised setting, the smallest singular value of  $DF$  gives a pessimistic lower bound on the ratio  $\| \nabla \ell \| ^2 /(\ell -\ell^*)$ , and hence a pessimistic lower bound on training speed (cf. Theorems 2.4, 2.3). Indeed, this lower bound is only attained when the vector  $\nabla \gamma$  perfectly aligns with the smallest singular subspace of  $DF$ : a probabilistically unlikely occurrence. In general, since  $\nabla \ell = DF^T\nabla \gamma$ , the entire singular value distribution of  $DF$  at a given parameter will play a role in determining the ratio  $\| \nabla \ell \| ^2 /\left(\ell -\ell^{*}\right)$ , and hence training speed.
Now  $DF$  is partly determined by products. As such, the distribution of singular values of  $F$  is a distribution of singular values of such layer  $J$ , determined by the singular value distributions
of layer Jacobians (cf. Proposition 4.2). of  $DF$  is determined in part by the discobian products, which are themselves deof each of the individual layer Jacobians.
Figure 1: Singular value histogram of  $500 \times 500$  matrix  $A$  (left) with entries sampled iid from  $U(-1/\sqrt{500}, 1/\sqrt{500})$ . Adding an identity matrix (right) shifts the distribution upwards.
In particular, if through some architectural intervention each layer Jacobian could have its singular value distribution shifted upwards, we would expect the singular value distribution of  $DF$  to be shifted upwards, too. Our argument in the previous paragraph then suggests that such an intervention will result in faster training, provided of course that the upwards-shifting is not so large as to cause exploding gradients.
Figure 1 shows in the linear setting that a skip connection constitutes precisely such an intervention. We hypothesise that this continues to hold even in the nonlinear setting.
Hypothesis 6.1. The addition of a deterministic
skip connection, all of whose singular values are 1, across a composite of possibly nonlinear random layers, shifts upwards the singular value distribution of the corresponding composite Jacobian<sup>6</sup>, thereby improving convergence speed at least initially.
We test Hypothesis 6.1 in the next subsections.
6.2 MNIST
Recall that the ResNet architecture [20] consists in part of a composite of residual blocks
$$
f (\theta , A, X) = A X + g (\theta , X), \tag {14}
$$
where  $A$  is either the identity transformation, in the case when the dimension of the output of  $g(\theta, X)$  is the same as the dimension of its input; or a randomly initialised 1x1 convolution otherwise.
At least for some common initialisation schemes.
(a) Chain. Mean: 2.36
(b) Res. Mean: 2.91
(c) ResAvg. Mean: 3.17
(d) Loss
Figure 2: (a)-(c) Singular value histograms of composite layer Jacobians averaged over first 10 training iterations. Distributions are shifted upwards as deterministic skip connections are added, resulting in faster convergence (d). Means over 10 trials shown.
(a) ResNet18, CIFAR10
Figure 3: Mean training loss curves for ResNets on CIFAR (10 trials) and ImageNet (3 trials), with 1 standard deviation shaded. The modifications improve training speed as predicted.
(b) ResNet18, CIFAR100
(c) ResNet50, ImageNet
Hypothesis 6.1 predicts that the additions of the identity skip connections will shift upwards the singular value distributions of composite layer Jacobians relative to the equivalent chain network, thus improving convergence speed. It also predicts that replacing  $1 \times 1$  convolutional skip connections  $A$  with  $I + A$ , where  $I$  is deterministic with all singular values equal to 1, will do the same.
We first test this hypothesis by doing gradient descent, with a learning rate of 0.1, on 32 randomly chosen data points from the MNIST training set<sup>7</sup>. We compare three models, all with identical initialisations for each trial run using the default PyTorch initialisation:
1. (Chain) A batch-normed convolutional chain network, with six convolutional layers.
2. (Res) The same as (1), but with convolutional layers 2-3 and 5-6 grouped into two residual blocks, and with an additional  $1 \times 1$  convolution as the skip connection in the second residual block, as in Equation (14).
3. (ResAvg) The same as (2), but with the 1x1 convolutional skip connection of Equation (14) replaced by
$$
\tilde {f} (\theta , A, X) = (I + A) X + g (\theta , X), \tag {15}
$$
where  $I$  is an average pool, rescaled to have all singular values equal to 1.
Hypothesis 6.1 predicts that the singular value distributions of the composite layer Jacobians will be more positively shifted going down the list, resulting in faster convergence. This is indeed what we observe (Figure 2).
6.3 CIFAR and ImageNet
We now test Hypothesis 6.1 on CIFAR and ImageNet. We replace all of the convolution-skip connections in the ResNet architecture [20] with sum(average pool, convolution)-skip connections as in Equation (15) above, leaving all else unchanged. Hypothesis 6.1 predicts that these modifications will improve training speed, which we verify using default PyTorch initialisation schemes.
We trained PreAct-ResNet18 on CIFAR10/100 and PreAct-ResNet50 [20] on ImageNet using standard training regimes (details can be found in the appendix), performing respectively 10 and 3 trial runs
The input-output Jacobians of MPS layers scale with the square of the number of training points, making their computation with a large number of data points prohibitively expensive computationally.
on CIFAR and ImageNet. We performed the experiments at a range of different learning rates. We have included figures for the best performing learning rates on the original model (measured by loss value averaged over the final epoch) in Figure 3, with additional plots and validation accuracies in the appendix. Validation accuracies were not statistically significantly different between the two models on CIFAR10/100. Although the modified version had statistically significantly better validation accuracy in the ImageNet experiment, we believe this is only due to the faster convergence, as the training scheme was not sufficient for the model to fully converge.
Our work suggests some open research problems. First, the recently-developed edge of stability theory [9] could be used in place of our Lipschitz bounds to more realistically characterise training of practical nets with large learning rates. Second, like in pervious works [7], the heavy-lifting for our PL-type bounds is all done by the parameter-derivative of a single layer, and the bounds would be significantly improved by an analysis that considers all layers. Third, extension of the theory to SGD is desirable. Fourth, the dependence of Hypothesis 6.1 on weight variance should be investigated. Fifth, our empirical results on the impact of skip connections on singular value distributions suggest future work using random matrix theory [18].
Beyond these specifics, our formal framework provides a setting in which all neural network layers can be analysed in terms of their effect on the key loss landscape properties of smoothness and regularity, and is the first to demonstrate that a uniform bound on regularity is not necessary to prove convergence. We hope the tools we provide in this paper will be of use in extending deep learning optimisation theory to more practical settings than has so far been the case.
We gave a formal theoretical framework for studying the optimisation of multilayer systems. We used the framework to give the first proof that a class of deep neural networks can be trained by gradient descent even to global optima at infinity. Our theory generates the novel insight that skip connections aid optimisation speed by improving loss regularity, which we verified empirically using practical datasets and architectures.
We thank the anonymous reviewers for their time reviewing the manuscript. Their critiques helped to improve the paper.
[1] Z. Allen-Zhu, Y. Li, and Z. Song. A Convergence Theory for Deep Learning via Over-Parameterization. In ICML, pages 242–252, 2019.
[2] S. Arora, N. Cohen, N. Golowich, and W. Hu. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. In ICLR, 2019.
[3] S. Arora, Z. Li, and A. Panigrahi. Understanding Gradient Descent on Edge of Stability in Deep Learning. In ICML, 2022.
[4] P. L. Bartlett, D. P. Humbold, and P. M. Long. Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks. In ICML, 2018.
[5] M. Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203-248, 2021.
[6] K. Bhardwaj, G. Li, and R. Marculescu. How does topology influence gradient propagation and model performance of deep networks with densenet-type skip connections? In CVPR, 2021.
[7] S. Bombari, M. H. Amani, and M. Mondelli. Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization. In NeurIPS, 2022.
[32] Q. Nguyen. On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths. In NeurIPS, 2021.
[33] Q. Nguyen and M. Mondelli. Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology. In NeurIPS, 2020.
[34] Q. Nguyen, M. Mondelli, and G. Montufar. Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks. In ICML, 2021.
[35] A. E. Orhan and X. Pitkow. Skip Connections Eliminate Singularities. In ICLR, 2018.
[36] S. Oymak and M. Soltanolkotabi. Towards moderate overparameterization: global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory, 1:84-105, 2020.
[37] J. Pennington and P. Worah. Nonlinear random matrix theory for deep learning. In NeurIPS, 2017.
[38] J. Pennington, S. S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NeurIPS, 2017.
[39] J. Pennington, S. Schoenholz, and S. Ganguli. The emergence of spectral universality in deep networks. In AISTATS, 2018.
[40] S. Qiao, H. Wang, C. Liu, W. Shen, and A. Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv:1903.10520, 2019.
[41] T. Salismans and D. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In NeurIPS, 2016.
[42] S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry. How does batch normalization help optimization? In NeurIPS, 2018.
[43] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR, 2014.
[44] Z. Sun, M. Lin, X. Sun, Z. Tan, H. Li, and R. Jin. Mae-det: Revisiting maximum entropy principle in zero-shot nas for efficient object detection. In ICML, 2022.
[45] D. Terjek and D. González-Sánchez. A framework for overparameterized learning, 2022. arXiv:2205.13507.
[46] L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In ICML, 2018.
[47] G. Yang. Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes. In NeurIPS, 2019.
[48] G. Yang. Tensor Programs II: Neural Tangent Kernel for Any Architecture. arXiv:2006.14548, 2020.
[49] G. Yang and S. S. Schoenholz. Mean Field Residual Networks: On the Edge of Chaos. In NeurIPS, 2017.
[50] G. Zhang, A. Botev, and J. Martens. Deep learning without shortcuts: Shaping the kernel with tailored rectifiers. In ICLR, 2022.
[51] D. Zou and Q. Gu. An improved analysis of training over-parameterized deep neural networks. In NeurIPS, 2019.
[52] D. Zou, Y. Cao, D. Zhou, and Q. Gu. Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks. Machine Learning, 109:467-492, 2020.
Computational resources: Both the exploratory and final experiments for this paper were conducted using a desktop machine with two Nvidia RTX A6000 GPUs, for a total running time of approximately 150 hours.
A Examples of MPS
All commonly used neural network layers fit into the framework of multilayer parameterised systems (MPS). Here we list the examples of most relevance to our work.
Example A.1. An affine layer aff:  $\mathbb{R}^{d_1\times d_0 + d_1}\times \mathbb{R}^{d_0\times N}\to \mathbb{R}^{d_1\times N}$  is given by the formula
$$
\operatorname {a f f} (A, b, X) := A X + b 1 _ {N} ^ {T}. \tag {16}
$$
A routine calculation shows that
$$
J \operatorname {a f f} (A, b, X) = A \otimes \operatorname {I d} _ {N}, \tag {17}
$$
$$
D \operatorname {a f f} (A, b, X) = \left(\operatorname {I d} _ {d _ {1}} \otimes X ^ {T}, \operatorname {I d} _ {d _ {1}} \otimes 1 _ {N}\right). \tag {18}
$$
More generally, if  $P: \mathbb{R}^p \to \mathbb{R}^{d_1 \times d_0}$  denotes any continuously differentiable map, then one obtains a  $P$ -parameterised affine layer
$$
\operatorname {a f f} _ {P} (w, b, X) := \operatorname {a f f} (P (w), b, X) = P (w) X + b 1 ^ {T}. \tag {19}
$$
One has
$$
J \operatorname {a f f} _ {P} (w, b, X) = P (w) \otimes \operatorname {I d} _ {N} \tag {20}
$$
and, by the chain rule,
$$
D \operatorname {a f f} _ {P} (w, b, X) = \left(\left(\operatorname {I d} _ {d _ {1}} \otimes X ^ {T}\right) D P (w), \operatorname {I d} _ {d _ {1}} \otimes 1\right), \tag {21}
$$
where  $DP(w) \in \mathbb{R}^{d_0N \times p}$  is the derivative of  $P$  at  $w$ . Common examples include  $\epsilon$ -weight normalisation  $\mathrm{wn}(w) \coloneqq (\epsilon + \| w\|_{row}^2)^{-\frac{1}{2}}w$  [41] and convolutions, which send convolutional kernels to associated Toeplitz matrices. We will also consider  $\epsilon$ -entry normalisation  $\mathrm{en}(w) \coloneqq (\epsilon + w^2)^{-\frac{1}{2}}w$ , with operations applied entrywise.
Example A.2. A (parameter-free) elementwise nonlinearity  $\Phi : \mathbb{R}^{d_0 \times N} \to \mathbb{R}^{d_0 \times N}$  defined by a continuously differentiable function  $\phi : \mathbb{R} \to \mathbb{R}$  is given by applying  $\phi$  to every component of a matrix  $X \in \mathbb{R}^{d_0 \times N}$ . Extension to the parameterised case is straightforward.
Example A.3. A (parameter-free) batch normalisation (BN) layer  $\mathbf{bn}:\mathbb{R}^{d_0\times N}\to \mathbb{R}^{d_0\times N}$  is given by the formula
$$
\operatorname {b n} (X) := \frac {X - \mathbb {E} [ X ]}{\sqrt {\epsilon + \sigma [ X ] ^ {2}}}, \tag {22}
$$
where  $\epsilon > 0$  is some fixed hyperparameter and  $\mathbb{E}$  and  $\sigma$  denote the row-wise mean and standard deviation. The parameterised BN layer from [21], with scaling and bias parameters  $\gamma$  and  $\beta$  respectively, is given simply by postcomposition  $\mathrm{aff}_{\mathrm{diag}}(\gamma, \beta, \cdot) \circ \mathrm{bn}$  with a diag-parameterised affine layer (Example A.1).
Example A.4. A residual block  $f: \mathbb{R}^p \times \mathbb{R}^{d_0 \times N} \to \mathbb{R}^{d_1 \times N}$  can be defined given any other layer (or composite thereof)  $g: \mathbb{R}^p \times \mathbb{R}^{d_0 \times N} \to \mathbb{R}^{d_1 \times N}$  by the formula
$$
f (\theta , X) := I X + g (\theta , X), \tag {23}
$$
where  $I:\mathbb{R}^{d_0\times N}\to \mathbb{R}^{d_1\times N}$  is some linear transformation. In practice,  $I$  is frequently the identity map [20]; our main theorem will concern the case where  $I$  has all singular values equal to 1.
Proof of Theorem 2.4. Using the fact that  $D\ell = (D\gamma \circ F) \cdot DF$ , we compute:
$$
\begin{array}{l} \left\| \nabla \ell (\theta) \right\| ^ {2} = \langle D F (\theta) ^ {T} \nabla \gamma (F (\theta)), D F (\theta) ^ {T} \nabla \gamma (F (\theta)) \rangle \\ = \left\langle \nabla \gamma (F (\theta)), D F (\theta) D F (\theta) ^ {T} \nabla \gamma (F (\theta)) \right\rangle \\ \geq \lambda (D F (\theta)) \| \nabla \gamma (F (\theta)) \| ^ {2} \\ \geq \mu \lambda (D F (\theta)) \left(\gamma (F (\theta)) - \inf _ {\theta^ {\prime}} \gamma (F (\theta^ {\prime}))\right), \\ \end{array}
$$
where the first inequality follows from the standard estimate  $\langle v,AA^T v\rangle \geq \lambda_{min}(AA^T)\| v\|^2$ , and the final inequality follows from the fact that  $\gamma$  is  $\mu$ -P-L over the set  $\{F(\theta):\theta \in \mathbb{R}^p\}$ .
Our proof of Proposition 4.3 requires the following standard lemma.
Lemma B.1. Let  $\{g_i : \mathbb{R}^p \to \mathbb{R}^{m_i \times m_{i-1}}\}_{i=1}^n$  be a family of matrix-valued functions. If, with respect to some submultiplicative matrix norm, each  $g_i$  is bounded by  $b_i$  and Lipschitz with constant  $c_i$  on a set  $S \subset \mathbb{R}^p$ , then their pointwise matrix product  $\theta \mapsto \prod_{i=1}^n g_i(\theta)$  is also bounded and Lipschitz on  $S$ , with bound  $\prod_{i=1}^n b_i$  and Lipschitz constant  $\sum_{i=1}^n c_i \left( \prod_{j \neq i} b_j \right)$ .
Proof of Lemma B.1. We prove the lemma by induction. When  $n = 2$ , adding and subtracting a copy of  $g_{1}(\theta)g_{2}(\theta^{\prime})$  and using the triangle inequality implies that  $\| g_{1}g_{2}(\theta) - g_{1}g_{2}(\theta^{\prime})\|$  is bounded by