SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we consider four commonly used datasets for our experiments: CIFAR10 [32], CelebA [33], ImageNet [34], and MS-COCO [35]. For the unconditional setting, we use CIFAR10, which consists of 50,000 training images, and CelebA 64, which contains 162,770 training images of human faces. For the class-conditional settings, we utilize the well-known ImageNet dataset at  $64 \times 64$  resolutions, which contains 1,281,167 training images from 1,000 different classes. For the text-to-image setting, we consider the MS-COCO dataset at  $256 \times 256$  resolution. Unless otherwise specified, all of our experiments are conducted on A100 GPU and follow the experimental settings in [36].
Table 5: The summary of the datasets for the experiments.
Dataset	#class	#training	#type	#Image size
CIFAR10	10	50,000	Uncondition	32 x 32
CelebA	100	162,770	Uncondition	64 x 64
ImageNet 64	1000	1,281,167	Class-condition	64 x 64
MS-COCO	-	82,783	Text-to-Image	256 x 256
A.2 Network architecture details.
The scaling strategies we proposed on long skip connections, namely CS and LS, are straightforward to implement and can be easily transferred to neural network architectures of different diffusion models. Fig. 9 and Fig. 10 compare the original UViT with the code incorporating CS or LS, demonstrating that our proposed methods require minimal code modifications.
Table 6: The training settings for different dataset.
Dataset	CIFAR10 64×64	CelebA 64×64	ImageNet 64×64	MS-COCO
Latent shape	-	-	-	32×32×4
Batch size	128/64/32	128	1024	256
Training iterations	500k	500k	300k	1M
Warm-up steps	2.5k	4k	5k	5k
Optimizer	AdamW	AdamW	AdamW	AdamW
Learning rate	2.00E-04	2.00E-04	3.00E-04	2.00E-04
Weight decay	0.03	0.03	0.03	0.03
Beta	(0.99,0.999)	(0.99,0.99)	(0.99,0.99)	(0.9,0.9)
Origin
skipx = []
for blk in self.in_blocks:
    x = blk(x)
    skips.append(x)
x = self.mid.block(x)
for blk in self.out_blocks:
    x = blk(x, skips.pop())
x = self(norm(x))
x = self.decoder_pred(x)
assert x.size(1) == self.extras + L
x = x[:, self.extras:, :] x = unpatchify(x, self.in_chans)
x = self.final_layer(x)
return x
CS (ours)
skip  $\equiv$  []
for blk in self.in_blocks:
    x = blk(x)
    skips.append(x)
x = self.mid.block(x)
layer_cnt = 0
for blk in self.out_blocks:
    x = blk(x, skips.pop() * k** (self.depth - 1 - layer_cnt))
    layer_cnt += 1
x = self(norm(x)
x = self.decoder_pred(x)
assert x.size(1) == self.extras + L
x = x[;, self.extras:, :]
x = unpatchify(x, self.in_chans)
x = self.final_layer(x)
return x
Figure 9: The comparison between the original UViT and our proposed CS.
CS requires no additional parameters and can rapidly adapt to any UNet-alike architecture and dataset. In our experiments, we have chosen values of  $\kappa$  as 0.5, 0.5, 0.7, and 0.8 for the four datasets illustrated in Table 5, respectively. In various training environments, we recommend setting  $\kappa$  within the range of 0.5 to 0.95. LS designs a calibration network shared by all long skip connections to predict scaling coefficients  $\{\kappa_i\}$ . Specifically, for the  $i$ -th long skip connection, its input feature is  $x_{i} \in \mathbb{R}^{B \times N \times D}$ , where  $B$  denotes the batch size. For convolution-based UNet,  $N$  and  $D$  respectively denote the channel number and spatial dimension ( $H \times W$ ) of the feature map; for transformer-based UViT [36],  $N$  and  $D$  respectively represent the token number and token dimension.
In the main text, we define the LS as
$$
\kappa_ {i} = \sigma \left(\zeta_ {\phi} [ \mathrm {G A P} (x _ {i}) ]\right) \in \mathbb {R} ^ {B \times N \times 1}, 1 \leq i \leq N, \tag {10}
$$
For  $\zeta_{\phi}$ , we employ a neural network structure based on SENet [79], given by:
$$
\zeta_ {\phi} (x) = \mathbf {W} _ {1} \circ \psi \left(\mathbf {W} _ {2} \circ x\right), \tag {11}
$$
where  $x \in \mathbb{R}^N$ ,  $\circ$  denotes matrix multiplication, and  $\psi$  represents the ReLU activation function. The matrices  $\mathbf{W}_1 \in \mathbb{R}^{N \times N / r}$  and  $\mathbf{W}_1 \in \mathbb{R}^{N / r \times N}$ , where  $r$  is the compression ratio, are set to  $r = 16$  in this paper. In the ablation study of the main text, we have demonstrated that the choice of  $r$  has a minimal impact on the performance of the diffusion model. It is worth noting that if LS is applied to long skip connections where the feature dimensions remain unchanged, such as in UViT, then the calibration network is only a shared  $\zeta_{\phi}(x)$ .
However, when LS is applied to neural networks where the dimension of features in certain long skip connections may change, such as in UNet with downsample and upsample operations, we need to provide a set of encoders and decoders in the calibration network to ensure dimension alignment for different feature dimensions.
Assuming there are  $d$  different feature dimensions in the long skip connections, where the dimension of the  $i$ -th feature is  $(B \times N_i \times D)$ ,  $1 \leq i \leq d$ , the input dimension of  $\zeta_{\phi}(x)$  is set to  $N_{\min} = \min N_i$ . Additionally, we set up  $d$  sets of encoders and decoders, where features with the same dimension
Figure 10: The comparison between the original UViT and our proposed LS.
LS (ours)
share an encoder and a decoder. The  $i$ -th set of encoder  $\zeta_{\mathrm{en}}^{(i)}$  and decoder  $\zeta_{\mathrm{de}}^{(i)}$  also adopt the SENet structure, given by:
$$
\zeta_ {\mathrm {e n}} ^ {(i)} (x) = \mathbf {W} _ {N _ {\min } \times N _ {i} / r _ {i}} \circ \psi \left(\mathbf {W} _ {N _ {i} / r _ {i} \times N _ {i}} \circ x\right), \quad x \in \mathbb {R} ^ {N _ {i}} \tag {12}
$$
where  $r_i$  is the compression ratio, typically set to  $\lfloor N_i / 4\rfloor$ , and
$$
\zeta_ {\mathrm {d e}} ^ {(i)} (x) = \mathbf {W} _ {N _ {i} \times N _ {i} / r _ {i}} \circ \psi \left(\mathbf {W} _ {N _ {i} / r _ {i} \times N _ {\min }} \circ x\right), \quad x \in \mathbb {R} ^ {N _ {\min }}. \tag {13}
$$
In fact, since the parameter count of  $\zeta_{\phi}(x)$  itself is not substantial, to make LS more versatile, we can also individually set a scaling module into each long skip connection. This configuration typically does not lead to significant performance degradation and can even result in improved performance, while maintaining the same inference speed as the original LS.
B The proof of the Theorems and Lemma
Diffusion Model (DM). DDPM-alike DMs [1-6, 8, 9, 43] generates a sequence of noisy samples  $\{\mathbf{x}_i\}_{i=1}^T$  by repeatedly adding Gaussian noise to a sample  $\mathbf{x}_0$  until attaining  $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ . This noise injection process, a.k.a. the forward process, can be formalized as a Markov chain  $q(\mathbf{x}_{1:T}|\mathbf{x}_0) = \prod_{t=1}^T q(\mathbf{x}_t|\mathbf{x}_{t-1})$ , where  $q(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t|\sqrt{\alpha_t}\mathbf{x}_{t-1}, \beta_t\mathbf{I})$ ,  $\alpha_t$  and  $\beta_t$  depend on  $t$  and satisfy  $\alpha_t + \beta_t = 1$ . By using the properties of Gaussian distribution, the forward process can be written as
$$
q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right) = \mathcal {N} \left(\mathbf {x} _ {t}; \sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right), \tag {14}
$$
$$
\ell_ {\text {s i m p l e}} ^ {t} (\theta) = \mathbb {E} \| \epsilon_ {t} - \hat {\epsilon} _ {\theta} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon_ {t}, t\right) \| _ {2} ^ {2}. \tag {15}
$$
UNet-alike Network. Since the network  $\hat{\epsilon}_{\theta}(\cdot, t)$  in DMs predicts the noise to denoise, it plays a similar role of UNet-alike network widely in image restoration tasks, e.g. image de-raining, image denoising [22-26, 44, 45]. This inspires DMs to use UNet-alike network in (16) that uses LSCs to connect distant parts of the network for long-range information transmission and aggregation
$$
\mathbf {U N e t} (x) = f _ {0} (x), \quad f _ {i} (x) = b _ {i + 1} \circ \left[ \kappa_ {i + 1} \cdot a _ {i + 1} \circ x + f _ {i + 1} \left(a _ {i + 1} \circ x\right) \right], \quad 0 \leq i \leq N - 1 \tag {16}
$$
W.l.o.g, in this paper, we consider  $\circ$  as matrix multiplication, and set the block  $a_{i}$  and  $b_{i}$  as the stacked networks [46, 47] which is implemented as
$$
a _ {i} \circ x = \mathbf {W} _ {l} ^ {a _ {i}} \phi \left(\mathbf {W} _ {l - 1} ^ {a _ {i}} \dots \phi \left(\mathbf {W} _ {1} ^ {a _ {i}} x\right)\right), \quad b _ {i} \circ x = \mathbf {W} _ {l} ^ {b _ {i}} \phi \left(\mathbf {W} _ {l - 1} ^ {b _ {i}} \dots \phi \left(\mathbf {W} _ {1} ^ {b _ {i}} x\right)\right) \tag {17}
$$
with a ReLU activation function  $\phi$  and learnable matrices  $\mathbf{W}_j^{a_i},\mathbf{W}_j^{b_i}\in \mathbb{R}^{m\times m}$ $(j\geq 1)$ . Moreover, let  $f_{N}$  also have the same architecture, i.e.  $f_{N}(x) = \mathbf{W}_{l}^{f_{i}}\phi (\mathbf{W}_{l - 1}^{f_{i}}\dots \phi (\mathbf{W}_{1}^{f_{i}}x))$
Proposition B.1 (Amoroso distribution). The Amoroso distribution is a four parameter, continuous, univariate, unimodal probability density, with semi-infinite range [83]. And its probability density function is
$$
\operatorname {A m o r o s o} (X | a, \theta , \alpha , \beta) = \frac {1}{\Gamma (\alpha)} \left| \frac {\beta}{\theta} \right| \left(\frac {X - a}{\theta}\right) ^ {\alpha \beta - 1} \exp \left\{- \left(\frac {X - a}{\theta}\right) ^ {\beta} \right\}, \tag {18}
$$
for  $x, a, \theta, \alpha, \beta \in \mathbb{R}, \alpha > 0$  and range  $x \geq a$  if  $\theta > 0$ ,  $x \leq a$  if  $\theta < 0$ . The mean and variance of Amoroso distribution are
$$
\mathbb {E} _ {X \sim \mathbf {A m o r o s o} (X | a, \theta , \alpha , \beta)} X = a + \theta \cdot \frac {\Gamma (\alpha + \frac {1}{\beta})}{\Gamma (\alpha)}, \tag {19}
$$
$$
\operatorname {V a r} _ {X \sim \mathbf {A m o r o s o} (X | a, \theta , \alpha , \beta)} X = \theta^ {2} \left[ \frac {\Gamma (\alpha + \frac {2}{\beta})}{\Gamma (\alpha)} - \frac {\Gamma (\alpha + \frac {1}{\beta}) ^ {2}}{\Gamma (\alpha) ^ {2}} \right]. \tag {20}
$$
Proposition B.2 (Stirling's formula). For big enough  $x$  and  $x \in \mathbb{R}^+$ , we have an approximation of Gamma function:
$$
\Gamma (x + 1) \approx \sqrt {2 \pi x} \left(\frac {x}{e}\right) ^ {x}. \tag {21}
$$
Proposition B.3 (Scaled Chi distribution). Let  $X = (x_{1}, x_{2}, \ldots, x_{k})$  and  $x_{i}, i = 1, \ldots, k$  are  $k$  independent, normally distributed random variables with mean 0 and standard deviation  $\sigma$ . The statistic  $\ell_{2}(X) = \sqrt{\sum_{i=1}^{k} x_{i}^{2}}$  follows Scaled Chi distribution [83]. Moreover,  $\ell_{2}(X)$  also follows Amoroso $(x|0, \sqrt{2}\sigma, \frac{k}{2}, 2)$ . By Eq. (19) and Eq. (20), the mean and variance of Scaled Chi distribution are
$$
\mathbb {E} _ {X \sim N \left(\mathbf {0}, \sigma^ {2}. \mathbf {I} _ {\mathbf {k}}\right)} [ \ell_ {2} (X) ] ^ {j} = 2 ^ {j / 2} \sigma^ {j} \cdot \frac {\Gamma (\frac {k + j}{2})}{\Gamma (\frac {k}{2})}, \tag {22}
$$
$$
\mathbf {V a r} _ {X \sim N \left(\mathbf {0}, \sigma^ {2} \cdot \mathbf {I} _ {\mathbf {k}}\right)} \ell_ {2} (X) = 2 \sigma^ {2} \left[ \frac {\Gamma (\frac {k}{2} + 1)}{\Gamma (\frac {k}{2})} - \frac {\Gamma (\frac {k + 1}{2}) ^ {2}}{\Gamma (\frac {k}{2}) ^ {2}} \right]. \tag {23}
$$
Lemma B.4. For big enough  $x$  and  $x \in \mathbb{R}^+$ , we have
$$
\lim  _ {x \rightarrow + \infty} \left[ \frac {\Gamma (\frac {x + 1}{2})}{\Gamma (\frac {x}{2})} \right] ^ {2} \cdot \frac {1}{x} = \frac {1}{2}. \tag {24}
$$
$$
\lim  _ {x \rightarrow + \infty} \frac {\Gamma (\frac {x}{2} + 1)}{\Gamma (\frac {x}{2})} - \left[ \frac {\Gamma (\frac {x + 1}{2})}{\Gamma (\frac {x}{2})} \right] ^ {2} = \frac {1}{4}. \tag {25}
$$
$$
\begin{array}{l} \lim  _ {x \rightarrow + \infty} \left[ \frac {\Gamma (\frac {x + 1}{2})}{\Gamma (\frac {x}{2})} \right] ^ {2} \cdot \frac {1}{x} \approx \lim  _ {x \rightarrow + \infty} \left(\frac {\sqrt {2 \pi (\frac {x - 1}{2})} \cdot (\frac {x - 1}{2 e}) ^ {\frac {x - 1}{2}}}{\sqrt {2 \pi (\frac {x - 2}{2})} \cdot (\frac {x - 2}{2 e}) ^ {\frac {x - 2}{2}}}\right) ^ {2} \cdot \frac {1}{x} \quad \text {f r o m P r o p o s i t i o n .} \\ = \lim _ {x \to + \infty} \left(\frac {x - 1}{x - 2}\right) \cdot \frac {\left(\frac {x - 1}{2 e}\right) ^ {x - 2}}{\left(\frac {x - 2}{2 e}\right) ^ {x - 2}} \cdot \left(\frac {x - 1}{2 e}\right) \cdot \frac {1}{x} \\ = \lim  _ {x \rightarrow + \infty} \left(1 + \frac {1}{x - 2}\right) ^ {x - 2} \cdot \frac {x - 1}{x - 2} \cdot \frac {x - 1}{2 e} \cdot \frac {1}{x} \\ = \frac {1}{2} \\ \end{array}
$$
on the other hand, we have
$$
\begin{array}{l} \lim  _ {x \rightarrow + \infty} \frac {\Gamma (\frac {x}{2} + 1)}{\Gamma (\frac {x}{2})} - \left[ \frac {\Gamma (\frac {x + 1}{2})}{\Gamma (\frac {x}{2})} \right] ^ {2} = \lim  _ {x \rightarrow + \infty} \frac {x}{2} - \left(1 + \frac {1}{x - 2}\right) ^ {x - 2} \cdot \frac {x - 1}{x - 2} \cdot \frac {x - 1}{2 e} \\ = \lim  _ {x \rightarrow + \infty} \frac {x}{2 e} \left(e - (1 + \frac {1}{x}) ^ {x}\right) \\ = \frac {1}{2} \left(- \frac {\frac {1}{e} (- e)}{2}\right) \\ = \frac {1}{4} \\ \end{array}
$$
Lemma B.5. [84, 85] Let  $\mathbf{x} = (x_{1}, x_{2}, \ldots, x_{N})$ , where  $x_{i}, 1 \leq i \leq N$ , are independently and identically distributed random variables with  $x_{i} \sim \mathcal{N}(\mu, \sigma^{2})$ . In this case, the mathematical expectation  $\mathbb{E}\left(\sum_{i=1}^{N} x_{i}^{2}\right)$  can be calculated as  $\sigma^{2}N + \mu^{2}N$ .
Lemma B.6. [86] Let  $F_{ij} \in \mathbb{R}^{N_i \times k \times k}$  be the  $j$ -th filter of the  $i$ -th convolutional layer. If all filters are initialized as Kaiming's initialization, i.e.,  $\mathcal{N}(0, \frac{2}{N_i \times k \times k})$ , in  $i$ -th layer,  $F_{ij}$  ( $j = 1, 2, \dots, N_{i+1}$ ) are i.i.d and approximately follow such a distribution during training:
$$
F _ {i j} \sim \mathcal {N} \left(\mathbf {0}, \mathcal {O} \left[ \left(k ^ {2} N _ {i}\right) ^ {- 1} \right] \cdot \mathbf {I} _ {N _ {i} \times k \times k}\right). \tag {26}
$$
Lemma B.7. [47, 87] For the feedforward neural network  $\mathbf{W}_l\phi (\mathbf{W}_{l - 1}\dots \phi (\mathbf{W}_1x))$  with initialization  $\mathcal{N}(0,\frac{2}{m})$ , and  $W_{i},1\leq i\leq l,\in \mathbb{R}^{m\times m}$ .  $\forall \rho \in (0,1)$ , the following inequality holds with at least  $1 - \mathcal{O}(l)\exp [-\Omega (m\rho^2 /l)]$ :
$$
(1 - \rho) \| h _ {0} \| _ {2} \leq \| h _ {a} \| _ {2} \leq (1 + \rho) \| h _ {0} \| _ {2}, \quad a \in \{1, 2, \dots , l - 1 \}, \tag {27}
$$
where  $h_a$  is  $\phi(\mathbf{W}_a h_{a-1})$  for  $a = 1, 2, \ldots, l-1$  with  $h_0 = x$ .
Lemma B.8. For a matrix  $\mathbf{W} \in \mathbb{R}^{m \times m}$  and its elements are independently follow the distribution  $\mathcal{N}(0, c^2)$ , we have the following estimation while  $m$  is large enough
$$
\| \mathbf {W} s \| _ {2} \approx \mathcal {O} (\| s \| _ {2}), \tag {28}
$$
where  $s\in \mathbb{R}^m$
Proof. Let  $v_{i}, 1 \leq i \leq m$  be the  $i$ -th column of matrix  $\mathbf{W}$ , and we have  $v_{i} \sim \mathcal{N}(\mathbf{0}, c^{2}\mathbf{I}_{m})$ . We first prove the following two facts while  $m \to \infty$ :
(1)  $||v_{i}||_{2}\approx ||v_{j}||_{2}\to \sqrt{2} c\cdot \frac{\Gamma((m + 1) / 2)}{\Gamma(m / 2)},1\leq i <   j\leq m;$
(2)  $\langle v_i,v_j\rangle \to 0,1\leq i <   j\leq m;$
For the fact (1), since Chebyshev inequality, for  $1 \leq i \leq m$  and a given  $M$ , we have
$$
P \left\{\left| \left| \left| v _ {i} \right| \right| _ {2} - \mathbb {E} \left(\left| \left| v _ {i} \right| \right| _ {2}\right) \right| \geq \sqrt {M \mathbf {V a r} \left(\left| \left| v _ {i} \right| \right| _ {2}\right)} \right\} \leq \frac {1}{M}. \tag {29}
$$
from Proposition B.3 and Lemma. B.4, we can rewrite Eq. (29) when  $m\to \infty$
$$
P \left\{\left| \left| v _ {i} \right| \right| _ {2} \in \left[ \sqrt {2} c \cdot \frac {\Gamma ((m + 1) / 2)}{\Gamma (m / 2)} - \sqrt {\frac {M}{2}} c, \sqrt {2} c \cdot \frac {\Gamma ((m + 1) / 2)}{\Gamma (m / 2)} + \sqrt {\frac {M}{2}} c \right] \right\} \geq 1 - \frac {1}{M}. \tag {30}
$$
For a small enough  $\epsilon > 0$ , let  $M = 1 / \epsilon$ . Note that  $\sqrt{\frac{M}{2}} c = c / \sqrt{2\epsilon}$  is a constant. When  $m \to \infty$ ,  $\sqrt{2} c \cdot \frac{\Gamma((m + 1) / 2)}{\Gamma(m / 2)} \gg \sqrt{\frac{M}{2}} c$ . Hence, for any  $i \in [1, m]$  and any small enough  $\epsilon$ , we have
$$
P \left\{\left| \left| v _ {i} \right| \right| _ {2} \approx \sqrt {2} c \cdot \frac {\Gamma ((m + 1) / 2)}{\Gamma (m / 2)} \right\} \geq 1 - \epsilon . \tag {31}
$$
So the fact (1) holds. Moreover, we consider the  $\langle v_i,v_j\rangle ,1\leq i <   j\leq m$
Let  $v_{i} = (v_{i1}, v_{i2}, \dots, v_{im})$  and  $v_{j} = (v_{j1}, v_{j2}, \dots, v_{jm})$ . So  $\langle v_{i}, v_{j} \rangle = \sum_{p=1}^{m} v_{ip} v_{jp}$ . Note that,  $v_{i}$  and  $v_{j}$  are independent, hence
$$
\mathbb {E} \left(v _ {i p} v _ {j p}\right) = 0, \tag {32}
$$
$$
\mathbf {V a r} \left(v _ {i p} v _ {j p}\right) = \mathbf {V a r} \left(v _ {i p}\right) \mathbf {V a r} \left(v _ {j p}\right) + \left(\mathbb {E} \left(v _ {i p}\right)\right) ^ {2} \mathbf {V a r} \left(v _ {j p}\right) + \left(\mathbb {E} \left(v _ {j p}\right)\right) ^ {2} \mathbf {V a r} \left(v _ {i p}\right) = c ^ {4}, \tag {33}
$$
since central limit theorem, we have
$$
\sqrt {m} \cdot c ^ {- 2} \cdot \left(\frac {1}{m} \sum_ {p = 1} ^ {m} v _ {i p} v _ {j p} - 0\right) \sim N (0, 1), \tag {34}
$$
According to Eq. (22), Lemma B.4 and Eq. (34), when  $m \to \infty$ , we have
$$
\frac {\left\langle v _ {i} , v _ {j} \right\rangle}{\left\| v _ {i} \right\| _ {2} \cdot \left\| v _ {j} \right\| _ {2}} \rightarrow \frac {1}{\sqrt {m}} \cdot \frac {\left\langle v _ {i} , v _ {j} \right\rangle}{\sqrt {m}} \sim N \left(0, \mathcal {O} (\frac {1}{m})\right)\rightarrow N (0, 0). \tag {35}
$$
Therefore, the fact (2) holds and we have  $\langle v_i,v_j\rangle \to 0,1\leq i < j\leq m$
Next, we consider  $\| \mathbf{W}s\| _2$  for any  $s\in \mathbb{R}^m$
$$
\begin{array}{l} \left\| \mathbf {W} s \right\| _ {2} ^ {2} = s ^ {\mathrm {T}} \left[ v _ {1}, v _ {2},..., v _ {m} \right] ^ {\mathrm {T}} \left[ v _ {1}, v _ {2},..., v _ {m} \right] s \\ = s ^ {\mathrm {T}} \left( \begin{array}{c c c c} \| v _ {1} \| _ {2} ^ {2} & v _ {1} ^ {\mathrm {T}} v _ {2} & \dots & v _ {1} ^ {\mathrm {T}} v _ {m} \\ v _ {2} ^ {\mathrm {T}} v _ {1} & \| v _ {2} \| _ {2} ^ {2} & & \vdots \\ \vdots & & \ddots & \\ v _ {m} ^ {\mathrm {T}} v _ {1} & \dots & & \| v _ {m} \| _ {2} ^ {2} \end{array} \right) s \tag {36} \\ \end{array}
$$
From the facts (1) and (2), when  $m$  is large enough, we have  $v_{i}^{\mathrm{T}}v_{j}\to 0,i\neq j,$  and  $\| v_1\| _2^2\approx \| v_2\| _2^2\approx$ $\dots \approx \| v_m\| _2^2 \coloneqq c_v$ . Therefore, we have
$$
\left\| \mathbf {W} s \right\| _ {2} ^ {2} = s ^ {\mathrm {T}} c _ {v} \mathbf {I} _ {\mathbf {m}} s = \mathcal {O} \left(\left\| s \right\| _ {2} ^ {2}\right). \tag {37}
$$
Let  $k = 1$  in Lemma B.6, and the convolution kernel degenerates into a learnable matrix. In this case, we can observe that if  $W \in \mathbb{R}^{m \times m}$  is initialized using Kaiming initialization, i.e.,  $\mathcal{N}(0,2 / m)$ , during the training process, the elements of  $W$  will approximately follow a Gaussian distribution  $\mathcal{N}(0,\mathcal{O}(m^{-1}))$ . In this case, we can easily rewrite Lemma B.7 as the following Corollary B.9:
Corollary B.9. For the feedforward neural network  $\mathbf{W}_l\phi (\mathbf{W}_{l - 1}\dots \phi (\mathbf{W}_1x))$  with kaiming's initialization during training.  $\forall \rho \in (0,1)$ , the following inequality holds with at least  $1 - \mathcal{O}(l)\exp [-\Omega (m\rho^2 /l)]$ :
$$
\mathcal {O} ((1 - \rho) \| h _ {0} \| _ {2}) \leq \| h _ {a} \| _ {2} \leq \mathcal {O} ((1 + \rho) \| h _ {0} \| _ {2}), \quad a \in \{1, 2, \dots , l - 1 \}, \tag {38}
$$
Moreover, considering Lemma B.6 and Lemma B.8, we can conclude that the matrix  $W_{l}$  in Corollary B.9 will not change the magnitude of its corresponding input  $\phi (\mathbf{W}_{l - 1}\dots \phi (\mathbf{W}_1x))$ . In other words, we have the following relationship:
$$
\mathcal {O} ((1 - \rho) \| h _ {0} \| _ {2}) \leq \| \mathbf {W} _ {l} h _ {l - 1} \| _ {2} \leq \mathcal {O} ((1 + \rho) \| h _ {0} \| _ {2}), \tag {39}
$$
Therefore, for  $a_i$  and  $b_i$  defined in Eq. (17), we have following corollary:
Corollary B.10. For  $a_i$  and  $b_i$  defined in Eq. (17),  $1 \leq i \leq N$ , and their corresponding input  $x_a$  and  $x_b$ .  $\forall \rho \in (0, 1)$ , the following each inequality holds with at least  $1 - \mathcal{O}(l) \exp[-\Omega(m\rho^2 / l)]$ :
$$
\left\| a _ {i} \circ x _ {a} \right\| _ {2} = \mathcal {O} \left(\left\| x _ {a} \right\| _ {2}\right), \quad \left\| b _ {i} \circ x _ {b} \right\| _ {2} = \mathcal {O} \left(\left\| x _ {b} \right\| _ {2}\right), \tag {40}
$$
Lemma B.11. [47] Let  $\mathbf{D}$  be a diagonal matrix representing the ReLU activation layer. Assume the training loss is  $\frac{1}{n}\sum_{s=1}^{n}\ell_s(\mathbf{W})$ , where  $\ell_s(\mathbf{W})$  denotes the training loss of the  $s$ -th sample among the  $n$  training samples. For the  $\mathbf{W}_p^q$  defined in Eq. (17) and  $f_0(\mathbf{x}_t)$  defined in Eq. (16), where  $p \in \{1,2,\dots,l\}$ ,  $q \in \{a_i,b_i\}$ , the following inequalities hold with at least 1- $\mathcal{O}(nN)$ $\exp[-\Omega(m)]$ :
$$
\left\| \mathbf {D} \left(\mathbf {W} _ {p + 1} ^ {a _ {i}}\right) ^ {\mathrm {T}} \dots \mathbf {D} \left(\mathbf {W} _ {l} ^ {a _ {i}}\right) ^ {\mathrm {T}} \dots \mathbf {D} \left(\mathbf {W} _ {l} ^ {b _ {1}}\right) ^ {\mathrm {T}} v \right\| _ {2} \leq \mathcal {O} (\| v \| _ {2}), \tag {41}
$$
$$
\left\| \mathbf {D} \left(\mathbf {W} _ {p + 1} ^ {b _ {i}}\right) ^ {\mathrm {T}} \dots \mathbf {D} \left(\mathbf {W} _ {l} ^ {b _ {1}}\right) ^ {\mathrm {T}} v \right\| _ {2} \leq \mathcal {O} (\| v \| _ {2}), \tag {42}
$$
where  $v\in \mathbb{R}^m$
B.2 The proof of Theorem 3.1.
Theorem 3.1. Assume that all learnable matrices of UNet in Eq. (16) are independently initialized as Kaiming's initialization, i.e.,  $\mathcal{N}(0,\frac{2}{m})$ . Then for any  $\rho \in (0,1]$ , by minimizing the training loss Eq. (15) of DMs, with probability at least  $1 - \mathcal{O}(N)\exp [-\Omega (m\rho^2)]$ , we have
$$
(1 - \rho) ^ {2} \left[ c _ {1} \| \mathbf {x} _ {t} \| _ {2} ^ {2} \cdot \sum_ {j = i + 1} ^ {N} \kappa_ {j} ^ {2} + c _ {2} \right] \lesssim \| h _ {i} \| _ {2} ^ {2} \lesssim (1 + \rho) ^ {2} \left[ c _ {1} \| \mathbf {x} _ {t} \| _ {2} ^ {2} \cdot \sum_ {j = i + 1} ^ {N} \kappa_ {j} ^ {2} + c _ {2} \right], \tag {43}
$$
where the hidden feature  $h_i$  is the output of  $f_{i}$  defined in Eq. (16),  $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\epsilon_t$  is the input of UNet;  $c_{1}$  and  $c_{2}$  are two constants;  $\kappa_{i}$  is the scaling coefficient of the  $i$ -th LSC in UNet.
Proof. Let  $s$  represent the input of  $f_{i}$  in Eq. (16), i.e.,  $s = a_{i} \circ a_{i-1} \circ \dots \circ a_{1} \circ \mathbf{x}_{t}$ , and let  $h_{i} = f_{i}(s)$ . By expanding Eq. (16), we can observe that  $h_{i} = f_{i}(s)$  is mainly composed of the following  $N - i$  terms in the long skip connection:
$$
\left\{ \begin{array}{l} \kappa_ {i + 1} \cdot b _ {i + 1} \circ a _ {i + 1} \circ s; \\ \kappa_ {i + 2} \cdot b _ {i + 1} \circ b _ {i + 2} \circ a _ {i + 2} \circ a _ {i + 1} \circ s; \\ \dots \\ \kappa_ {N} \cdot b _ {i + 1} \circ b _ {i + 2} \dots \circ b _ {N} \circ a _ {N} \dots a _ {i + 1} \circ s; \end{array} \right. \tag {44}
$$
When  $N$  is sufficiently large,  $h_i$  is predominantly dominated by  $N$  long skip connection terms, while the information on the residual term is relatively minor. Hence, we can approximate Eq. (16) as follows:
$$
h _ {i} = f _ {i} (s) \approx \sum_ {j = i + 1} ^ {N} \kappa_ {j} \cdot \pi_ {b} ^ {j} \circ \pi_ {a} ^ {j} \circ s, \quad 0 \leq i \leq N - 1, \tag {45}
$$
where  $\pi_b^j \coloneqq b_{i+1} \circ b_{i+2} \cdots \circ b_j$ ,  $\pi_a^j \coloneqq a_j \circ a_{j-1} \cdots \circ a_{i+1}$ .