SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pages 234-241. Springer, 2015.
[2] Nahian Siddique, Sidike Paheding, Colin P Elkin, and Vijay Devabhaktuni. U-net and its variants for medical image segmentation: A review of theory and applications. IEEE Access, 9:82031-82057, 2021.
[3] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, pages 3-11. Springer, 2018.
[4] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999, 2018.
[5] Zoe Landgraf, Fabian Falck, Michael Bloesch, Stefan Leutenegger, and Andrew J Davison. Comparing view-based and map-based semantic labelling in real-time slam. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6884-6890. IEEE, 2020.
[6] Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616, 2022.
[7] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-1611, 2022.
[8] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[9] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[10] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
[30] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011.10650, 2020.
[31] Shmuel Agmon. Lectures on elliptic boundary value problems, volume 369. American Mathematical Soc., 2010.
[32] Susanne C Brenner, L Ridgway Scott, and L Ridgway Scott. The mathematical theory of finite element methods, volume 3. Springer, 2008.
[33] Bernardo Cockburn, George E Karniadakis, and Chi-Wang Shu. The development of discontinuous Galerkin methods. Springer, 2000.
[34] Barry Boots, Kokichi Sugihara, Sung Nok Chiu, and Atsuyuki Okabe. Spatial tessellations: concepts and applications of voronoi diagrams. 2009.
[35] Boris Delaunay. Sur la sphère vide. Izv. Akad. Nauk SSSR, Otdelenie Matematicheskii i Estestvennyka Nauk, 7(793-800):1-2, 1934.
[36] Der-Tsai Lee and Bruce J Schachter. Two algorithms for constructing a Delaunay triangulation. International Journal of Computer & Information Sciences, 9(3):219-242, 1980.
[37] Aryan Verma. Triangulations Using Matplotlib. https://www.scaler.com/topics/matplotlib/matplotlib-triangulation/.
[38] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[39] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, volume 34, 2021.
[40] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models. In Advances in Neural Information Processing Systems, volume 34, 2021.
[41] Chitwan Sahara, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
[42] Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. Wavelet score-based generative modeling. arXiv preprint arXiv:2208.05003, 2022.
[43] David L Donoho. De-noising by soft-thresholding. IEEE transactions on information theory, 41(3):613-627, 1995.
[44] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[45] Thorvald A Sorensen. A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on danish commons. Biol. Skar., 5:1-34, 1948.
[46] Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297-302, 1945.
[47] Yann LeCun, Corinna Cortes, and C. J. Burges. MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/, 2010.
[48] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[49] Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616, 2022.
[66] Patrick Esser, Ekaterina Sutter, and Björn Ommer. A variational u-net for conditional appearance and shape generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8857-8866, 2018.
[67] Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1):67-70, 2019.
[68] Zhengxin Zhang, Qingjie Liu, and Yunhong Wang. Road extraction by deep residual u-net. IEEE Geoscience and Remote Sensing Letters, 15(5):749-753, 2018.
[69] Nahian Siddique, Paheding Sidike, Colin Elkin, and Vijay Devabhaktuni. U-net and its variants for medical image segmentation: theory and applications. arXiv preprint arXiv:2011.01118, 2020.
[70] Nabil Ibtehaz and M Sohel Rahman. Multiresunet: Rethinking the u-net architecture for multimodal biomedical image segmentation. Neural networks, 121:74-87, 2020.
[71] Mert Lostar and Islem Reik. Deep hypergraph u-net for brain graph embedding and classification. arXiv preprint arXiv:2008.13118, 2020.
[72] Wenchong He, Zhe Jiang, Chengming Zhang, and Arpan Man Sainju. Curvanet: Geometric deep learning based on directional curvature for 3d shape analysis. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2214-2224, 2020.
[73] Ingrid Daubechies. Ten lectures on wavelets. SIAM, 1992.
[74] Michele Benzi. Preconditioning techniques for large linear systems: a survey. Journal of computational Physics, 182(2):418-477, 2002.
[75] A. J. Wathen. Preconditioning. Acta Numerica, 24:329-376, 2015.
[76] Ke Chen. Matrix preconditioning techniques and applications, volume 19. Cambridge University Press, 2005.
[77] Eli Turkel. Preconditioning techniques in computational fluid dynamics. Annual Review of Fluid Mechanics, 31(1):385-416, 1999.
[78] Wolfgang Dahmen and Angela Kunoth. Multilevel preconditioning. Numerische Mathematik, 63(1):315-344, 1992.
[79] O Axclsson and Panayot S Vassilevski. Algebraic multilevel preconditioning methods. i. Numerische Mathematik, 56(2-3):157-177, 1989.
[80] Owe Axelsson and Panayot S Vassilevski. Algebraic multilevel preconditioning methods, ii. SIAM Journal on Numerical Analysis, 27(6):1569-1590, 1990.
[81] Li Zhang, Hengyuan Ma, Xiatian Zhu, and Jianfeng Feng. Preconditioned score-based generative models. arXiv preprint arXiv:2302.06504, 2023.
[82] Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
[83] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035, 2019.
[84] Fergal Cotter. Uses of complex wavelets in deep convolutional neural networks. PhD thesis, University of Cambridge, 2020.
[85] Gregory Lee, Ralf Gommers, Filip Waselewski, Kai Wohlfahrt, and Aaron O'Leary. Pywavelets: A python package for wavelet analysis. Journal of Open Source Software, 4(36):1237, 2019.
[86] William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019.
[87] J. D. Hunter. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3):90-95, 2007.
[88] Charles R. Harris, K. Jarrod Millman, Stefan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, et al. Array programming with NumPy. Nature, 585(7825):357-362, 2020.
[89] Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
[90] pyyaml contributors. pyyaml. https://github.com/yyaml/ppyaml, 2006.
[91] tqdm contributors. Imageio. https://github.com/tqdm/tqdm, 2022.
[92] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[93] Guido Van Rossum. The Python Library Reference, release 3.8.2. Python Software Foundation, 2020.
[94] OpenAI. Gpt-4 technical report, 2023.
Appendix for A Unified Framework for U-Net Design and Analysis
A Theoretical Details and Technical Proofs
A.1 Proofs of theoretical results in the main text
Theorem 1 Suppose  $U_{i}^{*}$  and  $U^{*}$  are solutions of the  $L^2$  regression problem. Then,  $\mathcal{L}_{i|j}(U_i^*)^2 \leq \mathcal{L}_{|j}(U^*)^2$  with equality as  $i \to \infty$ . Further, if  $Q_{i}U^{*}$  is  $V_{i}$ -measurable, then  $U_{i}^{*} = Q_{i}U^{*}$  minimises  $\mathcal{L}_i$ .
Proof. Let  $\mathcal{U} = (\mathcal{V},\mathcal{W},\mathcal{E},\mathcal{D},\mathcal{P},U_0)$  be a U-Net in canonical form, that is where each encoder map  $E_{i}$  in  $\mathcal{E}$  is set to be the identity map, and further assume  $W$  is a Hilbert space with  $\mathcal{W}$  being a sequence of subspaces spanned by orthogonal basis vectors for  $W$ . For a  $V$ -valued random variable  $v$ , define the  $\sigma$ -algebra
$$
\mathcal {H} _ {j} := \sigma \left(P _ {j} v\right) = \sigma \left(v _ {j}\right). \tag {4}
$$
Now the filtration
$$
\mathcal {H} _ {1} \subset \mathcal {H} _ {2} \subset \dots \tag {5}
$$
increases to  $\mathcal{H} \coloneqq \sigma(v)$ . These are the  $\sigma$ -algebras generated by  $v_j$ . For data  $w \in W$ , we may define the losses
$$
\mathcal {L} _ {i | j} ^ {2} \left(U _ {i | j}\right) := \frac {1}{| \mathcal {S} |} \sum_ {\left(w _ {i}, v _ {j}\right) \in \mathcal {S}} \| w _ {i} - U _ {i | j} \left(v _ {j}\right) \| ^ {2}, \tag {6}
$$
for  $U_{i|j}: V_j \mapsto W_i$  where  $w_i = Q_i w$ . Let  $\mathcal{F}_{i|j}$  be the set of measurable functions from  $V_j$  to  $W_i$ , then the conditional expectation  $\mathbb{E}(w_i | v_j)$  is given by the solution of the regression problem
$$
\mathbb {E} \left(w _ {i} \mid v _ {j}\right) = \underset {U _ {i \mid j} \in \mathcal {F} _ {i \mid j}} {\arg \min } \mathcal {L} _ {i \mid j} ^ {2} \left(U _ {i \mid j}\right), \tag {7}
$$
where the conditioning is respect to the  $\sigma$ -algebra  $\mathcal{H}_j$  for the response variable  $w_i$ . For a fixed  $i < i'$  and any  $j > 0$  we have that
$$
\begin{array}{l} \mathcal {L} _ {i ^ {\prime} | j} ^ {2} \left(U _ {i ^ {\prime} | j}\right) = \frac {1}{| \mathcal {S} |} \sum_ {\left(w _ {i ^ {\prime}}, v _ {j}\right) \in \mathcal {S}} \| w _ {i ^ {\prime}} - U _ {i ^ {\prime} | j} \left(v _ {j}\right) \| ^ {2} (8) \\ = \frac {1}{| \mathcal {S} |} \sum_ {\left(w _ {i ^ {\prime}}, v _ {j}\right) \in \mathcal {S}} \left(\left\| Q _ {i} \left(w _ {i ^ {\prime}} - U _ {i ^ {\prime} | j} \left(v _ {j}\right)\right) \right\| ^ {2} + \left\| Q _ {i} ^ {\perp} \left(w _ {i ^ {\prime}} - U _ {i ^ {\prime} | j} \left(v _ {j}\right)\right) \right\| ^ {2}\right) (9) \\ = \mathcal {L} _ {i | j} ^ {2} \left(Q _ {i} U _ {i ^ {\prime} | j}\right) + \frac {1}{| \mathcal {S} |} \sum_ {\left(w _ {i ^ {\prime}}, v _ {j}\right) \in \mathcal {S}} \| Q _ {i} ^ {\perp} \left(w _ {i ^ {\prime}} - U _ {i ^ {\prime} | j} (v _ {j})\right) \| ^ {2}. (10) \\ \end{array}
$$
Any  $U_{i'|j}$  admits the parameterisation  $U_{i'|j} = U_{i|j} + U_{i'|j}^{\perp,i}$  where  $U_{i|j}$  is in  $\mathcal{F}_{i|j}$  and  $U_{i'|j}^{\perp,i} \in \mathcal{F}_{i'|j}$  and vanishes on  $W_i$ . Under this parameterisation we gain
$$
\mathcal {L} _ {i ^ {\prime} | j} ^ {2} \left(U _ {i ^ {\prime} | j}\right) = \mathcal {L} _ {i | j} ^ {2} \left(U _ {i | j}\right) + \frac {1}{| \mathcal {S} |} \sum_ {\left(w _ {i ^ {\prime}}, v _ {j}\right) \in \mathcal {S}} \| Q _ {i} ^ {\perp} \left(w _ {i ^ {\prime}} - U _ {i ^ {\prime} | j} ^ {\perp , i} \left(v _ {j}\right)\right) \| ^ {2}, \tag {11}
$$
and further,
$$
\underset {U _ {i ^ {\prime} | j} \in \mathcal {F} _ {i ^ {\prime} | j}} {\arg \min } \mathcal {L} _ {i ^ {\prime} | j} ^ {2} \left(U _ {i ^ {\prime} | j}\right) = \underset {U _ {i | j} \in \mathcal {F} _ {i | j}} {\arg \min } \mathcal {L} _ {i | j} ^ {2} \left(U _ {i, j}\right) + \underset {U _ {i ^ {\prime} | j} ^ {\perp , i}} {\arg \min } \frac {1}{| \mathcal {S} |} \sum_ {\left(w _ {i ^ {\prime}}, v _ {j}\right) \in \mathcal {S}} \| Q _ {i} ^ {\perp} \left(w _ {i ^ {\prime}} - U _ {i ^ {\prime} | j} ^ {\perp , i} \left(v _ {j}\right)\right) \| ^ {2}. \tag {12}
$$
In particular, when  $i' \to \infty$ , if we define  $\mathcal{L}_{|j}$  to be the loss on  $W$  then
$$
\underset {U _ {| j} \in \mathcal {F} _ {| j}} {\arg \min } \mathcal {L} _ {| j} ^ {2} (U _ {| j}) = \underset {U _ {i | j} \in \mathcal {F} _ {i | j}} {\arg \min } \mathcal {L} _ {i | j} ^ {2} (U _ {i | j}) + \underset {U _ {| j} ^ {\perp , i}} {\arg \min } \sum_ {(w, v _ {j}) \in \mathcal {S}} \| Q _ {i} ^ {\perp} (w - U _ {| j} ^ {\perp , i} (v _ {j})) \| ^ {2}. \tag {13}
$$
So for any  $j$ , we have that  $\mathcal{L}_{i|j}(U_{i|j}^{*}) \leq \mathcal{L}_{|j}(U_{|j}^{*})$ . Further, as  $i \to \infty$ , the truncation error
$$
\sum_ {(w, v _ {j}) \in \mathcal {S}} \| Q _ {i} ^ {\perp} (w - U _ {| j} ^ {\perp , i} (v _ {j})) \| ^ {2}, \tag {14}
$$
tends to zero. Now assume that  $Q_{i}U^{*}$  is  $V_{i}$ -measurable then  $Q_{i}U^{*} \in \mathcal{F}_{i|j}$  and if this did not minimise  $\mathcal{L}_{i|j}$ , then choosing the minimiser of  $\mathcal{L}_{i|j}$  and constructing  $U = U_{i}^{*} + U^{\perp,i}$  will have  $\mathcal{L}^2 (U) < \mathcal{L}^2 (U^*)$ , contradicting  $U^{*}$  being optimal.
Proposition 1 If  $\mathcal{U}$  is a residual U-Net, then  $U_{i}$  is a ResNet preconditioned on  $U_{i}^{\mathrm{pre}}(v_{i}) = U_{i - 1}(\tilde{v}_{i - 1})$  where  $\tilde{v}_{i - 1} = P_{i - 1}(E_i(v_i))$ .
Proof. For a ResNet  $R(v_{i}) = R^{\mathrm{pre}}(v_{i}) + R^{\mathrm{res}}(v_{i})$  select  $R^{\mathrm{pre}}(v_{i}) = U_{i - 1}(\tilde{v}_{i - 1})$  where  $\tilde{v}_{i - 1} = P_{i - 1}(E_i(v_i))$ .
Theorem 2 For time  $t \geq 0$  and  $j \geq i$ ,  $Q_{i}X_{j}(t) \stackrel{d}{=} X_{i}(t)$ . Furthermore if  $X_{i}(t) = \sum_{j=0}^{i} \widehat{X}^{(j)}(t) \cdot \widehat{\phi}_{j}$  be the decomposition of  $X_{i}(t)$  in its Haar wavelet frequencies (see Appendix C). Each component  $\widehat{X}^{(j)}(t)$  of the vector has variance  $2^{j-1}$  relative to the variance of the base Haar wavelet frequency.
Proof. Let  $X_{i} \in V_{i}$  represented in the standard basis  $\Phi = \{\phi_k : k = 1, \dots, 2^i\}$  giving
$$
X _ {i} = \sum_ {k} X _ {i} ^ {(k)} \phi_ {k}. \tag {15}
$$
To transform this into its Haar wavelet representation where average-pooling is conjugate to basis projection, we can use the map  $T_{i}:V_{i}\mapsto V_{i}$  defined by
$$
T _ {i} = \Lambda_ {i} H _ {i}, \tag {16}
$$
where  $H_{i}$  is the Haar-matrix on resolution  $i$  and  $\Lambda_{i}$  is a diagonal scaling matrix with entries
$$
\left(\Lambda_ {i}\right) _ {k, k} = 2 ^ {- i + j - 1}, \quad \text {i f} k \in \{2 ^ {j - 1}, \dots , 2 ^ {j} \}, \tag {17}
$$
for  $j > 0$  and equal to  $2^{-i}$  for the initial case  $j = 0$ . For example, the matrix for a four-pixel image is
$$
2 ^ {- 2} \cdot \left( \begin{array}{c c c c} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 2 ^ {1} & 0 \\ 0 & 0 & 0 & 2 ^ {1} \end{array} \right) \left( \begin{array}{c c c c} 1 & 1 & 1 & 1 \\ 1 & 1 & - 1 & - 1 \\ 1 & - 1 & 0 & 0 \\ 0 & 0 & 1 & - 1 \end{array} \right) = \Lambda_ {2} H _ {2}. \tag {18}
$$
Now given the vector of coefficients  $\mathbf{X}_i = (X_i^{(k)})_{k=1}^{2^i}$ , the coefficients in the Haar frequencies can be given by
$$
\widehat {\mathbf {X}} _ {i} := T _ {i} \left(\mathbf {X} _ {i}\right). \tag {19}
$$
We define the vectors
$$
\left(\widehat {X} _ {j}\right) _ {k} = \left(T _ {i} \left(\mathbf {X} _ {i}\right)\right) _ {k + 2 ^ {j - 1}}, \tag {20}
$$
for  $j \in \{1, \dots, 2^{j-1}\}$ . Now we may write  $X_{i}$  in terms of its various frequency components as
$$
X _ {i} = \sum_ {j = 0} ^ {i} \widehat {X} ^ {(j)} \cdot \widehat {\phi} _ {j}. \tag {21}
$$
Suppose that  $X_{i}$  is a time varying random variable on  $V_{i}$  that evolves according to
$$
X _ {i} ^ {(k)} (t) := \sqrt {1 - \alpha_ {t}} X _ {i} ^ {(k)} + \sqrt {\alpha_ {t}} \varepsilon^ {(k)}, \tag {22}
$$
when represented in its pixel basis where  $\varepsilon^{(k)}$  is a standard normally distributed random variable. We may represent  $X_{i}$  in its frequencies, with a random and time dependence on the coefficient vectors,
$$
X _ {i} = \sum_ {j = 0} ^ {i} \widehat {X} ^ {(j)} (t) \cdot \widehat {\phi} _ {j}. \tag {23}
$$
To analyse the variance of  $\widehat{X}^{(j)}(t)$ , we may analyse the variance of a standard normally distributed random vector  $\varepsilon_{i} = (\varepsilon^{k})_{k=1}^{2^{i}}$  under the mapping  $T_{i}$ . Due to the symmetries in the matrix representation for  $T_{i}$ , the variance for each element of  $\widehat{X}^{(j)}(t)$  is the same, and by direct computation
$$
\operatorname {V A R} \left(T _ {i} \varepsilon_ {i}\right) _ {k} = 2 ^ {- i + j - 1} \quad \text {f o r} i > 0, k \in \{2 ^ {j - 1}, \dots , 2 ^ {j} \}, \tag {24}
$$
and  $2^{-i}$  for when  $i = 0$ . To see this note that
$$
\operatorname {V A R} \left(T _ {i} \varepsilon_ {i}\right) = \operatorname {V A R} \left(\Lambda_ {i} H _ {i} \varepsilon_ {i}\right) = \Lambda_ {i} ^ {2} \operatorname {V A R} \left(H _ {i} \varepsilon_ {i}\right), \tag {25}
$$
as  $\Lambda_{i}$  is a diagonal matrix. Now each of the elements of  $\varepsilon_{i}$  are independent and normally distributed, so if  $(H_{i})_{k,:}$  is the  $k^{th}$  row of  $H_{i}$ , then
$$
\operatorname {V A R} \left(H _ {i} \varepsilon_ {i}\right) _ {k} = \operatorname {V A R} \left(\left(H _ {i}\right) _ {k,:} \varepsilon_ {i}\right) = \| \left(H _ {i}\right) _ {k,:} \| _ {0}, \tag {26}
$$
where  $\| (H_i)_{k,:} \|_0$  is the amount of non-zero elements of  $(H_i)_{k,:}$  as each entry of  $H_i$  is 0, or  $\pm 1$ . For  $(H_i)_{k,:}$  where  $k \in \{2^{j-1}, \ldots, 2^j\}$  there are  $2^{i-j+1}$  entries. Further, in this position we have  $(\Lambda_i)_k = 2^{-i+j-1}$ , so putting this together finishes the count. Now see that for the evolution of  $X_i^{(k)}(t)$ , the random part of Eq. (22) is simply a Gaussian random vector multiplied by  $\sqrt{\alpha_t}$ . Therefore we have that  $\mathrm{VAR}X_i^{(k)}(t) = \alpha_t 2^{-i+j-1}$ . To get the relative frequency, for  $k$  we divide this variance by the variance of the base frequency, the zeroth resolution. This yields  $\alpha_t 2^{-i+j-1} / \alpha_t 2^{-i} = 2^{j-1}$ .
Now we show how the diffusion on a given resolution  $i$  can be embedded into a higher resolution of  $i + 1$ , which maintains consistency with the projection map  $Q_{i}$ . Define the diffusion on  $i + 1$  to be
$$
X _ {i + 1} ^ {(k)} (t) := \sqrt {1 - \alpha_ {t}} X _ {i + 1} ^ {(k)} + \sqrt {2 \alpha_ {t}} \varepsilon^ {(k)} \tag {27}
$$
Let  $\widehat{X}_i^{(j)}(t)$  be the Haar coefficients for the initial diffusion defined on resolution  $i$  and  $\widehat{X}_i^{(j)}(t)$  be the coefficients defined on resolution  $i + 1$ . Both of these random vectors are linear transformations of Gaussian random vectors, so we need only confirm that the means and variances on the first  $j$  entries agree to show that these are equal in distribution. If  $X_i^{(k)} = \frac{X_{i + 1}^{(k)} + X_{i + 1}^{(k + 1)}}{2}$ , that is the initial data on resolution  $i + 1$  averages to the initial data on resolution  $i$ , then by construction the means of the random vectors  $\widehat{X}_i^{(j)}(t)$  and  $\widehat{X}_{i + 1}^{(j)}(t)$  agree for the first  $i$  resolutions. Note that the variances for the components of the noise for either process are  $2^{-i + j - 1}$  when the diffusion on  $i + 1$  is given by Eq. (27). In general, for the process on  $i + \ell$  to embed into our original diffusion on resolution  $i$  we require scaling the noise term by a factor of  $2^{\ell /2}$ . Again, this shows that the noising process on a frequency  $i$  has the natural extension noising the high-frequency components exponentially faster than the reference frequency  $i$ .
We may comment on how this is represented in its natural sequence space. Assume that  $X_{i} \in V_{i} \subset L^{2}([0,1])$ . There is a natural mapping from  $V_{i}$  to  $\ell_2$  defined by
$$
\ell_ {2} := \{s | \sum_ {k = 0} ^ {\infty} s _ {k} ^ {2} <   \infty \} \tag {28}
$$
through the mapping  $\pi :L^2 ([0,1])\mapsto \ell_2$  via
$$
\pi \left(X _ {i}\right) = \left(\widehat {X} ^ {(1)}, \widehat {X} ^ {(2)}, \dots\right). \tag {29}
$$
For elements  $X_{i}\in V_{i}$  the image of  $\pi$  is not surjective and has image contained in  $\ell_{00}$  - the set of infinite sequences which eventually are zero. In this case we have a finite dimensional mapping. Theorem 2 simply states that the forward noising process in a diffusion model, under transformation into its Haar basis, has variance
$$
\operatorname {V A R} \left(\pi X _ {i}\right) _ {k} = 2 ^ {- i + j - 1} \quad \text {f o r} i > 0, k \in \{2 ^ {j - 1}, \dots , 2 ^ {j} \}, \tag {30}
$$
implying that the sequence image of the datum has monotonically increasing variance in the Haar wavelet sequence space.
A.2 Diffusions on the sphere.
Suppose that we have  $L^2(\mathbb{X})$  valued data on the globe and we would like to form a diffusion model on this domain. Instead of constructing a local change of coordinate system over a manifold and projecting our diffusion to a square domain[16], we could simply design our U-Net to encode our geometry. As an example of this, we can take a standard triangulation of the globe, then apply our triangular basis U-Net directly to our data.
For the data on the globe, we could then choose a resolution  $i$  to refine our triangular domain to, see Figure 9 for a refinement level of  $i = 2$  which could be used to tessellate the globe. For each 'triangle pixel' in our refinement, we could define the diffusion process