SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Such as  $\sigma$  being smaller than the nonzero eigenvalues of  $\pmb{\Sigma}_k$  and the normalization assumption  $\pi_i\operatorname*{det}(\pmb{\Sigma}_i + \sigma^2\pmb {I})^{-1 / 2} = \pi_j\operatorname*{det}(\pmb {\Sigma}_j + \sigma^2\pmb {I})^{-1 / 2}$  for all  $i,j\in [K]$ , where  $\pi_{k}$  is the mixture proportion for the  $k^{\mathrm{th}}$  Gaussian.
This statement can be made mathematically rigorous by exploiting a deep connection between neural ODEs and diffusion models, following ideas in Song et al. [44] and Chen et al. [72].
mixture of subspaces. So we may obtain an upper bound of the coding rate for the token set  $Z$  by projecting its tokens onto these subspaces and summing up the respective coding rates:
$$
R ^ {c} (\boldsymbol {Z}; \boldsymbol {U} _ {[ K ]}) = \sum_ {k = 1} ^ {K} R \left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right) = \frac {1}{2} \sum_ {k = 1} ^ {K} \log \det  \left(\boldsymbol {I} + \frac {p}{N \epsilon^ {2}} \left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right) ^ {*} \left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right)\right). \tag {8}
$$
We would like to compress (or denoise) the set of tokens against these subspaces by minimizing the coding rate. The gradient of  $R^{c}(Z;U_{[K]})$  is
$$
\nabla_ {\boldsymbol {Z}} R ^ {c} (\boldsymbol {Z}; \boldsymbol {U} _ {[ K ]}) = \frac {p}{N \epsilon^ {2}} \sum_ {k = 1} ^ {K} \boldsymbol {U} _ {k} \boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z} \left(\boldsymbol {I} + \frac {p}{N \epsilon^ {2}} \left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right) ^ {*} \left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right)\right) ^ {- 1}. \tag {9}
$$
The above expression approximates the residual of each projected token  $U_{k}^{*}z_{i}$  regressed by other tokens  $U_{k}^{*}z_{j}$  [55]. But, differently from [55], not all tokens in  $Z$  are from the same subspace. Hence, to denoise each token with tokens from its own group, we can compute their similarity through an auto-correlation among the projected tokens as  $(U_{k}^{*}Z)^{*}(U_{k}^{*}Z)$  and convert it to a distribution of membership with a softmax, namely  $\text{softmax}((U_k^* Z)^* (U_k^* Z))$ . Then, as we show in Appendix A.2, if we only use similar tokens to regress and denoise each other, then a gradient step on the coding rate with learning rate  $\kappa$  can be naturally approximated as follows:
$$
\boldsymbol {Z} ^ {\ell + 1 / 2} = \boldsymbol {Z} ^ {\ell} - \kappa \nabla_ {\boldsymbol {Z}} R ^ {c} \left(\boldsymbol {Z} ^ {\ell}; \boldsymbol {U} _ {[ K ]}\right) \approx \left(1 - \kappa \cdot \frac {p}{N \epsilon^ {2}}\right) \boldsymbol {Z} ^ {\ell} + \kappa \cdot \frac {p}{N \epsilon^ {2}} \cdot \mathrm {M S S A} \left(\boldsymbol {Z} ^ {\ell} \mid \boldsymbol {U} _ {[ K ]}\right), \tag {10}
$$
where MSSA is defined through an SSA operator as:
$$
\mathbf {S S A} (\boldsymbol {Z} \mid \boldsymbol {U} _ {k}) \doteq \left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right) \operatorname {s o f t m a x} \left(\left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right) ^ {*} \left(\boldsymbol {U} _ {k} ^ {*} \boldsymbol {Z}\right)\right), \quad k \in [ K ], \tag {11}
$$
$$
\operatorname {M S S A} \left(\boldsymbol {Z} \mid \boldsymbol {U} _ {[ K ]}\right) \doteq \frac {p}{N \epsilon^ {2}} \cdot \left[ \boldsymbol {U} _ {1}, \dots , \boldsymbol {U} _ {K} \right] \left[ \begin{array}{c} \operatorname {S S A} \left(\boldsymbol {Z} \mid \boldsymbol {U} _ {1}\right) \\ \vdots \\ \operatorname {S S A} \left(\boldsymbol {Z} \mid \boldsymbol {U} _ {K}\right) \end{array} \right]. \tag {12}
$$
Here the SSA operator in (11) resembles the attention operator in a typical transformer [28], except that here the linear operators of value, key, and query are all set to be the same as the subspace basis, i.e.,  $\mathbf{V} = \mathbf{K} = \mathbf{Q} = \mathbf{U}_k^*$ . Hence, we name  $\mathrm{SSA}(\cdot | U_k) : \mathbb{R}^{d \times N} \to \mathbb{R}^{p \times N}$  the Subspace Self-Attention (SSA) operator (more details and justification can be found in (72) in Appendix A.2). Then, the whole MSSA operator in (12), formally defined as  $\mathrm{MSSA}(\cdot | U_{[K]}) : \mathbb{R}^{d \times N} \to \mathbb{R}^{d \times N}$  and called the Multi-Head Subspace Self-Attention (MSSA) operator, aggregates the attention head outputs by averaging using model-dependent weights, similar in concept to the popular multi-head self-attention operator in existing transformer networks. The overall gradient step (10) resembles the multi-head self-attention implemented with a skip connection in transformers.
Notice that if we have  $N = 1$  tokens as well as take an aggressive gradient step ( $\kappa = 1$ ) and tune the quantization error ( $\epsilon = \sqrt{p / N}$ ), the multi-head subspace self-attention operator in (12) becomes the ideal denoiser defined in (6), with the one minor difference that the aggregation of the heads is done by a linear function here, while in (6) it is done by a nonlinear mixture-of-experts type function. This provides two very related interpretations of the multi-head self-attention operator, as denoising and compression against a mixture of low-dimensional subspaces.
2.4 MLP via Iterative Shrinkage-Thresholding Algorithms (ISTA) for Sparse Coding
In the previous subsection, we focused on how to compress a set of tokens against a set of (learned) low-dimensional subspaces. Optimizing the remaining terms in the sparse rate reduction objective (1), including the non-smooth term, serves to sparsify the compressed tokens, hence leading to a more compact and structured (i.e., parsimonious) representation. From (1) and (7), this term is
$$
\max  _ {\boldsymbol {Z}} \left[ R (\boldsymbol {Z}) - \lambda \| \boldsymbol {Z} \| _ {0} \right] = \min  _ {\boldsymbol {Z}} \left[ \lambda \| \boldsymbol {Z} \| _ {0} - \frac {1}{2} \log \det  \left(\boldsymbol {I} + \frac {d}{N \epsilon^ {2}} \boldsymbol {Z} ^ {*} \boldsymbol {Z}\right) \right], \tag {13}
$$
<sup>6</sup>We note a recent suggestion of Hinton [51] that it is more sensible to set the "value, key, and query" projection matrices in a transformer to be equal. Our derivation in this section confirms this mathematically.
This suggests that we could also consider such a mixture of expert type aggregation of the multiple attention heads. In this work, we use linear aggregation, and leave evaluation of more variants for future work.
Figure 2: One layer of the CRATE architecture. The full architecture is simply a concatenation of such layers, with some initial tokenizer and final task-specific architecture (i.e., a classification head).
where  $R(Z)$  denotes the coding rate of the whole token set, as defined in (7). In addition to sparsification via the  $\| Z\| _0$  term, the expansion term  $R(Z)$  in (13) promotes diversity and non-collapse of the representation, a highly desirable property. However, prior work has struggled to realize this benefit on large-scale datasets due to poor scalability of the gradient  $\nabla_{Z}R(Z)$ , which requires a matrix inverse [55].
To simplify things, we therefore take a different approach to trading off between representational diversity and sparsification: we posit a (complete) incoherent or orthogonal dictionary  $\pmb{D} \in \mathbb{R}^{d \times d}$ , and ask to sparsify the intermediate iterates  $Z^{\ell + 1/2}$  with respect to  $\pmb{D}$ . That is,  $Z^{\ell + 1/2} = DZ^{\ell + 1}$  where  $Z^{\ell + 1}$  is more sparse. The dictionary  $\pmb{D}$  is global, i.e., is used to sparsify all tokens simultaneously. By the incoherence assumption, we have  $D^{*}D \approx I_{d}$ ; thus from (7) we have  $R(Z^{\ell + 1}) \approx R(DZ^{\ell + 1}) = R(Z^{\ell + 1/2})$ . Thus we approximately solve (13) with the following program:
$$
Z ^ {\ell + 1} = \underset {Z} {\arg \min } \| Z \| _ {0} \quad \text {s u b j e c t t o} \quad Z ^ {\ell + 1 / 2} = D Z. \tag {14}
$$
The above sparse representation program is usually solved by relaxing it to an unconstrained convex program, known as LASSO:
$$
\boldsymbol {Z} ^ {\ell + 1} = \underset {\boldsymbol {Z}} {\arg \min } \left[ \lambda \| \boldsymbol {Z} \| _ {1} + \| \boldsymbol {Z} ^ {\ell + 1 / 2} - \boldsymbol {D} \boldsymbol {Z} \| _ {F} ^ {2} \right]. \tag {15}
$$
In our implementation, motivated by Sun et al. [33] and Zarka et al. [35], we also add a non-negative constraint to  $Z^{\ell +1}$
$$
\boldsymbol {Z} ^ {\ell + 1} = \underset {\boldsymbol {Z} \geq \mathbf {0}} {\arg \min } \left[ \lambda \| \boldsymbol {Z} \| _ {1} + \| \boldsymbol {Z} ^ {\ell + 1 / 2} - \boldsymbol {D} \boldsymbol {Z} \| _ {F} ^ {2} \right], \tag {16}
$$
which we then incrementally optimize by performing an unrolled proximal gradient descent step, known as an ISTA step [8], to give the update:
$$
\boldsymbol {Z} ^ {\ell + 1} = \operatorname {R e L U} \left(\boldsymbol {Z} ^ {\ell + 1 / 2} + \eta \boldsymbol {D} ^ {*} \left(\boldsymbol {Z} ^ {\ell + 1 / 2} - \boldsymbol {D} \boldsymbol {Z} ^ {\ell + 1 / 2}\right) - \eta \lambda \mathbf {1}\right) \doteq \mathrm {I S T A} \left(\boldsymbol {Z} ^ {\ell + 1 / 2} \mid \boldsymbol {D}\right). \tag {17}
$$
In Appendix A.3, we will show one can arrive at a similar operator to the above ISTA-like update for optimizing (13) by properly linearizing and approximating the rate term  $R(Z)$ .
2.5 The Overall White-Box CRATE Architecture
By combining the above two steps:
1. (Sections 2.2 and 2.3) Local denoising and compression of tokens within a sample towards a mixture-of-subspace structure, leading to the multi-head subspace self-attention block - MSSA;
2. (Section 2.4) Global compression and sparsification of token sets across all samples through sparse coding, leading to the sparsification block - ISTA;
we can get the following rate-reduction-based transformer layer, illustrated in Figure 2,
$$
\boldsymbol {Z} ^ {\ell + 1 / 2} \doteq \boldsymbol {Z} ^ {\ell} + \operatorname {M S S A} \left(\boldsymbol {Z} ^ {\ell} \mid \boldsymbol {U} _ {[ K ]} ^ {\ell}\right), \quad \boldsymbol {Z} ^ {\ell + 1} \doteq \operatorname {I S T A} \left(\boldsymbol {Z} ^ {\ell + 1 / 2} \mid \boldsymbol {D} ^ {\ell}\right). \tag {18}
$$
Composing multiple such layers following the incremental construction of our representation in (2), we obtain a white-box transformer architecture that transforms the data tokens towards a compact and sparse union of incoherent subspaces.
This model has the parameters  $(\pmb{U}_{[K]}^{\ell})_{\ell = 1}^{L}$  and  $(D^{\ell})_{\ell = 1}^{L}$ , which are learned from data via backpropagation. Notably, in each layer  $\ell$ , the learned  $U_{[K]}^{\ell}$  retain their interpretation as incoherent bases for supporting subspaces for the mixture-of-Gaussians model at layer  $\ell$ , and the learned  $D^{\ell}$  retains its interpretation as a sparsifying dictionary at layer  $\ell$ . We emphasize that the parameters  $U_{[K]}^{\ell}$  and  $D^{\ell}$  are dependent on the layer  $\ell$  — that is, we learn a different set of parameters at each layer. This is because at each layer we learn an approximate local parametric model for the input data distribution, then use that learned model to construct the layer operators that transform the distribution. Our procedure of parameterizing the data distribution at each layer distinguishes this work from previous works on unrolled optimization for neural networks such as the ReduNet [55]. Our interpretation clarifies the roles of the network forward pass (given local signal models at each layer, denoise/compress/sparsify the input) and the backward pass (learn the local signal models from data via supervision).
We note that in this work, at each stage of our construction, we have chosen arguably the simplest possible construction to use. We can substitute each part of this construction, so long as the new part maintains the same conceptual role, and obtain another white-box architecture. Nevertheless, our such-constructed architecture, called CRATE (i.e., Coding RAte TransformEr), connects to existing transformer models, obtains competitive results on real-world datasets, and is fully mathematically interpretable.
In this section, we conduct experiments to study the performance of our proposed white-box transformer CRATE on real-world datasets and tasks. As the analysis in Section 2 suggests, either the compression or the sparsification step can be achieved through various alternative design choices or strategies. CRATE arguably adopts the most basic choices and so our goal with the experiments is not simply to compete with other heavily engineered transformers while using such a rudimentary design. Rather, our goals are twofold. First, unlike any empirically designed black-box networks that are usually evaluated only on end-to-end performance, the white-box design of our network allows us to look inside the deep architecture and verify if layers of the learned network indeed perform their design objective—say performing incremental optimization for the objective (1). Second, despite their simplicity, our experiments will actually reveal the vast practical potential of our so-derived CRATE architectures since, as we will show, they already achieve very strong performance on large-scale real-world datasets and tasks. In the remainder of this section we highlight a selection of results; additional experimental details and results can be found in Appendix B.
Model architecture. We implement the architecture that is described in Section 2.5, with minor modifications that are described in Appendix B.1. We consider different model sizes of CRATE by varying the token dimension  $d$ , number of heads  $K$ , and the number of layers  $L$ . We consider four model sizes in this work: CRATE-Tiny, CRATE-Small, CRATE-Base, and CRATE-Large. A PyTorch-style pseudocode can be found in Appendix B.1, which contains more implementation details. For training using supervised classification, we first take the CLS token  $\overline{z}_b = z_{1,b}^{L + 1}$  of for each sample, then apply a linear layer; the output of this linear layer  $\pmb{u}_b \doteq \pmb{W}\overline{z}_b$  is used as input to the standard cross-entropy loss. The overall loss averages over all samples  $b \in [B]$ .
Datasets and optimization. We mainly consider ImageNet-1K [9] as the testbed for our architecture. Specifically, we apply the Lion optimizer [73] to train CRATE models with different model sizes. Meanwhile, we also evaluate the transfer learning performance of CRATE: by considering the models trained on ImageNet-1K as pre-trained models, we fine-tune CRATE on several commonly used downstream datasets (CIFAR10/100, Oxford Flowers, Oxford-IIT-Pets). More details about the training and datasets can be found in Appendix B.1.
Figure 3: Left: The compression term  $R^{c}(Z^{\ell +1 / 2})$  of the MSSA outputs at different layers. Right: the sparsity of the ISTA output block,  $\| Z^{\ell +1}\| _0 / (d\cdot N)$ , at different layers. (Model: CRATE-Small).
Figure 4: The compression term  $R^{c}(\mathbf{Z})$  (left) and sparsification term  $\| \mathbf{Z}\|_{0} / (d\cdot N)$  (right) across models trained with different numbers of epochs. (Model: CRATE-Base).
3.1 In-depth Layer-wise Analysis of CRATE
Do layers of CRATE achieve their design goals? As described in Section 2.3 and Section 2.4, the MSSA block is designed to optimize the compression term  $R^{c}(Z)$  and the ISTA block to sparsify the token representations (corresponding to the sparsification term  $\| Z\| _0$ ). To understand whether CRATE indeed optimizes these terms, for each layer  $\ell$ , we measure (i) the compression term  $R^{c}(Z^{\ell +1 / 2})$  on the MSSA block outputs  $Z^{\ell +1 / 2}$ ; and (ii) sparsity  $\| Z^{\ell +1}\| _0$  on the ISTA block outputs  $Z^{\ell +1}$ . Specifically, we evaluate these two terms by using training/validation samples from ImageNet-1K. Both terms are evaluated at the per-sample level and averaged over  $B = 10^{3}$  samples.
Figure 3 shows the plots of these two key measures at all layers for the learned CRATE-small model. We find that as the layer index  $\ell$  increases, both the compression and the sparsification terms improve in most cases. The increase in the sparsity measure of the last layer is caused by the extra linear layer for classification. These results suggest that CRATE aligns well with the original design goals: once learned, it essentially learns to gradually compress and sparsity the representations through its layers. In addition, we also measure the compression and sparsification terms on CRATE models with different model sizes as well as intermediate model checkpoints and the results are shown by plots in Figure 5 of Appendix B.2. The observations are very consistent across all different model sizes—both the compression and sparsification terms improve in most scenarios. Models with more layers tend to optimize the objectives more effectively, confirming our understanding of each layer's roles.
To see the effect of learning, we present the evaluations on CRATE-Small trained with different number of epochs in Figure 4. When the model is not trained enough (e.g. untrained), the architecture does not optimize the objectives effectively. However, during training—learning better subspaces  $\pmb{U}_{[K]}^{\ell}$  and dictionaries  $D^{\ell}$ —the designed blocks start to optimize the objectives much more effectively.
Visualizing layer-wise token representations. To gain a better understanding of the token representations of CRATE, we visualize the output of each ISTA block at layer  $\ell$  in Figure 6 of Appendix B.2. Specifically, we visualize the  $Z^{\ell +1}$  via heatmap plots. We observe that the output  $Z^{\ell +1}$  becomes more sparse as the layer increases. Moreover, besides the sparsity, we also find that  $Z^{\ell +1}$  becomes
Note that the learned sparse (tokens) features need to be mixed in the last layer for predicting the class. The phenomenon of increase in the sparsity measure at the last layer suggests that each class of objects may be associated with a number of features, and some of these features are likely to be shared across different classes.
Table 1: Top 1 accuracy of CRATE on various datasets with different model scales when pre-trained on ImageNet. For ImageNet/ImageNetReaL, we directly evaluate the top-1 accuracy. For other datasets, we use models that are pre-trained on ImageNet as initialization and the evaluate the transfer learning performance via fine-tuning.
Datasets	CRATE-T	CRATE-S	CRATE-B	CRATE-L	ViT-T	ViT-S
# parameters	6.09M	13.12M	22.80M	77.64M	5.72M	22.05M
ImageNet	66.7	69.2	70.8	71.3	71.5	72.4
ImageNet RealL	74.0	76.0	76.5	77.4	78.3	78.4
CIFAR10	95.5	96.0	96.8	97.2	96.6	97.2
CIFAR100	78.9	81.0	82.7	83.6	81.8	83.2
Oxford Flowers-102	84.6	87.1	88.7	88.3	85.1	88.5
Oxford-IIIT-Pets	81.4	84.9	85.3	87.4	88.5	88.6
more structured (i.e., low-rank), which indicates that the set of token representations become closer to linear subspaces, confirming our mental picture of the geometry of each layer (as in Figure 1).
Visualizing layer-wise subspaces in multi-head self-attention. We now visualize the  $\mathbf{U}_{[K]}^{\ell}$  matrices used in the MSSA block. In Section 2.3, we assumed that  $\mathbf{U}_{[K]}^{\ell}$  were incoherent to capture different "views" of the set of tokens. In Fig. 7 of Appendix B.2, we first normalize the columns in each  $\mathbf{U}_k^\ell$ , then we visualize the  $[\mathbf{U}_1^\ell, \ldots, \mathbf{U}_K^\ell]^{*}[\mathbf{U}_1^\ell, \ldots, \mathbf{U}_K^\ell] \in \mathbb{R}^{pK \times pK}$ . The  $(i,j)$ -th block in each sub-figure corresponds to  $(\mathbf{U}_i^\ell)^*\mathbf{U}_j^\ell$  for  $i,j \in [K]$  at a particular layer  $\ell$ . We find that the learned  $\mathbf{U}_{[K]}^{\ell}$  are approximately incoherent, which aligns well with our assumptions. One interesting observation is that the  $\mathbf{U}_{[K]}^{\ell}$  becomes more incoherent when the layer index  $\ell$  is larger, which suggests that the token representations are more separable. This mirrors the situation in other popular deep networks [58].
3.2 Evaluations of CRATE on Large Real-World Datasets and Tasks
We now study the empirical performance of the proposed networks by measuring their top-1 accuracy on ImageNet-1K as well as transfer learning performance on several widely used downstream datasets. We summarize the results in Table 1. As our designed architecture leverages parameter sharing in both the attention block (MSSA) and the MLP block (ISTA), our CRATE-Base model (22.08 million) has a similar number of parameters to the ViT-Small (22.05 million).
From Table 1, we find that with a similar number of model parameters, our proposed network achieves similar ImageNet-1K and transfer learning performance as ViT, despite the simplicity and interpretability of our design. Moreover, with the same set of training hyperparameters, we observe promising scaling behavior in CRATE—we consistently improve the performance by scaling up the model size. For comparison, directly scaling ViT on ImageNet-1K does not always lead to consistent performance improvement measured by top-1 accuracy [40]. To summarize, we achieve promising performance on real-world large-scale datasets by directly implementing our principled architecture.
In this paper, we propose a new theoretical framework that allows us to derive deep transformer-like network architectures as incremental optimization schemes to learn compressed and sparse representation of the input data (or token sets). The so derived and learned deep architectures are not only fully mathematically interpretable, but also consistent on a layer-by-layer level with their design objective. Despite being arguably the simplest among all possible designs, these networks already demonstrate performance on large-scale real-world datasets and tasks close to seasoned transformers. We believe this work truly helps bridge the gap between theory and practice of deep neural networks as well as help unify seemingly separate approaches to learning and representing data distributions. Probably more importantly for practitioners, our framework provides theoretical guidelines to design and justify new, potentially more powerful, deep architectures for representation learning.
We thank the anonymous reviewers for their helpful comments. Yaodong Yu would like to thank Kwan Ho Ryan Chan for the valuable discussions we had regarding visualizing tokens in vision transformers. Yaodong Yu acknowledges support from the joint Simons Foundation-NSF DMS grant #2031899. Yi Ma acknowledges support from ONR grant N00014-22-1-2102 and the joint Simons Foundation-NSF DMS grant #2031899. This work was partially supported by NSF 1704458, the Northrop Grumman Mission Systems Research in Applications for Learning Machines (REALM) initiative, NIH NIA 1R01AG067396, and ARO MURI W911NF-17-1-0304.
[78] Hongkang Li, Meng Wang, Sijia Liu, and Pin-Yu Chen. “A Theoretical Understanding of shallow Vision Transformers: Learning, Generalization, and Sample Complexity”. arXiv preprint arXiv:2302.06015 (2023). 2.
[79] Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar. "The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers". The Eleventh International Conference on Learning Representations. 2023. 23.
[80] Ravid Shwartz-Ziv and Yann LeCun. "To Compress or Not to Compress-Self-Supervised Learning and Information Theory: A Review". arXiv preprint arXiv:2304.09355 (2023). 2.
[81] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. "Consistency models". arXiv preprint arXiv:2303.01469 (2023). 2.
A Technical Details from Section 2
A.1 Companion to Section 2.2
We first wish to re-iterate the core contributions of our approach in Section 2.2 at a slightly more technical level. Connections between denoising and score matching are well-understood [60], and computing the optimal denoising function (i.e., the conditional expectation) against a mixture-of-Gaussians model is a rather simple computation giving existing tools such as Tweedie's formula [13]. These are not our main contributions. Instead, the main contributions of Section 2.2 are two-fold: