SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that the above formulation is by no means limited to these two examples. Many other popular sampling strategies can equally well be considered. This includes radial line sampling, spiral sampling and various other strategies used in practice. We remark in passing that the setup described here is closely related to the block sampling setup in compressed sensing. See [18, 22].
Remark C.3 It is worth noting two extensions of this setup. First, in practice one may wish to consider overlapping partitions  $\Delta_1,\ldots ,\Delta_M$ . Here, one requires a small modification of the sampling operators to ensure Assumption 2.4 holds. For an index  $j\in \{1,\dots ,N\}$  let  $r_j\in \mathbb{N}$  be the number of partition elements in which  $j$  appears, i.e.,  $r_j = |\{i:j\in \Delta_i\} |$ . Then define
$$
L (i) (x) = \sqrt {M} \left(\left(F x\right) _ {j} / \sqrt {r _ {j}}\right) _ {j \in \Delta_ {i}}. \tag {C.3}
$$
We now argue as in (C.2) to deduce that empirical nondegeneracy holds with  $\alpha = \beta = 1$ .
Second, it is sometime useful in practice to allow the partition elements to have different cardinalities, i.e.,  $|\Delta_i| = p_i$ ,  $i = 1, \ldots, M$ . This can be handled straightforwardly by zero padding. Let  $\mathbb{Y} = \mathbb{C}^p$  where  $p = \max_{i=1,\dots,M} \{p_i\}$ . Then we define  $L(i): \mathbb{C}^N \to \mathbb{C}^p$  by appending  $p - p_i$  zeros to the vector defined in (C.1) (or (C.3) if the partition is overlapping).
C.2 The approximation space: compressive imaging with generative models
The approach considered in this paper is based on the work [20] which introduced the generative compressed sensing framework. Their subsequent work on posterior sampling further delved into the problem [68]. In compressed sensing, the primary objective is to leverage the inherent structure in the solutions of underdetermined systems of noisy linear measurements. Most of the existing literature in this field focuses on imposing various forms of sparsity, such as standard sparsity, joint sparsity, hierarchical sparsity, or block sparsity [5, 48]. However, the generative compressed sensing framework takes a different approach. It assumes that the solutions to these problems lie near the range of a generative model  $G: \mathbb{R}^p \to \mathbb{R}^N$ . By employing a well-trained generative model  $G$  that is relevant to the reconstruction task at hand, one can solve an optimization problem to reconstruct images or other signals. In this case, the approximation space  $\mathbb{F} = \mathrm{range}(G) \subset \mathbb{R}^N$ , i.e., the range of the generative model  $G$ . This space can be seen as a low-dimensional manifold embedded in a higher-dimensional space. Even for simple DNN models this space is typically nonlinear, and certainly for the complex models considered for the experiments related to this section.
Importantly, the generative compressed sensing framework is not limited to image reconstruction but has broader applicability. The recent work [15] extends the analysis from [20] to the problem of demixing with subgaussian matrices. Other notable contributions include variational networks [59], generative adversarial networks trained with least-squares and pixel-wise  $\ell_1 / \ell_2$  cost functions [87], generative adversarial network frameworks for structure preserving compressive sensing MRI [40], and model-based approaches combined with regularization and unrolled network architectures [7]. Furthermore, a recent study considers the semi-parametric single-index model with a generative prior under a Gaussian measurement assumption [81]. These studies, along with other diverse works, demonstrate the impressive versatility of the generative compressed sensing approach.
C.3 The generalized Christoffel function
Let  $\mathbb{F} \subseteq \mathbb{X}$ , where, as stated above,  $\mathbb{X} = \mathbb{C}^N$  with the Euclidean inner product. Denote the Euclidean norm by  $\| \cdot \|$ . Then the generalized Christoffel function is defined by
$$
K (\mathbb {F}) (i) = \sup  \left\{\frac {\| L (i) (f) \| ^ {2}}{\| f \| ^ {2}}: f \in \mathbb {F} \right\} = M \sup  \left\{\frac {\sum_ {j \in \Delta_ {i}} | (F f) _ {j} | ^ {2}}{\| f \| ^ {2}}: f \in \mathbb {F} \right\} \tag {C.4}
$$
for  $i \in \{1, \ldots, M\}$ . Since  $D = \{1, \ldots, M\}$  is a discrete set, we consider discrete probability measures. Denote such a measure by  $(\mu_i)_{i=1}^M$ , i.e.,  $I \sim \mu$  if  $\mathbb{P}(I = i) = \mu_i$ . Then sampling according to the Christoffel function gives the values
$$
\mu_ {i} = \frac {K (\mathbb {F}) (i)}{M \kappa (\mathbb {F})}, \quad i \in \{1, \dots , M \},
$$
$$
\kappa (\mathbb {F}) = \int_ {D} K (\mathbb {F}) (\theta) \mathrm {d} \rho (\theta) = \frac {1}{M} \sum_ {i = 1} ^ {M} K (\mathbb {F}) (i).
$$
Algorithm 1 Computing  $\widetilde{K}$
input: Generative model  $G$  , number of iterations  $t$  input distribution  $\rho$  on the domain of  $G$  DFT matrix  $F$    
initialize:  $\widetilde{K} = 0$    
for  $l = 1$  to t do Generate two i.i.d. realizations  $f_{1},f_{2}$  via the generative model, i.e.,  $f_{i} = G(z_{i})$  for  $z_{i}\sim \mathrm{i.i.d.}\rho$  Compute  $a_{i} = M\sum_{j\in \Delta_{i}}|(Fg)_{j}|^{2} / \| v\|^{2}$  for  $g = f_{1} - f_{2}$  and  $i = 1,\dots,M$  for  $i = 1$  to  $M$  do Replace  $\tilde{K} (i) = \max \{\tilde{K} (i),a_i\}$    
end for   
end for   
output: Vector  $\widetilde{K}$  with updated entries.
Note that the corresponding weight function is
$$
\nu (i) = M \mu_ {i} = \frac {K (\mathbb {F}) (i)}{\kappa (\mathbb {F})}, \quad i \in \{1, \dots , M \}.
$$
In order to apply Theorem 4.2 to this example, we need to identify a (subspace) cover  $\widetilde{\mathbb{F}} = \mathbb{F}_1 \cup \dots \cup \mathbb{F}_d$  of the set  $\mathbb{F} - \mathbb{F}$ , where, as above  $\mathbb{F}$ , is the range of a generative model. It is therefore important to first ask when the set  $\mathbb{F} - \mathbb{F}$  admits such a cover for a reasonable value of  $d$  (since  $d$  appears logarithmically in the bound). This question has been addressed in [95, Lem. 3.1] in the case of ReLU generative models (see also [16, Lem. S2.2]). Roughly speaking, if  $G: \mathbb{R}^p \to \mathbb{R}^N$  has  $L$  layers and the widths of its hidden layers are  $\mathcal{O}(p)$ , then  $\mathbb{F} - \mathbb{F}$  can be covered exactly with  $\log(d) \lesssim pL$  subspaces of dimension  $2p$ .
There are two factors that prohibit direct application of Theorem 4.2 to this example. First, the generative models we use are not simple ReLU DNNs. Second, even if they were, identifying the aforementioned union of subspaces that form the cover and computing the constant  $K(\widetilde{\mathbb{F}})$  is likely computationally infeasible, since  $d$  grows exponentially in  $pL$ .
As noted in Section 5, however, we believe the estimate involving  $K(\widetilde{\mathbb{F}})$  in Theorem 4.2 to be non-sharp, and conjecture that it should suffice to consider  $K(\mathbb{F} - \mathbb{F})$  instead. Therefore, in what follows, we compute with (an approximation to) the former. Thus, in this example, the theory does not completely apply. Nevertheless, as noted, we see significant benefits from CS for this example.
C.4 Numerical approximation to the generalized Christoffel function
In view of (C.4), computing  $K(\mathbb{F} - \mathbb{F})$  involves solving  $M$  (nonlinear, nonconvex) maximization problems over the input space  $\mathbb{R}^p$  of the generative model  $G$ . This is extremely computationally expensive, since both  $p$  and  $M$  are large in practice. In our examples,  $M = n^3$  (Theorem C.1) or  $M = n^2$  (Theorem C.2), where  $n = 128$ , and  $p = 1024$  (see next).
Therefore, in this work we consider a different approach, which is based on an iterative, random sampling procedure for computing an approximation  $\widetilde{K}$  to  $K$ . This procedure is presented in Algorithm 1 and involves repeated random samples from the input distribution of the generative model to construct candidate elements  $f_{1} - f_{2} \in \mathbb{F} - \mathbb{F}$ . It has the advantages of being simple and not computationally intensive. Moreover, as our examples show, sampling from the resulting  $\widetilde{K}$  achieves the objective of significantly outperforming MCS.
The convergence of Algorithm 1 in computing  $\tilde{K}$  can be seen in Fig. 7 where the relative  $\ell_2$  error of the iterates are computed with the final  $\tilde{K}$  after the algorithm returns.
C.5 Additional experimental details
The experiments in Figures 2 & 8 were conducted using a single generative neural network model trained on brain MRI data. The braingen-0.1.0 generative model was obtained from https://github.com/neuronets/trained-models/releases and was trained using the nobrainer
Figure 7: Convergence of the relative  $\ell_2$  error of iterations of Algorithm 1 in computing  $\widetilde{K}$  for (left) Example C.2 and (right) Example C.1.
framework [70] available under Apache License, Version 2.0. This model was trained using a progressively growing strategy [71] for generation of T1-weighted brain MR scans. The model takes as input a latent vector  $\pmb{v} \in \mathbb{R}^{1024}$ , here generated by sampling from the standard normal distribution, i.e.  $v_{i} \sim N(0,1)$ ,  $i = 1, \ldots, 1024$ . The output dimension of the selected braingen-0.1.0 model is  $N = n^3$  where  $n = 128$ .
First we generate  $\widetilde{K}$  using 1000 iterations of Algorithm 1. To generate the results, we chose an image from the range of the generator using random seed '12345'. We then ran 20 random trials reconstructing this image with the same computed  $\widetilde{K}$ , solving problem (2.2) using 5000 iterations of the Adam optimizer [72], implemented in TensorFlow. A constant learning rate multiplier of 100 was used to improve the convergence in the gradient update step. The experiments were conducted on a single computer with an AMD Ryzen 9 5950x, 64 GB of DDR4-3600 RAM, and a NVIDIA GeForce RTX 3090 Ti. The total runtime for these experiments was approximately 2 weeks. In Fig. 2 we consider the setup of Example C.2. In Fig. 8 we consider the setup of Example C.1. As in Section B.5, our error plots display the mean PSNR value and also a shaded region corresponding to one standard deviation. The PSNR is defined as
$$
\mathrm {P S N R} = 2 0 \cdot \log_ {1 0} \left(\frac {\mathrm {M A X} _ {f ^ {*}}}{\sqrt {\mathrm {M S E}}}\right),
$$
where MSE is the Mean Squared Error
$$
\mathrm {M S E} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left| f _ {i} ^ {*} - \hat {f} _ {i} \right| ^ {2}.
$$
Here  $f^{*}$  is the ground truth image,  $\hat{f}$  is its reconstruction and  $\mathrm{MAX}_{f^{*}}$  is the maximum possible pixel value of the image, here equal to 255.
C.6 Further experiments and discussion
In Fig. 2 consider the strategy of sampling horizontal lines in  $k$ -space, see Example C.2. In Fig. 8 we perform the same experiment with the strategy of sampling individual frequencies in  $k$ -space, see Example C.1. While CS with line sampling performs similarly to CS with individual frequency sampling, MCS line sampling in  $k$ -space can be seen to have a  $15 - 20\%$  drop in image reconstruction quality over MCS individual frequency sampling in  $k$ -space.
Although, as noted, the theory does not directly apply to this example, it is worth examining the relevant constants to gain some further insight into the performance of CS over MCS. For MCS, Theorem 4.2 suggests that the sample complexity depends on the term
$$
C_{\mathsf{MCS}} = \operatorname *{ess}\sup_{i\sim \rho}\widetilde{K} (\mathbb{F} - \mathbb{F})(i) = \max_{i = 1,\ldots ,M}\widetilde{K} (\mathbb{F} - \mathbb{F})(i),
$$
Figure 8: CS for MRI reconstruction using generative models. Plots of (left) the average PSNR vs. image number of the 3-dimensional brain MR image for both CS and MCS methods at  $0.125\%$ ,  $0.375\%$ , and  $0.625\%$  sampling percentages, (middle) the average PSNR computed over the frames 30 to 90 of the image vs. sampling percentage for both CS and MCS methods, and (right) the empirical function  $\widetilde{K}$  used for the CS procedure. In this figure, we consider Example C.1 instead of Example C.2, which was used in Fig. 2.
Table 1: Values of  ${C}_{\mathrm{{MCS}}}$  and  ${C}_{\mathrm{{CS}}}$  for experiments with Examples C.1 & C.2.
Example C.1	Example C.2
CMCS	3.2794×105	4616.4
CCS	7.6029	3.9595
i.e., the maximum of the approximate Christoffel function  $\widetilde{K}$ . Conversely, for CS it suggests that the sample complexity depends on its average, i.e.,
$$
C _ {\mathbb {C S}} = \int_ {D} \widetilde {K} (\mathbb {F} - \mathbb {F}) (\theta) \mathrm {d} \rho (\theta) = \frac {1}{M} \sum_ {i = 1} ^ {M} \widetilde {K} (\mathbb {F} - \mathbb {F}) (i).
$$
In Table 1 we report these values for the two experiments. As is evident, CS leads to a greatly reduced constant in both cases over MCS, which is in agreement with the general improvement in PSNR for this method. We note, however, that these constants do not capture all the phenomena seen in the experiments. For example, MCS performs worse for Example C.2 than it does for Example C.1. Yet the size of the constants would suggest otherwise. This is an interesting gap between the theory and practice.
C.7 Variations involving  $C > 1$
The setup described so far (sampling the Fourier transform) is a discrete model for so-called single-coil MRI. However, modern MR scanners are typically employ parallel coils. Here, multiple receive coils acquire Fourier measurements in parallel. Let  $C \geq 1$  be the number of coils. Then one can model the measurements in parallel MRI as
$$
b _ {c} = P _ {\Omega_ {c}} F S _ {c} f ^ {*} + e _ {c}, \quad c = 1, \dots , C.
$$
Here  $b_{c} \in \mathbb{C}^{m_{c}}$  are the measurements in the  $c$ th coil,  $\Omega_{c} \subseteq \{1, \dots, N\}$  with  $|\Omega_{c}| = m_{c}$  and  $S_{c}$  is a diagonal matrix, known as the sensitivity profile of the coil. See, e.g., [76, 116].
This is a multimodal sampling model that can be cast into our framework. Indeed, we replace the single sampling operator (C.1) with  $C$  sampling operators defined by
$$
L _ {c} (i) (x) = \sqrt {M / N} \big ((F S _ {c} x) _ {j} \big) _ {j \in \Delta_ {i}} \in \mathbb {C} ^ {p}, \quad \forall x \in \mathbb {C} ^ {N}, c = 1, \ldots , C.
$$
Note that empirical nondegeneracy (2.5) is easily shown to be equivalent to the condition
$$
\alpha \leq \sum_ {c = 1} ^ {C} | (S _ {c}) _ {i i} | ^ {2} \leq \beta , \quad \forall i = 1, \dots , N.
$$
Sensitivity profiles are often designed so that this property holds with moderate  $\alpha$ ,  $\beta$ . Note that in practice the sensitivity profiles are not known, and have to be computed either via a calibration pre-scan or learned during the process of recovering the image. See, e.g., [5, Sec. 3.5.2-3.5.3] and references therein.
D Solving PDEs using PINNs
In this appendix, we provide further details for the example considered in Section 3.3 involving solving the nonlinear Burger's equation using PINNs. As mentioned, we implement CS in an adaptive manner for this example, using a strategy we term Christoffel Adaptive Sampling (CAS). This strategy not only applies to the numerical solution of PDEs, but to any regression problem with DNNs and general, multimodal data. We then apply this to the numerical solution of PDEs through the framework of PINNs.
D.1 PINNs for solving PDEs
Consider a spatial domain  $D \subset \mathbb{R}^d$  and time interval  $I = [0, T]$ , where  $d \geq 1$  and  $T \geq 0$ , respectively. Then we seek to learn an unknown function  $u^* : D \times [0, T] \colon \to \mathbb{C}$  given as the solution of a PDE of the form
$$
\begin{array}{l} \mathcal {L} u = f \quad \text {i n} D \times I \\ u = u _ {i} \quad \text {i n} D \times \{0 \} \\ u = u _ {\mathrm {b}} \quad \text {o n} \partial D \times (0, T ]. \\ \end{array}
$$
Here  $\mathcal{L}$  is a differential operator in the space and time variables,  $u_{\mathrm{i}}$  is a known initial condition and  $u_{\mathrm{b}}$  is a known boundary condition. Throughout this section, we use  $f$  to denote the right-hand side of the PDE and  $u^{*}$  to denote the target function (instead of  $f^{*}$ ), as is customary in the numerical PDE literature.
The general idea behind PINNs [69, 105] is to consider training data of the form
$$
\left\{ \right.\left( \right.\left(x _ {i} ^ {\mathrm {d}}, t _ {i} ^ {\mathrm {d}}\right), f \left(x _ {i} ^ {\mathrm {d}}, t _ {i} ^ {\mathrm {d}}\right)\left. \right\} _ {i = 1} ^ {m _ {\mathrm {d}}} \cup \left\{ \right.\left( \right.\left(x _ {i} ^ {\mathrm {i}}, t _ {i} ^ {\mathrm {i}}\right), u ^ {*} \left(x _ {i} ^ {\mathrm {i}}, t _ {i} ^ {\mathrm {i}}\right)\left. \right\} _ {i = 1} ^ {m _ {\mathrm {i}}} \cup \left\{ \right.\left( \right.\left(x _ {i} ^ {\mathrm {b}}, t _ {i} ^ {\mathrm {b}}\right), u ^ {*} \left(x _ {i} ^ {\mathrm {b}}, t _ {i} ^ {\mathrm {b}}\right)\left. \right\} _ {i = 1} ^ {m _ {\mathrm {b}}} \left. \right.\left. \right.
$$
where  $(x_i^{\mathrm{d}}, t_i^{\mathrm{d}}) \in D \times (0, T]$  are sample points in the interior of the domain,  $(x_i^{\mathrm{i}}, t_i^{\mathrm{i}}) \in D \times \{0\}$  are sample points at the initial condition and  $(x_i^{\mathrm{b}}, t_i^{\mathrm{b}}) \in \partial D \times (0, T]$  are sample points on the space-time boundary of the domain. Typically, these sample points are generated via MCS. One then seeks a minimizer of the problem
$$
\begin{array}{l} \min  _ {u \in \mathbb {U}} \frac {1}{m _ {\mathsf {d}}} \sum_ {i = 1} ^ {m _ {\mathsf {d}}} | \mathcal {L} u (x _ {i} ^ {\mathsf {d}}, t _ {i} ^ {\mathsf {d}}) - f (x _ {i} ^ {\mathsf {d}}, t _ {i} ^ {\mathsf {d}}) | ^ {2} + \frac {\lambda_ {\mathrm {i}}}{m _ {\mathrm {i}}} \sum_ {i = 1} ^ {m _ {\mathrm {i}}} | u (x _ {i} ^ {\mathrm {i}}, t _ {i} ^ {\mathrm {i}}) - u _ {0} (x _ {i} ^ {\mathrm {i}}, t _ {i} ^ {\mathrm {i}}) | ^ {2} \\ + \frac {\lambda_ {\mathsf {b}}}{m _ {\mathsf {b}}} \sum_ {i = 1} ^ {m _ {\mathsf {b}}} | u (x _ {i} ^ {\mathsf {b}}, t _ {i} ^ {\mathsf {b}}) - u _ {\mathsf {b}} (x _ {i} ^ {\mathsf {b}}, t _ {i} ^ {\mathsf {b}}) | ^ {2}, \\ \end{array}
$$
where  $\mathbb{U}$  is a class of DNNs  $u:\mathbb{R}^d\times \mathbb{R}\to \mathbb{R}$  and  $\lambda_{\mathrm{i}},\lambda_{\mathrm{b}} > 0$  are hyperparameters that control fitting the PDE versus fitting the boundary/initial conditions. Note that we write  $\mathbb{U}$  and  $u\in \mathbb{U}$  throughout this section instead of  $\mathbb{F}$  and  $f\in \mathbb{F}$ , to avoid confusion with the right-hand side of the PDE.
D.2 PINNs formulation in terms of the general sampling framework
We now show how to reformulate PINNs in terms of the general sampling framework. Since there are three types of data, we let  $C = 3$ . Then we define  $D_{1} = D \times (0,T)$ ,  $D_{2} = D \times \{0\}$  and  $D_{3} = \partial D \times (0,T]$ , and let  $\rho_{1}, \rho_{2}$  and  $\rho_{3}$  be measures on these domains. We then consider the sampling operators
$$
\begin{array}{l} L _ {1} (\theta) (u) = \mathcal {L} u (\theta), \quad y = (x, t) \in D _ {1} \\ L _ {2} (\theta) (u) = \sqrt {\lambda_ {2}} u (\theta), \quad y = (x, t) \in D _ {2} \tag {D.1} \\ L _ {3} (\theta) (u) = \sqrt {\lambda_ {3}} u (\theta), \quad y = (x, t) \in D _ {3}. \\ \end{array}
$$
Next, we let  $\nu_{c}:D_{c}\to \mathbb{R}$  and  $\mu_c$  be as in Assumption 2.2. Finally, for each  $c = 1,2,3$ , we draw  $m_{c}$  samples  $(\Theta_{ic})_{i = 1}^{m_c}$ , where  $\Theta_{ic} = (X_{ic},T_{ic})$ , and consider the weighted regression problem
$$
\min  _ {u \in \mathbb {U}} \sum_ {c = 1} ^ {3} \frac {1}{m _ {c}} \sum_ {c = 1} ^ {m _ {c}} \frac {1}{\nu_ {c} (\Theta_ {i c})} \left| L _ {c} \left(\Theta_ {i c}\right) (u) - L _ {c} \left(\Theta_ {i c}\right) \left(u ^ {*}\right) \right| ^ {2}. \tag {D.2}
$$
There are several factors that prohibit direct application of CS to (D.2). The first is that the sampling operator  $L_{1}$  is potentially nonlinear, since it involves the generally nonlinear PDE operator  $\mathcal{L}$ . This means that there is no theoretical basis for the use of CS. However, even if it were linear (e.g., for a linear PDE), it could be very difficult to compute the generalized Christoffel functions  $K_{c}(\mathbb{U})$  in practice, since evaluating  $K_{c}(\mathbb{U})$  at a single point  $\theta$  involves solving a maximization problem over the space  $\mathbb{U}$  (a class of DNNs). The second, and arguably more critical issue, is that PINNs are typically implemented in the highly overparametrized regime, where the number of DNN parameters greatly exceeds the amount of training data. CS is designed to provide sampling measures that ensure 'good' recovery over the whole of the space  $\mathbb{U}$ . But this is an extremely conservative approach for DNN-based learning problems, since only certain DNNs are likely to be encountered in training.
We tackle these problems in reverse order. For the second, we present an adaptive sampling approach for deep learning. This is described in the next subsection. Note that this does not assume the underlying problem is a PDE. Our presentation follows the lines of the general sampling framework. Having done this, we then describe its application to PINNs, including suitable modifications of the sampling measures and loss function that take into account the nonlinearity of the PDE operator.
D.3 CAS for deep learning with general, multimodal samples
In this section, we consider DL based on general, multimodal samples as introduced in our general sampling framework. What follows is an extension of a method first introduced in [3], which in turn is based on ideas from [35]. Let  $\mathbb{X}$  be a Hilbert space of functions  $\mathbb{R}^{d_1} \to \mathbb{R}^{d_2}$  and  $\mathbb{U} \subset \mathbb{X}$  be a class of DNNs of a fixed architecture. As in Section 2, we consider sampling operators
$$
L _ {c}: D _ {c} \to \mathcal {B} (\mathbb {X} _ {0}, \mathbb {Y} _ {c}), \quad c = 1, \ldots , C,
$$
where the  $\mathbb{Y}_c$  are Hilbert spaces. We also write  $u^{*}\in \mathbb{X}$  for the unknown function we seek to learn.
We now describe the adaptive sampling method. This exploits the so-called adaptive basis viewpoint of DNNs, as developed in [35]. Let  $n \in \mathbb{N}$  be the number of nodes in the penultimate layer of the network architecture used to define  $\mathbb{U}$  and, for convenience, assume that the output is not subject to a bias term, just a weight. Therefore, we can write any  $u \in \mathbb{U}$  as
$$
u = C ^ {\top} H, \tag {D.3}
$$
where  $H: \mathbb{R}^{d_1} \to \mathbb{R}^n$  is a DNN of the same architecture as  $u$ , except for the final layer, and  $C = (c_{ij})_{i,j=1}^{n,d_2}$  is the weight matrix on the output layer. Now let  $u_i: \mathbb{R}^{d_1} \to \mathbb{R}$  be the  $i$ th component of  $u$ ,  $i = 1,\ldots,d_2$ , and  $H_i: \mathbb{R}^{d_1} \to \mathbb{R}$  be the  $i$ th component of  $H$ ,  $i = 1,\ldots,n$ . Then we can express  $h_j$  as
$$
u _ {j} = \sum_ {i = 1} ^ {n} c _ {i j} H _ {i}, \quad j = 1, \dots , d _ {2}.
$$
In particular, each  $u_{j}$  is an element of the subspace  $\operatorname{span}\{H_1,\dots ,H_n\}$ .
Now consider an initial DNN  $\hat{u} = \hat{u}^{(0)} = (\hat{u}_j^{(0)})_{j=1}^{d_2}$  obtained via a standard (random) initialization strategy. Write  $\hat{u}$  as in (D.3) as
$$
\hat {u} ^ {(0)} = (\widehat {C} ^ {(0)}) ^ {\top} \widehat {H} ^ {(0)},
$$
let  $\widehat{H}^{(0)} = (\widehat{H}_i^{(0)})_{i=1}^n$  and consider the subspace
$$
\mathbb {H} ^ {(0)} = \operatorname {s p a n} \{\widehat {H} _ {1} ^ {(0)}, \ldots , \widehat {H} _ {n} ^ {(0)} \}
$$
so that  $\hat{u}_j^{(0)}\in \mathbb{H}^{(0)},j = 1,\ldots ,d_2$  . Let  $p_0 = \dim (\mathbb{H}^{(0)})\leq n$  . We now perform CS using this subspace. In other words, we define sampling measures
$$
\mathrm {d} \mu_ {c} ^ {(1)} (\theta) = \nu_ {c} ^ {(1)} (\theta) \mathrm {d} \rho_ {c} (\theta), \quad \nu_ {c} ^ {(1)} (\theta) = \frac {K _ {c} (\mathbb {H} ^ {(0)}) (\theta)}{p _ {0}}, \qquad c = 1, \dots , C,
$$
then, for each  $c$ , we generate  $m_c^{(1)}$  sample points  $\Theta_{ic}$  i.i.d. from  $\mu_c^{(1)}$  and generate samples as
$$
y _ {i c} = L _ {c} (\Theta_ {i c}) (u ^ {*}), \quad i = 1, \ldots , m _ {c} ^ {(1)}, c = 1, \ldots , C.
$$