SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given these samples, we then train a new DNN by applying a standard optimizer to the minimization problem
$$
\min  _ {u \in \mathbb {U}} \sum_ {c = 1} ^ {C} \frac {1}{m _ {c} ^ {(1)}} \sum_ {i = 1} ^ {m _ {c} ^ {(1)}} \frac {1}{\nu_ {c} ^ {(1)} (\Theta_ {i c})} \| y _ {i c} - L _ {c} (\Theta_ {i c}) (u) \| _ {\mathbb {Y} _ {c}} ^ {2}.
$$
Let  $\hat{u}^{(1)}\in \mathbb{U}$  be resulting DNN. The idea now is to repeat this process by drawing  $m_c^{(2)} - m_c^{(1)}$  new samples using CS for the new subspace  $\mathbb{H}^{(1)}$  and then using the full set of  $m_c^{(2)}$  samples to compute a new DNN  $\hat{u}^{(2)}$ . Continuing in this way, we generate a sequence of DNNs  $\hat{u}^{(1)},\hat{u}^{(2)},\ldots \in \mathbb{U}$  approximating the unknown function  $u^{*}$ . To be precise, consider step  $l\geq 1$ . Let  $\hat{u}^{(l - 1)}$  be given, write
$$
\hat {u} ^ {(l - 1)} = (\hat {C} ^ {(l - 1)}) ^ {\top} \hat {H} ^ {(l - 1)}, \qquad \hat {H} ^ {(l - 1)} = (\hat {H} _ {i} ^ {(l - 1)}) _ {i = 1} ^ {n}
$$
and define
$$
\mathbb {H} ^ {(l - 1)} = \operatorname {s p a n} \{\widehat {H} _ {1} ^ {(l - 1)}, \dots , \widehat {H} _ {n} ^ {(l - 1)} \}, \qquad p _ {l - 1} = \dim (\mathbb {H} ^ {(l - 1)}).
$$
We use this to construct sampling measures
$$
\mathrm {d} \mu_ {c} ^ {(l)} (\theta) = \nu_ {c} ^ {(l)} (\theta) \mathrm {d} \rho_ {c} (\theta), \quad \nu_ {c} ^ {(l - 1)} (\theta) = \frac {K _ {c} (\mathbb {H} ^ {(l - 1)}) (\theta)}{p _ {l - 1}}, \qquad c = 1, \ldots , C,
$$
and then, for each  $c$ , we generate  $m_c^{(l)} - m_c^{(l-1)}$  new sample points  $\Theta_{ic}$ ,  $i = m_c^{(l-1)} + 1, \ldots, m_c^{(l)}$ , i.i.d. from  $\mu_c^{(l-1)}$  and corresponding samples
$$
y _ {i c} = L _ {c} (\Theta_ {i c}) (u ^ {*}), \quad i = m _ {c} ^ {(l - 1)} + 1, \ldots , m _ {c} ^ {(l)}, c = 1, \ldots , C.
$$
We then combine these with the existing  $m_c^{(l-1)}$  samples, giving a total of  $m_c^{(l)}$  samples for each  $c$ , and then use this to train a new DNN via the minimization problem
$$
\min _ {u \in \mathbb {U}} \sum_ {c = 1} ^ {C} \frac {1}{m _ {c} ^ {(l)}} \sum_ {i = 1} ^ {m _ {c} ^ {(l)}} \frac {1}{\nu_ {c} ^ {(l)} (\Theta_ {i c})} \| y _ {i c} - L _ {c} (\Theta_ {i c}) (u) \| _ {\mathbb {Y} _ {c}} ^ {2}.
$$
We write  $\hat{u}^{(l)}\in \mathbb{U}$  for the resulting DNN.
D.4 CAS for PINNs
Our aim now is to apply the CAS method in the previous subsection to PINNs. In this case, we set  $d_{1} = d + 1$  and  $d_{2} = 1$ . As we now describe, we make one main modification to CAS in this case, which is to replace the true generalized Christoffel functions  $K_{c}$  with surrogates  $\widetilde{K}_{c}$  that are more amenable to computations.
Recall that  $C = 3$  in this case and that the sampling operators are as in (D.1). To define  $\widetilde{K}_1$  we simply forget the differential operator  $\mathcal{L}$  and define
$$
\widetilde {K} _ {1} (\mathbb {H} ^ {(l - 1)}) (x, t) = \sup  \left\{\frac {| h (x , t) | ^ {2}}{\| h \| _ {L _ {\rho_ {1}} ^ {2} (D _ {1})} ^ {2}}: h \in \mathbb {H} ^ {(l - 1)} \backslash \{0 \} \right\}, \quad (x, t) \in D _ {1} = D \times (0, T).
$$
Next, for the initial  $(c = 2)$  and boundary  $(c = 3)$  data, we define
$$
\widetilde {K} _ {2} (\mathbb {H} ^ {(l - 1)}) (x, t) = \sup  \left\{\frac {| h (x , t) | ^ {2}}{\| h \| _ {L _ {\rho_ {2}} ^ {2} (D _ {2})} ^ {2}}: h \in \mathbb {H} ^ {(l - 1)} \backslash \{0 \} \right\}, \quad (x, t) \in D _ {2} = D \times \{0 \}
$$
$$
\widetilde {K} _ {2} \left(\mathbb {H} ^ {(l - 1)}\right) (x, t) = \sup  \left\{\frac {| h (x , t) | ^ {2}}{\| h \| _ {L _ {\rho_ {3}} ^ {2} (D _ {3})} ^ {2}}: h \in \mathbb {H} ^ {(l - 1)} \backslash \{0 \} \right\}, \quad (x, t) \in D _ {3} = \partial D \times (0, T).
$$
Let  $\tilde{\nu}_c^{(l)}$  and  $\tilde{\mu}_c^{(l)}$  denote the corresponding weight function and sampling measures, i.e.,
$$
\mathrm {d} \tilde {\mu} _ {c} ^ {(l)} (x, t) = \tilde {\nu} _ {c} ^ {(l)} (x, t) \mathrm {d} \rho_ {c} (x, t), \quad \tilde {\nu} _ {c} ^ {(l - 1)} (x, t) = \frac {\widetilde {K} _ {c} (\mathbb {H} ^ {(l - 1)}) (x , t)}{p _ {l - 1 , c}}, \qquad c = 1, 2, 3. \tag {D.4}
$$
Then at step  $l$  of the procedure, we generate  $m_c^{(l)} - m_c^{(l-1)}$  new sample points  $(X_{ic}, T_{ic})$ ,  $i = m_c^{(l-1)} + 1, \ldots, m_c^{(l)}$ , i.i.d. from  $\tilde{\mu}_c^{(l-1)}$  and compute the corresponding samples
$$
y _ {i c} = L _ {c} \left(X _ {i c}, T _ {i c}\right) \left(u ^ {*}\right), \quad i = m _ {c} ^ {(l - 1)} + 1, \dots , m _ {c} ^ {(l)}, c = 1, 2, 3,
$$
where the  $L_{c}$  are as in (D.1). We then combine these with the existing  $m_c^{(l - 1)}$  samples and use this to train a new DNN via the minimization problem
$$
\min  _ {u \in \mathbb {U}} \sum_ {c = 1} ^ {3} \frac {1}{m _ {c} ^ {(l)}} \sum_ {i = 1} ^ {m _ {c} ^ {(l)}} \frac {1}{\tilde {\nu} _ {c} ^ {(l)} \left(\Theta_ {i c}\right)} \left| y _ {i c} - L _ {c} \left(X _ {i c}, T _ {i c}\right) (u) \right| ^ {2}. \tag {D.5}
$$
We term this method CAS for PINNs. It is summarized in Algorithm 2.
Steps 2-3 of this algorithm require some further explanation. First, as in previous examples (see Appendix B), we introduce finite grids  $Z_{c} = \{z_{ic}\}_{i = 1}^{N_{c}}$ ,  $c = 1,2,3$ , to represent the domains  $D_{c}$ ,  $c = 1,2,3$ . This is to enable efficient sampling from the measures  $\tilde{\mu}_c$ , which are now discrete measures defined over the corresponding grids. See the next section for the construction of these grids.
Second, recall from Lemma E.1 that in order to compute the functions  $\widetilde{K}_c$ , we need to construct orthonormal bases for space
$$
\operatorname {s p a n} \{\widehat {H} _ {1} ^ {(l - 1)}, \dots , \widehat {H} _ {N} ^ {(l - 1)} \}
$$
over the finite grids  $Z_{ic}$ . Previously, in Appendix B.4, we did this via QR factorization. However, the dictionary  $\{\widehat{H}_i^{(l-1)}\}_{i=1}^N$  generated in Step 1 of the algorithm is often numerically redundant. Therefore, in this case we use a truncated SVD instead. Specifically, if
$$
B _ {c} = \left(\frac {1}{\sqrt {N _ {c}}} \widehat {H} _ {j} ^ {(l - 1)} (z _ {i c})\right) _ {i, j = 1} ^ {N _ {c}, N},
$$
then we compute its SVD  $U_{c}\Sigma_{c}V_{c}^{*}$  and define the numerical dimension
$$
p _ {l - 1, c} = \max  \left\{i: \sigma_ {i c} / \sigma_ {1 c} > \delta_ {\mathrm {t o l}} \right\} \in \{1, \dots , N \}.
$$
Using this, we now obtain an orthonormal basis over the grid  $Z_{c}$  using the first  $p_{l - 1,c}$  left singular vectors, i.e., the first  $p_{l - 1,c}$  columns of  $U_{c}$ . Given this basis, we can then compute the function  $\widetilde{K}_c(\mathbb{H}^{(l - 1)})$  over the grid via Lemma E.1. To precise,
$$
\widetilde {K} _ {c} \left(\mathbb {H} ^ {(l - 1)}\right) \left(z _ {i c}\right) = \sum_ {j = 1} ^ {p _ {l - 1, c}} | (U _ {c}) _ {i j} | ^ {2}, \quad i = 1, \dots , N _ {c},
$$
and the corresponding measures  $\tilde{\mu}_c$  are given by  $\theta \sim \tilde{\mu}_c$  if
$$
\mathbb {P} (\theta = z _ {i c}) = \frac {\widetilde {K} _ {c} \left(\mathbb {H} ^ {(l - 1)}\right) \left(z _ {i c}\right)}{p _ {l - 1 , c}}, \quad i = 1, \dots , N _ {c}.
$$
D.5 Additional experimental details
In this section, we describe the setup for our experiments. The problem we consider is the Burger's equation with  $D = (-1, 1)$ ,  $T = 1$ ,
$$
\mathcal {L} u = \partial_ {t} u + u \partial_ {x} u - (0. 0 1 / \pi) \partial_ {x x} u,
$$
and initial and boundary conditions
$$
u (x, 0) = - \sin (\pi x), \quad x \in D,
$$
Algorithm 2 CAS for PINNs
input: PDE operator  $\mathcal{L}$ , initial and boundary condition  $u_{\mathrm{i}}$  and  $u_{\mathrm{b}}$ . Class of DNNs  $\mathbb{U}$  with fixed architecture, no bias on the output layer and penultimate layer of with  $N$ . Finite grids  $Z_{c} = \{z_{ic}\}_{i=1}^{N_{c}} \subset D_{c}$ , where  $N_{c} \geq N$ ,  $c = 1,2,3$ , numbers of samples  $0 = m_{c}^{(0)} < m_{c}^{(1)} < m_{c}^{(2)} < \ldots$ ,  $c = 1,2,3$ , and tolerance  $\delta_{\mathrm{tol}} > 0$ .
initialize: Initialize the DNN  $\widehat{h}^{(0)}$
for  $l = 1$  to  $t$  do
Construct the dictionary  $\{\widetilde{H}_1^{(l - 1)},\ldots ,\widetilde{H}_N^{(l - 1)}\}$  via the relation
$$
\hat {u} ^ {(l - 1)} = \sum_ {j = 1} ^ {N} \left(\hat {c} _ {j} ^ {(l - 1)}\right) \widehat {H} _ {j} ^ {(l - 1)}.
$$
Construct the sampling measures  $\tilde{\mu}_c^{(l)}, c = 1,2,3,$  as in (D.4).
For each  $c = 1,2,3$  , draw  $m_c^{(l)} - m_c^{(l - 1)}$  sample points  $(X_{i,c},T_{i,c})$ $i = m_c^{(l - 1)} + 1,\ldots ,m_c^{(l)}$
Train the DNN  $\hat{u}^{(l)}\in \mathbb{U}$  by applying a standard optimizer to (D.5) with initialization  $\hat{u}^{(l - 1)}$  end for
output: A sequence of DNN approximations  $\widehat{u}^{(1)},\widehat{u}^{(2)},\ldots ,\widehat{u}^{(t)}$  to  $u$
$$
u (- 1, t) = u (1, t) = 0, \quad t \in (0, 1 ],
$$
respectively. As mentioned, we employ three finite grids
$$
Z _ {1} = \left\{z _ {i 1} \right\} _ {i = 1} ^ {N _ {1}} = \left\{\left(x _ {i 1}, t _ {i 1}\right) \right\} _ {i = 1} ^ {N _ {1}} \in (- 1, 1) \times (0, 1 ], \quad N _ {1} = 1 6, 0 0 0,
$$
$$
Z _ {2} = \left\{z _ {i 2} \right\} _ {i = 1} ^ {N _ {2}} = \left\{\left(x _ {i 2}, t _ {i 2}\right) \right\} _ {i = 1} ^ {N _ {2}} \in (- 1, 1) \times \{0 \}, \quad N _ {2} = 2 0 0 0,
$$
$$
Z _ {3} = \left\{z _ {i 3} \right\} _ {i = 1} ^ {N _ {3}} = \left\{\left(x _ {i 3}, t _ {i 3}\right) \right\} _ {i = 1} ^ {N _ {3}} \in \{- 1, 1 \} \times (0, 1 ], \quad N _ {3} = 2 0 0 0.
$$
Following Appendix B, these are generated once beforehand by MCS with respect to the uniform measure over each domain.
We now describe further aspects of our experiments.
Methods considered. As in previous appendices, we compare CS with MCS. Specifically, we consider DNNs trained using training data constructed via CAS (i.e., Algorithm 2), and DNNs trained from standard training data obtained via MCS from the uniform measure over each domain. As in previous example, we labels these as 'CS' and 'MCS', respectively. For Algorithm 2 we use the threshold parameter  $\delta_{\mathrm{tol}} = 10^{-6}$ , since our computations are performed in single precision.
Choice of architectures and initialization. We consider fully-connected DNNs with activation function  $\rho$ ,  $L \geq 1$  hidden layers and an equal number of  $N \geq 1$  nodes per hidden layer. We term these  $\rho$ $L \times N$  DNNs. Since we are solving a regression problem, we assume the output layer is not subject to  $\rho$ . We also set the bias to be zero in this layer. For the activation function  $\rho$ , we consider either the Rectified Linear Unit (ReLU), hyperbolic tangent (tanh), Exponential Linear Unit (ELU), or sigmoid. These are defined as follows:
ReLU:  $\rho (x)\coloneqq \max \{x,0\} ,$
tanh:  $\rho (x)\coloneqq \frac{e^{2x} - 1}{e^{2x} + 1},$
ELU:  $\rho (x)\coloneqq \{x:x\geq 0,(e^x -1):x <   0\} ,$
sigmoid:  $\rho (x)\coloneqq \frac{1}{1 + e^x}.$
Note that we choose DNNs with a fixed number of nodes  $N$  for each layer. Moreover, we also focus on DNNs with depth  $L$  such that the ratio  $L / N$  is small. This is motivated by results in [62]. To be precise,  $L / N = 0.1$  in our experiments.
For the initialization strategy, we consider weights and biases with entries drawn from the Normal initialization with mean 0 and variance 0.1. Having in mind the size of architectures described in this
Table 2: Number of samples per iteration for each domain
parameter	Number of samples mc(l)
l	1	2	3	4	5	6	7	8	9
c = 1	400	1350	2300	3250	4200	5150	6100	7050	8000
c = 2	100	200	300	400	550	650	750	850	1000
c = 3	100	200	300	400	550	650	750	850	1000
Total	600	1750	2900	4050	5300	6450	7600	8750	10000
work, this initialization leads to smaller variance than commonly used initializations used PINNs such as the well-known Xavier and He Normal initializations. Besides, it has been shown that is an effective choice for function approximation, see [63, 65]. In all our experiments, the DNNs were initialized with the same seed to compare the effect of varying the samples.
Optimizers for training and parametrization. We use Adam as the optimizer in combination with an exponentially decaying learning rate with respect to the number of epochs. For both CAS and the procedure based on MCS, we train the networks following a schedule number of epochs, described as follows:
$$
n ^ {(0)} = 5 0 0 0, \quad n ^ {(l)} = n ^ {(0)} \times l, \quad l \geq 1.
$$
For the CAS strategy, we train the networks for  $n^{(l)}$  epochs on each new set of samples, using the weights and biases from the previous set as initialization. We repeat this procedure 9 times, i.e. we train for a total of 225,000 epochs. For MCS, we train the network for a total of 225,000 epochs as well, stopping every  $n^{(l)}$  epochs to add more data points drawn from the uniform distribution, and continuing to train from the previous point.
Training data and design of experiments. To measure the performance of these methods, we present the average testing error and run statistics over 20 trials for both sampling strategies. We train our DNNs over a set of training data consisting of the values
$$
\bigcup_ {c = 1} ^ {3} \{((X _ {i c}, T _ {i c}), u (X _ {i c}, T _ {i c})) \} _ {i = 1} ^ {m _ {c} ^ {(l)}}.
$$
For each  $c = 1,2,3$ , we generate each data set of size  $m_c^{(l)}$ , with  $0 < m_c^{(1)} < \ldots < m_c^{(l)} < \ldots < m_c^{(9)}$ , we sample 20 i.i.d. sets of points  $\{\Theta_{ic}\}_{i=1}^{m_c^{(9)}}$  from the sampling measures for each strategy. The precise values for the numbers of samples  $m_c^{(l)}$  are given in Table 2.
Testing data and error metric. Let  $u^{*}$  be the solution of the PDE and  $\hat{u}$  be a DNN approximation to it. We compute the (discrete) relative error
$$
E (u ^ {*}) = \sqrt {\frac {\frac {1}{M} \sum_ {z \in Z ^ {t}} | u ^ {*} (z) - \hat {u} (z) | ^ {2}}{\frac {1}{M} \sum_ {z \in Z ^ {t}} | u ^ {*} (z) | ^ {2}}} = \sqrt {\frac {\sum_ {z \in Z ^ {t}} | u ^ {*} (z) - \hat {u} (z) | ^ {2}}{\sum_ {z \in Z ^ {t}} | u ^ {*} (z) | ^ {2}}}.
$$
where  $Z^t = Z_1^t \cup Z_2^t \cup Z_3^t$  and  $M = M_1 + M_2 + M_3$ . Here, for each  $c = 1, 2, 3$ ,  $Z_c^t$  is a test grid of size  $M_c$ , with points drawn randomly from the uniform measure over the corresponding domain. We set  $M_1 = 8000$ ,  $M_2 = 1000$  and  $M_3 = 1000$  in this work.
To generate the training and testing data, we have to evaluate the true PDE solution  $u$  over the corresponding grids. We compute this solution to an accuracy of  $10^{-7}$  using the numerical solver BURGERS SOLUTION, which is implemented in [14]. This is available from https://people.math.sc.edu/Burkardt/m_src/burgers_solution/burgers_solution.html under the GNU LGPL license.
Plots. As in Appendix B.5, our error plots display the geometric mean error value and a shaded region corresponding to one (geometric) standard deviation. We also plot the points generated by the final iteration of CAS. We visualize the density of these points using the package densityScatterChart in MATLAB using 'ksdensity' as density method and  $\alpha \in [0.2, 0.5]$ . See https://www.mathworks.com/matlabcentral/fileexchange/95828-densityscatterchart?s_tid=FX_rc3_behav for more details.
Figure 9: CAS for solving PDEs with PINNs. Plots of (left) the average test error  $E(u^{*})$  versus the number of samples used for training ReLU and ELU  $5 \times 50$  DNNs using CAS (solid line) and MCS (dashed line), (middle) the samples generated by the CAS for the ReLU  $5 \times 50$  DNN and (right) the samples generated by CAS for the ELU  $5 \times 50$  DNN. The colour indicates the density of the points.
Figure 10: CAS for solving PDEs with PINNs. Plots of (left) the average test error  $E(u^{*})$  versus the number of samples used for training  $3 \times 30$  DNNs using CAS (solid line) and MCS (dashed line), (middle) the samples generated by the CAS for the tanh (top) and ReLU (bottom)  $3 \times 30$  DNNs and (right) the samples generated by CAS for the sigmoid (top) and ELU (bottom)  $3 \times 30$  DNNs. The colour indicates the density of the points.
Implementation, hardware and compute time. Our experiments have been implemented in TensorFlow version 2.5. This implementation has been inspired by works [3, 19].
The computations were performed in single precision using NVIDIA Tesla P100 GPUs on Compute Canada's Cedar compute cluster. To be precise, each node used in these experiments has 24 CPU cores and 128GB of memory and four Tesla P100 GPUs with 16GB memory. See https://docs. alliancecan.ca/wiki/Cedar for further information.
The compute time is roughly one day for the computations involving the  $\rho 3\times 30$  and  $\rho 5\times 50$  DNNs and 2 days for those involving the  $\rho 10\times 100$  DNNs. Therefore, the total compute time is approximately four days.
Figure 11: CAS for solving PDEs with PINNs. Plots of (left) the average test error  $E(u^{*})$  versus the number of samples used for training  $10 \times 100$  DNNs using CAS (solid line) and MCS (dashed line), (middle) the samples generated by the CAS for the tanh (top) and ReLU (bottom)  $10 \times 100$  DNNs and (right) the samples generated by CAS for the sigmoid (top) and ELU (bottom)  $10 \times 100$  DNNs. The colour indicates the density of the points.
D.6 Further experiments and discussion
In Fig. 9 we show the same results as in Fig. 3, except using the ReLU and ELU activation functions instead. Here, none of the methods lead to a decreasing test error as the number of samples increases, although the errors for MCS are somewhat larger than those of CAS. As is evident from the plots of the samples points, CAS fails to adapt the sampling measures to the PDE solution in this experiment, unlike in the case of Fig. 3.
This experiment indicates that the performance of CAS in relation to MCS is heavily influenced by the choice of architecture. To further explore this phenomenon, in Fig. 10 and Fig. 11 we consider  $3 \times 30$  and  $10 \times 100$  DNNs, respectively. In the former case, we once more see no decrease in the test error for the ReLU and ELU activation functions. The test error for the tanh and sigmoid activation functions does decrease with increasing  $m$ . Yet, CAS leads only to a marginal benefit over MCS for the tanh activation, and leads to worse performance for the sigmoid activation. As can be seen from the plot of the points, the CAS fails to adequately cluster the points in the case of the latter.
In Fig. 11, we consider  $10 \times 100$  DNNs and see decreasing errors for the tanh and sigmoid activations, with CAS giving better performance than MCS. For the ELU activation function, while the error is (slowly) decreasing, CAS actually performs worse than MCS. Once more, for the ReLU activation function, we see no convergence.
The discrepancy in performance between the different activation functions is mirrored when we examine the training loss. In Fig. 12 we show the training loss versus number of epochs. For the best-performing tanh and sigmoid activations, we see significant decrease of the training loss, whereas for the other activations we see slower or sometimes no decrease. Note that all DNNs are trained using the same hyperparameters. This behaviour is in some senses unsurprising, since the worst-performing activation functions are only piecewise smooth and the training loss involves the PDE operator. Another notable feature is that MCS generally attains a smaller training error than CAS. Therefore, there is scope to further improve the performance of CAS by improving the training procedure.
Figure 12: Plots of the training loss versus the number of epochs for the CAS and MCS methods with  $5 \times 50$  DNNs using 20 trials. The vertical black lines indicate the number of epochs at which more samples are added in the training.
Finally, in Fig. 13 we plot the functions  $\widetilde{K}_c$ ,  $c = 1,2,3$ , that are used to generate the sample points at the final iteration (see Appendix D.4). Here, once again we see that the tanh and sigmoid activation functions lead to functions  $\widetilde{K}_c$  that are peaked according to the behaviour of the true PDE solution  $u^*$ . However, the ReLU and ELU activation functions fail to capture this behaviour.
E Proofs of the theoretical results
In this appendix, we give the proofs of the results from the main paper. We commence with the following lemma, which gives several properties of the generalized Christoffel function that will be needed later.
Lemma E.1 (Properties of  $K(\mathbb{F})(\theta)$ ). The generalized Christoffel function  $K$  (see Definition 2.6) has the following properties.
Figure 13: Plots of the functions  $\widetilde{K}_c$ ,  $c = 1, 2, 3$  (left to right), that are used to generate the sample points in the final iteration of CAS in the case of the  $5 \times 50$  DNNs with tanh, sigmoid, ReLU and ELU activation functions (top to bottom). In the third column, the blue and red curves correspond to the  $x = 1$  and  $x = -1$  boundaries, respectively.
(i) Let  $\mathbb{F}$  be a finite-dimensional subspace of  $\mathbb{X}$ . Then
$$
K (\mathbb {F}) (\theta) \leq \sum_ {i = 1} ^ {n} \| L (\theta) \left(f _ {i}\right) \| _ {\mathbb {Y}} ^ {2} \tag {E.1}
$$
for any orthonormal basis  $\{f_i\}_{i=1}^n$  of  $\mathbb{F}$ . Furthermore, if  $\mathbb{Y}$  has dimension  $p$  then
$$
K (\mathbb {F}) (\theta) \geq \frac {1}{p} \sum_ {i = 1} ^ {n} \| L (\theta) (f _ {i}) \| _ {\mathbb {Y}} ^ {2}.
$$
In particular, (E.1) holds with equality when  $\dim (\mathbb{Y}) = 1$ .
(ii) Let  $\mathbb{F}_1, \mathbb{F}_2 \subseteq \mathbb{X}$ . Then
$$
K \left(\mathbb {F} _ {1} \cup \mathbb {F} _ {2}\right) (\theta) = \max  \left\{K \left(\mathbb {F} _ {1}\right) (\theta), K \left(\mathbb {F} _ {2}\right) (\theta) \right\}.
$$
(iii) Let  $\mathbb{F}_1, \mathbb{F}_2$  be finite-dimensional subspaces of  $\mathbb{X}$ . Then
$$
K (\mathbb {F} _ {1} + \mathbb {F} _ {2}) (\theta) \leq 2 \left(K (\mathbb {F} _ {1}) (\theta) + K (\mathbb {F} _ {2}) (\theta)\right).
$$
(iv) Let  $\mathbb{F} \subseteq \mathbb{X}$  and  $\mathcal{B}(\mathbb{F}) = \{f / \| f\|_{\mathbb{X}} : f \in \mathbb{F} \backslash \{0\}\}$ . Then
$$
K (\mathcal {B} (\mathbb {F})) (\theta) = K (\mathbb {F}) (\theta).
$$
Proof. We commence with the proof of (i). Write  $f \in \mathbb{F}$ ,  $f \neq 0$ , as  $f = \sum_{i=1}^{n} c_i f_i$  for  $c_i \in \mathbb{C}$  with  $\sum_{i=1}^{n} |c_i|^2 = \|f\|_{\mathbb{X}}^2$ . Then
$$
\| L (\theta) (f) \| _ {\mathbb {Y}} ^ {2} = \left\| \sum_ {i = 1} ^ {n} c _ {i} L (\theta) (f _ {i}) \right\| _ {\mathbb {Y}} ^ {2} \leq \sum_ {i = 1} ^ {n} | c _ {i} | ^ {2} \sum_ {i = 1} ^ {n} \| L (\theta) (f _ {i}) \| _ {\mathbb {Y}} ^ {2} = \| f \| _ {\mathbb {X}} ^ {2} \sum_ {i = 1} ^ {n} \| L (\theta) (f _ {i}) \| _ {\mathbb {Y}} ^ {2},
$$
$$
K (\mathbb {F}) (\theta) \leq \sum_ {i = 1} ^ {n} \| L (\theta) (f _ {i}) \| _ {\mathbb {Y}} ^ {2}.
$$
For the lower bound, let  $\{e_1,\ldots ,e_p\}$  be an orthonormal basis of  $\mathbb{Y}$ . Fix  $j\in \{1,\dots ,p\}$  and consider  $f = \sum_{i = 1}^{n}c_{i}f_{i}$ , where  $c_{i} = \langle e_{j},L(\theta)(f_{i})\rangle_{\mathbb{Y}}$ . Then
$$
\| L (\theta) (f) \| _ {\mathbb {Y}} ^ {2} = \left\| \sum_ {i = 1} ^ {n} \langle e _ {j}, L (\theta) (f _ {i}) \rangle_ {\mathbb {Y}} L (\theta) (f _ {i}) \right\| _ {\mathbb {Y}} ^ {2} \geq \left(\sum_ {i = 1} ^ {n} | \langle e _ {j}, L (\theta) (f _ {i}) \rangle_ {\mathbb {Y}} | ^ {2}\right) ^ {2}.
$$
We also have
$$
\left\| f \right\| _ {\mathbb {X}} ^ {2} = \sum_ {i = 1} ^ {n} \left| \langle e _ {j}, L (\theta) (f _ {i}) \rangle_ {\mathbb {Y}} \right| ^ {2}.
$$
Therefore, we deduce that
$$
K (\mathbb {F}) (\theta) \geq \sum_ {i = 1} ^ {n} | \langle e _ {j}, L (\theta) (f _ {i}) \rangle_ {\mathbb {Y}} | ^ {2}.
$$
Since  $j$  was arbitrary, we now sum over  $j$  and use Parseval's identity to obtain
$$
p K (\mathbb {F}) (\theta) \geq \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {p} | \langle e _ {j}, L (\theta) (f _ {i}) \rangle_ {\mathbb {Y}} | ^ {2} = \sum_ {i = 1} ^ {n} \| L (\theta) (f _ {i}) \| _ {\mathbb {Y}} ^ {2},
$$