SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\boldsymbol {\beta}} \frac {1}{n} \sum_ {i = 1} ^ {n} | y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta} | ^ {2} + \left(\delta \| \boldsymbol {\beta} \| _ {*} + \frac {1}{n} \| \boldsymbol {y} \| _ {1}\right) ^ {2}.
$$
The proposition is proved in Appendix B. It establishes the equivalence between  $\ell_{\infty}$ -adversarial training and Lasso (even though there is not a closed-form expression for the map between  $\delta$  and  $\lambda'$ ). Indeed, under the assumptions of the propositions,  $\widehat{\beta}$  is the solution of  $\ell_{\infty}$ -adversarial problem only if it is the solution of
$$
\min  _ {\beta} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta}\right) + \lambda^ {\prime} \| \boldsymbol {\beta} \| _ {1}, \tag {8}
$$
for some  $\lambda'$ . We can prove this statement using the proposition: under the appropriate assumptions, the  $\ell_{\infty}$ -adversarial problem solution is also a solution to the problem
$$
\min  _ {\boldsymbol {\beta}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta}\right) ^ {2} + \left(\delta | | \boldsymbol {\beta} | | _ {1} + \frac {1}{n} | | \boldsymbol {y} | | _ {1}\right) ^ {2},
$$
which in turn, is the Lagrangian formulation of the following constrained optimization problem
$$
\min  _ {\boldsymbol {\beta}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta}\right) ^ {2} \text {s u b j e c t} \left(\delta \| \boldsymbol {\beta} \| _ {1} + \frac {1}{n} \| \boldsymbol {y} \| _ {1}\right) ^ {2} \leq \Delta ,
$$
for some  $\Delta \geq \frac{1}{n} ||\pmb{y}||_1$ . The constraint can be rewritten as  $\delta ||\beta ||_1 \leq \sqrt{\Delta} - \frac{1}{n} ||\pmb{y}||_1$ . And, the result follows because (8) is the Lagrangian formulation of this modified problem. A similar reasoning could be used to connect the result for  $\ell_2$ -adversarial training with ridge regression.
More generally, even when the condition on  $\hat{\beta}$  is not satisfied and if  $\pmb{y}$  is negative, we show in Appendix B that for zero-mean and symmetrically distributed covariates, i.e.,  $\mathbb{E}[x] = 0$  and  $(\pmb{x} \sim -\pmb{x})$ , adversarial training has approximately the same solution as
$$
\min _ {\boldsymbol {\beta}} \frac {1}{n} \sum_ {i = 1} ^ {n} | y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta} | ^ {2} + \left(\delta \| \boldsymbol {\beta} \| _ {*} + \frac {1}{n} \boldsymbol {s} ^ {\top} \boldsymbol {y}\right) ^ {2},
$$
for  $s = \mathrm{sign}(\pmb{y} - \pmb{X}^{\top}\widehat{\beta})$ . This explains the similarities in the regularization paths.
5.2 Transition into the interpolation regime
The discussion above hints at the similarities between adversarial training and parameter-shrinking methods. Interestingly, Lasso and ridge regression are also connected to minimum-norm interpolators. The ridge regression solution converges to the minimum-norm solution as the parameter vanishes
Figure 4: Training mean squared error vs inverse adversarial radius. Left: for ridge and  $\ell_2$ -adversarial training. Right: for Lasso and  $\ell_{\infty}$ -adversarial training. The error bars give the median and the 0.25 and 0.75 quantiles from 5 realizations. The vertical black lines show  $\bar{\delta}$  in (5). Figure S.9 (appendix) shows the test MSE. For ridge regression or Lasso (see Section 3), the  $x$ -axis should be read as  $1 / \lambda$ , rather than  $1 / \delta$ . We generate the data synthetically using an isotropic Gaussian feature model (see Section 7) with  $n = 60$  training data points and  $p = 200$  features.
i.e.,  $\widehat{\beta}^{\mathrm{ridge}}(\lambda) \to \widehat{\beta}^{\mathrm{min}-\ell_2}$  as  $\lambda \to 0^{+}$ . Similarly, there is a relation between the minimum  $\ell_1$ -norm solution and Lasso. The relation requires additional constraints because, for the overparameterized case, Lasso does not necessarily have a unique solution. Nonetheless, it is proved in [41, Lemma 7] that the Lasso solution by the LARS algorithm satisfies  $\widehat{\beta}^{\mathrm{lasso}}(\lambda) \to \widehat{\beta}^{\mathrm{min}-\ell_1}$  as  $\lambda \to 0^{+}$ .
For a sufficiently small  $\delta$ , the solution of  $\ell_2$ -adversarial training equals the minimum  $\ell_2$ -norm solution; and, the solution of  $\ell_{\infty}$ -adversarial training equals the minimum  $\ell_1$ -norm solution. There is a notable difference though: while this happens only in the limit for ridge regression and Lasso, for adversarial training this happens for all values  $\delta$  smaller than the threshold  $\bar{\delta}$ . We illustrate this phenomenon next. Unlike ridge regression and Lasso which converge towards the interpolation solution, adversarial training goes through abrupt transitions and suddenly starts to interpolate the data. Figure 4 illustrates this phenomenon in synthetically generated data (with isotropic Gaussian feature, see Section 7).
Intuitive explanation for abrupt transitions. To get insight into why the abrupt transition occurs we present the case where  $n = 1$ , i.e., the dataset has a single data point  $(\boldsymbol{x}, y)$ , where  $y = 1$  and  $\| \boldsymbol{x} \|_2 = 1$ . We can write  $R^{\mathrm{adv}}(\beta; \delta, \| \cdot \|_2) = (\mathcal{L}(\beta))^2$ , where
$$
\mathcal {L} (\boldsymbol {\beta}) = | y - \boldsymbol {x} ^ {\top} \boldsymbol {\beta} | + \delta \| \boldsymbol {\beta} \| _ {2}.
$$
Figure 5: Function  $\mathcal{L}(\beta)$  for  $\beta = \alpha x$ .
The function  $\mathcal{L}$  is necessarily minimized along the subspace spanned by the vector  $\pmb{x}$ . Now, along this line, the
function  $\mathcal{L}$  is piecewise linear with three segments, see Figure 5. One of the three segments changes the slope sign as  $\delta$  decreases and the minimum of the function changes abruptly, thus explaining why abrupt transitions occur.
6 Relation to robust regression and square-root Lasso
In this section we establish that both adversarial training and square-root Lasso can be integrated into the robust regression framework. We further investigate the similarities between the methods by analyzing the prediction error upper bounds.
Robust regression framework. Robust linear regression considers the minimization of the following cost function
$$
R ^ {\text {r o b u s t}} (\boldsymbol {\beta}; \mathcal {S}) = \max  _ {\boldsymbol {\Delta} \in \mathcal {S}} \| \boldsymbol {y} - (\boldsymbol {X} + \boldsymbol {\Delta}) \boldsymbol {\beta} \| _ {2}, \tag {9}
$$
where the disturbance matrix  $\Delta$  is constrained to belong to the 'disturbance set'  $S$ . We connect robust regression, square-root Lasso and adversarial training. The next proposition gives the equivalence between robust regression for a row-bounded disturbance sets  $\mathcal{R}_{p,\delta}$  and  $\ell_p$ -adversarial training, see the appendix for the proof.
Proposition 5. For a disturbance set with the rows bounded by  $\delta$ :
$$
\mathcal {R} _ {p, \delta} = \left\{\left[ \begin{array}{c c c} - & \boldsymbol {\Delta x} _ {1} & - \\ & \vdots \\ - & \boldsymbol {\Delta x} _ {n} & - \end{array} \right]: \| \boldsymbol {\Delta x} _ {i} \| \leq \delta , \forall i \right\},
$$
we have that  $\arg \min_{\beta} R^{\mathrm{robust}}(\beta, \mathcal{R}_{p,\delta}) = \arg \min_{\beta} R^{\mathrm{adv}}(\beta; \delta, \| \cdot \|$
On the other hand, there is an equivalence between square-root Lasso and robust regression for column-bounded disturbance sets  $\mathcal{C}_{2,\delta}$ . This was established by Xu et al. [33, Theorem 1] and we repeat it below.
Proposition 6. [33] For a disturbance set with columns bounded by  $\delta$ :
$$
\mathcal {C} _ {2, \delta} = \left\{\left[ \begin{array}{c c c} | & \dots & | \\ | & & | \end{array} \right]: \| \boldsymbol {\zeta} _ {i} \| _ {2} \leq \delta , \forall i \right\},
$$
we have that  $\arg \min_{\beta} R^{\mathrm{robust}}(\beta; \mathcal{C}_{2,\delta}) = \arg \min_{\beta} R^{\sqrt{\mathrm{lasso}}}(\beta, \delta)$ .
The above discussion hints at the similarities between adversarial training and square-root Lasso, as instances of robust regression under different constraints. Square-root Lasso minimizes  $R^{\sqrt{\mathrm{lasso}}}(\beta, \lambda) = n^{-1/2} \| \pmb{y} - \pmb{X}\pmb{\beta} \|_2 + \lambda \| \pmb{\beta} \|_1$  (See Section 3). The main motivation for the square-root Lasso is that it is a 'pivotal' method for sparse recovery: that is, it attains near-oracle performance without knowledge of the variance levels to set the regularization parameter [32]. As we will show in the next section, a similar property applies to  $\ell_{\infty}$ -adversarial training.
Fixed-design analysis and similarities with square-root Lasso. In this section, we assume that the data was generated as:  $y_{i} = \pmb{x}_{i}^{\top}\pmb{\beta}^{*} + \varepsilon_{i}$  where  $\beta^{*}$  is the parameter vector used to generate the data. Under these assumptions, we can derive an upper bound for the (in-sample) prediction error:
Theorem 2. Let  $\delta >\delta^{*} = 3\frac{\|\boldsymbol{X}^{\top}\boldsymbol{\varepsilon}\|_{\infty}}{\|\boldsymbol{\varepsilon}\|_{1}}$ , the prediction error of  $\ell_{\infty}$ -adversarial training satisfies the bound:
$$
\frac {1}{n} \| \boldsymbol {X} (\widehat {\boldsymbol {\beta}} - \boldsymbol {\beta} ^ {*}) \| _ {2} ^ {2} \leq 8 \delta \| \boldsymbol {\beta} ^ {*} \| _ {1} \left(\frac {1}{n} \| \boldsymbol {\varepsilon} \| _ {1} + 1 0 \delta \| \boldsymbol {\beta} ^ {*} \| _ {1}\right). \tag {10}
$$
For comparison, we also provide the result for Lasso: (Adapted from [31, Thm. 7.13, p. 210])
Theorem 3. [31] Let  $\lambda >\lambda^{*} = 3\| \frac{X^{\top}\varepsilon}{n}\|_{\infty}$ , the prediction error of Lasso satisfies the bound:
$$
\frac {1}{n} \| \boldsymbol {X} (\widehat {\boldsymbol {\beta}} - \boldsymbol {\beta} ^ {*}) \| _ {2} ^ {2} \leq 8 \lambda \| \boldsymbol {\beta} ^ {*} \| _ {1}. \tag {11}
$$
Without additional constraints (see Remark 5) it can be shown that for Lasso it is not possible to improve the above bound. To satisfy this bound, however, Lasso requires knowledge of the magnitude of the noise  $\varepsilon$ . In Theorem 3,  $\lambda^{*} = \frac{3}{n}\|X^{\top}\varepsilon\|$ . Hence, if we rescale  $\varepsilon$  (i.e.,  $\varepsilon \rightarrow \eta \varepsilon$ ) then a correspondent change in magnitude follows in  $\lambda^{*}$  (i.e.,  $\lambda^{*} \rightarrow \eta \lambda^{*}$ ). Square-root Lasso [32] avoids this problem and achieves a similar rate even without knowing the variance: i.e., it is a 'pivotal' method. Our method has similar properties and allows us to set  $\delta$  without estimating the variance. This can be seen in Theorem 2, where re-scaling  $\varepsilon$  does not alter the value of  $\delta^{*} = 3\|X^{\top}\varepsilon\|_{\|\varepsilon\|_{1}}$  because it affects the numerator and denominator simultaneously.
For instance, if we assume  $\varepsilon$  has i.i.d.  $\mathcal{N}(0,\sigma^2)$  entries and that the matrix  $X$  is fixed with  $\max_{j = 1,\ldots ,m}\| x_j\|_{\infty}\leq M$ . For  $\lambda \propto M\sigma \sqrt{(\log p) / n}$ , we (with high-probability) satisfy the condition in Theorem 3, obtaining:  $\frac{1}{n}\| X(\widehat{\beta} -\beta^{*})\|_{2}^{2}\lesssim M\sigma \sqrt{(\log p) / n}$ . For adversarial training, we can set:  $\delta \propto M\sqrt{(\log p) / n}$ , and (with high-probability) satisfy the theorem condition, obtaining the same bound. Notice that the choice of  $\delta$  is not dependent on  $\sigma$ . We provide a full analysis in Appendix C.3.
Remark 4 (On the relation between  $\bar{\delta}$  and  $\delta^{*}$ ). For sufficiently large  $n$ , we have  $\delta^{*} > \bar{\delta}$  in the scenario above—the noise  $\varepsilon$  has i.i.d. normal entries  $\mathcal{N}(0, \sigma^2)$  and the matrix  $X$  is fixed. We can prove it by contradiction: if  $\delta^{*} \leq \bar{\delta}$  then we could apply the bound from Theorem 2 to the minimum  $\ell_{1}$ -norm
interpolator (due to Theorem 1). Hence,  $\sigma^2\approx \frac{1}{n}\| \varepsilon \| _2^2 = \frac{1}{n}\| X(\widehat{\beta} -\beta^*)\| _2^2\lesssim M\sigma \sqrt{(\log p) / n}$  and we can always choose a sufficiently large value of  $n$  for which the inequality is false.
Remark 5. In the original square-root Lasso paper [32], a bound is obtained for the  $\ell_2$  parameter distance:  $\| \widehat{\beta} -\beta^{*}\|_{2}^{2}$  for the case  $X$  satisfy the restricted eigenvalue condition and  $\beta^{*}$  is sparse. Under these more strict assumptions (restricted eigenvalue condition) we can also obtain a faster convergence rate for the prediction error. Here we focus on the fixed-design prediction error without additional assumptions on  $X$ . For other predictors, these other proofs follow similar steps and yield similar requirements on  $\lambda^{*}$ , see [31, Chapter 7]. However, we leave these and other analysis (such as variable selection consistency) of adversarial training for future work.
We study five different examples. The main goal is to experimentally confirm our theoretical findings. For each scenario, we compute and plot: train and test MSE for different choices of  $p$ ,  $n$  and  $\delta$ . For comparison purposes, we also compute and plot train and test MSE for Lasso and ridge regression and minimum-norm interpolators. Finally, in line with our discussion in Section 4, we compute and plot  $\bar{\delta}$  as a function of  $p / n$ . In all the numerical examples the adversarial training solution is implemented by minimizing (2) using CVXPY [42]. The scenarios under consideration are described below (see Appendix D for additional details)
1. Isotropic Gaussian feature model. The output is a linear combination of the features plus additive noise:  $y_{i} = \pmb{x}_{i}^{\top}\pmb{\beta} + \epsilon_{i}$ , for Gaussian noise and covariates:  $\epsilon_{i} \sim \mathcal{N}(0,\sigma^{2})$  and  $\pmb{x}_i \sim \mathcal{N}(0,\pmb{I}_p)$ .
2. Latent-space feature model [26, Section 5.4]. The features  $\pmb{x}$  are noisy observations of a lower-dimensional subspace of dimension  $d$ . A vector in this latent space is represented by  $\pmb{z} \in \mathbb{R}^d$ ,  $\pmb{x} = \pmb{W}\pmb{z} + \pmb{u}$ . The output is a linear combination of the latent space plus noise.
3. Random Fourier features model [43]. We apply a random Fourier feature map to inputs of the Diabetes dataset [18]. Random fourier features are obtained by the transformation  $\boldsymbol{x}_i = \sqrt{2 / m}\cos (\boldsymbol {W}\boldsymbol {z}_i + \boldsymbol {b})$  where the entries of  $\mathbf{W}$  and  $\pmb{b}$  are independently sampled. It can be seen as a one-layer untrained neural network and approximates the Gaussian kernel feature map [43].
4. Random projection model [40]. The data is generated as in the isotropic Gaussian feature model, but we only observe  $S\pmb{x}_i$ , i.e., a projection of the inputs.
5. Phenotype prediction from genotype. We illustrate our method on the Diverse MAGIC wheat dataset [44] from the National Institute for Applied Botany. We use a subset of the genotype to predict one of the continuous phenotypes.
Results. We experimentally corroborate Theorem 1 in all scenarios: In Figures 4 and S.8 (Appendix) we show the train MSE as we change the adversarial radius  $\delta$ , confirming the abrupt transitions into the interpolation regime and also that  $\bar{\delta}$  is the transition point. In Figures S.9 and S.10 we show the corresponding test MSE without and in the presence of an adversary, respectively. Interestingly, the adversarial radius that yields the best results is not always equal to the radius  $\delta_{\mathrm{test}}$  the model will be evaluated on.
Figures 2 and S.5 (Appendix) display  $\bar{\delta}$  as a function of the ratio  $p / n$ . We observe that  $\bar{\delta} /\mathbb{E}[\| \boldsymbol {x}\| ]$  is growing in all examples and we would still expect improved robustness in light of Proposition 2. The Random Fourier features model is the only case where  $\bar{\delta}$  seems to decrease (in absolute value). However,  $\mathbb{E}\left[\| \boldsymbol {x}\| _1\right]$  is also decreasing (and at a faster rate). Figures S.6 and S.7 give test MSE without and in the presence of an adversary for the minimum-norm interpolator as a function of the ratio  $p / n$ , confirming improved adversarial robustness as  $p$  grows.
Figure 6: Test normalized MSE (NMSE) in the MAGIC dataset.
Figure 6 provides a comparison of the test error of the different methods under study. For Lasso, ridge and
adversarial training, we use the best  $\delta$  or  $\lambda$  available for each method (obtained via grid search). We note that while for  $p = 1000$  optimally tuned Lasso and  $\ell_{\infty}$ -adversarial training significantly outperform the corresponding minimum  $\ell_1$ -norm interpolator. As  $p$  increases, the performance of the three different methods becomes quite similar. It is also an example where the minimum  $\ell_1$ -norm outperforms the minimum  $\ell_2$ -norm interpolator (for large  $p$ ). In the same setup, Figure S.12
study a choice of adversarial radius inspired by Theorem 2. We use  $\delta \propto \|X\xi\|_{\infty}/\|\xi\|_{1}$  for  $\xi$  a vector with zero-mean normal entries. We use a random  $\xi$ , since we do not know the true additive noise. Even with this approximation,  $\ell_{\infty}$ -adversarial training performs comparably with Lasso with the regularization parameter set using 5-fold cross-validation doing a full search in the hyperparameter space. The figure also provides a comparison with square-root Lasso under a similar setting.
8 Results for general loss functions
The following theorem can be used to generalize Proposition 1.
Theorem 4. Let  $\mathcal{L}:\mathbb{R}\to \mathbb{R}$  be a convex and lower-semicontinuous function, for every  $\delta \geq 0$
$$
\max  _ {\| \boldsymbol {\Delta} \boldsymbol {x} \| \leq \delta} \mathcal {L} \left(\left(\boldsymbol {x} + \boldsymbol {\Delta} \boldsymbol {x}\right) ^ {\top} \boldsymbol {\beta}\right) = \max  _ {s \in \{- 1, 1 \}} \mathcal {L} \left(\boldsymbol {x} ^ {\top} \boldsymbol {\beta} + \delta s \| \boldsymbol {\beta} \| _ {*}\right). \tag {12}
$$
We can find a closed-form expression for  $s_* = \arg \max_{s \in \{-1,1\}} \mathcal{L}(\pmb{\beta}^\top x + \delta s \| \pmb{\beta} \|_*)$  in many cases of interest. For regression problems, given any non-decreasing function  $\ell : \mathbb{R}^+ \to \mathbb{R}^+$ , and  $\mathcal{L}(\pmb{x}^\top \pmb{\beta}) = \ell(|\pmb{x}^\top \pmb{\beta} - y|)$ . If  $\ell$  is lower semicontinuous and convex then so will  $\mathcal{L}$  and the result of the theorem holds (the squared loss  $\ell(z) = z^2$  is a special case). Then  $s_* = -\mathrm{sign}(y - \pmb{x}^\top \pmb{\beta})$  and
$$
\max  _ {\| \boldsymbol {\Delta} \boldsymbol {x} \| \leq \delta} \ell (| (\boldsymbol {x} + \boldsymbol {\Delta} \boldsymbol {x}) ^ {\top} \boldsymbol {\beta} - y |) = \ell (| y - x ^ {\top} \boldsymbol {\beta} | + \delta \| \boldsymbol {\beta} \| _ {*}).
$$
For classification, let  $y \in \{-1, 1\}$  and  $\mathcal{L}(\pmb{x}^\top \pmb{\beta}) = \ell(y(\pmb{x}^\top \pmb{\beta}))$  where  $\ell$  is a non-increasing function. If  $\ell$  is lower semicontinuous and convex then so will  $\mathcal{L}$  and the result of the theorem holds. Then  $s_* = -y$  and we obtain:
$$
\max  _ {\| \boldsymbol {\Delta} \boldsymbol {x} \| \leq \delta} \ell (y ((\boldsymbol {x} + \boldsymbol {\Delta} \boldsymbol {x}) ^ {\top} \boldsymbol {\beta})) = \ell (y (\boldsymbol {x} ^ {\top} \boldsymbol {\beta}) - \delta \| \boldsymbol {\beta} \| _ {*}).
$$
The above result can also be applied to unsupervised learning. In the Appendix, we illustrate how Theorem 4 can be used for dimensionality reduction. We consider the problem of finding  $P$  that minimizes the reconstruction error  $\| \pmb{x} + \pmb{P}\pmb{P}^{\top}\pmb{x}\|_{2}^{2}$  (that yields PCA algorithm) and derive an adversarial version of it.
Remark 6. Over this paper we consider  $\pmb{x}, \pmb{\beta} \in \mathbb{R}^p$ , but Theorem 4 holds generally for  $\pmb{x} \in \mathcal{X}$  a vector in a Banach space endowed with norm  $\| \cdot \|$  and  $\beta \in \mathcal{X}^*$  a continuous linear map  $\beta : \mathcal{X} \to \mathbb{R}$ . Just define  $\pmb{x}^\top \pmb{\beta} \coloneqq \pmb{\beta}(\pmb{x})$  and take  $\| \cdot \|_*$  to be the norm of the dual space  $\mathcal{X}^*$ .
We study adversarial training in linear regression. Adversarial training allows us to depart from the traditional regularization setting where we can decompose the loss and the regularization terms, minimizing  $\frac{1}{n}\sum_{i=0}^{n}\ell(\boldsymbol{x}_i^\top\boldsymbol{\beta},y_i) + \Omega(\boldsymbol{\beta})$  for some penalization function  $\Omega$ . We show how it provides new insights into the minimum-norm interpolator, by proving a new equivalence between the two methods (Section 4). While adversarially-trained linear regression arrives at similar solutions to parameter shrinking methods in some scenarios (Section 5), it also has some advantages, for instance, the adversarial radius might be set without knowing the noise variance (Section 6). Unlike, squared-root Lasso it achieves this while still minimizing a sum of squared terms. We believe a natural next step is to provide a tailored solver that can allow for efficient solutions for models with many features, rendering adversarially-trained linear regression useful, for instance, in modeling genetics data (see one minimal example of phenotype prediction from the genotype in Section 7).
Another interesting direction is generalizing our results to classification and nonlinear models. Indeed, adversarial training is very often used in the context of neural networks and it is natural to be interested in the behavior of this class of models. Our work allows for the analysis of simplified theoretical models commonly used to study neural networks. Random Fourier feature models can be analyzed using our theory and are they studied in the numerical examples. These models can be seen as a simplified neural network model (one-layer neural network where only the last layer weights are adjusted). In Appendix A.4, we provide adversarial training after linear projections, and could, for instance, be used to analyze deep linear networks, as the ones studied in [45], [46]. Finally, Section 8 provides a result for infinite spaces, and one could attempt to use it to analyze kernel methods and infinitely-wide neural networks [47], [48]. Overall we believe that our results provide an interesting set of tools also for the analysis of nonlinear models, and could provide insight into the empirically observed interplay (see [49]) between robustness and regularization in deep neural networks.
The authors would like to thank Carl Nettelblad for very fruitful discussions throughout the work with this research. We also thank Daniel Gedon and Dominik Baumann for their feedback on the first version of the manuscript.
TBS and DZ are financially supported by the Swedish Research Council, with the projects Deep probabilistic regression – new models and learning algorithms (contract number: 2021-04301) and Counterfactual Prediction Methods for Heterogeneous Populations (contract number:2018-05040) and by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by Knut and Alice Wallenberg Foundation. TBS, AHR and DZ by the Kjell och Märta Beijer Foundation. FB by the Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA0001 (PRAIRIE 3IA Institute); and by the European Research Council (grant SEQUOIA 724063).
The research was partially conducted during AHR's research visit to INRIA. The visit was financially supported by the Institute François de Suède through the SFVE-A mobility program; and, by the European Network of AI Excellence Centres through ELISE mobility program.