SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To guarantee the boundedness of the considered validation loss function classes, we will have the following assumptions for the data and regularization parameters. The first assumption is that all features and target values in the training and validation examples are bounded. The second assumption is that we only consider regularization coefficient values  $\lambda$  within an interval  $[\lambda_{\min}, \lambda_{\max}]$ . In practice, those assumptions are naturally satisfied by data normalization.
Assumption 1 (Bounded covariate and label). We assume that all the feature vectors and target values in training and validation set is upper-bounded by absolute constants  $R_{1}$  and  $R_{2}$ , i.e.  $\max \{\| X\|_{\infty}, \| X_{val}\|_{\infty}\} \leq R_1$ , and  $\max \{\| y\|_{\infty}, \| y_{val}\|_{\infty}\} \leq R_2$ .
Assumption 2 (Bounded Coefficient). We assume that  $\lambda \in [\lambda_{\min}, \lambda_{\max}]^2$  with  $\lambda_{\min} > 0$ .
Under Assumptions 2, 1, Theorem 3.3 immediately implies the following generalization guarantee for Elastic Net hyperparameter tuning.
Theorem 3.4. Let  $\mathcal{D}$  be an arbitrary distribution over the problem instance space  $\Pi_{m,p}$ . Under Assumptions 1, 2, the loss functions in  $\mathcal{H}_{EN}$  have range bounded by some constant  $H$  (Lemma C.1). Then there exists an algorithm s.t. for any  $\epsilon, \delta > 0$ , given  $N = O\left(\frac{H^2}{\epsilon^2} (p + \log \left(\frac{1}{\delta}\right))\right)$  sample problem instances drawn from  $\mathcal{D}$ , the algorithm outputs a regularization parameter  $\hat{\lambda}$  such that with probability at least  $1 - \delta$ ,  $\mathbb{E}_{P\sim \mathcal{D}}h_{EN}(\hat{\lambda}, P) < \min_{\lambda} \mathbb{E}_{P\sim \mathcal{D}}h_{EN}(\lambda, P) + \epsilon$ .
Proof. Denote  $\lambda^{*} = \operatorname{argmin}_{\lambda}\mathbb{E}_{P\sim \mathcal{D}}h_{\mathrm{EN}}(\lambda ,P)$ . From Theorems 3.3 and A.2, given  $n = O(\frac{H^2}{\epsilon^2} (p + \log (\frac{1}{\delta})))$  problem instances  $P^{(i)}$  for  $i\in [N]$  drawn from  $\mathcal{D}$ , w.p.  $1 - \delta$ , we have  $\mathbb{E}_{P\sim \mathcal{D}}h_{\mathrm{EN}}(\hat{\lambda},P) < \frac{1}{N}\sum_{i = 1}^{N}h_{\mathrm{EN}}(\hat{\lambda},P^{(i)}) + \frac{\epsilon}{2} < \frac{1}{N}\sum_{i = 1}^{N}h_{\mathrm{EN}}(\lambda^{*},P^{(i)}) + \frac{\epsilon}{2} < \mathbb{E}_{P\sim \mathcal{D}}h_{\mathrm{EN}}(\lambda^{*},P) + \epsilon$ .
3.3 Lower bound
Remarkably, we are able to establish a matching lower bound on the pseudo-dimension of the Elastic Net loss function class, parameterized by the regularization parameters. Note that every Elastic Net problem can be converted to an equivalent LASSO problem [1]. In fact, we show something stronger, that the pseudo-dimension of even the LASSO regression loss function class (parameterized by regression coefficient  $\lambda_{1}$ ) is  $\Omega(p)$ , from which the above observation follows (by taking  $\lambda_{2} = 0$  in our construction). Our proof of the lower bound adapts the "adversarial strategy" of [35] which is used to design a worst-case LASSO regularization path. While [35] construct a single dataset to bound the number of segments in the piecewise-linear LASSO solution path, we create a collection of problem instances for which all above-below sign patterns may be achieved by selecting regularization parameters from different segments of the solution path.
Theorem 3.5. Let  $\mathcal{H}_{LASSO}$  be a set of functions  $\{h_{LASSO}(\lambda, \cdot): \Pi_{m,p} \to \mathbb{R}_{\geq 0} \mid \lambda \in \mathbb{R}^{+}\}$  that map a regression problem instance  $P \in \Pi_{m,p}$  to the validation loss  $h_{LASSO}(\lambda, P)$  of LASSO trained with regularization parameter  $\lambda$ . Then  $\mathrm{Pdim}(\mathcal{H}_{LASSO})$  is  $\Omega(p)$ .
Proof Sketch. Consider  $N = p$  problem instances for LASSO regression given by  $P^{(i)} = (X^{(i)}, y^{(i)}, X_{\mathrm{val}}^{(i)}, y_{\mathrm{val}}^{(i)})$ , where the training set  $(X^{(i)}, y^{(i)}) = (X^*, y^*)$  is fixed and set using the "adversarial strategy" of [35], Proposition 2. The validation sets are given by single examples  $(X_{\mathrm{val}}^{(i)}, y_{\mathrm{val}}^{(i)}) = (\mathbf{e}_i, 0)$ , where  $\mathbf{e}_i$  are standard basis vectors in  $\mathbb{R}^p$ . We will now proceed to provide the witnesses  $r_1, \ldots, r_N$  and  $\lambda$  values to exhibit a pseudo-shattering of these problem instances.
Corresponding to subset  $T \subseteq [p]$  of problem instances, we will provide a value of  $\lambda_T$  such that, we have  $\ell_{\mathrm{LASSO}}(\lambda_T, P^{(i)}) > r_i$  iff  $i \in T$ , for each  $i \in [p]$  and each  $T \subseteq [p]$ . We set all witnesses  $r_i = 0$  for all  $i \in [p]$ . As a consequence of Theorem 1 in [35], the regularization path of  $(X^*, y^*)$  consists of a linear segment corresponding all  $2^p$  unsigned sparsity patterns in  $\{0, 1\}^p$  (we will not need all the segments in the construction, but note that it is guaranteed to contain all distinct unsigned sparsity patterns) and we select  $\lambda_T$  as any interior point corresponding to a linear segment with sparsity pattern  $\{(c_1, \ldots, c_p) \mid c_i = 0$  iff  $i \in T\}$ , i.e. elements in  $T$  are exactly the ones with sparsity pattern 0. Therefore,  $|\beta_T^* \cdot \mathbf{e}_i| = 0$  iff  $i \in T$ , where  $\beta_T^*$  is the LASSO regression fit for regularization parameter  $\lambda_T$ . This implies the desired shattering condition w.r.t. witnesses  $r_1 = 0, \ldots, r_N = 0$ . Therefore,  $\operatorname{Pdim}(\mathcal{H}_{\mathrm{LASSO}}) \geq p$ . See Appendix B.2 for a full proof.
3.4 Hyperparameter tuning in Regularized Kernel Regression
The Kernel Least Squares Regression ([4]) is a natural generalization of the linear regression problem, which uses a kernel to handle non-linearity. In this problem, each sample has  $p_1$  feature, corresponding to a real-valued target. Formally, each problem instance  $P$  drawn from  $\Pi$  can be described as
$$
P = \left(X, y, X _ {\text {v a l}}, y _ {\text {v a l}}\right) \in \mathbb {R} ^ {m \times p _ {1}} \times \mathbb {R} ^ {m} \times \mathbb {R} ^ {m ^ {\prime} \times p _ {1}} \times \mathbb {R} ^ {m ^ {\prime}}.
$$
A common issue in practice is that the relation between  $y$  and  $X$  is non-linear in the original space. To overcome this issue, we consider the mapping  $\phi : \mathbb{R}^{p_1} \to \mathbb{R}^{p_2}$  which maps the original input space to a new feature space in which we hopefully can perform linear regression. Define  $\phi(X) = (\phi(x_1), \ldots, \phi(x_m))_{m \times p_2}$ , our goal is to find a vector  $\theta \in \mathbb{R}^{p_2}$  so that the squared loss  $\frac{1}{2} \|y - \phi(X)\theta\|_2^2 + R(\|\theta\|)$  is minimized, where the regularization term  $R(\|\theta\|)$  is any strictly monotonically increasing function of the Hilbert space norm. It is well-known from the literature (e.g. [36]) that under the Representer Theorem's conditions, the optimal value  $\theta^*$  can be linearly represented by row vectors of  $\phi(X)$ , i.e.,  $\theta^* = \phi(X)\beta = \sum_{i=1}^{m} \phi(x_i)\beta_i$ , where  $\beta = (\beta_1, \ldots, \beta_m) \in \mathbb{R}^m$ . This directly includes the  $\ell_2$  regularizer but does not include  $\ell_1$  regularization. To overcome this issue, Roth ([3]) proposed an alternative approach to regularized kernel regression, which directly restricts the representation of coefficient  $\theta$  via a linear combination of  $\phi(x_i)$ , for  $i \in [m]$ . The regularized kernel regression hence can be formulated as
$$
\hat {\beta} _ {l, \lambda} ^ {(X, y)} = \operatorname * {a r g m i n} _ {\beta \in \mathbb {R} ^ {m}} \frac {1}{2} \| y - K \beta \| _ {2} ^ {2} + \lambda_ {1} \| \beta \| _ {1} + \lambda_ {2} \| \beta \| _ {2} ^ {2},
$$
where  $k(x,x^{\prime}) = \langle \phi (x),\phi (x^{\prime})\rangle$  is the kernel mapping, and the Gram matrix  $K$  satisfies  $[K]_{i,j} = k(x_i,x_j)$  for all  $i,j\in [m]$ .
Clearly, the problem above is a linear regression problem. Formally, denote  $h_{\mathrm{KER}}(\lambda, P) = \frac{1}{2} \| y - K\hat{\beta}_{(X,y)}(\lambda)\|_2$  and let  $\mathcal{H}_{\mathrm{KER}} = \{h_{\mathrm{KER}}(\lambda, \cdot): \Pi_{m,p} \to \mathbb{R}_{\geq 0} \mid \lambda \in \mathbb{R}_+^2\}$ . The following result is a direct corollary of Theorem 3.3, which gives an upper bound for the pseudo-dimension of  $\mathcal{H}_{\mathrm{KER}}$ .
Corollary 3.6.  $\operatorname{Pdim}(\mathcal{H}_{KER}) = O(m)$ .
Note that  $m$  here denotes the training set size for a single problem instance, and Corollary 3.6 implies a guarantee on the number of problem instances needed for learning a good regularization parameter for kernel regression via classic results [33, 34]. Our results do not make any assumptions on the  $m$  samples within a problem instance/dataset; if these samples within problem instances are further assumed to be i.i.d. draws from some data distribution (distinct from problem distribution  $\mathcal{D}$ ), then well-known results imply that  $m = O(k\log p)$  samples are sufficient to learn the optimal LASSO coefficient [37, 38], where  $k$  denotes the number of non-zero coefficients in the optimal regression fit.
4 Hyperparameter tuning for Regularized Logistic Regression
Logistic regression is more naturally suited to applications modeling probability of an event, like medical risk for a patient [39], predicting behavior in markets [40], failure probability of an engineering system [41] and many more applications [42]. It is a fundamental statistical technique for classification, and regularization is again crucial for avoiding overfitting and estimating variable importance. In this section, we will present learning guarantees for tuning the Regularized Logistic Regression (RLR) regularization coefficients across instances. Given a problem instance  $P$  drawn from a problem distribution  $\mathcal{D}$  over  $\Pi_{m,p}$ , let  $h_{\mathrm{RLR}}(\lambda, P) = l_{\mathrm{RLR}}(\hat{\beta}_{(X,y)}(\lambda), (X_{\mathrm{val}}, y_{\mathrm{val}}))$  be the RLR validation loss function class (defined in Section 2), and let  $\mathcal{H}_{\mathrm{RLR}} = \{h_{\mathrm{RLR}}(\lambda, \cdot) : \Pi_{m,p} \to \mathbb{R}_{\geq 0} \mid \lambda \in \mathbb{R}_{>0}\}$  be the RLR validation loss function class, our goal is to provide a learning guarantee for  $\mathcal{H}_{\mathrm{RLR}}$ . Besides, we also study the commonly used 0-1 validation loss function class  $\mathcal{H}_{\mathrm{RLR}}^{0-1} = \{h_{\mathrm{RLR}}^{0-1}(\lambda, \cdot) : \Pi_{m,p} \to \mathbb{R}_{\geq 0} \mid \lambda \in \mathbb{R}_{>0}\}$ , where  $h_{\mathrm{RLR}}^{0-1}(\lambda, P) = \frac{1}{m'} \sum_{i=1}^{m'} \mathbb{1}\{y_i x_i^\top \hat{\beta}_{X,y}(\lambda) \leq 0\}$ , which we will cover in Section 4.3. Similarly, to guarantee the boundedness of  $\mathcal{H}_{\mathrm{RLR}}$ , we also assume that Assumptions 1 and 2 also hold in this setting.
4.1 Approximate solutions of Regularized Logistic Regression
The main challenge in analyzing the regularized logistic regression, unlike the regularized logistic regression problem, is that the solution  $\hat{\beta}_{(X,y)}(\lambda)$  corresponding to a problem instance  $P$  and particular value  $\lambda > 0$  does not have a closed form depending on  $\lambda$ . We then propose an alternative approach to this end, which is examining via the approximation  $\beta_{(X,y)}^{(\epsilon)}(\lambda)$  of the solution  $\hat{\beta}_{(X,y)}(\lambda)$ .
Algorithm 1 Approximate incremental quadratic algorithm for RLR with  $\ell_1$  penalty, [2]
Set  $\beta_0^{(\epsilon)} = \hat{\beta}_{(X,y)}(\lambda_{\mathrm{min}}), t = 0$ , small constant  $\delta \in \mathbb{R}_{>0}$ , and  $\mathcal{A} = \{j \mid [\hat{\beta}_{(X,y)}(\lambda_{\mathrm{min}})]_j \neq 0\}$ .
while  $\lambda_t < \lambda_{\max}$  do
$$
\left\{ \begin{array}{l} \lambda_ {t + 1} = \lambda_ {t} + \epsilon \\ \left(\beta_ {t + 1} ^ {(\epsilon)}\right) _ {\mathcal {A}} = \left(\beta_ {t} ^ {(\epsilon)}\right) _ {\mathcal {A}} - \left[ \nabla^ {2} l \left(\beta_ {t} ^ {(\epsilon)}, (X, y)\right) _ {\mathcal {A}} \right] ^ {- 1} \cdot \left[ \nabla l \left(\beta_ {t} ^ {(\epsilon)}, (X, y)\right) _ {\mathcal {A}} + \lambda_ {t + 1} \operatorname {s g n} \left(\beta_ {t} ^ {(\epsilon)}\right) _ {\mathcal {A}} \right] \\ \left(\beta_ {t + 1} ^ {(\epsilon)}\right) _ {- \mathcal {A}} = \vec {0} \\ \mathcal {A} = \mathcal {A} \cup \{j \neq \mathcal {A} | \nabla l (\beta_ {t + 1} ^ {(\epsilon)}, (X, y)) > \lambda_ {t + 1} \} \\ \mathcal {A} = \mathcal {A} \setminus \{j \in \mathcal {A} | \left| \beta_ {t + 1, j} ^ {(\epsilon)} \right| <   \delta \} \\ t = t + 1 \end{array} \right.
$$
The approximation Algorithm 1 (Algorithm 2) for the solution  $\hat{\beta} (\lambda)$  of RLR under  $\ell_{1}$  (or  $\ell_2$ ) constraint were first proposed by Rosset [26, 2]. Given a problem instance  $P$ , and a sufficiently small step-size  $\epsilon >0$ , using Algorithms 1, 2 yields an approximation  $\beta_{(X,y)}^{(\epsilon)}$  of  $\hat{\beta}_{(X,y)}$  that are piece-wise linear functions of  $\lambda$  in total  $(\lambda_{\mathrm{max}} - \lambda_{\mathrm{min}}) / \epsilon$  [26]. Moreover, it is also guaranteed that the error between  $\beta_{(X,y)}^{(\epsilon)}$  and  $\hat{\beta}_{(X,y)}$  is uniformly upper bounded for all  $\lambda \in [\lambda_{\mathrm{min}},\lambda_{\mathrm{max}}]$ .
Algorithm 2 Approximate incremental quadratic algorithm for RLR with  $\ell_2$  penalty, [2]
Set  $\beta_0^{(\epsilon)} = \hat{\beta}_{(X,y)}(\lambda_{\min}), t = 0.$
while  $\lambda_t < \lambda_{\max}$  do
$$
\left\{ \begin{array}{l} \lambda_ {t + 1} = \lambda_ {t} + \epsilon \\ \beta^ {(\epsilon)} (\lambda) = \beta_ {t} ^ {(\epsilon)} - \left[ \nabla^ {2} l \left(\beta_ {t} ^ {(\epsilon)}, (X, y)\right) + 2 \lambda_ {t + 1} I \right] ^ {- 1} \cdot \left[ \nabla l \left(\beta_ {t} ^ {(\epsilon)}, (X, y)\right) + 2 \lambda_ {t + 1} \beta_ {t} ^ {(\epsilon)} \right] \\ t = t + 1 \end{array} \right.
$$
Theorem 4.1 (Theorem 1, [2]). Given a problem instance  $P$ , for small enough  $\epsilon$ , there is a uniform bound  $O(\epsilon^2)$  on the error  $\| \hat{\beta}_{(X,y)}(\lambda) - \beta_{(X,y)}^{(\epsilon)}(\lambda)\|_2$  for any  $\lambda \in [\lambda_{\min}, \lambda_{\max}]$ .
Denote  $h_{\mathrm{RLR}}^{(\epsilon)}(\lambda, P) = l_{\mathrm{RLR}}(\beta^{\epsilon}(\lambda), (X_{\mathrm{val}}, y_{\mathrm{val}}))$  the approximation function of the validation loss  $h_{\mathrm{RLR}}(\lambda, P)$ . Using Theorem 4.1 and note that the loss  $f(z) = \log(1 + e^{-z})$  is 1-Lipschitz, we can show that the difference between  $h_{\mathrm{RLR}}^{(\epsilon)}(\lambda, P)$  and  $h_{\mathrm{RLR}}(\lambda, P)$  is uniformly upper-bounded.
Lemma 4.2. The approximation error of the validation loss function is uniformly upper-bounded  $|h_{RLR}^{(\epsilon)}(\lambda, P) - h_{RLR}(\lambda, P)| = O(\epsilon^2)$ , for all  $\lambda \in [\lambda_{\min}, \lambda_{\max}]$ .
We now present one of our main results, which is the pseudo-dimension bound of the approximate validation loss function class  $\mathcal{H}_{\mathrm{RLR}}^{(\epsilon)}$ .
Theorem 4.3. Consider the RLR under  $\ell_1$  (or  $\ell_2$ ) constraint with parameter  $\lambda \in [\lambda_{\min}, \lambda_{\max}]$  that take a problem instance  $P$  drawn from an unknown problem distribution  $\mathcal{D}$  over  $\Pi_{m,p}$ . Under Assumptions 1 and 2,  $\mathcal{H}_{RLR}$  is bounded by some constant  $H$  (Lemma C.2). Suppose that we use Algorithm 1 (or Algorithm 2) to approximate the solution  $\hat{\beta}_{(X,y)}(\lambda)$  by  $\beta_{(X,y)}^{(\epsilon)}(\lambda)$  with a uniform error  $O(\epsilon^2)$  for any  $\lambda \in [\lambda_{\min}, \lambda_{\max}]$ , where  $\epsilon$  is the approximation step-size. Consider the approximation validation loss function class  $\mathcal{H}_{RLR}^{(\epsilon)} = \{h_{RLR}^{(\epsilon)}(\lambda, \cdot): \Pi_{m,p} \to \mathbb{R}_{\geq 0} \mid \lambda \in [\lambda_{\min}, \lambda_{\max}]\}$ , where
$$
h _ {R L R} ^ {(\epsilon)} (\lambda , P) = \frac {1}{m ^ {\prime}} \sum_ {i = 1} ^ {m ^ {\prime}} \log (1 + \exp (- y _ {i} x _ {i} ^ {\top} \beta_ {(X, y)} ^ {(\epsilon)} (\lambda)))
$$
is the approximate validation loss. Then we have  $\mathrm{Pdim}(\mathcal{H}_{RLR}^{(\epsilon)}) = O(m^2 +\log (1 / \epsilon))$ . Further, we assume that  $\epsilon = O(\sqrt{H})$  where  $H$  is the upperbound of  $\mathcal{H}_{RLR}$  under Assumptions 1 and 2. Given any set  $\mathcal{S}$  of  $T$  problem instances drawn from a problem distribution  $\mathcal{D}$  over  $\Pi_{m,p}$ , the empirical Rademacher complexity  $\hat{\mathcal{R}} (\mathcal{H}_{RLR}^{(\epsilon)},\mathcal{S}) = O(H\sqrt{(m^2 + \log(1 / \epsilon)) / T})$ .
The key observation here is that the approximation solution  $\hat{\beta}_{(X,y)}^{(\epsilon)}$  is piece-wise linear over  $(\lambda_{\max} - \lambda_{\min}) / \epsilon$  pieces, leading to the fact that the approximate validation loss function  $h_{\mathrm{RLR}}^{(\epsilon)}(\lambda,\cdot)$  is a "special function" (Pfaffian function [43]) in each piece, which is a combination of exponentiation of linear functions of  $\lambda$ . The detailed proof of Theorem 4.3 can be found on the Appendix D.3.
4.2 Learning guarantees for Regularized Logistic Regression hyperparameter tuning
Our goal now is to use the upper bound for empirical Rademacher complexity of the validation loss function class  $\mathcal{H}_{\mathrm{RLR}}$ . We use techniques for approximate data-driven algorithm design due to [29], combining the uniform error upper bound between validation loss function  $h_{\mathrm{RLR}}(\lambda, P)$  and its approximation  $h_{\mathrm{RLR}}^{(\epsilon)}(\lambda, P)$  (Lemma 4.2) and empirical Rademacher complexity of approximation validation loss function class  $\mathcal{H}_{\mathrm{RLR}}^{(\epsilon)}$  (Theorem 4.3), to obtain a bound on the empirical Rademacher complexity of  $\mathcal{H}_{\mathrm{RLR}}$ . This allows us to give a learning guarantee for the regularization parameters  $\lambda$ , which is formalized by the following theorem.
Theorem 4.4. Consider the RLR under  $\ell_1$  (or  $\ell_2$ ) constraint. Under Assumptions 1, 2,  $\mathcal{H}_{RLR}$  is bounded by some constant  $H$  (Lemma C.2). Consider the class function  $\mathcal{H}_{RLR} = \{h_{RLR}(\lambda,\cdot):\Pi_{m,p}\to \mathbb{R}_{\geq 0}\mid \lambda \in [\lambda_{\min},\lambda_{\max}]\}$  where  $h_{RLR}(\lambda,P)$  is the validation loss corresponding to problem
instance  $P$  and the  $\ell_1$  ( $\ell_2$ ) parameter  $\lambda$ . Given any set  $S$  of  $T$  problem instances drawn from a problem distribution  $\mathcal{D}$  over  $\Pi_{m,p}$ , for any  $h_{RLR}(\lambda, \cdot) \in \mathcal{H}_{RLR}$ , w.p.  $1 - \delta$  for any  $\delta \in (0,1)$ , we have
$$
\left| \frac {1}{T} \sum_ {i = 1} ^ {T} h _ {R L R} (\lambda , P ^ {(i)}) - \mathbb {E} _ {P \sim \mathcal {D}} [ h _ {R L R} (\lambda , P) ] \right| \leq O \left(H \sqrt {\frac {m ^ {2} + \log (1 / \epsilon)}{T}} + \epsilon^ {2} + \sqrt {\frac {1}{T} \log \frac {1}{\delta}}\right),
$$
for some sufficiently small  $\epsilon$ .
The proof detail of Theorem 4.4 is included in the Appendix D.3. The above generalization guarantee gives a bound on the average error on RLR validation loss over the problem distribution, for the parameter  $\lambda$  learned from  $T$  problem instances. In commonly used approaches, the validation set size is small or a constant, and our result can be interpreted as the upper bound on the generalization error in terms of the number of problem instances  $T$  and the step length  $\epsilon$ . We only consider RLR under  $\ell_1$  (or  $\ell_2$ ) constraints, which are commonly studied in the literature, our analysis could be easily extended to RLR under  $\ell_q$  constraint for any  $q \geq 1$ .
4.3 An extension to 0-1 loss
Since logistic regression is often used for binary classification tasks, it is interesting to consider the 0-1 loss as the validation loss function. It has been shown that  $\mathbb{1}\{z\leq 0\} \leq 4\log (1 + e^{-z})$  for any  $z$  [44]. This inequality, combined with Theorem 4.4, directly provides a learning guarantee for the 0-1 validation loss function.
Theorem 4.5. Let  $\tau > 2\epsilon^2$  and  $\delta \in (0,1)$ , where  $\epsilon$  is the approximation step-size. Then for any  $n \geq s(\tau/2, \delta) = \Omega\left(\frac{H^2(m^2 + \log\frac{1}{\epsilon}) + \log\frac{1}{\delta}}{(\tau/2 - \epsilon^2)^2}\right)$ , if we have  $n$  problem instances  $\{P^{(i)}, \ldots, P^{(n)}\}$  drawn i.i.d. from some problem distribution  $\mathcal{D}$  over  $\Pi_{m,p}$  to learn the regularization parameter  $\lambda^{ERM}$  for RLR via  $ERM$ , then
$$
\mathbb {E} _ {P \sim \mathcal {D}} (h _ {R L R} ^ {0 - 1} (\lambda^ {E R M}, P))) \leq 4 \min _ {\lambda \in [ \lambda_ {\min }, \lambda_ {\max } ]} \mathbb {E} _ {P \sim \mathcal {D}} (h _ {R L R} (\lambda , P)) + 4 \tau .
$$
The detailed proof of Theorem 4.5 can be found on Appendix D.4. It is worth noting that we are providing learning guarantee for 0-1 validation loss function class  $\mathcal{H}_{\mathrm{RLR}}^{0 - 1}$  indirectly via the validation loss function class  $\mathcal{H}_{\mathrm{RLR}}$  with cross-entropy objective function, which is arguably not optimal. The question of how to provide a true PAC-learnable guarantee for  $\mathcal{H}_{\mathrm{RLR}}^{0 - 1}$  remains an interesting challenge.
In this work, we present novel learning guarantees for tuning regularization parameters for both the Elastic Net and Regularized Logistic Regression models, across problem instances. For the Elastic Net, we propose fine-grained structural results that pertain to the tuning of regularization parameters. We use them to give an improved upper bound on the pseudo-dimension of the relevant validation loss function class of and we prove that our new bound is tight.
For the problem of tuning regularization parameters in regularized logistic regression, we propose an alternative approach that involves analyzing the approximation of the original validation loss function class. This approximation, characterized by a piece-wise linear representation, provides a useful analytical tool in the absence of an exact dependence of the logistic loss on the regularization parameters. Additionally, we employ an upper bound on the approximation error between the original and approximated functions, to obtain a learning guarantee for the original validation loss function class. Remarkably, our proposed approach is not restricted solely to regularized logistic regression but can be extended to a wide range of other problems, demonstrating its generality and applicability.
It is worth noting that this work only focuses on the sample complexity aspect of the hyperparameter tuning in the Elastic Net and Regularized Logistic Regression. The question of computational complexity in this setting is an interesting future direction. Other interesting questions include designing hyperparameter tuning techniques for this setting that are robust to adversarial attacks, and hyperparameter tuning for Regularized Logistic Regression with both  $\ell_1$  and  $\ell_2$  constraints.
We thank Yingyu Liang for useful discussions during early stages of this work and Mikhail Khodak for helpful feedback. This work was supported in part by NSF grants CCF-1910321 and SES-1919453, the Defense Advanced Research Projects Agency under cooperative agreement HR00112020003, and a Simons Investigator Award.