Title: New Bounds for Hyperparameter Tuning of Regression Problems Across Instances
Venue: NeurIPS 2023
Accepted: yes
Keywords: Elastic Net

logistic regression

data-driven algorithm design

learning theory

regularization
Abstract: The task of tuning regularization coefficients in regularized regression models with provable guarantees across problem instances still poses a significant challenge in the literature. This paper investigates the sample complexity of tuning regularization parameters in linear and logistic regressions under $\ell_1$ and $\ell_2$-constraints in the data-driven setting. For the linear regression problem, by more carefully exploiting the structure of the dual function class, we provide a new upper bound for the pseudo-dimension of the validation loss function class, which significantly improves the best-known results on the problem. Remarkably, we also instantiate the first matching lower bound, proving our results are tight. For tuning the regularization parameters of logistic regression, we introduce a new approach to studying the learning guarantee via an approximation of the validation loss function class. We examine the pseudo-dimension of the approximation class and construct a uniform error bound between the validation loss function class and its approximation, which allows us to instantiate the first learning guarantee for the problem of tuning logistic regression regularization coefficients.

New Bounds for Hyperparameter Tuning of Regression Problems Across Instances

Abstract

The task of tuning regularization coefficients in regularized regression models with provable guarantees across problem instances still poses a significant challenge in the literature. This paper investigates the sample complexity of tuning regularization parameters in linear and logistic regressions under ℓ 1 and ℓ 2 -constraints in the data-driven setting. For the linear regression problem, by more carefully exploiting the structure of the dual function class, we provide a new upper bound for the pseudo-dimension of the validation loss function class, which significantly improves the best-known results on the problem. Remarkably, we also instantiate the first matching lower bound, proving our results are tight. For tuning the regularization parameters of logistic regression, we introduce a new approach to studying the learning guarantee via an approximation of the validation loss function class. We examine the pseudo-dimension of the approximation class and construct a uniform error bound between the validation loss function class and its approximation, which allows us to instantiate the first learning guarantee for the problem of tuning logistic regression regularization coefficients.

1 Introduction

Regularized linear models, including the Elastic Net [1], and Regularized Logistic Regression [2, 3, 4], as well as their variants [5, 6, 7], have found widespread use in diverse fields and numerous application domains. Thanks to their simplicity and interpretability, those methods are popular choices for controlling model complexity, improving robustness, and preventing overfitting by selecting relevant features [4, 8, 9]. Moreover, regularized linear models can be adapted to the non-linear regime using kernel methods [10, 11], significantly expanding their applicability to a wide range of problems. In typical applications, one needs to solve not only a single regression problem instance, but several related problems from the same domain. Can we learn how to regularize with good generalization across the related problem instances?

Suppose we have a regression dataset ( X,y ) ∈ R m × p × Y m , where X is a design matrix with m samples and p features, and y is a target vector. Regularized linear models aim to compute an estimator ˆ β ( X,y ) ( λ ) by solving the optimization problem

where ( λ 1 , λ 2 ) ∈ R 2 ≥ 0 are the regularization coefficients. For instance, if λ ∈ R 2 > 0 , y ∈ R m , and l ( β, ( X,y )) = 1 2 ∥ y -Xβ ∥ 2 2 (squared-loss function), we get the well-known Elastic Net [1]. On the other hand, if λ ∈ { ( λ 1 , 0) , (0 , λ 2 ) } for λ 1 , λ 2 > 0 , y ∈ {± 1 } m , and l ( β, ( X,y )) = 1 m ∑ m i =1 log(1 + exp( -y i x ⊤ i β )) , we obtain regularized logistic regression.

In regularized linear models, the parameters λ play a crucial role in controlling the sparsity ( ℓ 1 ) and shrinkage ( ℓ 2 ) constraints, and are essential in ensuring better generalization and robustness [9, 4, 12]. A popular approach in practice is cross-validation, which involves choosing a finite grid of values of λ and iteratively solving the regression problem for multiple values of λ and evaluating on held-out validation sets to determine the optimal parameter. Principled techniques with theoretical guarantees suffer from various limitations, for example require strong assumptions about the original problem [13], or aim to search the optimal parameter over a discrete subset instead of the whole continuous domain. Moreover, repeatedly solving the regression problem is particularly inefficient if we have multiple problem instances from the same problem domain.

In this work, we investigate an alternative setting for tuning regularization parameters, namely data-driven algorithm design , following the previous line of work by Balcan et al. [14]. Unlike the traditional approach, which involves considering a single dataset ( X,y ) , in the data-driven approach, we analyze a collection of datasets or problem instances ( X ( i ) , y ( i ) , X ( i ) val , y ( i ) val ) drawn from an underlying problem distribution D . Our objective is to determine the optimal regularization parameters λ so that when using the training set ( X ( i ) , y ( i ) ) and λ to select a model in Optimization problem 1, the selected model minimizes loss on the validation set ( X ( i ) val , y ( i ) val ) . As remarked by Balcan et al. [14], data-driven algorithm design can handle more diverse data generation scenarios in practice, including cross validation and multitask-learning [15, 16]. We emphasize that the data-driven setting differs significantly from the standard single dataset setting.

In this paper, we consider the problem of tuning regularization parameters in regularized logistic regression and the Elastic Net across multiple problem instances. Our contributions are:

We present an improved upper bound (Theorem 3.3) on the pseudo-dimension for tuning the Elastic Net regularization parameters across problem instances by establishing a novel structural result for the validation loss function class (Theorem 3.2). We provide a crucial refinement to the piecewise structure of this function class established by Balcan et al. [14], by providing a bound on the number of distinct functional behaviors across the pieces. This enables us to describe the computation of the validation loss function as a GJ algorithm [17], which yields an upper-bound of O ( p ) on the pseudo-dimension, a significant improvement of the prior best bound of O ( p 2 ) by Balcan et al. [14], and a corresponding improvement in the sample complexity (Theorem 3.4).

Furthermore, we establish the tightness of our result by providing the first asymptotically matching lower bound of Ω( p ) on the pseudo-dimension (Theorem 3.5). It is worth noting that our results have direct implications for other specialized cases, such as LASSO and Ridge Regression.

We further extend our results on the Elastic Net to regularized kernel linear regression problem (Corollary 3.6).

We propose a novel approach to analyze the problem of tuning regularization parameters in regularized logistic regression, which involves indirectly investigating an approximation of the validation loss function class. Using this approach, we instantiate the first learning guarantee for this problem in the data-driven setting (Theorem 4.4).

1.1 Related work

Model selection for regularized linear models. Extensive research has focused on the selection of optimal parameters for regularized linear models, including the Elastic Net and regularized logistic regression. This process usually entails choosing the appropriate regularization coefficients for a given dataset [18, 19]. Nevertheless, a substantial proportion of this research relies on heuristic approaches that lack theoretical guarantees [20, 21]. Others have concentrated on creating tuning objectives that go beyond validation error [22, 23], but with no clearly defined procedures for provably optimizing them. The conventional method for selecting a tuning regularization parameter is through grid-based selection, which aims to choose the parameter from a subset, known as a grid, within the parameter space. While this approach provides certain guarantees [24], it falls short in delivering an optimal solution across the entire continuous parameter space, particularly when using tuning objectives that exhibit numerous discontinuities. Additionally, the grid-based technique is highly sensitive to density, as selecting a grid that is either too dense or too coarse might result in inefficient search or highly inaccurate solutions. Other guarantees require strong assumptions on the data distribution, such as sub-Gaussian noise [25, 13]. Some studies focus on evaluating regularized linear models by

...

Fit

using

training data

Train problem instances

(drawn from

)

Hyper-parameter

tuning algorithm

Compute validation loss

for

on

Hyper-parameter tuning algorithm

Minimizing validation loss

constructing solution paths [26, 2, 27]. However, it is important to note that these approaches are primarily computational in nature and do not provide theoretical guarantees.

Data-driven algorithm design. Data-driven algorithms can adapt their internal structure or parameters to problem instances from unknown application-specific distributions. It is proved to be effective for a variety of combinatorial problems, such as clustering, integer programming, auction design, and graph-based semi-supervised learning [28, 29, 30, 31]. Balcan et al. [14] recently introduced a novel approach to tuning regularization parameters in regularized linear regression models, such as Elastic Net and its variants. They applied data-driven analysis to reveal the underlying discrete structure of the problem and leveraged a general result from [32] to obtain an upper bound on the pseudo-dimension of the problem. To provably tune the regularization parameters across problem instances, they proposed a simple ERM learner and provided sample complexity guarantee for such learner. However, the general techniques from [32] do not always lead to optimal bounds on the pseudodimension. Our paper is an example of a problem where these bounds (as derived in [14]) are sub-optimal, and more specialized techniques due to [31] result in the tighter bounds that we obtain. Also prior work does not establish any lower bound on the pseudodimension. Furthermore, it should be noted that their analysis heavily relies on the assumption of having a closed-form representation of the Elastic Net estimator [27]. This approach may not be applicable in analyzing other regularized linear models, such as regularized logistic regression, for which we propose an alternative approach.

2 Problem setting

In this section, we provide a formal definition of the problem of tuning regularization parameters in the Elastic Net and regularized logistic regression (RLR) across multiple problem instances, which follows the settings by Balcan et al. [14]. Given a problem instance P = ( X,y,X val , y val ) , where ( X,y ) ∈ R m × p × Y m represents the training dataset with m samples and p features, and ( X val , y val ) ∈ R m ′ × p ×Y m ′ denotes the validation split with m ′ samples, we consider the estimator ˆ β ( X,y ) ( λ ) defined as:

where l ( β, ( X,y )) represents the objective loss function, λ denotes the regularization coefficients, and R ( β ) = ( ∥ β ∥ 1 , ∥ β ∥ 2 2 ) represents the regularization vector function.

For instance, if λ ∈ R 2 > 0 , Y ≡ R , and l ( β, ( X,y )) = l EN ( β, ( X,y )) = 1 2 m ∥ y -Xβ ∥ 2 2 , we get the well-known Elastic Net. On the other hand, if λ ∈ { ( λ 1 , 0) , (0 , λ 2 ) } for λ 1 , λ 2 > 0 , y ∈ {± 1 } m , and l ( β, ( X,y )) = l RLR ( β, ( X,y )) = 1 m ∑ m i =1 log(1 + exp( -y i x ⊤ i β )) , we obtain RLR with ℓ 1 or ℓ 2 regularization. Note that for the Elastic Net hyperparameter tuning problem, we allows the regularization coefficients of both ℓ 1 , ℓ 2 are positive, while in the Regularized Logistic Regression

problem, we consider either ℓ 1 or ℓ 2 as the regularization term. We then use the validation set ( X val , y val ) to calculate the validation loss h ( λ, P ) = l ( ˆ β ( X,y ) ( λ ) , ( X val , y val )) corresponding to the problem instance P and learned regularization parameters λ .

In the data-driven setting , we receive a collection of n problem instances P ( i ) = ( X ( i ) , y ( i ) , X ( i ) val , y ( i ) val ) ∈ R m i ,p i ,m ′ i for i ∈ [ n ] , where R m i ,p i ,m ′ i := R m i × p i ×Y m i × R m ′ i × p i ×Y m ′ i . The problem space Π m,p is given by Π m,p = ∪ m 1 ≥ 0 ,m 2 ≤ m,p 1 ≤ p R m 1 ,p 1 ,m 2 , and we assume that problem instance P is drawn i.i.d from the problem distribution D over Π m,p . Remarkably, in this setting, problem instances can have varying training and validation sample sizes, as well as different sets of features. This general framework applies to practical scenarios where the feature sets differ among instances and allows one to learn regularization parameters that effectively work on average across multiple different but related problem instances. See Figure 1 for an illustration of the setting.

The goal here is to learn the value ˆ λ s.t. with high probability over the draw of n problem instances, the expected validation loss E P ∼D h ( ˆ λ, P ) is close to min λ E P ∼D [ h ( λ, P )] . This paper primarily focuses on providing learning guarantees in terms of sample complexity for the problem of tuning regularization parameters in the Elastic Net and regularized logistic regression (RLR). Specifically, we aim to address the question of how many problem instances are required to learn a value of λ that performs well across all problems P drawn from the problem distribution D . To achieve this, we analyze the pseudo-dimension (in the case of the Elastic Net) or the Rademacher Complexity (for RLR) of the validation loss function class H = { h ( λ, · ) | λ ∈ Λ } , where Λ represents the search space for λ .

3 Tight pseudo-dimension bounds for Elastic Net hyperparameter tuning

In this section, we will present our results on the pseudo-dimension upper and lower bounds for the regularized linear regression problem in the data-driven setting. Classic learning-theoretic results [33, 34] connect the pseudo-dimension of the validation loss function class (parameterized by the regularization coefficient) with the sample complexity of the number of problem instances { P (1) , . . . , P ( n ) } drawn i.i.d. from some unknown problem distribution D needed for learning good regularization parameters with high confidence. Let h EN ( λ, P ) = l EN ( ˆ β ( X,y ) ( λ ) , ( X val , y val )) be the validation loss function of the Elastic Net, and H EN = { h EN ( λ, P ) : Π m,p → R ≥ 0 | λ ∈ R 2 > 0 } be the corresponding validation loss function class, we now present tight bounds for Pdim ( H EN ) .

3.1 The Goldberg-Jerrum framework

Recently, Bartlett et al. [31] instantiate a simplified version of the well-known Goldberg-Jerrum (GJ) Framework [17]. The GJ framework offers a general pseudo-dimension upperbound for a wide class of functions in which each function can be computed by a GJ algorithm . We provide a brief overview of the GJ Framework which is useful in establishing our improved pseudo-dimension upper bound.

Definition 1 (GJ Algorithm, [31]) . A GJ algorithm Γ operates on real-valued inputs, and can perform two types of operations:

Arithmetic operators of the form v ′′ = v ⊙ v ′ , where ⊙ ∈ { + , -, × , ÷} , and

Conditional statements of the form ' if v ≥ 0 . . . else . . . ' .

In both cases, v and v ′ are either inputs or values previously computed by the algorithm.

General speaking, each intermediate value of the GJ algorithm Γ can be described by a rational function , which is a fractional between two polynomials, of the algorithm's inputs. The degree of a rational function is equal to the maximum degree of the polynomials in its numerator and its denominator. We can define two quantities that represent the complexity of GJ algorithms.

Definition 2 (Complexity of GJ algorithm, [31]) . The degree of a GJ algorithm is the maximum degree of any rational function it computes of the inputs. The predicate complexity of a GJ algorithm is the number of distinct rational functions that appear in its conditional statements.

The following theorem essentially shows that for any function class F , if we can describe any function f ∈ F by a GJ algorithm of which the degree and predicate complexity are at most ∆ and Λ , respectively, then we can automatically obtain the upper bound for the pseudo-dimension of F .

Theorem 3.1 ([31]) . Suppose that each function f ∈ F is specified by n real parameters. Suppose that for every x ∈ X and r ∈ R , there is a GJ algorithm Γ x,r that given f ∈ F , returns "true" if f ( x ) ≥ r and "false" otherwise. Assume that Γ x,r has degree ∆ and predicate complexity Λ . Then, Pdim ( F ) = O ( n log(∆Λ)) .

3.2 Upper bound

Our work improves on prior research [14] by presenting an upper bound on the pseudo-dimension of Elastic Net validation loss function class H EN parameterized by λ . We extend the previous piecewise-decomposable structure of the loss function by providing a bound on the number of distinct rational piece functions for any fixed problem instance (Definition 3). This allows us to use a GJ algorithm and Theorem 3.1 to obtain better bounds on the number of distinct predicates that need to be computed. While prior research only used a bound on the number of distinct loss function pieces generated by algebraic boundaries, our new observation that the loss function has a limited number of possible distinct functional behaviors yields a tighter upper bound on the pseudo-dimension (Theorem 3.2). In Theorem 3.5, we will demonstrate the tightness of our upper bound by providing a novel lower bound for the problem.

We first provide a refinement of the piece-wise decomposable function class terminology introduced by [32] which is useful for establishing our improved upper bound. Intuitively, this corresponds to real-valued functions for which the domain is partitioned by finitely many boundary functions such that the function is well-behaved in each piece in the partition, i.e. can be computed using a piece function from another function class.

r(,2)= 0

f(l)

f2(M)

f4(l)

fs(l)

fs(l)

r2( 2) = 0

Definition 3. A function class H ⊆ R Y that maps a domain Y to R is ( F , k F , G , k G ) -piecewise decomposable for a class G of boundary functions and a class F ∈ R Y of piece functions if the following holds: for every h ∈ H , (1) there are k G functions g (1) , . . . , g ( k G ) ∈ G and a function f b ∈ F for each bit vector b ∈ { 0 , 1 } k G s.t. for all y ∈ Y , h ( y ) = h b y ( y ) where b y = { ( g (1) ( y ) , . . . , g ( k G ) ( y )) } ∈ { 0 , 1 } k G , and (2) there is at most k F different functions in F .

A key distinction from [32] is the finite bound k F on the number of different piece functions needed to define any function in the class H . Under this definition we give the following more refined structure for the Elastic Net loss function class by extending arguments from [14].

Theorem 3.2. Let H EN = { h EN ( λ, · ) : Π m,p → R ≥ 0 | λ ∈ R 2 > 0 } be the class of Elastic Net validation loss function class. Consider the dual class H ∗ EN = { h ∗ P : H EN → R ≥ 0 | P ∈ Π m,p } , where h ∗ P ( h EN ( λ, · )) = h EN ( λ, P ) . Then H ∗ EN is ( F , 3 p , G , p 3 p ) -piecewise decomposable, where the piece function class F = { f q : H EN → R } consists at most 3 p rational functions f q 1 ,q 2 : h EN ( λ, · ) ↦→ q 1 ( λ 1 ,λ 2 ) q 2 ( λ 1 ,λ 2 ) of degree at most 2 p , and the boundary function class G = { g r : H EN →{ 0 , 1 }} consists

of semi-algebraic sets bounded by at most p 3 p algebraic curves g r : h EN ( λ, · ) ↦→ ✶ { r ( λ 1 , λ 2 ) < 0 } , where r is a polynomial of degree at most p .

Figure 2 demonstrates the piece-wise structure of H ∗ EN , which allows us to establish an improved upper bound on the pseudo-dimension.

Theorem 3.3. Let H EN = { h EN ( λ, · ) : Π → R ≥ 0 | λ ∈ R 2 > 0 } be the Elastic Net validation loss function class that maps problem instance P to validation loss h val ( λ, P ) . Then Pdim ( H EN ) is O ( p ) .

Proof Sketch. For every problem instance P ∈ Π m,p , and a threshold r ∈ R , consider the computation ✶ { h EN ( λ, P ) -r ≥ 0 } for any h EN ( λ, · ) ∈ H EN. From Theorem 3.2, we can describe ✶ { h EN ( λ, P ) -r ≥ 0 } as a GJ algorithm Γ P,r which is specified by 2 parameters λ 1 , λ 2 , has degree of at most 2 p , and has predicate complexity of at most ( p +1)3 p (See Figure 3). Then Theorem 3.1 implies that Pdim ( H EN ) = O ( p ) . □

r1(入1,2)< 0?

False

r2(入1,入2)< 0?

r2(入1,入2)< 0?

Tkg(入1,入2)< 0?

rkg(入1,入2)<0?

True

False

True

False

f1(入1,入2)-r ≥ 0?

f2(入1,入2)-r≥0？

fM-1(入1,入2)-r ≥ 0?

fM(入1,入2)-r ≥0?

True

K

False

True

K

False

Y

True

K

False

True

K

False

The detailed proof of Theorem 3.3 can be found on Appendix B.1.2. Recent work by Balcan et al. [14] also studied the Elastic Net, and showed the piece-wise structure of the dual function of the validation loss function which implies an upper bound of O ( p 2 ) by employing the general tool from [32]. We establish a tighter bound of O ( p ) in Theorem 3.3 by establishing additional properties of the loss function class and giving a GJ algorithm for computing the loss functions.

To guarantee the boundedness of the considered validation loss function classes, we will have the following assumptions for the data and regularization parameters. The first assumption is that all features and target values in the training and validation examples are bounded. The second assumption is that we only consider regularization coefficient values λ within an interval [ λ min , λ max ] . In practice, those assumptions are naturally satisfied by data normalization.

Assumption 1 (Bounded covariate and label) . We assume that all the feature vectors and target values in training and validation set is upper-bounded by absolute constants R 1 and R 2 , i.e. max {∥ X ∥ ∞ , ∥ X val ∥ ∞ } ≤ R 1 , and max {∥ y ∥ ∞ , ∥ y val ∥ ∞ } ≤ R 2 .

Assumption 2 (Bounded Coefficient) . We assume that λ ∈ [ λ min , λ max ] 2 with λ min > 0 .

Under Assumptions 2, 1, Theorem 3.3 immediately implies the following generalization guarantee for Elastic Net hyperparameter tuning.

Theorem 3.4. Let D be an arbitrary distribution over the problem instance space Π m,p . Under Assumptions 1, 2, the loss functions in H EN have range bounded by some constant H (Lemma C.1). Then there exists an algorithm s.t. for any ϵ, δ > 0 , given N = O ( H 2 ϵ 2 ( p +log( 1 δ ))) sample problem instances drawn from D , the algorithm outputs a regularization parameter ˆ λ such that with probability at least 1 -δ , E P ∼D h EN ( ˆ λ, P ) < min λ E P ∼D h EN ( λ, P ) + ϵ .

Proof. Denote λ ∗ = argmin λ E P ∼D h EN ( λ, P ) . From Theorems 3.3 and A.2, given n = O ( H 2 ϵ 2 ( p + log( 1 δ ))) problem instances P ( i ) for i ∈ [ N ] drawn from D , w.p. 1 -δ , we have E P ∼D h EN ( ˆ λ, P ) < 1 N ∑ N i =1 h EN ( ˆ λ, P ( i ) ) + ϵ 2 < 1 N ∑ N i =1 h EN ( λ ∗ , P ( i ) ) + ϵ 2 < E P ∼D h EN ( λ ∗ , P ) + ϵ.

3.3 Lower bound

Remarkably, we are able to establish a matching lower bound on the pseudo-dimension of the Elastic Net loss function class, parameterized by the regularization parameters. Note that every Elastic Net problem can be converted to an equivalent LASSO problem [1]. In fact, we show something stronger, that the pseudo-dimension of even the LASSO regression loss function class (parameterized by regression coefficient λ 1 ) is Ω( p ) , from which the above observation follows (by taking λ 2 = 0 in our construction). Our proof of the lower bound adapts the 'adversarial strategy' of [35] which is used to design a worst-case LASSO regularization path. While [35] construct a single dataset to bound the number of segments in the piecewise-linear LASSO solution path, we create a collection of problem instances for which all above-below sign patterns may be achieved by selecting regularization parameters from different segments of the solution path.

Theorem 3.5. Let H LASSO be a set of functions { h LASSO ( λ, · ) : Π m,p → R ≥ 0 | λ ∈ R + } that map a regression problem instance P ∈ Π m,p to the validation loss h LASSO ( λ, P ) of LASSO trained with regularization parameter λ . Then Pdim ( H LASSO ) is Ω( p ) .

Proof Sketch. Consider N = p problem instances for LASSO regression given by P ( i ) = ( X ( i ) , y ( i ) , X ( i ) val , y ( i ) val ) , where the training set ( X ( i ) , y ( i ) ) = ( X ∗ , y ∗ ) is fixed and set using the 'adversarial strategy' of [35], Proposition 2. The validation sets are given by single examples ( X ( i ) val , y ( i ) val ) = ( e i , 0) , where e i are standard basis vectors in R p . We will now proceed to provide the witnesses r 1 , . . . , r N and λ values to exhibit a pseudo-shattering of these problem instances.

Corresponding to subset T ⊆ [ p ] of problem instances, we will provide a value of λ T such that, we have ℓ LASSO ( λ T , P ( i ) ) > r i iff i ∈ T , for each i ∈ [ p ] and each T ⊆ [ p ] . We set all witnesses r i = 0 for all i ∈ [ p ] . As a consequence of Theorem 1 in [35], the regularization path of ( X ∗ , y ∗ ) consists of a linear segment corresponding all 2 p unsigned sparsity patterns in { 0 , 1 } p (we will not need all the segments in the construction, but note that it is guaranteed to contain all distinct unsigned sparsity patterns) and we select λ T as any interior point corresponding to a linear segment with sparsity pattern { ( c 1 , . . . , c p ) | c i = 0 iff i ∈ T } , i.e. elements in T are exactly the ones with sparsity pattern 0. Therefore, | β ∗ T · e i | = 0 iff i ∈ T , where β ∗ T is the LASSO regression fit for regularization parameter λ T . This implies the desired shattering condition w.r.t. witnesses r 1 = 0 , . . . , r N = 0 . Therefore, Pdim ( H LASSO ) ≥ p . See Appendix B.2 for a full proof.

3.4 Hyperparameter tuning in Regularized Kernel Regression

The Kernel Least Squares Regression ([4]) is a natural generalization of the linear regression problem, which uses a kernel to handle non-linearity. In this problem, each sample has p 1 feature, corresponding to a real-valued target. Formally, each problem instance P drawn from Π can be described as

A common issue in practice is that the relation between y and X is non-linear in the original space. To overcome this issue, we consider the mapping ϕ : R p 1 → R p 2 which maps the original input space to a new feature space in which we hopefully can perform linear regression. Define ϕ ( X ) = ( ϕ ( x 1 ) , . . . , ϕ ( x m )) m × p 2 , our goal is to find a vector θ ∈ R p 2 so that the squared loss 1 2 ∥ y -ϕ ( X ) θ ∥ 2 2 + R ( ∥ θ ∥ ) is minimized, where the regularization term R ( ∥ θ ∥ ) is any strictly monotonically increasing function of the Hilbert space norm. It is well-known from the literature (e.g. [36]) that under the Representer Theorem's conditions, the optimal value θ ∗ can be linearly represented by row vectors of ϕ ( X ) , i.e., θ ∗ = ϕ ( X ) β = ∑ m i =1 ϕ ( x i ) β i , where β = ( β 1 , . . . , β m ) ∈ R m . This directly includes the ℓ 2 regularizer but does not include ℓ 1 regularization. To overcome this issue, Roth ([3]) proposed an alternative approach to regularized kernel regression, which directly restricts the representation of coefficient θ via a linear combination of ϕ ( x i ) , for i ∈ [ m ] . The regularized kernel regression hence can be formulated as

where k ( x, x ′ ) = ⟨ ϕ ( x ) , ϕ ( x ′ ) ⟩ is the kernel mapping, and the Gram matrix K satisfies [ K ] i,j = k ( x i , x j ) for all i, j ∈ [ m ] .

Clearly, the problem above is a linear regression problem. Formally, denote h KER ( λ, P ) = 1 2 ∥ y -K ˆ β ( X,y ) ( λ ) ∥ 2 and let H KER = { h KER ( λ, · ) : Π m,p → R ≥ 0 | λ ∈ R 2 + } . The following result is a direct corollary of Theorem 3.3, which gives an upper bound for the pseudo-dimension of H KER.

Corollary 3.6. Pdim ( H KER ) = O ( m ) .

Note that m here denotes the training set size for a single problem instance, and Corollary 3.6 implies a guarantee on the number of problem instances needed for learning a good regularization parameter for kernel regression via classic results [33, 34]. Our results do not make any assumptions on the m samples within a problem instance/dataset; if these samples within problem instances are further assumed to be i.i.d. draws from some data distribution (distinct from problem distribution D ), then well-known results imply that m = O ( k log p ) samples are sufficient to learn the optimal LASSO coefficient [37, 38], where k denotes the number of non-zero coefficients in the optimal regression fit.

4 Hyperparameter tuning for Regularized Logistic Regression

Logistic regression is more naturally suited to applications modeling probability of an event, like medical risk for a patient [39], predicting behavior in markets [40], failure probability of an engineering system [41] and many more applications [42]. It is a fundamental statistical technique for classification, and regularization is again crucial for avoiding overfitting and estimating variable importance. In this section, we will present learning guarantees for tuning the Regularized Logistic Regression (RLR) regularization coefficients across instances. Given a problem instance P drawn from a problem distribution D over Π m,p , let h RLR ( λ, P ) = l RLR ( ˆ β ( X,y ) ( λ ) , ( X val , y val )) be the RLR validation loss function class (defined in Section 2), and let H RLR = { h RLR ( λ, · ) : Π m,p → R ≥ 0 | λ ∈ R > 0 } be the RLR validation loss function class, our goal is to provide a learning guarantee for H RLR. Besides, we also study the commonly used 0-1 validation loss function class H 0-1 RLR = { h 0-1 RLR ( λ, · ) : Π m,p → R ≥ 0 | λ ∈ R > 0 } , where h 0-1 RLR ( λ, P ) = 1 m ′ ∑ m ′ i =1 ✶ { y i x ⊤ i ˆ β X,y ( λ ) ≤ 0 } , which we will cover in Section 4.3. Similarly, to guarantee the boundedness of H RLR, we also assume that Assumptions 1 and 2 also hold in this setting.

4.1 Approximate solutions of Regularized Logistic Regression

The main challenge in analyzing the regularized logistic regression, unlike the regularized logistic regression problem, is that the solution ˆ β ( X,y ) ( λ ) corresponding to a problem instance P and particular value λ > 0 does not have a closed form depending on λ . We then propose an alternative approach to this end, which is examining via the approximation β ( ϵ ) ( X,y ) ( λ ) of the solution ˆ β ( X,y ) ( λ ) .

̸

̸

The approximation Algorithm 1 (Algorithm 2) for the solution ˆ β ( λ ) of RLR under ℓ 1 (or ℓ 2 ) constraint were first proposed by Rosset [26, 2]. Given a problem instance P , and a sufficiently small step-size ϵ > 0 , using Algorithms 1, 2 yields an approximation β ( ϵ ) ( X,y ) of ˆ β ( X,y ) that are piece-wise linear functions of λ in total ( λ max -λ min ) /ϵ [26]. Moreover, it is also guaranteed that the error between β ( ϵ ) ( X,y ) and ˆ β ( X,y ) is uniformly upper bounded for all λ ∈ [ λ min , λ max ] .

Algorithm 2 Approximate incremental quadratic algorithm for RLR with ℓ 2 penalty, [2]

Theorem 4.1 (Theorem 1, [2]) . Given a problem instance P , for small enough ϵ , there is a uniform bound O ( ϵ 2 ) on the error ∥ ˆ β ( X,y ) ( λ ) -β ( ϵ ) ( X,y ) ( λ ) ∥ 2 for any λ ∈ [ λ min , λ max ] .

Denote h ( ϵ ) RLR ( λ, P ) = l RLR ( β ϵ ( λ ) , ( X val , y val )) the approximation function of the validation loss h RLR ( λ, P ) . Using Theorem 4.1 and note that the loss f ( z ) = log(1 + e -z ) is 1 -Lipschitz, we can show that the difference between h ( ϵ ) RLR ( λ, P ) and h RLR ( λ, P ) is uniformly upper-bounded.

Lemma 4.2. The approximation error of the validation loss function is uniformly upper-bounded | h ( ϵ ) RLR ( λ, P ) -h RLR ( λ, P ) | = O ( ϵ 2 ) , for all λ ∈ [ λ min , λ max ] .

We now present one of our main results, which is the pseudo-dimension bound of the approximate validation loss function class H ( ϵ ) RLR .

Theorem 4.3. Consider the RLR under ℓ 1 (or ℓ 2 ) constraint with parameter λ ∈ [ λ min , λ max ] that take a problem instance P drawn from an unknown problem distribution D over Π m,p . Under Assumptions 1 and 2, H RLR is bounded by some constant H (Lemma C.2). Suppose that we use Algorithm 1 (or Algorithm 2) to approximate the solution ˆ β ( X,y ) ( λ ) by β ( ϵ ) ( X,y ) ( λ ) with a uniform error O ( ϵ 2 ) for any λ ∈ [ λ min , λ max ] , where ϵ is the approximation step-size. Consider the approximation validation loss function class H ( ϵ ) RLR = { h ( ϵ ) RLR ( λ, · ) : Π m,p → R ≥ 0 | λ ∈ [ λ min , λ max ] } , where

is the approximate validation loss. Then we have Pdim ( H ( ϵ ) RLR ) = O ( m 2 +log(1 /ϵ )) . Further, we assume that ϵ = O ( √ H ) where H is the upperbound of H RLR under Assumptions 1 and 2. Given any set S of T problem instances drawn from a problem distribution D over Π m,p , the empirical Rademacher complexity ˆ R ( H ( ϵ ) RLR , S ) = O ( H √ ( m 2 +log(1 /ϵ )) /T ) .

The key observation here is that the approximation solution ˆ β ( ϵ ) ( X,y ) is piece-wise linear over ( λ max -λ min ) /ϵ pieces, leading to the fact that the approximate validation loss function h ( ϵ ) RLR ( λ, · ) is a "special function" (Pfaffian function [43]) in each piece, which is a combination of exponentiation of linear functions of λ . The detailed proof of Theorem 4.3 can be found on the Appendix D.3.

4.2 Learning guarantees for Regularized Logistic Regression hyperparameter tuning

Our goal now is to use the upper bound for empirical Rademacher complexity of the validation loss function class H RLR. We use techniques for approximate data-driven algorithm design due to [29], combining the uniform error upper bound between validation loss function h RLR ( λ, P ) and its approximation h ( ϵ ) RLR ( λ, P ) (Lemma 4.2) and empirical Rademacher complexity of approximation validation loss function class H ( ϵ ) RLR (Theorem 4.3), to obtain a bound on the empirical Rademacher complexity of H RLR. This allows us to give a learning guarantee for the regularization parameters λ , which is formalized by the following theorem.

Theorem 4.4. Consider the RLR under ℓ 1 (or ℓ 2 ) constraint. Under Assumptions 1, 2, H RLR is bounded by some constant H (Lemma C.2). Consider the class function H RLR = { h RLR ( λ, · ) : Π m,p → R ≥ 0 | λ ∈ [ λ min , λ max ] } where h RLR ( λ, P ) is the validation loss corresponding to problem

instance P and the ℓ 1 ( ℓ 2 ) parameter λ . Given any set S of T problem instances drawn from a problem distribution D over Π m,p , for any h RLR ( λ, · ) ∈ H RLR, w.p. 1 -δ for any δ ∈ (0 , 1) , we have

for some sufficiently small ϵ .

The proof detail of Theorem 4.4 is included in the Appendix D.3. The above generalization guarantee gives a bound on the average error on RLR validation loss over the problem distribution, for the parameter λ learned from T problem instances. In commonly used approaches, the validation set size is small or a constant, and our result can be interpreted as the upper bound on the generalization error in terms of the number of problem instances T and the step length ϵ . We only consider RLR under ℓ 1 (or ℓ 2 ) constraints, which are commonly studied in the literature, our analysis could be easily extended to RLR under ℓ q constraint for any q ≥ 1 .

4.3 An extension to 0-1 loss

Since logistic regression is often used for binary classification tasks, it is interesting to consider the 0-1 loss as the validation loss function. It has been shown that ✶ { z ≤ 0 } ≤ 4 log(1 + e -z ) for any z [44]. This inequality, combined with Theorem 4.4, directly provides a learning guarantee for the 0-1 validation loss function.

Theorem 4.5. Let τ > 2 ϵ 2 and δ ∈ (0 , 1) , where ϵ is the approximation step-size. Then for any n ≥ s ( τ/ 2 , δ ) = Ω ( H 2 ( m 2 +log 1 ϵ )+log 1 δ ( τ/ 2 -ϵ 2 ) 2 ) , if we have n problem instances { P ( i ) , . . . , P ( n ) } drawn i.i.d. from some problem distribution D over Π m,p to learn the regularization parameter λ ERM for RLR via ERM, then

The detailed proof of Theorem 4.5 can be found on Appendix D.4. It is worth noting that we are providing learning guarantee for 0-1 validation loss function class H 0-1 RLR indirectly via the validation loss function class H RLR with cross-entropy objective function, which is arguably not optimal. The question of how to provide a true PAC-learnable guarantee for H 0-1 RLR remains an interesting challenge.

5 Conclusion and future work

In this work, we present novel learning guarantees for tuning regularization parameters for both the Elastic Net and Regularized Logistic Regression models, across problem instances. For the Elastic Net, we propose fine-grained structural results that pertain to the tuning of regularization parameters. We use them to give an improved upper bound on the pseudo-dimension of the relevant validation loss function class of and we prove that our new bound is tight.

For the problem of tuning regularization parameters in regularized logistic regression, we propose an alternative approach that involves analyzing the approximation of the original validation loss function class. This approximation, characterized by a piece-wise linear representation, provides a useful analytical tool in the absence of an exact dependence of the logistic loss on the regularization parameters. Additionally, we employ an upper bound on the approximation error between the original and approximated functions, to obtain a learning guarantee for the original validation loss function class. Remarkably, our proposed approach is not restricted solely to regularized logistic regression but can be extended to a wide range of other problems, demonstrating its generality and applicability.

It is worth noting that this work only focuses on the sample complexity aspect of the hyperparameter tuning in the Elastic Net and Regularized Logistic Regression. The question of computational complexity in this setting is an interesting future direction. Other interesting questions include designing hyperparameter tuning techniques for this setting that are robust to adversarial attacks, and hyperparameter tuning for Regularized Logistic Regression with both ℓ 1 and ℓ 2 constraints.

References

Hui Zou and Trevor Hastie. Regularization and variable selection via the Elastic Net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 67(2):301-320, 2005.

Saharon Rosset. Following curved regularized optimization solution paths. Advances in Neural Information Processing Systems , 17, 2004.

Volker Roth. The generalized LASSO. IEEE transactions on neural networks , 15(1):16-28, 2004.

Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Arthur Hoerl and Robert Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics , 12(1):55-67, 1970.

Robert Tibshirani. Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics , 46(1):247-279, 2018.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Gang Wang, Dit-Yan Yeung, and Frederick H Lochovsky. The kernel path in kernelized LASSO. In Artificial Intelligence and Statistics , pages 580-587. PMLR, 2007.

Yunlong Feng, Shao-Gao Lv, Hanyuan Hang, and Johan Suykens. Kernelized Elastic Net regularization: generalization bounds, and sparse recovery. Neural computation , 28(3):525-562, 2016.

Kevin P Murphy. Machine learning: a probabilistic perspective . MIT press, 2012.

Tong Zhang. Some sharp performance bounds for least squares regression with L1 regularization. The Annals of Statistics , pages 2109-2143, 2009.

Maria-Florina Balcan, Mikhail Khodak, Dravyansh Sharma, and Ameet Talwalkar. Provably tuning the Elastic Net across instances. In Advances in Neural Information Processing Systems , 2022.

Rich Caruana. Multitask learning. Machine learning , 28:41-75, 1997.

Mervyn Stone. Cross-validation: A review. Statistics: A Journal of Theoretical and Applied Statistics , 9(1):127-139, 1978.

Paul Goldberg and Mark Jerrum. Bounding the Vapnik-Chervonenkis dimension of concept classes parameterized by real numbers. In Proceedings of the sixth annual conference on Computational learning theory , pages 361-369, 1993.

Li Wang, Michael D Gordon, and Ji Zhu. Regularized least absolute deviations regression and an efficient algorithm for parameter tuning. In Sixth International Conference on Data Mining (ICDM'06) , pages 690-700. IEEE, 2006.

Denny Wu and Ji Xu. On the optimal weighted L2 regularization in overparameterized linear regression. Advances in Neural Information Processing Systems , 33:10112-10123, 2020.

Diane Galarneau Gibbons. A simulation study of some ridge estimators. Journal of the American Statistical Association , 76(373):131-139, 1981.

Lisa-Ann Kirkland, Frans Kanfer, and Sollie Millard. LASSO tuning parameter selection. In Annual Proceedings of the South African Statistical Association Conference , volume 2015, pages 49-56. South African Statistical Association (SASA), 2015.

Yosiyuki Sakamoto, Makio Ishiguro, and Genshiro Kitagawa. Akaike information criterion statistics. Dordrecht, The Netherlands: D. Reidel , 81(10.5555):26853, 1986.

Gideon Schwarz. Estimating the dimension of a model. The annals of statistics , pages 461-464, 1978.

Michael Chichignoud, Johannes Lederer, and Martin Wainwright. A practical scheme and fast algorithm to tune the LASSO with optimality guarantees. The Journal of Machine Learning Research , 17(1):8162-8181, 2016.

Denis Chetverikov, Zhipeng Liao, and Victor Chernozhukov. On cross-validated LASSO in high dimensions. The Annals of Statistics , 49(3):1300-1317, 2021.

Saharon Rosset. Topics in regularization and boosting . PhD thesis, Stanford University, 2003.

Ryan Tibshirani. The LASSO problem and uniqueness. Electronic Journal of Statistics , 7:1456-1490, 2013.

Rishi Gupta and Tim Roughgarden. A PAC approach to application-specific algorithm selection. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science , pages 123-134, 2016.

Maria-Florina Balcan, Tuomas Sandholm, and Ellen Vitercik. Refined bounds for algorithm configuration: The knife-edge of dual class approximability. In International Conference on Machine Learning , pages 580-590. PMLR, 2020.

Maria-Florina Balcan, Siddharth Prasad, Tuomas Sandholm, and Ellen Vitercik. Structural analysis of branch-and-cut and the learnability of gomory mixed integer cuts. Advances in Neural Information Processing Systems , 35:33890-33903, 2022.

Peter Bartlett, Piotr Indyk, and Tal Wagner. Generalization bounds for data-driven numerical linear algebra. In Conference on Learning Theory , pages 2013-2040. PMLR, 2022.

Maria-Florina Balcan, Dan DeBlasio, Travis Dick, Carl Kingsford, Tuomas Sandholm, and Ellen Vitercik. How much data is sufficient to learn high-performing algorithms? Generalization guarantees for data-driven algorithm design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 919-932, 2021.

Martin Anthony and Peter Bartlett. Neural Network Learning: Theoretical Foundations . Cambridge University Press, USA, 1st edition, 2009.

Maria-Florina Balcan. Book chapter Data-Driven Algorithm Design. In Beyond Worst Case Analysis of Algorithms, T. Roughgarden (Ed) . Cambridge University Press, 2020.

Julien Mairal and Bin Yu. Complexity analysis of the LASSO regularization path. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1835-1842, 2012.

Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In Computational Learning Theory: 14th Annual Conference on Computational Learning Theory, COLT 2001 and 5th European Conference on Computational Learning Theory, EuroCOLT 2001 Amsterdam, The Netherlands, July 16-19, 2001 Proceedings 14 , pages 416-426. Springer, 2001.

Martin Wainwright. Information-theoretic bounds on sparsity recovery in the high-dimensional and noisy setting. In 2007 IEEE International Symposium on Information Theory , pages 961-965. IEEE, 2007.

Martin Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using L1-constrained quadratic programming (LASSO). IEEE transactions on information theory , 55(5):2183-2202, 2009.

Peter Armitage, Geoffrey Berry, and John Nigel Scott Matthews. Statistical methods in medical research . John Wiley & Sons, 2008.

Michael JA Berry and Gordon S Linoff. Data mining techniques: for marketing, sales, and customer relationship management . John Wiley & Sons, 2004.

Sanjay Kumar Palei and Samir Kumar Das. Logistic regression model for prediction of roof fall risks in bord and pillar workings in coal mines: an approach. Safety science , 47(1):88-96, 2009.

David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. Applied logistic regression , volume 398. John Wiley & Sons, 2013.

Askold G Khovanski. Fewnomials , volume 88. American Mathematical Soc., 1991.

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems , 32, 2019.

Richard M Dudley. Universal, Donsker classes and metric entropy. The Annals of Probability , 15(4):1306-1326, 1987.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of hyperparameter tuning across problem instances, as proposed by Balcan et al. (2022). They propose three novel learning guarantees regarding the sample complexity of tuning regularization parameters: i) an improved upper bound on the pseudo-dimension for elastic net; ii) a matching lower bound in the same setting; iii) a bound on the pseudo-dimension and on the generalization error for logistic regression.\n\nSTRENGTHS:\nThe paper presents new results, which either improve the state-of-the-art (i) or pave a new way (ii and iii) in analyzing common learning techniques. I think these results are topical and interesting for the community.\n\nWEAKNESSES:\nAs an unschooled researcher regarding the theory of hyperparameter tuning across instances, I found the paper not that easy to read. I particular, I missed the target the authors aimed at (it is stated at the end), which would help me to understand. Here are some remarks which may help in improving the paper:\n1) Some elementary definitions are missing and curb the reader understanding (particularly the pseudo-dimension, the Rademacher complexity and the ERM principle in Theorem 4.5). Also, a comment for Theorem 3.1 would be appreciated. As for me, a simple improvement of the paper would be to state informally a kind of Theorem 4.4 at the beginning, such that the reader is more able to understand Sections 3 and 4.\n2) Minor comments: “optimization problem 1” should have capital letters, a “be” is missing Line 188, an “of” is missing Line 191, “an GJ algorithm” in the caption of Figure 1 should be “a GJ algorithm”, “Roth et al. ([3])” Line 251 should be “Roth [3]”, increasingness of $R(||\\theta||)$ would be better place before stating the representer expansion, the update of $\\beta_t^{(\\epsilon)}$ in Algorithm 1 should be stored in $\\beta_{t+1}^{(\\epsilon)}$ instead of $\\beta^{(\\epsilon)}(\\lambda)$, moreover it involves the difference of two vectors of different sizes.\n\nQUESTIONS:\n1) Can the authors define formally the probability space used for the problems (Line 117)? It is a bit disturbing to manipulate objects of varying size.\n2) Should it be $h_{b_y}$ and $g^{(1)}(y)$ in Lines 183-184?\n\nLIMITATIONS:\nLimitations are not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyses the complexity of hypotheses classes where the hyperparameter of logistic regression and linear regression are tuned, in the setting where multiple datasets are available.\n\nSTRENGTHS:\nThe paper derives tighter upperbounds for the elastic net setting and proves a matching lowerbound, which are extended to KRR, for the setting where regularization parameters are learned from problem instances (e.g. metalearning). The regularized logistic regression setting is tackled by approximating the function class.\n\nWEAKNESSES:\n1 While I am quite familiar with learning theory (e.g. VC bounds, PAC, pseudo-dimension) I really have a hard time grasping the problem setting. I had to consult [15] and [27] for that. I think the paper could be greatly improved by clarifying the problem setting in more detail, with some example settings machine learners will be familiar with. E.g. the example of cross validation of [15]. It would also be good to clarify generally what are the inputs to the learner; it is still a bit unclear to me. So are [X_train, y_train, X_test] the inputs? And the output is $\\lambda$? The strange thing is, is that the hypothesis seems to take $\\lambda$ as input (definition of h), while the $\\lambda$ seems the thing we want to learn. \n\n2 Furthermore, in line 124 it is stated \"the learning algorithm in this scenario has been mentioned in [15], which is based on simulated annealing\" this seems wrong, as the learner in that work [15] is operating in the online learning setting, while the current work seems more to focus on the statistical learning setting. Shouldn't instead be the ERM learner or CV learner of [15] be mentioned? \n\n3 Generally, I think more examples are necessary to grasp the main point of the setting. It would also be clear to further clarify why this is an important problem to study; for which learners can we now derive theoretical guarantees? \n\n4 Once I had clarified the setting I had no time to read the paper furthermore in detail, so I cannot judge the other aspects in more detail.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nA challenge is to give a tighter PAC bound for the 01 loss for RLR", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main idea of this paper is to address the challenge of tuning regularization coefficients in regression models with provable guarantees across problem instances. The authors investigate the sample complexity of tuning regularization parameters in linear and logistic regressions under l1 and l2 constraints in the data-driven setting. They provide new upper bounds for the pseudo-dimension of the validation loss function class, which significantly improves the existing results on the problem. Additionally, the paper introduces a new approach for studying the learning guarantee in logistic regression regularization coefficients. Overall, the paper aims to contribute to the literature by providing improved bounds and advancements in the field of hyperparameter tuning for regression problems.\n\nSTRENGTHS:\n1. Improved Bounds: One strength of this paper is that it provides improved upper bounds for the pseudo-dimension of the validation loss function class in linear and logistic regressions. These improved bounds contribute to a better understanding of the sample complexity of tuning regularization parameters in regression models.\n2. Generality and Applicability: The proposed approach for analyzing the approximation of the original validation loss function class is not limited to regularized logistic regression but can be extended to a wide range of other problems. This demonstrates the generality and applicability of the approach, making it a valuable contribution to the field.\n\nWEAKNESSES:\n1. Lack of Experimental Validation: The paper focuses on theoretical analysis and bounds but does not provide experimental validation or empirical results to support the proposed approach. Including experimental validation would have added practical relevance and strengthened the paper's findings by demonstrating the effectiveness of the approach in real-world scenarios.\n2. Lack of Clarity and Visual Support: One weakness of this paper is the insufficient clarity in explaining the problem it addresses. The excessive repetition of formulas and the absence of visual aids, such as graphs or charts, make it difficult for readers to understand the main ideas and practical implications of the research. The lengthy theoretical derivations, while comprehensive, may hinder the overall comprehension of the paper's intended message.\n\nQUESTIONS:\nIs it possible to provide any experimental study?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the sample complexity of tuning regularization parameters in linear and logistic regressions under $\\ell_1$ and $\\ell_2$ constraints in the data-driven setting. Theoretically, it provides a new bound for the pseudo-dimension of the validation loss function class, which significantly improves the best-known results on the problem. Besides, it provides the matching lower bound. Moreover, it also provides a new bound for tuning the regularization parameters of logistic regression, which previous work cannot do.\n\nSTRENGTHS:\nFirst of all, I have to admit that I am not an expert in the area of data-driven algorithm design and may miss some related work. Overall, I think this is a solid theoretical work.\n\n- Overall, this paper is well-written and discussions about the comparisons with the related work are clear.\n- The considered problem (e.g., hyperparameter tuning or model selection) is important for the machine learning community. \n- The theoretical results seem right although I have not carefully checked the proofs line-by-line.\n- Theoretically, the obtained upper bound is tight since the (nearly) matched lower bound is provided.\n- In contrast with previous work, it is good to provide the analysis for the logistic regression as it might provide techniques to analyze more general settings with non-closed solution forms.\n\nWEAKNESSES:\n- Although this is a pure theory work, it would be better to add experimental results to illustrate its applications in practice.\n- While the considered setting is the hyperparameter tuning across instances, more discussions can be added to illustrate its practicality over previous work with a single instance.\n\n\n\nMinor issues:\n\ntypos：\n- In lines 503-507 (i.e., Lemma A.1), it is not consistent about the expression of non-increasing (or decreasing) and non-decreasing (or increasing).\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nPlease see the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}