Title: Beyond Tikhonov: faster learning with self-concordant losses, via iterative regularization
Authors: Gaspard Beugnot
Julien Mairal
Alessandro Rudi
Venue: NeurIPS 2021
Keywords: Kernel methods
learning theory
self concordance
iterated tikhonov
proximal point

Abstract: The theory of spectral filtering is a remarkable tool to understand the statistical properties of learning with kernels. For least squares, it allows to derive various regularization schemes that yield faster convergence rates of the excess risk than with Tikhonov regularization. This is typically achieved by leveraging classical assumptions called source and capacity conditions, which characterize the difficulty of the learning task. In order to understand estimators derived from other loss functions, Marteau-Ferey et al. have extended the theory of Tikhonov regularization to generalized self concordant loss functions (GSC), which contain, e.g., the logistic loss. In this paper, we go a step further and show that fast and optimal rates can be achieved for GSC by using the iterated Tikhonov regularization scheme, which is intrinsically related to the proximal point method in optimization, and overcomes the limitation of the classical Tikhonov regularization.

TL;DR: The iterative Thikonov regularization scheme achieves optimal sample complexity on self concordant losses. 

Review discussion:
This paper gives optimal learning rates for regression with generalized self-concordant convex loss functions and the saturation effect is overcome. The goal is achieved by iterations with proximal operators with the number of iterations depending on the order r of approximation in the source condition as t \geq r+1/2. The optimal rates stated in Theorem 1 are significant. These rates and the choice of the regularization parameter are stated in terms of the source condition and capacity condition imposed on the underlying probability measure. The authors also present sufficient conditions for the optimal learning rates when implementing the scheme in practical applications. The ideas in the paper are novel and clearly presented.
No
We thank this reviewer for the positive comments and for recognizing the added value of our work.
We agree with the reviewer about the explicit dependence on the confidence. We recognize that the mentioned references provide an interesting quantitative bound in terms of the confidence with respect to [3] in the context of squared loss and we will add them to the literature review.
In any case explicit dependence on the confidence is an interesting topic and it would be nice to obtain a similar analysis for estimator based on GSC. This could constitute an interesting future work.
In our proof strategy, all the quantities are vectors or operators of the Hilbert space. This makes the analysis simpler when $\theta^\star$ is in $\mathcal{H}$.  Clearly, if $\theta^\star$ is not in $\mathcal{H}$, we cannot use the same tools, and we need instead to express all these quantities as vectors or operators in $L^2$.
In particular, we would also need to find an equivalent of the integral operator for GSC loss functions, which would constitute a very interesting future work.
I read the authors' response and would like to keep my rating.
There are two points which would in my opinion improve the paper: $i$) notation and $ii$) discussion of prior work on sequential greedy approximations.
I have tried to list below some of the parts that were confusing to me when reading for the first time.
- Assumption 1: The Hessian is only defined in Assumption 2 and yet it is for the first time used in Assumption 1. Given the definition in Assumption 2, I think it should be $Tr\ H(0)$ instead of the existing expression?
- Definition 2 (generalized self-concordance): I fail to parse this definition of self-concordance and the reference provided is not helping. It is not clear what the operator $[...]$ does and what the role of h and k is here.
When referring to Boyd (Convex optimization), the self-concordant loss is introduced first for 1D functions and then an extension to Euclidean spaces requires that "... it is self-concordant along every line in its domain...". Given that the latter deals with vector inputs, there is a reparametrization $\hat{l}(t)=l(u + tv)$ where $t \in \mathbb{R}$ and $u + tv$ are all lines. The definition then reduces to that of the 1D case.
This work is introducing the self-concordance over reproducing kernel Hilbert spaces and it would be good to do this in details and as clearly as possible.
Moreover, the right-hand side has a term $\sup | k\ \dot\   g|$ which requires clarification and motivation (e.g., the explicit impact it will play in the bounds, in my understanding it is fundamental for sharp rates). Along these lines, the map $\phi$ needs motivation so that one is able to carry this definition with some understanding/motivation to the proofs.
- Definition 3: The norm has only been defined in the appendix and it would be good to have it as part of this definition. That would also allow for an easy walk through the paragraph that follows.
- Eq. (12): This is an unfortunate overload of symbols, t denoting the function and iteration number in the iterated Tikhonov scheme.
- Eq. (33): Which operator takes the precedence, $+$ or $\wedge$?
The second comment is regarding prior work on sequential greedy approximations, which is especially related to the second part of the paper and Proposition 1. If my understanding is correct, the iterated Tikhonov scheme has been studied previously under the name of sequential greedy approximations and it would be great if the authors could expand the discussion of related work by comparing their contributions to the two papers below. It would be a great way of bridging the two disparate directions that have significant overlap. Here, I would like to point out that the theoretical contributions of this paper are significant and augment prior understanding of the iterated scheme.
[1] M. Donahue, C. Darken, L. Gurvits, and E. Sontag (Constructive Approximation, 1997). Rates of convex approximation in non-Hilbert spaces.
[2] T. Zhang (IEEE Transaction on Information Theory, 2003). Sequential greedy approximations for certain convex optimization problems.
No
We thank the reviewer for very positive feedback, the constructive comments and the suggestions regarding the notation. They will certainly improve the readability and will be included in the article. Some details are provided below.
We thank the reviewer for highlighting the connections with the literature of sequential greedy approximation. We will include the proposed references and explore such connections in our previous works section. It seems indeed that sparse greedy approximation relates to proximal point optimization, since the update rule can be written as
\begin{aligned}
\theta_k &= \theta_{k-1} + \arg \min_{\theta} L(\theta + \theta_{k-1}) + \lambda/2 || \theta ||^2 \\\\
&= \theta_{k-1} + \arg \min_{|| \theta || \leq \rho_{k-1}} L(\theta + \theta_{k-1})
\end{aligned}
with $\lambda$ being the dual variable associated to the constraint $|| \theta || \leq \rho_{k-1}$ at step $k$. This is indeed similar to the update rule (*) in algorithm 2.1 of [2]. In the previous works section, we will elaborate on the comment above.
### Minor edits
* Assumption 1: Indeed, this should read $Tr \nabla^2 \ell_z(0)$.
* Definition 2: Thanks for the clarifying suggestions.
Self concordant functions were introduced to conduct a general analysis of the Newton method in optimization. They are adapted in [a,b] to encompass a larger class of loss function, “$\nu$-self concordant functions” which are different from Nesterov’s concordant function.
The map $\phi$ is indeed important and governs to what extent the loss function can be approximated by its quadratic approximation. In appendix C.1, the radius $R$ is introduced l. 782 along with the radius of the Dikin ellipsoid l. 783. Intuitively, as soon as the estimator $\hat{\theta}$ enters this ellipsoid, we can bound quantities depending on $\mathsf{t}$ in the inequalities.
When it’s close to the optimum, deep in this ellipsoid, $\mathsf{t} \to ½$ and the bounds in Prop. 3 and Lemma 2 tend to the bounds we have for the square loss.
On the contrary, when we leave this ellipsoid, the upper bound diverges exponentially to infinity while the lower bound goes exponentially to 0, making the bounds of Prop. 3 very loose.
To conclude, a GSC function with high R has a small Dikin ellipsoid, and is far from its quadratic approximation.
On the contrary, a GSC function with low R will be close to its quadratic approximation ; the Dikin ellipsoid is large. The extreme case is obtained when \ell is the square loss.
Then, $\phi = \{0\}$, so $R = 0$, so the Dikin ellipsoid spans the whole space. This implies e.g. that the lower and upper bound on the gradient matches, making the quadratic approximation tight.
We can also refer to [c], example 1, which provides multiple example of values of $\phi$ for usual loss functions.
[a] Tianxiao Sun, and Quoc Tran-Dinh. Generalized self-concordant functions: a recipe for Newton-type methods. Mathematical Programming 178(1): 145-213, 2019.
[b] Francis Bach. Self-Concordant Analysis for Logistic Regression. Electronic Journal of Statistics, 4, 384-414, 2010.
[c] U. Marteau-Ferey, D. Ostrovskii, F. Bach, and A. Rudi. Beyond least-squares: Fast rates for regularized empirical risk minimization through self-concordance. In Conference on Learning Theory (COLT), 2019.
* Definition 3, Eq. (12): these will be corrected.
* Eq. (33): “∧” takes precedence, parenthesis will be added to avoid confusion.
Thank you very much for a detailed response. I look forward to reading the next version, this is a very good submission.
## Detailed Comments
This is a strong submission which continues a sequence of well-known existing works on learning-theoretic bounds for self-concordant functions [1,2].
The main convergence rate for iterated Tikhonov regularization is novel and addresses a specific open question from Marteau-Ferey et al., who posited that alternative regularizers could overcome the _saturation effect_ of Tikhonov regularization.
Despite the mathematical complexity, the result is fairly intuitive: Tikhonov regularization is not flexible enough to handle cases where the bias decays very quickly in the regularization strength (ie, $r$ in the source condition is large), while iterated Tikhonov can exploit this structure in the data.
I particularly enjoyed the results on excess risk from sub-optimality of the optimization procedure.
Bias from optimization is easy to ignore, but a problem of significant concern when composing optimization problems, as in iterated Tikhonov.
The requirement that accuracy decay geometrically is a nice result that matches our intuition: early accuracy is critical to avoid propagating errors.
The main weaknesses of the submission are the following: i) it is an iterative improvement on existing work, and ii) the complexity of the theoretical developments, which often lack suitable intuition.
As to the first point, the major assumptions used by the authors --- generalized self-concordance, capacity condition, and source condition --- were previously leveraged by Marteau-Ferey et al. and are not new contributions.
Similarly, the approach based on upper and lower second-order Taylor bounds using (generalized) self-concordance is due Bach [2].
On the second point, some clearer intuition on the source\capacity conditions in the main text would be very useful for the reader.
For example, Marteau-Ferey [1] provide fairly detailed developments connecting the conditions to spectral properties of the expected Hessian after they are introduced.
With that said, these are small issues that can be addressed before the camera-ready deadline.
### Writing
The writing in main paper is polished with only a few typographical issues. See "Minor Comments" below.
### Theory
I briefly checked the appendix and did not see any issues.
However, the proofs provided with the paper are 22 pages long and I could not fully examine them during the review period.
### Experiments
The experiments are toy-sized, but do a fair job illustrating the main theorem.
The match between theoretical predictions and estimated learning rates in Table 1 is surprisingly tight.
## Minor Comments
- Line 139: "As we mentioned earlier, the key hindsight on" -> "As we mentioned earlier, the key \*\*insight\*\* on"
- Line 332 "This paper study a well-known" -> "This paper \*\*studies\*\* a well-known".
# References
[1] Marteau-Ferey, Ulysse, et al. "Beyond least-squares: Fast rates for regularized empirical risk minimization through self-concordance." Conference on Learning Theory. PMLR, 2019.
[2] Bach, Francis. "Self-concordant analysis for logistic regression." Electronic Journal of Statistics 4 (2010): 384-414.
No
We thank the reviewer for this very positive feedback.
Indeed, we hope this paper will pave the way for more results on the statistical properties of estimators without a closed-form expression, notably when using different regularization schemes with GSC loss functions.
Regarding the first comment, this paper focuses indeed on the qualification of IT regularization and sufficient conditions for approximating it in practice. We used the mathematical toolbox on GSC functions of [1] which we slightly adapted to our needs (Lemma 1 for the upper bound on gradients, along with different concentration bounds and inequalities in Appendix D to account for larger source condition parameter $r$). Exending this toolbox beyond reproducing kernel Hilbert spaces is not trivial and deserves a work on its own, similar to the effort in the community of kernel least squares to go from the attainable, to the non-attainable case.
Notably, the case where $\theta^\star$ is not contained in the reproducing kernel Hilbert space but is only squared integrable will require to define the Hessian and the source condition in terms of an integral operator in $L^2(\rho_x)$ with the consequence that many concentration result à la Caponnetto de Vito [a] could not be used.
In any case this is a very interesting direction that we will explore in a future work.
Regarding the second comment, we can include the following comment in Sec. 2.2, on top of references to [1], to better describe the role of the source and capacity condition.
> To understand the source and capacity condition, one must pay attention to the counterpart of the covariance operator for GSC loss function, that is the expected hessian at optimality. It is denoted with ${\bf H} (\theta^\star)$ in the paper. The source and capacity conditions are assumptions on the eigendecomposition of this operator.
> The source condition is a smoothness assumption on $\theta^\star$. It amounts to assuming that the eigendecomposition of $\theta^\star$ on the basis of the Hessian decays faster than its spectrum. Bigger $r$ implies that the optimum can be well approximated by a few eigenvectors. Assuming $r =0$ simplifies to $\theta^\star \in \mathcal{H}$.
> Similarly, the capacity condition is an assumption on the decay of the spectrum of the Hessian.
Specifically, it assumes that the spectrum decays polynomially, i.e $\sigma_j \sim j^{-\alpha}$. As this operator is compact, we have $\alpha > 1$ for the $\sum_j j^{-\alpha}$ to be summable. Bigger $\alpha$ gives easier input space $\mathcal{X}$.
### References
[a] A. Caponnetto, E. De Vito.  Optimal Rates for the Regularized Least-Squares Algorithm. Foundations of Computational Mathematics, (2007)
Thanks for the response.
- I recognize that this work extends the GSC framework to a more general setting. My point is only that the analysis based on GSC functions originates in previous work. I don't think that this significantly detracts from the contributions of the submission.
- I agree that the proposed comment is sufficient to clarify the source and capacity conditions.
# Pros:
+ The paper is written well.
+ I think the presented results are solid that extend existing results for both Tikhonov regularization and least squares.
# Cons:
- Even though the computational issues are discussed a bit in the paper, the results are built upon Newtons' method which poorly scales with the problem dimensionality. Discussions for obtaining the inexact solutions by first-order methods could be interesting, especially observing that the proximal algorithm is closely related to gradient descent.
- Eq. (19) is hard to interpret. Could you elaborate what is $\bar{L}$?
- l.153. Based on my calculation I think the qualification for Tikhonov regularization is $1$ instead of $1/2$. This is also supported by the references provided. Since this property is used for multiple times in the later results, could you please elaborate a bit? I might miss important point here as I have little background on inverse problems.
# Overall
I am temporarily holding a weak reject to respect the potential technical mistake mentioned above & I will be happy to re-evaluate the paper based on the future feedback from the authors.
# Typos
* l.105. The last $\theta^*$ should be $\theta$
* l.115. I am not sure if I missed something, but I feel here it should write $df \propto Tr TT_{\lambda}^{-1}$?
For example, consider least square with additive Gaussian noise with variance level $\sigma$?
* The paragraph title style (capitalization) could be aligned.
Post-Rebuttal: after discussion with the authors, it turns out the issue of the qualification of Tikhonov is a typo, but does not affect the final established results. I am therefore happy to vote for acceptance to this work.
No
We thank the reviewer for their constructive comments, in particular for recognizing that the presented results are solid and extend existing results for both Tikhonov regularization and least squares. We remark briefly here that there is no error in the qualification for Tikhonov filter, but just a different notation, as discussed in the answer to Question 3, below.
### Question 1
Newton method or first order methods. We agree with the reviewer that generally speaking first order methods are considered more performant than Newton methods on large scale problems.
Indeed, in the experimental section, we also used first order methods (with Cyanure toolbox [a]) to reach machine precision, which provided analogous results to 2nd order scheme. We will add a discussion about it in the paper.
However, two reasons motivates the use of Newton method:
* Practical considerations: For the specific case of learning with GSC losses, it is possible to leverage techniques from randomized linear algebra and preconditioned iterative solvers, to scale up in a very efficient way up to datasets of $10^9$ examples.
For example there is Falkon [b] an advanced library for large scale computations, that already implements the Thikonov scheme for GSC losses via Newton method as described in [c]. This library achieved the remarkable result of running Newton method for GSC losses (in particular for logistic loss) on a large scale dataset of $10^9$, being order of magnitudes faster than competitors that were using first order methods.
* Theoretical considerations: the analysis of GSC functions is well suited to 2nd order optimization schemes.
In particular, the Newton decrement becomes a quite natural measure to track the error for GSC losses. If we measured the error differently (e.g by the difference in function values between the computed quantity and the value at the global minimum), we would need additional assumptions on the loss function.
### Question 2
The quantity $\bar{L}$ is the empirical function we minimize when we compute the proximal operator of the loss applied to the current estimate.
In practice, when computing the IT’s estimator, this is the function we manipulate at step $k$. If the optimization was carried without error, we would have $\bar{L}^k = \hat{L}^k$, as defined in Eq. (18). Note that at the first step, $\hat{L}^0 = \bar{L}^0$. We will add a comment explaining this function more in detail.
### Question 3
The value of the saturation depends on how the source condition is defined in the filter literature.
In particular, the source condition parameter $r$ defined in terms of the integral operator in $L: L^2(\rho_x) \to L^2(\rho_x)$ corresponds to $r = s + 1/2$, where $s$ is the parameter of the source condition expressed with respect to the covariance operator $T: \mathcal{H} \to \mathcal{H}$. This is explained in detail in [d] (Eq. 9, and 10). Tikhonov saturates at $1$ in the formulation with the integral operator $L$, which then corresponds to $1/2$ in the formulation with the covariance operator $T$, that is the one we use here in the paper.
In particular, in this paper, by taking just one single step of Iterated Tikhonov, we recover the saturation of Tikhonov in Thm. 1, which was unveiled by [e].
We will add the above discussion after the introduction of the source condition in Section 2.3, to better clarify the context to the reader.
### References
[a] J. Mairal. Cyanure: An open-source toolbox for empirical risk minimization for Python, C++, and soon more, 2019.
[b] G. Meanti, L. Carratino, L. Rosasco, and A. Rudi. Kernel methods through the roof: handling billions of points efficiently. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[c] U. Marteau-Ferey, F. Bach, and A. Rudi. Globally convergent newton methods for ill-conditioned generalized self-concordant losses. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[d] F. Bauer, S. Pereverzev, and L. Rosasco. On regularization algorithms in learning theory. Journal of complexity, 23(1):52–72, 2007.
[e] U. Marteau-Ferey, D. Ostrovskii, F. Bach, and A. Rudi. Beyond least-squares: Fast rates for regularized empirical risk minimization through self-concordance. In Conference on Learning Theory (COLT), 2019.
Dear authors,
There is still some confusion about Q3.
While it is clear that source condition could be defined with respect to the "identify" matrix or the "covariance" matrix, I am not sure how this fact is related to the qualification for Tikhonov filter. By comparing Definition 4 in this paper to Eq. 16 of Definition 1 in [e], I see no essential difference except for some boundedness requirement. Could you confirm that line 153 is correct that the qualification for Tikhonov filter is $1/2$, not $1$? If this is indeed a typo, could you elaborate how it affects the results of the paper?
Dear reviewer,
Thanks for allowing some more discussion on the topic and posing again Question 3.
Indeed we got it wrong the first time: you ask about the **qualification of Tikhonov**, while we talked about the **saturation effect**.
* The qualification is specific to spectral filters, and obtained by solving Eq. (7).
* The saturation effect, instead, depends on the notation that we use for the parameters of the source condition. It is the maximum rate at which decays the bias in the bound on the excess risk.
* Spectral filter theory establishes optimal rates with the square loss for a spectral filter with qualification $q$ and a source condition of parameter $r$ expressed with the covariance operator.
That is, assuming $f = T^r v$ for some $v \in \mathcal{H}$. In this case, the excess risk decays as $n^{-\frac{2s \alpha}{2s \alpha +1}}$, with $s = \min\\{r+1/2, q\\}$ and $\alpha$ is the parameter of the capacity condition. **This shows that the saturation effect of a spectral method starts when $r + 1/2 > q$**. This is derived, for example, in Theorem 3.4 in [15] (to be read with interpolation norm parameter $s = 1/2$ corresponding to the convergence of the estimator in the $L^2$ norm i.e. the prediction error;
moreover the covariance operator is defined in Proposition 2.5 calling it $B_\nu$ and the source condition in Eq. (2.5)).
We agree with you, thanks for spotting the typo of line 153. Tikhonov has qualification of 1, and not 1/2 as incorrectly stated. This is immediate to see by solving Eq. (7) with $g_\lambda(\sigma) = (\sigma + \lambda)^{-1}$.
Also, Iterated Tikhonov with $t$ steps has qualification $t$. Qualification and saturation are different concepts and in the last stages of the submission, to reduce the length of the paper, we compressed the lines 150-153 too much, making them inaccurate. Fortunately, this error does not propagate in the proofs, but it is only a typo in the presentation of previous results for spectral filters.
The lines 150-153 must read as
"[...] an optimal rate in $n^{ - \frac{\alpha(1+2r)}{1+\alpha(1+2r)}}$ if $r + 1/2 \leq q$ (where r comes from the source condition).
When $r + 1/2 > q$, the rate is instead of order $n^{-\frac{2\alpha q}{1+2\alpha q}}$, which is suboptimal. See for example Theorem 3.4 in [15] in the case of the $L^2$ norm (set the parameter $s$ to $1/2$). This illustrates the saturation effect of some regularization schemes.
For example, Tikhonov regularization amounts to filtering with $g_\lambda(\sigma) =  (\sigma + \lambda)^{-1}$ and has qualification $1$, so the parameter $r$ {\em saturates} at $r = 1/2$.
Thus, even if $r \gg 1/2$, the excess risk of $\hat{\theta}_\lambda$ will decay in $n^{-\frac{2\alpha}{1+2\alpha}}$ , which is suboptimal."
To further confirm that there is no error, note indeed that the final learning rate is the correct one.
Indeed if we consider our algorithm in the case of Tikhonov, i.e. by running it with only one step of iterated Tikhonov, we achieve exactly the optimal learning rate for Tikhonov that is $$n^{-\frac{2\alpha \min(r+1/2,1)}{1 + 2\alpha \min(r+1/2,1)} }$$
The rate above, in the case of squared loss, matches [15] and is minimax optimal. In the case of GSC losses it matches [e] (see their Assumption 6 and Corollary 9, where the condition $r \leq 1/2$ is highlighted). Note that [e] also expresses the source condition with respect to the analog of the covariance operator in the case of GSC losses, as we also do.
Moreover, since they use Tikhonov, they experience saturation effect starting at $r = 1/2$ as we wanted to report originally in lines 150-153.
Thanks again for the opportunity of this interaction. We hope that this could help to clarify the question about Q3 and possibly to re-evaluate our manuscript positively.
In any case we are open to further discussion.
Best wishes,
the authors
Thanks for the detailed clarifications and confirm the typo, as well as confirm that the typo does not affect the correctness of the final results. I am happy to re-evaluate the paper.

Meta-review:
role: PC
data: comment: The topic of the submission is supervised learning in RKHSs (reproducing kernel Hilbert space) with generalized self-concordant (GSC; such as the logistic one) losses. Particularly, the authors raise the question if spectral filtering techniques can be leveraged to achieve faster/optimal convergence rates under source and capacity conditions in case of GSC losses when the regularization is changed from the classical Tikhonov one (as studied in [1] and suffering from saturation) to the iterative Tikhonov scheme (also known as proximal point algorithm). The authors answer this question affirmatively (Theorem 1) with extension to inexact proximal solvers (Proposition 2). The efficiency of the approach is illustrated on synthetic examples.

Kernel techniques are omnipresent in machine learning with a large number of successful applications. The reviewers agreed that the authors deliver a solid theoretical contribution in this direction in a well-written paper which can be of clear interest to the NeurIPS audience.

Decision: Accept (Spotlight)