Title: Stability and Deviation Optimal Risk Bounds with Convergence Rate $O(1/n)$
Authors: Yegor Klochkov
Nikita Zhivotovskiy
Venue: NeurIPS 2021
Keywords: algorithmic stability
generalization bounds
concentration inequalities
stochastic convex optimization

Abstract: The sharpest known high probability generalization bounds for uniformly stable algorithms (Feldman, Vondrak, NeurIPS 2018, COLT, 2019), (Bousquet, Klochkov, Zhivotovskiy, COLT, 2020) contain a generally inevitable sampling error term of order $\Theta(1/\sqrt{n})$. When applied to excess risk bounds, this leads to suboptimal results in several standard stochastic convex optimization problems. We show that if the so-called Bernstein condition is satisfied, the term $\Theta(1/\sqrt{n})$ can be avoided, and high probability excess risk bounds of order up to $O(1/n)$ are possible via uniform stability. Using this result, we show a high probability excess risk bound with the rate $O(\log n/n)$ for strongly convex and Lipschitz losses valid for \emph{any} empirical risk minimization method. This resolves a question of Shalev-Shwartz, Shamir, Srebro, and Sridharan (COLT, 2009). We discuss how $O(\log n/n)$ high probability excess risk bounds are possible for projected gradient descent in the case of strongly convex and Lipschitz losses without the usual smoothness assumption.

TL;DR: The first high-probability deviation risk bounds for uniformly stable algorithms with nearly optimal rate $O(\log n/ n)$

Review discussion:
1. To achieve the main result, a useful Lemma 3.1 is shown (this is a minor variation of a Lemma from BKZ): this result avoids the sampling error term $O(1/\sqrt{n})$ by showing a moment bound of a "shifted" generalization error. This shifting includes the risk of the algorithm output and an average conditional loss. It would be interesting if the authors would elaborate further about the intuition on this refinement, and what are its consequences for the excess risk bounds (the technical proofs are clear, but
nevertheless it might be useful a brief discussion).
2. The proof of the main result (Thm 1.1) is obtained nontrivially from Lemma 3.1. This involves a (seemingly novel) decomposition of the excess risk and the (shifted) generalization error, along with various estimates based on concentration by the Bernstein condition.
3. Theorem 1.2 provides what they call a variance-type bound (not sure why is it called like this), which bounds the risk in terms of the empirical risk and a deviation bound which leverages faster rates by uniform stability. This bound seems useful in realizable settings, where the empirical risk can be made arbitrarily small.
4. The consequences for strongly convex stochastic optimization appear in Proposition 2. 1 and its following discussion. As mentioned above, this general bound for stable algorithms answers a question raised in Shalev-Shwartz et al (2009) which is quite interesting on its own. In this result, the most problematic term is $\sqrt{L^2\overline{\Delta}/\lambda}$, where $\overline{\Delta}$ is a bound on the empirical error: this leads to optimal excess risk in this case for GD with O(n^2) iterations. It is worth pointing out here that a similar iteration requirement appears in the nonsmooth case, as shown in Bassily, Feldman Guzman and Talwar (2020).
That paper also includes a lower bound on stability, showing that for GD $\Omega(n^2)$ steps needed to obtain optimal rates with uniform stability. It is reasonable to conjecture that a similar lower bound may hold in the strongly convex setting, and in fact some limitations of GD in terms of excess risk have already been shown in recent work by Amir, Koren and Livni (https://arxiv. org/pdf/2102. 01117. pdf). The latter result is shown for GD on the regularized ERM, but I suspect ideas there might be useful for the strongly convex setting as well.
Finally, I would suggest the authors to include a more detailed statement of the last paragraph before section 3 (lines 204-206 in page 6). Here it is claimed a new high-probability excess risk bound for GD on smooth and strongly convex losses. For future use of this result, it might be a good idea to have a proper statement.
No
"Deviation optimal" refers mainly to the bound from Proposition 2.1, although rather ambitiously we ignore the logarithmic term. The lower bound $\Omega(\log(1/\delta)/n)$ is a standard excess risk lower bound for strongly convex losses (one may consider $d = 1, X = 1$ a.s., then $(y - w)^2$ is strongly convex with respect to $w$ whenever $Y$ is equal to $\pm L$ with probability $1/2$ respectively. In this case, ERM is a sample mean estimator and its excess risk is $\Omega(\log(1/\delta)/n)$. We will add a comment and a reference on this. Lemma 3.1 as a probabilistic statement that is also sharp up to a logarithmic factor as the lower bound of Proposition 9 in (BKZ) shows.
We call it the variance-type bound because of the equivalent form presented in equation (12). The second moment of the risk is bounded there by the term $MR(w_n)$. This notion is quite standard for the bounds based on the Bernstein inequality.
Thanks a lot, we were not aware of the recent work of Amir, Koren and Livni (2021) and some of the details of Bassily, Feldman, Guzman and Talwar (2020). We will add this discussion and references in the updated text. It is indeed very plausible that $\Omega(n^2)$ steps are needed in the strongly convex, non-smooth case.
Thank you for this suggestion, we will expand the discussion of the strongly convex smooth case in the final version.
Thank you for the clarifications.
Strengths:
+ The deviation analysis of uniformly stable algorithms as focused in this paper is an important and timely topic in learning theory.
+ I am particularly impressed with the strength and originality of the results obtained in this paper for strongly convex optimization problems. The main idea of proof is to make a smart use of an extended bounded difference inequality from Bousquet, et al. (2020) to separate the sampling term from the generalization error, so that the randomness of model and data sample can be decoupled in a way that the Bernstain inequality is applicable to get tighter bounds.
While built largely upon the previous results, IMO the technical part still appears novel enough in the considered problem regime.
+ The paper is generally well organized and clearly presented, with all steps in the proofs very carefully explained.
Weakness:
My main concern is about the benefit of main results in Theorem 1.1 when substantialized to ERM and PGD with arbitrary convex losses. Basically, the improved $O(\frac{\log(n)}{n})$ rate of convergence seems only possible for strongly convex loss functions with zero optimization error achievable over training data. For generic convex losses which are of more interest in practice, we usually need to deal with regularized ERM of which the empirical risk error $\Delta_{opt}$ is typically $O(\frac{1}{\sqrt{n}})$ due to a trade-off between optimization and generalization. In this case, while the population risk can still be assumed strongly convex so that Theorem 1.1 is applicable, the $O(\frac{1}{\sqrt{n}})$ optimization error term will dominate the risk bound which makes the  $O(\frac{1}{n})$ term much less interesting. The authors have partially  addressed such a limitation for Theorem 1.2 with respect to the empirical risk.
However, it looks like that similar or even more severe limitations exist for Theorem 1.1 as well in terms of optimization error.
=== Post rebuttal update ===
The major concern  on the limitation of results raised in my initial review has been properly addressed in author response. I would like to maintain my rating.
No
Thank you for the detailed review. General convex losses are indeed of greater practical interest.
However, we respectfully disagree that the fact that our improvements are not very useful in this setup is the weakness of our work.
In particular, there are fundamental limitations for achieving the rates faster than $O(1/\sqrt n)$ for general convex Lipschitz losses (for some high probability bounds achieving this rate, see the work of Feldman and Vondrak, 2019), whereas we focus on getting $O(1/n)$ high probability bounds whenever it is theoretically possible. We also remark that for strongly convex losses the improved convergence rates work if the optimization error is small but not necessarily zero (see Section 2.3).
Finally, we briefly discuss possible applications in the non-strongly convex cases (see last paragraph of Section 2.1).
Thank you for the feedback. My concern on the limitation of results is completely clarified. I would maintain my rating and provide an additional comment on the possibility of removing the $\log(n)$ factor in the risk bound for $\lambda$-strongly convex ERM. I note that this seems possible if the loss function is additionally assumed to be smooth. In this case, the $\gamma$-stability of $w_n$ implies that of $\|\|w_{n} – w^* \|\|$.
Then we have
$E[\|\|w_n – w^* \|\|] \le \sqrt{E[\|\|w_n – w^* \|\|^2]} \le \sqrt{E\left[\frac{2(R(w^n) – R(w^*))}{\lambda}\right]} \le O(\sqrt{\frac{\gamma}{\lambda}})$
Applying McDiamid’s inequality yields that $\|\|w_n – w^* \|\| \le O\left(\sqrt{\frac{\gamma}{\lambda}} + \gamma\sqrt{n\log(1/\delta)}\right)$ holds with probability at least $1-\delta$. The smoothness the leads to $R(w_n) – R(w^*) \le O(\|\|w_n – w^*\|\|^2) \le O(\frac{\gamma}{\lambda} + \gamma^2 n\log(1/\delta)) \le O(\frac{\log(1/\delta)}{\lambda^2 n} ) $, keeping in mind that $\gamma\le O(\frac{1}{\lambda n})$.
I’m wondering if the $\log(n)$ factor can also be improved for Lipschitz losses, at least for strongly convex ERM when $\gamma$ scales as fast as  $O(1/n)$?
The question of log factor is very thrilling but so far the technique we use does not allow us to get rid of it.
Thank you for this interesting and short proof. It seems however that such an approach is limited to $\lambda$ of order one since the term $1/(\lambda^{2} n)$ can be worse than ours $1/ (\lambda n) $ when $\lambda$ is somewhere between $1/\sqrt{n}$ and $1$.
To my knowledge, this type of result was only proved in particular cases and this is the first general result on fast rates of convergence obtained via stability. The article is clear and well written and I think this is significant for the Neurips community. This article may generalize to a lot of learning problems and may become a tool to prove fast rates in  a lot of different settings, this may be an important step to assess the rate of convergence of algorithms and to have theoretical rates that match the rates that we witness in practice. The authors also illustrate this generalization capacity on the problem of stochastic convex optimization with strongly convex losses.
No
The paper uses tools developed by Bousquet et al. [8] heavily to prove their results. It is rather difficult to find a strongly convex function that is Lipschitz continuous and bounded. An easy way to justify these conditions together is to assume that \mathcal{X} is a compact set. The contribution is of good value from a theoretical point of view. As the authors pointed out, the generalized Bernstein assumption might be useful when considering optimization under PL-condition which is widely studied in recent years due to its usefulness for modern ML applications.
Overall, the paper makes a good contribution and might serve as a first step to thoroughly explore the new assumption in different settings.
No
Thank you, it is a good question about the relatively large $O(n^2)$ number of iterations in the non-smooth case. We believe that without the smoothness assumption not much can be improved.
In particular, very recently Amir, Koren and Livni (https://arxiv.org/pdf/2102.01117.pdf) considered the problem of convex Lipschitz optimization with a quadratic regularization and showed that in this case, GD requires $\Omega(n^2)$ steps (we thank the reviewer 4(1bRR) for pointing this out). In their Theorem 3.2 the term $\sqrt{\Delta / \lambda}$ appears in the lower bound.
Even though they consider a slightly different problem, this number of iterations might not be improvable in our case as well.
Additionally, see the discussions on strongly convex Lipschitz optimization problems in Section 2.
Thank you for the clarifications.
We thank all the referees for their appreciation of our work and their useful comments.

Meta-review:
role: PC
data: comment: The reviewers are unanimous that this work provides a significant contribution in the area of uniform stability, which should interest the learning theory community as well as the wider NeurIPS community. A key consequence of the authors' results is the positive resolution of a long-standing open question of Shalev-Shwartz et al.'s COLT 2009 paper: this question is about whether ERM can obtain fast rates of convergence for Lipschitz, stochastic strongly convex optimization with high probability, where the dependence on the failure probability $\delta$ is logarithmic. While the authors do heavily draw from previous work, certain technical components like the novel excess risk decomposition are highly original, provide an important technical contribution, and might lead to progress in future works. In addition, the work is very well written and clear. This work would be a welcome contribution to NeurIPS.

Decision: Accept (Oral)