Nonvacuous Loss Bounds with Fast Rates for Neural Networks via Conditional Information Measures

Abstract: We present a framework to derive bounds on the test loss of randomized learning algorithms for the case of bounded loss functions. This framework leads to bounds that depend on the conditional information density between the the output hypothesis and the choice of the training set, given a larger set of data samples from which the training set is formed. Furthermore, the bounds pertain to the average test loss as well as to its tail probability, both for the PAC-Bayesian and the single-draw settings. If the conditional information density is bounded uniformly in the size n of the training set, our bounds decay as 1/n, which is referred to as a fast rate. This is in contrast with the tail bounds involving conditional information measures available in the literature, which have a less benign 1/n dependence.  We demonstrate the usefulness of our tail bounds by showing that they lead to estimates of the test loss achievable with several neural network architectures trained on MNIST and Fashion-MNIST that match the state-of-the-art bounds available in the literature.

Reviews and author responses:

The paper tackles an important concern when it comes to PAC-Bayes bounds, i.e., the fact that such bounds apply to the risk of a stochastic predictor rather than a deterministic one. The authors propose bounds on a "single draw" of a predictor according to the PAC-Bayesian predictor. 

However, I would like the author to discuss to which extent the derivation of this "single draw'' bound differs from the result of Hellstrõm & Durisi (2020) they refer to. Indeed, the latter is a "slow rate" $O(1/\sqrt{n)}$ bound, while the proposed result is a "fast rate" $O(1/n)$ one. However, such bounds for randomized predictors already exist in the PAC-Bayes literature for randomized predictors. As a matter of fact, the PAC-Bayes theorem of Seeger (2002), involving the KL divergence between two Bernoulli distributions, is tighter than both the slow rate result of Eq. (1) and the fast rate result of Eq. (2); the Seeger's bound achieving fast rate when the empirical loss is zero (see Letarte et al., 2019 (Thm 3), for an explicit connection between Eq (2) and Seeger's bound). 

In other words, I wonder if the proposed "fast rate" bound is a particular case of a general analysis obtainable from a slight generalization of Hellstrõm & Durisi (2020).

The experiments show promising results but would benefit by being extended. Figure 1 shows the bound values according to the training epoch up to 30 epochs. The training and test loss are still decreasing at this point. In order to have the complete picture, it would be important to compare the results for fully trained models. Moreover, I would like to see the bounds values in the overfitting regime, when the training error is close to zero, as the experiments are performed in Dziugate & Roy (2017). Note that the proposed fast rate bounds only converge in this setting.

Finally, a nice addition would be to comment on the possibly to learn a predictor by directly minimizing a single draw bound, as it is frequently done in the PAC-Bayes works (e.g., Dziugate & Roy (2017). 

### Minor comments:
- I do not understand the subscript $i + S_i n$ of $\tilde Z$ (middle of page 3)
- It is strange that the authors cite an unpublished paper (Guedj and Pujol 2019) for the canonical PAC-Bayes bound stated in the early PAC-Bayes works of McAllester.

### References:
Letarte et al., Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networks. NeurIPS 2019
Seeger, PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification. JMLR 2002


We thank the reviewer for your continuing constructive feedback.

The manner in which we presented the numerical results in our initial submission was not chosen with deceptive intent. Indeed, for the optimization procedure that we originally used (SGD without momentum and a decaying learning rate), the metrics that we plot do not change much between 30 and 50 epochs (which is the highest number of epochs we initially studied). To get an idea of how Figures 1a--d would look if plotted for more epochs, we ran experiments for LeNet-5 on MNIST with SGD without momentum and with a decaying learning rate. The data from one run is shown in the following link:

https://ibb.co/vDjBT86

In particular, after 200 epochs, we obtained the following results when we used a threshold of 0.05 to select $\sigma_1$ and $\sigma_2$:

***
Deterministic NN training loss: 	0.026  
Stochastic NN training loss:		0.043  
Fast-rate bound on test loss:		0.179  
Slow-rate bound on test loss:		0.304
***	

Thus, the drastic performance deterioration as the number of training epochs increases shown in Figures 1e--f does not occur to the same degree when momentum is not used as in Figures 1a--d. For the fast-rate bound, no significant deterioration can be seen.
The most likely reason is the following: the deterministic NNs that are found with the training procedure used in Figures 1a--d (SGD with decaying learning rate and no momentum) are significantly weaker than the deterministic NNs we find when using momentum.

In response to your initial feedback on our experiments, we then studied networks trained with SGD with momentum for many more epochs, in order to be able to achieve smaller training losses (also for randomized labels). This is when we observed the phenomenon reported in Figures 1e--f. We thank you for encouraging us to perform these experiments, as this has helped to shed light on the limitations of our bounding techniques, and, more generally, on the limitations of the PAC-Bayesian bounds that are currently available.

We have updated the discussion in the experiments and conclusion sections: we now highlight more prominently the limitations of the bounds that we present. 
Specifically, we state explicitly that our bounds are loose for NNs that are trained so as to obtain a very small training error; furthermore, as shown in the experiments described in Appendix D, they overestimate the number of training samples needed to achieve good generalization performance, and are vacuous when randomized labels are considered. 
In the final version of the paper, we intend to extend Figures 1a--d to 200 epochs to illustrate what happens when training progresses for the case in which we use SGD without momentum with a decaying learning rate. We will also replicate Figures 1e--f for the case of Fashion-MNIST. Preliminary results indicate that the behavior is similar to Figures 1e--f, but that, as expected, the bounds are much weaker for Fashion-MNIST. These results can be found at the following links. The plots in the two different links are identical apart from the range of the y-axis.

https://ibb.co/6w3kTLh

https://ibb.co/ByBMw2p

Thank you for your response on this issue. You are correct in pointing out that we should give Catoni more recognition for the result in (2). We have done so by adjusting the text in the manuscript to read as follows:

***
By adapting a proof technique introduced by Catoni (2007, Thm. 1.2.6), McAllester (2013, Eq. (21)) derived the following alternative bound:

$$\dots$$

This bound, with $d_\gamma(q||p)$ replaced by $d(q||p)$, slightly improves the dependence on the sample size $n$ as compared to an earlier bound reported in (Seeger, 2002, Thm. 1), but again at the cost of losing uniformity over posteriors.
***

The point you make about uniformity in the PAC-Bayesian bounds of McAllester is true, and an important one to note. We have added the following sentence to the paper to highlight this.

***
The version of the bound given in (1) slightly improves the dependence on the sample size $n$ as compared to the bound reported in (McAllester, 2003, Thm. 1), at the cost of not holding uniformly over all posterior distributions. 
***

I had a closer look to  Guedj and Pujol (2019, Prop 3) , and it seems that the result doesn't contain the dependence on the sample size $n$ inside the $\log(1/\delta)$ term because it holds for one posterior $Q$ with prob. $\geq 1-\delta$. Indeed, in the proof of Guedj and Pujol, the $\lambda$ term is chosen in terms of the posterior. In contrast, most seminal PAC-Bayes bounds of  McAllester  hold with prob. $\geq 1-\delta$ *simultaneously for all distributions $Q$*. This detail is worth mentioning in the paper.

The new experiments are welcomed. They provide a more honest and complete picture of the proposed bounds behavior. It is not completely surprising that the bounds increase when the error becomes very low, as it has been observed in other works deriving similar guarantees.

That being said, it is somewhat unpleasant that these deceptive results were hidden in the initial submission; it is unlikely that the authors did not observe such behavior during their experiments. Note that such deceptive results should not lead to paper rejection provided that the observed behaviors are properly analyzed. The community needs these rigorous analysis in order to progress towards more accurate generalization guarantees! 

I encourage the authors to replicate Figure 1 (e-f) experiments on the Fashion-MNIST dataset, and to clearly state the limitations of their bounds.


I acknowledge this addition to the paper, which helps to situate the results. However, Catoni (2007) deserves more credit for above Eq (2). Indeed, it is explicitly written McAllester (2013) that this result is adapted from Catoni (2007). In comparison, Seeger (2002, Thm 1) seems remotely connected to above  Eq (2).

We have now uploaded a revised version of our manuscript, which includes the corrections we have listed below and the promised additional numerical results.
All modified parts are marked in blue for a better visualization.

Specifically, in Figure 1e--f, we illustrate the behavior of the bounds for larger training epochs. In agreement with what previously reported in the literature, the bounds become progressively looser as the number of training epochs increases. This negative behavior of our bounds can be mitigated by an appropriate choice of the variances $\sigma_1$ and $\sigma_2$.

In Figure 2, we plot the bounds as a function of the size of the training set $n$. As expected, the fast-rate bound is superior to the slow-rate bound for MNIST, while the two bounds exhibit similar performance for Fashion-MNIST.

Finally, in Table 2, we report results for the scenario in which a fraction of the labels in a binarized version of MNIST is randomly flipped.
Although the bounds are vacuous (as all bounds we are aware of for this scenario), they correctly capture the expected trend: the test loss increases as the fraction of randomly flipped labels increases.

The results reported in Fig. 1f and Fig. 2 are preliminary: we are currently performing additional simulations to reduce the size of the confidence interval. Note that the order of the figures in the revised version of the paper, as well as the specific cutoff values used when examining the dependence of our bounds on the training set size, differs from what we stated in the comments below.

We thank the reviewer for their positive words and insightful comments.

### Comments regarding novelty of contributions and relation to previous literature
The contributions of this paper are concerned with the *random-subset setting* introduced in (Steinke \& Zakynthinou, 2020), wherein the $n$ training samples in $\boldsymbol{Z}(\boldsymbol{S})$ are chosen from a set $\widetilde{\boldsymbol{Z}}$ of $2n$ samples through a random variable $\boldsymbol{S}$. In our literature survey, we also review bounds obtained in the *standard setting*, where no such structure is assumed, and we simply have $n$ training samples $\boldsymbol{Z}$. As the reviewer correctly pointed out, the fast-rate PAC-Bayesian bound for the standard setting that we refer to in (4) below (Eq. (2) in the original submission) is not the tightest available bound. As shown in (Letarte et al., 2019), the bound in (Seeger, 2002), which uses the KL divergence between two Bernoulli distributions, is generally tighter. 
A slight improvement of this bound can be found in (Eq. (21), McAllester, 2013). By relaxing the bound in (Eq. (21), McAllester, 2013), McAllester derives the fast-rate bound that we report in (4) below (Eq. 2 in the original submission). 
In the revised version of the paper, we have expanded the discussion around fast-rate PAC-Bayesian bounds in the standard setting to accommodate for these observations, and clarified what the tightest known PAC-Bayesian bound is for the standard setting. The relevant excerpt from the revised version of the paper is given below.
* * * 
In the vein of (Catoni, 2007, Thm. 1.2.6) and (Seeger, 2002, Thm. 1), McAllester derived the following bound (McAllester, 2013, Eq. (21)):
for all $\gamma\in\mathbb{R}$, and with probability at least $1-\delta$ under $P_{\boldsymbol{Z}}$, 

$$ d_\gamma ( \mathbb{E}\_\{P_\{W\vert Z\}\} [L_Z (W)]|| \mathbb{E}\_\{P_\{W\vert Z\}\} [L_\{P_Z\} (W)])\leq \frac{1}{n}\left( D(P_\{W\vert\boldsymbol{Z}\}||Q_W) + \log \frac{1}{\delta} \right).\qquad (2)$$

Here, $d_\gamma(q ||p )=\gamma q -\log(1-p+p e^\gamma)$, and one can show that $\sup_\gamma d_\gamma(q ||p )=d(q ||p )$, where $d(q||p )$ indicates the KL divergence between two Bernoulli distributions with parameters $p$ and $q$ respectively. Let $q=\mathbb{E} \_{P\_{W\vert \boldsymbol{Z}} }[ L_Z(W) ] $ and let $c$ denote the right-hand side of (2). To use this inequality to bound the population loss, we need to find 
$$ p^*(q,c)=\sup\\{p: p\in[0,1], d(q ||p)\leq c \\} .\qquad(3)$$
This is the largest population loss that satisfies the inequality (2). For small $q$ and $c$, we have $p^*(q,c)\approx c$, which gives us a fast-rate bound. More generally, for any permissible values of $q$ and $c$, the bound in (2) can be weakened to obtain the fast-rate bound reported in (McAllester, 2013, Thm. 2): for all $\lambda\in (0,1)$, and with probability at least $1-\delta$ under $P_{\boldsymbol{Z}}$
$$
\mathbb{E}\_\{P_\{W\vert Z\}\} [ L_\{P_Z\} (W)]\leq\frac{1}{\lambda}\left[\mathbb{E}\_\{P_\{W\vert Z\}\} [L_Z(W)] +\frac{D(P_{W\vert \boldsymbol{Z}}||Q_W)+\log\frac{1}{\delta}}{2(1-\lambda)n} \right].\qquad(4)
$$
Note that the faster decay in $n$ of this bound comes at the price of a multiplication of the training loss and the KL term by a constant that is larger than $1$.
As a consequence, if the training loss or the KL term are large, this multiplicative constant may make the fast-rate bound quantitatively worse than the slow-rate bound for a fixed $n$. Additional insights on the tightness of these bounds are provided in (Letarte et al., 2019, Thm. 3).

* * *

We thank the reviewers for their constructive comments. We have tried to address all of them, and we believe that this has improved the paper substantially.

Below, we clarify how each of the comments will be addressed in the revised version of the paper, which we are currently finalizing. The revision will include the new numerical results requested by the reviewers. The figures describing the outcome of these numerical results are currently under preparation and will be uploaded together with a revised version of the manuscript within the next days. 

### Comments regarding numerical results
In response to these comments, we are currently extending our numerical results. For all experiments, in order to achieve smaller training errors, we changed our optimization procedure to SGD with momentum. We are running experiments for more epochs than what was presented in the original version of this paper. In a new section in the appendices, we will add a plot to show how the bounds behave for these later epochs. Since the training error achieved towards the later epochs is close to zero, this also illustrates their behavior in the overfitting regime that you refer to.
Furthermore, we will update the figure in the main body of the paper to show the dependence of the bounds on the sample size $n$ rather than the number of epochs. For a given $n$, we will show results for networks that are trained until a certain threshold is reached. In line with the results in (Dziugaite et al., 2020, Fig. 4), we will use a threshold based on a training error of $0.03$ for MNIST and $0.125$ for Fashion-MNIST.
Finally, in order to show a situation where the training and test errors differ significantly, we are running experiments with partially corrupted labels. Specifically, we consider the binarized version of MNIST where the digits $0,\dots,4$ are combined into one class and $5,\dots,9$ into another, and we set the labels of a fixed proportion of both the training and test sets randomly. All of these results will be included in a revised version of the paper that we intend to upload in the coming days.

Unfortunately, predictors cannot be selected by directly minimizing the bounds we present, since, for the bounds to hold, $W$ has to be conditionally independent of $\widetilde{\boldsymbol{Z}}$ and $\boldsymbol{S}$ given $\boldsymbol{Z}(\boldsymbol{S})$. In other words, the Markov chain $(\widetilde{\boldsymbol{Z}}, \boldsymbol{S})-\boldsymbol{Z}(\boldsymbol{S})-W$ has to hold. Since the bounds depend on $(\widetilde{\boldsymbol{Z}}, \boldsymbol{S})$ through more than just $\boldsymbol{Z}(\boldsymbol{S})$, computing $W$ by directly minimizing the bounds would result in a violation of this Markov property. 
### Minor comments
1. Since $S_i$ is uniformly distributed on $\{0,1\}$, the subscript $i+S_i n$ will take the value $i$ with probability $1/2$ and the value $i+n$ with probability $1/2$. We added the following sentence to the paper to clarify this point.
* * *
Thus, the binary variable $S_i$ determines whether the training set $\boldsymbol{Z}(\boldsymbol{S})$ will contain the sample $\tilde Z_i$ or the sample $\tilde{Z}_{i+n}$.
* * *
2. Regarding the PAC-Bayes bound in (1) (Eq. 1 both in the original and revised versions of the paper), this version of the bound cannot be found in McAllester (1998), nor, to the best of our knowledge, in any other work by McAllester. The closest bound is the one reported in (Thm. 1, McAllester, 2003), where the dependence on the sample size $n$ is, however, less favorable than in (1). Indeed, we were unable to find any published paper which gives the bound in the specific form given in (1). However, since this is a minor improvement on a previous bound obtained by McAllester, we modified the text to reflect this point by adding the following sentence after (1). 
***
The version of the bound given in (1) slightly improves the dependence on the sample size $n$ as compared to the bound reported in (Thm. 1, McAllester, 2003).
***
### References
- G. Letarte, P. Germain, B. Guedj, and F. Laviolette. Dichotomize and generalize: PAC-Bayesian binary activated deep neural networks. Dec 2019
- D.A. McAllester. A PAC-Bayesian tutorial with a dropout bound. July 2013.
- D.A. McAllester. PAC-Bayesian Stochastic Model Selection. April 2003.

### Comments regarding novelty of contributions and relation to previous literature (continued)
As far as the contributions of the current paper are concerned, on the theoretical side, we provide new fast-rate bounds for the PAC-Bayesian and single-draw scenarios for the random-subset setting introduced in (Steinke \& Zakynthinou, 2020). As discussed above, such fast-rate bounds were previously available for the standard setting (Seeger 2002, Catoni 2007, McAllester 2013), but not for the random-subset setting. 

Differently from the standard setting, it seems difficult to further tighten the bounds for the random-subset setting derived in our paper by obtaining bounds similar to (McAllester, 2013, Eq. (21)). Indeed, unlike in the standard setting, in the random-subset setting, the training superset $\widetilde{\boldsymbol{Z}}$ and the output hypothesis $W$ are dependent *both before and after* the change of measure argument. This means that the exponential inequality used in (McAllester, 2013, Eq. (17)) and (Seeger, 2002, Eq. (22)) to prove (McAllester, 2013, Eq. (21)) cannot be used in the random-subset setting.

We would also like to point out that the fast-rate bounds reported in the present paper are not a trivial extension of the slow-rate bounds derived in (Hellström \& Durisi, 2020b) for the random-subset setting. Indeed, since the proof of (Thm. 4, Hellström \& Durisi, 2020b) relies on a sub-Gaussianity argument, the slow rate is unavoidable. This is because sub-Gaussanity results in a quadratic term involving the parameter $\lambda$ within the exponential inequality reported in (Hellström \& Durisi, 2020b). When this parameter is optimized, the resulting exponential inequality depends on the square of the generalization gap. This yields the slow-rate dependence. 

In contrast, in Theorem 1 of the submitted paper, the population loss only appears linearly in the exponential inequality when the parameters $\lambda$ and $\gamma$ are suitably chosen. This linearity is what allows us to derive fast-rate bounds from this new exponential inequality. We clarified this point by adding the following text after Theorem 1. Note that (10) refers to the exponential inequality in Thm. 1 (Eq. (8) in the original submission).
* * *
Note that the exponential function in (10) depends linearly on the population loss. In contrast, the exponential inequality derived in (Hellström & Durisi, 2020b, Thm. 4) to establish slow-rate generalization bounds for the random-subset setting depends quadratically on the population loss (after the parameter $\lambda$ therein is suitably optimized). This difference explains why Theorem 1 allows for the derivation of fast-rate bounds, whereas (Hellström & Durisi, 2020b, Thm. 4) unavoidably leads to slow-rate bounds. Also note that, since in the random-subset setting $W$ and $\widetilde{\boldsymbol{Z}}$ are dependent both before and after any change of measure argument, the proof techniques used in (McAllester, 2013, App. A) and, previously in (Seeger, 2002, Thm. 1), to derive (McAllester, 2013, Eq. (21)) cannot be used in the random-subset setting.
* * * 

We thank the reviewer for the positive feedback and relevant comments.

1. The main contribution, in terms of the proof technique, is that we show how to combine two recent developments to harness the benefits from each. Specifically, we combine the fast-rate derivation of (Steinke \& Zakynthinou, 2020), which itself builds upon (McAllester, 2013), with the exponential-inequality approach of (Hellström \& Durisi, 2020b), which allows one to derive bounds on the average, PAC-Bayes, and single-draw loss. We added the following sentence after Theorem 1 to better highlight these aspects.

* * *

We now present an exponential inequality from which several test loss bounds can be derived, in a similar manner as was done in (Hellström & Durisi, 2020b). The derivation, which echos part of the proof of (Steinke & Zakynthinou, 2020, Thm. 2.(3)), is provided in Appendix A.2.  This result and its proof illustrate how to combine the exponential-inequality approach from (Hellström & Durisi, 2020b) with fast-rate derivations, like those employed in (Steinke & Zakynthinou, 2020, Thm. 2.(3)) and (McAllester, 2013, Thm. 2) for the standard setting.

* * *

2.  When constructing the prior, we build 10 subsets from the $2n$ elements of $\widetilde{\boldsymbol{Z}}$, each of size $n$, and train a neural network on each of these. Thus, we get 10 different parameter vectors $\boldsymbol U_1,\dots,\boldsymbol{U}_{10}$, each containing a list of the weights of the trained networks. To compute $\boldsymbol\mu_2$, we average over these vectors: $\boldsymbol\mu_2 = \frac{1}{10} (\boldsymbol U_1+\dots+\boldsymbol{U}_{10})$. Thus, $\boldsymbol{\mu}_2$ is a vector, the size of which equals the number of parameters in the network, and when performing the averaging, we only sum weights that correspond to the same unit in the network architecture. Thus, this method of computing $\boldsymbol{\mu}_2$ respects the structure of the network, including its hierarchical nature. The parameter $\boldsymbol{\mu}_2$ is typically not close to the zero vector.

3. In response to these comments, we are currently extending our numerical results. For all experiments, in order to achieve smaller training errors, we changed our optimization procedure to SGD with momentum. We will update the figure in the main body of the paper to show the dependence of the bounds on the sample size $n$ rather than the number of epochs. For a given $n$, we will show results for networks that are trained until a certain threshold is reached. In line with the results in (Dziugaite et al., 2020, Fig. 4), we will use a threshold based on a training error of $0.03$ for MNIST and $0.125$ for Fashion-MNIST. Furthermore, in order to show a situation where the training and test errors differ significantly, we are running experiments with partially corrupted labels. Specifically, in order to make training easier, we consider a binarized version of MNIST where the digits $0,\dots,4$ are combined into one class and $5,\dots,9$ into another. We then set the labels of a fixed proportion of both the training and test sets randomly. In a new section in the appendices, we will add a plot to show how the bounds behave for different proportions of randomized labels. Finally, we are running experiments for more epochs than what was presented in the original version of this paper. In the aforementioned new appendix, we will also add a plot of this to illustrate the bounds for more fully trained networks, where the training error is close to zero.
All of these results will be included in a revised version of the paper that we intend to upload in the coming days.

We thank the reviewer for these kind words. The approach with conditional information measures indicates that a good prior $Q_{W\vert \widetilde{\boldsymbol{Z}}}$ can be obtained by marginalizing out the subset choice variable $\boldsymbol{S}$. However, for neural networks, this is much too computationally expensive, since it would involve training the network $2^n$ times. Still, this observation leads to a principled approach to choose a good prior, by averaging over a couple of instances of $\boldsymbol{S}$. We altered the text in the conclusion section to highlight this point. Unfortunately, predictors cannot be selected by directly minimizing the bounds we present, since, for the bounds to hold, $W$ has to be conditionally independent of $\widetilde{\boldsymbol{Z}}$ and $\boldsymbol{S}$ given $\boldsymbol{Z}(\boldsymbol{S})$. In other words, the Markov chain $(\widetilde{\boldsymbol{Z}}, \boldsymbol{S})-\boldsymbol{Z}(\boldsymbol{S})-W$ has to hold. Since the bounds depend on $(\widetilde{\boldsymbol{Z}}, \boldsymbol{S})$ through more than just $\boldsymbol{Z}(\boldsymbol{S})$, computing $W$ by directly minimizing the bounds would result in a violation of this Markov property. The modified text reads as follows:

* * *
In particular, the random-subset setting provides a natural way to select data-dependent priors, namely by marginalizing the learning algorithm $P_{W\vert\widetilde{\boldsymbol{Z}}\boldsymbol{S}}$ over $\boldsymbol{S}$, either exactly or approximately. Such data-dependent priors are a key element in obtaining tight information-theoretic generalization bounds (Dziugaite et al., 2020).

* * *

This paper extends results of prior work by Steinke and Zakynthinou, by providing generalization bounds in the PAC-Bayesian and single-draw settings that depend on the conditional mutual information. The emphasis in this work is on obtaining fast rates ($1/n$ vs. $1/\sqrt{n}$). The authors also conduct empirical experiments showing how the fast rate bounds they propose can be useful for obtaining non-vacuous generalization bounds in the context of over-parameterized neural networks.

I think Theorem 1 and its corollaries are a nice contribution. The paper is very well written and clear. The authors do an excellent job in explaining the relevant related work, and how their results expand upon the results of earlier work. 

It might be useful if the authors could indicate or explain whether their bounds suggest or motivate improved learning algorithms. For example, how should one choose a prior distribution Q over the hypothesis based on knowledge of the full training examples?



This paper derives bounds on the test loss under the random-subset setting, which are expressed with conditional information measures, and have fast rates with respect to the sample size n. The derived bounds are compared with the practical performance of deep neural networks (DNNs) trained on the MNIST and Fashion-MNIST data sets.

Although the derived bounds can be somewhat loose practically as it can be larger than those of slow-rates, the paper provides a steady theoretical progress on the evaluation of loss bounds using conditional information measures.

Although the experiments with NNs and practical data sets are interesting, they are a little unsatisfactory for demonstrating the significance of the derived fast-rate bounds.

p.5, main theorem: It would be nicer to explain what novel important proof techniques are required, if any, to derive the fast-rate bounds in Section 3.

p.6, computation of \mu_2: Since NNs have hierarchical structures, computing the average of their weights does not seem so good. I wonder if computed \mu_2 are close to zero and if there is any other possible ways to compute \mu_2.

p.7, experiments on MNISTs: Although the experiments demonstrate that the fast-rate bound can in fact be tighter than the slow-rate bound in practical cases, I wonder if some experiments should be designed with changing the sample size n in order to compare fast and slow rate bounds. 
As the training and test errors reported in Fig. 1 are close to each other, isn’t it better to produce any overfitting situation, even for synthetic data?


Meta-review: The paper extends results from the recent work of Steinke and Zakynthinou (SZ) for the test loss of randomized learning algorithms. They provide bounds in the single draw as well as PAC-Bayes setting. The main result is about fast rates the proof of which follows with minor modifications from the corresponding result in SZ. It is unclear to me the contribution over existing work is sufficient to merit acceptance.

Decision: withdrawn-rejected-submissions