Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper extends previous results on individual privacy accounting from Rényi DP to Gaussian DP. It also considers methods to maintain approximate privacy filters.
strength_and_weaknesses: ## Strengths
- The paper has adequate review of the relevant literature, is mostly self-contained, and clearly states the prior results it builds upon.
- It provides a proof for a natural (and perhaps not surprising) extension of the prior individual privacy accounting in the RDP case.

## Weaknesses
- The technical contribution seems incremental (proof of Theorem 11), but I will leave the assessment of the paper's novelty to the experts in this particular topic.
- Experiments should better highlight the practical use of individual GDP accounting compared to RDP. Figure C.3 shows one example for private GD, with a fixed number of steps. Perhaps showing similar plots under different settings (e.g. different number of steps) can help.
- Experiments in Section 6 merit further discussion. Besides the disparate impact problem, the experiments suggest that even in terms of overall AUC (Figure 4), very soon after filtering starts, the overall AUC begins to drop, which suggest a limited benefit from individual accounting. Did the authors compare to not using individual accounting?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clearly positions itself within the relevant literature. Code for the experiments is provided in the supplement.
summary_of_the_review: A nice extension of previous individual accounting results, from the Rényi DP to the Gaussian DP case.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work derives privacy accounting for Gaussian-DP under fully adaptive composition, which gives tighter privacy bounds than RDP analysis. Efficient numerical techniques are proposed to compute the privacy parameters in practice. The theoretical findings are verified by experiments.
strength_and_weaknesses: Strengths
- The proposed accountant method achieves tighter bounds for the Gaussian mechanism than the commonly used RDP accounting. 
- Efficient numerical calculation method is proposed.

Weakness
- The writing can be improved. Overall the paper looks very technical. The main results and their implications do not stand out well. 
- Technically this work mainly uses existing techniques from previous works. 
- More experiments can help to sufficiently demonstrate the effectiveness of the proposed PLD method 
- The disparate loss among subgroups for DP is observed in previous works and this part is somewhat orthogonal to the main contribution of this work.
clarity,_quality,_novelty_and_reproducibility: The paper is technically solid. However, the presentation can be improved to highlight the main results. The techniques used are mainly from existing works.
summary_of_the_review: This work derives tighter privacy accounting for Gaussian mechanisms with Gaussian-DP than RDP accounting. However the presentation needs improvement, and technical contributions is somewhat limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper generalizes the Renyi Filter techniques and individual privacy accounting results by Feldman and Zrnic (2021) from Renyi DP to Gaussian DP. At the end of this paper, the authors use experiments with MIMIC-III to obtain a similar result to Yu et al. (2022) which shows a correlation between the average of individual privacy guarantees and the accuracies among different groups. 
strength_and_weaknesses: Major strengths: 
1. The Gaussian DP is a special case of f-DP which is one of the optimal DP definitions for tight privacy analysis, and the Renyi filter is also one of the most important breakthroughs for fine-grained DP analysis. It is great to see this work connect these two ideas.

Minor strengths: 
1. The paper is well-written and easy to read. 

Major weaknesses: 
1. The results in this paper are not as comprehensive and novel as I expected. It is natural to use the idea of dominating pair of distributions which is equivalent to f-DP. Since Gaussian DP is mostly useful in asymptotic composition results, and the tight privacy analysis often requires f-DP, I would like to know why the proof of Theorem 11 cannot be generalized to f-DP. 
2. The Blackwell theorem used in Section 5 is not very appropriate since it is based on the relationship between two tradeoff functions (as shown in the Gaussian DP paper). There is no such result in the dominating pair paper (Zhu et al. 2022). I mean, the authors should provide or cite such a result that if $(P,Q)$ is a dominating pair, then the tradeoff function $T_{P,Q}$ is a lower bound for the tradeoff functions of neighboring datasets. By the way, I guess the stochastic transformation $T$ is a Markov kernel instead of a Markov process, therefore there is a typo on Page 8.
3. The experimental results are not very related to the main topic of this paper.

Minor weaknesses:
1. The composition result of Gaussian DP is not stated in the main text (the proof description of Theorem 11) but it must be used somewhere (I know it is shown at the end of Appendix Section B, but I cannot find it in the main text.). This confuses me since in the proof of Theorem 12, the authors used $\sum_{i=1}^t \mu_i \geq B$ instead of $\sum_{i=1}^t \mu_i^2 \geq B^2$ (which I think is a typo and appears again in the line above Equation (5.1).)
2. Algorithm 1 may need to be revised. I suggest the authors check 'Algorithm 2 by Feldman and Zrnic (2021)' again since we need $(X, X^{-i})$ to be any $(S_1,S_2) \in \mathcal{S}(X_i,n)$. By the way, I think $\mu_{j+1}^{(i)}$ should be $\mu_{j}^{(i)}$
3. Theorem 12 should be put before Algorithm 1 since the two conditions, HALT and CONT, need to be defined.
4. In Section 4.1, the worse-case analysis is said to be $(\varepsilon=1.0, \delta=10^{-5})$, but this is not shown in Figure 3. Is Figure 3 an average result of individual privacy accounting?
5. In Section 6, the authors claim that 'we find that while the individual filtering leads to more equal group-wise $\varepsilon$ values, ...' I cannot find any result that shows the 'more' equal values. There seems to be no such comparison in Figure 2.

clarity,_quality,_novelty_and_reproducibility: This paper is mostly well-written. The statements in the paper are clear and concise. The novelty of this paper is not very high (see the weaknesses). I believe that the experimental results are reproducible, and I have checked the theoretical results which mostly look good to me.
summary_of_the_review: This paper is easy to read and follow. The authors connect the existing work together (Gaussian DP, dominating pairs, and Renyi Filter), so their results are natural and correct. It is like an exact follow-up to the work done by Feldman and Zrnic (2021). There are minor issues in this paper but I believe all are fixable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper gives new methods for privacy loss accounting in differentially private (DP) algorithms, and illustrates the improvements of its methods over prior work through experimental results. The experiments are mainly performed via the case of running private gradient descent (DP-GD) in various learning tasks. As the paper lists in Section 1.1, it contains four main contributions.
1. Analysis of fully adaptive compositions of DP mechanisms using Gaussian differential privacy (GDP), which in turn gives tight $(\varepsilon, \delta)$ bounds for compositions of Gaussian mechanisms.
2. Using the idea of dominating pairs of distributions and the Blackwell Theorem, they provide an approximative $(\varepsilon, \delta)$ accountant, which frequently leads to better $\varepsilon$ bounds than the traditional Renyi differential privacy (RDP) accountants.
3. It provides efficient numerical techniques to account for individual privacy parameters using privacy loss distributions (PLD's) and and FFT, which improves on the running time of existing accountants.
4. It gives experimental results to show the improvements in privacy accounting over RDP for both the aforementioned methods. It also observes that a result of individual filtering is that it leads to disparate loss in accuracies among subgroups while training using a neural network via DP-GD.
strength_and_weaknesses: Strengths:
1. The paper is well-written, with the flow and the preliminary ideas being easy to follow. The definitions and the ideas are explained quite nicely, along with supporting examples at different places. I also like the idea of keeping the appendix within the same document because for me, personally, it makes it easier to read.
2. The ideas are backed by rigorous proofs, which seem believable to me. So, the paper appears strong enough to me mathematically.
3. The experiments support the claims, too, as the improvements over the usual RDP accountants are clearly visible.
4. On a high level, I think this does seem to be an important problem to work on because of the existing gap between deployable DP systems and the theoretical foundations. Of course, more progress has been made, especially in the last few years to invent more practical algorithms for various applications, but tighter accounting is a great step here. It feels like a significant problem to study to learn about the limits of the existing DP tools or even for what we could possibly hope to achieve privately. Essentially, all I'm saying is that I like the paper also because of the problem they're trying to solve.

Weaknesses:
1. I did see comparisons with prior work, but only through experiments. I'm not *super* familiar with this particular aspect of DP literature, so I'm wondering if it were possible at all to have more in-depth theoretical comparisons, as well?
2. The experiments were somewhat convincing, but not fully, if I'm honest. For example, in Figure 1, the plot of $\delta$-vs-$\varepsilon$ doesn't give me that much details as to how it compares with the RDP accountant. Also, I'm not sure how much we gain by improving $\delta$ because any non-trivial DP algorithm usually has polylogarithmic dependence on $\delta$ in its accuracy guarantees. Isn't $\varepsilon$ the quantity we should be focussing on more? More clarity on these details would be nice, unless I'm missing something obvious.
3. I wish I could see more applications of their work besides DP-GD. That would have given me a better idea of how useful this is for other techniques that don't involve the Gaussian mechanism, for example, those that use stability-based techniques. What do I expect the results to look like in those cases, and why?
clarity,_quality,_novelty_and_reproducibility: As mentioned earlier, I think the paper is well-written, so I don't have much issues with clarity, barring the questions I raised in the same section of the review.

There seems to be sufficient novelty in the techniques -- sure, many ideas do seem to be based on recent prior work, but the idea of trying GDP did the trick to improve. I'm satisfied by that!
summary_of_the_review: I do vote for acceptance of this paper based on what I wrote earlier -- the positives seem to outweigh the negatives, in my opinion. That said, my review might change depending on the authors response.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work extends fully adaptive differential privacy filter and individual privacy techniques of Feldman and Zrnic (2019) to Gaussian DP. It demonstrates a numerical improvement in the parameters that can be achieved through this generalization and a faster way to perform individual DP accounting via existing techniques. DP composition and its individual version are one of the key tools in private ML. While the novelty of the techniques needed for this extension is relatively modest this tool is likely to be useful in ML applications.
justification_for_why_not_higher_score: Technical novelty is limited.
justification_for_why_not_lower_score: Clean theoretical result that is also likely to be useful practically
Decision: Accept: poster