SlowGuess/NeurIPS_2023_OCR huggingface dataset:
J. M. Abowd, R. Ashmead, R. Cumings-Menon, S. Garfinkel, M. Heineck, C. Heiss, R. Johns, D. Kifer, P. Leclerc, A. Machanavajjhala, B. Moran, W. Sexton, M. Spence, and P. Zhuravlev. The 2020 census disclosure avoidance system topdown algorithm, 2022. arXiv: 2204.08986.
M. Bun and T. Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography, pages 635-658. Springer Berlin Heidelberg, 2016. URL https://link.springer.com/chapter/10.1007/978-3-662-53641-4_24.
M. Cesar and R. Rogers. Bounding, concentrating, and truncating: Unifying privacy loss composition for data analytics. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 of Proceedings of Machine Learning Research, pages 421-457. PMLR, 2021. URL https://proceedings.mlr.press/v132/cesar21a.html.
C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology - EUROCRYPT 2006, pages 486-503. Springer Berlin Heidelberg, 2006a. ISBN 978-3-540-34547-3. URL https://www.iacr.org/archive/eurocrypt2006/40040493/40040493.pdf.
C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography, TCC'06, page 265-284. Springer-Verlag, 2006b. ISBN 3540327312. doi: 10.1007/11681878_14. URL https://doi.org/10.1007/11681878_14.
C. Dwork, G. N. Rothblum, and S. P. Vadhan. Boosting and differential privacy. In 51st Annual Symposium on Foundations of Computer Science, pages 51-60, 2010.
V. Feldman and T. Zrnic. Individual privacy accounting via a rényi filter. In Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=PBctz6_47ug.
S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingales. *Probability Surveys*, 17:257 – 317, 2020.
S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055 - 1080, 2021.
P. Kairouz, S. Oh, and P. Viswanath. The composition theorem for differential privacy. volume 63, pages 4037-4049, 2017. doi: 10.1109/TIT.2017.2685505. URL https://doi.org/10.1109/TIT.2017.2685505.
F. Koufogiannis, S. Han, and G. J. Pappas. Gradual release of sensitive data under differential privacy. Journal of Privacy and Confidentiality, 7(2), 2017.
K. Ligett, S. Neel, A. Roth, B. Waggoner, and S. Z. Wu. Accuracy first: Selecting a differential privacy level for accuracy constrained ERM. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/file/86df7dcbd896fcaf2674f757a2463eba-Paper.pdf.
X. Lyu. Composition theorems for interactive differential privacy. In Advances in Neural Information Processing Systems, volume 35, pages 9700-9712, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/3f52b555967a95ee850fcecbd29ee52d-Paper-Conference.pdf.
F. McSherry and K. Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07), pages 94-103, 2007. doi: 10.1109/FOCS.2007.66. URL http://dx.doi.org/10.1109/FOCS.2007.66.
J. Murtagh and S. Vadhan. The complexity of computing the optimal composition of differential privacy. In Proceedings, Part I, of the 13th International Conference on Theory of Cryptography - Volume 9562, TCC 2016-A, pages 157-175, Berlin, Heidelberg, 2016. Springer-Verlag. ISBN 978-3-662-49095-2. doi: 10.1007/978-3-662-49096-9_7. URL https://doi.org/10.1007/978-3-662-49096-9_7.
D. Revuz and M. Yor. Continuous martingales and Brownian motion, volume 293. Springer Science & Business Media, 2013.
R. M. Rogers, A. Roth, J. Ullman, and S. P. Vadhan. Privacy odometers and filters: Pay-as-you-go composition. In Advances in Neural Information Processing Systems, pages 1921-1929, 2016. URL http://papers.nips.cc/paper/6170-privacy-odometers-and-filters-pay-as-you-go-composition.
S. Vadhan and T. Wang. Concurrent composition of differential privacy. In K. Nissim and B. Waters, editors, Theory of Cryptography, pages 582-604, Cham, 2021. Springer International Publishing. ISBN 978-3-030-90453-1.
S. Vadhan and W. Zhang. Concurrent composition theorems for differential privacy. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing. ACM, jun 2023. doi: 10.1145/3564246.3585241. URL https://doi.org/10.1145%2F3564246.3585241.
H. Wang, X. Peng, Y. Xiao, Z. Xu, and X. Chen. Differentially private data aggregating with relative error constraint, 2022. URL https://doi.org/10.1007/s40747-021-00550-3.
J. Whitehouse, A. Ramdas, S. Wu, and R. Rogers. Brownian noise reduction: Maximizing privacy subject to accuracy constraints. In Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=J-IZQLQZdYu.
J. Whitehouse, A. Ramdas, R. Rogers, and Z. S. Wu. Fully adaptive composition in differential privacy. In International Conference on Machine Learning. PMLR, 2023.
X. Xiao, G. Bender, M. Hay, and J. Gehrke. Ireduct: Differential privacy with reduced relative errors. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data, page 229-240. Association for Computing Machinery, 2011. doi: 10.1145/1989323.1989348.
A Concurrent Composition
We will use concurrent composition for differential privacy, introduced by Vadhan and Wang [2021] and with subsequent work in Lyu [2022], Vadhan and Zhang [2023], in our analysis, which we define next. We first define an interactive system.
Definition A.1. An interactive system is a randomized algorithm  $S: (\mathcal{Q} \times \mathcal{Y})^* \times \mathcal{Q} \to \mathcal{Y}$  with input an interactive history  $(q_1, y_1), (q_2, y_2), \dots, (q_t, y_t) \in (\mathcal{Q} \times \mathcal{Y})^t$  with a query  $q_{t+1} \in \mathcal{Q}$ . The output of  $S$  is denoted  $y_{t+1} \sim S((q_i, y_i)_{i \in [t]}, q_{t+1})$ .
Note that an interactive system may also consist of an input dataset  $x$  that the queries are evaluated on, which will then induce an interactive system. In particular, we will consider two neighboring datasets  $x^{(0)}$  and  $x^{(1)}$ , which will then induce interactive systems  $S_{i}^{(b)}$  corresponding to input data  $x^{(b)}$  for  $b \in \{0, 1\}$  and  $i \in [k]$ . We will use the concurrent composition definition from Lyu [2022].
Definition A.2 (Concurrent Composition). Suppose  $S_1, \dots, S_k$  are  $k$  interactive systems. The concurrent composition of them is an interactive system  $\mathrm{COMP}(S_1, \dots, S_k)$  with query domain  $[k] \times \mathcal{Q}$  and response domain  $\mathcal{V}$ . An adversary is a (possibly randomized) query algorithm  $\mathcal{A} : ([k] \times \mathcal{Q} \times \mathcal{V})^* \to [k] \times \mathcal{Q}$ . The interaction between  $\mathcal{A}$  and  $\mathrm{COMP}(S_1, \dots, S_k)$  is a stochastic process that runs as follows.  $\mathcal{A}$  first computes a pair  $(i_1, q_1) \in [k] \times \mathcal{Q}$ , sends a query  $q_1$  to  $S_{i_1}$  and gets the response  $y_1$ . In the t-th step,  $\mathcal{A}$  calculates the next pair  $(i_t, q_t)$  based on the history, sends the t-th query  $q_t$  to  $S_{i_t}$  and receives  $y_t$ . There is no communication or interaction between the interactive systems. Each system  $S_i$  can only see its own interaction with  $\mathcal{A}$ . Let IT  $(\mathcal{A} : S_1, \dots, S_k)$  denote the random variable recording the transcript of the interaction.
We will be interested in how much the distribution of the transcript of interaction changes for interactive systems with neighboring inputs. We will say  $\text{COMP}(S_1, \dots, S_k)$  is  $(\varepsilon, \delta)$ -DP if for all neighboring datasets  $x^{(0)}, x^{(1)}$ , we have for any outcome set of transcripts  $E$ , we have for  $b \in \{0, 1\}$
$$
\operatorname * {P r} [ \mathrm {I T} (\mathcal {A}: S _ {1} ^ {(b)}, \dots , S _ {k} ^ {(b)}) \in E ] \leq e ^ {\varepsilon} \operatorname * {P r} [ \mathrm {I T} (\mathcal {A}: S _ {1} ^ {(1 - b)}, \dots , S _ {k} ^ {(1 - b)}) \in E ] + \delta .
$$
We then have the following result from Lyu [2022].
Theorem 6. Let  $A_{1},\dots ,A_{k}$  be  $k$  interactive mechanisms that run on the same data set. Suppose that each mechanism  $A_{i}$  satisfies  $(\varepsilon_{i},\delta_{i})$  -DP. Then  $\mathrm{COMP}(\mathcal{A}:A_1,\dots ,A_k)$  is  $(\varepsilon ,\delta)$  -DP, where  $\varepsilon ,\delta$  are given by the optimal (sequential) composition theorem Kairouz et al. [2017], Murtagh and Vadhan [2016].
This result is very powerful because we can consider the privacy of each interactive system separately and still be able to provide a differential privacy guarantee for the more complex setting that allows an analyst to interweave queries to different interactive systems.
B Missing Proofs and Figures
B.1 Missing Proofs from Section 5
We start with a simple result that states that ex-post mechanisms compose by just adding up the ex-post privacy functions. Similar to our notation for privacy filters, we will denote  $\mathcal{E}_n(\cdot ;x)$  as the privacy loss bound for algorithm  $A_{n}$  conditioned on the outcomes of  $A_{1:n - 1}(x)$ .
Lemma B.1. Fix a sequence  $\delta_1, \dots, \delta_m \geq 0$ . Let there be a probability measure  $\mu_n$  on  $\mathcal{Y}_n$  for each  $n$  and the product measure on  $\mathcal{Y}_1 \times \dots \times \mathcal{Y}_m$ . Consider mechanisms  $A_n: \mathcal{X} \times \prod_{i=1}^{n-1} \mathcal{Y}_i \to \mathcal{Y}_n$  for  $n \in [m]$  where each  $A_n(\cdot; y_{1:n-1})$  is  $(\mathcal{E}_n(\cdot; y_{1:n-1}), \delta_n)$ -ex-post private for all prior outcomes  $y_{1:n-1}$ . Then the overall mechanism  $A_{1:m}(\cdot)$  is  $(\sum_{i=1}^{m} \mathcal{E}_i(A_i(\cdot); \cdot), \sum_{i=1}^{m} \delta_i)$ -ex-post private with respect to the product measure.
Proof. We consider neighboring inputs  $x, x'$  and write the privacy loss random variable  $\mathcal{L}_A(A(x))$  for  $A$  in terms of the privacy losses  $\mathcal{L}_i(A_i(x))$  of each  $A_i$  for  $i \in [m]$
$$
\Pr \left[ \mathcal {L} _ {A} (A (x)) <   \sum_ {i = 1} ^ {m} \mathcal {E} _ {i} (A _ {i} (x); x) \right] = \Pr \left[ \sum_ {i = 1} ^ {m} \mathcal {L} _ {i} (A _ {i} (x)) <   \sum_ {i = 1} ^ {m} \mathcal {E} _ {i} (A _ {i} (x); x) \right]
$$
Because each mechanism  $A_{i}$  is  $(\mathcal{E}_i(\cdot ;x),\delta_i)$ -ex post private, we have  $\operatorname*{Pr}[\mathcal{L}_i(A(x))\geq \mathcal{E}_i(A_i(x);x)]\leq \delta_i$  and hence
$$
\begin{array}{l} \Pr \left[ \sum_ {i = 1} ^ {m} \mathcal {L} _ {i} (A _ {i} (x)) \leq \sum_ {i = 1} ^ {m} \mathcal {E} _ {i} (A _ {i} (x); x) \right] \geq \Pr \left[ \bigcap_ {i \in [ m ]} \left\{\mathcal {L} _ {i} (A _ {i} (x)) \leq \mathcal {E} _ {i} (A _ {i} (x); x) \right\} \right] \\ = 1 - \Pr \left[ \bigcup_ {i \in [ m ]} \left\{\mathcal {L} _ {i} \left(A _ {i} (x)\right) > \mathcal {E} _ {i} \left(A _ {i} (x); x\right) \right\} \right] \\ \geq 1 - \sum_ {i \in [ m ]} \Pr \left[ \mathcal {L} _ {i} \left(A _ {i} (x)\right) > \mathcal {E} _ {i} \left(A _ {i} (x); x\right) \right] \\ \geq 1 - \sum_ {i \in [ m ]} \delta_ {i}. \\ \end{array}
$$
Hence, we have  $\operatorname*{Pr}\left[\mathcal{L}_A(A(x))\geq \sum_{i = 1}^m\mathcal{E}_i(A_i(x);x)\right]\leq \sum_{i\in [m]}\delta_i$  , as desired.
For general ex-post private mechanisms, this basic composition cannot be improved. We can simply pick  $\mathcal{E}_i$  to be the same as the privacy loss  $\mathcal{L}_i$  at each round with independently selected mechanisms  $A_i$  at each round  $i \in [m]$ . We now show how we can obtain a privacy filter from a sequence of ex-post mechanisms as long as each selected ex-post privacy mechanism selected at each round cannot exceed the remaining privacy budget. We will write  $\delta_n(x)$  to denote the parameter  $\delta_n$  selected as a function of prior outcomes from  $A_1(x), \dots, A_{n-1}(x)$ .
Lemma B.2. Let  $\varepsilon > 0$  and  $\delta \geq 0$  be fixed privacy parameters. Let  $(A_{n} : \mathcal{X} \to \mathcal{Y})_{n \geq 1}$  be a sequence of  $(\mathcal{E}_{n}(\cdot ; x), \delta_{n}(x))$ -ex-post private conditioned on prior outcomes  $y_{1:n-1} = A_{1:n-1}(x)$  where for all  $y_{i}$  we have
$$
\sum_ {i = 1} ^ {n - 1} \mathcal {E} _ {i} (y _ {i}; x) \leq \varepsilon .
$$
Consider the function  $N:\mathcal{Y}^{\infty}\to \mathbb{N}$  where
$$
N \left(\left(y _ {n}\right) _ {n \geq 1}\right) = \inf  \left\{n: \varepsilon = \sum_ {i \in [ n ]} \mathcal {E} _ {i} \left(y _ {i}; x\right) o r \delta <   \sum_ {i \in [ n + 1 ]} \delta_ {i} \left(y _ {1: i - 1}\right) \right\}.
$$
Then the algorithm  $A_{1:N(\cdot)}(\cdot)$  is  $(\varepsilon, \delta)$ -DP where
$$
N (x) = N \left(\left(A _ {n} (x)\right) _ {n \geq 1}\right).
$$
Proof. We follow a similar analysis in Whitehouse et al. [2023] where they created a filter for probabilistic DP mechanisms, that is the privacy loss of each mechanism can be bounded with high probability. We will write the corresponding privacy loss variables of  $A_{n}$  to be  $\mathcal{L}_n$  and for the full sequence of algorithms  $A_{1:n} = (A_1,\dots ,A_n)$ , the privacy loss is denoted as  $\mathcal{L}_{1:n}$ .
Define the events
$$
D := \left\{\exists n \leq N (x): \mathcal {L} _ {1: n} > \varepsilon \right\}, \text {a n d} E := \left\{\exists n \leq N (x): \mathcal {L} _ {n} > \mathcal {E} _ {n} \left(A _ {n} (x); x\right) \right\}.
$$
Using Bayes rule, we have that
$$
\Pr [ D ] = \Pr [ D \cap E ^ {c} ] + \Pr [ D \cap E ] \leq \Pr [ D | E ^ {c} ] + \Pr [ E ] = \Pr [ E ],
$$
where the last inequality follows from how we defined the stopping function  $N(x)$ . Now, we show that  $\operatorname*{Pr}[E]\leq \delta$ . Define the modified privacy loss random variables  $(\widetilde{\mathcal{L}}_n)_{n\in \mathbb{N}}$  by
$$
\widetilde {\mathcal {L}} _ {n} := \left\{ \begin{array}{l l} \mathcal {L} _ {n} & n \leq N (x) \\ 0 & \text {o t h e r w i s e} \end{array} \right..
$$
Likewise, define the modified privacy parameter random variables  $\widetilde{\mathcal{E}}_n(\cdot ;x)$  and  $\widetilde{\delta}_n(x)$  in an identical manner. Then, we can bound  $\operatorname*{Pr}[E]$  in the following manner:
$$
\begin{array}{l} \Pr \left[ \exists n \leq N (x): \mathcal {L} _ {n} > \mathcal {E} _ {n} \left(A _ {n} (x); x\right) \right] = \Pr \left[ \exists n \in \mathbb {N}: \widetilde {\mathcal {L}} _ {n} > \widetilde {\mathcal {E}} _ {n} \left(A _ {n} (x); x\right) \right] \\ \leq \sum_ {n = 1} ^ {\infty} \Pr \left[ \widetilde {\mathcal {L}} _ {n} > \widetilde {\mathcal {E}} _ {n} (A _ {n} (x); x) \right] \\ = \sum_ {n = 1} ^ {\infty} \mathbb {E} \left[ \Pr \left[ \widetilde {\mathcal {L}} _ {n} > \widetilde {\mathcal {E}} _ {n} (A _ {n} (x); x) \mid \mathcal {F} _ {n - 1} \right] \right] \\ \leq \sum_ {n = 1} ^ {\infty} \mathbb {E} \left[ \widetilde {\delta} _ {n} (x) \right] = \mathbb {E} \left[ \sum_ {n = 1} ^ {\infty} \widetilde {\delta} _ {n} (x) \right] = \mathbb {E} \left[ \sum_ {n \leq N (x)} \delta_ {n} (x) \right] \leq \delta , \\ \end{array}
$$
Proof of Theorem 2. We will separate out the approximate zCDP mechanisms,  $(A_{n}^{\mathrm{zCDP}})_{n\geq 1}$  from the ex-post private mechanisms  $(A_{n}^{\mathrm{post}})_{n\geq 1}$  to form two separate interactive systems. In this case, the parameters that are selected can only depend on the outcomes from the respective interactive system, e.g.  $\rho_{n}(x)$  can only depend on prior outcomes to mechanisms  $A_{i}^{\mathrm{zCDP}}(x)$  for  $i < n$ . From Theorem 1, we know that  $A_{1:N_{\mathrm{zCDP}}(\cdot)}^{\mathrm{zCDP}}(\cdot)$  is  $(\varepsilon, \delta + \delta'')$ -DP. We denote  $\mathcal{L}_n^{\mathrm{post}}$  to be the privacy loss random variable for the ex-post private mechanism at round  $n$ . We will also write the stopping time for the ex-post private mechanisms as  $N_{\mathrm{post}}(x)$ . From Lemma B.2, we know that  $A_{1:N_{\mathrm{post}}(\cdot)}^{\mathrm{post}}(\cdot)$  is  $(\varepsilon', \delta')$ -DP.
From Theorem 6, we know that the concurrent composition, which allows for both  $A_{1:N_{\mathrm{post}}(\cdot)}^{\mathrm{post}}(\cdot)$  and  $A_{1:N_{\mathrm{zCDP}}(\cdot)}^{\mathrm{zCDP}}(\cdot)$  to interact arbitrarily, will still be  $(\varepsilon + \varepsilon', \delta + \delta' + \delta'')$ -DP.
Remark 1. Note that although DP is closed under post-processing, ex-post privacy is not. We typically say that any post processing function of a DP outcome is also DP with the same parameter. Our privacy analysis depends on getting actual outcomes from BM, rather than a post-processed value of it. However, we also say post processing ensures that no one can compromise privacy by taking the outcome and trying to learn more about the dataset. With ex-post privacy, we are not trying to ensure privacy of each outcome, but rather the full interaction, so the full interaction will still be DP under any post processing of any intermediate outcomes.
B.2 Missing Remarks from Section 6
We now write out some remarks regarding Theorem 4.
Remark 2. We note here that the stopping time of each Brownian noise reduction is with respect to  $\mathcal{H} = (\mathcal{H}^{(t)})$  where  $\mathcal{H}^{(t)} = \sigma (f(x) + B^{(u)};u > t)$ , but  $f(x)$  is constant so the shifted process is equivalent to the of the non-shifted process  $\mathcal{G}$  as defined above.
Remark 3. For the multivariate case we have  $B^{(u)} = \left(B^{(u)}[i]: i \in [d]\right)$  where each coordinate is an independent Brownian motion, and we will write the filtration  $\mathcal{G}[i]$  to be the natural reverse filtration corresponding to the Brownian motion for index  $i$ . We then define  $\mathcal{G}^{(t)} = \sigma\left(B^{(u)}[1], \dots, B^{(u)}[d]: u \geq t\right)$ . From Lemma 6.1, we have the following for all  $\lambda \in \mathbb{R}$ ,  $i \in [d]$ , and  $0 < s < t$
$$
\mathbb {E} \left[ \exp \left(\lambda B ^ {(t)} [ i ] / t - \lambda^ {2} / (2 t)\right) \mid \mathcal {G} ^ {(s)} [ i ] \right] \leq 1.
$$
We then consider the full  $d$ -dimensional Brownian motion so that with a unit vector  $v = (v[1], \dots, v[d]) \in \mathbb{R}^d$  we have
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(\lambda \cdot \sum_ {i = 1} ^ {d} v [ i ] \cdot B ^ {(t)} [ i ] / t - \frac {\lambda^ {2}}{2 t}\right) \mid \mathcal {G} ^ {(s)} \right] \\ = \mathbb {E} \left[ \prod_ {i = 1} ^ {d} \exp \left(\lambda \cdot v [ i ] \cdot B ^ {(t)} [ i ] / t - \frac {\lambda^ {2} \cdot v [ i ] ^ {2}}{2 t}\right) \mid \mathcal {G} ^ {(s)} \right] \\ = \prod_ {i = 1} ^ {d} \mathbb {E} \left[ \exp \left(\lambda \cdot v [ i ] \cdot B ^ {(t)} [ i ] / t - \frac {\lambda^ {2} \cdot v [ i ] ^ {2}}{2 t}\right) \mid \mathcal {G} ^ {(s)} [ i ] \right] \\ \leq 1. \\ \end{array}
$$
B.3 Missing Proofs from Section 6
Proof of Lemma 6.2. We use the decomposition of the privacy loss  $\mathcal{L}_n^{(1:k)}$  at round  $n\geq 1$  for Brownian noise reduction in Theorem 3 that is stopped at time value  $t_n^{(k)}$  to get the following with the natural filtration  $(\mathcal{F}_n(x))_{n\geq 1}$  generated by  $(A_{n}(x))_{n\geq 1}$ . Recall that we have for the Brownian motion  $(B_n^{(t)})_{t > 0}$  used in the Brownian noise reduction
$$
\mathcal {L} _ {n} ^ {(1: k)} = \mathcal {L} _ {n} ^ {(k)} = \frac {1}{2 t _ {n} ^ {(k)}} + \frac {B _ {n} ^ {\left(t _ {n} ^ {(k)}\right)}}{t _ {n} ^ {(k)}}.
$$
From Lemma 6.1 we have for all  $\lambda \in \mathbb{R}$  and  $n\geq 1$  with time value  $t_n^{(k)}$
$$
\mathbb {E} \left[ \exp \left(\lambda X _ {n} ^ {(k)} - \frac {\lambda^ {2}}{2 t _ {n} ^ {(k)}}\right) \mid \mathcal {F} _ {n - 1} (x) \right] \leq 1, \quad \text {w h e r e} \quad X _ {n} ^ {(k)} = \mathcal {L} _ {n} ^ {(k)} - \frac {1}{2 t _ {n} ^ {(k)}}.
$$
We then form the following process
$$
M _ {n} ^ {\left(k _ {1}, \dots , k _ {n}\right)} = \exp \left(\lambda \sum_ {i = 1} ^ {n} X _ {i} ^ {\left(k _ {i}\right)} - \sum_ {i = 1} ^ {n} \frac {\lambda^ {2}}{2 t _ {i} ^ {\left(k _ {i}\right)}}\right).
$$
Hence, with fixed time values  $(t_n^{(k_n)})_{n\geq 1}$  for  $n\geq 1$  we have for all  $\lambda$
$$
\mathbb {E} \left[ M _ {n} ^ {\left(k _ {1}, \dots , k _ {n}\right)} \mid \mathcal {F} _ {n - 1} (x) \right] \leq 1.
$$
We then replace  $(k_{i})_{i\leq n}$  with an adaptive stopping time functions  $(T_{i})_{i\leq n}$ , rather than fixing them in advance, in which case we rename  $M_n^{(k_1,\dots ,k_n)}$  as  $M_n^{\mathrm{BM}}$ . We know from Lemma 6.1 that  $M_n^{\mathrm{BM}}$  is still an e-value for any  $n$ . We then apply the optional stopping theorem to conclude that with the stopping time  $N(x)$  that  $\mathbb{E}\left[M_N^{\mathrm{BM}}(x)\right]\leq 1$ . By the definition of our stopping time, so that  $\sum_{i = 1}^{N(x)}\frac{1}{2t_i^{T_i(x)}} = \rho$ , we have for all  $\lambda$ ,
$$
\mathbb {E} \left[ \right. \exp \left( \right.\left. (\lambda - 1) \sum_ {i = 1} ^ {N (x)} L _ {i} ^ {(T _ {i} (x))}\right)\left. \right] \leq e ^ {\lambda (\lambda - 1) \rho}.
$$
We then set  $\lambda = \frac{2\rho + 2\sqrt{\rho\log(1 / \delta)}}{2\rho}$  to get
$$
\Pr \left[ \sum_ {i = 1} ^ {N (x)} L _ {i} ^ {\left(T _ {i} (x)\right)} \geq \rho + 2 \sqrt {\rho \log \left(1 / \delta\right)} \right] \leq \delta .
$$
We then have a high probability bound on the overall privacy loss, which then implies differential privacy.
Proof of Theorem 5. We will show that  $A_{1:N(\cdot)}(\cdot)$  is  $\delta$ -approximate  $\rho$ -zCDP and then use Lemma 2.1 to obtain a DP guarantee stopping at  $N(x)$  as defined in the theorem statement.
We follow a similar analysis to the proof of Theorem 1 in Whitehouse et al. [2023]. Let  $P_{1:n}$  and  $Q_{1:n}$  denote the joint distributions of  $(A_1, \ldots, A_n)$  with inputs  $x$  and  $x'$ , respectively. We overload notation and write  $P_{1:n}(y_1, \ldots, y_n)$  and  $Q_{1:n}(y_1, \ldots, y_n)$  for the likelihood of  $y_1, \ldots, y_n$  under input  $x$  and  $x'$  respectively. We similarly write  $P_n(y_n \mid y_{1:n-1})$  and  $Q_n(y_n \mid y_{1:n-1})$  for the corresponding conditional densities.
For any  $n\in \mathbb{N}$  , we have
$$
P _ {1: n} \left(y _ {1}, \dots , y _ {n}\right) = \prod_ {m = 1} ^ {n} P _ {m} \left(y _ {m} \mid y _ {1: m - 1}\right),
$$
$$
Q _ {1: n} \left(y _ {1}, \dots , y _ {n}\right) = \prod_ {m = 1} ^ {n} Q _ {m} \left(y _ {m} \mid y _ {1: m - 1}\right).
$$
When then show that the two likelihoods can be decomposed as weighted mixtures of  $P'$  and  $P''$ , as well as  $Q'$  and  $Q''$ , respectively such that the mixture weights on  $P'$  and  $Q'$  are at least  $(1 - \delta)$ , and for all  $\lambda \geq 1$ ,
$$
\max  \left\{D _ {\lambda} \left(P ^ {\prime} \| Q ^ {\prime}\right), D _ {\lambda} \left(Q ^ {\prime} \| P ^ {\prime}\right) \right\} \leq \rho \lambda . \tag {4}
$$
By our assumption of approximate zCDP at each step  $n$ , we can write the conditional likelihoods of  $P_{n}$  and  $Q_{n}$  as the following convex combinations:
$$
P _ {n} (y _ {n} \mid y _ {1: n - 1}) = (1 - \delta_ {n} (y _ {1: n - 1})) P _ {n} ^ {\prime} (y _ {n} \mid y _ {1: n - 1}) + \delta_ {n} (y _ {1: n - 1}) P _ {n} ^ {\prime \prime} (y _ {n} \mid y _ {1: n - 1}),
$$
$$
Q _ {n} \left(y _ {n} \mid y _ {1: n - 1}\right) = \left(1 - \delta_ {n} \left(y _ {1: n - 1}\right)\right) Q _ {n} ^ {\prime} \left(y _ {n} \mid y _ {1: n - 1}\right) + \delta_ {n} \left(y _ {1: n - 1}\right) Q _ {n} ^ {\prime \prime} \left(y _ {n} \mid y _ {1: n - 1}\right),
$$
such that for all  $\lambda \geq 1$  and all prior outcomes  $y_{1:n - 1}$ , we have both
$$
D _ {\lambda} \left(P _ {n} ^ {\prime} (\cdot \mid y _ {1: n - 1}) \| Q _ {n} ^ {\prime} (\cdot \mid y _ {1: n - 1})\right) \leq \rho_ {n} \left(y _ {1: n - 1}\right) \lambda , \tag {5}
$$
$$
D _ {\lambda} \left(Q _ {n} ^ {\prime} (\cdot \mid y _ {1: n - 1}) \| P _ {n} ^ {\prime} (\cdot \mid y _ {1: n - 1})\right) \leq \rho_ {n} \left(y _ {1: n - 1}\right) \lambda . \tag {6}
$$