SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.4 (Generalization). Given two functions  $\delta_1: \mathbb{R} \to [0,1]$ ,  $\delta_2: \mathbb{R} \to [0,1]$ , and an analyst  $A$ , if a mechanism  $M$  is  $(\epsilon, \delta_1(\epsilon))$ -Bayes stable and  $(\epsilon, \delta_2(\epsilon))$ -sample accurate with respect to  $A$ , then  $M$  is  $(\epsilon, \delta'(\epsilon))$ -distribution accurate for  $\delta'(\epsilon) := \inf_{\epsilon' \in (0,\epsilon), \xi \in (0,\epsilon - \epsilon')} \left( \delta_1(\epsilon') + \frac{1}{\xi} \int_{\epsilon - \epsilon' - \xi}^{\infty} \delta_2(t) dt \right)$ .
Since achieving posterior accuracy is relatively straightforward, guaranteeing Bayes stability is the main challenge in leveraging this theorem to achieve distribution accuracy with respect to adaptively chosen queries. The following lemma gives a useful and intuitive characterization of the quantity that the Bayes stability definition requires be bounded. Simply put, the Bayes factor  $K(\cdot, \cdot)$  (defined in the lemma below) represents the amount of information leaked about the dataset during the interaction with an analyst, by moving from the prior distribution over data elements to the posterior induced by some view  $v$ . The degree to which a query  $q$  overfits to the dataset is expressed by the correlation between the query and that Bayes factor. This simple lemma is at the heart of the progress that we make in this paper, both in our intuitive understanding of adaptive data analysis, and in the concrete results we show in subsequent sections. Its corresponding version for arbitrary queries are presented in Section C.2
Lemma 3.5 (Covariance stability). Given a view  $v \in \mathcal{V}$  and a linear query  $q$ ,
$$
q \left(D ^ {v}\right) - q \left(D _ {\mathcal {X}}\right) = \underset {X \sim D _ {\mathcal {X}}} {C o v} \left(q \left(X\right), K \left(X, v\right)\right).
$$
Furthermore, given  $\Delta, \sigma > 0$ ,
$$
\sup_{q\in \mathcal{Q}}|q\left(D^{v}\right) - q\left(D_{\mathcal{X}}\right)| = \Delta \cdot \mathbf{D}_{TV}\left(D^{v}\| D_{\mathcal{X}}\right)
$$
$$
\sup_{q\in \mathcal{Q}}\sup_{s.t.  \sigma_{q}^{2}\leq \sigma^{2}}|q\left(D^{v}\right) - q\left(D_{\mathcal{X}}\right)| = \sigma \sqrt{\mathbf{D}_{\mathcal{X}^{2}}\left(D^{v}\| D_{\mathcal{X}}\right)},
$$
where  $K(x, v) \coloneqq \frac{D(x|v)}{D(x)} = \frac{D(v|x)}{D(v)}$  is the Bayes factor of  $x$  given  $v$  (and vice-versa),  $\mathbf{D}_{TV}$  is the total variation distance (Definition B.1), and  $\mathbf{D}_{\chi^2}$  is the chi-square divergence (Definition B.2).
Proof. By definition,  $q(D^v) = \mathbb{E}_{X \sim D^v}[q(X)] = \mathbb{E}_{X \sim D}[K(X,v)q(X)]$ , so
$$
q \left(D ^ {v}\right) - q (D) = \underset {X \sim D} {\mathbb {E}} [ K (X, v) q (X) ] - \overbrace {\underset {X \sim D} {\mathbb {E}} [ K (X , v) ]} ^ {= 1} \cdot \overbrace {\underset {X \sim D} {\mathbb {E}} [ q (X) ]} ^ {= q (D)} = \underset {X \sim D} {\operatorname {C o v}} \left(q (X), K (X, v)\right).
$$
The second part is a direct result of the known variational representation of total variation distance and  $\chi^2$  divergence, which are both  $f$ -divergences (see Equations 7.88 and 7.91 in Polyanskiy and Wu [2022] for more details).
Using the first part of the lemma, we guarantee Bayes stability by bounding the correlation between specific  $q$  and  $K(\cdot ,v)$  as discussed in Section 6. The second part of this Lemma implies that bounding the appropriate divergence is necessary and sufficient for bounding the Bayes stability of the worst query in the corresponding family, which is how the main theorems of this paper are all achieved, using the next corollary.
Corollary 3.6. Given an analyst  $A$ , for any  $\epsilon > 0$  we have
$$
\underset { \begin{array}{c} S \sim D ^ {(n)} \\ V \sim M (S, A), Q \sim A (V) \end{array} } {P r} \left[ \left| Q \left(D ^ {V}\right) - Q \left(D\right) \right| > \sigma_ {Q} \cdot \epsilon \right] \leq \underset { \begin{array}{c} S \sim D ^ {(n)} \\ V \sim M (S, A) \end{array} } {P r} \left[ \mathbf {D} _ {\chi^ {2}} \left(D _ {\mathcal {X}} ^ {v} \| D _ {\mathcal {X}}\right) > \epsilon^ {2} \right].
$$
The corresponding version of this corollary for bounded range queries provides an alternative proof to the generalization guarantees of the LS stability notion [Shenfeld and Ligett, 2019].
4 Pairwise concentration
In order to leverage Lemma 3.5, we need a stability notion that implies Bayes stability of query responses in a manner that depends on the actual datasets and the actual queries (not just the worst case). In this section we propose such a notion and prove several key properties of it. Missing proofs from this section can be found in Appendix D
We start by introducing a measure of the stability loss of the mechanism.
Definition 4.1 (Stability loss). Given two sample sets  $s, s' \in \mathcal{X}^n$ , a query  $q$ , and a response  $r \in \mathcal{R}$ , we denote by  $\ell_q(s, s'; r) \coloneqq \ln \left( \frac{D^q(r|s)}{D^q(r|s')} \right)$  the stability loss of  $r$  between  $s$  and  $s'$  for  $q$  (we omit  $q$  from notation for simplicity). Notice that if  $R \sim M(s, q)$ , the stability loss defines a random variable. Similarly, given two elements  $x, y \in \mathcal{X}$ , we denote  $\ell(x, y; r) \coloneqq \ln \left( \frac{D(r|x)}{D(r|y)} \right)$  and  $\ell(x; r) \coloneqq \ln \left( \frac{D(r|x)}{D(r)} \right)$ . This definition extends to views as well.
Next we introduce a notion of a bound on the stability loss, where the bound is allowed to depend on the pair of swapped inputs and on the issued queries.
Definition 4.2 (Pairwise concentration). Given a non-negative function  $\varphi : \mathcal{X}^n \times \mathcal{X}^n \times \mathcal{Q} \to \mathbb{R}^+$  which is symmetric in its first two arguments, a mechanism  $M$  will be called  $\varphi$ -Pairwise Concentrated (or PC, for short), if for any two datasets  $s, s' \in \mathcal{X}^n$  and  $q \in \mathcal{Q}$ , for any  $\alpha \geq 0$ .
$$
\underset {R \sim M (s, q)} {\mathbb {E}} \left[ \exp \left((\alpha - 1) \ell (s, s ^ {\prime}; R)\right) \right] \leq \exp \left(\alpha (\alpha - 1) \varphi (s, s ^ {\prime}; q)\right).
$$
Given a non-negative function  $\varphi : \mathcal{X}^n \times \mathcal{X}^n \times \mathcal{V} \to \mathbb{R}^+$  which is symmetric in its first two arguments and an analyst  $A$ , a mechanism  $M$  will be called  $\varphi$ -Pairwise Concentrated with respect to  $A$ , if for any  $s, s' \in \mathcal{X}^n$  and  $\alpha \geq 0$ ,
$$
\underset {V \sim M (s, A)} {\mathbb {E}} \left[ \exp \left((\alpha - 1) \left(\ell \left(s, s ^ {\prime}; V\right) - \alpha \varphi \left(s, s ^ {\prime}; V\right)\right)\right) \right] \leq 1.
$$
We refer to such a function  $\varphi$  as a similarity function over responses or views, respectively.
The similarity function serves as a measure of the local sensitivity of the issued queries with respect to the replacement of the two datasets, by quantifying the extent to which they differ from each other with respect to the query  $q$ . The case of noise addition mechanisms provides a natural intuitive interpretation, where the  $\varphi$  scales with the difference between  $q(s)$  and  $q(s')$ , which governs how much observing the response  $r$  distinguishes between the two datasets, as stated in the next lemma.
We note that the first part of this definition can be viewed as a refined version of zCDP (Definition B.18), where the bound on the Rényi divergence (Definition B.5) is a function of the sample sets
This quantity is sometimes referred to as privacy loss in the context of differential privacy, or information density in the context of information theory.
The requirement for the range  $0 \leq \alpha \leq 1$  might appear somewhat counter-intuitive, and results from the fact that  $\varphi$  provides simultaneously a bound on the stability loss random variable's mean (the  $-( \alpha - 1)$  coefficient) and variance proxy (the  $(\alpha - 1)^2$  coefficient), as discusses in Definitions B.17, B.18
and the query. As for the second part, since the bound depends on the queries, which themselves are random variables, it should be viewed as a bound on the Rényi dissimilarity notion that we introduce in the appendix (Definition B.9). This kind of extension is not limited to Rényi divergence, as discussed in Appendix B.
Lemma 4.3 (Gaussian mechanism is PC). Given  $\eta >0$  and an analyst  $A$ , if  $M$  is a Gaussian mechanism with noise parameter  $\eta$ , then it is  $\varphi -PC$  for  $\varphi (s,s';q)\coloneqq \frac{(q(s) - q(s'))^2}{2\eta^2}$  and  $\varphi -PC$  with respect to  $A$  for  $\varphi (s,s';v)\coloneqq \frac{\left\|\bar{q}_v(s) - \bar{q}_v(s')\right\|^2}{2\eta^2}$ .
To leverage this stability notion in an adaptive setting, it must hold under adaptive composition, which we prove in the next theorem.
Theorem 4.4 (PC composition). Given  $k \in \mathbb{N}$ , a similarity function over responses  $\varphi$ , and an analyst  $A$  issuing  $k$  queries, if a mechanism  $M$  is  $\varphi$ - $PC$ , then it is  $\tilde{\varphi}$ - $PC$  with respect to  $A$ , where  $\tilde{\varphi}(s, s'; v) := \sum_{i=1}^{k} \varphi(s, s'; q_i)$ .
Before we state the stability properties of PC mechanisms, we first transition to its element-wise version, which while less general than the original definition, is more suited for our use.
Lemma 4.5 (Element-wise PC). Given a similarity function  $\varphi$  and an analyst  $A$ , if a mechanism  $M$  that is  $\varphi$ -PC with respect to  $A$  receives an iid sample from  $\mathcal{X}^n$ , then for any two elements  $x, y \in \mathcal{X}$  and  $\alpha \geq 1$  we have
$$
\underset {V \sim D (\cdot \mid x)} {\mathbb {E}} \left[ \exp \left((\alpha - 1) \left(\ell (x, y; V) - \alpha \varphi (x, y; V)\right)\right) \right] \leq 1
$$
$$
\underset {V \sim D (\cdot \mid x)} {\mathbb {E}} \left[ \exp \left((\alpha - 1) \left(\ell (x; V) - \alpha \varphi (x; V)\right)\right) \right] \leq 1,
$$
where  $\varphi (x,y;v)\coloneqq \sup_{s\in \mathcal{X}^{n - 1}}\bigl (\varphi \left((s,x),(s,y);v)\right)$  and  $\varphi (x;v)\coloneqq \ln \Big(\underset {Y\sim D}{\mathbb{E}}\big[e^{\varphi (x,Y;v)}\big]\Big).$
Finally, we bound the Bayes stability of PC mechanisms.
Theorem 4.6 (PC stability). Given a similarity function over views  $\varphi$  and an analyst  $A$ , if a mechanism  $M$  that is  $\varphi$ -PC with respect to  $A$  receives an iid sample from  $\mathcal{X}^n$ , then for any  $\epsilon, \delta > 0$ ,  $\phi \geq \underset{\substack{X \sim D, S \sim D^{(n)} \\ V \sim M(S, A)}}{\mathbb{E}}[\varphi(X; V)]$ , we have
$$
\begin{array}{l} \underset { \begin{array}{c} S \sim D ^ {(n)} \\ V \sim M (S, A), Q \sim A (V) \end{array} } {P r} \left[ | Q \left(D _ {\mathcal {X}} ^ {v}\right) - Q \left(D _ {\mathcal {X}}\right) | > \sigma_ {Q} \epsilon \right] \\ \leq \underset { \begin{array}{c} S \sim D ^ {(n)} \\ V \sim M (S, A) \end{array} } {P r} \left[ \underset {X \sim D} {\mathbb {E}} \left[ e ^ {2 7 \ln \left(\frac {1}{\delta}\right) (\varphi (X; V) + \phi)} \right] > 1 + \frac {\epsilon^ {2}}{6} \right] + O \left(\frac {e ^ {\phi} \delta}{\phi \epsilon^ {2}}\right), \\ \end{array}
$$
where  $\varphi (x;v)$  is as defined in Lemma 4.5
An exact version of the bound can be found in Theorem D.10.
Proof outline. From Corollary 3.6,
$$
\operatorname *{Pr}_{\substack{S\sim D^{(n)}\\ V\sim M(S,A),Q\sim A(V)}}\left[|Q\left(D^{\upsilon}_{\mathcal{X}}\right) - Q\left(D_{\mathcal{X}}\right)| > \sigma_{Q}\epsilon \right]\leq \operatorname *{Pr}_{\substack{S\sim D^{(n)}\\ V\sim M(S,A)}}\left[\mathbf{D}_{\mathcal{X}^{2}}\left(D^{\upsilon}_{\mathcal{X}}\| D_{\mathcal{X}}\right) > \epsilon^{2}\right].
$$
Denoting  $\varepsilon(x; v) \coloneqq \varphi(x; v) + \phi + 4\sqrt{\ln\left(\frac{1}{\delta}\right)(\varphi(x; v) + \phi)}$ , this can be bounded by
$$
\operatorname *{Pr}_{\substack{S\sim D^{(n)}\\ V\sim M(S,A)}}\left[\mathbf{D}_{\mathcal{X}^{2}}\left(D_{\mathcal{X}}^{v}\| D_{\mathcal{X}}\right) > \underset {X\sim D}{\mathbb{E}}\left[\left(e^{\varepsilon (X;V)} - 1\right)^{2}\right] + \frac{\epsilon^{2}}{2}\right] + \operatorname *{Pr}_{\substack{S\sim D^{(n)}\\ V\sim M(S,A)}}\left[\underset {X\sim D}{\mathbb{E}}\left[\left(e^{\varepsilon (X;V)} - 1\right)^{2}\right] > \frac{\epsilon^{2}}{2}\right].
$$
We then show via algebraic manipulation that this second term can be bounded using the inequality
$$
\underset {X \sim D} {\mathbb {E}} \left[ \left(e ^ {\varepsilon (X; v)} - 1\right) ^ {2} \right] \leq 3 \left(\underset {X \sim D} {\mathbb {E}} \left[ e ^ {2 7 \ln \left(\frac {1}{\delta}\right) (\varphi (X; v) + \phi)} \right] - 1\right),
$$
and, recalling the definition of the chi square divergence between  $P$  and  $Q$  (Definition B.2)  $\mathbf{D}_{\chi^2}(P\| Q)\coloneqq \underset {X\sim D_2}{\mathbb{E}}\left[\left(\frac{P(X)}{Q(X)} -1\right)^2\right]$ , the first term can be bounded using Markov's inequality followed by the Cauchy-Schwarz inequality by the term:
$$
\begin{array}{l} \frac{2}{\epsilon^{2}}\underset { \begin{array}{c}X\sim D,S\sim D^{(n)}\\ V\sim M(S,A) \end{array} }{\mathbb{E}}\left[\left(\frac{D\left(V\mid X\right)}{D\left(V\right)} -1\right)^{2}\cdot \mathbb{1}_{B(\varepsilon)}\left((X,V)\right)\right] \\ \leq \frac{2}{\epsilon^{2}}\sqrt{\overbrace{\left(\underset{\begin{array}{c}X\sim D,S\sim D^{(n)}\\V\sim M(S,A)\end{array}}\right.}\left[\left(\frac{D(V|X)}{D(V)}\right)^{4}\right] + 1\left. \right)\cdot\overbrace{\underset{\begin{array}{c}X\sim D,S\sim D^{(n)}\\V\sim M(S,A)\end{array}}{\operatorname*{Pr}}[(X,V)\in B(\varepsilon)]}^{*}}^{\ast}} \\ \end{array}
$$
where  $B(\varepsilon) \coloneqq \left\{(x, v) \in \mathcal{X} \times \mathcal{V} \mid \left| \ln \left( \frac{D(v \mid x)}{D(v)} \right) \right| > \varepsilon (x; v) \right\}$ .
We then bound the quantity  $^*$  by  $O\left(e^{\phi}\right)$ , and we bound  $\ast$  by  $O\left(\frac{\delta^2}{\phi}\right)$ , which completes the proof.
Combining this theorem with Theorem 3.4 we get that PC and sample accuracy together imply distribution accuracy.
5 Variance-based generalization guarantees for the Gaussian mechanism
In this section we leverage the theorems of the previous sections to prove variance-based generalization guarantees for the Gaussian mechanism under adaptive data analysis. All proofs from this section appear in Appendix E.
We first provide generalization guarantees for bounded queries.
Theorem 5.1 (Generalization guarantees for bounded queries). Given  $k \in \mathbb{N}$ ;  $\Delta, \sigma, \epsilon \geq 0$ ;  $0 < \delta \leq \frac{1}{e}$ ; and an analyst  $A$  issuing  $k \Delta$ -bounded linear queries with variance bounded by  $\sigma^2$ , if  $M$  is a Gaussian mechanism with noise parameter  $\eta = \Theta \left( \sigma \sqrt{\frac{\sqrt{k}}{n}} \right)$  that receives an iid dataset of size  $n = \Omega \left( \max \left\{ \frac{\Delta}{\epsilon}, \frac{\sigma^2}{\epsilon^2} \right\} \sqrt{k} \cdot \ln \left( \frac{k\sigma}{\delta\epsilon} \right) \right)$ , then  $M$  is  $(\epsilon, \delta)$ -distribution accurate.
An exact version of the bound can be found in Theorem E.4
This theorem provides significant improvement over similar results that were achieved using differential privacy (see, e.g., Theorem 13 in Jung et al. [2020], which is defined for  $\Delta = 1$ ), by managing to replace the  $\Delta^2$  term with  $\sigma^2$ . The remaining dependence on  $\Delta$  is inevitable, in the sense that such a dependence is needed even for non-adaptive analysis. This improvement resembles the improvement provided by Bernstein's inequality over Hoeffding's inequality.
This generalization guarantee, which nearly avoids dependence on the range of the queries, begs the question of whether it is possible to extend these results to handle unbounded queries. Clearly such a result would not be true without some bound on the tail distribution for a single query, so we focus in the next theorem on the case of sub-Gaussian queries. Formally, we will consider the case where  $q(X) - q(D)$  is a sub-Gaussian random variable, for all queries.
Definition 5.2 (Sub-Gaussian random variable). Given  $\sigma >0$ , a random variable  $X\in \mathbb{R}$  will be called  $\sigma^2$ -sub-Gaussian if for any  $\lambda \in \mathbb{R}$  we have  $\mathbb{E}\left[e^{\lambda X}\right]\leq e^{\frac{\lambda^2}{2}\nu}$ .
We can now state our result for unbounded queries.
Theorem 5.3 (Generalization guarantees for sub-Gaussian queries). Given  $k \in \mathbb{N}$ ;  $\sigma, \epsilon \geq 0$ ;  $0 < \delta \leq \frac{1}{e}$ ; and an analyst  $A$  issuing  $k \sigma^2$ -sub-Gaussian linear queries, if  $M$  is a Gaussian mechanism with noise parameter  $\eta = \Theta \left( \sigma \sqrt{\frac{\sqrt{k}}{n}} \right)$  that receives an iid dataset of size  $n = \Omega \left( \frac{\sigma^2}{\epsilon^2} \sqrt{k} \ln \left( \frac{k \sigma}{\delta \epsilon} \right) \right)$ , then  $M$  is  $(\epsilon, \delta)$ -distribution accurate.
These results extend to the case where the variance (or variance proxy) of each query  $q_{i}$  is bounded by a unique value  $\sigma_{i}^{2}$ , by simply passing this value to the mechanism as auxiliary information and scaling the added noise  $\eta_{i}$  accordingly. Furthermore, using this approach we can quantify the extent to which incorrect bounds affect the accuracy guarantee. Overestimating the bound on a query's variance would increase the error of the response to this query by a factor of square root of the ratio between the assumed and the correct variance, while the error of the other responses would only decrease. On the other hand, underestimating the bound on a query's variance would only decrease the error of the response to this query, while increasing the error of each subsequent query by a factor of the square root of the ratio between the assumed and the correct variance, divided by the number of subsequent queries. A formal version of this claim can be found in Section E.3
The contribution of this paper is two-fold. In Section 3 we provide a tight measure of the level of overfitting of some query with respect to previous responses. In Sections 4 and 5 we demonstrate a toolkit to utilize this measure, and use it to prove new generalization properties of fundamental noise-addition mechanisms. The novelty of the PC definition stems replacing the fixed parameters that appear in the differential privacy definition with a function of the datasets and the query. The definition presented in this paper provides a generalization of zero-concentrated differential privacy, and future work could study similar generalizations of other privacy notions, as discussed in Section B.4.
One small extension of the present work would be to consider queries with range  $\mathbb{R}^d$ . It would also be interesting to extend our results to handle arbitrary normed spaces, using appropriate noise such as perhaps the Laplace mechanism. It might also be possible to relax our assumption that data elements are drawn iid to a weaker independence requirement. Furthermore, it would be interesting to explore an extension from linear queries to general low-sensitivity queries.
We hope that the mathematical toolkit that we establish in Appendix B to analyze our stability notion may find additional applications, perhaps also in context of privacy accounting. Furthermore, the max divergence can be generalized analogously to the "dynamic" generalization of Rényi divergence proposed in this paper (B.9), perhaps suggesting that this approach may be useful in analyzing other mechanisms as well.
Our Covariance Lemma (3.5) shows that there are two possible ways to avoid adaptivity-driven overfitting—by bounding the Bayes factor term, which induces a bound on  $|q(D^v) - q(D)|$ , as we do in this work, or by bounding the correlation between  $q$  and  $K(\cdot, v)$ . This second option suggests interesting directions for future work. For example, to capture an analyst that is non-worst-case in the sense that she "forgets" some of the information that she has learned about the dataset, both the posterior accuracy and the Bayes stability could be redefined with respect to the internal state of the analyst instead of with respect to the full view. This could allow for improved bounds in the style of Zrnic and Hardt [2019].
We gratefully acknowledge productive discussions with Etam Benger, Vitaly Feldman, Yosef Rinott, Aaron Roth, and Tomer Shoham. This work was supported in part by a gift to the McCourt School of Public Policy and Georgetown University, Simons Foundation Collaboration 733792, Israel Science Foundation (ISF) grant 2861/20, and a grant from the Israeli Council for Higher Education. Shenfeld's work was also partly supported by the Apple Scholars in AI/ML PhD Fellowship. Part this work was completed while Ligett was visiting Princeton University's Center for Information Technology Policy.
Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning gaussians and beyond. In Conference on Learning Theory, pages 1075-1076. PMLR, 2022.
An exact version of the bound can be found in Theorem E.10
Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-3820, 2014.
H Victor, Michael J Klass, and Tze Leung Lai. Pseudo-maximization and self-normalized processes. Probability Surveys, 4:172-192, 2007.
Yu-Xiang Wang. Per-instance differential privacy. Journal of Privacy and Confidentiality, 9(1), 2019.
Justin Whitehouse, Aaditya Ramdas, Ryan Rogers, and Steven Wu. Fully-adaptive composition in differential privacy. In International Conference on Machine Learning, pages 36990-37007. PMLR, 2023.
Bin Yang, Issei Sato, and Hiroshi Nakagawa. Bayesian differential privacy on correlated data. In Proceedings of the 2015 ACM SIGMOD international conference on Management of Data, pages 747-762, 2015.
Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pages 4782-4817. PMLR, 2022.
Tijana Zrnic and Moritz Hardt. Natural analysts in adaptive data analysis. In International Conference on Machine Learning, pages 7703-7711. PMLR, 2019.