SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and  $a_i - a_j$  and  $b_i - b_j$  are either both nonnegative or both nonpositive, depending on whether  $i \leq j$ .
Lemma D.5. Let  $X = Z + \sum_{i=1}^{n} S_i$  where  $Z, S_1, \ldots, S_n$  are independent random variables, and  $Z \sim \mathrm{TGeom}(e^{-\varepsilon})$  for some  $\varepsilon > 0$  and each  $S_i$  is drawn from the uniform distribution on  $\{-1,1\}$ . Then
$$
\mathbf {E} [ \operatorname {c l i p} (X) ] = 0
$$
$$
\mathbf {E} [ \operatorname {c l i p} (X + 1) ] > \min  \left\{\frac {\varepsilon}{3}, \frac {1}{6 \lceil \sqrt {n} \rceil + 6} \right\}
$$
Proof. The distribution of  $X$  is symmetric around 0:  $\operatorname*{Pr}[X = x] = \operatorname*{Pr}[X = -x]$  for any  $x$ . Thus,  $\operatorname*{Pr}[X \leq -1] = \operatorname*{Pr}[X \geq 1]$ , so  $\mathbf{E}[\mathrm{clip}(X)] = \operatorname*{Pr}[X \geq 1] - \operatorname*{Pr}[X \leq 1] = 0$ .
It remains to bound  $\mathbf{E}[\mathrm{clip}(X + 1)]$ , which equals  $\operatorname*{Pr}[X \geq 0] - \operatorname*{Pr}[X \leq -2]$ . Once again using the symmetry of  $X$ , we see  $\operatorname*{Pr}[X \leq -2] = \operatorname*{Pr}[X \geq 2]$ , and so
$$
\mathbf {E} [ \operatorname {c l i p} (X + 1) ] = \Pr [ X \geq 0 ] - \Pr [ X \geq 2 ] = \Pr [ X \in \{0, 1 \} ]
$$
To avoid some repetition, let  $T = \sum_{i=1}^{n} S_i$ . Since  $T$  has the same parity as  $n$ , we'll pay some attention to parities in what follows. Let  $n = 2k + r$  where  $r \in \{0,1\}$  is  $n$ 's parity. Then
$$
\begin{array}{l} \Pr [ X \in \{0, 1 \} ] = \sum_ {z = - \infty} ^ {\infty} \Pr [ T = 2 z + r ] \Pr [ Z - 2 z - r \in \{0, 1 \} ] \\ \geq \sum_ {z = - z _ {0} - r} ^ {z _ {0}} \Pr [ T = 2 z + r ] \Pr [ Z - 2 z - r \in \{0, 1 \} ] \\ \end{array}
$$
where  $z_0$  will be chosen later. For an integer  $z$ , let  $a_{z} = \operatorname{Pr}[T = 2z + r]$  and  $b_{z} = \operatorname{Pr}[Z - 2z - r \in \{0,1\}]$ , so
$$
\mathbf {E} [ \operatorname {c l i p} (X + 1) ] = \Pr [ X \in \{0, 1 \} ] \geq \sum_ {z = - z _ {0}} ^ {z _ {0}} a _ {z} b _ {z}
$$
Consider the sequence  $z = 0, -1, 1, -2, 2, -3, 3, \ldots$ : that is 0 and then  $-i, i$  for each positive integer  $i$ . Both  $(a_z)$  and  $(b_z)$  are nondecreasing with respect to this order:
$$
a _ {0} \geq a _ {- 1} \geq a _ {1} \geq a _ {- 2} \geq a _ {2} \geq \dots
$$
$$
b _ {0} \geq b _ {- 1} \geq b _ {1} \geq b _ {- 2} \geq b _ {2} \geq \dots
$$
This nondecreasing property of  $(a_{z})$  follows because  $(T + n) / 2$  is a binomial random variable with mean  $n / 2$ , and so its probability mass falls away monotonically from the mode(s) at  $(T + n) / 2 = \lfloor n / 2 \rfloor, \lceil n / 2 \rceil$ , that is,  $T = -r, r$ . The nondecreasing property for  $(b_{z})$  follows from the fact that the probability mass of  $Z$  falls away monotonically from the mode at  $Z = 0$ .
Using the above order, we may apply Lemma D.4 to the sequences  $(a_{z}),(b_{z})$ , to get:
$$
\begin{array}{l} \mathbf {E} [ \operatorname {c l i p} (X + 1) ] = \sum_ {z = - z _ {0} - r} ^ {z _ {0}} a _ {z} b _ {z} \\ \geq \frac {1}{2 z _ {0} + r + 1} \left(\sum_ {z = - z _ {0} - r} ^ {z _ {0}} a _ {z}\right) \left(\sum_ {z = - z _ {0} - r} ^ {z _ {0}} b _ {z}\right) \\ = \frac {1}{2 z _ {0} + r + 1} A B \\ \end{array}
$$
where  $A = \operatorname{Pr}[T \in [-2z_0 - r, 2z_0 + r]]$ ,  $B = \operatorname{Pr}[Z \in [-2z_0 - r, 2z_0 + r + 1]]$ . From the probability mass function of  $\mathrm{TGeom}(e^{-\varepsilon})$  we directly compute  $B = 1 - e^{-(2z_0 + r + 1)\varepsilon}$ . We apply Hoeffding's inequality to see
$$
A > 1 - 2 \exp \left(- \frac {\left(2 z _ {0} + r\right) ^ {2}}{2 n}\right)
$$
Taking  $z_0 = \lceil \sqrt{n} \rceil$  gives  $A > 1 - 2e^{-2}$  and  $B = 1 - e^{-(2\lceil \sqrt{n} \rceil + r + 1)\varepsilon}$ . If  $2\lceil \sqrt{n} \rceil + r + 1 \geq \varepsilon$ , then  $B \geq 1 - e^{-1}$ , so we have
$$
\mathbf {E} [ \mathrm {c l i p} (X + 1) ] > \frac {(1 - 2 e ^ {- 2}) (1 - e ^ {- 1})}{2 \lceil \sqrt {n} \rceil + r + 1} \geq \frac {1}{6 \lceil \sqrt {n} \rceil + 6}
$$
Otherwise  $(2\lceil \sqrt{n}\rceil +r + 1 <   \varepsilon)$  we use the fact that for  $x <   0,e^{x} <   1 + x + \frac{x^{2}}{2}$  to get  $B > C\varepsilon -\frac{1}{2} C^2\varepsilon^2$  where  $C = 2\lceil \sqrt{n}\rceil +r + 1 <   \varepsilon$  , and so
$$
\begin{array}{l} \mathbf {E} [ \operatorname {c l i p} (X + 1) ] > \frac {1}{C} (1 - 2 e ^ {- 2}) \left(C \varepsilon - \frac {1}{2} C ^ {2} \varepsilon^ {2}\right) \\ = (1 - 2 e ^ {- 2}) (1 - \frac {1}{2} C \varepsilon) \varepsilon \\ > \frac {1}{3} \varepsilon \\ \end{array}
$$
Lemma D.6. If  $X$  is a binomially-distributed random variable  $X \sim \mathrm{Binom}(n,p)$  then
$$
\operatorname * {P r} [ X \geq \max  \{5 n p, 1 \} ] <   \frac {1}{3}
$$
Proof. Let  $P = \operatorname{Pr}[X \geq \max\{5np, 1\}] = \min\{\operatorname{Pr}[X \geq 5np], \operatorname{Pr}[X \geq 1]\}$ .
If  $np < \frac{1}{3}$ , then using Markov's inequality,
$$
P \leq \Pr [ X \geq 1 ] \leq \mathbf {E} [ X ] = n p <   \frac {1}{3}
$$
Otherwise  $(np\geq \frac{1}{3})$  , we apply the Chernoff bound in the following form:
$$
\operatorname * {P r} [ X > (1 + \delta) {\bf E} [ X ] ] <   \left(\frac {e ^ {\delta}}{(1 + \delta) ^ {1 + \delta}}\right) ^ {{\bf E} [ X ]}
$$
taking  $\delta = 4$  to get
$$
P \leq \operatorname * {P r} [ X \geq 5 n p ] <   \left(\frac {e ^ {4}}{5 ^ {5}}\right) ^ {n p} \leq \left(\frac {e ^ {4}}{5 ^ {5}}\right) ^ {\frac {1}{3}} <   \frac {1}{3}
$$
Definition D.7. Given datasets  $D_R, D_S$  and parameters  $b, \varepsilon$ , we define
$$
\beta (D _ {R}, D _ {S}, b, \varepsilon) = \frac {\mathbf {E} [ \Phi_ {D _ {R} , C} (\mathbf {1}) ]}{\mathrm {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (\mathbf {1})}
$$
where  $\mathbf{1}$  is the constant function  $\mathbf{1}(x,y) = 1$ , and the expectation is taken over the random choice of hash functions  $h, s$  and the noise from  $\mathrm{TGeom}(e^{-\varepsilon})$  that party S added to  $C$ . (Note that  $\mathrm{SumOverJoin}_{D_R,D_S}(\mathbf{1}) = |D_R \bowtie D_S|$ .)
Lemma D.8.  $\beta (D_R,D_S,b,\varepsilon)\geq \frac{1}{18} W_RW_S$  where  $W_{R}$  and  $W_{S}$  are as in Definition D.1.
Proof. For any  $\mathfrak{id} \in \mathcal{U}$  and  $i \in \{1..k\}$ , define
$$
T \left(D _ {R}, C, \mathrm {i d}, i\right) := \frac {1}{N _ {R} (h (\mathrm {i d} , i))} s (\mathrm {i d}, i) \operatorname {c l i p} (C) _ {h (\mathrm {i d}, i)}
$$
$$
\begin{array}{l} \beta (D _ {R}, D _ {S}, b, \varepsilon) = \frac {\mathbf {E} [ \Phi_ {D _ {R} , C} (\mathbf {1}) ]}{| D _ {R} \bowtie D _ {S} |} \\ = \frac {1}{\left| D _ {R} \bowtie D _ {S} \right|} \sum_ {\left(\mathrm {i d}, x\right) \in D _ {R}} \sum_ {i = 1} ^ {k} \mathbf {E} \left[ T \left(D _ {R}, C, \mathrm {i d}, i\right) \right] \\ \end{array}
$$
It suffices to show that for every  $(\mathrm{id},x)\in D_R$  and  $i\in \{1..k\}$ , if  $(\mathrm{id},y_i)\in D_S$ , then
$$
\mathbf {E} \left[ T \left(D _ {R}, C, \operatorname {i d}, i\right) \right] \geq \frac {1}{1 8} W _ {R} W _ {S} \tag {8}
$$
and otherwise  $\mathbf{E}[T(D_R,C,\mathrm{id},i)] = 0$
Suppose  $(\mathfrak{id},y_i)\notin D_S$ . Then  $s(\mathfrak{id},i)$  is independent of  $C$  and  $N_{R}(h(\mathfrak{id},i))$ , and so the probability distribution of  $T(D_R,C,\mathfrak{id},i)$  is symmetric about 0, so  $\mathbf{E}[T(D_R,C,\mathfrak{id},i)] = 0$  as required.
Now suppose  $(\mathrm{id},y_i)\in D_S$ . Let  $N_{R} = N_{R}(h(\mathrm{id},i))$  and define  $N_{S}$  to be the number of rows  $(\mathrm{id}',y_{i'})\in D_S$  such that  $h(\mathrm{id}',i') = h(\mathrm{id},i)$ . Note  $N_{R}\geq 1$  because  $(\mathrm{id},x)\in D_R$ , and  $N_{S}\geq 1$  because  $(\mathrm{id},y_i)\in D_S$ . Then
$$
\mathbf {E} [ T (D _ {R}, C, \mathbf {i d}, i) ] = \sum_ {n _ {R} = 1} ^ {| D _ {R} |} \sum_ {n _ {S} = 1} ^ {| D _ {S} |} \operatorname * {P r} [ N _ {R} = n _ {R} \land N _ {S} = n _ {S} ] E _ {n _ {R}, n _ {S}}
$$
$$
E _ {n _ {R}, n _ {S}} = \mathbf {E} [ T (D _ {R}, C, \mathrm {i d}, i) | N _ {R} = n _ {R} \wedge N _ {S} = n _ {S} ]
$$
Conditioned on a particular value  $N_S = n_S \geq 1$ ,  $s(\mathrm{id}, i) C_{h(\mathrm{id}, i)}$  is distributed as
$$
1 + Z + \sum_ {j = 1} ^ {n _ {S} - 1} S _ {j}
$$
where  $Z \sim \mathrm{TGeom}(e^{-\varepsilon})$  is the noise party S added to  $C$  for privacy, and  $S_{1}, \ldots, S_{N_{S} - 1}$  are independent uniform samples from  $\{-1, 1\}$ , each having the value  $s(\mathrm{id}, i)s(\mathrm{id}', i')$  for some other  $(\mathrm{id}', i') \in D_{S}$  for which  $h(\mathrm{id}', i') = h(\mathrm{id}, i)$ . So, by Lemma D.5,
$$
E _ {n _ {R}, n _ {S}} > \frac {1}{n _ {R}} \min  \left\{\frac {\varepsilon}{3}, \frac {1}{6 \lceil \sqrt {n _ {S} - 1} \rceil + 6} \right\} \tag {9}
$$
In particular,  $E_{n_R,n_S} \geq 0$ , so in our effort to lower bound  $\mathbf{E}[T(D_R,C,\mathrm{id},i)]$ , we are free to focus our attention on a subset of the terms:
$$
\begin{array}{l} \mathbf {E} [ T (D _ {R}, C, \mathbf {i d}, i) ] \geq \sum_ {n _ {R} = 1} ^ {\lfloor B _ {R} \rfloor} \sum_ {n _ {S} = 1} ^ {\lfloor B _ {S} \rfloor} \Pr [ N _ {R} = n _ {R} \wedge N _ {S} = n _ {S} ] E _ {n _ {R}, n _ {S}} \\ \geq \Pr [ N _ {R} \leq B _ {R} \wedge N _ {S} \leq B _ {S} ] E _ {\min } \\ \end{array}
$$
where we choose
$$
B _ {R} = \min \left\{| D _ {R} |, 1 + \max \left\{\frac {5 k | D _ {R} |}{b}, 1 \right\} \right\}
$$
$$
B _ {S} = \min \left\{| D _ {S} |, 1 + \max \left\{\frac {5 | D _ {S} |}{b}, 1 \right\} \right\}
$$
and  $E_{\mathrm{min}}$  is the minimum value of  $E_{n_R,n_S}$  over all  $n_R \leq B_R, n_S \leq B_S$ .
Recall that to complete our proof it suffices to prove (8). For this, it suffices to show that
$$
\Pr \left[ N _ {R} \leq B _ {R} \wedge N _ {S} \leq B _ {S} \right] \geq \frac {1}{3} \tag {10}
$$
$$
E _ {\min } \geq \frac {1}{6} W _ {R} W _ {S} \tag {11}
$$
(10) follows from Lemma D.6 since  $N_R - 1 \sim \mathrm{Binom}(n = k|D_R| - 1, p = \frac{1}{b})$  and  $N_S - 1 \sim \mathrm{Binom}(n = |D_S| - 1, p = \frac{1}{b})$ .
It remains to prove (11).
From (9) we have
$$
E _ {\min } \geq \frac {1}{\left\lfloor B _ {R} \right\rfloor} \min  \left\{\frac {\varepsilon}{3}, \frac {1}{6 \left\lceil \sqrt {\left\lfloor B _ {S} \right\rfloor - 1} \right\rceil + 6} \right\}
$$
By considering the cases  $\frac{5k|D_R|}{b} \geq 1$ ,  $\frac{5k|D_R|}{b} \leq 1$  separately it can be seen that  $\frac{1}{[B_R]} \geq \min\{\frac{1}{2}, \frac{b}{10k|D_R|}\} = \frac{1}{2} W_R$ . It remains only to be shown that
$$
\min \left\{\frac {\varepsilon}{3}, \frac {1}{6 \lceil \sqrt {\lfloor B _ {S} \rfloor - 1} \rceil + 6} \right\} \geq \frac {1}{3} W _ {S}
$$
We consider two cases. If  $\frac{5|D_S|}{b} \leq 1$ , then  $B_{S} = 2$  and
$$
\min \left\{\frac {\varepsilon}{3}, \frac {1}{6 \lceil \sqrt {\lfloor B _ {S} \rfloor - 1} \rceil + 6} \right\} = \min \left\{\frac {\varepsilon}{3}, \frac {1}{1 2} \right\}
$$
Otherwise  $\left(\frac{5|D_S|}{b} > 1\right)$  we have  $B_{S} = 1 + \frac{5|D_{S}|}{b} > 2$ , and so
$$
\begin{array}{l} \min \left\{\frac {\varepsilon}{3}, \frac {1}{6 \lceil \sqrt {\lfloor B _ {S} \rfloor - 1} \rceil + 6} \right\} \geq \min \left\{\frac {\varepsilon}{3}, \frac {1}{6 \left\lceil \sqrt {\frac {5 | D _ {S} |}{b}} \right\rceil + 6} \right\} \\ \geq \min \left\{\frac {\varepsilon}{3}, \frac {1}{1 8 \sqrt {\frac {5 | D _ {S} |}{b}}} \right\} \\ \geq \min  \left\{\frac {\varepsilon}{3}, \frac {1}{4 2} \sqrt {\frac {b}{| D _ {S} |}} \right\} \\ \end{array}
$$
In both cases, we find
$$
\min  \left\{\frac {\varepsilon}{3}, \frac {1}{6 \lceil \sqrt {\lfloor B _ {S} \rfloor - 1} \rceil + 6} \right\} \geq \min  \left\{\frac {1}{1 2}, \frac {\varepsilon}{3}, \frac {1}{4 2} \sqrt {\frac {b}{| D _ {S} |}} \right\} = \frac {1}{3} W _ {S}
$$
Lemma D.9. For any datasets  $D_R, D_S$  and parameters  $b, \varepsilon$ , for uniformly random functions  $h: \mathcal{U} \to \{1..b\}$  and  $s: \mathcal{U} \to \{-1, 1\}$ ,
$$
\mathbf {E} \left[ \Phi_ {D _ {R}, C} (f) \right] = \beta \left(D _ {R}, D _ {S}, b, \varepsilon\right) \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f)
$$
where  $C = \mathrm{PrivCountSketch}_{b,h,s,\varepsilon}(D_S)$ .
$$
\begin{array}{l} \mathbf {E} [ \Phi_ {D _ {R}, C} (f) ] = \sum_ {(\mathbf {i d}, x) \in D _ {R}} \sum_ {i = 1} ^ {k} f (x, y _ {i}) \mathbf {E} \left[ \frac {1}{N _ {R} (h (\mathbf {i d} , i))} s (\mathbf {i d}, i) \operatorname {c l i p} (C) _ {h (\mathbf {i d}, i)} \right] \\ = \sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {i = 1} ^ {k} f (x, y _ {i}) \mathbf {E} [ T (D _ {R}, C, \mathrm {i d}, i) ] \\ \end{array}
$$
with  $T$  as defined in the proof of Lemma D.8. As observed in that proof, when  $(\mathrm{id},y_i)\notin D_S$  we have  $\mathbf{E}[T(D_R,C,\mathrm{id},i) = 0$ , and so,
$$
\mathbf {E} \left[ \Phi_ {D _ {R}, C} (f) \right] = \sum_ {\left(\mathrm {i d}, x, y _ {i}\right) \in D _ {R} \bowtie D _ {S}} f \left(x, y _ {i}\right) \mathbf {E} \left[ T \left(D _ {R}, C, \mathrm {i d}, i\right) \right] \tag {12}
$$
By the definition of  $\beta$
$$
\sum_ {(\mathrm {i d}, x, y _ {i}) \in D _ {R} \bowtie D _ {S}} \mathbf {E} [ T (D _ {R}, C, \mathrm {i d}, i) ] = \beta \left(D _ {R}, D _ {S}, b, \varepsilon\right) | D _ {R} \bowtie D _ {S} | \tag {13}
$$
Now, observe that  $T(D_{R}, C, \mathrm{id}, i)$  has exactly the same distribution for every  $(\mathrm{id}, x, y_{i}) \in D_{R} \rtimes D_{S}$ : it is distributed as  $\frac{1}{N_{R}} (1 + Z + \sum_{i=1}^{N_{S}} S_{i})$  as described in the proof of Lemma D.8, where the random variables  $N_{R}$  and  $N_{S}$  are counts of hash collisions with  $(\mathrm{id}, i)$ . So,  $\mathbf{E}[T(D_{R}, C, \mathrm{id}, i)]$  has the same value for any  $(\mathrm{id}, x, y_{i}) \in D_{R} \rtimes D_{S}$ . From (13) we see that value is exactly  $\beta(D_{R}, D_{S}, b, \varepsilon)$ , and so from (12) we have
$$
\begin{array}{l} \mathbf {E} \left[ \Phi_ {D _ {R}, C} (f) \right] = \sum_ {\left(\mathrm {i d}, x, y _ {i}\right) \in D _ {R} \bowtie D _ {S}} f (x, y _ {i}) \beta \left(D _ {R}, D _ {S}, b, \varepsilon\right) \\ = \beta \left(D _ {R}, D _ {S}, b, \varepsilon\right) \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f) \\ \end{array}
$$
Proof of Lemma D.3. We begin with an account of the randomness affecting  $\Phi_{D_R,C}(f)$ , as being in three parts: hash values  $h(\mathrm{id},i)\in \{1..b\}$ , hash values  $s(\mathrm{id},i)\in \{-1,1\}$ , and the noise values  $Z\sim \mathrm{TGeom}(e^{-\varepsilon})$  that were added by party S to each coordinate of the private count sketch  $C$ . We define a sequence of random variables representing each of these parts.
Let  $\mathrm{id}_1, \mathrm{id}_2, \ldots, \mathrm{id}_{|D_R|} \in \mathcal{U}$  be the identities that appear in  $D_R$ . Let  $s = |D_S| - |D_R \bowtie D_S|$  (the number of identities that appear in  $D_S$  but not  $D_R$ ) and let  $(\mathrm{id}_1', y_{i_1}), \ldots, (\mathrm{id}_n', y_{i_s}) \in D_S$  be the entries in  $D_S$  which do not participate in the join  $D_R \bowtie D_S$ . We define the following random variables:
-  $H_{i,j} \coloneqq h(\mathrm{id}_i, j) \sim \mathrm{Unif}(\{1..b\})$ , for  $i = 1, \dots, |D_R|$  and  $j \in \{1..k\}$ ;
-  $S_{i,j} \coloneqq s(\mathrm{id}_i, j) \sim \mathrm{Unif}(\{-1, 1\})$ , for  $i = 1, \dots, |D_R|$  and  $j \in \{1..k\}$ ;
-  $H_i' \coloneqq h(\mathfrak{id}_j', i_j) \sim \mathrm{Unif}(\{1..b\})$  for  $j = 1, \ldots, s$ ;
-  $S_{i}^{\prime} \coloneqq s(\mathbf{id}_{j}^{\prime}, i_{j}) \sim \mathrm{Unif}(\{-1, 1\})$  for  $j = 1, \ldots, s$ ; and
-  $Z_{i} \sim \mathrm{TGeom}(e^{-\varepsilon})$  for  $i \in \{1..b\}$ .
We will complete the proof in two steps: first, we use Hoeffding's inequality to deal with the variables  $Z_{i}$ , and then we use McDiarmid's inequality to deal with the other four kinds of random variable  $(H,S,H^{\prime},S^{\prime})$ . The value  $C$ , defined next, connects the two steps together.
Let  $A$  be the expected value of  $\Phi_{D_R,C}(f)$  with respect to the random variables  $Z_{i}$ :
$$
A = \mathbf {E} _ {Z _ {1}, \dots , Z _ {b}} [ \Phi_ {D _ {R}, C} (f) ]
$$
Note that  $A$  is a random variable still depending on the values  $H_{i,j}, S_{i,j}, H_i', S_i'$ , and
$$
\mathbf {E} [ A ] = \mathbf {E} \left[ \Phi_ {D _ {R}, C} (f) \right.
$$
$$
\begin{array}{l} \Pr \left[ \frac {| \beta^ {- 1} \Phi_ {D _ {R} , C} (f) - \mathrm {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (f) |}{| D _ {R} \bowtie D _ {S} |} \geq \mu \right] \\ = \Pr \left[ \left| \Phi_ {D _ {R}, C} (f) - \beta \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f) \right| \geq \beta \mu \mid D _ {R} \bowtie D _ {S} \right] \\ \leq \underbrace{\Pr[|\Phi_{D_{R},C}(f) - A|\geq\frac{\beta\mu|D_{R}\bowtie D_{S}|}{2}]}_{(*)} \\ + \underbrace {\Pr [ | A - \beta \operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (f) | \geq \frac {\beta \mu | D _ {R} \bowtie D _ {S} |}{2} ]} _ {(\#)} \\ \end{array}
$$
We'll begin with  $(\ast)$ . The probability is taken over the choice of  $h, s$  and  $Z$ , but we will look at the worst-case hash functions  $h$  and  $s$ :
$$
(*) \leq \max  _ {h _ {0}, s _ {0}} \Pr [ | \Phi_ {D _ {R}, C} (f) - A | > \frac {\beta \mu | D _ {R} \bowtie D _ {S} |}{2} | h = h _ {0}, s = s _ {0} ]
$$
For fixed functions  $h_0, s_0$ , we can write
$$
\Phi_ {D _ {R}, C} (f) = \sum_ {a = 1} ^ {b} X _ {a}
$$
where  $X_{a} = 0$  when  $N_R(a) = 0$  and otherwise
$$
X _ {a} = \frac {\operatorname {c l i p} (C) _ {a}}{N _ {R} (a)} \sum_ {(\mathbf {i d}, x) \in D _ {R}} \sum_ {i = 1} ^ {k} [ h (\mathbf {i d}, i) = a ] s (\mathbf {i d}, i) f (x, y _ {i})
$$
Here,  $[h(\mathrm{id},i) = a]$  is the indicator with value 1 if  $h(\mathrm{id},i) = a$  and 0 otherwise.  $X_{a}$  is bounded between  $-M$  and  $M$ , since  $N_{R}(a)$  equals the number of terms in the sum for which  $[h(\mathrm{id},i) = a] = 1$ . Therefore, changing the value of one variable  $Z_{a}$  while leaving the others the same changes the corresponding  $X_{a}$  by at most  $2M$ . Now,  $\Phi_{D_R,C}(f)$  depends on at most  $k|D_R|$  random variables  $Z_{a}$ , namely the ones for which at least one pair  $(\mathrm{id},i)$  hashes to  $a$  for some  $\mathrm{id} \in D_R$ . We apply Hoeffding's inequality to get:
$$
(*) \leq 2 \exp \left(- \frac {\beta^ {2} \mu^ {2} | D _ {R} \bowtie D _ {S} | ^ {2}}{8 k | D _ {R} | M ^ {2}}\right)
$$
To bound (#) we use McDiarmid's inequality (Thm. B.3). We will view  $A$  as being a function of the random variables  $(H_{i,j})_{i,j}$ ,  $(S_{i,j})_{i,j}$ ,  $(H_i')_i$ ,  $(S_i')_i$ , described above and call this function  $r$ :
$$
r \left(\left(H _ {i, j}\right) _ {i, j}, \left(S _ {i, j}\right) _ {i, j}, \left(H _ {i} ^ {\prime}\right) _ {i}, \left(S _ {i} ^ {\prime}\right) _ {i}\right) := A = \mathbf {E} _ {Z _ {1}, \dots , Z _ {b}} \left[ \Phi_ {D _ {R}, C} (f) \right]
$$
Changing the value of one hash value  $H_{i,j}$  or  $H_{i}'$  affects at most two of the variables  $X_{a}$ , and so changes (#) by at most  $4M$ . Changing the value of one hash value  $S_{i,j}$  or  $S_{i}'$  affects at most one of the variables  $X_{a}$ , and so changes (#) by at most  $2M$ . Summing up the squares of these difference values, we get  $(4^{2} + 2^{2})(k|D_{R}| + s)M^{2} = 20M^{2}$ , so applying Thm. B.3, we have:
$$
(\#) \leq 2 \exp \left(- \frac {\beta^ {2} \mu^ {2} | D _ {R} \bowtie D _ {S} | ^ {2}}{4 0 (k | D _ {R} | + s) M ^ {2}}\right)
$$
Combining these bounds with Lemma D.8, we have
$$
\begin{array}{l} (*) + (\#) \leq 4 \exp \left(- \frac {\beta^ {2} \mu^ {2} | D _ {R} \bowtie D _ {S} | ^ {2}}{4 0 (k | D _ {R} | + s) M ^ {2}}\right) \\ \leq 4 \exp \left(- \frac {W _ {R} W _ {S} \mu^ {2} | D _ {R} \bowtie D _ {S} |}{7 2 0 (k \phi + \psi) M ^ {2}}\right) \\ \end{array}
$$
D.2 Proof of Theorem 6.4
We prove Thm. 6.4, repeated here for convenience. Recall that  $\mathcal{F}$  is a set of classifiers, and for a function  $f\in \mathcal{F},f:X\to \{0,1\}$ , we define the corresponding error function  $e_{f}(x,y) = |f(x) - y|$ ; for the full context, see §6.3.
Theorem 6.4. For any datasets  $D_R \subseteq \mathcal{U} \times X$ ,  $D_S \times \mathcal{U} \times \{1..k\}$ , function class  $\mathcal{F}$ , privacy parameter  $\varepsilon > 0$ , number of hash buckets  $b$  and accuracy parameter  $\mu > 0$ , if  $h$  and  $s$  are drawn uniformly at random from the set of all  $^4$  functions  $\mathcal{U} \times \{1..k\} \to \{1..b\}$ , the following holds. Let  $f_{\mathrm{opt}}$  be the classifier that minimizes  $L_f$ , and let  $\hat{f}$  be the classifier corresponding to the function  $e_{\hat{f}}$  output by Algorithm 3. Let  $d$  be the VC dimension of  $\mathcal{F}$ ,  $n = |D_R \rtimes D_S|$ , and
We prove the theorem by combining Thm. D.2 with a standard result relating covering number to VC dimension (Lemma B.8), as well as Lemma B.9 to relate the VC dimension of  $\mathcal{F}$  to that of the class  $\mathcal{G}$  of error functions  $e_f$ .
Proof. Taking  $M = 1$ , Thm. D.2 (applied using the function class  $\mathcal{G} = \{e_f \mid f \in \mathcal{F}\}$ ) says:
$$
\begin{array}{l} \Pr \left[ \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} \left(e _ {\hat {f}}\right)}{n} > \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} \left(e _ {f _ {\text {o p t}}}\right)}{n} + \mu \right] \\ \leq (c + 1) \exp (- \Omega (\mu^ {2} \kappa (D _ {R}, D _ {S}, k, b, \varepsilon))) \\ \end{array}
$$
Where  $c$  is the uniform covering number
$$
c := \mathcal {N} _ {\infty} \left(\frac {\mu W _ {R} W _ {S}}{7 2 k \phi}, \mathcal {G}, n\right)
$$
For any  $f \in \mathcal{F}$ , we have  $\mathrm{SumOverJoin}_{D_R, D_S}(e_f) = L_f$ , so we can rewrite that as:
$$
\Pr \left[ \frac {L _ {\hat {f}} - L _ {f _ {\mathrm {o p t}}}}{n} > \mu \right] \leq (c + 1) \exp (- \Omega (\mu^ {2} \kappa (D _ {R}, D _ {S}, k, b, \varepsilon)))
$$
All that remains is to bound the uniform covering number  $c$ . Note that the range of functions in  $\mathcal{G}$  is  $\{0,1\}$ , and the  $\ell_{\infty}$  distance between points in  $\{0,1\}^n$  is always 1. An examination of Definitions B.4 and B.5 then shows:
If  $\frac{\mu W_R W_S}{72k\phi} \geq 1$  then  $c \leq 1$ .
- If  $\frac{\mu W_R W_S}{72k\phi} < 1$  then for any  $\vec{y} \in (X \times \{0,1\})^n$  we have  $\mathcal{N}\left(\frac{\mu W_R W_S}{72k\phi}, \vec{y}, \ell_{\infty}\right) = |\mathcal{G}_{\vec{y}}|$ , and so  $c = \max_{\vec{y} \in (X \times \{0,1\})^n} |\mathcal{G}_{\vec{y}}| = \Pi_{\mathcal{G}}(n)$ .
$$
c \leq \Pi_ {\mathcal {G}} (n)
$$
Apply Lemma B.8.
$$
\leq n ^ {\operatorname {V C d i m} (\mathcal {G})} + 1
$$
Apply Lemma B.9 (recall  $d = \mathrm{VCdim}(\mathcal{F})$ ).
$$
= n ^ {d} + 1
$$
$$
\begin{array}{l} \operatorname * {P r} \left[ \frac {L _ {\hat {f}} - L _ {f _ {\mathrm {o p t}}}}{n} > \mu \right] \leq (n ^ {d} + 2) \exp (- \Omega (\mu^ {2} \kappa (D _ {R}, D _ {S}, k, b, \varepsilon))) \\ = \exp (- \Omega (\mu^ {2} \kappa (D _ {R}, D _ {S}, k, b, \varepsilon) - d \log n)) \\ \end{array}
$$
D.3 Proof of Corollary 6.5
We prove Cor. 6.5, repeated here for convenience:
$$
\Pr \left[ \frac {L _ {\hat {f}} - L _ {f _ {\mathrm {o p t}}}}{n} > \mu \right] \leq \exp (- \Omega (\mu^ {2} n \varepsilon - d \log n))
$$
Proof. By Thm. 6.4, it suffices to show
$$
\frac {W _ {R} W _ {S}}{\phi + \psi} = \Omega (\varepsilon) \tag {14}
$$
Where  $W_{R}, W_{S}, \phi, \psi$  are given in the theorem statement. Indeed, since by assumption  $|D_{R}| = |D_{S}| = n$ , we have
$$
\phi = 1, \psi = 0
$$
and since by assumption  $b = \Omega(n) = \Omega(|D_R|) = \Omega(|D_S|)$ , we have
$$
W _ {R} = \Omega (1), W _ {S} = \Omega (\varepsilon)
$$
and (14) follows.
E Training a binary classifier with a difference sketch
If R wishes to train a binary classification model to predict labels in  $\{-1,1\}$  that appear in  $D_{S}$ , the approach described in this section might give better results than the one described in §6.2. The reconstructed dataset it produces does not have weights, so will work with a broader class of optimization algorithms. Also, the lack of negative weights avoids a problem with unbounded loss functions: since the approach in §6.2 can produce a dataset with negative weights, if the loss function is not bounded above, then the training loss may not be bounded below, causing the optimization algorithm to diverge.