SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition B.5 (Uniform covering number of a function class [5]). Given a function  $f: X \to \mathbf{R}$  and a sequence  $\vec{x} = (x_1, \ldots, x_n) \in X^n$ , the restriction of  $f$  to  $\vec{x}$  is
$$
f | _ {\vec {x}} = \left(f \left(x _ {1}\right), \dots , f \left(x _ {n}\right)\right) \in \mathbf {R} ^ {n}
$$
The restriction of a class of functions  $\mathcal{F}$  is the set of restrictions of the functions in it:
$$
\mathcal {F} | _ {\vec {x}} = \{f | _ {\vec {x}}: f \in \mathcal {F} \}
$$
The uniform covering number  $\mathcal{N}_{\infty}(\mu, \mathcal{F}, n)$  is the largest covering number of any restriction  $\mathcal{F}|_{\vec{x}}$  to a length- $n$  sequence:
$$
\mathcal {N} _ {\infty} (\mu , \mathcal {F}, n) = \max _ {\vec {x} \in X ^ {n}} \mathcal {N} (\mu , \mathcal {F} | _ {\vec {x}}, \ell_ {\infty})
$$
The Vapnik-Chervonenkis dimension or VC-dimension is another measure of complexity that applies to boolean-valued functions. In the case of boolean-valued functions, the uniform covering number is called the growth function:
Definition B.6 (Growth function [5]). Let  $\mathcal{F}$  be a class of functions  $f:X\to \{0,1\}$ . The growth function of  $\mathcal{F}$  is a function  $\Pi_{\mathcal{F}}:\mathbf{N}\rightarrow \mathbf{N}$  defined by  $\Pi_{\mathcal{F}}(m) = \mathcal{N}_{\infty}(\frac{1}{2},\mathcal{F},m)$ .
$(\frac{1}{2}$  could be replaced by any number less than 1.) Note that  $\Pi_{\mathcal{F}}(m)\leq 2^{m}$  since there are only  $2^{m}$  possible restrictions  $f|_{\vec{x}}$  for a length-  $m$  sequence  $\vec{x}$ .
Definition B.7 (VC Dimension [5]). Let  $\mathcal{F}$  be a class of functions  $f:X\to \{0,1\}$ . The VC dimension of  $\mathcal{F}$ , denoted  $\mathrm{VCdim}(\mathcal{F})$ , is the largest  $d$  such that  $\Pi_{\mathcal{F}}(d) = 2^d$ .
The VC dimension summarizes the behaviour of the growth function in a way captured by the following lemma:
Lemma B.8 (Theorem 3.7 of [5]). Let  $\mathcal{F}$  be a class of boolean-valued functions with VC dimension  $d$ . Then for any  $m \geq 1$ ,  $\Pi_{\mathcal{F}}(m) \leq m^d + 1$ .
We will make use of a simple connection between the complexity of a collection of classifiers  $f$  and of the corresponding set of error functions  $e_f$ :
Lemma B.9. Let  $\mathcal{F}$  be a class of functions  $f: X \to \{0,1\}$ . For  $f \in \mathcal{F}$  define  $e_f: X \times \{0,1\} \to \{0,1\}$  by  $e_f(x,y) = |f(x) - y|$ , and let  $\mathcal{G} = \{e_f: f \in \mathcal{F}\}$ . Then
$$
\operatorname {V C d i m} (\mathcal {G}) = \operatorname {V C d i m} (\mathcal {F})
$$
Proof. Let  $d_{\mathcal{G}} = \mathrm{VCdim}(\mathcal{G})$  and  $d_{\mathcal{F}} = \mathrm{VCdim}(\mathcal{F})$ .
First we'll show  $d_{\mathcal{G}} \geq \mathrm{VCdim}(\mathcal{G})$ . Unpacking the definitions of VC dimension and growth function, we see there is  $\vec{x} = (x_1, \ldots, x_{d_{\mathcal{F}}}) \in X^{d_{\mathcal{F}}}$  such that  $\mathcal{F}|_{\vec{x}}$  includes every one of the  $2^{d_{\mathcal{F}}}$  sequences in  $\{0, 1\}^{d_{\mathcal{F}}}$ . Let  $\vec{z} = ((x_1, 0), \ldots, (x_{d_{\mathcal{F}}}, 0))$ . Observe that for any  $f \in \mathcal{F}$ ,  $f|_{\vec{x}} = e_f|_{\vec{z}} \in \{0, 1\}^{d_{\mathcal{F}}}$ . Therefore  $\mathcal{G}_{\vec{z}}$  includes all  $2^{d_{\mathcal{F}}}$  sequences, and so  $\mathrm{VCdim}(\mathcal{G}) \geq d_{\mathcal{F}}$ .
To show  $d_{\mathcal{G}} \geq \mathrm{VCdim}(\mathcal{G})$ , note there is some  $\vec{z} = ((x_1, y_1), \ldots, (x_{d_{\mathcal{G}}}, y_{d_{\mathcal{G}}})) \in (X \times \{0, 1\}^{d_{\mathcal{G}}}$  such that  $\mathcal{G}|_{\vec{z}}$  includes every sequence in  $\{0, 1\}^{d_{\mathcal{G}}}$ . Let  $\vec{x} = (x_1, \ldots, x_{d_{\mathcal{G}}}) \in X^{d_{\mathcal{G}}}$ . Given a sequence  $s \in \{0, 1\}^{x_{d_{\mathcal{G}}}}$ , we would like to show  $s \in \mathcal{F}|_{\vec{x}}$ . Indeed, define the sequence  $s' \in \{0, 1\}^{x_{d_{\mathcal{G}}}}$  by  $s_i' = |s_i - y_i|$ , and let  $e_f$  be a function in  $\mathcal{G}$  (corresponding to some  $f \in \mathcal{F}$  such that  $e_f|_{\vec{z}} = s'$ ). Then for any  $i \in \{1..d_{\mathcal{G}}\}$ ,  $f(x_i) = |e_f(x_i) - y_i| = |s_i' - y_i| = s_i$ , and so  $f|_{\vec{x}} = s$ . We were able to do this for an arbitrary  $s$ , so  $\mathcal{F}_{\vec{x}}$  includes all  $2^{d_{\mathcal{G}}}$  sequences, so  $\mathrm{VCdim}(\mathcal{F}) \geq d_{\mathcal{G}}$ .
C Linear Queries: Proof of Theorem 5.1
We prove Thm. 5.1, repeated here for convenience:
Theorem 5.1. For any datasets  $D_R \subseteq \mathcal{U} \times X$ ,  $D_S \subseteq \mathcal{U} \times \{1..k\}$ , function  $f: X \times \{1..K\} \to \mathbf{R}$ , privacy parameter  $\varepsilon > 0$ , number of hash buckets  $b$ , and accuracy parameter  $\mu > 0$ , if  $h$  and  $s$  are drawn from a mutually 4-way independent hash family, the following holds. Let  $M$  be an upper bound on  $|f(x,y)|$ ,  $k$  be the range of the value column of  $D_S$ ,  $n = |D_R \bowtie D_S|$ ,  $C = \mathrm{PrivCountSketch}_{b,h,s,\varepsilon}(D_S)$ . Then
$$
\Pr \left[ \frac {| Q ^ {\mathsf {T}} C - \mathrm {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (f) |}{n} > \mu \right] \leq \frac {M ^ {2} | D _ {R} | (k + 1)}{\mu^ {2} n ^ {2}} \left(\frac {2 e ^ {- \varepsilon}}{(1 - e ^ {- \varepsilon}) ^ {2}} + \frac {| D _ {S} |}{b ^ {2}}\right)
$$
The proof follows by applying Chebyshev's inequality to the following lemma:
Lemma C.1. If  $h$  and  $s$  are drawn from a mutually 4-way independent hash family (Definition B.2), then
$$
\mathbf {E} \left[ Q ^ {\mathsf {T}} C \right] = \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f)
$$
$$
\operatorname {V a r} \left[ Q ^ {\mathsf {T}} C \right] \leq | f | _ {\infty} ^ {2} \left| D _ {R} \right| (k + 1) \left(\frac {2 e ^ {- \varepsilon}}{(1 - e ^ {- \varepsilon}) ^ {2}} + \frac {\left| D _ {S} \right|}{b ^ {2}}\right)
$$
where  $k$  is the number of possible values allowed in  $S$ 's value column,  $|f|_{\infty} = \sup_{x,y} f(x,y)$ ,  $C = \operatorname{PrivCountSketch}_{b,h,s,\varepsilon}(D_S)$ , and the randomness is over the choice of  $h$  and  $s$  and the noise from TGeom( $e^{-\varepsilon}$ ) that was added to  $C$ .
Proof. We write  $C = Z + \sum_{(\mathrm{id},y) \in D_S} C(\mathrm{id},y)$  where  $C(\mathrm{id},y)$  is a vector with  $s(\mathrm{id},y)$  at coordinate  $h(\mathrm{id},y)$  and 0 everywhere else, and  $Z$  is a vector of independent TGeom( $e^{-\varepsilon}$ ) random variables.
For every id,  $x,y$  , we have
$$
q (\mathrm {i d}, x, y) ^ {\top} C (\mathrm {i d}, y) = f (x, y) \tag {1}
$$
On the other hand, if  $(\mathrm{id},y^{\prime})\neq (\mathrm{id}^{\prime},y)$  then  $\mathbf{E}[q(\mathrm{id},x,y^{\prime})^{\top}C(\mathrm{id}^{\prime},y)] = 0$  . This follows from the facts that  $s$  is independent of  $h$  and that  $s$  is two-way independent and so  $\mathbf{E}[s(\mathrm{id},y^{\prime})s(\mathrm{id}^{\prime},y)] = 0$  Finally,  $\mathbf{E}[q(\mathrm{id},x,y)^{\top}Z]$  is zero, since  $Z$  is independent of  $Q$  and has expectation zero. So:
$$
\begin{array}{l} \mathbf {E} [ Q ^ {\top} C ] = \mathbf {E} \left[ \left(\sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {y ^ {\prime} = 1} ^ {k} q (\mathrm {i d}, x, y ^ {\prime})\right) ^ {\top} \cdot \left(Z + \sum_ {(\mathrm {i d} ^ {\prime}, y) \in D _ {S}} C (\mathrm {i d} ^ {\prime}, y)\right) \right] \\ = \sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {y ^ {\prime} = 1} ^ {k} \sum_ {(\mathrm {i d} ^ {\prime}, y) \in D _ {S}} \mathbf {E} [ q (\mathrm {i d}, x, y ^ {\prime}) ^ {\top} C (\mathrm {i d} ^ {\prime}, y) ] \\ = \sum_ {(\mathbf {i d}, x, y) \in D _ {R} \rtimes D _ {S}} f (x, y) \\ = \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f) \\ \end{array}
$$
Moving on to the variance, we have
$$
\begin{array}{l} \operatorname {V a r} \left[ Q ^ {\mathsf {T}} C \right] = \mathbf {E} \left[ \left(Q ^ {\mathsf {T}} C - \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f)\right) ^ {2} \right] \\ = \mathbf {E} \left[ (A + B - \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f)) ^ {2} \right] \\ \end{array}
$$
where  $A = \sum_{(\mathbf{id},x)\in D_R}\sum_{y = 1}^k q(\mathbf{id},x,y)^\top Z$  and
$$
B = \sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {y ^ {\prime} = 1} ^ {k} \sum_ {(\mathrm {i d} ^ {\prime}, y) \in D _ {S}} q (\mathrm {i d}, x, y ^ {\prime}) ^ {\mathsf {T}} C (\mathrm {i d} ^ {\prime}, y)
$$
Using (1) can write
$$
\begin{array}{l} B - \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f) = \sum_ {(\mathbf {i d}, x) \in D _ {R}} \sum_ {y ^ {\prime} = 1} ^ {k} \sum_ {(\mathbf {i d} ^ {\prime}, y) \in D _ {S}} [ (\mathbf {i d}, y ^ {\prime}) \neq (\mathbf {i d} ^ {\prime}, y) ] q (\mathbf {i d}, x, y ^ {\prime}) ^ {\top} C (\mathbf {i d} ^ {\prime}, y) \\ = \sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {y ^ {\prime} = 1} ^ {k} \sum_ {(\mathrm {i d} ^ {\prime}, y) \in D _ {S}} \\ [ (\mathrm {i d}, y ^ {\prime}) \neq (\mathrm {i d} ^ {\prime}, y) ] [ h (\mathrm {i d}, y ^ {\prime}) = h (\mathrm {i d} ^ {\prime}, y) ] \cdot \\ s \left(\operatorname {i d}, y ^ {\prime}\right) s \left(\operatorname {i d} ^ {\prime}, y\right) f (x, y) \tag {2} \\ \end{array}
$$
where  $[( \mathrm{id}, y') \neq (\mathrm{id}', y) ]$  denotes a function which is 0 whenever  $\mathrm{id} = \mathrm{id}'$  and  $y' = y$  and 1 otherwise, and similarly  $[h(\mathrm{id}, y') = h(\mathrm{id}', y)]$  is 1 when the hashes are equal. We write,
$\operatorname{Var}[Q^{\mathsf{T}}C] = \mathbf{E}[A^{2}] + 2\mathbf{E}[A(B - \mathrm{SumOverJoin}_{D_{R},D_{S}}(f))] + \mathbf{E}[(B - \mathrm{SumOverJoin}_{D_{R},D_{S}}(f))^{2}]$
and compute each term separately.
$$
\mathbf {E} \left[ A ^ {2} \right] = \sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {y = 1} ^ {k} f (x, y) ^ {2} \frac {2 e ^ {- \varepsilon}}{(1 - e ^ {- \varepsilon}) ^ {2}} \leq \frac {2 k \left| D _ {R} \right| e ^ {- \varepsilon} | f | _ {\infty} ^ {2}}{(1 - e ^ {- \varepsilon}) ^ {2}} \tag {3}
$$
Note that cross terms in  $A^2$  are zero in expectation because each includes a product  $s(\mathrm{id},y)s(\mathrm{id}',y')$  where  $(\mathrm{id},y)\neq (\mathrm{id}',y')$ . The variance of the two-sided geometric distribution comes from Lemma B.1. We have
$$
2 \mathbf {E} [ A (B - \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f)) ] = 0 \tag {4}
$$
since (referring to (2)) each term includes a product of distinct ID-y pairs  $s(\mathrm{id},y)s(\mathrm{id}',y')s(\mathrm{id}'',y'')$  or  $s(\mathrm{id},y)s(\mathrm{id}',y')^2$  which are both 0 in expectation since  $s$  is 3-way independent. Finally, looking at (2), we see
$$
\mathbf{E}[(B - \mathrm{SumOverJoin}_{D_{R},D_{S}}(f))^{2}] = \sum_{\substack{(\mathrm{id},x)\in D_{R}\\ y^{\prime}\in \{1..k\} \\ (\mathrm{id}^{\prime},y)\in D_{S}\\ (\mathrm{id}^{\prime \prime},x^{\prime})\in D_{R}\\ y^{\prime \prime \prime}\in \{1..k\} \\ (\mathrm{id}^{\prime \prime \prime},y^{\prime \prime})\in D_{S}}}C\cdot D\cdot f(x,y)\cdot f(x^{\prime},y^{\prime \prime})
$$
where  $C = [(\mathrm{id},y^{\prime})\neq (\mathrm{id}^{\prime},y)][(\mathrm{id}^{\prime \prime},y^{\prime \prime \prime})\neq (\mathrm{id}^{\prime \prime \prime},y^{\prime \prime})]\operatorname *{Pr}[h(\mathrm{id},y^{\prime}) = h(\mathrm{id}^{\prime},y)\wedge h(\mathrm{id}^{\prime \prime},y^{\prime \prime \prime}) =$ $h(\mathrm{id}^{\prime \prime \prime},y^{\prime \prime})]$  and
$$
D = \mathbf {E} [ s (\mathrm {i d}, y ^ {\prime}) s (\mathrm {i d} ^ {\prime}, y) s (\mathrm {i d} ^ {\prime \prime}, y ^ {\prime \prime \prime}) s (\mathrm {i d} ^ {\prime \prime \prime}, y ^ {\prime \prime}) ]
$$
Since  $s$  is 4-way independent,  $D$  is 1 when every pair in  $\{(\mathrm{id},y^{\prime}),(i\mathrm{d}^{\prime},y),(i\mathrm{d}^{\prime \prime},y^{\prime \prime}),(i\mathrm{d}^{\prime \prime \prime},y^{\prime \prime})\}$  appears at least twice, and 0 otherwise. So,  $C\cdot D$  is  $1 / b^{2}$  when either  $(\mathrm{id},y^{\prime}) = (\mathrm{id}^{\prime \prime},y^{\prime \prime \prime})\neq$ $(\mathrm{id}^{\prime},y) = (\mathrm{id}^{\prime \prime \prime},y^{\prime \prime})$  or  $(\mathrm{id},y^{\prime}) = (\mathrm{id}^{\prime \prime \prime},y^{\prime \prime})\neq (\mathrm{id}^{\prime},y) = (\mathrm{id}^{\prime \prime},y^{\prime \prime \prime})$  and 0 otherwise, so
$$
\begin{array}{l} \mathbf {E} \left[ (B - \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f)) ^ {2} \right] = \frac {1}{b ^ {2}} \sum_ {(\mathrm {i d}, x) \in D _ {R}} \left[ \left(\mathrm {i d}, y ^ {\prime}\right) \neq \left(\mathrm {i d} ^ {\prime}, y\right) \right] f (x, y) ^ {2} \\ y ^ {\prime} \in \{1.. k \} \\ \left(\mathrm {i d} ^ {\prime}, y\right) \in D _ {S} \\ +\frac{1}{b^{2}}\sum_{\substack{(\mathbf{id},x,y)\in D_{R}\rtimes D_{S}\\ (\mathbf{id}^{\prime},x^{\prime},y^{\prime})\in D_{R}\rtimes D_{S}}}[(\mathbf{id},y^{\prime})\neq (\mathbf{id}^{\prime},y)]f(x,y)f(x^{\prime},y^{\prime}) \\ \leq \frac {1}{b ^ {2}} \left(k \left| D _ {R} \right| \left| D _ {S} \right| + \left| D _ {R} \bowtie D _ {S} \right| ^ {2}\right) | f | _ {\infty} ^ {2} \tag {5} \\ \end{array}
$$
Combining (3), (4) and (5) gives:
$$
\begin{array}{l} \operatorname {V a r} \left[ Q ^ {\mathsf {T}} C \right] \leq | f | _ {\infty} ^ {2} \left(\frac {2 k e ^ {- \varepsilon} \left| D _ {R} \right|}{(1 - e ^ {- \varepsilon}) ^ {2}} + \frac {1}{b ^ {2}} \left(k \left| D _ {R} \right| \left| D _ {S} \right| + \left| D _ {R} \bowtie D _ {S} \right| ^ {2}\right)\right) \\ \leq | f | _ {\infty} ^ {2} | D _ {R} | (k + 1) \left(\frac {2 e ^ {- \varepsilon}}{(1 - e ^ {- \varepsilon}) ^ {2}} + \frac {| D _ {S} |}{b ^ {2}}\right) \\ \end{array}
$$
where for the last inequality we use the fact that  $|D_R \bowtie D_S| \leq \min \{|D_R|, |D_S|\}$  since by assumption identities are unique.
D Optimization: Proof and generalization of Theorem 6.4 (Optimization)
Here we state and prove Thm. D.2, a generalization of Thm. 6.4. For the proof of Thm. 6.4 itself, see Section D.2.
Our general optimization theorem states that our optimization method, Algorithm 3 from §6, produces close-to-optimal results under certain conditions. Specifically, we show that the estimated optimal function  $\hat{f}$ , which we define in Definition 6.3 and show how to compute in §6.2, has a value close to the true optimum  $f_{\mathrm{opt}} = \arg \min_{f\in \mathcal{F}}\operatorname {SumOverJoin}_{D_R,D_S}(f)$ . Our bound on the accuracy of  $\hat{f}$  given in Thm. D.2 depends on a quality score (Definition D.1) and the uniform covering number of the function class.
Definition D.1. The quality rating  $\kappa(D_R, D_S, k, b, \varepsilon) > 0$  depends on a pair of datasets  $D_R \subseteq \mathcal{U} \times X$  and  $D_S \subseteq \mathcal{U} \times \{1..k\}$ , an integer  $b > 0$  (number of hash buckets) and a real number  $\varepsilon > 0$  (privacy parameter), and is defined as:
$$
\kappa (D _ {R}, D _ {S}, k, b, \varepsilon) = \frac {| D _ {R} \bowtie D _ {S} | W _ {R} W _ {S}}{k \phi + \psi}
$$
$$
\phi = \frac {\left| D _ {R} \right|}{\left| D _ {R} \bowtie D _ {S} \right|} \quad \psi = \frac {\left| D _ {S} \right|}{\left| D _ {R} \bowtie D _ {S} \right|} - 1
$$
$$
W _ {R} = \min  \left\{1, \frac {b}{5 k | D _ {R} |} \right\} W _ {S} = \min  \left\{\frac {1}{4}, \varepsilon , \frac {1}{1 4} \sqrt {\frac {b}{| D _ {S} |}} \right\}
$$
Theorem D.2 (Generalization of Thm. 6.4). Fix a pair of datasets  $D_R, D_S$ , parameters  $b$  (number of hash buckets) and  $\varepsilon > 0$ , a bound  $M > 0$ , a class  $\mathcal{F}$  of functions  $f: X \times Y \to [-M, M]$ , and an error parameter  $\mu > 0$ . Let  $C = \mathrm{PrivCountSketch}_{b,h,s,\varepsilon}(D_S)$  be the sketch sent by  $S$ , and
$$
f_{\mathrm{opt}} = \operatorname *{arg  min}_{f\in \mathcal{F}}\operatorname {SumOverJoin}_{D_{R},D_{S}}(f)
$$
$$
\hat{f} = \operatorname *{arg  min}_{f\in \mathcal{F}}\Phi_{D_{R},C}(f)
$$
That is,  $f_{\mathrm{opt}}$  and  $\hat{f}$  are the true optimal function and the function returned by Algorithm 3, respectively. Then
$$
\begin{array}{l} \Pr \left[ \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (\hat {f})}{| D _ {R} \bowtie D _ {S} |} > \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (f _ {\text {o p t}})}{| D _ {R} \bowtie D _ {S} |} + \mu \right] \\ \leq (c + 1) \exp \left(- \Omega \left(\frac {\mu^ {2} \kappa \left(D _ {R} , D _ {S} , k , b , \varepsilon\right)}{M ^ {2}}\right)\right) \\ \end{array}
$$
where  $c$  is the uniform covering number
$$
c = \mathcal {N} _ {\infty} \left(\frac {\mu W _ {R} W _ {S}}{7 2 k \phi}, \mathcal {F}, | D _ {R} \bowtie D _ {S} |\right)
$$
and  $\kappa, \phi, W_R, W_S$  are given in Definition D.1.
For reference, we define  $\mathcal{N}_{\infty}$  in Appendix B.4. We prove Thm. D.2 through a careful combination of tail bounds. It has similarities to classic machine learning results relating empirical loss to expected loss on an underlying distribution, but note that we instead relate two empirical values: the sum of a function over a true join, versus what we recover from the sketch.
D.1 Proof of Theorem D.2
Central to the proof is the following bound on the error in the score of a single function:
Lemma D.3. Fix a pair of datasets  $D_R, D_S$ , parameters  $b$  (number of hash buckets) and  $\varepsilon > 0$  and a bound  $M > 0$ . Then there is a real number  $\beta = \beta(D_R, D_S, b, \varepsilon) \geq \frac{1}{18} W_R W_S$ , where  $W_R$  and  $W_S$  are as in Definition D.1, such that for every function  $f: X \times Y \to [-M, M]$ , the value  $\beta^{-1} \Phi_{D_R, C}(f)$  approximates  $\mathrm{SumOverJoin}_{D_R, D_S}(f)$  in the following sense.
Let the hash functions  $h: \mathcal{U} \to \{1..b\}$  and  $s: \mathcal{U} \to \{-1,1\}$  be sampled uniformly at random, and let  $C = \operatorname{PrivCountSketch}_{b,h,s,\varepsilon}(D_S)$ . Then for every  $\mu > 0$ ,
$$
\begin{array}{l} \operatorname * {P r} \left[ \frac {| \beta^ {- 1} \Phi_ {D _ {R} , C} (f) - \mathrm {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (f) |}{| D _ {R} \bowtie D _ {S} |} \geq \mu \right] \\ \leq 4 \exp \left(- \frac {\mu^ {2} \kappa (D _ {R} , D _ {S} , k , b , \varepsilon)}{7 2 0 M ^ {2}}\right) \\ \end{array}
$$
We first show how to prove Thm. D.2 from Lemma D.3, and then prove Lemma D.3 in §D.1.1.
Proof of Thm. D.2 from Lemma D.3. The proof can be summarized as a chain of two approximate equalities and one inequality:
$$
\begin{array}{l} \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (\hat {f}) \approx \beta^ {- 1} \Phi_ {D _ {R}, C} (\hat {f}) \leq \beta^ {- 1} \Phi_ {D _ {R}, C} (f _ {\text {o p t}}) \\ \approx \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} \left(f _ {\text {o p t}}\right) \\ \end{array}
$$
where each of the  $\approx$  is proved by Lemma D.3, and the  $\leq$  follows by the definition of  $\hat{f}$ . Unfortunately, since  $\hat{f}$  depends on some randomness (the noise added for privacy, and the hash functions), we cannot directly prove the first  $\approx$  through an application of Lemma D.3, and so we first find a cover of functions, and apply the Lemma to each function in the cover.
Let  $J \subseteq X \times Y$  be the inner join  $D_R \bowtie D_S$  excluding the iid column. Let  $V \subseteq \mathbf{R}^{|J|}$  be a set of size  $c$  which is an  $\ell_{\infty} \frac{\mu W_R W_S}{72k\phi}$ -cover of  $\mathcal{F}|_J$ .
For any  $f, f' \in \mathcal{F}$  satisfying the "closeness" condition guaranteed by the cover; that is, as long as
$$
\forall (x, y) \in J | f (x, y) - f ^ {\prime} (x, y) | <   \frac {\mu}{4 k \phi}
$$
is true, we have
$$
\begin{array}{l} | \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} (f) - \operatorname {S u m O v e r J o i n} _ {D _ {R}, D _ {S}} \left(f ^ {\prime}\right) | \leq \sum_ {(\mathrm {i d}, x, y) \in D _ {R} \bowtie D _ {S}} | f (x, y) - f ^ {\prime} (x, y) | \\ \leq \left| D _ {R} \bowtie D _ {S} \right| \frac {\mu W _ {R} W _ {S}}{7 2 k \phi} \\ \leq \frac {\mu}{4} \left| D _ {R} \bowtie D _ {S} \right| \tag {6} \\ \end{array}
$$
and similarly, from Definition 6.2 and the triangle inequality, together with Lemma D.8 (which says  $\beta \geq \frac{1}{18} W_R W_S$ ),
$$
\begin{array}{l} \left| \beta^ {- 1} \Phi_ {D _ {R}, C} (f) - \beta^ {- 1} \Phi_ {D _ {R}, C} \left(f ^ {\prime}\right) \right| \leq \beta^ {- 1} \sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {y = 1} ^ {k} \frac {1}{N _ {R} (h (\mathrm {i d} , y))}. \\ | s (\mathrm {i d}, y) \operatorname {c l i p} (C) _ {h (\mathrm {i d}, y)} \left(f (x, y) - f ^ {\prime} (x, y)\right) | \\ \leq \frac {1 8}{W _ {R} W _ {S}} k | D _ {R} | \frac {\mu W _ {R} W _ {S}}{7 2 k \phi} \\ = \frac {\mu}{4} \left| D _ {R} \bowtie D _ {S} \right| \tag {7} \\ \end{array}
$$
(For the last step, recall  $\phi = \frac{|D_R|}{|D_R \rtimes D_S|}$ .)
Let  $E$  be the event that every function in  $V$ , as well as  $f_{\mathrm{opt}}$ , has its value approximated by  $\beta^{-1}$  times its score within  $\frac{\mu}{4}$ :
$$
E = \left\{\forall f \in V \cup \left\{f _ {\mathrm {o p t}} \right\} \left| \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (f)}{| D _ {R} \bowtie D _ {S} |} - \frac {\beta^ {- 1} \Phi_ {D _ {R} , C} (f)}{| D _ {R} \bowtie D _ {S} |} \right| <   \frac {\mu}{4} \right\}
$$
By combining Lemma D.3 (with  $\frac{\mu}{4}$  in the place of  $\mu$ ) with a union bound over every  $f \in V \cup \{f_{\mathrm{opt}}\}$ ,
$$
\Pr \left[ E ^ {c} \right] \leq 4 (| V | + 1) \exp \left(- \frac {\mu^ {2} \kappa \left(D _ {R} , D _ {S} , k , b , \varepsilon\right)}{1 1 5 2 0 M ^ {2}}\right)
$$
Assume  $E$  holds and let  $\hat{f}'$  be a function in  $V$  which is close to  $\hat{f}$ ; that is,  $\forall (x,y) \in J|\hat{f}(x,y) - \hat{f}'(x,y)| < \frac{\mu}{4k\phi}$ . Then, using (6) and (7), and the fact that by definition  $\Phi_{D_R,C}(\hat{f}) \leq \Phi_{D_R,C}(f_{\mathrm{opt}})$ :
$$
\begin{array}{l} \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (\hat {f})}{| D _ {R} \bowtie D _ {S} |} \leq \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (\hat {f} ^ {\prime})}{| D _ {R} \bowtie D _ {S} |} + \frac {\mu}{4} \\ \leq \frac {\beta^ {- 1} \Phi_ {D _ {R} , C} (\hat {f} ^ {\prime})}{| D _ {R} \rtimes D _ {S} |} + 2 \frac {\mu}{4} \\ \leq \frac {\beta^ {- 1} \Phi_ {D _ {R} , C} (\hat {f})}{| D _ {R} \bowtie D _ {S} |} + 3 \frac {\mu}{4} \\ \leq \frac {\beta^ {- 1} \Phi_ {D _ {R} , C} (f _ {\text {o p t}})}{| D _ {R} \bowtie D _ {S} |} + 3 \frac {\mu}{4} \\ \leq \frac {\operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} \left(f _ {\text {o p t}}\right)}{\left| D _ {R} \bowtie D _ {S} \right|} + 4 \frac {\mu}{4} \\ \end{array}
$$
D.1.1 Proof of Lemma D.3
Restated for convenience:
$$
\begin{array}{l} \Pr \left[ \frac {\left| \beta^ {- 1} \Phi_ {D _ {R} , C} (f) - \operatorname {S u m O v e r J o i n} _ {D _ {R} , D _ {S}} (f) \right|}{\left| D _ {R} \bowtie D _ {S} \right|} \geq \mu \right] \\ \leq 4 \exp \left(- \frac {\mu^ {2} \kappa \left(D _ {R} , D _ {S} , k , b , \varepsilon\right)}{7 2 0 M ^ {2}}\right) \\ \end{array}
$$
To prove Lemma D.3, we begin by describing the factor  $\beta$  in (Definition D.7) and bounding it (Lemma D.8). Next, we show  $\Phi_{D_R,C}(f)$  has the correct value in expectation (Lemma D.9), and finally we complete the proof using a combination of tail bounds.
A technical lemma:
Lemma D.4. For any two nondecreasing sequences  $a_1 \geq a_2 \geq \dots \geq a_n$  and  $b_1 \geq b_2 \geq \dots \geq b_n$ ,
$$
\sum_ {i = 1} ^ {n} a _ {i} b _ {i} \geq \frac {1}{n} \left(\sum_ {i = 1} ^ {n} a _ {i}\right) \left(\sum_ {i = 1} ^ {n} b _ {i}\right)
$$
Proof. First, we rewrite each side of the inequality:
$$
\sum_ {i = 1} ^ {n} a _ {i} b _ {i} = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \left(a _ {i} b _ {i} + a _ {j} b _ {j}\right)
$$
$$
\frac {1}{n} \left(\sum_ {i = 1} ^ {n} a _ {i}\right) \left(\sum_ {i = 1} ^ {n} b _ {i}\right) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \left(a _ {i} b _ {j} + a _ {j} b _ {i}\right)
$$
It suffices to show that each term in the first sum is greater than or equal to the corresponding term in the second sum. To see this, note that
$$
\left(a _ {i} b _ {i} + a _ {j} b _ {j}\right) - \left(a _ {i} b _ {j} + a _ {j} b _ {i}\right) = \left(a _ {i} - a _ {j}\right) \left(b _ {i} - b _ {j}\right)
$$