SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f_{\mathrm{opt}}:= \operatorname *{arg  min}_{f\in \mathcal{F}}\operatorname {SumOverJoin}_{D_{R},D_{S}}(f) = \operatorname *{arg  min}_{f\in \mathcal{F}}\sum_{(\operatorname{id},x,y)\in D_{R}\rtimes D_{S}}f(x,y)
$$
in the sense that  $\left| \operatorname{SumOverJoin}_{D_R, D_S}(\hat{f}) - \operatorname{SumOverJoin}_{D_R, D_S}(f_{\mathrm{opt}}) \right|$  is small.
For example, suppose  $D_S$  has labels  $y \in \{0,1\}$  and  $D_R$  has feature vectors  $x \in \mathbf{R}^d$ , and  $\mathbf{R}$  would like to train a logistic regression model on  $D_R \bowtie D_S$ . Then  $\mathbf{R}$  can take  $\mathcal{F} = \{\ell_\theta : \theta \in \mathbf{R}^d\}$ , where  $\ell_\theta$  is the logistic loss  $\ell_\theta(x,y) = y \log(1 + e^{-\theta^\top x}) + (1 - y) \log(1 + e^{\theta^\top x})$ .
In §6.1 and 6.2 we show how to compute the estimate  $\hat{f}$ , assuming we already have a non-private optimization algorithm that accepts weighted inputs. (For an alternative without weights, see Appendix E.) In §6.3 we analyze the loss in accuracy due to privacy.
6.1 Evaluating one function
Our optimization method is based on a score  $\Phi_{D_R,C}(f)$  for individual functions  $f\in \mathcal{F}$ , which R computes based on their dataset  $D_R$  and the sketch  $C$  sent by S. The score is a proxy for the true objective SumOverJoin $_{D_R,D_S}(f)$ . It is less sensitive to noise and hash collisions than the estimate  $Q^{\mathrm{T}}C$  from §5. The cost is that  $\Phi_{D_R,C}(f)$  does not actually approximate SumOverJoin $_{D_R,D_S}(f)$ .
Instead, the two have a linear relationship:  $\Phi_{D_R,C}(f) \approx \beta \operatorname{SumOverJoin}_{D_R,D_S}(f)$  for a positive number  $\beta = \beta(D_R,D_S,b,\varepsilon)$  which does not depend on  $f$ . (We prove this in the appendix in Lemma D.3, as part of the proof of Thm. 6.4.) R does not know  $\beta$ , but does not need to: the optimization method described in §6.2 boils down to computing  $\arg \min_{f \in \mathcal{F}} \Phi_{D_R,C}(f)$ .
There are two differences between the score  $\Phi_{D_R,C}(f)$  and the linear estimate  $Q^{\mathsf{T}}C$ , developed while proving Theorem 6.4. First, we clip coordinates of the sketch  $C$  to be between  $-1$  and  $1$ .
Definition 6.1. For  $x \in \mathbf{R}$ ,  $\operatorname{clip}(x)$  is 1 or -1 if  $x \geq 1$  or  $x \leq -1$ ; otherwise  $\operatorname{clip}(x) = x$ . The clipped version of a vector  $C \in \mathbf{R}^b$  is a vector with clipped entries:  $\operatorname{clip}(C)_i = \operatorname{clip}(C_i)$ .
(Since our sketches have integer coordinates,  $\operatorname{clip}(x)$  equals the sign  $\operatorname{sgn}(x) \in \{-1, 0, 1\}$ . In §7.2 we try a sketch with non-integer coordinates.) Second, when  $h$  hashes more than one row ( $\operatorname{id}, x) \in D_R$  to the same coordinate of  $C$ , we reweight the terms corresponding to those rows.
Definition 6.2. The score of  $f \in \mathcal{F}$  is defined as follows. Let  $N_R(a)$  be the number of possible pairs  $(\mathrm{id}, y)$  where  $\mathrm{id}$  appears in  $D_R$ ,  $y$  is any value in  $\{1..k\}$ , and  $h(\mathrm{id}, y) = a$ :
$$
\Phi_ {D _ {R}, C} (f) := \sum_ {(\mathrm {i d}, x) \in D _ {R}} \sum_ {y = 1} ^ {k} \frac {s (\mathrm {i d} , y) \operatorname {c l i p} (C) _ {h (\mathrm {i d} , y)} f (x , y)}{N _ {R} (h (\mathrm {i d} , y))}
$$
6.2 Performing the optimization
Here we show how R can compute the estimate  $\hat{f} \in \mathcal{F}$ , given their own dataset  $D_R$ , the parameters  $k, b, h, s, \varepsilon$ , and  $C = \mathrm{PrivCountSketch}_{b,h,s,\varepsilon}(D_S)$  downloaded from the repository. In short, our method is simply to find the function  $f \in \mathcal{F}$  that minimizes the score defined in the previous section:
Definition 6.3. The estimated optimal function in  $\mathcal{F}$  is  $\hat{f} \coloneqq \arg \min_{f \in \mathcal{F}} \Phi_{D_R, C}(f)$ .
If  $\mathcal{F}$  is a small finite set, this definition directly leads to an algorithm:  $\mathbf{R}$  computes  $\Phi_{D_R,C}(f)$  for every  $f\in \mathcal{F}$  and chooses the arg min directly. For larger  $\mathcal{F}$ s we will assume  $\mathbf{R}$  has an algorithm WeightedOpt to find the optimal  $f$  on a weighted dataset. A weighted dataset  $\tilde{D}$  is a collection of  $(w,x,y)$  triples where  $w\in \mathbb{R}$  is called the weight and  $(x,y)$  is a valid input to functions in  $\mathcal{F}$ . To be precise, WeightedOpt minimizes
$$
\operatorname {S u m O v e r D a t a s e t} _ {\tilde {D}} (f) := \sum_ {(w _ {\mathrm {i d}, y}, x, y) \in \tilde {D}} w _ {\mathrm {i d}, y} f (x, y)
$$
Many algorithms for training machine learning models admit a weight associated with each training example; such an algorithm could play the role of WeightedOpt. R can now find the optimal function by implementing Algorithm 3. The time complexity of this algorithm is  $O(k|D_R|) + W(k|D_R|)$ , where  $W(n)$  is the run-time complexity of WeightedOpt on a dataset with  $n$  elements.
By Definitions 6.2 and 6.3, the output of WeightedOpt is identical to the estimated optimal function  $\hat{f}$ . The question of whether  $\hat{f}$  is any good is addressed in §6.3.
Algorithm 3 ReceiverOptimize
1: Download  $C = \mathrm{PrivCountSketch}_{b,h,s,\varepsilon}(D_S)$  and the parameters  $k, b, h, s$ .
2:  $w_{\mathrm{id},y}\gets s(\mathrm{id},y)\mathrm{clip}(C_{h(\mathrm{id},y)}) / N_R(h(\mathrm{id},y)$  for  $y = 1,\ldots ,k$
3: Construct  $\tilde{D} = \tilde{D}(b, h, s, D_R, C)$  by creating  $(w_{\mathrm{id},y}, x, y)$  for each element  $(\mathrm{id}, x) \in D_R$  and  $y \in \{1..k\}$ . // note:  $|\tilde{D}| = k|D_R|$
4: Return WeightedOpt( $\tilde{D}, \mathcal{F}$ )
6.3 Analysis
Here we analyze the accuracy of our method in the context of learning a classifier. We state and prove a more general result in Appendix D, and prove the below results in Appendices D.2 and D.3.
We bound the classification error of a model trained using our method on features from  $D_R$  and binary labels from  $D_S$  (so  $k = 2$ ). Let  $\mathcal{F}$  be a set of classifiers  $f: X \to \{0,1\}$ , and for  $f \in \mathcal{F}$  let
$L_{f}\in \mathbf{N}$  be the number of classifications errors on  $D_R\bowtie D_S$  . To apply Algorithm 3, we convert this to an equivalent problem: let  $\mathcal{G}$  be the set of corresponding error functions  $e_f(x,y) = |f(x) - y|$  for  $f\in \mathcal{F}$  , so  $L_{f} = \mathrm{SumOverJoin}_{D_{R},D_{S}}(e_{f})$  . Then we can ask Algorithm 3 to find the best  $e_f\in \mathcal{G}$  which corresponds to the best classifier  $f$
Theorem 6.4. For any datasets  $D_R \subseteq \mathcal{U} \times X$ ,  $D_S \times \mathcal{U} \times \{1..k\}$ , function class  $\mathcal{F}$ , privacy parameter  $\varepsilon > 0$ , number of hash buckets  $b$  and accuracy parameter  $\mu > 0$ , if  $h$  and  $s$  are drawn uniformly at random from the set of all functions  $\mathcal{U} \times \{1..k\} \rightarrow \{1..b\}$ , the following holds. Let  $f_{\mathrm{opt}}$  be the classifier that minimizes  $L_f$ , and let  $\hat{f}$  be the classifier corresponding to the function  $e_{\hat{f}}$  output by Algorithm 3. Let  $d$  be the VC dimension of  $\mathcal{F}$ ,  $n = |D_R \rtimes D_S|$ , and
$$
\phi = \frac {| D _ {R} |}{n} \quad \psi = \frac {| D _ {S} |}{n} - 1 \quad W _ {R} = \min \left\{1, \frac {b}{5 k | D _ {R} |} \right\} \quad W _ {S} = \min \left\{\frac {1}{4}, \varepsilon , \frac {1}{1 4} \sqrt {\frac {b}{| D _ {S} |}} \right\}
$$
$$
\operatorname * {P r} \left[ \frac {L _ {\hat {f}} - L _ {f _ {\mathrm {o p t}}}}{n} > \mu \right] \leq \exp \left(- \Omega \left(\frac {W _ {R} W _ {S}}{\phi + \psi} \mu^ {2} n - d \log n\right)\right)
$$
For reference, we define VC dimension in Appendix B.4. Adding assumptions can simplify the bound. This corollary highlights the dependence on the privacy parameter  $\varepsilon$ :
Corollary 6.5. Assume  $D_R$  and  $D_S$  have the same set of identities, so the join is "perfect" in the sense that  $|D_R| = |D_S| = |D_R \bowtie D_S|$ . Let  $n = |D_R \bowtie D_S|$  and let  $d$  be the VC dimension of  $\mathcal{F}$ . Assume also that  $b = \Omega(n)$ . Let  $f_{\mathrm{opt}}$  be the classifier that minimizes  $L_f$ , and let  $\hat{f}$  be the classifier corresponding to the function  $e_{\hat{f}}$  output by Algorithm 3. Then for any  $\mu > 0$ ,
$$
\operatorname * {P r} \left[ \frac {L _ {\hat {f}} - L _ {f _ {\mathrm {o p t}}}}{n} > \mu \right] \leq \exp (- \Omega (\mu^ {2} n \varepsilon - d \log n))
$$
The goal of the experiments is to evaluate the accuracy of our solutions and quantify the effect of noise and join size on downstream tasks. To simulate parties S and R, for each dataset, we add a unique iid to each row, then split its columns into  $D_R$  with all of the features and  $D_S$  with labels. In §7.1 we learn a function that, given features  $x$ , predicts labels  $y \in \{1..k\}$  using Algorithm 3 and a private count sketch. In Appendix F we estimate a joint distribution using Algorithm 2. In both cases, we find that the test error drops as we increase  $\varepsilon$  (meaning the privacy requirement is relaxed) or the sketch dimension, or as the join size increases ( $D_R$ ,  $D_S$  have more ids in common). In §7.2 we compare our results with the private linear queries protocol of Zhao et al. [39]. We also conduct experiments with multi-way joins and with non-unique identifiers in Appendix I.
Datasets. Two datasets are used in the experiments. (1) UC Irvine Adult dataset [1]. We predict if income is greater than 50K based on categorical features such as age, workclass and education. (2) Extended MNIST, for which we predict labels based on images of handwritten digits.
7.1 Optimization: Prediction Findings
We evaluate our method with two machine learning tasks: logistic regression on the UCI Adult dataset, and a neural net for Extended MNIST. We capped the cross-entropy loss by replacing predicted probabilities  $\hat{y}$  with  $\max \{\hat{y}, 10^{-4}\}$ . This is necessary since Algorithm 3 can produce negative weights, which would lead to divergence in the optimization if loss were not bounded.
In each case, we conduct several experiments varying the privacy parameter $^3$ $\varepsilon$  and the sketch dimension  $b$ . All accuracy numbers are measured on the test set directly, with no simulated join. The right halves of Figs. 2 and 3 show the results, with different lines for different sketch dimensions  $b$ . A dashed line shows the test accuracy of a model trained directly on the training set, without privacy.
In practice, cryptographically secure hash functions should give the same guarantees.
Ponomareva et al. [29, §5.2.1] suggest  $\varepsilon \leq 1$  gives a "strong" privacy guarantee and  $\varepsilon < 10$  is "reasonable".
Figure 2: Test accuracy of logistic regression on the UCI Adult salary prediction task (§7.1).
Figure 3: Test accuracy of a deep neural net on the Extended MNIST dataset (§7.1).
Imperfect joins, where  $D_R$  has some IDs not present in  $D_S$ , are simulated by randomly removing most of the rows in the training set, and then adding some back only to  $D_R$ . The left halves of Figs. 2 and 3 show the results. The performance drops as this fraction increases, which is expected since unmatched rows in  $D_R$  are assigned random labels that add noise to the training. In practice, party R would benefit by filtering  $D_R$  to contain only identities likely to be in  $D_S$ , e.g., if  $D_S$  contains people in Ohio, R should also filter  $D_R$  to Ohio. Note that on the right panel of Fig. 3, when  $\varepsilon = 1$ , the test accuracy is around  $92\%$ , however, on the left panel of Fig. 3, the performance drops to  $80\%$  for  $\varepsilon = 1$ . This is due to the smaller training set size (240K on the right vs 20K on the left).
We also train a neural net on the EMNIST by merge dataset consisting of 760K images of handwritten digits and uppercase and lowercase letters, with 47 classes. We conduct this experiment to investigate the effects of large label domain sizes, i.e., large  $k$ . Fig. 4, shows test accuracy as a function of the number of labels. For each run, we randomly chose out of 47 classes, and applied our method with  $\varepsilon = 1$ . The figure shows that, as expected, the performance degrades significantly as  $k$  increases, but the method is still viable with  $k = 45$ . Note that in this experiment, the size of the dataset, and thus the join size, changes as we change  $k$ .
Our results are consistent with the dependence of Thm. D.2 on the parameters  $\varepsilon$ ,  $b$ , and  $\phi = |D_R| / |D_R \bowtie D_S|$ : test accuracy improves as  $\varepsilon$  or  $b$  increase, or  $|D_R| / |D_R \bowtie D_S|$  or  $k$  decrease. Training details can be found in Appendix G.
7.2 Comparison with other methods
To our knowledge, there is no past work on non-interactive private joins in the repository setting. However, one can replace our simplified private count sketch (Definition 3.5) with other private sketches. To this end, we incorporate Zhao et al's [39] private linear sketches into our method. Party S generates a sketch of (id, value) pairs as before, but builds the sketch using Algorithms 1 and 3 from Zhao et al. instead of our Definition 3.5. Then, line 2 of Algorithm 3 is replaced with LinearSketchQuery from Zhao et al.'s Algorithm 2:  $w_{\mathrm{id},y} \gets \mathrm{clip}(\mathrm{LinearSketchQuery}(\mathrm{id},y))$ .
The EMNIST experiment is run with different combinations of Zhao et al.'s parameters  $\gamma, \beta, \rho$ . Their protocol satisfies  $\rho$ -zCDP; we use Proposition 1.4 of [8] to compare our results by equating  $\rho = \frac{1}{2}\varepsilon^2$ . The number of hash buckets  $b$  is related to  $\gamma$  as  $b = 1 / \gamma$  and the number of hash functions (one for us) is given by  $\log(2 / \beta)$ .
Figure 4: Test accuracy as the domain size of the labels,  $k$ , increases.
Figure 5: Comparisons with other baselines. The definition of neighboring datasets is different for LabelDP and ours:  $2\varepsilon_{\mathrm{ours}} \approx \varepsilon_{\mathrm{theirs}}$ , roughly. We have taken this into account in the right panel.
Zhao et al. describe two sketch implementations, Count-Min and CountSketch. Count-Min sketch is a random classifier on EMNIST for  $\varepsilon \lesssim 10$ , possibly because Count-Min is designed to avoid underestimating frequencies, whereas in our case it is more important to be correct on average. CountSketch leads to better results, Fig. 5. For  $\beta = 0.3$ , there is one hash function, as with our method, and the remaining differences between the two methods is the kind of added noise (Gaussian vs two-sided geometric), and that LinearSketchQuery omits the denominator  $N(h(\mathrm{id},y))$  on Line 2 of our Algorithm 3. Our method provides better results, especially for smaller  $\varepsilon$ .
The accuracy of the model is roughly proportional to the number of correct vs. incorrect labels reconstructed in the join. For the corresponding values of  $\varepsilon$  and  $(\rho, \beta, \gamma)$  in the two methods, the number of incorrect labels reconstructed is fewer for the two-sided geometric noise than Gaussian. Gaussian noise has thinner tails which is helpful for reducing the probability of large error values but not helpful in our application where queries are in any case clipped to the range  $[-1, 1]$ .
We also compare with LabelDP [21], which is a Local DP protocol that protects the labels. LabelDP is not designed to perform joins, and so we run it without splitting the training set. Results are shown in the right panel of Fig. 5. The definition of "neighbouring datasets" differs in the two settings: we use the "add/remove" definition where a single row is added or removed, but in LabelDP's definition, a single value is changed. To account for this, we double the privacy budget for LabelDP, so for example at  $\varepsilon = 1$  on the  $x$ -axis, we actually give LabelDP a privacy budget of  $\varepsilon = 2$ .
The LabelDP method benefits from not needing to perform a join, and so we expect it to perform better than our method, where the parties must join using a non-interactive protocol. We were surprised to see our method perform better for small  $\varepsilon$ , and do not understand why this happens.
Future Work. This work focused on when the sender has a single private label column. To broaden applicability, data with multiple columns and data of different types, such as real-valued or text, deserve consideration. New privacy challenges arise in the streaming setting where organizations continuously receive new data and discard outdated data.
We would like to thank Joan Feigenbaum, Bill Horne, Yonatan Naamad, Aaron Roth, Doug Terry, Tal Wagner, and the anonymous reviewers for helpful feedback, comments and suggestions that improved this paper.
[1] Adult. Adult. UCI Machine Learning Repository, 1996.
[2] F. Alda and B. I. Rubinstein. The bernstein mechanism: Function release under differential privacy. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[3] N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. Journal of Computer and system sciences, 58(1):137-147, 1999.
[4] N. Alon, P. B. Gibbons, Y. Matias, and M. Szegedy. Tracking join and self-join sizes in limited storage. Journal of Computer and System Sciences, 64(3):719-747, 2002.
[5] M. Anthony and P. L. Bartlett. Neural network learning: Theoretical foundations. Cambridge university press, 1999.
[6] R. Bassily, K. Nissim, U. Stemmer, and A. Thakurta. Practical locally private heavy hitters. J. Mach. Learn. Res., 21:16-1, 2020.
[7] J. Blocki, A. Blum, A. Datta, and O. Sheffet. The johnson-lindenstrauss transform itself preserves differential privacy. In 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science, pages 410-419. IEEE, 2012.
[8] M. Bun and T. Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In M. Hirt and A. Smith, editors, Theory of Cryptography, pages 635-658, Berlin, Heidelberg, 2016. Springer Berlin Heidelberg. ISBN 978-3-662-53641-4.
[9] M. Byali, H. Chaudhari, A. Patra, and A. Suresh. Flash: Fast and robust framework for privacy-preserving machine learning. Cryptology ePrint Archive, Paper 2019/1365, 2019. URL https://eprint.iacr.org/2019/1365. https://eprint.iacr.org/2019/1365.
[10] M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. In International Colloquium on Automata, Languages, and Programming, pages 693-703. Springer, 2002.
[11] H. Chaudhari, A. Choudhury, A. Patra, and A. Suresh. Astra: High throughput 3pc over rings with application to secure prediction. In Proceedings of the 2019 ACM SIGSAC Conference on Cloud Computing Security Workshop, CCSW'19, page 81-92, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450368261. doi: 10.1145/3338466.3358922. URL https://doi.org/10.1145/3338466.3358922.
[12] S. G. Choi, D. Dachman-Soled, M. Kulkarni, and A. Yerukhimovich. Differentially-private multi-party sketching for large-scale statistics. Cryptology ePrint Archive, 2020.
[13] E. F. Codd. Further normalization of the data base relational model. Research Report / RJ / IBM / San Jose, California, RJ909, 1971.
[14] B. Coleman and A. Shrivastava. A one-pass private sketch for most machine learning tasks. arXiv preprint arXiv:2006.09352, 2020.
[15] B. Coleman and A. Shrivastava. A one-pass distributed and private sketch for kernel sums with applications to machine learning at scale. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3252-3265, 2021.
[16] G. Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms, 55(1):58-75, 2005.
[34] S. Wagh, D. Gupta, and N. Chandran. Securenn: 3-party secure computation for neural network training. Proceedings on Privacy Enhancing Technologies, 2019:26 - 49, 2019. URL https://apisemantic scholar.org/CorpusID:78090069.
[35] S. Wagh, S. Tople, F. Benhamouda, E. Kushilevitz, P. Mittal, and T. Rabin. Falcon: Honest-majority maliciously secure framework for private deep learning. Proceedings on Privacy Enhancing Technologies, 2021(1):188-208, Nov 2020. ISSN 2299-0984. doi: 10.2478/popets-2021-0011. URL http://dx.doi.org/10.2478/popets-2021-0011.
[36] D. Xu, S. Yuan, and X. Wu. Achieving differential privacy in vertically partitioned multiparty learning. In 2021 IEEE International Conference on Big Data (Big Data), pages 5474-5483. IEEE, 2021.
[37] A. C.-C. Yao. How to generate and exchange secrets. In 27th Annual Symposium on Foundations of Computer Science (sfcs 1986), pages 162-167. IEEE, 1986.
[38] Q. Ye, H. Hu, X. Meng, and H. Zheng. Privkv: Key-value data collection with local differential privacy. In 2019 IEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May 19-23, 2019, pages 317-331. IEEE, 2019. doi: 10.1109/SP.2019.00018. URL https://doi.org/10.1109/SP.2019.00018.
[39] F. Zhao, D. Qiao, R. Redberg, D. Agrawal, A. E. Abbadi, and Y.-X. Wang. Differentially private linear sketches: Efficient implementations and applications. Neurips, 2022.
A Illustration of Algorithm 1
Figure 6 provides a visual depiction of the private count sketch that the sender uploads to the public repository. The noise values  $Z_{\varepsilon}$  are sampled from TGeom( $e^{-\varepsilon}$ ) (Def. 3.3). For example, if we choose  $\varepsilon = 1$  (which gives a "strong" privacy guarantee [29, §5.2.1]), then the variance of each  $Z_{\varepsilon}$  is about 1.84.
Figure 6: Visual depiction of Algorithm 1
B Some Useful Definitions and Theorems
B.1 More on the two-sided geometric distribution
We repeat the definition for convenience:
$\mathrm{TGeom}(\alpha)$  is the same distribution as the difference between two independent geometric random variables,  $\mathrm{Geom}(p = 1 - \alpha)$ . This fact can be verified directly from the probability mass functions, and it provides a convenient way to sample from the distribution when using a library that provides the geometric distribution. From this it immediately follows that the expectation and variance of  $\mathrm{TGeom}(\alpha)$  are given by:
Lemma B.1. For  $Z \sim \mathrm{TGeom}(\alpha)$ ,  $\mathbf{E}[Z] = 0$  and  $\operatorname{Var}[Z] = \frac{2\alpha}{(1 - \alpha)^2}$ .
B.2 Mutually independent hash families
Definition B.2. A distribution  $H$  over pairs of hash functions  $h: U \to \{1..b\}, s: U \to \{-1, 1\}$  is a mutually  $k$ -way independent hash family iff for any fixed distinct inputs  $x_1, \ldots, x_k$  and outputs  $y_1, \ldots, y_k \in \{1..b\}$  and  $z_1, \ldots, z_k \in \{-1, 1\}$ ,
$$
\Pr [ \forall i \in \{1.. k \} h (x _ {i}) = y _ {i} \wedge s (x _ {i}) = z _ {i} ] = \left(\frac {1}{2 b}\right) ^ {k}
$$
In other words, the outputs  $h(x_1), \ldots, h(x_k), s(x_1), \ldots, s(x_k)$  are distributed uniformly over  $\{1..b\}^k \times \{-1, 1\}^k$ .
B.3 McDiarmid's Inequality
Theorem B.3 (McDiarmid's Inequality). Let  $f: \mathcal{X}_1 \times \dots \times \mathcal{X}_n \to \mathbf{R}$  have the property that for every index  $i \in \{1..n\}$  and every pair of sequences  $\vec{x}, \vec{y} \in \mathcal{X}_1 \times \dots \times \mathcal{X}_n$  that differ only in the  $i$ -th coordinate,  $|f(\vec{x}) - f(\vec{y})| \leq c_i$ . Then if  $X_1 \sim \mathcal{X}_1, \ldots, X_n \sim \mathcal{X}_n$  are independent random variables, then for any  $\mu \geq 0$ ,
$$
\operatorname * {P r} [ | f (X _ {1}, \ldots , X _ {n}) - \mathbf {E} [ f (X _ {1}, \ldots , X _ {n}) ] | \geq \mu ] \leq 2 \exp \left(- \frac {2 \mu^ {2}}{\sum_ {i = 1} ^ {n} c _ {i}}\right)
$$
B.4 Measuring the complexity of a function class
When analyzing an optimization algorithm, it is useful to have a measure of the complexity of a class of functions. One such measure is the uniform covering number.
Definition B.4 (Covering number in  $\mathbf{R}^n$  [5]). An  $\ell_{\infty}$ $\mu$ -cover of a set  $S \subseteq \mathbf{R}^n$  is a set  $C \subseteq S$  such that  $\forall x \in S \exists c \in C |x - c|_{\infty} < \mu$ . The  $\ell_{\infty}$ $\mu$ -covering number of  $S$ , denoted  $\mathcal{N}(\mu, S, \ell_{\infty})$ , is the cardinality of the smallest such  $C$ .