SlowGuess/NAACL_2022_OCR huggingface dataset:
Nearest-neighbor projections are arguably natural, computationally efficient, and useful in practice. Although we have considered them in the Euclidean space so far, they can be defined for an arbitrary distance metric  $m$  on  $\mathbb{R}^d$ . As before, consider an embedding of a lexicon of size  $V$  into  $\mathbb{R}^d$  and the corresponding one-hot vectors in  $\{0,1\}^V$ . We call  $d$ -proj a nearest-neighbor projection operator w.r.t.  $m$  if it maps each  $x \in \mathbb{R}^d$  to the one-hot vector in  $\{0,1\}^V$  that corresponds to the lexicon item whose embedding is closest to  $x$  under metric  $m$  (breaking ties arbitrarily).
Proposition 1 Every nearest-neighbor projection operator, under any metric, maps infinitely many elements of  $\mathbb{R}^d$ , forming one or more continuous subspaces, to every one-hot vector in  $\{0,1\}^V$ .
A proof is included in Appendix B.1. In effect, the projection operators induce a clustering of the space of continuous prompts  $\mathbb{R}^{d\times L}$  into regions that have the same discrete projection (Fig.6).
The infinite-to-one mapping aspect is not limited to the class of nearest-neighbor projection operators. It is rather an inherent property of the interaction between continuous and discrete spaces, and
holds for a broader family consisting of all but a negligible portion of possible projection operators:
Proposition 2 Let  $\mathbb{D}$  denote the space of all projection operators that map  $\mathbb{R}^d$  to one-hot vectors in  $\{0,1\}^V$ . Let  $d$ -proj be a random projection drawn uniformly from  $\mathbb{D}$ . Then, with probability 1,  $d$ -proj maps infinite elements of  $\mathbb{R}^d$  to every one-hot vector in  $\{0,1\}^V$ .
Proof of Prop. 1: Let  $c_{i} \in \mathbb{R}^{d}$  for  $i \in \{1, \ldots, V\}$  be fixed vectors (denoting the embedding of words in a lexicon of size  $V$ ). Let  $e_{i} \in \{0, 1\}^{V}$  denote the one-hot vector with 1 in the  $i$ -th position and 0 elsewhere. Since  $d$ -proj is a nearest-neighbor projection operator w.r.t.  $m$ , by definition it maps  $x \in \mathbb{R}^{d}$  to  $e_{i}$  whenever  $x$  is closest to  $c_{i}$ , i.e.,  $i = \arg \min_{j} m(x, c_{j})$  (breaking ties arbitrarily).
Let  $S_{i} \subseteq \mathbb{R}^{d}$  denote the pre-image of  $e_{i}$ , i.e., the elements that the nearest-neighbor projection  $d$ -proj maps to the  $i$ -th one-hot vector. By definition,  $c_{i} \in S_{i}$ . Let  $d' = \min_{j} m(c_{i}, c_{j}) > 0$  denote the distance of  $c_{i}$  to the nearest  $c_{j}$  w.r.t. the metric  $m$ . Consider the subspace  $C_{i} = \{x \mid m(x, c_{i}) < d'/2\}$ . By design, we have  $C_{i} \subseteq S_{i}$ . Further, moving  $x$  by some small distance  $\epsilon$  (w.r.t.  $m$ ) to another point  $x'$  changes its distance to  $c_{i}$  only by at most  $\epsilon$  (by the triangle inequality property of  $m$ ). This implies that if  $\epsilon$  is chosen to be small enough such that  $m(x, c_{i}) + \epsilon < d'/2$ , then  $x'$  must also be in  $C_{i}$ . In other words, if  $x \in C_{i}$ , then, for a small enough  $\epsilon$ , the entire  $\epsilon$ -neighborhood of  $x$  is also in  $C_{i}$ . It follows that  $C_{i}$  is an open subset of  $R^{d}$  and thus contains infinitely many elements forming a continuous subspace. Hence  $S_{i}$ , which contains  $C_{i}$ , e.g., also has infinite elements in one or more continuous subspaces.
Proof of Prop. 2: For simplicity, assume  $V = 2$ . A projection operator  $d$ -proj  $\in \mathbb{D}$  can then be fully characterized by the subset  $S \subseteq \mathbb{R}^d$  that it maps to any one arbitrarily chosen one-hot vector. Choosing  $d$ -proj uniformly at random from  $\mathbb{D}$  thus amounts to choosing the subset  $S$  uniformly at random from  $\mathbb{R}^d$ . We show that the probability of choosing an  $S$  such that  $|S|$  is finite, is 0. (The same argument applies to  $|\mathbb{R} \setminus S|$  being finite.)
To see this, let  $\mathbb{S}_i$  denote the set of all (finite) subsets of  $\mathbb{R}^d$  that have size exactly  $i$ . First, observe that the probability of choosing an  $S$  that lies in  $\mathbb{S}_0 \cup \mathbb{S}_1$  (i.e., a subset of  $\mathbb{R}^d$  that has at
$^{12}$ https://en.wikipedia.org/wiki/Metric_(mathematics)
3642 12
most 1 element) is 0; this is a degenerate case in the underlying continuous probability space. Second, for any  $i \geq 2$ ,  $\mathbb{S}_i$  has the same "size" (in the measure theoretic sense) as  $\mathbb{S}_1$ , because one can construct an injective map from either one to the other—which follows from the fact that they both have the same cardinality as the set  $\mathbb{R}$ .<sup>13</sup> Lastly, the space  $\mathbb{S}$  of all finite subsets of  $\mathbb{R}^d$  is the countable union  $\cup_i \mathbb{S}_i$  of disjoint sets. Therefore,  $\operatorname*{Pr}[S \in \mathbb{S}] = \sum_i \operatorname*{Pr}[S \in \mathbb{S}_i] = 0$ .
This can be proved using the rules of cardinal multiplication applied to  $\mathbb{S}_i$  viewed as (a subset of) the Cartesian product of  $\mathbb{S}_1$  with itself,  $i$  times.
3643 13