SlowGuess/NAACL_2022_OCR huggingface dataset:
In the numerical study, we randomly sample 10,000 inputs texts and obtain their corresponding neighbor texts from Equation (4) with  $M = 10$  using each of the 12 hidden layers in BERT as the encoder  $\mathcal{H}_{\text{encoder}}$  under  $L_{2}$  norm. Figure 7 shows the precision result from different encoders, where we observe that the fifth hidden layer has the highest precision in terms of both category and subcategory, which is consistent with the intuition that the middle layer is a trade-off of token-alike and output-alike inputs. In the following experiment, we adopt the fifth layer as the encoder. In general, when no external labels are provided, we may choose a different encoder depending on the use case.
E Ablation Study on Kernel Function
In Case III, we conduct an ablation study with different choices of kernel functions using different mask ratio to find out if different kernel yields different learning speed:
1. Radial basis function kernel (RBF):
$$
k _ {R B F} (a, b) = e x p (- | | a - b | | _ {2} / l ^ {2}),
$$
where larger hyper-parameter  $l$  indicates lower impact from neighbors and vice versa. In the numerical study, we choose  $l = 2$  based on the range of embedding  $a$  and  $b$ .
2. Cubic kernel (Cubic):
$$
k _ {C u b i c} (a, b) = (\gamma a ^ {T} b + c _ {0}) ^ {d},
$$
where  $\gamma = 7$ ,  $c_{0} = 0$  and  $d = 3$ , smaller  $\gamma$  means lower impact from neighbors.
3. Cosine kernel (Cosine):
$$
k _ {C o s} (a, b) = a ^ {T} b / | | a | | | b | | |
$$
This kernel function havee no parameter.
4. Laplacian kernel (Laplacian):
$$
k _ {L a p l a c i a n} (a, b) = e x p (- | | a - b | | _ {1} / l ^ {2}),
$$
in the numerical study, we choose  $l = 2$ .
5. L2 norm based similarity  $(L2)$ :
$$
k _ {L 2} (a, b) = 1 / c l i p (\left\| a - b \right\| _ {2}, \lambda_ {l e f t}, \lambda_ {r i g h t}),
$$
where  $clip(\cdot, \lambda_{left}, \lambda_{right})$  denotes clip function with  $\lambda_{left} = 0.3$  and  $\lambda_{right} = 3$  as clip boundary in numerical study.
6. Indicator function based similarity (Indicator):
$$
k _ {I n d i c a t o r} (a, b) = \mathbb {I} (a, b),
$$
where  $\mathbb{I}(\cdot ,\cdot)$  denotes indicator function.
The results are shown in Figure 8. We observe that no single kernel function outperforms all other kernel functions under all mask ratios in this study. Indicator function shows a good performance when the masked ratio is greater than  $10\%$ , while RBF kernel shows a good performance when the masked ratio is smaller than  $5\%$ . This can due to the reason that the indicator function only aggregates identical words and this conservative manner helps when we lost most important words.