SlowGuess/NAACL_2022_OCR huggingface dataset:
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316.
Ivan Vulic, Edoardo M. Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pages 7222-7240.
Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv preprint arXiv:1909.10430.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019. BERT-based lexical substitution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3368-3373, Florence, Italy. Association for Computational Linguistics.
A Derivation of Eqn. (4) and (5)
By plugging Eqn. (2) and (3) into Eqn. (4), we have
$$
\begin{array}{l} \operatorname {P M I} (x; w | \mathbf {c}) \approx \log \frac {\hat {p} (x | w , \mathbf {c})}{p (x | \mathbf {c})} \\ = \log \frac {\operatorname {s o f t m a x} \left(\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x}\right)}{\operatorname {s o f t m a x} \left(\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x}\right)} \\ = \log \frac {e ^ {\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}} - \log \frac {e ^ {\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}} \\ = \left(\boldsymbol {h} _ {s} - \boldsymbol {h} _ {c}\right) ^ {\top} \boldsymbol {v} _ {x} + \log \frac {\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}}. \tag {14} \\ \end{array}
$$
The second term in Eqn. (14) can be denoted as
$$
\varphi (w, \mathbf {c}) = \log \frac {\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}},
$$
since  $\varphi (w,\mathbf{c})$  is a function w.r.t. only  $w$  and  $\mathbf{c}$ , and is constant to  $x$ .
For the CBOW model, by applying  $\pmb{h}_{\pmb{w}} = \pmb{h}_{s} - \pmb{h}_{c}$ , the PMI function given by Eqn. (14) can be transformed into
$$
\begin{array}{l} \mathrm {P M I} _ {\mathrm {C B O W}} (x; w | \mathbf {c}) \approx \boldsymbol {h} _ {w} ^ {\top} \boldsymbol {v} _ {\boldsymbol {x}} + \log \frac {\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}} \\ = \log \frac {e ^ {\boldsymbol {h} _ {w} ^ {\top} \boldsymbol {v} _ {x}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {w} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}} + \log \frac {\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}} \sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {w} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}} \\ = \log p (x | w) + \psi (w, \mathbf {c}), \tag {15} \\ \end{array}
$$
$$
\psi (w, \mathbf {c}) = \log \frac {\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {c} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}} \sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {w} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}}{\sum_ {x ^ {\prime}} e ^ {\boldsymbol {h} _ {s} ^ {\top} \boldsymbol {v} _ {x ^ {\prime}}}}.
$$
$\psi (w,\mathbf{c})$  , like  $\varphi (w,\mathbf{c})$  , is also constant w.r.t.  $x$
B Pre-training Details
We employ the BERT-base uncased model and RoBERTa-base model to initialize the context and gloss encoders in our experiments. Both models are pre-trained on released Oxford dictionary data for around 10 epochs. We evaluate the model every epoch by the gloss matching accuracy on the randomly divided evaluation set. In the pre-training process, we set the GR loss weight as  $\lambda = 2.0$ . We use cosine similarity between gloss embedding and target word contextual embedding. As the setting in SimCSE (Gao et al., 2021) training process, we also use the temperature  $\tau = 0.05$  in softmax. Taking the MT GR model as an
example, the softmax of the gloss matching is  $\mathrm{softmax}(\cos \mathrm{ine}(h_s,e_t) / \tau)$ .
We conduct the pre-training on 8 Tesla V100 GPUs. For each GPU, the batch size (related to in-batch negative sampling) is set as 48 for BERT and 36 for RoBERTa model. The learning rate is set  $2 \times 10^{-5}$  with warm-up setting in the first  $10\%$  training steps. The AdamW optimizer is used in the training with default hyper-parameters.
C Lexical Substitution Details
As Arefyev et al. (2020) reported, the process on the format of word candidates influences the metrics. We thus (almost) follow their code and fix the word process in all experiments. In our experiments, the word process includes lemmatization (went->go), filtering the candidates having the same lemmatization output with the original word and removing duplicate lemmatization of candidates. Additionally we filter out the candidates according to the PoS information. For example, the word good can be used as noun or adj, but it would be unreasonable to serve as verb. We then check the possible PoSs for each candidate and filter those words with unmatched PoS with the target word.
In the post-process, the hyper-parameters in (+emb) and validation are tuned in LS07 data. Follow the implementation of Arefyev et al. (2020), we use cosine similarity and the temperature for similarity is set 1/15 in all our experiments. For the validation process, we follow the idea of Zhou et al. (2019), but use BERT-base uncased model for validation. Following their work, we pick the first 50 candidates to re-rank (it has little influence when the number is above 20 in our experiments). The values in propose and validate scores are in different scales, as one is from logits and the other is from cosine similarity. We then adjust the weight of propose score to let its standard deviation be in the same level with the cosine similarity. We set the weight as 0.009 for RoBERTa and 0.004 for BERT.
Table 5 gives examples of LS task and compares our model outputs with the baseline.
D Sentence Similarity
We extend the contextual token similarity measurement into sentence similarity. As stated in (Li et al., 2020), the dot product similarity between sentence
$^{6}$ https://github.com/Samsung/LexSubGen
target word	tell
sentence	He held Obi-Wan loosely, gently stroking his back He knew now that it did n't matter what Sampris said, or what Yoda told him.
labels	said to (4), inform (2)
RoBERTa	teach, say, give, call, have
SC GR-RoBERTa	teach, say, warn, instruct, promise
+ post-process	inform, teach, warn, say, instruct
target word	think
sentence	Shafer thinks we're going to cry, "he doesn't get it!" in reply to his piece" "it" being the amazing world of the Web and new media.
labels	believe (3), feel (1), suspect (1), reckon (1), assume (1)
RoBERTa	say, know, hop, believe, worry
SC GR-RoBERTa	believe, say, hop, expect, suspect
+ post-process	believe, say, hop, expect, know
target word	thus
sentence	The kind of control he exercises is thus likely to be limited to " passive " control such as inspection of produced goods and testing to insure that quality standards are being met.
labels	therefore (5), accordingly (1), consequently (1)
RoBERTa	typically, therefore, then, so, similarly
SC GR-RoBERTa	therefore, consequently, so, accordingly, hence
+ post-process	therefore, consequently, hence, thereby, so
target word	clean
sentence	Dog and horse owners should be encouraged to clean up after their animals.
labels	scrape (1), clear (2), tidy (2)
RoBERTa	wash, pick, wake, keep, clear
SC GR-RoBERTa	groom, walk, look, care, do
+ post-process	tidy, wash, groom, care, walk
target word	late
sentence	We were late doing this since I refused to use someone else's " shopping cart " system that I did n't write and could n't trust .
labels	delayed (3), tardy (2), behind schedule (1), behind time (1), behind (1)
RoBERTa	also, early, just, still, already
SC GR-RoBERTa	early, slow, not, long, behindo
+ post-process	early, slow, prematurely, long, not
target word	new
sentence	The lecture itself went well, but a new problem arose .
labels	different (1), extra (1), additional (1), fresh (4)
RoBERTa	different, big, small, fresh, great
SC GR-RoBERTa	fresh, big, previous, further, different
+ post-process	fresh, renewed, different, previous, recent
Table 5: Examples from LS07 benchmark to show the task and model outputs. The number follows each label is the frequency count indicating the number of annotators that provided this substitute. For each model, we report the top 5 candidates in the first 50 predictions in lemmatized form.
representations  $h_c^\top h_{c'}$  is difficult to derived theoretically, since it is not explicitly involved in the BERT pre-training process. Therefore, inspired by token-level lexical substitution task using contextual probability distribution, we consider the probability distribution of a sentence  $\mathbf{s}_1$  given another sentence  $\mathbf{s}_2$ , i.e.  $p(\mathbf{s}_1|\mathbf{s}_2)$ .
Proposition 1. Let  $w_{1}, \dots, w_{n}$  be  $n$  tokens sampled from a sentence  $\mathbf{s}$ , and  $\mathbf{c}_i$  be the rest of tokens in  $\mathbf{s}$  except for  $w_{i}$ . Let  $x_{1}, \dots, x_{n}$  denote the tokens that can replace  $w_{1}, \dots, w_{n}$  in  $\mathbf{s}$ , respectively. The joint probability distribution of  $x_{1}, \dots, x_{n}$  given  $\mathbf{s}$  is formulated as
$$
\log p \left(x _ {1}, \dots , x _ {n} \mid \mathbf {s}\right) = \sum_ {i = 1} ^ {n} P _ {i}, \tag {16}
$$
$$
P _ {i} = \log p \left(x _ {i} \mid \mathbf {c} _ {i}, x _ {<   i}\right) + \operatorname {P M I} \left(x _ {i}; w _ {i} \mid \mathbf {c} _ {i}, x _ {<   i}\right), \tag {17}
$$
and  $x_{< i}$  denotes  $x_{1},\dots ,x_{i - 1}$
Proof We use the mathematical induction to proof the proposition.
When  $n = 1$ ,  $\log p(x_1|\mathbf{s}) = P_1$  is equivalent as Eqn. (1).
When  $n > 1$ , we make an assumption that Eqn. (16) holds true for  $n = k - 1$ , i.e.  $\log p(x_{<k}|\mathbf{s}) = \sum_{i=1}^{k-1} P_i$ . Then,
$$
\begin{array}{l} \log p \left(x _ {<   k}, x _ {k} \mid \mathbf {s}\right) \\ = \log p (x _ {k} | \mathbf {c} _ {k}, x _ {<   k}) + \log \frac {p (x _ {k} | w _ {k} , \mathbf {c} _ {k} , x _ {<   k})}{p (x _ {k} | \mathbf {c} _ {k} , x _ {<   k})} \dots \\ + \log \frac {p \left(x _ {k} , x _ {<   k} \mid w _ {k} , \mathbf {c} _ {k}\right)}{p \left(x _ {k} \mid w _ {k} , \mathbf {c} _ {k} , x _ {<   k}\right)} \\ = \log p (x _ {k} | \mathbf {c} _ {k}, x _ {<   k}) + \operatorname {P M I} (x _ {k}; w _ {k} | \mathbf {c} _ {k}, x _ {<   k}) \dots \\ + \log p (x _ {<   k} | \mathbf {s}) \\ = P _ {k} + \sum_ {i = 1} ^ {k - 1} P _ {i} = \sum_ {i = 1} ^ {k} P _ {i}, \tag {18} \\ \end{array}
$$
which means Eqn. (16) is also true for  $n = k$ .
Proposition 1 indicates one sentence can be transformed into another sentence through a series of token substitution operations, and the sentence transforming probability can be decomposed into the sum of a series of contextual token probabilities and contextual token similarities, i.e.
$$
p \left(\mathbf {s} _ {1} \mid \mathbf {s} _ {2}\right) = \sum_ {i = 1} ^ {n} P _ {i}, \tag {19}
$$
where  $P_{i}$  is defined in Eqn. (17), and  $\mathbf{s}_1 = [x_1,\dots ,x_n],\mathbf{s}_2 = [w_1,\dots ,w_n]$ . We ignore the case when  $\mathbf{s}_1$  and  $\mathbf{s}_2$  have different lengths, since a simple solution is to pad the shorter sentence to the length of the longer one.
Eqn. (19) and (17) show that the sentence-level tasks also benefits from our gloss regularizer, since the contextual token similarity modeled by gloss matching task also contributes to sentence representation.