SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhewei Sun<sup>1</sup>, Richard Zemel<sup>1,2,4</sup>, Yang Xu<sup>1,3,4</sup>
$^{1}$ Department of Computer Science, University of Toronto, Toronto, Canada
$^{2}$ Department of Computer Science, Columbia University, New York, USA
$^{3}$ Cognitive Science Program, University of Toronto, Toronto, Canada  $^{4}$ Vector Institute for Artificial Intelligence, Toronto, Canada
{zheweisun, zemel, yangxu}@cs.toronto.edu
Slang is a predominant form of informal language making flexible and extended use of words that is notoriously hard for natural language processing systems to interpret. Existing approaches to slang interpretation tend to rely on context but ignore semantic extensions common in slang word usage. We propose a semantically informed slang interpretation (SSI) framework that considers jointly the contextual and semantic appropriateness of a candidate interpretation for a query slang. We perform rigorous evaluation on two large-scale online slang dictionaries and show that our approach not only achieves state-of-the-art accuracy for slang interpretation in English, but also does so in zero-shot and few-shot scenarios where training data is sparse. Furthermore, we show how the same framework can be applied to enhancing machine translation of slang from English to other languages. Our work creates opportunities for the automated interpretation and translation of informal language.
Slang is one of the most common forms of informal language, but interpreting slang can be difficult for both humans and machines. Empirical studies have shown that, although it is done instinctively, interpretation and translation of unfamiliar or novel slang expressions can be quite hard for humans (Braun and Kitzinger, 2001; Mattiello, 2009). Similarly, slang interpretation is also notoriously difficult for state-of-the-art natural language processing (NLP) systems, which presents a critical challenge to downstream applications such as natural language understanding and machine translation.
Consider the sentence "I got really steamed when my car broke down". As illustrated in Figure 1, directly applying a translation system such as Google Translate on this raw English sentence would result in a nonsensical translation of the
Figure 1: Illustrations of slang interpretation in English (top panel) and slang translation (bottom panel) from English to French on the original sentence (nonsensical), or on the interpreted version of the sentence (sensical).
slang term steamed in French. This error is due partly to the underlying language model that fails to recognize the flexible extended use of the slang term from its conventional meaning (e.g., "vapor") to the slang meaning of "angry". However, if knowledge about such semantic extensions can be incorporated into interpreting the slang prior to translation, as Figure 1 shows the system would be quite effective in translating the intended meaning.
Here we consider the problem of slang interpretation illustrated in the top panel of Figure 1. Given a target slang term like steamed in a novel query sentence, we want to automatically infer its intended meaning in the form of a definition (e.g., "angry"). Tackling this problem has implications in both machine interpretation and understanding of informal language within individual languages and translation between languages.
One natural solution to this problem is to use contextual information to infer the meaning of a slang term. Figure 2 illustrates this idea by showing the top infilled words predicted under a GPT-2
Human Language Technologies, pages 5213 - 5231
Figure 2: Workflow of the proposed framework.
(Radford et al., 2019) based language infill model (Donahue et al., 2020). Each of these words can be considered a candidate paraphrase for the target slang steamed conditioned on its surrounding words. Although the groundtruth meaning "angry" is among the list of top candidates, this model infers "sick" as the most probable interpretation. A similar context-based approach has been explored in a previous study led by Ni and Wang (2017) showing that a sequence-to-sequence model trained directly on a large number of pairs of slang-contained sentences along with their corresponding definitions from Urban Dictionary can be a useful starting point toward the automated interpretation of slang.
We present an alternative approach to slang interpretation that builds on but goes beyond the context-based models. Inspired by recent work on generative models of slang (Sun et al., 2019, 2021), we consider slang interpretation to be the inverse process of slang generation and propose a semantically informed framework that takes into account both contextual information and knowledge about slang meaning extensions (e.g., "vapor"  $\rightarrow$  "angry") in inferring candidate interpretations. Our framework incorporates a semantic model of slang that uses contrastive learning to capture semantic extensions that link conventional and slang meanings of words (Sun et al., 2021). Under this framework, meanings that are otherwise far apart can be brought close, resulting in a semantic space that is sensitive to the flexible extended usages of slang. Rather than using this learned semantic space to generate novel slang usages, we apply it to the inverse problem of slang interpretation by checking whether a candidate interpretation may be suitably expressed as a slang using the to-be-interpreted slang expression. For example, "sick" and "angry" can both replace the slang steamed in a given context, but "angry" may be a more appropriate meaning to be expressed using steamed in the slang context. As such, we build a computational framework that takes into account the semantic knowledge of words as well as the context of slang in the interpretation process.
Figure 2 illustrates the workflow of our approach. We begin with a set of candidate interpretations informed by a context-based model (e.g., a language infill model), where the set would contain a list of possible meanings that fit reasonably in the given context. We then rerank this set of candidate interpretations by selecting the meaning that is most likely to be extended as slang from the to-be-interpreted slang expression.
For the scope of this work, we focus on interpreting slang expressions with existing word forms because extensive studies in slang have suggested that a high proportion of slang usages relies on the extended reuse of existing word forms (Warren, 1992; Green, 2010; Eble, 2012). We show that our framework can enhance state-of-the-art language models in slang interpretation in English and slang translation from English to other languages.
2.1 Natural Language Processing for Slang
Existing approaches in the natural language processing for slang focus on efficient construction, extension, and retrieval from dictionary-based resources for detection (Pal and Saha, 2013; Dhuliawala et al., 2016), interpretation (Gupta et al., 2019), and sentiment analysis of slang (Dhuliawala et al., 2016; Wu et al., 2018). These studies often rely on heuristic measures to determine or retrieve the meaning of slang and cannot generalize beyond what was available in the training data. Recent work such as Kulkarni and Wang (2018) and Pei et al. (2019) proposed deep learning based approaches to generalize toward unseen slang.
Closely related to our study is Ni and Wang (2017) that formulated English slang interpretation as a translation task (although they did not tackle slang machine translation per se). In this work, each slang query sentence in English is paired with the groundtruth slang definition (also in English), and such pairs are fed into a translation model. In addition, the spellings of slang word forms are also considered as input. In their model, both the context and the slang form are encoded using separate LSTM encoders. The two encoded representations are then linearly combined to form the encoded input for a sequence-to-sequence network (Sutskever et al., 2014). During training, the combined state is passed onto an LSTM decoder to train against
1Code and data available at: https://github.com/ zhewei-sun/slanginterp
the corresponding definition sentence. During test time, beam search (Graves, 2012) is applied to decode a set of candidate definition sentences.
One key problem with this approach is that the Dual Encoder tends to rely on the contextual features surrounding the target slang but does not model flexible meaning extensions of the slang word itself. Similar issues are present in a language-model based approach, whereby one can use an infill model to infer the meaning of a target slang based solely on its surrounding words. Our work extends these context-based approaches by jointly considering the contextual and semantic appropriateness of a slang expression in a sentence, using generative semantic models of slang.
2.2 Generative Semantic Models of Slang
Recent work by Sun et al. (2019, 2021) proposed a neural-probabilistic generative framework for modeling slang word choice in novel context. Given a query sentence with the target slang blanked out and the intended meaning of that slang, their framework predicts which word(s) would be appropriate slang choices that fill in the blank. Relevant to their framework is a semantic model of slang that uses contrastive learning from Siamese networks (Baldi and Chauvin, 1993; Bromley et al., 1994) to relate conventional and slang meanings of words. This model yields a semantic embedding space that is sensitive to flexible slang meaning extensions. For example, it may learn that meanings associated with "vapor" can extend to meanings about "angry" (as in the steamed example in Figure 1).
Differing from slang generation, our work concerns the inverse problem of slang interpretation that has more direct applications in natural language processing particularly machine translation (e.g., of informal language). Building on work of slang generation, we incorporate the generative semantic model of slang in a semantically informed interpretation framework that integrates context to infer the intended meaning of a target slang.
3 Computational Framework
Our computational framework is comprised of three key components following the workflow illustrated in Figure 2: 1) A context-based baseline interpreter that generates an n-best list of candidate interpretations for a target slang in a query sentence; 2) A semantic model of slang that checks the appropriateness of a candidate interpretation to
the slang context; 3) A reranker informed by the semantic model in 2) that re-prioritizes the candidate interpretations from the context-based interpreter in 1). We use this framework for both interpreting slang within English and translating slang from English to other languages.
3.1 Context-based Interpretation
We define slang interpretation formally as follows. Given a target slang term  $S$  in context  $C_S$  of a query sentence, interpret the meaning of  $S$  by a definition  $M$ . The context is an important part of the problem formulation since a slang term  $S$  may be polysemous and context can be used to constrain the interpretation of its meaning. We define a slang interpreter  $I$  probabilistically as:
$$
I (S, C _ {S}) = \underset {M} {\arg \max } P (M | S, C _ {S}) \tag {1}
$$
Given this formulation, we retrieve an n-best list of candidate interpretations  $\mathcal{K}$  (i.e.,  $|\mathcal{K}| = n$ ) based on an interpretation model of choice  $P(M|S,C_S)$ . Here, we consider two alternative models for  $P(M|S,C_S)$ : 1) a language-model (LM) based approach that treats slang interpretation as a cloze task, and 2) a sequence-to-sequence based approach similar to work by Ni and Wang (2017).
LM-based interpreter. The first model we consider is a language infill model in a cloze task, in which the model itself is based on large pre-trained language models such as GPT-2 (Radford et al., 2019). Although slang expressions may make sporadic appearances during training, this model is not trained specifically on a slang related task and thus serves as a baseline that reflects the state-of-the-art language-model based NLP systems (e.g., Donahue et al., 2020).
Given context  $C_S$  containing target slang  $S$ , we blank out  $S$  in the context and ask the language infill model to infer the most likely words to fill in the blank. This results in a probability distribution  $P(w|C_S\backslash S)$  over candidate words  $w$ . The filled words can then be viewed as candidate interpretations of the slang  $S$ :
$$
\begin{array}{l} I(S,C_{S}) = D[\arg \max_{w}LM(w|C_{S}\backslash S) \\ + \mathbb {1} _ {T (w)} [ T (C _ {S} \backslash S) ] ] \tag {2} \\ \end{array}
$$
Here,  $D$  is a dictionary lookup function that maps a candidate word  $w$  to a definition sentence. In this case, we constrain the space of meanings considered to the set of all meanings corresponding
to words in the lexicon. Additionally, we apply a Part-of-Speech (POS) tagger  $T$  to check whether the candidate word  $w$  shares the same POS tag as the blanked-out word in the usage context. Words that share the same POS tags are preferred in the list of n-best retrievals.
This baseline approach by itself does not take into account any (semantic) information from the target slang  $S$ . In the case where two distinctive slang terms may be placed in the same context, the model would generate the exact same output. However, this LM based approach does not require task-specific data to train. We show later that by reranking language model outputs, it is possible to achieve state-of-the-art performance using much less on-task data than existing approaches.
Dual encoder. Ni and Wang (2017) partly addressed the context-only limitation by encoding the slang term using a character-level recurrent neural network in an end-to-end model inspired by the sequence-to-sequence architecture for neural machine translation (Sutskever et al., 2014). We implement their dual encoder architecture as an alternative context-based interpreter to LM. In this model, separate LSTM encoders are applied on the context  $C_S$  and the character encoding of the to-be-interpreted slang  $S$  respectively. The two encoders are then linearly combined using learned parameters. The combined state is passed onto an LSTM decoder to train against the corresponding definition sentence in Urban Dictionary (as in the original work of Ni and Wang 2017). For inference, beam search (Graves, 2012) is applied to decode an n-best list of candidate definition sentences.
While this approach is trained directly on slang data and considers the slang word forms, it requires a large on-task dataset to be trained effectively. This model also does not take into account the appropriateness of meaning extension in slang usage. We next describe how a semantic model of slang can be incorporated to enhance the context-based interpreters.
3.2 Semantic Model of Slang
Given an n-best list of candidate interpretations  $\mathcal{K}$  for the target slang  $S$  in context  $C_S$ , we wish to model the semantic plausibility of each candidate interpretation  $k \in \mathcal{K}$ . Specifically, we ask how likely one would relate the (conventional meaning of) target slang expression  $S$  to a candidate interpretation  $k$ . Sun et al. (2019, 2021) modeled the
relationship between a to-be-expressed meaning and a word form using the prototype model (Rosch, 1975; Snell et al., 2017). We adapt this model in the context of slang interpretation:
$$
\begin{array}{l} f (k, S) = s i m (E _ {k}, E _ {S}) \\ = \exp \left(- \frac {d \left(E _ {k} , E _ {S}\right)}{h _ {m}}\right) \tag {3} \\ \end{array}
$$
$E_{k}$  is an embedding for a candidate interpretation  $k$  and  $E_{S}$  is the prototypical conventional meaning of  $S$  computed by averaging the embeddings of its conventional meanings in dictionary  $(\mathcal{E}_S)$ :
$$
E _ {S} = \frac {1}{\left| \mathcal {E} _ {S} \right|} \sum_ {E _ {S _ {i}} \in \mathcal {E} _ {S}} E _ {S _ {i}} \tag {4}
$$
The similarity function  $f$  can then be computed by taking the negative exponential of the Euclidean distance between the two resulting semantic embeddings.  $h_m$  is a kernel width hyperparameter.
Following Sun et al. (2021), we learn semantic embeddings  $E_{k}$  and  $E_{S_{i}}$  under a max-margin triplet loss scheme, where embeddings of slang sense definitions  $(E_{SL})$  are brought close in Euclidean space to those of their conventional sense definitions  $(E_{P})$  yet kept apart from irrelevant word senses  $(E_{N})$  by a pre-specified margin  $m$ :
$$
L o s s = \left[ d \left(E _ {S L}, E _ {P}\right) - d \left(E _ {S L}, E _ {N}\right) + m \right] _ {+} \tag {5}
$$
The resulting contrastive sense encodings are shown to be sensitive to slang semantic extensions that have been observed during training. We leverage this knowledge to check whether pairing a candidate interpretation  $k$  with the slang expression  $S$  is likely given the common semantic extensions observed in slang usages.
3.3 Semantically Informed Reranking
We define a semantic scorer  $g$  over the set of candidate interpretations  $\mathcal{K}$  and the to-be-interpreted slang  $S$ . The candidates are reranked based on the resulting scores to obtain semantically informed slang interpretations (SSI):
$$
S S I (\mathcal {K}) = \arg \max  g (k, S) \tag {6}
$$
We define  $g(\mathcal{K}, S)$  as a score distribution over the set of candidates  $\mathcal{K}$  given slang  $S$ , where each score is computed by checking the semantic appropriateness of a candidate meaning  $k \in \mathcal{K}$  with respect to
target slang  $S$  by querying the semantic model  $f$  from Equation 3:
$$
g (k, S) = P (k | S) \propto f (k, S) \tag {7}
$$
In addition, we apply collaborative filtering (Goldberg et al., 1992) to account for a small neighborhood of words  $L(S)$  akin to the slang expression  $S$  in conventional meaning:
$$
\begin{array}{l} g ^ {*} (k, S) \propto \sum_ {S ^ {\prime} \in L (S)} \operatorname {s i m} (S, S ^ {\prime}) g (k, S ^ {\prime}) (8) \\ \operatorname {s i m} \left(S, S ^ {\prime}\right) = \exp \left(- \frac {d \left(S , S ^ {\prime}\right)}{h _ {c f}}\right) (9) \\ \end{array}
$$
Here,  $d(S,S^{\prime})$  is the cosine distance between the two slang's word vectors and  $h_{cf}$  is a hyperparameter controlling the kernel width. The collaborative filtering step encodes intuition from studies in historic semantic change that similar words tend to extend to express similar meanings (Lehrer, 1985; Xu and Kemp, 2015), which was found to extend well in the case of slang (Sun et al., 2019, 2021).
We use two online English slang dictionary resources to train and evaluate our proposed slang interpretation framework: 1) the Online Slang Dictionary  $(\mathrm{OSD})^2$  dataset from Sun et al. (2021) and 2) a collection of Urban Dictionary  $(\mathrm{UD})^3$  entries from 1999 to 2014 collected by Ni and Wang (2017). Each dataset contains slang gloss entries including a slang's word form, its definition, and at least one corresponding example sentence containing the slang term. We use the same training and testing split provided by the original authors and only use entries where a corresponding non-informal entry can be found in the online version of the Oxford Dictionary (OD) for English<sup>4</sup>, which allows the retrieval of conventional senses for all slang expressions considered. We also filter out entries where the example usage sentence contains none or more than one exact references of the corresponding slang expression. When a definition entry has multiple example usage sentences, we treat each example sentence as a separate data entry, but all data entries corresponding to the same definition entry will only appear in the same data split. Table 1 shows the size of the datasets after pre-processing.
While OSD contains higher quality entries, UD offers a much larger dataset. We thus use OSD to evaluate model performance in a low resource scenario and UD for evaluation of larger neural network based approaches.
5 Evaluation and Results
5.1 Evaluation on Slang Interpretation
We first evaluate the semantically informed and baseline interpretation models in a multiple choice task. In this task, each query is paired with a set of definitions that construe the meaning of the target slang in the query. One of these definitions is the groundtruth meaning of the target slang, while the other definitions are incorrect or negative entries sampled from the training set (i.e., all taken from the slang dictionary resources described). To score a model, each definition sentence is first compared with the model-predicted definition by computing the Euclidean distance between their respective Sentence-BERT (Reimers and Gurevych, 2019) embeddings. The ideal model should produce a definition that is semantically closer to the groundtruth definition, more so than the other competing negatives. For each dataset, we sample two sets of negatives. The first set of negative candidates contains only definition sentences from the training set that are distinct from the groundtruth definition. We consider two definition sentences to be distinct if the overlap in the number of content words is less than  $50\%$ . The other set of negative definitions is sampled randomly. We measure the performance of the models by computing the standard mean reciprocal rank (MRR) of the groundtruth definition's rank when checked against 4 other sampled negative definitions.
We train the semantic reranker on all definition entries in the respective training sets from the two data resources. When training the Dual Encoder, we use 400,431 out-of-vocabulary slang entries (i.e., entries with a slang expression that does not contain a corresponding lexical entry in the standard dictionary) from UD in addition to the invocabulary entries used to train the reranker. This is necessary since the baseline Dual Encoder performs poorly without a large number of training entries. Similarly, training the Dual Encoder directly on the OSD training set does not result in an adequate model for comparison. We instead train the Dual Encoder on all UD entries and experiment with the resulting interpreter on OSD. Any UD
$^{2}$ OSD: http://onlineslangdictionary.com
$^{3}$ UD: https://www.urbandictionary.com
$^{4}$ OD: https://en.oxforddictionaries.com
Dataset	# of unique slang word forms	# of slang definition entries	# of context sentences	# of definitions in the test set	# of context sentences in the test set
OSD	1,635	2,979	3,718	299	405
UD	9,474	65,478	65,478	1,242	1,242
Table 1: Summary of basic statistics for the two online slang dictionaries used in the study.
Model	Distinctively sampled candidates	Randomly sampled candidates
Dataset 1: Online Slang Dictionary (OSD) (Sun et al., 2021)
Language Infill Model (LM Infill) (Donahue et al., 2020), n = 50	0.532	0.502
+ Semantically Informed Slang Interpretation (SSI)	0.557	0.563
Dual Encoder* (Ni and Wang, 2017), n = 5	0.584	0.583
+ SSI	0.592	0.588
Dual Encoder*, n = 50	0.568	0.602
+ SSI	0.616	0.607
* Dual Encoders trained on UD data after filtering out slang in OSD test set.
Dataset 2: Urban Dictionary (UD) (Ni and Wang, 2017)
LM Infill, n = 50	0.517	0.521
+ SSI	0.569	0.579
Dual Encoder, n = 5	0.556	0.555
+ SSI	0.573	0.572
Dual Encoder, n = 50	0.547	0.550
+ SSI	0.582	0.584
Table 2: Evaluation of English slang interpretation measured in mean-reciprocal rank (MRR). Predictions are ranked against 4 negative candidates distinctively or randomly sampled, yielding MRR=0.457 for the random baseline.
entries corresponding to words found in the OSD testset are filtered out in this particular experiment. Detailed training procedures for all models can be found in Appendix A.