SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2 summarizes the multiple-choice evaluation results on both slang datasets. In all cases, applying the semantically informed slang interpretation framework improves the MRR of the respective baselines under both types of negative candidate sampling. On the UD evaluation, even though the language infill model (LM Infill) is not trained on this specific task, LM infill based SSI is able to select better and more appropriate interpretations than the dual encoder baseline, which is trained specifically on slang interpretation with more than 7 times the number of definition entries for training. We also find that while increasing the beam size (specified by  $n$ ) in the sequence-to-sequence based Dual Encoder model impairs its performance, SSI can take advantage of the additional variation in
the generated candidates and outperform its counterpart with a smaller beam size.
Table 3 provides example interpretations predicted by the models. The lit example shows a case where the semantically informed models were able to correctly pinpoint the intended definition, among alternative definitions that describe individuals. The lush example suggests that the SSI model is not perfect and points to common errors made by the model including predicting definitions that are more general and applying incorrect semantic extensions. In this case, the model predicts the slang lush to mean "something that is not cool" because polarity shift is a common pattern in slang usage (Eble, 2012), even though the groundtruth definition does not make such a polarity shift in this specific example.
Note that the improvement brought by SSI is less prominent in the OSD experiment where the Dual Encoder trained on UD was used. This is
Query (target slang in bold italic):	That chick is lit!
Groundtruth definition of target slang:	Attractive.
LM Infill baseline prediction:	Cute, beautiful, adorable.
LM Infill + SSI prediction:	Hot, cool, fat.
Dual Encoder baseline prediction:	Another word for bitch.
Dual Encoder + SSI prediction:	Word used to describe someone who is very attractive.
Query:	That Louis Vuitton purse is lush!
Groundtruth definition of target slang:	High quality, luxurious. (British slang.)
LM Infill baseline prediction:	Amazing, beautiful, unique.
LM Infill + SSI prediction:	Lovely, stunning, expensive.
Dual Encoder baseline prediction:	Something that is cool or awesome.
Dual Encoder + SSI prediction:	An adjective used to describe something that is not cool.
expected because the Dual Encoder is trained to generate definition sentences in the style of UD entries, whereas the SSI is trained on OSD definition sentences instead. The mismatch in style between the two datasets might have caused the difference in performance gain.
5.2 Zero-shot and Few-shot Interpretation
Recent studies in deep learning have shown that large neural network based models such as GPT-3 excel at learning new tasks in a few-shot learning setting (Brown et al., 2020). We examine to what extent the superior performance of our SSI framework may be affected by fine-tuning the LM baseline model in zero-shot and few-shot scenarios. We finetune the language infill model (LM Infill) on the first example usage sentence that correspond to each definition entry in the OSD dataset, resulting in 2,979 sentences. Given an example sentence, we mask out the slang expression and train the language infill model to predict the corresponding slang term. We randomly shuffle all examples and finetune LM Infill for one epoch. We then compare the resulting model with the off-the-shelf LM using examples in the test set that were not used in finetuning (i.e., entries with usage sentences that do not correspond to the first example usage sentence of a definition entry). This results in 106 novel examples for evaluation.
Table 4 shows the result of this experiment. While finetuning does improve test performance (a 6 point gain in MRR), it remains beneficial to consider semantic information in slang context. In both the zero-shot and the few-shot cases, SSI brings
Table 3: Example queries from OSD and top predictions made from both the baseline language infill models (LM Infill) and the Dual Encoder models with  $n = 50$ , along with top predictions from the enhanced semantically informed slang interpretation (SSI) models. Additional examples can be found in Appendix B.1.
Model	Distinct negatives	Random negatives
LM Zero-shot, n = 50	0.444	0.443
+ SSI	0.571	0.565
LM Few-shot, n = 50	0.504	0.513
+ SSI	0.567	0.564
Table 4: Interpretation results on OSD measured in mean-reciprocal rank (MRR) before and after finetuning the language infill model.
significant performance gain even though SSI itself is only trained on entries from the training set.
5.3 Evaluation on Slang Translation
We next apply the slang interpretation framework to neural machine translation. Existing machine translation systems have difficulty in translating source sentences containing slang usage partly because they lack the ability to properly decode the intended slang meaning. We make a first attempt in addressing this problem by exploring whether machine interpretation of slang can lead to better translation of slang. Given a source English sentence containing a slang expression  $S$ , we apply the LM based slang interpreters to generate a paraphrased word to replace  $S$ . The paraphrased sentence would then contain the intended meaning of the slang in its literal form. Here, we take advantage of the LM-based approaches' ability to directly generate a paraphrase instead of a definition sentence (i.e., without dictionary lookup  $D$  in Equation 2), which allows direct insertion of the resulting interpretation into the original sentence.
(a) English to French
(b) English to German
Figure 3: Translation scores of translated sentences with the slang replaced by n-best interpretations. Curves show sentence-level BLEU, BLEURT, and COMET scores of the best translation within the top-n retrievals. Aggregate scores integrated over the first 20 retrievals are shown in parenthesis. Baselines are obtained by directly translating the original sentence containing slang.
We perform our experiment on the OSD test set because it contains higher quality example sentences than UD. To mitigate potential biases, we consider only entries that correspond to single word slang expressions, and that the slang has not been seen during training (where the slang attaches to a different slang meaning than the one in the test set). For the remaining 102 test entries, we obtain gold-standard translations by first manually replacing the slang word in the example sentence with its intended definition, condensed to a word or short phrase to fit into the context sentence. We then translate the sentences to French and German using machine translation.
We make all machine translations using pretrained 6-layer transformer networks (Vaswani et al., 2017) from MarianMT (Tiedemann and Thotingal, 2020), which are trained on a collection of web-based texts in the OPUS dataset (Tiedemann, 2012). Here, we select models pre-trained on web-based texts to maximize the baseline model's ability
to correctly process slang. We evaluate the translated sentences using three metrics: 1) Sentence-level BLEU scores (Papineni et al., 2002) computed using sentence_bleu implementation from NLTK (Bird et al., 2009) with smoothing (method4 in NLTK, Chen and Cherry, 2014) to account for sparse n-gram overlaps; 2) BLEURT scores (Sellam et al., 2020) computed using the pre-trained BLEURT-20 checkpoint; 3)COMET scores (Rei et al., 2020) computed using the pre-trained wmt20-comet-da checkpoint. For COMET scores, we replace slang expressions in the source sentences with their literal equivalents to reduce confusion that the COMET model might have on slang.
Figure 3 summarizes the results. Overall, the semantically informed approach tends to outperform the baseline approaches for the range of top retrievals (from 1 to 20) under all three metrics considered, with the exception of BLEURT evaluated on German where the semantically informed approach gives very similar performance as the
Query (target slang in bold italic):	I want to go get coffee but it's bitter outside.
Definition of target slang:	Abbreviated form of bitterly cold.
Groundtruth interpreted sentence:	I want to go get coffee but it's bitterly cold outside.
Original query sentence translation:	Je veux aller prendre un café mais c'est amer dehors.
(BLEU: 65.0, BLEURT: 59.8,COMET: 0.77)
Gold-standard translation:	Je veux aller prendre un café, mais il fait très froid dehors.
LM Infill interpretation & translation:
(1) I want to go get coffee but it's raining outside.	Je veux aller prendre un café mais il pleut dehors.
(BLEU: 68.1, BLEURT: 79.9,COMET: 0.97)
(2) I want to go get coffee but it's closed outside.	Je veux aller prendre un café mais il est fermé dehors.
(BLEU: 70.7, BLEURT: 53.9,COMET: -0.15)
LM Infill + SSI interpretation & translation:
(1) I want to go get coffee but it's cold outside.	Je veux aller prendre un café, mais il fait froid dehors.
(BLEU: 90.3, BLEURT: 92.7,COMET: 1.20)
(2) I want to go get coffee but it's warm outside.	Je veux aller prendre un café mais il fait chaud dehors.
(BLEU: 78.1, BLEURT: 79.1,COMET: 1.12)
Table 5: An example of machine translation of slang, without or with the application of the SSI framework. The top 2 interpreted and translated sentences are shown for each model with BLEU, BLEURT, and COMET scores against the gold-standard translation shown in parentheses. More examples can be found in Appendix B.4.
language model baseline. While not all predicted interpretations correspond to the groundtruth definitions, the set of interpreted sentences often contain plausible interpretations that result in improved translation of slang. Table 5 provides some example translations. We observe that quality translations can be found reliably with a small number of interpretation retrievals (i.e., around 5) and the quality generally improves as we retrieve more candidate interpretations. Our approach may be ultimately integrated with a slang detector (e.g., Pei et al. 2019) to produce fully automated translations in natural context that involves slang.
The flexible nature of slang is a hallmark of informal language, and to our knowledge we have presented the first principled framework for automated slang interpretation that takes into account both contextual information and knowledge about semantic extensions of slang usage. We showed that our framework is more effective in interpreting and translating the meanings of English slang terms in natural sentences in comparison to existing approaches that rely more heavily on context to infer slang meaning.
Future work in this area may benefit from principled approaches that model the coinage of slang expressions with novel word forms and multi-word expressions with complex formation strategies, as
well as how slang terms emerge in specific individuals and groups. Our current study shows promise for advancing methodologies in informal language processing toward these avenues of future research.
We analyze entries of slang usage in our work and acknowledge that such usages may contain offensive information. We retain such entries in our datasets to preserve the scientific validity of our results, as a significant portion of slang usage aligns to possibly offensive usage context. In the presentation our of results, however, we strive to select examples or illustrations that minimize the extent to which offensive content is represented. We also acknowledge that models trained on datasets such as the Urban Dictionary have a greater tendency to generate offensive language. All model outputs shown are results of model learning and do not reflect opinions of the authors and their affiliated organizations. We hope that our work will contribute to the greater good by enhancing AI system's ability to comprehend such offensive language use, allowing better filtering of online content that may be potentially harmful.
We thank the ARR reviewers for their constructive comments and suggestions, and Walter Rader for
permission to use The Online Slang Dictionary. This work was supported by a NSERC Discovery Grant RGPIN-2018-05872, a SSHRC Insight Grant #435190272, and an Ontario ERA Award to YX.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pages 1638-1649.
Pierre Baldi and Yves Chauvin. 1993. Neural networks for fingerprint recognition. *Neural Computation*, 5(3):402-418.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc."
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Virginia Braun and Celia Kitzinger. 2001. "Snatch," "Hole," or "Honey-pot"? Semantic categories and the problem of nonspecificity in female genital slang. The Journal of Sex Research, 38(2):146-158.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1994. Signature verification using a "siamese" time delay neural network. In J. D. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems 6, pages 737-744. Morgan-Kaufmann.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362-367, Baltimore, Maryland, USA. Association for Computational Linguistics.
Shehzaad Dhuliawala, Diptesh Kanojia, and Pushpak Bhattacharyya. 2016. SlangNet: A WordNet like resource for English slang. In Proceedings of the
Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 4329-4332, Portoroz, Slovenia. European Language Resources Association (ELRA).
Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2492-2501, Online. Association for Computational Linguistics.
Connie C Eble. 2012. Slang & sociability: In-group language among college students. University of North Carolina Press, Chapel Hill, NC.
David Goldberg, David Nichols, Brian M. Oki, and Douglas Terry. 1992. Using collaborative filtering to weave an information tapestry. *Commun. ACM*, 35:61-70.
Alex Graves. 2012. Sequence transduction with recurrent neural networks. In International Conference of Machine Learning (ICML) 2012 Workshop on Representation Learning.
Jonathan Green. 2010. *Green's Dictionary of Slang*. Chambers, London.
Anshita Gupta, Sanya Bathla Taneja, Garima Malik, Sonakshi Vij, Devendra K. Tayal, and Amita Jain. 2019. Slangzy: a fuzzy logic-based algorithm for english slang meaning selection. Progress in Artificial Intelligence, 8(1):111-121.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Vivek Kulkarni and William Yang Wang. 2018. Simple models for word formation in slang. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1424-1434, New Orleans, Louisiana. Association for Computational Linguistics.
Adrienne Lehrer. 1985. The influence of semantic fields on semantic change. *Historical Semantics: Historical Word Formation*, 29:283-296.
Elisa Mattiello. 2009. Difficulty of slang translation. In *Translation Practices*, pages 65-83. Brill Rodopi.
Ke Ni and William Yang Wang. 2017. Learning to explain non-standard English words and phrases. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 413-417, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Alok Ranjan Pal and Diganta Saha. 2013. Detection of slang words in e-data using semi-supervised learning. International Journal of Artificial Intelligence and Applications, 4(5):49-61.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. ACL '02, page 311-318, USA. Association for Computational Linguistics.
Zhengqi Pei, Zhewei Sun, and Yang Xu. 2019. Slang detection and identification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 881-889, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Radim Rehurek and Petr Sojka. 2011. Gensim-python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2).
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Eleanor Rosch. 1975. Cognitive representations of semantic categories. Journal of Experimental Psychology: General, 104:192-233.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information
Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 4080-4090.
Zhewei Sun, Richard Zemel, and Yang Xu. 2019. Slang generation as categorization. In Proceedings of the 41st Annual Conference of the Cognitive Science Society, pages 2898-2904. Cognitive Science Society.
Zhewei Sun, Richard Zemel, and Yang Xu. 2021. A computational framework for slang generation. Transactions of the Association for Computational Linguistics, 9:462-478.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27, pages 3104-3112. Curran Associates, Inc.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT - building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479-480, Lisboa, Portugal. European Association for Machine Translation.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Beatrice. Warren. 1992. Sense Developments: A Contrastive Study of the Development of Slang Senses and Novel Standard Senses in English. Acta Universitatis Stockholmiensis: Stockholm studies in English. Almqvist & Wiksell International.
Liang Wu, Fred Morstatter, and Huan Liu. 2018. SlangSD: Building, expanding and using a sentiment dictionary of slang words for short-text sentiment classification. Lang. Resour. Eval., 52(3):839-852.
Yang Xu and Charles Kemp. 2015. A computational evaluation of two laws of semantic change. In Proceedings of the 37th Annual Conference of the Cognitive Science Society, pages 2703-2708. Cognitive Science Society.
A Training Procedures
A.1 Baseline Models
We train two context-based slang interpreters described in Section 3.1 as our baseline models. For the LM-based interpreter, we use a pre-trained language infill model from Donahue et al. (2020) based on the GPT-2 (Radford et al., 2019) architecture. Here, we obtain the n-best list of interpretations by retrieving the list of inflated words with the highest infill probability. Words containing non-alphanumeric characters are filtered out. For the dictionary lookup function  $D$  in Equation 2, if a matching dictionary entry can be found in Oxford Dictionary (OD), the top definition sentence is retrieved as the definition sentence for the input word. Otherwise, the word itself is used as the definition. In addition to the word's original form, we apply lemmatization or stemming to the original form using NLTK (Bird et al., 2009) to find matching dictionary entries. To check for Part-of-Speech (POS) tags, we apply the Flair tagger (Akbik et al., 2018) on the context sentence with the slang expression replaced by a mask token and use counts from Histwords (Hamilton et al., 2016) to determine POS tags for individual words.
To train the Dual Encoder, we use LSTM encoders with 256 and 1024 hidden units to encode a slang expression's spelling and its usage context respectively, with 100 and 300 dimensional input embeddings for the characters and words respectively. Following Ni and Wang (2017), we use random initialization for the input embeddings and use stochastic gradient descent (SGD) with an adaptive learning rate. We train the model for 20 epochs beginning with a learning rate of 0.1 and add an exponential decay of 0.9 every epoch. We reserve  $5\%$  of the training examples as a development set for hyperparameter tuning. We train the model for 20 epochs on a Nvidia Titan V GPU and took 12 hours to complete. During inference, we obtain the n-best list of interpretations by running a beam search of corresponding beam width on the LSTM decoder.
A.2 Semantic Reranker
We obtain the contrastive sense encodings (CSE) described in Section 3.2 by using 768-dimensional Sentence-BERT (Reimers and Gurevych, 2019) embeddings as our baseline embedding. Following Sun et al. (2021), we train the contrastive network with a 1.0 margin ( $m$  in Equation 5) using