SlowGuess/ACL_2020_OCR huggingface dataset:
Yichao Zhou, Jyun-yu Jiang, Jieyu Zhao, Kai-Wei Chang and Wei Wang
{yz, jyunyu, jyzhao, kwchang, weiwang}@cs.ucla.edu
Humor plays an important role in human languages and it is essential to model humor when building intelligence systems. Among different forms of humor, puns perform wordplay for humorous effects by employing words with double entendre and high phonetic similarity. However, identifying and modeling puns are challenging as puns usually involved implicit semantic or phonological tricks. In this paper, we propose Pronunciation-attentive Contextualized Pun Recognition (PCPR) to perceive human humor, detect if a sentence contains puns and locate them in the sentence. PCPR derives contextualized representation for each word in a sentence by capturing the association between the surrounding context and its corresponding phonetic symbols. Extensive experiments are conducted on two benchmark datasets. Results demonstrate that the proposed approach significantly outperforms the state-of-the-art methods in pun detection and location tasks. In-depth analyses verify the effectiveness and robustness of PCPR.
During the last decades, social media has promoted the creation of a vast amount of humorous web contents (Nijholt et al., 2017). Automatic recognition of humor has become an important task in the area of figurative language processing, which can benefit various downstream NLP applications such as dialogue systems, sentiment analysis, and machine translation (Melby and Warner, 1995; Augello et al., 2008; Ghosh et al., 2015; Bertero and Fung, 2016; Blinov et al., 2019). However, humor is one of the most complicated behaviors in natural language semantics and sometimes it is even difficult for humans to interpret. In most cases, understanding humor requires adequate background knowledge and a rich context.
Homographic Puns
1. Did you hear about the guy whose whole left side was cut off? He's all right now.
2. I'd tell you a chemistry joke but I know I wouldn't get a reaction.
Heterographic Puns
1. The boating store had its best sail (sale) ever.
2. I lift weights only on Saturday and Sunday because Monday to Friday are weak (week) days.
Table 1: Examples of homographic and heterographic puns.
Puns are a form of humorous approaches using the different meanings of identical words or words with similar pronunciations to explain texts or utterances. There are two main types of puns. Homographic puns rely on multiple interpretations of the same word. As shown in Table 1, the phrase all right means good condition or opposite to left; the word reaction means chemical change or action. The two meanings of the same expression are consistent with its context, which creates a humorous pun in both sentences when there is a clear contrast between two meanings. On the other hand, heterographic puns take advantage of phonologically same or similar words. For example, the word pairs sale and sail, weak and week in Table 1 have the same or similar pronunciations. The sentences are funny because both words fit the same context. Understanding puns is a big fish to fry for deep comprehension of complex semantics.
These two forms of puns have been studied in literature from different angles. To recognize puns in a sentence, word sense disambiguation techniques (WSD) (Navigli, 2009) have been employed to identify the equitable intention of words in utterances (Pedersen, 2017). External knowledge bases such as WordNet (Miller, 1998b) have been applied in determining word senses of pun words (Oele and Evang, 2017). However, these methods cannot tackle heterographic puns with distinct word
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 813-822 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
spellings and knowledge bases that only contain a limited vocabulary. To resolve the issues of sparseness and heterographics, the word embedding techniques (Mikolov et al., 2013; Pennington et al., 2014) provide flexible representations to model puns (Hurtado et al., 2017; Indurthi and Oota, 2017; Cai et al., 2018). However, a word may have different meanings regarding its contexts. Especially, an infrequent meaning of the word might be utilized for creating a pun. Therefore, static word embeddings are insufficient to represent words. In addition, some puns are created by replacing a word with another word with the same or similar pronunciation as examples shown in Table 1. Therefore, to recognize puns, it is essential to model the association between words in the sentence and the pronunciation of words. Despite existing approaches attempt to leverage phonological structures to understand puns (Doogan et al., 2017; Jaech et al., 2016), there is a lack of a general framework to model these two types of signals in a whole.
In this paper, we propose Pronunciation-attentive Contextualized Pun Recognition (PCPR) to jointly model the contextualized word embeddings and phonological word representations for pun recognition. To capture the phonological structures of words, we break each word into a sequence of phonemes as its pronunciation so that homophones can have similar phoneme sets. For instance, the phonemes of the word pun are  $\{\mathrm{P},\mathrm{AH},$  N}. In PCPR, we construct a pronunciation attentive module to identify important phonemes of each word, which can be applied in other tasks related to phonology. We jointly encode the contextual and phonological features into a self-attentive embedding to tackle both pun detection and location tasks. We summarize our contributions as following.
- To the best of our knowledge, PCPR is the first work to jointly model contextualized word embeddings and pronunciation embeddings to recognize puns. Both contexts and phonological properties are beneficial to pun recognition.
- Extensive experiments are conducted on two benchmark datasets. PCPR significantly outperforms existing methods in both pun detection and pun location. In-depth analyses also verify the effectiveness and robustness of PCPR.
- We release our implementations and pre-trained phoneme embeddings at https://github.com/joey1993/pun-recognition to facilitate future research.
Pun Recognition and Generation To recognize puns, Miller et al. (2017) summarize several systems for the SemEval 2017 tasks. To detect the pun, Pedersen (2017) supposes that if there is one pun in the sentence, when adopting different Word Sense Disambiguation (WSD) methods, the sense assigned to the sentence will be different. To locate the pun, based on the WSD results for pun detection, they choose the last word which changes the senses between different WSD runs. Even though this method can tackle both homographic and heterographic pun detection, it does not use any pretrained embedding model. Xiu et al. (2017) detect the pun in the sentence using similarity features which are calculated on sense vectors or cluster center vectors. To locate the pun, they use an unsupervised system by scoring each word in the sentence and choosing the word with the smallest score. However, this model exclusively relies on semantics to detect the heterographic puns but ignores the rich information embedded in the pronunciations. Doogan et al. (2017) leverage word embeddings as well as the phonetic information by concatenating pronunciation strings, but the concatenation has limited expression ability. They also mention that their systems suffer for short sentences as word embeddings do not have much context information.
Besides, Zou and Lu (2019) jointly detect and locate the pun from a sequence labeling perspective by employing a new tagging schema. Diao et al. (2018) expand word embeddings using WordNet to settle the polysemy of homographic puns, following by a neural attention mechanism to extract the collocation to detect the homographic pun. However, all these methods only make use of limited context information. Other than the pun recognition, Yu et al. (2018) generate homographic puns without requiring any pun data for training. He et al. (2019) improve the homographic pun generation based on the "local-global surprisal principle" which posits that the pun word and the alternative word have a strong association with the distant and immediate context respectively.
Pronunciation Embeddings Word embeddings assign each word with a vector so that words with similar semantic meanings are close in the embedding space. Most word embedding models only make use of text information and omitting the rich information contained in the pronunciation. How-
ever, the pronunciation is also an important part of the language (Zhu et al., 2018). Prior studies have demonstrated that the phonetic information can be used in speech recognition (Bengio and Heigold, 2014), spell correction (Toutanova and Moore, 2002) and speech synthesis (Miller, 1998a). By projecting to the embedding space, words sound alike are nearby to each other (Bengio and Heigold, 2014). Furthermore, Kamper et al. (2016) make use of word pairs information to improve the acoustic word embedding. Zhu et al. (2018) show that combining the pronunciation with the writing texts can help to improve the performance of word embeddings. However, these pronunciation embeddings are word-level features, while in our approach, we make use of syllabic pronunciations which is phoneme-level and could help with the out-of-vocabulary (OOV) situation. Luo et al. (2019) also propose an adversarial generative network for pun generation, which does not require any pun corpus.
Contextualized Word Embeddings Traditional word embeddings assign a fixed vector to one word even if the word has multiple meanings under different contexts (e.g., "the river bank" v.s. "the commercial bank"). McCann et al. (2017) combine the pivot word embeddings as well as the contextual embeddings generated by an encoder from a supervised neural machine translation task. Peters et al. (2017) enrich the word embeddings by the contextual information extracted from a bidirectional language model. (Devlin et al., 2018) learn the language embedding by stacking multiple transformer layers with masked language model objective which advances the state-of-the-art for many NLP tasks. Yang et al. (2019) enable learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and solve the problem of pretrain-finetune discrepancy.
3 Pronunciation-attentive Contextualized Pun Recognition
In this section, we first formally define the problem and then introduce the proposed method, PCPR.
Suppose the input text consists of a sequence of  $N$  words  $\{w_{1},w_{2},\dots ,w_{N}\}$ . For each word  $w_{i}$  with  $M_{i}$  phonemes in its pronunciation, the phonemes are denoted as  $R(w_{i}) = \{r_{i,1},r_{i,2},\dots ,r_{i,M_{i}}\}$ ,
where  $r_{i,j}$  is the  $j$ -th phoneme in the pronunciation of  $w_i$ . These phonemes are given by a dictionary. In this paper, we aim to recognize potential puns in the text with two tasks, including pun detection and pun location, as described in the following.
Task 1: Pun Detection. The pun detection task identifies whether a sentence contains a pun. Formally, the task is modeled as a classification problem with binary label  $y^{D}$ .
Task 2: Pun Location. Given a sentence containing at least a pun, the pun location task aims to unearth the pun word. More precisely, for each word  $w_{i}$ , we would like to predict a binary label  $y_{i}^{L}$  that indicates if  $w_{i}$  is a pun word.
In addition to independently solving the above two tasks, the ultimate goal of pun recognition is to build a pipeline from scratch to detect and then locate the puns in texts. Hence, we also evaluate the end-to-end performance by aggregating the solutions for two tasks.
Figure 1 shows the overall framework of the proposed Pronunciation-attentive Contextualized Pun Recognition (PCPR). For each word in the input text, we first derive two continuous vectors, including contextualized word embedding and pronunciation embedding, as representations in different aspects. Contextualized word embeddings derive appropriate word representations with consideration of context words and capture the accurate semantics in the text. To learn the phonological characteristics, each word is divided into phonemes while each phoneme is projected to a phoneme embedding space, thereby obtaining pronunciation embeddings with the attention mechanism (Bahdanau et al., 2015). Finally, a self-attentive encoder blends contextualized word embeddings and pronunciation embeddings to capture the overall semantics for both pun detection and location.
3.3 Contextualized Word Embeddings
The context is essential for interpreting a word in the text. Hence, we propose to apply contextualized word embeddings to derive word representations. In the framework of PCPR, any contextualized word embedding method, such as BERT (Devlin et al., 2018), ELMo (Peters et al., 2018), and XLNet (Yang et al., 2019), can be utilized. Here, we choose BERT to derive contextualized word embeddings without loss of generality.
Figure 1: The overall framework of PCPR. We leverage the self-attention mechanism to jointly model contextualized embeddings and phonological representations. PCPR can tackle both pun detection and pun location tasks.
BERT deploys a multi-layer bidirectional encoder based on transformers with multi-head self-attention (Vaswani et al., 2017) to model words in the text after integrating both word and position embeddings (Sukhbaatar et al., 2015). As a result, for each word, a representative contextualized embedding is derived by considering both the specific word and all contexts in the document. Here we denote  $T_{i}^{C}$  as the  $d_{C}$ -dimensional contextualized word embedding for the word  $w_{i}$ . In addition, BERT contains a special token [CLS] with an embedding vector in BERT to represent the semantics of the whole input text.
3.4 Pronunciation Embeddings
To learn the phonological characteristics of words, PCPR models the word phonemes. For each phoneme  $r_{i,j}$  of the word  $w_i$ , we project  $r_{i,j}$  to a  $d_P$ -dimensional embedding space as a trainable vector  $u_{i,j}$  to represent its phonological properties.
Based on the phoneme embeddings of a word, we apply the attention mechanism (Bahdanau et al., 2015) to simultaneously identify important phonemes and derive the pronunciation embedding  $T_{i}^{P}$ . Specifically, the phoneme embeddings are transformed by a fully-connected hidden layer to measure the importance scores  $\alpha_{i}^{P}$  as follows:
$$
v _ {i, j} = \tanh (\mathcal {F} _ {P} (u _ {i, j})),
$$
$$
\alpha_ {i, j} ^ {P} = \frac {v _ {i , j} ^ {\mathsf {T}} v _ {s}}{\sum_ {k} v _ {i , k} ^ {\mathsf {T}} v _ {s}},
$$
where  $\mathcal{F}_P(\cdot)$  is a fully-connected layer with  $d_A$  outputs and  $d_A$  is the attention size;  $v_{s}$  is a  $d_{A}$ -dimensional context vector that estimates the importance score of each pronunciation embedding. Finally, the pronunciation embeddings  $T_{i}^{P}$  can
be represented as the weighted combination of phoneme embeddings as follows:
$$
T _ {i} ^ {P} = \sum_ {j} \alpha_ {i, j} u _ {i, j}.
$$
Moreover, we can further derive the joint embedding  $T_{i}^{J}$  to indicate both word semantics and phonological knowledge for the word  $w_{i}$  by concatenating two different embeddings as follows:
$$
T _ {i} ^ {J} = \left[ T _ {i} ^ {C}; T _ {i} ^ {P} \right].
$$
Note that the joint embeddings are  $d_J$ -dimensional vectors, where  $d_J = d_C + d_P$ .
3.5 Pronunciation-attentive Contextualized Embedding with Self-attention
For the task of pun detection, understanding the meaning of input text is essential. Due to its advantages of interpretability over convolutional neural network (LeCun et al., 1995) and recurrent neural network (Schuster and Paliwal, 1997), we deploy the self-attention mechanism (Vaswani et al., 2017) to capture the overall semantics represented in the joint embeddings. For each word  $w_{i}$ , the self-attention mechanism estimates an importance vector  $\alpha_{i}^{S}$ :
$$
\mathcal {F} _ {S} (T) = \mathrm {S o f t m a x} (\frac {T T ^ {\intercal}}{\sqrt {d}}) T,
$$
$$
\alpha_ {i} ^ {S} = \frac {\exp (\mathcal {F} _ {S} (T _ {i} ^ {J}))}{\sum_ {j} \exp (\mathcal {F} _ {S} (T _ {j} ^ {J}))},
$$
where  $\mathcal{F}_S(\cdot)$  is the function to estimate the attention for queries, and  $d$  is a scaling factor to avoid extremely small gradients. Hence, the self-attentive embedding vector is computed by aggregating joint embeddings:
$$
T _ {[ \mathrm {A T T} ]} ^ {J} = \sum_ {i} \alpha_ {i} ^ {S} \cdot T _ {i} ^ {J}.
$$
Note that the knowledge of pronunciations is considered by the self-attentive encoder but not the contextualized word encoder. Finally, the pronunciation-attentive contextualized representation for the whole input text can be derived by concatenating the overall contextualized embedding and the self-attentive embedding:
$$
T _ {[ \mathrm {C L S} ]} ^ {J} = \left[ T _ {[ \mathrm {C L S} ]} ^ {C}; T _ {[ \mathrm {A T T} ]} ^ {J} \right].
$$
Moreover, each word  $w_{i}$  is benefited from the selfattentive encoder and is represented by a joint embedding:
$$
T _ {\mathrm {i}, [ \mathrm {A T T} ]} ^ {J} = \alpha_ {i} ^ {S} \cdot T _ {i} ^ {J}.
$$
3.6 Inference and Optimization
Based on the joint embedding for each word and the pronunciation-attentive contextualized embedding for the whole input text, both tasks can be tackled with simple fully-connected layers.
Pun Detection. Pun detection is modeled as a binary classification task. Given the overall embedding for the input text  $T_{[CLS]}^{J}$ , the prediction  $\hat{y}^{D}$  is generated by a fully-connected layer and the softmax function:
$$
\hat {y} ^ {D} = \operatorname * {a r g m a x} _ {k \in \{0, 1 \}} \mathcal {F} _ {D} (T _ {[ \mathrm {C L S} ]} ^ {J}) _ {k},
$$
where  $\mathcal{F}_D(\cdot)$  derives the logits of two classes in binary classification.
Pun Location. For each word  $w_{i}$ , the corresponding self-attentive joint embedding  $T_{\mathrm{i},[\mathrm{ATT}]}^{J}$  is applied as features for pun location. Similar to pun detection, the prediction  $\hat{y}_i^L$  is generated by:
$$
\hat{y}_{i}^{L} = \operatorname *{argmax}_{k\in \{0,1\}}\mathcal{F}_{L}(T_{i,\text{[ATT]}}^{J})_{k},
$$
where  $\mathcal{F}_L(\cdot)$  derives two logits for classifying if a word is a pun word.
Since both tasks focus on binary classification, we optimize the model with cross-entropy loss.
In this section, we describe our experimental settings and explain the results and interpretations. We will verify some basic assumptions of this paper: (1) the contextualized word embeddings and pronunciation embeddings are both beneficial to the pun detection and location tasks; (2) the attention mechanism can improve the performance.
Dataset	SemEval	PTD
Homo	Hetero
Examples w/ Puns	1,607	1,271	2,423
Examples w/o Puns	643	509	2,403
Total Examples	2,250	1,780	4,826
Table 2: Data statistics. "Homo" and "Hetero" denote homographic and heterographic puns. Pun detection employs all of the examples in the two datasets while pun location only exploits the examples with puns in SemEval due to the limitation of annotations.
Experimental Datasets. We conducted experiments on the SemEval 2017 shared task 7 dataset<sup>1</sup> (SemEval) (Miller et al., 2017) and the Pun of The Day dataset (PTD) (Yang et al., 2015). For pun detection, the SemEval dataset consists of 4,030 and 2,878 examples for pun detection and location while each example with a pun can be a homographic or heterographic pun. In contrast, the PTD dataset contains 4,826 examples without labels of pun types. Table 2 further shows the data statistics. The two experimental datasets are the largest publicly available benchmarks that are used in the existing studies. SemEval-2017 dataset contains punning and non-punning jokes, aphorisms, and other short texts composed by professional humorists and online collections. Hence, we assume the genres of positive and negative examples should be identical or extremely similar.
Evaluation Metrics. We adopt precision (P), recall (R), and  $F_{1}$  score (Schütze et al., 2007; Powers, 2011) to compare the performance of PCPR with previous studies in both pun detection and location. More specifically, we apply 10-fold cross-validation to conduct evaluation. For each fold, we randomly select  $10\%$  of the instances from the training set for development. To conduct fair comparisons, we strictly follow the experimental settings in previous studies (Zou and Lu, 2019; Cai et al., 2018) and include their reported numbers in the comparisons.
Implementation Details. For data pre-processing, all of the numbers and punctuation marks are removed. The phonemes of each word are derived by the CMU Pronouncing Dictionary<sup>2</sup>. We initialize the phoneme embeddings by using the fastText
1http://alt.qcri.org/semeval2017/ task7/ 2http://svn.code.sf.net/p/cmusphinx/ code/trunk/cmudit/
(a) Phoneme emb. size  $d_P$
(b) Attention size  $d_A$
Figure 2: Pun location performance over different phoneme embedding sizes  $d_{P}$  and attention sizes  $d_{A}$  on the SemEval dataset.
word embedding (Mikolov et al., 2018) trained on Wikipedia articles<sup>3</sup> crawled in December, 2017. The PCPR is implemented in PyTorch while the fused Adam optimizer (Kingma and Ba, 2014) optimizes the parameters with an initial learning rate of  $5 \times 10^{-5}$ . The dropout and batch size are set as  $10^{-1}$  and 32. We follow BERT (BASE) (Devlin et al., 2018) to use 12 Transformer layers and self-attention heads. To clarify, in PCPR, tokens and phonemes are independently processed, so the tokens processed with WordPiece tokenizer (Wu et al., 2016) in BERT are not required to line up with phonemes for computations. To deal with the out-of-vocabulary words, we use the output embeddings of the first WordPiece tokens as the representatives, which is consistent with many state-of-the-art named entity recognition approaches (Devlin et al., 2018; Lee et al., 2019). We also create a variant of PCPR called CPR by exploiting only the contextualized word encoder without considering phonemes to demonstrate the effectiveness of pronunciation embeddings.
To tune the hyperparameters, we search the phoneme embedding size  $d_{P}$  and the attention size  $d_{A}$  from  $\{8,16,32,64,128,256,512\}$  as shown in Figure 2. For the SemEval dataset, the best setting is  $(d_{P} = 64,d_{A} = 256)$  for the homographic puns while heterographic puns favor  $(d_{P} = 64,d_{A} = 32)$ . For the PTD dataset,  $(d_{P} = 64,d_{A} = 32)$  can reach the best performance.
Baseline Methods. We compare PCPR with several baseline methods.
For the SemEval dataset, nine baseline methods are compared in the experiments, including Duluth (Pedersen, 2017), JU_CES_NLP (Pramanick and Das, 2017), PunFields (Mikhalkova and Karyakin, 2017), UWAV (Vadehra, 2017), Fermi (Indurthi and Oota, 2017), and
UWaterloo (Vechtomova, 2017). While most of them extract complicated linguistic features to train rule based and machine learning based classifiers. In addition to task participants, Sense (Cai et al., 2018) incorporates word sense representations into RNNs to tackle the homographic pun location task. The CRF (Zou and Lu, 2019) captures linguistic features such as POS tags, n-grams, and word suffix to model puns. Moreover, the Joint (Zou and Lu, 2019) jointly models two tasks with RNNs and a CRF tagger.
For the PTD dataset, four baseline methods with reported performance are selected for comparisons. MCL (Mihalcea and Strapparava, 2005) exploits word representations with multiple stylistic features while HAE (Yang et al., 2015) applies a random forest model with Word2Vec and human-centric features. PAL (Chen and Lee, 2017) trains a convolutional neural network (CNN) to learn essential feature automatically. Based on existing CNN models, HUR (Chen and Soo, 2018) improves the performance by adjusting the filter size and adding a highway layer.