SlowGuess/ACL_2020_OCR huggingface dataset:
Pun Detection. Table 3 presents the pun detection performance of methods for both homographic and heterographic puns on the SemEval dataset while Table 4 shows the detection performance on the PTD dataset. For the SemEval dataset, compared to the nine baseline models, PCPR achieves the highest performance with  $3.0\%$  and  $6.1\%$  improvements of  $F_{1}$  against the best among the baselines (i.e. Joint) for the homographic and heterographic datasets, respectively. For the PTD dataset, PCPR improves against HUR by  $9.6\%$ . Moreover, the variant CPR beats all of the baseline methods and shows the effectiveness of contextualized word embeddings. In addition, PCPR further improves the performances by  $2.3\%$  and  $1.1\%$  with the attentive pronunciation feature for detecting homographic and heterographic puns, respectively. An interesting observation is that pronunciation embeddings also facilitate homographic pun detection, implying the potential of pronunciation for enhancing general language modeling.
Pun Location. Table 3 shows that the proposed PCPR model achieves highest  $F_{1}$ -scores on both homographic and heterographic pun location tasks with  $10.9\%$  and  $15.9\%$  incredible increment against the best baseline method. The improvement is
3https://dumps.wikimedia.org/enwiki/latest/
Model	Homographic Puns	Heterographic Puns
Pun Detection	Pun Location	Pun Detection	Pun Location
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Duluth	78.32	87.24	82.54	44.00	44.00	44.00	73.99	86.62	68.71	-	-	-
JU_CSE_NLP	72.51	90.79	68.84	33.48	33.48	33.48	73.67	94.02	71.74	37.92	37.92	37.92
PunFields	79.93	73.37	67.82	32.79	32.79	32.79	75.80	59.40	57.47	35.01	35.01	35.01
UWAV	68.38	47.23	46.71	34.10	34.10	34.10	65.23	41.78	42.53	42.80	42.80	42.80
Fermi	90.24	89.70	85.33	52.15	52.15	52.15	-	-	-	-	-	-
UWaterloo	-	-	-	65.26	65.21	65.23	-	-	-	79.73	79.54	79.64
Sense	-	-	-	81.50	74.70	78.00	-	-	-	-	-	-
CRF	87.21	64.09	73.89	86.31	55.32	67.43	89.56	70.94	79.17	88.46	62.76	73.42
Joint	91.25	93.28	92.19	83.55	77.10	80.19	86.67	93.08	89.76	81.41	77.50	79.40
CPR	91.42	94.21	92.79	88.80	85.65	87.20	93.35	95.04	94.19	92.31	88.24	90.23
PCPR	94.18	95.70	94.94	90.43	87.50	88.94	94.84	95.59	95.22	94.23	90.41	92.28
Table 3: Performance of detecting and locating puns on the SemEval dataset. All improvements of PCPR and CPR over baseline methods are statistically significant at a  $95\%$  confidence level in paired  $t$ -tests. Comparing to PCPR, CPR does not model word pronunciations. Results show that both PCPR and CPR outperform baselines. With modeling pronunciations, PCPR performs the best.
Model	P	R	F1
MCL	83.80	65.50	73.50
HAE	83.40	88.80	85.90
PAL	86.40	85.40	85.70
HUR	86.60	94.00	90.10
CPR	98.12	99.34	98.73
PCPR	98.44	99.13	98.79
Table 4: Performance of pun detection on the PTD dataset.
Model	Homographic Puns	Heterographic Puns
P	R	F1	P	R	F1
Joint	67.70	67.70	67.70	68.84	68.84	68.84
PCPR	87.21	81.72	84.38	85.16	80.15	82.58
much larger than that on pun detection task. We posit the reason is that predicting pun locations relies much more on the comparative relations among different tokens in one sentence. As a result, contextualized word embeddings acquire an enormous advantage. By applying the pronunciation-attentive representations, different words with similar pronunciations are linked, leading to a much better pinpoint of pun word for the heterographic dataset. We notice that some of the baseline models such as UWaterloo, UWAV and PunFields have poor performances. These methods consider the word position in a sentence or calculate the inverse document frequency of words. We suppose such rule-based recognition techniques can hardly capture the deep semantic and syntactic properties of words.
Pipeline Recognition. The ultimate goal of pun
Table 5: Performance of pipeline recognition in the SemEval dataset.
Model	P	R	F1
PCPR	90.43	87.50	88.94
w/o Pre-trained Phoneme Emb.	89.37	85.65	87.47
w/o Self-attention Encoder	89.17	86.42	87.70
w/o Phonological Attention	89.56	87.35	88.44
Table 6: Ablation study on different features of PCPR for homographic pun detection on the SemEval dataset.
recognition is to establish a pipeline to detect and then locate puns. Table 5 shows the pipeline performances of PCPR and Joint, which is the only baseline with reported pipeline performance for recognizing the homographic and heterographic puns in the SemEval dataset. Joint achieves suboptimal performance and the authors of Joint attribute the performance drop to error propagation. In contrast, PCPR improves the  $F_{1}$ -scores against Joint by  $24.6\%$  and  $20.0\%$  on two pun types.
Ablation Study. To better understand the effectiveness of each component in PCPR, we conduct an ablation study on the homographic puns of the SemEval dataset. Table 6 shows the results on taking out different features of PCPR, including pre-trained phoneme embeddings, the self-attentive encoder, and phonological attention. Note that we use the average pooling as an alternative when we remove the phonological attention module. As a result, we can see the drop after removing each of the three features. It shows that all these components are essential for PCPR to recognize puns.
Attentive Weights Interpretation. Figure 3 illustrates the self-attention weights  $\alpha_{i}^{S}$  of three ex
Figure 3: Visualization of attention weights of each pun word (marked in pink) in the sentences. A deeper color indicates a higher attention weight.
Sentence	Pun	CPR	PCPR
In the dark? Follow the son.
He stole an invention and then told patent lies.
A thief who stole a calendar got twelve months.	son
patent
got	-
patent	son
lies
-
Table 7: A case study of the model predictions for the pun location task of SemEval 2017.
(a) Pun Detection
(b) Pun Location
Figure 4: Pun recognition performance over different text lengths for homographic and heterographic puns on the SemEval dataset.
samples from heterographic puns in the SemEval dataset. The word highlighted in the upper sentence (marked in pink) is a pun while we also color each word of the lower sentence in blue according to the magnitude of its attention weights. The deeper colors indicate higher attention weights. In the first example, busy has the largest weight because it has the most similar semantic meaning as harried. The barber also has relatively high weights. We suppose it is related to hairy which should be the other word of this double entendre. Similar, the zoo is corresponded to lion while phone and busy indicate line for the pun. Moreover, boating confirms sail while store supports sale. Interpreting the weights out of our self-attentive encoder explains the significance of each token when the model detects the pun in the context. The phonemes are essential in these cases because they strengthen the relationship among words with distant semantic meanings but similar phonological expressions.
Sensitivity to Text Lengths. Figure 4 shows the performance of pun detection and location over different text lengths for homographic and heterographic puns in the SemEval dataset. For both tasks, the performance gets higher when the text lengths are longer because the context informa
tion is richer. Especially in the pun detection task, we observe that our model requires longer contexts (more than 20 words) to detect the homographic puns. However, shorter contexts (less than 10 words) are adequate for heterographic pun detection, which indicates the contribution from phonological features. In short, the results verify the importance of contextualized embeddings and pronunciation representations for pun recognition.
Case Study and Error Analysis. Table 7 shows the results of a case study with the outputs of CPR and PCPR. In the first case, the heterographic pun comes from the words son and sun. CPR fails to recognize the pun word with limited context information while the phonological attention in PCPR helps to locate it. However, the pronunciation features in some cases can mislead the model to make wrong predictions. For example, patent in the second sentence is a homographic pun word and has several meanings, which can be found with the contextual features. Besides, the phonemes in lies are ubiquitous in many other words like laws, thereby confusing the model. In the last case, got is a widely used causative with dozens of meanings so that the word is hard to be recognized as a pun word with its contextual and phonological features.
In this paper, we propose a novel approach, PCPR, for pun detection and location by leveraging a contextualized word encoder and modeling phonemes as word pronunciations. Moreover, we would love to apply the proposed model to other problems, such as general humor recognition, irony discovery, and sarcasm detection, as the future work.
Agnese Augello, Gaetano Saccone, Salvatore Gaglio, and Giovanni Pilato. 2008. Humorist bot: Bringing computational humour in a chat-bot system. In CI-SIS 2008, pages 703-708.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015.
Samy Bengio and Georg Heigold. 2014. Word embeddings for speech recognition. In ISCA 2014.
Dario Bertero and Pascale Fung. 2016. Predicting humor response in dialogues from tv sitcoms. In ICASSP 2016, pages 5780-5784.
Vladislav Blinov, Valeria Bolotova-Baranova, and Pavel Braslavski. 2019. Large dataset and language model fun-tuning for humor recognition. In ACL 2019, pages 4027-4032.
Yitao Cai, Yin Li, and Xiaojun Wan. 2018. Sense-aware neural models for pun location in texts. In ACL 2018, pages 546-551.
Lei Chen and Chong MIn Lee. 2017. Predicting audience's laughter using convolutional neural network. arXiv preprint arXiv:1702.02584.
Peng-Yu Chen and Von-Wun Soo. 2018. Humor recognition using deep learning. In *NAACL* 2018, pages 113-117.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Yufeng Diao, Hongfei Lin, Di Wu, Liang Yang, Kan Xu, Zhihao Yang, Jian Wang, Shaowu Zhang, Bo Xu, and Dongyu Zhang. 2018. Weca: A wordnet-encoded collocation-attention network for homographic pun recognition. In EMNLP 2018, pages 2507-2516.
Samuel Doogan, Aniruddha Ghosh, Hanyang Chen, and Tony Veale. 2017. Idiom savant at semeval-2017 task 7: Detection and interpretation of english puns. In SemEval-2017, pages 103-108.
Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. Semeval-2015 task 11: Sentiment analysis of figurative language in twitter. In SemEval 2015, pages 470-478.
He He, Nanyun Peng, and Percy Liang. 2019. Pun generation with surprise. In *NAACL* 2019.
Lluis-F Hurtado, Encarna Segarra, Ferran Pla, Pascual Carrasco, and José-Angel González. 2017. Elirf-upv at semeval-2017 task 7: Pun detection and interpretation. In SemEval-2017, pages 440-443.
Vijayasaradhi Indurthi and Subba Reddy Oota. 2017. Fermi at semeval-2017 task 7: Detection and interpretation of homographic puns in english language. In SemEval-2017, pages 457-460.
Aaron Jaech, Rik Koncel-Kedzierski, and Mari Ostendorf. 2016. Phonological pun-derstanding. In ACL 2016, pages 654-663.
Herman Kamper, Weiran Wang, and Karen Livescu. 2016. Deep convolutional acoustic word embeddings using word-pair side information. In ICASSP 2016, pages 4950-4954. IEEE.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Yann LeCun, Yoshua Bengio, et al. 1995. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746.
Fuli Luo, Shunyao Li, Pengcheng Yang, Lei Li, Baobao Chang, Zhifang Sui, and Xu SUN. 2019. Pun-GAN: Generative adversarial network for pun generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NeurIPS 2017, pages 6294-6305.
Alan K Melby and Terry Warner. 1995. The possibility of language: A discussion of the nature of language, with implications for human and machine translation, volume 14. John Benjamins Publishing.
Rada Mihalcea and Carlo Strapparava. 2005. Making computers laugh: Investigations in automatic humor recognition. In EMNLP 2005, pages 531-538.
Elena Mikhalkova and Yuri Karyakin. 2017. Punfields at semeval-2017 task 7: Employing roget's thesaurus in automatic pun recognition and interpretation. arXiv preprint arXiv:1707.05479.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In LREC 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Corey Andrew Miller. 1998a. Pronunciation modeling in speech synthesis.
George Miller. 1998b. WordNet: An electronic lexical database. MIT press.
Tristan Miller, Christian Hempelmann, and Iryna Gurevych. 2017. Semeval-2017 task 7: Detection and interpretation of english puns. In SemEval-2017, pages 58-68.
Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM computing surveys (CSUR), 41(2):10.
Anton Nijholt, Andreea I Niculescu, Valitutti Alessandro, and Rafael E Banchs. 2017. Humor in humancomputer interaction: a short survey.
Dieke Oele and Kilian Evang. 2017. Buzzsaw at semeval-2017 task 7: Global vs. local context for interpreting and locating homographic english puns with sense embeddings. In SemEval-2017, pages 444-448.
Ted Pedersen. 2017. Duluth at semeval-2017 task 7: Puns upon a midnight dreary, lexical semantics for the weak and weary. arXiv preprint arXiv:1704.08388.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP 2014, pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In *NAACL* 2018, pages 2227-2237.
Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. arXiv preprint arXiv:1705.00108.
David Martin Powers. 2011. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation.
Aniket Pramanick and Dipankar Das. 2017. Ju cse nlp @ semeval 2017 task 7: Employing rules to detect and interpret english puns. In SemEval-2017, pages 432-435.
Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-2681.
Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2007. An introduction to information retrieval. Cambridge University Press,.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
Kristina Toutanova and Robert C Moore. 2002. Pronunciation modeling for improved spelling correction.
Ankit Vadehra. 2017. Uwav at semeval-2017 task 7: Automated feature-based system for locating puns. In SemEval-2017, pages 449-452.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Olga Vechtomova. 2017. Uwaterloo at semeval-2017 task 7: Locating the pun using syntactic characteristics and corpus-based metrics. In SemEval-2017, pages 421-425.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Yuhuan Xiu, Man Lan, and Yuanbin Wu. 2017. Ecnu at semeval-2017 task 7: Using supervised and unsupervised methods to detect and locate english puns. In SemEval-2017, pages 453-456.
Diyi Yang, Alon Lavie, Chris Dyer, and Eduard Hovy. 2015. Humor recognition and humor anchor extraction. In EMNLP 2015, pages 2367-2376.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Zhiwei Yu, Jiwei Tan, and Xiaojun Wan. 2018. A neural approach to pun generation. In ACL 2018.
Wenhao Zhu, Xin Jin, Jianyue Ni, Baogang Wei, and Zhiguo Lu. 2018. Improve word embedding using both writing and pronunciation. PloS one, 13(12):e0208785.
Yanyan Zou and Wei Lu. 2019. Joint detection and location of english puns. In *NAACL* 2019, pages 2117-2123.