SlowGuess/NAACL_2021_OCR huggingface dataset:
LOANT differs from MAML since, in the inner update, LOANT updates the sentence-level features  $z_{s}$  and  $z_{t}$  instead of the model parameters  $w$ . As  $z_{s}$  and  $z_{t}$  are usually of much smaller dimensions than  $w$ , this leads to accelerated training and reduced memory footprint. For example, in the BERT-base model (Devlin et al., 2019),  $L$  is 110 million and  $D$  is 768. Within the regular range of batch size  $B$ ,  $BD \ll L$ . In the experiments, we verify the benefits of LOANT in terms of accuracy and time and space complexity.
We conduct four cross-domain sarcasm detection experiments by transferring from an automatically collected dataset to a manually annotated dataset. The two automatically collected datasets include Ptáček (Ptáček et al., 2014) and Ghosh<sup>1</sup> (Ghosh and Veale, 2016), which treat tweets having particular hastags such as #sarcastic, #sarcasm or #not as sarcastic and others as not sarcastic. We crawled the Ptáček dataset using the NLTK API<sup>2</sup> according to the tweet ids published online<sup>3</sup>.
The two manually annotated datasets include SemEval-  $18^{4}$  (Van Hee et al., 2018) and iSarcasm
Dataset	Train	Val	Test	% Sarcasm
Ptáček	51009	5668	6298	49.50%
Ghosh	33373	3709	4121	44.84%
SemEval-18	3398	378	780	49.12%
iSarcasm	3116	347	887	17.62%
Table 1: Dataset statistics, including number of samples in each split and the proportion of sarcastic texts.
(Oprea and Magdy, 2020). SemEval-18 consists of both sarcastic and ironic tweets supervised by third-party annotators and thus is used for perceived sarcasm detection. The iSarcasm dataset contains tweets written by participants of an online survey and thus is an example of intended sarcasm detection.
Table 1 summarizes the statistics of the four datasets. The SemEval-18 dataset is balanced while the iSarcasm dataset is imbalanced. The two source datasets are more than ten times the size of the target datasets. For all datasets, we use the predefined test set and use a random  $10\%$  split of the training set as the development set.
We preprocessed all datasets using the lexical normalization tool for tweets from Baziotis et al. (2017). We cleaned the four datasets by dropping all the duplicate tweets within and across datasets, and trimmed the texts to a maximum length of 100. To deal with class imbalance, we performed upsampling on the target-domain datasets, so that both the sarcastic and non-sarcastic classes have the same size as source domain datasets.
We compare LOANT with several competitive single-task and multi-task baselines.
MIARN (Tay et al., 2018): A state-of-the-art short
$^{1}$ https://github.com/AniSkywalker/SarcasmDetection/tree/master/resource $^{2}$ http://www.nltk.org/howto/twitter.html $^{3}$ http://liks.fav.zcu.cz/sarcasm/ $^{4}$ https://github.com/Cyvhee/SemEval2018-Task3/tree/master/datasets
text sarcasm detection model ranked top-1 on the iSarcam dataset. The model is a co-attention based LSTM model which uses the word embeddings pretrained on Twitter data<sup>5</sup>.
Dense-LSTM (Wu et al., 2018): A state-of-the-art single-task sarcasm detection model ranked top-1 on the SemEval-18 dataset. The model is a densely connected LSTM network consisting of four BiLSTM layers and the word embeddings pretrained on two Twitter datasets.
BERT: We finetune the BERT model (Devlin et al., 2019) with an additional simple classifier directly on the target dataset.
S-BERT is a two-stage finetuning of the BERT model. We first finetune BERT on the source dataset and the best model is selected for further fine-tuning on the target dataset.
MTL: We implemented a multi-task learning (MTL) model, which has the same architecture as LOANT except that the domain discriminator is removed. We use BERT as the shared text encoding network.
MTL+LO: In this baseline, we applied latent optimization to MTL. As MTL does not have the adversarial discriminator, we use the domain-specific losses to optimize latent representations:
$$
z _ {s} ^ {\prime} = z _ {s} - \gamma \frac {\partial \mathcal {L} _ {s} \left(z _ {s}\right)}{\partial z _ {s}} \tag {18}
$$
$$
z _ {t} ^ {\prime} = z _ {t} - \gamma \frac {\partial \mathcal {L} _ {t} \left(z _ {t}\right)}{\partial z _ {t}} \tag {19}
$$
We use the above to replace Equations 5 and 6 and keep the rest training steps unchanged. This model is compared against MTL to study the effects of LO in non-adversarial training for cross-domain sarcasm detection.
ANT: This is the conventional adversarial neural transfer model with the same architecture as LOANT. The only difference is that we do not apply latent optimization. For fair comparisons, we use BERT as the text encoder.
ANT+MAML: In Section 3.3, we discussed the similarity between LO and MAML. Therefore, we create a baseline that uses a MAML-like strategy for encouraging the collaboration of different loss terms. Instead of optimizing the latent representation  $z_{s}$  and  $z_{t}$ , we first take a SGD step in the
parameter space of  $w_{b}$
$$
w _ {b} ^ {\prime} = w _ {b} + \gamma \frac {\partial \mathcal {L} _ {d} \left(z _ {s} , z _ {t}\right)}{\partial w _ {b}}. \tag {20}
$$
After that, we use  $w_{b}^{\prime}$  to compute the gradients used in the actual updates to all model parameters, including  $w_{b}$ .
Model Settings. For all models using the BERT text encoder, we use the uncased version of the BERT-base model and take the 768-dimensional output from the last layer corresponding to the [CLS] token to represent a sentence. The BERT parameters are always shared between domains. For other network components, we randomly initialize the dense layers and classifiers. To minimize the effect of different random initializations, we generate the same set of initial parameters for each network component and use them across all baselines wherever possible.
The source dense layer, the shared dense layer, and the target dense layer are single linear layers with input size of 768 and output size of 768 followed by the tanh activation. The classifier in all models consists of two linear layers. The first linear layer has input size of  $768 \times 2$  (taking both shared and domain-specific features) and output size of 768 followed by the ReLU activation. The second linear layer has input size 768 and output size 2 for binary classification. After that we apply the softmax operation. More details can be found in Appendix B.
Training Setting. We optimize all models using Adam (Kingma and Ba, 2014) with batch size of 128. We tune the learning rate (LR) on the development set from 1e-5 to 1e-4 in increments of 2e-5. To objectively assess the effects of latent optimization (LO), we first find the best LR for the base models such as ANT and MTL. After that, with the best LR unchanged, we apply LO to ANT and MTL. We use the cosine learning rate schedule for all models. All models are trained for 5 epochs on Nvidia V100 GPUs with 32GB of memory in mixed precision. Due to the large model size and pretrained weights of BERT, 5 epochs are sufficient for convergence.
Evaluation Metrics. Following (Wu et al., 2018; Van Hee et al., 2018; Oprea and Magdy, 2020), we select and compare models using the F-score on the sarcastic class in each dataset. We additionally
$^{5}$ https://nlp.stanford.edu/projects/glove/
Target: SemEval-18	Target: iSarcasm
Model	F-score	Recall	Precision	Model	F-score	Recall	Precision
Single-task	Random†	0.3730	0.3730	0.3730	SIARN‡	0.3420	0.7820	0.2190
Unigram SVM†	0.5890	0.6590	0.5320	MIARN‡	0.3640	0.7930	0.2360
LSTM†	0.5260	0.4440	0.6450	LSTM‡	0.3360	0.7470	0.2170
DenseLSTM *	0.6510	0.7106	0.6005	DenseLSTM‡	0.3180	0.2760	0.3750
BERT	0.6626	0.7055	0.6246	BERT	0.3492	0.4904	0.2711
Source: Ptáče	S-BERT	0.6676	0.7055	0.6337	S-BERT	0.3710	0.5541	0.2788
MTL	0.6404	0.7896	0.5386	MTL	0.3767	0.3503	0.4074
ANT	0.6348	0.8187	0.5184	ANT	0.3857	0.5159	0.3079
MTL+LO	0.6598	0.7346	0.5989	MTL+LO	0.4379	0.4267	0.4496
ANT+MAML	0.6454	0.7540	0.5641	ANT+MAML	0.3951	0.5605	0.2923
LOANT (ours)	0.6702	0.8025	0.5754	LOANT (ours)	0.4642	0.4968	0.4357
Source: Ghosh	S-BERT	0.6512	0.7766	0.5607	S-BERT	0.3383	0.5732	0.2400
MTL	0.6525	0.7475	0.5789	MTL	0.3838	0.5159	0.3056
ANT	0.6626	0.8899	0.5278	ANT	0.4063	0.4904	0.3468
MTL+LO	0.6622	0.8058	0.5620	MTL+LO	0.3987	0.4012	0.3962
ANT+MAML	0.6338	0.7281	0.5610	ANT+MAML	0.3589	0.4904	0.2830
LOANT (ours)	0.6818	0.7734	0.6096	LOANT (ours)	0.4101	0.4649	0.3668
† Results reported in (Van Hee et al., 2018), * in (Wu et al., 2018) and ‡ in (Oprea and Magdy, 2020).
report the corresponding Recall and Precision. In all our experiments, we use the development set for model selection and report their performance on the test set. To evaluate the efficiency of LOANT versus MAML-based training, we also compare their required GPU memory and average training time in each epoch. We compare models on the target domain datasets. Additional multi-domain performance can be found in Appendix C.
4.4 Comparison with the States of the Art
We compare LOANT with state-of-the-art methods on the SemEval-18 dataset (Van Hee et al., 2018) and the iSarcasm dataset (Oprea and Magdy, 2020). Table 2 presents the test performance of LOANT and all baseline models. Our LOANT model consistently outperforms all single-task baselines by large margins. In particular, LOANT outperforms MIARN by  $10.02\%$  on iSarcasm (Oprea and Magdy, 2020) whereas the fine-tuned BERT achieved  $1.48\%$  lower than MIARN. On SemEval-18, the fine-tuned BERT achieves better test performance than other four single-task baselines. The results indicate that fine-tuning BERT, a popular baseline, does not always outperform the traditional LSTM networks specifically designed for the task. We hypothesize that the large BERT model can easily overfit the small datasets used, which highlights the challenge of sarcasm detection.
Table 2: Performance on the sarcastic class reported by single-task and multi-task models on the same test sets. The best performed F-score on the four groups of transfer learning are in bold. The best single task learning results are underlined.
SemEval-18	iSarcasm
Model	RAM/Time	RAM/Time
Source: Ptáče	LOANT	1.01x/2.41x	1.01x/2.55x
MTL+LO	1.01x/1.92x	1.01x/1.91x
ANT	1.00x/1.00x	1.00x/1.00x
ANT + MAML	1.99x/8.31x	1.93x/10.2x
Source: Ghosh	LOANT	1.01x/2.44x	1.01x/1.94x
MTL+LO	1.01x/1.94x	1.01x/1.89x
ANT	1.00x/1.00x	1.00x/1.00x
ANT + MAML	1.99x/8.41x	1.93x/10.7x
Table 3: Running time and maximum memory footprint for different transfer learning methods.
4.5 Transfer Learning Performance
The middle and bottom sections of Table 2 present the test performance of six transfer learning models (S-BERT, MTL, ANT, MTL+LO, ANT+MAML, and LOANT) under four groups of transfer learning experiments. These models generally outperform the single-task models, demonstrating the importance of transfer learning. Among these, we have the following observations.
Effects of the Domain Discriminator. The performance differences between MTL and ANT can be explained by the addition of the domain discriminator, which encourages the shared features under the source domain and the target domain to have the same distributions. In the four pairs of experiments, ANT marginally outperforms MTL by an average of  $0.9\%$  F-score. In the Ptcek  $\rightarrow$  SemEval-18 experiment, the domain discriminator causes F-score to decrease by  $0.56\%$ . Overall, the
benefits of the adversarial discriminator to transfer learning appear to be limited. As discussed earlier, the competition between the domain-specific losses and the negative domain discrimination loss may have contributed to the ineffectiveness of ANT.
Effects of Latent Optimization. We can observe the effects of LO by comparing ANT with LOANT and comparing MTL with MTL+LO. Note that in these experiments we adopted the best learning rates for the baseline models ANT and MTL rather than the latent-optimized models. On average, LOANT outperforms ANT by  $3.42\%$  in F-score and MTL+LO outperforms MTL by  $2.63\%$ , which clearly demonstrates the benefits provided by latent optimization.
Latent Space vs. Model Parameter Space. In the ANT+MAML baseline, we adopt a MAML-like optimization strategy, which performs the look-ahead in the BERT parameter space instead of the latent representation space. Interestingly, this strategy does not provide much improvements and on average performs  $1.40\%$  worse than ANT. LOANT clearly outperforms ANT+MAML.
In addition, optimization in the latent space also provides savings in computational time and space requirements. Table 3 shows the time and memory consumption for different transfer learning methods. Adding LO to ANT has minimal effects on the memory usage, but adding MAML nearly doubles the memory consumption. On average, ANT+MAML increases the running time of LOANT by 3.1 fold.
The Influence of Domain Divergence. In transfer learning, the test performance depends on the similarity between the domains. We thus investigate the dissimilarity between datasets using the Kullback-Leibler (KL) divergence between the unigram probability distributions,
$$
d _ {K L} = \sum_ {g \in V} P _ {t} (g) \log \frac {P _ {t} (g)}{P _ {s} (g)}. \tag {21}
$$
where  $P_{s}(g)$  and  $P_{t}(g)$  are the probabilities of unigram  $g$  for the source domain and target domain respectively.  $V$  is the vocabulary. Table 4 shows the results. Ptáček is more similar to the two target datasets than Ghosh. Among the two target datasets, iSarcasm is more similar to Ptáček than SemEval-18.
Comparing LOANT and ANT, we observe that the largest improvement,  $7.85\%$ , happens in the
SemEval-18	iSarcasm
Ptáček	0.1631	0.0521
Ghosh	0.2300	0.2217
Table 4: The KL divergence of word probability over the overlapped vocabulary for each pair of domains.
Ptáček → iSarcasm transfer where domain divergence is the smallest. The Ptáček → SemEval-18 transfer comes in second with  $3.54\%$ . Transferring from Ghosh yields smaller improvements. Further, we observe the same trend in the comparison between MTL+LO and MTL. The largest improvement brought by LO is  $6.12\%$  in the Ptáček → iSarcasm transfer. As one may expect, applying LO leads to greater performance gains when the two domains are more similar.
Transfer learning holds the promise for the effective utilization of multiple datasets for sarcasm detection. In this paper, we propose a latent optimization (LO) strategy for adversarial transfer learning for sarcasm detection. By providing look-ahead in the gradient updates, the LO technique allows multiple losses to accommodate each other. This proves to be particularly effective in adversarial transfer learning where the domain-specific losses and the adversarial loss potentially conflict with one another. With the proposed LOANT method, we set a new state of the art for the iSarcasm dataset. We hope the joint utilization of multiple datasets will contribute to the creation of contextualized semantic understanding that is necessary for successful sarcasm detection.
This research is supported by the National Research Foundation, Singapore under its the AI Singapore Programme (AISG2-RP-2020-019), NRF Investigatorship (NRF-NRFI05-2019-0002), and NRF Fellowship (NRF-NRFF13-2021-0006); the Joint NTU-WeBank Research Centre on Fintech (NWJ2020-008); the Nanyang Assistant/Associate Professorships (NAP); the RIE 2020 Advanced Manufacturing and Engineering Programmatic Fund (A20G8b0102), Singapore; NTU-SDU-CFAIR (NSC-2019-011). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the funding agencies.
Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the ACL 2016 student research workshop, pages 107-113.
Roee Aharoni, Melvin Johnson, and Orhan First. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Héctor Martínez Alonso and Barbara Plank. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In EACL.
Silvio Amir, Byron C. Wallace, Hao Lyu, and Paula Carvalho Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In CoNLL.
Waiss Azizian, Ioannis Mitliagkas, Simon Lacoste-Julien, and Gauthier Gidel. 2020. A tight and unified analysis of gradient-based methods for a whole spectrum of games. In AISTATS.
David Balduzzi, Sébastien Racanière, James Martens, Jakob N. Foerster, Karl Tuyls, and Thore Graepel. 2018. The mechanics of n-player differentiable games. In ICML.
David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Ninth international AAAI conference on web and social media. Citeseer.
Christos Baziotis, Nikos Pelekis, and Christos Doukkeridis. 2017. Datastories at SemEval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747-754, Vancouver, Canada. Association for Computational Linguistics.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine learning, 79(1):151-175.
Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL.
Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multi-modal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506-2515, Florence, Italy. Association for Computational Linguistics.
Soravit Changpinyo, Hexiang Hu, and Fei Sha. 2018. Multi-task learning for sequence tagging: An empirical study. In COLING.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. 2019. Bam! born-again multi-task networks for natural language understanding. In ACL.
Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, 12:2493-2537.
Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, page 107-116, USA. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615-1625, Copenhagen, Denmark. Association for Computational Linguistics.
Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, Istanbul, Turkey.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR.
Jean E. Fox Tree, J. Trevor D'Arcey, Alicia A. Hammond, and Alina S. Larson. 2020. The sarchasm: Sarcasm production and identification in spontaneous conversation. Discourse Processes, 57(5-6):507-533.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In ICML, volume 37, pages 1180-1189.
Ian Gemp and Sridhar Mahadevan. 2019. Global convergence to the equilibrium of gans using variational inequalities. arXiv 1808.01531.
Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 161-169.
Raymond W. Gibbs. 2000. Irony in talk among friends. Metaphor and Symbol, 15(1-2):5-27.
Roberto González-Ibanez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.
Roberto González-Ibanez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581-586.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2019. Autosem: Automatic task selection and mixing in multi-task learning. In NAACL.
Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757-762.
Anush Kamath, Sparsh Gupta, and Vitor Carvalho. 2019. Reversing gradients in adversarial domain adaptation for question dedduplication and textual entailment tasks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5545-5550.
Sandeepa Kannangara. 2018. Mining twitter for fine-grained political opinion polarity classification, ideology detection and sarcasm detection. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 751-752.
Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018a. A large self-annotated corpus for sarcasm. In Proceedings of the Eighth International Conference on Language Resources and Evaluation.
Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrachalli. 2018b. A large self-annotated corpus for sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Young-Bum Kim, Karl Stratos, and Dongchan Kim. 2017. Adversarial adaptation of synthetic or stale data. In Proceedings of the 55th Annual Meeting of