SlowGuess/NAACL_2022_OCR huggingface dataset:
Processing, EMNLP '21, pages 698-718, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social Chemistry 101: Learning to Reason about Social and Moral Norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP '20, pages 653-670, Online. Association for Computational Linguistics.
Batya Friedman, Peter H. Kahn, and Alan Borning. 2008. Value Sensitive Design and Information Systems. In The Handbook of Information and Computer Ethics, pages 69-101. John Wiley & Sons, Inc., Hoboken, NJ, USA.
Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kathleen McKeown, Mohit Bansal, and Avi Sil. 2021. InfoSurgeon: Cross-media fine-grained information consistency checking for fake news detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP '21, Online. Association for Computational Linguistics.
Jason Gabriel. 2020. Artificial Intelligence, Values, and Alignment. *Minds and Machines*, 30(3):411-437.
Justin Garten, Joe Hoover, Kate M. Johnson, Reihane Boghrati, Carol Iskiwitch, and Morteza Dehghani. 2018. Dictionaries and distributions: Combining Expert Knowledge and Large Scale Textual Data Content Analysis: Distributed Dictionary Representation. Behavior Research Methods, 50(1):344-361.
Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P. Wojcik, and Peter H. Ditto. 2013. Moral Foundations Theory: The Pragmatic Validity of Moral Pluralism. In Advances in Experimental Social Psychology, volume 47, pages 55-130. Elsevier, Amsterdam, the Netherlands.
Jesse Graham, Jonathan Haidt, and Brian A. Nosek. 2009. Liberals and Conservatives Rely on Different Sets of Moral Foundations. Journal of Personality and Social Psychology, 96(5):1029-1046.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning AI With Shared Human Values. In Proceedings of the 2021 International Conference on Learning Representations, ICLR '21, pages 1-29.
Myles Hollander and Douglas A. Wolfe. 1999. Nonparametric Statistical Methods. Wiley, New York, USA.
Joe Hoover, Gwenyth Portillo-Wightman, Leigh Yeh, Shreya Havaldar, Aida Mostafazadeh Davani, Ying Lin, Brendan Kennedy, Mohammad Atari, Zahra Kamel, Madelyn Mendlen, Gabriela Moreno,
Christina Park, Tingyee E. Chang, Jenna Chin, Christian Leong, Jun Yen Leung, Arineh Mirinjian, and Morteza Dehghani. 2020. Moral Foundations Twitter Corpus: A Collection of 35k Tweets Annotated for Moral Sentiment. Social Psychological and Personality Science, 11(8):1057-1071.
Frederic R. Hopp, Jacob T. Fisher, Devin Cornell, Richard Huskey, and René Weber. 2021. The extended Moral Foundations Dictionary (eMFD): Development and Applications of a Crowd-Sourced Approach to Extracting Moral Intuitions from Text. Behavior Research Methods, 53:232-246.
Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-Tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL '18, pages 328-339.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and Efficient Fine-Tuning for Pretrained Natural Language Models through Principled Regularized Optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL '20, pages 2177-2190. Association for Computational Linguistics.
Kristen Johnson and Dan Goldwasser. 2018. Classification of Moral Foundations in Microblog Political Discourse. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL '18, pages 720-730, Melbourne, Australia. Association for Computational Linguistics.
Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Joe Hoover, Ali Omrani, Jesse Graham, and Morteza Dehghani. 2021. Moral Concerns are Differentially Observable in Language. Cognition, 212:104696.
Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the Human Values behind Arguments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL '22, pages 1-13, Dublin, Ireland. Association for Computational Linguistics.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences of the United States of America, 114(13):3521-3526.
Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. 2018. Algorithmic Fairness. AEA Papers and Proceedings, 108:22-27.
Jonathan Kobbe, Ines Rehbein, Ioana Hulpus, and Heiner Stuckenschmidt. 2020. Exploring Morality in Argumentation. In Proceedings of the 7th Workshop on Argument Mining, pages 30-40, online. Association for Computational Linguistics.
Christopher A. Le Dantec, Erika Shehan Poole, and Susan P. Wyche. 2009. Values as Lived Experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, pages 1141-1150. ACM Press.
Zhizhong Li and Derek Hoiem. 2018. Learning without Forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2947.
Ying Lin, Joe Hoover, Gwenyth Portillo-Wightman, Christina Park, Morteza Dehghani, and Heng Ji. 2018. Acquiring Background Knowledge to Improve Moral Value Prediction. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining,ASONAM '18, pages 552-559, Barcelona, Spain. IEEE.
Enrico Liscio, Michiel van der Meer, Luciano C. Siebert, Catholijn M. Jonker, Niek Mouter, and Pradeep K. Murukannaiah. 2021. *Axes: Identifying and Evaluating Context-Specific Values*. In *Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems*, AAMAS '21, pages 799–808, Online. IFAAMAS.
Enrico Liscio, Michiel van der Meer, Luciano C. Siebert, Catholijn M. Jonker, and Pradeep K. Murukannaaiah. 2022. What Values Should an Agent Align With? Autonomous Agents and Multi-Agent Systems, 36(23):32.
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI '21, pages 13470-13479.
Ilia Markov and Walter Daelemans. 2021. Improving cross-domain hate speech detection by reducing the false positive rate. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 17-22, Online. Association for Computational Linguistics.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys, 54(6).
Siddharth Mehrotra, Catholijn M. Jonker, and Myrthe L. Tielman. 2021. More Similar Values, More Trust? The Effect of Value Similarity on Trust in Human-Agent Interaction. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES '21, pages 777-783. Association for Computing Machinery.
Nieves Montes and Carles Sierra. 2021. Value-Guided Synthesis of Parametric Normative Systems. In Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '21, pages 907-915, Online. IFAAMAS.
Marlon Mooijman, Joe Hoover, Ying Lin, Heng Ji, and Morteza Dehghani. 2018. Moralization in social networks and the emergence of violence during protests. Nature Human Behaviour, 2(6):389-396.
Niek Mouter, Jose Ignacio Hernandez, and Anatol Valerian Itten. 2021. Public Participation in Crisis Policymaking. How 30,000 Dutch Citizens Advised Their Government on Relaxing COVID-19 Lockdown Measures. PLoS ONE, 16(5):1-42.
Pradeep K. Murukannaiah, Nirav Ajmeri, Catholijn J. M. Jonker, and Munindar P. Singh. 2020. New Foundations of Ethical Multiagent Systems. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '20, pages 1706-1710, Auckland. IFAAMAS.
Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP '21, pages 5414-5426, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
James W. Pennebaker, Martha E. Francis, and Roger J. Booth. 2001. Linguistic Inquiry and Word Count (LIWC). Mahway: Lawrence Erlbaum Associates, 71.
Alina Pomeranz, Christian Detweiler, Pascal Wiggers, and Catholijn M. Jonker. 2012. Elicitation of Situated Values: Need for Tools to Help Stakeholders and Designers to Reflect and Communicate. Ethics and Information Technology, 14(4):285-303.
Vladimir Ponizovskiy, Murat Ardag, Lusine Grigoryan, Ryan Boyd, Henrik Dobewall, and Peter Holtz. 2020. Development and Validation of the Personal Values Dictionary: A Theory-Driven Tool for Investigating References to Basic Human Values in Text. European Journal of Personality, 34(5):885-902.
Xiaoye Qu, Zhikang Zou, Yu Cheng, Yang Yang, and Pan Zhou. 2019. Adversarial category alignment network for cross-domain sentiment classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL '19, pages 2496-2508, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Rezvaneh Rezapour, Saumil H. Shah, and Jana Diesner. 2019. Enhancing the Measurement of Social Effects by Capturing Morality. In Proceedings of the 10th
Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 35-45, Minneapolis, Minnesota, USA.
Subendhu Rongali, Beiye Liu, Liwei Cai, Konstantine Arkoudas, Chengwei Su, and Wael Hamza. 2021. Exploring Transfer Learning For End-to-End Spoken Language Understanding. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI '21, pages 13754–13761, Online.
Shamik Roy, Maria Leonor Pacheco, and Dan Goldwasser. 2021. Identifying Morality Frames in Political Tweets using Relational Learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP '21, pages 9939-9958. Association for Computational Linguistics.
Sebastian Ruder. 2019. Neural Transfer Learning for Natural Language Processing. Ph.D. thesis, NUI Galway.
Stuart J. Russell, Daniel Dewey, and Max Tegmark. 2015. Research Priorities for Robust and Beneficial Artificial Intelligence. AI Magazine, 36(4):105-114.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL '20, pages 5477-5490, Online. Association for Computational Linguistics.
Shalom H. Schwartz. 2012. An Overview of the Schwartz Theory of Basic Values. Online readings in Psychology and Culture, 2(1):1-20.
Marc Serramia, Maite López-Sánchez, Stefano Moretti, and Juan Antonio Rodríguez-Aguilar. 2021. On the dominant set selection problem and its application to value alignment. *Autonomous Agents and Multi-Agent Systems*, 35(2):1-38.
Yiting Shen, Steven R. Wilson, and Rada Mihalcea. 2019. Measuring Personal Values in Cross-Cultural User-Generated Content. In Proceedings of the 10th International Conference on Social Informatics, SocInfo '19, pages 143-156. Springer.
Luciano C. Siebert, Enrico Liscio, Pradeep K. Murukannaaiah, Lionel Kaptein, Shannon L. Spruit, Jeroen van den Hoven, and Catholijn M. Jonker. 2022. Estimating Value Preferences in a Hybrid Participatory System. In Proceedings of the first International Conference on Hybrid Human-Artificial Intelligence, HHAI '22, pages 1-14, Amsterdam, the Netherlands. IOS Press.
Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie. 2021. Embracing Domain Differences in Fake News: Cross-domain Fake News Detection using Multi-modal Data. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI '21, pages 557-565.
Nate Soares and Benya Fallenstein. 2017. Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda. In *The Technological Singularity: Managing the Journey*, pages 103–125. Springer, Berlin.
Brian Thompson, Jeremy G Winnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL '19, pages 2062-2068, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Terne Sasha Thorn Jakobsen, Maria Barrett, and Anders Søgaard. 2021. Spurious correlations in cross-topic argument mining. In Proceedings of the Tenth Joint Conference on Lexical and Computational Semantics, *SEM* 2021, pages 263-277, Online. Association for Computational Linguistics.
Anna Tigunova, Andrew Yates, Paramita Mirza, and Gerhard Weikum. 2019. Listening Between the Lines: Learning Personal Attributes from Conversations. In Proceedings of the 2019 World Wide Web Conference, WWW '19, pages 1818-1828.
Alexandra N Uma, Dirk Hovy, Barbara Plank, and Massimo Poesio. 2021. Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72:1385-1470.
Michiel van der Meer, Enrico Liscio, Catholijn M. Jonker, Aske Plaat, Piek Vossen, and Pradeep K. Murukannaiah. 2022. HyEnA: A Hybrid Method for Extracting Arguments from Opinions. In Proceedings of the first International Conference on Hybrid Human-Artificial Intelligence, HHAI '22, pages 1-15, Amsterdam, the Netherlands. IOS Press.
Francisco Vargas and Ryan Cotterell. 2020. Exploring the Linear Subspace Hypothesis in Gender Bias Mitigation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP '20, pages 2902-2913.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proceedings of the 31st Conference on Neural Information Processing Systems, NeurIPS '17, pages 5998-6008, Long Beach, CA, USA.
Steven R. Wilson, Yiting Shen, and Rada Mihalcea. 2018. Building and Validating Hierarchical Lexicons with a Case Study on Personal Values. In Proceedings of the 10th International Conference on Social Informatics, SocInfo '18, pages 455-470, St. Petersburg, Russia. Springer.
Hua Yuan, Jie Zheng, Qiongwei Ye, Yu Qian, and Yan Zhang. 2021. Improving Fake News Detection with Domain-Adversarial and Graph-Attention Neural Network. Decision Support Systems, 53:113633.
As we train deep learning models, reproducibility is an issue due to the inherent randomness of the training procedure. Nevertheless, we seek to provide all possible tools for reproducing our experimental results. To do so, we attach our code and the complete set of results. Furthermore, the following sections describe our data preprocessing, the hyperparameters, the computing infrastructure, and the random seeds used in our experiments.
We choose to use the datasets as they are, despite their imbalanced label distribution (Table 3), since such imbalance is representative of realistic applications. We preprocess the tweets by removing URLs, emails, usernames and mentions. Next, we employ the Ekphrasis package<sup>2</sup> to correct common spelling mistakes and unpack contractions. Finally, emojis are transformed into their respective words using the Python Emoji package<sup>3</sup>.
To select the hyperparameters, we trained and evaluated the model on the entire MFTC corpus with 10-fold cross-validation. Table A1 shows the hyperparameters that were compared in this setting, highlighting in bold the best performing option that we then used in the experiments described in the paper. If a parameter is not present in the table, the default value supplied by the framework was used.
Table A1: Hyperparameters tested and selected
Hyperparameters	Options
Model name	bert-base-uncased
Number of parameters	110M
Max sequence length	64
Epochs	2, 3, 5
Batch size	16, 32, 64
Dropout	0.05, 0.1, 0.02
Optimizer	AdamW
Learning Rate	5*10-5
Loss function	Binary Cross Entropy
A.3 Computing Infrastructure
The following are the main libraries and computing environment used in our experiments.
PyTorch: 1.8.1
$^{2}$ https://github.com/cbaziotis/ekphrasisis  
 $^{3}$ https://pypi.org/project/emoji/
- TensorFlow: 2.5.0
- FastText: 0.8.22
Hugginface's Transformers: 4.6.0
- NVIDIA GeForce RTX 2080 Ti GPU
CUDA: 11.2
cuDNN:8.1.1.33
Refer to the code base for a detailed list of the libraries we used, and their versions.
The following list details the amount of GPU hours spent for obtaining our results:
- Tables 4, B1, and B2: 44 hours
- Figures 2 and 3: 33 hours
- Tables B3, B4, and B5: 24 hours
Table B7: 26 hours
The error analysis (Tables 5, 6, and 7) did not require additional GPU time.
A.4 Random Seeds
In our experiments, we ensured that the same train-test splits are used across different runs of each experiment. Further, to control for randomness, we fixed the random seeds in the following libraries:
- Python (random.seed);
- NumPy (numpy.random.seed);
- PyTorch (torchmanual_seed);
- TensorFlow (tensorflow.random.seed_seed).
A.5 Artifacts Usage
We have mainly used two artifacts in this research: the MFTC and BERT.
The MFTC was collected with the intent of facilitating NLP research on moral values (Hoover et al., 2020). It can be downloaded and used under the Creative Commons Attribution 4.0 license.
BERT (Devlin et al., 2019) was created with the intent of performing, among others, text classification. Thus, we are using it as originally intended, under its Apache 2.0 distribution license<sup>5</sup>.
4https://osf.io/k5n7y/ 5https://github.com/google-research/ bert/blob/master/License
In this Appendix we extend the results presented in the paper. The following results are not crucial for supporting our conclusions. Nevertheless, they provide additional details on our experiments.
B.1 Model Comparison
We have presented the results of the transferability analysis with the BERT model. In order to evaluate whether our conclusions generalize to other model architectures, we repeat the experiment conducted in the paper (see Sections 3 and 4) with the following two additional models:
- Long Short Term Memory (LSTM), a category of Recurrent Neural Networks (RNN). We choose LSTM as a baseline model since it is commonly used in moral value classification (Lin et al., 2018; Mooijman et al., 2018; Rezapour et al., 2019; Hoover et al., 2020).
- fastText, a machine learning approach that learns character-level information, in contrast to the whole word representations LSTM employs. This flexibility makes fastText a good candidate for transfer learning. Further, we choose fastText as it attains performances on par with state-of-the-art deep learning methods, but is considerably faster.
Tables B1 and B2 present the results of the transferability analysis, performed and presented analogously to Table 4, for LSTM, fastText, and BERT. We observe that BERT outperforms fastText and LSTM in most settings. This is not surprising, since BERT is state-of-the-art for text classification. Both BERT and fastText outperform LSTM, the model extensively used for predicting moral values. Further, we notice that the general trends observed in Section 4.1 hold for all three models.
Generalizability All three models achieve better average  $F_{1}$ -scores in the  $\mathcal{C}(\text{source}, \text{target})$  setting than the majority (target) baseline. However, compared to the majority (source) baseline,  $\mathcal{C}(\text{target}, \text{source})$  performs worse with LSTM, comparably with fastText, and much better with BERT. This suggests that a contextualized representation, as in BERT, is necessary for value classification in novel domains, especially for the novel domains with a large moral vocabulary as is the case in  $\mathcal{C}(\text{target}, \text{source})$ .
Transferability From the average  $F_{1}$ -scores in Table B2, we observe that  $\mathcal{C}(\text{finetune}, \text{target})$  performs better than or on par with  $\mathcal{C}(\text{target}, \text{target})$  across all three models. The benefits of finetuning are most evident for LSTM (7% increase in the average  $m$  and 17% increase in  $M$ ). The benefits can also be observed for fastText (similar  $m$  and 8% increase of  $M$ ) and BERT (similar  $m$  and 8% increase of  $M$ ), but to a lesser degree than LSTM.
Catastrophic Forgetting We observe that all three models suffer from catastrophic forgetting since finetuning on  $\mathcal{T}_{\text {target }}$  reduces the performance on  $\mathcal{T}_{\text {source }}$ . As mentioned in the paper, the degree of catastrophic forgetting is most evident when finetuning on unbalanced datasets such as DAV than balanced datasets such as BLM.
B.1.1 Training Time
In some applications, e.g., estimating value trends on Twitter, value classifiers need to be re-trained frequently since the trends can shift fast. Similarly, to employ techniques such as active learning for value annotation requires training a classifier at every iteration to prompt for new labels. In such cases, training time is an important factor for selecting an approach and model. Figure B1 shows the average training time in logarithmic scale, for different models and scenarios (Appendix A.3 describes our computing infrastructure).
Figure B1: Average training time per model and scenario
Two considerations are evident. First, fastText trains significantly faster than the other two models. Second, for all three models, the training time is approximately proportional to the amount of data in the training set—the target and finetune scenarios employ a similar amount of data, which is roughly six times smaller than in the source and all scenarios.
Table B1: Results of the four training scenarios and three models evaluated on  $\mathcal{T}_{source}$ . The columns indicate the dataset used as  $\mathcal{T}_{target}$ . For each experiment we report micro  $F_{1}$ -score ( $m$ , left-hand column) and macro  $F_{1}$ -score ( $M$ , right-hand column).
Classifier Setting	ALM	BLT	BLM	DAV	ELE	MT	SND	Average
m	M	m	M	m	M	m	M	m	M	m	M	m	M	m	M
LSTM
Csource, source)	64.1	45.7	64.0	52.1	61.1	39.6	59.2	48.0	63.5	46.5	66.4	47.1	65.6	46.8	63.4	46.5
C(target, source)	47.8	19.3	41.0	6.1	53.5	25.6	38.8	5.1	51.1	20.2	39.1	11.9	35.1	16.1	43.8	14.9
C(finetune, source)	61.4	37.4	48.3	25.1	60.0	39.6	41.6	11.0	60.7	40.5	55.1	39.1	52.3	36.6	54.2	32.8
C(all, source)	64.5	46.7	63.2	49.2	62.3	41.4	59.3	47.7	64.2	48.6	66.4	48.7	65.8	48.1	63.7	47.2
fastText
Csource, source)	66.8	56.0	65.9	57.8	64.4	51.5	63.1	56.9	66.6	56.7	69.5	59.5	67.8	56.8	66.3	56.5
C(target, source)	54.5	30.9	42.7	8.5	56.4	33.1	38.7	5.1	52.2	30.0	48.9	22.0	41.3	20.3	47.8	21.4
C(finetune, source)	62.1	48.8	54.4	39.5	62.6	46.4	52.9	39.9	61.4	50.8	57.3	45.7	56.7	49.7	58.2	45.8
C(all, source)	66.9	56.3	66.0	57.5	64.8	52.1	63.1	56.7	66.9	57.0	68.7	58.2	67.5	56.4	66.3	56.3
BERT
Csource, source)	73.9	65.6	73.9	68.3	71.2	61.8	71.1	66.4	73.3	66.4	75.7	68.0	74.5	66.5	73.4	66.1
C(target, source)	61.6	37.7	43.8	13.1	62.6	43.0	38.8	5.1	59.3	40.4	52.4	39.1	54.4	36.6	53.3	30.7
C(finetune, source)	70.3	57.2	61.2	47.8	69.2	54.9	56.6	41.9	70.5	61.5	67.7	60.5	68.0	60.8	66.2	54.9
C(all, source)	73.7	65.6	73.7	68.0	71.3	62.1	71.0	66.4	73.6	66.7	75.6	67.7	74.3	66.6	73.3	66.2
Majority (source)	47.0	6.1	42.3	5.6	49.0	6.2	38.8	5.3	46.1	6.0	49.0	6.2	48.9	6.2	45.9	5.9