Authorship Verification Based on SimCSE
Notebook for PAN at CLEF 2023

Yong Qiu, Haoliang Qi*, Yong Han, Kaicheng Huang
Foshan University, Foshan, Guangdong, ChinaAbstract
In the case of the Authorship verification task, where two texts belonging to different Discourse
Types (DT) are given, the objective is to determine if they are written by the same author(cross-DT authorship verification). In this paper, we use a framework based on contrastive
learning and a pre-trained language model to extract text features to solve the (authorshipverification) task. Compared with traditional machine learning methods, the method based oncontractive learning (SimCSE) can encode and get the comparison between the texts, enabling
better handling of semantic matching. In the authorship verification task, the model can capture
the author's writing style and characteristics more accurately by comparing the similarity
between multiple texts of the same author, thus improving the model’s classification
performance. The experiment demonstrated competitive performance, achieving an accuracyof 90.76% on our test dataset, which was manually created from a PAN-provided dataset.
Keywords 1
Authorship Verification, Contractive Learning, Pre-trained model, Classification

1. Introduction
The goal of the PAN@CLEF 2023 Authorship verification [1][2] task is to determine whether thetwo texts are written by the same author by comparing their writing styles. In earlier versions of PAN[3], the effectiveness of authorship verification techniques in multiple languages and text types wasinvestigated. In recent years, cross-domain authorship verification tasks (cross-DT authorshipverification) [3] have also been successfully implemented. The authorship verification task at PAN
2023, for the first time, focuses on authorship verification across spoken discourse genres. The aim isto investigate the effectiveness and robustness of style measurement methods under more challenging
and intriguing circumstances. This task provides a cross-DT author verification case using a new
English corpus of a diverse sample of approximately 100 native English speakers of similar ages (1822). The subject of the text is not restricted. Still, the forms and levels of formality vary, including
essays (written discourse), e-mails (written discourse), interviews (spoken discourse), and speechtranscriptions (spoken discourse).
In recent years, deep learning-based methods have been widely adopted due to their superiority inprocessing large-scale datasets. Neural networks have had many practices for judging text author
attribution [4]. Contractive learning [5] has gained increasing attention in deep learning, such as
SimCSE [6], which can use labeled and unlabeled data to train deep neural networks to alleviate thechallenges caused by data scarcity or inaccurate labels. In this authorship verification task, the lack ofsufficient labeled datasets is a problem that needs to be solved, and comparative learning can improve
the authentication performance of authorship verification by learning the similarity between tasks, tosolve these problems better. The sentence vector representation technique has always been a hot topic
in NLP. In the BERT [7] era, people generally used the [CLS] vector of the BERT model to represent
1

CLEF 2023 – Conference and Labs of the Evaluation Forum, September 18–21, 2023, Thessaloniki, Greece
EMAIL: Timmy_ong@163.com (A. 1); qihaoliang @fosu.edu.cn (A. 2) (*corresponding author); hanyong2005@fosu.edu.cn (A. 3);
teaslate@outlook.com (A. 4)
ORCID: 0000-0003-4620-4992 (A. 1); 0000-0003-1321-5820 (A. 2); 0000-0002-9416-2398 (A. 3); 0000-0002-3473-3355 (A. 4)©️ 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

CEUR Workshop Proceedings (CEUR-WS.org) Proceedings

the sentence vector by the inherent advantages of the pre-trained language model. In this paper, weapply BERT to obtain text representation, adopt contractive learning, and capture the author's writing
style and characteristics by comparing the similarity between multiple texts of the same author. By
experiment, we realized authorship verification.

2. Dataset
PAN@CLEF 2023 Authorship verification task belongs to the open set author verification. That is,
the training data set and the test data set have the same structure and similar properties. However, there
is no overlap in the author sets of training and test datasets. For the training set, there are 8836 text pairs
from the Aston 100 Idiolects Corpus in English covering DTs of both written and spoken language:
essays, emails, interviews, and speech transcriptions. Labels are given correspondingly, whose meaning
is to judge whether the two texts are written by the same author (label 1 means yes, and vice versa is0). To protect the author's privacy, information specific to both author and topic, such as named entities,
have been substituted with corresponding tags. Moreover, for spoken discourse genres, supplementarytags are utilized to indicate nonverbal vocalizations, such as coughing and laughing.
Dataset distribution types and quantities are shown in Table 1.
Table 1The distribution types and quantities of the PAN@CLEF 2023 Authorship verification task dataset
Type
Quantity
Essays93
450
Emails275
Interviews
Speech transcriptions

68

3. Method
This section will introduce our data processing approach and network framework.

3.1 Data Preprocessing
For the training set given by the task, we first separated the texts according to each author. We
randomly divided the texts according to 3:1 for the training set and the test set, including 42 and 14
different authors, respectively. For the training set, we make a text triplet (" pair ": [" text1 ", "text_pos,""text_neg"]), where " text_pos " indicates a positive sample, and " text_neg " indicates a negative sample.
The positive samples are from the texts written by the author of the first text of the triplet, while thenegative ones are from other authors.
The training set made of random combinations contains 22,090 pieces of data.

3.2 Network Architecture
In this paper, the novel comparative learning framework SimCSE(Simple Contrastive Sentence
Embedding Framework) is adopted, and the pre-training language model BERT is used to extract textfeatures. And the model parameters are constantly updated with training. Precisely, as for the BERT,
the stacking Transformer encoder can capture deeply bidirectional information between words in asentence. Take the hidden of the 0th position of hidden state in the last layer of output (that is, thehidden CLS, the vector of [CLS] tokens in the output layer) to represent the vector of the entire sentence.
Directly using supervised training set for comparative learning training, the positive sentence pair canbe regarded as a natural positive sample and regard other embedding in the same batch as negative
samples. So, we can think of authorship verification as a binary classification problem [8]. Note thatthe positive example of texti for each piece of data in the same batch is the only one, while all otherexamples in the batch are considered negative examples. However, if the batch contains two or more
texts from the same author (for example, assuming Text1 and Text2 in Figure 1 were written by the sameauthor), this may cause model confusion and performance degradation. Therefore, we scrambled thedataset to avoid the occurrence of the same authors in the same batch as much as possible, and we useda lower batch size further alleviates this issue.
The core of the SimCSE model is contractive learning, which aims to better learn the representationof the data by narrowing the distance between similar data and drawing the distance between dissimilar
data. This makes it more effective in text-matching tasks. For each training data(𝑥𝑖 , 𝑥𝑖+ , 𝑥𝑖− ), where 𝑥𝑖+
is the implied text and 𝑥𝑖− is the contradictory text. The training objective of supervised learning
SimCSE is:+𝑒 sim⁡(ℎ𝑖,ℎ𝑖 )/𝜏

𝑙𝑖 = −log⁡

𝑁(𝑒∑sim⁡(ℎ𝑖,ℎ+𝑗 )/𝜏+𝑒sim⁡(ℎ𝑖,ℎ−𝑗 )/𝜏

(1))

𝑗=1

Where ℎ𝑖 and ℎ𝑖+ is the sentence vector representation of 𝑥𝑖 and 𝑥𝑖+ , 𝑁 is the size of batch in thetraining process, sim⁡(ℎ𝑖 , ℎ𝑗+ ) is the cosine similarity of vectors ℎ𝑖 and ℎ𝑖+ , and 𝜏 is the temperature
hyperparameter.
The network architecture of the model is shown in Figure 1.

Figure 1: Network Architecture

4. Experiments and Results
4.1 Experimental Setting

In this paper, we choose BERT-base-uncased as an encoder with 12-layer, 768-hidden, 12-heads,
and 110M parameters. The vocab size is 28,996. The data set was expanded to 22,090 pieces of data
after partitioning. The maximum length of the encoder is set to 512. We used Adam optimizer with thelearning rate set to 2e-5. Our experiment was conducted on the A100 server. The best performance isachieved through 5 epoch models.

4.2 Evaluation
To evaluate the performance of our model, we used the evaluation platform provided by PAN,
which includes the following metrics:•AUC: the conventional area under the curve score.
•c@1: rewards systems that leave complicated problems unanswered [9].
•F_0.5u: focus on deciding same-author cases correctly [10].
•F1-score: harmonic way of combining the precision, and recall of the model [11].
•Brier: Brier Score evaluates the accuracy of probabilistic predictions [12].

4.3 Results
Table 2 presents the model performance. The first line shows the performance results of BERT only(without SimCSE), while the second line shows the results of additional SimCSE. By comparison, ourmethod (BERTBASE+SimCSE) performs better, which proves the effectiveness of SimCSE.
Table 2Results on the test set
ModelAUC

c@1

f_05_uF1

BrierOverall

BERTBASE
BERTBASE+SimCSE

0.846
0.908

0.846
0.908

0.853
0.871

0.844
0.915

0.846
0.908

0.847
0.902

Table 3 demonstrates the performance of our model evaluated on the TIRA [13] environment forPAN@CLEF 2023.
Table 3Results on pan23-authorship-verification-test
ModelAUC
c@1
pan23-cdav-baseline
BERTBASE+SimCSE

0.601
0.540

0.569
0.540

f_05_uF1BrierOverall

0.543
0.499

0.466
0.421

0.595
0.540

0.555
0.508

5. Conclusion
In this work, we adopted a contrastive learning framework and applied the BERT pre-trained
language model to extract text features to solve the PAN 2023 Authorship verification task. The
experimental results show that satisfactory results can be achieved by applying contrastive learning
to the field of natural language processing, such as authorship verification task. It also demonstrates thepowerful ability of the BERT model in text vector representation.
In future work, we will continue to improve our methods and strive to achieve better results in open
domain authorship verification.

6. Acknowledgements
This work is supported by the National Natural Science Foundation of China (No.62276064).

7. References[1] Stamatatos, E., Kredens, K., Pęzik, P., Heini, A., Bevendorff, J., Potthast, M., & Stein, B.
(2023). Overview of the Authorship Verification Task at PAN 2023. In CLEF 2023 Labs andWorkshops, Notebook Papers. Conference and Labs of the Evaluation Forum (CLEF 2022).
CEUR-WS.org.
[2] Bevendorff, J., Borrego-Obrador, I., Chinea-Ríos, M., Franco-Salvador, M., Fröbe, M., Heini,A., Kredens, K., Mayerl, M., Pęzik, P., Potthast, M., Rangel, F., Rosso, P., Stamatatos, E.,
Stein, B., Wiegmann, M., Wolska, M., & Zangerle, E. (2023). Overview of PAN 2023:
Authorship Verification, Multi-Author Writing Style Analysis, Profiling Cryptocurrency
Influencers, and Trigger Detection. In A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis,A. Giachanou, D. Li, M. Aliannejadi, M. Vlachos, G. Faggioli, & N. Ferro (Eds.), Experimental
IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the FourteenthInternational Conference of the CLEF Association (CLEF 2023). Thessalonikki, Greece:Springer.
[3] Mike Kestemont, Efstathios Stamatatos, Enrique Manjavacas, Janek Bevendorff, Martin
Potthast, and Benno Stein, Overview of the Cross-Domain Authorship Verification Task at
PAN 2021. Working notes of CLEF 2021 - Conference and Labs of the Evaluation Forum.
[4] Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship Attribution for Neural
Text Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 8384–8395, Online. Association for Computational
Linguistics.
[5] Ye, M., Zhang, X., Yuen, P. C., & Chang, S. F. (2019). Unsupervised embedding learning via
invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (pp. 6210-6219).
[6] Gao, T., Yao, X., & Chen, D. (2021). Simcse: Simple contrastive learning of sentenceembeddings. arXiv preprint arXiv:2104.08821.
[7] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
[8] M. Koppel, J. Schler, Authorship verification as a one-class classification problem, in: C. E.
Brodley (Ed.), Machine Learning, Proceedings of the Twenty-first International Conference(ICML 2004), Banff, Alberta, Canada, July 4-8, 2004, volume 69 of ACM InternationalConference.
[9] A. Peñas, A. Rodrigo, A simple measure to assess non-response (2011).
[10] J. Bevendorff, B. Stein, M. Hagen, M. Potthast, Generalizing unmasking for short texts, in:Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and ShortPapers), 2019, pp. 654–659.
[11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P.
Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, the Journalof machine Learning research 12 (2011) 2825–2830.
[12] G. W. Brier, et al., Verification of forecasts expressed in terms of probability, Monthly weather
review 78 (1950) 1–3.
[13] Fröbe, M., Wiegmann, M., Kolyada, N., Grahm, B., Elstner, T., Loebe, F., Hagen, M., Stein,B., & Potthast, M. (2023). Continuous Integration for Reproducible Shared Tasks with TIRA.io.
In J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz,& A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IRResearch (ECIR 2023) (pp. 236-241). Berlin Heidelberg New York: Springer.