Semi-Supervised Training for Biomedical Question
Answering
Dimitra Panou1 , Martin Reczko1,∗
1

Institute for Fundamental Biomedical Science, Biomedical Sciences Research Center “Alexander Fleming”, 34 Fleming
Street, 16672 Vari, Greece

Abstract
The recently introduced semi-supervised method GANBERT for finetuning large language models [1] hasbeen applied for document relevance prediction in biomedical question answering. The additional use ofunlabeled texts during training enhances the robustness of the prediction and outperforms our previoustransformer ELECTROLBERT [2]. The initial document selection phase used both for ELECTROLBERTand GANBERT has been improved using BM25 combined with RM3 query expansion with optimizedparameters. Both systems were continuously improved during the BioASQ11 [3] competition and in
the last batch, GANBERT ranked as the 3𝑟𝑑 team for document prediction. The previous version ofELECTROLBERT took the 1𝑠𝑡 place for the “yes/no” type questions in this years SYNERGY [4] prediction.

Keywords
Biomedical Question Answering, Semi-supervised learning, BioASQ, GANBERT, large language models

1. Introduction
One major bottleneck in the development of robust question answering systems is the lackof large volumes of high quality question answer pairs provided by human experts. Thoughtransfer learning by finetuning pretrained large language models (LLMs) alleviates this problem[5], the limited data jeopardizes finetuning through overfitting. A recently suggested remedy [1]transfers the successful paradigm of semi-supervised learning used in Generative Adversarial
Networks (GAN) for image processing [6] to the finetuning of LLMs. GANBERT extends thefine-tuning of BERT with unlabeled data using GAN framework, where a 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟(𝐺) is trained
to produce samples of the internal BERT representation resembling the distribution over theunlabeled data, and a 𝑑𝑖𝑠𝑐𝑟𝑖𝑚𝑖𝑛𝑎𝑡𝑜𝑟(𝐷) that is trained to distinguish samples of the generator
from the real instances. By generating only the internal representation of text, GANBERT
avoids the generation of “fake” text instances. It is an effective semi-supervised method that canimprove the generalization capability. Using vast amounts of unlabeled texts during training,
the scope of the language model can be expanded to facilitate the use of alternative formulations
for the same semantic content. In the original GANBERT paper [1] tests were performed fornews topic classification, question conceptual class prediction, sentiment analysis and text genre
CLEF 2023: Conference and Labs of the Evaluation Forum, September 18–21, 2023, Thessaloniki, Greece
Corresponding author.
Envelope-Open panou@fleming.gr (D. Panou); reczko@fleming.gr (M. Reczko)
Orcid 0000-0002-0005-8718 (M. Reczko)∗© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEURWorkshop
Proceedings

http://ceur-ws.org
ISSN 1613-0073

CEUR Workshop Proceedings (CEUR-WS.org)

classification. Two GANBERT variants were later successfully used for predicting he checkworthiness of potential fake news in tweets [7]. In [8], the noise generation in GANBERT wasoptimized for the task of discriminating correct paraphrases of Spanish texts. In the following
we describe optimized document selection and the application of GANBERT for document
relevance prediction in biomedical question answering in the BioASQ11 competition [9]. Wealso provide details for the additional predictions with our ELECTROLBERT algorithm [2] inthe same competition.

2. BM25 and RM3 hyperparameter optimization
To identify documents relevant for a question, we replace the TF/IDF method with the widelyused BM25 [10]. BM25 has two parameters 𝑘1 and 𝑏. 𝑘1 is intuitively related to the rateof increase in a document’s score from matching an additional occurrence of a term, wheresmaller 𝑘1 provides a faster increase. The parameter 𝑏 controls the extent of document-lengthnormalisation. The search is combined with RM3 [11], a classic pseudo-relevance feedback
based query expansion model, to find related concepts. RM3 has three parameters, 𝑡𝑒𝑟𝑚𝑠 isthe number of query expansion terms, 𝑑𝑜𝑐𝑠 is the number of top-ranked documents to obtain
the expansion terms and 𝑞𝑤 defines the weight of the original query. The efficient Python
implementation in the package Pyserini is used [12]. A gridsearch on these parameters tooptimize the mean average precision (𝑀𝐴𝑃) of the top 10 returned documents for the BioASQ11
training set provided the values that were used in all four batches of BioASQ11. A random
search optimizing the average 𝑀𝐴𝑃 of the top 10 returned documents for the 240 questions inthe first three batches of BioASQ11 indicates potential improvements. The optimized parameters
shown in table 1 clearly outperform the default settings.

3. Training, validation and test data
For finetuning GANBERT, all pairs of a question and its correct documents provided in thetraining set for BioASQ11 are used for the ’relevant’ class. As introduced in the ELECTROLBERT
training [2], the negative examples for the ’non-relevant’ class are generated using a range offalse positives from the initial document selection phase to better discriminate the relevant
documents obtained. All questions of the relevance training set were processed with BM25 and
RM3 using the settings marked with B3+4:EB0-4 in table 1 to select 1000 relevant documents foreach question. The documents were ranked according to their score and all documents between
rank 100 and 150 were used as negative examples, excluding potential positive examples inthese ranks. The values of the start and end rank positions for the negative set were optimized
by retraining and maximizing the mean average precision measured on all batches of BioASQ10.
For the unlabeled set, all pairs of a question and its ideal answer and all related snippets fromthe BioASQ10 training set were used. As a validation set, the top 100 documents scored withBM25 and RM3 (settings again as in B3+4:EB0-4) of the 240 questions in the first three batchesof BioASQ11 was used. A final independent test was made on the 90 questions of batch 4 ofBioASQ11.

Table 1BM25 & RM3 parameter optimization. 𝑘1 & 𝑏 are parameters of BM25 and the variables 𝑡𝑒𝑟𝑚𝑠 (Expansionterms), 𝑑𝑜𝑐𝑠 (number of top-ranked documents) and 𝑞𝑤 (Original query weight) are parameters of RM3model. 𝑛𝑎𝑠𝑛𝑜𝑘 specifies the number of questions (total 240) with at least one correct document. 𝑛𝑑𝑜𝑐𝑜𝑘
specifies the number of correctly identified documents (max. 647). In the column “used for”, Bx denotes
the BioASQ11 test batch x, and EBy denotes the system ELECTOLBERTy.
𝑘1𝑏𝑡𝑒𝑟𝑚𝑠

𝑑𝑜𝑐𝑠𝑞𝑤𝑀𝐴𝑃𝑟123

𝑛𝑎𝑛𝑠𝑜𝑘𝑛𝑑𝑜𝑐𝑜𝑘used for1.2
0.4
1.1
0.4
0.30
0.40
0.40
0.30
0.45
0.40
0.60
0.50
0.45
0.45
0.40
0.40
0.55
0.60
0.35
0.35

0.75
0.3
0.0
0.3
0.31
0.31
0.31
0.31
0.36
0.31
0.37
0.33
0.37
0.31
0.38
0.30
0.34
0.34
0.34
0.37

10
10
10
17
16
20
20
20
20
20
17
20
15
17
15
18
19
14
18
17

10
10
10
14
16
16
16
16
21
16
16
25
22
20
24
21
23
18
25
26

0.5
0.5
0.5
0.6
0.8
0.7
0.9
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.8
0.7
0.7
0.8
0.7
0.70.2734
0.26250.27520.29060.29320.2936
0.29400.29520.29800.29810.29830.29870.29920.29930.29990.30000.30020.30030.30040.3011

128
138
125
134
135
138
129
134
134
134
130
135
137
135
131
135
138
131
138
138

192
201
195
198
202
205
199
200
203
202
200
206
205
201
200
202
206
202
209
210

defaults
B1:EB2+3
B1:EB0+1
B2:EB0,2+3
B3+4:EB0-4

4. GANBERT finetuning and hyperparameter optimization
The adaptation of the GANBERT architecture introduced in [1] for document relevance classification is shown in figure 1. Using the labeled and unlabeled data described in the previous
section for finetuning and employing the large pretrained BERT model provided with theGANBERT implementation in the path for the real data (provided by the authors of GANBERTat https://github.com/crux82/ganbert), all relevant hyperparameters for GANBERT are optimized by multiple finetunings while monitoring the performance on the first three batchesof BioASQ11 as shown in table 2. All GANBERT models perform substantially better whencompared to the standard BERT model and the performance of GANBERT is quite stable for thedifferent hyperparameter settings, also for variations as suggested in [8] in the noise generation
part.

5. Results
In table 3 the performances of our document relevance submissions for the BioASQ11 competition are listed. All submissions marked with ’base model’ use the ELECTROLBERT model ofnoiseGF
relevant

real data

DRQ

non-relevant
is real?

RBERTUQNR

NR

Figure 1: The GANBERT architecture for question answering: The 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑜𝑟 G generates a set of
fake representations F given a random distribution. These and the unlabeled U and labeled L vector
representations computed by BERT are used as input for the 𝑑𝑖𝑠𝑐𝑟𝑖𝑚𝑖𝑛𝑎𝑡𝑜𝑟 D. The labeled examples areclassified into documents relevant (R) and non-relevant (NR) for a question Q. The real data should bediscriminated from the fake representations via the ’is real?’ output.
Table 2Hyperparameter optimization for question answering GANBERT models using the 𝑀𝐴𝑃 averaged
over the first three batches of BioASQ11 (𝑀𝐴𝑃𝑏123 ). The final test uses the 𝑀𝐴𝑃 of BioASQ11 batch4(𝑀𝐴𝑃𝑏4 ). Unless specified, the sequence length for prediction is 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175. All models arefinetuned for 32000 steps. 𝐿𝑅 denotes the learning rate, 𝑆𝐿𝐸𝑁 the sequence length during finetuningand 𝐿𝐴𝐵𝐸𝐿_𝑀𝐴𝑆𝐾 controls the ratio between the number of labeled and unlabeled examples. The boldmodel is GANBERT3, submitted as ELECTOLBERT-4 in batch4.
ModelLRSLEN

LABEL_MASK

Unlabeled set

Noise generation

BERT

2𝑒 − 6200---

GANBERT

2𝑒 − 62000.02

BioASQ10

uniform[0,1]” , 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 200”””””” , 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 150”””””””

175”””””225

”

”

”

”

”

200

0.01

”

”

”

”

”

0.05

”

”

”

1𝑒 − 6”

0.02

”

”

”

5𝑒 − 6”

”

”

”

”

2𝑒 − 6””

BioASQ10+BoolQ [13]

”

”

”

”

”

BioASQ10

uniform[-1,1]

”

”

”

”

”

normal[0,1]𝑀𝐴𝑃𝑏123𝑀𝐴𝑃𝑏4

0.3166
0.34680.34530.3456
0.34590.33630.33950.33800.32660.33340.33150.33330.32890.22310.23000.22230.22620.21660.21520.23010.21810.20660.22000.21660.22140.2220batch 4 in the BioASQ10 competition described in [2]. The models marked with ’large model’
use the large architecture in [2], where pretraining was continued for 30 million steps andfinetuning was performed with the labeled part of the training set for GANBERT for 180000
steps. It can be observed that the sequence length for predictions converged to an optimalvalue of 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175 during the competition. With the optimized first phase document
selection, it also became evident that the transformers in the second phase focus on the final
ranking of the results and the number of documents was gradually reduced from 𝑛𝑑𝑜𝑐𝑠 = 11500
to 𝑛𝑑𝑜𝑐𝑠 = 10. In batch 4, the names of the systems ELECTROLBERT-[1,2,3,4] are used for thedifferent GANBERT submissions.
Table 3
BioASQ11 document relevance prediction performance measured as mean average precision (𝑀𝐴𝑃). The
column ’model details’ specifies the type of the transformer architecture, the sequence length during
prediction and the number of documents to be ranked. The model GANBERT4 was trained for twice thenumber of steps as GANBERT3.
batch𝑀𝐴𝑃systemper team rank

model details

10.45900.38750.3732bioinfo-0
ELECTROLBERT-2,3
ELECTROLBERT-0,1

1
44

base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 200, 𝑛𝑑𝑜𝑐𝑠 = 11500

0.38520.32520.29420.27810.2513bioinfo-4
ELECTROLBERT-2
ELECTROLBERT-0
ELECTROLBERT-3
ELECTROLBERT-1

14
4
4
42

base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 250, 𝑛𝑑𝑜𝑐𝑠 = 11500

base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 6750
base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 250, 𝑛𝑑𝑜𝑐𝑠 = 6750
base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 275, 𝑛𝑑𝑜𝑐𝑠 = 6750
Query expansion using Roccio’s methond [14]base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 400, 𝑛𝑑𝑜𝑐𝑠 = 300

340.3185
0.25020.23360.23260.22960.2261dmiip2
ELECTROLBERT-0
ELECTROLBERT-4
ELECTROLBERT-2
ELECTROLBERT-1
ELECTROLBERT-3

1444440.3224
0.22790.22710.22420.21470.1849dmiip3
ELECTROLBERT-1
ELECTROLBERT-4
ELECTROLBERT-3
ELECTROLBERT-2
ELECTROLBERT-0

133333base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 60
base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 16
large model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 200, 𝑛𝑑𝑜𝑐𝑠 = 13
base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 150, 𝑛𝑑𝑜𝑐𝑠 = 300
large model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 150, 𝑛𝑑𝑜𝑐𝑠 = 11

GANBERT4, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 10
GANBERT3, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 10
GANBERT2, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 11
GANBERT1, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 16base model, 𝑆𝐿𝐸𝑁𝑝𝑟𝑒𝑑𝑖𝑐𝑡 = 175, 𝑛𝑑𝑜𝑐𝑠 = 60

6. Conclusion and Future Work
Our suggested GANBERT version for document relevance prediction has shown promising
performance, defeating our previous algorithm ELECTROLBERT. As can be seen at the published BioASQ11 results, both algorithms perform better than some of the other systems thatseem to employ ChatGPT [15]. One obvious extension would be the replacement of BERT in
the path processing the real data with ELECTROLBERT. This would also lead to the use ofa more appropriate scientific vocabulary, as the BERT model provided with the GANBERT
implementation uses a general purpose vocabulary. It should also be noted that the size ofthe unlabeled data set in this study is relatively small due to generation of this using only
text available with the BioASQ datasets and our limited computational resources. One way toincrease this could be the use of random segments from Pubmed abstracts.

Acknowledgments
GPU computations were offered by HYPATIA, the Cloud infrastructure of the Greek ELIXIR
node.

References[1] D. Croce, G. Castellucci, R. Basili, GAN-BERT: Generative Adversarial Learning for Robust
Text Classification with a Bunch of Labeled Examples, in: Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics, Association for Computational
Linguistics, Online, 2020, pp. 2114–2119. URL: https://aclanthology.org/2020.acl-main.191.
doi:10.18653/v1/2020.acl- main.191 .
[2] M. Reczko, ELECTROLBERT: Combining Replaced Token Detection and Sentence Order Prediction, in: Proc. of CLEF 2022: Conference and Labs of the Evaluation Forum,
September 5–8, 2022, Bologna, Italy, online http://ceur-ws.org/Vol-3180/paper-24.pdf,
urn:nbn:de:0074-3180-7, 2022.
[3] A. Nentidis, G. Katsimpras, A. Krithara, S. Lima-López, E. Farré-Maduell, L. Gasco,
M. Krallinger, G. Paliouras, Overview of BioASQ 2023: The eleventh BioASQ challenge on
Large-Scale Biomedical Semantic Indexing and Question Answering, in: Experimental
IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the FourteenthInternational Conference of the CLEF Association (CLEF 2023), 2023.
[4] A. Nentidis, G. Katsimpras, A. Krithara, G. Paliouras, Overview of BioASQ Tasks 11b andSynergy11 in CLEF2023, in: Working Notes of CLEF 2023 - Conference and Labs of theEvaluation Forum, 2023.
[5] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding, 2018. URL: https://arxiv.org/abs/1810.04805.
doi:10.48550/ARXIV.1810.04805 .
[6] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville,
Y. Bengio, Generative Adversarial Nets, in: Proceedings of the 27th International Confer-ence on Neural Information Processing Systems - Volume 2, NIPS’14, MIT Press, Cambridge,MA, USA, 2014, p. 2672–2680.
[7] R. A. Frick, I. Vogel, I. N. Grieser, Fraunhofer SIT at CheckThat! 2022: Semi-Supervised
Ensemble Classification for Detecting Check-Worthy Tweets, Working Notes of CLEF
(2022).
[8] T. Ta, A. B. S. Rahman, L. Najjar, A. Gelbukh, GAN-BERT, an Adversarial Learning
Architecture for Paraphrase Identification, in: Proc. of IberLEF 2022, September 2022, ACoruña, Spain, 2022.
[9] A. Krithara, A. Nentidis, K. Bougiatiotis, G. Paliouras, BioASQ-QA: A manually curated
corpus for Biomedical Question Answering, Scientific Data 10 (2023) 170.
[10] S. E. Robertson, S. Walker, M. Hancock-Beaulieu, Experimentation as a way of life: Okapi
at TREC, Inf. Process. Manag. 36 (2000) 95–108.
[11] V. Lavrenko, W. B. Croft, Relevance Based Language Models, in: Proceedings of the 24th
Annual International ACM SIGIR Conference on Research and Development in Information
Retrieval, SIGIR ’01, Association for Computing Machinery, New York, NY, USA, 2001, p.
120–127. URL: https://doi.org/10.1145/383952.383972. doi:10.1145/383952.383972 .
[12] J. Lin, X. Ma, S. Lin, J. Yang, R. Pradeep, R. F. Nogueira, Pyserini: An Easy-to-Use Python
Toolkit to Support Replicable IR Research with Sparse and Dense Representations, CoRR
abs/2102.10073 (2021). URL: https://arxiv.org/abs/2102.10073. arXiv:2102.10073 .
[13] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, K. Toutanova, BoolQ: Exploring
the Surprising Difficulty of Natural Yes/No Questions, in: NAACL, 2019.
[14] V. Lavrenko, W. B. Croft, Relevance-based language models, in: ACM SIGIR Forum,
volume 51, ACM New York, NY, USA, 2017, pp. 260–267.
[15] OpenAI, ChatGPT [Large language model] https://chat.openai.com/chat, 2023.