EARA: Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation
Authors: Ying Xiong
Xin Yang
Linjing Liu
Ka-Chun Wong
Qingcai Chen
Yang Xiang
Buzhou Tang
Venue: EMNLP 2023 Findings (2023)
Keywords: Biomedical semantic textual similarity
entity-aligned regularization
retrival augmentation

Abstract: Measuring Semantic Textual Similarity (STS) is a fundamental task in biomedical text processing, which aims at quantifying the similarity between two input biomedical sentences. Unfortunately, the STS datasets in the biomedical domain are relatively smaller but more complex in semantics than common domain, often leading to overfitting issues and insufficient text representation even based on Pre-trained Language Models (PLMs) due to too many biomedical entities. In this paper, we propose EARA, an entity-aligned, attention-based and retrieval-augmented PLMs. Our proposed EARA first aligns the same type of fine-grained entity information in each sentence pair with an entity alignment matrix. Then, EARA regularizes the attention mechanism with an entity alignment matrix with an auxiliary loss. Finally, we add a retrieval module that retrieves similar instances to expand the scope of entity pairs and improve the model's generalization. The comprehensive experiments reflect that EARA can achieve state-of-the-art performance on both in-domain and out-of-domain datasets. Source code is available \footnote{https://github.com/xy-always/EARA}.

EARA: Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation
Ying Xiong'. Xin Yang'. Linjing Liu², Ka-Chun Wong²
Qingcai Chen 13 Yang Xiang,, Buzhou Tangl 13x
Harhin Institute Technology, Shenzhen
City University of Hong Kong
Peng Cheng Laboratory
Abstract
BlunBERT
Measuring Semantic Textual Similarity (STS) is a fundamental task in biomedical text pro- cessing, which aims at quantifying the simi- larity between two input biomedical sentences. Unfortunately, the STS datasets in the biomedi- cal domain are relatively smaller but more com- plex in semantics than common domain, often leading to overfitting issues and insufficient text representation even based on Pre-traines Lan- guage Models (PLMs) due to too many biomed- ical entities. In this paper, we propose EARA, -aligdd,, attention-baa based and retrieval- augmented PLMs. Our proposed EARA first aligns same type fine-grained entity in- formation in each sentence pair with entity alignment matrix. Then, EARA regularizes the attention mechanism with an entity align- ment matrix with an auxiliary loss. Finally, add a retrieval module that retrieves similar in- stances to expand the scope of entity pairs and improve the model's generalization. The com- prehensive experiments reflect that EARA can
Serlenge
0.4
0.5
0.5
0.4
Probbblls
0.2
4.2
0.1
0.1
0
2
X5
Entitles
(a)
atenolal
[CLS]
ibuprote
200
Tou Nou
table
neede
tables table
mouth
ono grie
[SEP]
atona
(b)
Figure 1: (a) prediction errors of baseline model on N2C2STS datasets and the corresponding entity num- bers in sentences. Blue bars represent the propor- tion of sentences containing the number of entities k (k=0,1,2,3,44255) in the sentence; (b) Visualization the baseline model's attention the '[CLS]' token and 'atenolol" token of a piece of data in N2C2STS dataset.
achieve state-of- ofte--of-hh-art the-art performance both in-domain and out-of-domain datasets. Source code available
as EBMSASS (Hassanzadeh et al., 2019) and N2C2STS (Wang et al., 2020b), a growing num- ber scholars have paid attention those STS tasks (Xiong et al., 2020a,b; Chen et al., 2020) and achieved great performance by applying the Pre- trained Language Models (PLMs) (Devlin et al., 2019; Peng et al., 2019; Lee al., 2020) to those STS tasks.
1 Introduction
Biomedical Semantic Textual Similarity (STS) in- volves measuring the similarity between biomedi- cal texts, such as scientific articles, clinical notes, and Electronic Health Records (EHRs). This mea- surement can be utilized to identify the relevant information, reduce excessive redundant informa- tion in EHRs (Zhang et al., 201 O' D'Donnell et al., 2009; Cohen et al., 2013), and improve the accu- racy as well as the efficiency of Evidence-Baar Based Medicine (EBM) practices (Hassanzadeh et al., 2019).
Unfortunately, naively applying PLMs biomedical datasets may not yield promising re- sults, owing to the greater semantic complexity of such datasets than those in the general domain. A sentence is deemed semantically intricate when it contains fine-grained information of multiple types, such entities, which may distract the PLMs. For instance, in Figure (a), we observe a decline in the representational capacity of BlueBERT (as evi- denced by an increase the prediction error ratio) as the number of entities in sentences increases on the N2C2STS datasets. Upon visualizing the at-
As more biomedical STS datasets with de- tailed annotations have been published, such
Corresponding author. "hanss//gthhh..mmmyyaaawwy/////

---

tention matrix of the '[CLS]' token and 'atenolol" token of a specific instance in the N2C2STS with baseline BlueBERT, we discover that the model tends to focus on punctuation and the sentence con- taining the selected token.
2) PLM-enhanced Models Beyond Biomedical STS and 3) Retrieval-basee Methods.
Biomedical Semantic Textual Similarity A grow- ing number of works (Chen et al., 2021; Yang et al., 2020) based on PLMs (Devlin et al., 2019; Peng et al., 2019; Lee et al., 2020) have been exploited for biomedical STS datasets, such N2C2STS dataset (Wang al., 2020b), For example, Xiong et al. evaluated the pure Siamese CNN (Yin et al., 2016), Siamese RNN (Mueller and Thyagarajan, 2016), and BERT (Devlin et al., 2019) on these biomedical STS datasets, and improved the per- formance using a gate to fuse the one-hot features representation and deep semantic representation.
Besides, during the training process, we also observed that the Pearson Correlation Coefficient (PCC) of models on the training set could often approach 1.0. However, the PCC of models on the validation set increased, the PCC of models on the test set decreased. The reason might lie in that the size of the biomedical dataset is quite small domain dataset.
compared to the size of the And the complexity of PLMs is also too excessive for the biomedical dataset, which can easily result in overfitting.
There are lines of recent work enhance deep learning models on the biomedical STS task. The first used data augmentation strategies (Wang et al., 2020c; Li et al., 202 or multi-task Iearn- ing (Mulyar et al., 2021; Mahajan et al., 2020) to enhance the model's representation. The sec- ond introduced external knowledge into the neural network models, which can capture implicit infor- mation (Xiong et al., 2020a; Chang et al., 2021). These methods only integrate traditional features and lack interpretations.
In this paper, we propose EARA, which im- proves the biomedical semantic textual similar- ity with allined Attention and Retrieval Augmentation. EARA first makes PLMs pay more attention to important entities by aligning the same- type entities in sentence pairs. Secondly, EARA builds knowledge retriever on the MIMIC-III database (Johnson et al., 2016). Thirdly, EARA retrieves similar sentences expand the scope of entity pairs and feeds them into the entity-aware transformer. Finally, EARA fuses the logits of retrievals and inputs to improve the model's gener- alization. The experiments show that EARA can achieve otate-of-the-ar art performance on multiple biomedical tasks.
M-enhanceed Models Beyond Biomedical STS The PLMs are popularised on other fundamental tasks in NLP and augmented by various knowl- edge (Bugliarello and Okazaki, 2020; Zhou et al., 2020; Zhang et al., 2020a). Jia al. (Jia et al., 2020) proposed the Char- Shaa--Eatty-Tranffomma which injected lexical information into the char- level BERT and augmented the self-atteniion using a concatenation of char- and entity-level informa- tion. Liu et al. (Liu et al., 2021) proposed more efficient model than Char-Entity- Trantty-Tranorrmmr by adding lexicon adapter layer to inject the lexical information into the transformer layer of BERT. Their model achieved the SOTA performance on the named entity recognition, word segmentation, and POS tasks. Stacey et al. (Stacey et al., 2022) supervised the self-attention of BERT using hu- man explanation to the natural language inference tasks. Besides, some works (Zhang et al., 2020b; Li et al., 2021b) used syntax information to train the self-attention module in PLMs and showed promis- ing results on various tasks, like text classification tasks and named entity recognition tasks. Our work leveraged the entity information to regularize the multi-head attention module instead of directly in- troducing the information.
In summary, our work has the following contri- butions:
We make the model pay more attention to the same type of entity information by minimiz- ing the L2 loss between the entity alignment matrix and the multi-head attention matrix.
We add a retriever module and feed retrievals into transformers, which improves the model's generalization and eases the overfitting prob- lem for the biomedical STS datasets.
We make a comprehensive study on multi- ple biomedical STS datasets. Results show that our model can achieve promising perfor- mance both in-domain and out-of-domain datasets.
2
Related Work
In this section, we mainly introduce the related work from three perspectives: 1) Biomedical STS,

---

3.2 Multi-Type Fine-Grained Entity Information
Retriever
We use cTAKESSSSvova et al., 2010) to extract en- tity types and totally collect 12 types of default en- tity information. We extract 9 types of entity infor- mation, including 'SignSympoomMention,, 'Predi- cate', DDiseaseDisorddeMMeeniii 'MedicationMen- tion', 'RomanNumeenllnmottiionn SiteMention' "ractionAnnntattoon" 'Procedure- Mention', and 'DATE'. We add 'O' (standing for outside) and regard the special token '[CLS]' and '[SEP]' in PLMs as specific entity type. Thus, we have total of 12 entity types in this work.
Query example score
Eniity Aligned MHA
EMB
FFW
+
'Anatomical-
Retrieval instance score
aduer
Figure 2: The architecture of our EARA. The dark red rectangles are the newly added or modified modules.
3.3 Overall Architecture
Figure 2 shows the overall architecture of EARA. Firstly, the retriever module retrieves the nearest in- stance as retrieved instance. Secondly, an embed- ding layer encodes the tokens of the query exam- ple and retrieved instance. Next, entity aligned MHA module encodes the complex semantics of inputs. Finally, fully-connected layer predicts the similarity score. Figure shows the details of the retriever module and entity aligned MHA module.
Retrieval-basee Methods Retrieval-based meth- ods are now being utilized in various NLP tasks. As part of retrieval augmentation research, one ap- proach involved retrieving similar examples from internal training sets to act demonstrations for prompt learning in few-shot setting (Gao al., 2021; Kumar and Talukdar, 2021). Other works used retrieval augmentation for supervised tasks; they aimed to retrieve "answers" for their tasks, such as retrieving the next token in the language model pre-training task, retrieving summarizations from the nearest passages (Chen et al., 2022; Guu et al., 2020; Borgeaud et al., 2022b; Khandelwal et al., 2021; Wang et al., 2022a). While other mod- els rely nearest neighbours solely for enhanc- ing the prediction process, our approach retrieves similar instances to make the model consider a broader range of instances, which can help avoid overfitting and improve the models' generaliza- tion.
3.4 Entity-Aligned Multi- Heat Attention
Neural machine translation task (Bugliarello and Okazaki, 2020; Slobodkin et al., 2021) shows posi- tive results when introducing external structures in- formation into the attention matrix of transformers. Inspired by this, we introduce entity alignment matrix constructed from entity information to assist PLMS in learning more effective attention matrix, thereby facilitating the identification of differences between sentence pairs.
We define an entity alignment matrix M E {0, 1) as follows:
3 EARA
MiiJ] = {1 =00
5; and zy the same type, otherwise,
3.1 Problem Statement
where 0 < i,j < l, and 7 is the total length of sen- tence Sa and Sr, i.e., I n m mm+3 o, and x are tokens in sentence S, and Sy. Since xn is special token corresponding to '[CLS]' in PLMs, we de- fine M
Given tWo biomedical sentences Sa {aq. 2 2 and sentence 5b ). where I; is the token in the sentences, and m stands for the sentence's length of S and 5b, respectively. The goal of the biomedical STS task is to train regressor with biomedical STS datasets and predict the similarity score of two sentences. Specifically, the gold similarity score ranges from to in N2C2STS, where higher score means the given two sentences are more semantically similar.
(of,
(i, 0) or M [0.j] according to the type of the introduced entity information. we set M 0] = or M (0,j] = only in the cases that 2; or ty is an entity mention. Then, we normalize the entity alignment matrix M:
Vii = MW]
(2)
2j=0 MM

---

Retrieved
O00000
L2 loss
backward
forward Medication Alignment
Instance
Anatomical Site Alignment
XX Ahnstan
MatMul
ibuprofen 200mg by mouth [SEP]
Built by MIMIC-III
Instance Embeddings
SoftMax
Index
Mask
atenolol
50mg
tablet
MIMIC-III
Scale
Aligrment Matrix
by
Database
mouth
Input Instance
MatMul
200mg
mount
(SEP
alenolol
50mg
table
by
mauth
Quary
Key
Value
Encoder
S1: ibuprofen 200mg by moulh
Input Embedding
Query Example: (CLS) (SEP) (SEP) atenolol 50mg lablet by mouth
V
Retriever
Entity Aligned Multi-Head Attention
Figure The details of Retriever module and Entity Aligned MHA module.
where V E Rixi__ As shown in Figure 2, we define auxiliary L2 loss between multi-head attention scores and alignment matrix to regularize the atten- tion mechanism.
to encode the instances in the MIMIC-III. For each sentence 8; in the MIMIC-III, generate
key-value pair s1), 8;), where u, represents the embedding of the "[CLS]" token in the final layer of PLMs.
ΣH
Eh=1 A
Latign
H
3.5.2 Efficient Searching
where is the number of aligned attention heads, Va and A₁ are the normalized alignment matrix and the attention weight of the ftth attention head, respectively. Especially, the entity aligmment ma- trix Vn, of each head is equal to V. The attention weight A₁ is calculated by:
Efficient retrieval is crucial considering the po- tential enormity of the knowledge database. To achieve this, create matrix D E R|Clxd the index of MIMIC-III instances, previously demonstrated in the knowledge database creation process. When query example is presented dur- ing training, we encode it using the encoder to obtain the query vector y. This query vector is then utilized to search for the K nearest in- stances in the index /) through the maximum inner product search method. To perform this retrieval process, utilize FAISS (Johnson et al., 2019), excellent open-source toolkit, enable fast nearest- neighbour retrieval of MIMIC-III instances from the knowledge store.
O K Vd
An
where Qn and K₁, are query and key in the hth attention head, is the corresponding dimension.
3.5 Retriever
The retreeval-augmenteed module collects the most similar instance to the input from MIMIC-III (John-
son et al., 2016) knowledge store. In this work, we use MIMIC-III NOTEEVENTS.com as the knowl- edge database.
3.6 Final Similarity Score Prediction
Finally, given the input sentence pair semantic p and nearest retrieved sentence pair ro, they get sen- tence pair semantic representation O, and retrieved sentence pair semantic representation O We fully-connected layer with ReL.U activation predict the final similarity score Sp and Syg- During training, we minimize the mean square error loss
3.5.1 MIMIC-III Knowledge Database
To implement the retrieval module, the first step is construct knowledge database that can ef- fectively capture the semanties of the input from the MIMIC-III knowledge database This is achieved by utilizing a frozen BERT encoder

---

PCC. which measures the linear correlation be- tween two sets of data. PCC is calculated by the (5) ratio between the covariance of two variables and the product of their standard deviations. Therefore, and alignment loss below:
$p MMLP(O)) MLL((Oro)
L Lsts + X Latign.
(6) is basically a normalized measure of covariance such that the result always has a value between -1 and 1.
Lats = Ms + Sr, 8y);
where 8y is the gold score of the input sentence pair, À is hyper-parameter weigh the STS predic- tion loss and alignment loss. During the inference period, we do not use the retrieval instance.
4.2 Baselines and Implementation
We pick 3 representative PLMs as the baseline models and implement our EARA based these PLMs.
4 Experiments
BERT (Devlin al., 2019): BERT is a power- ful language model in general domain. is pre- trained on the BooksCorpus (Zhu et al., 2015) and Wikipedia, and reaches state-of-the-an results on the generic natural language processing tasks, like GLUE (Wang et al., 2019).
4.1 Benchmarks and Metrics
In this section, give brief introduction of datasets on which evaluate all models.
N2C2STS (Wang et al., 2020b): The N2C2STS data are expanded on the McdSTS (Wang et al., 2020a) data. All data the MedSTS dataset are used as the training set for N2C2STS, and 594 additional data are added as the new training set. The training set of N2C2STS has 1,642 sentence pairs in all, and the test set has 412 sentence pairs. The similarity score of N2C2STS ranges from 0 (low similarity) to (high similarity).
BioBERT (Lee et al., 2020): BioBERT is pre- trained with PubMed abstracts and PubMed Cen- tral full-text articles based on the pre-trained BERT model weights. It achieves a promising perfor- mance on the biomedical NLPP task.
BlueBERT (Peng et al., 2019): BlueBERT is a pre- trained language model based on pre-trained BERT with the addition of PubMed data or electronic medical record MIMIC-III data (Johnson et al., 2016).
BIOSSES (Sogancioglu et al., 2017): The BIOSSES comprises 100 sentence pairs, in which each sentence is selected from the Text Analy- sis Conference (TAC) Biomedical Summarization Track Training Dataset containing articles from the biomedical domain. The data is annotated by five experts and is scored from 0 (no relation) to (equivalent).
We apply the PLMs with 12 layers, hidden size 768 and 12 heads. For the pre-trained PLMs, we tune the learning rate in [le-5, 2e-5], batch size in [3, 5, 10J. For other parameters, we tune the initial learning rate to be [le-5, 2e-5], and tune the weighted loss À in [0.1, 0.3, 0.5, 0.7, 1.0]. The maximum sentence length varies from the dataset, 380 for N2C2STS, 200 for BIOSSES and EBM- SASS, We run 8 epoches entire model is implemented with Tensorflow and trained on 12G GTX 1080 GPU.
EBMSASS (Hassanzadeh et al., 2019): EBMSASS consists of 1,000 pairs of clinical evidence. These pairs of clinical evidence are generated from the NICTA-PIBOSO corpus, which is retrieved from 1000 biomedical abstracts. Ten annotators, includ- ing eight bioinformatics researchers, one clinical terminology, and one molecular biologist, manu- ally analyze the dataset and assign similarity scores, ranging from (low similarity) to 5 (high similar- ity).
both datasets. The
4.3 Experimental Results
In this section, we evaluate our EARA based on dif- ferent backbones on the N2C2STS, EBMSASS and BIOSSES and present the results transferred from N2C2STS to EBMSASS and BIOSSES. We com- pare our results to existing methods and conduct ablation studies.
The BIOSSES and EBMSASS datasets have no specific test set, so we random sample 80% training set and 20% are held out test set. For in-domain evaluation, following Xiong al. (Xiong al., 2020a), we train our super- vised model on the training set using a 5-fold cross- validation and evaluate our model on the gold test set. All experimental results are evaluated by the
4.3.1 In-Domain Performance
Table presents comprehensive summary of the results obtained by our proposed method, EARA,
http::///uhmme...hiiam....t

---

N2C2STS
BIOSSES
EBMSASS
Average
4%
BERT
0.8549
0.8570
0.9005
0.8708
EARA (BERT)
0.8650
0.9316
0.9244
0.9070
+3.62
BioBERT
0.8668
0.9057
0,9181
0.8969
EARA (BioBERT)
0.8782
0.9446
0.9263
0.9163
+1.94
BlueBERT
0.8630
0.7978
0.9226
0.8611
EARA (BlueBERT)
0.8872
0.8913
0.9313*
0.9033
+4.22
Table 1: Pearson Correlation Coefficient (PCC) indicate the best performance on each dataset. **' denotes significant improvement. A(%) represents the average improvement of the three datasets.
different STS datasets with various backbones. Bolded results
BIOSSES
EBMSASS
of BERT domain-specifet tasks. Overall, our findings highlight the effectiveness of our proposed approach and its potential for advancing the state- of-the-art in natural language processing.
BERT _CS
0.5326
0.5241
BlueBERT_CL
0.7752
0.6283
BioBERT_CLL
0.7759
0.6269
BERT
0.8559
0.8266
4.3.2
Out-of-Domain Performance
EARA (BERT)
0.8604
0.8460
BioBERT
0.8575
0.8236
To demonstrate the generalizability of our proposed model, trained it on the N2C2STS dataset and evaluated its performance on the entire BIOSSES and EBMSASS datasets. our evaluation, we compare our approach, EARA, with not-finetuned PLMs along with corresponding baseline PLMs trained on N2C2STS datasets. The experimen- tal results, as presented in Table 2, indicate that our best-performing EARA model outperforms the best not-finetuned PLMs by 9.14% and 23.18% the BIOSSES and EBMSASS datasets, respec- tively. Our EARA methods demonstrate promis- ing improvements the out-of-domain BIOSSES and EBMSASS datasets as compared to the base- line PLMs. Notably, EARA based on BioBERT achieves the highest PCC performance of 0.8829 on the BIOSSES dataset and obtains the best PCC performance of 0.83587 the EBMSASS dataset. Interestingly, our findings reveal that the model boosting observed on the BIOSSES and EBM- SASS datasets is inconsistent with the boosting trend observed on the N2C2STS dataset.
EARA (BioBERT)
0.8829
0.8587
BlueBERT
0.8403
0,8138
EARA (BlueBERT)
0.8558
0.8517
Table 2: Out-of oft-of-oee domain performance. Bolded results indicate the best results each model on each dataset.
Model
N2C2STS
(Xiong et al., 2020a)
0.868
(Ormerod et al., 2021)
0.870
(Chen et al., 2021) (single)
0.87
(Mulyar et al., 2021)
0.867
(Wang et al., 2022b)
0.875
EARA (BlueBERT)
0.887
Table PCC improvement compared previous work the N2C2STS dataset.
across three benchmark datasets: N2C2STS, EBM- SASS, and BIOSSES. Notably, our approach con- sistently outperforms the baseline models, with av- erage improvements ranging from 1.94% 4.22%. Our best model is significantly better than the base- line model (p-value<0.05)) Interestingly, we find that EARA based on BioBERT achieves the high- est performance most datasets. However, our results also demonstrate that EARA, incorporating retrieval augmentation and entity alignment mecha- nisms, can enable BERT achieve competitive re- sults with BioBERT and BlueBERT, both of which corpora. This observation suggests that these techniques hold sig- nificant potential for enhancing the performance
4.3.3 Comparison with Prior Works
We compare our model with several published state-of-the-art baselines under the same setting on different STS datasets. As illustrated in Table 3, our EARA based on BlueBERT achieved the high- est PCC performance of 0.887 on the N2C2STS dataset, surpassing the previous best model by 1.2%. Notably, the best results for BIOSSES and FBMSASS were reported by (Blagec al., 2019) and (Wang et al., 2022b), respectively, at 0.871 and 0.922. However, our EARA model outperforms
are pre-trained on domain-spectin

---

N2C2
BIOSSES
EBMSASS
5.1 Error Reduction
EbRA(bert)
0.8650
0.9316
0.9244
Figure 5 analyzes the enror rate of EARA in pre- dicting sentences with different numbers of entities. As shown in the figure, when the number of entities in a sentence is greater than 5, compared to Blue- BERT, EARA can significantly reduce the error probability by about 20%. Moreover, the overall error probability is also reduced from 25.7% to 18.9%. Unfortunately, when there are few entities (less than 3) in the sentence, the introduced entity alignment information slightly increase the ror probability. This inspires us to customize the learning strategy based on the numbers of entities in the future, determining when to introduce entity alignment information.
w/o RA
0.8593
0.8750
0.9013
w/o EA
0.8640
0.8737
0.9199
w/o All
0.8549
0.8570
0.9005
EARA(bio)
0.8782
0.9446
0.9263
w/o RA
0.8726
0.9397
0.9183
w/o EA
0.8793
0.9262
0.9184
w/o All
0.8668
0.9057
0.9181
EARA(blue)
0.8872
0.8913
0.9313
w/o RA
0.8699
0.8469
0.9256
w/o EA
0.8750
0.8162
0,9284
w/o All 0.8630
0.7978
0.9226
Table 4: Ablation studies. EA=Entity Alignment matrix; RA=Retrieval Augmentation; w/o=without; bio=BioBERT; blue=BlueBERT. N2C2 is the abbrevia- tion for the N2C2STS dataset, and the brackets indicate which baseline model EARA is based
5.2 Qualitative Analysis
We visualize the attention weights of baseline PLMs and EARA only with the entity alignment matrix module in Figure demonstrate that our model provides a clearer explanation. We select sentence pairs from the N2C2STS test sets.
them by a significant margin. Taken together, these experimental results demonstrate the effectiveness our proposed EARA model and its potential for enhancing the performance of state-of-thh--ar art mod- els in semantic textual similarity tasks.
As shown in Figure 4 (a), for the '[CLS]' token, the baseline model (B for '[CLS]') focused more on the first sentence and the punctuation. When we add the entity aligmment matrix (A for [[CLS]') the EARA pays more attention to important to- kens, such as medication brands 'ibuprofen' and 'atenolol', which is consistent with human judg- ment. Figure 4 (b) shows an example without entity alignment information during the inference period. The baseline model puts more attention stop words and punctuation, but our model learns focus on important nouns like 'consent', 'testing', 'risk', 'benefit', and 'altcrnatives' as well as verb information like 'read', 'accepted', 'explained', 'agreed', and 'proceed'.
4.3.4 Ablation Study
We conduct ablation studies to assess the individ- ual contributions of the entity alignment matrix and retricval augmentation to the final results. The results, presented in Table 4, show that removing each part and re-evaluating has a significant impact on the model's performance. Specifically, when we remove the retrieval augmentation module, the de- creased performance highlights the importance of adding the logits of retrieved instances during the training period. When we remove the entity align- ment matrix module, there is a substantial decrease in model performance, indicating that entity align- ment information makes it easier for the model to identify the similarity between two sentence seg- ments. Most notably, when we remove all modules, there is a larger decrease in the model's perfor- mance, suggesting that the entity alignment matrix module and retrieval augmentation module com- plement each other well in enhancing the model's overall performance.
We also pick a specific token show the token's 's contributions. Interestingly, as shown in Figure 4 (a) and (b), the baseline model only focused on the sentence containing the selected token. But with our alignment matrix, EARA can attend to not only the tokens in the sentence containing the selected token but the aligned tokens in the other sentence. Especially in Figure 4 (a), the tokens with the same type the selected token 'atenolol', such as 'ibuprofen' 'tablets', are paid more attention. And in Figure 4 (b), the token 'consent" attends to the token "testing" in the first sentence, and to- kens 'explained', 'risk', 'benefit', "alternatives', 'agreed', and 'proceed' in the second sentence.
5 Discussion
this section, we conduct further milysis to better understand the results of the experiments.

---

"ICLSI" "stenolol"
"ICLS "consent"
0 م- O m
0.
huppollnn
200
ma table
tablels
mou t eveng
hour
needed stend
ISEPT
tables table
moulfe
one tine
dally
[SEP]
(CL (CLS
following
consent
the resp
patient
and accepiag
ardet lessiing
explaines
the
banafi
and Aeamative
attematiie
amd Millee tad Sonto Froee
(a) The gold similarity: 2.50; The predicted similarity of B:
(b) The gold similarity: 1.00: The predicted similarity of B:
2.93; The predicted similarity of C: 2.41.
1.91; The predicted similarity of C: 1.54.
Figure 4: Token contribution heatmap. A: entity alignment matrix, B: baseline PLM, C: corresponding EARA. The three continuous of A, B, and C is the attention visualization of the token enclosed by quotation the left.
BLeBERT
EARA
1.0
1.0
0.8
0.8
* Sertterees
0.6
0.5
0.4
0.4
20
-01
(a) baseline model BioBERT
(b) EARA
Figure 6: The embeddings of the data in lower- dimensional space using t-SNE of the baseline model and EARA on very small dataset BIOSSES. 'o" stands for test data and 'x' is the training dataset.
2
3
>5
# Entitlis
Figure 5: The error probability EARA(BI))EETT) and BlueBERT on N2C2STS dataset.
as the sentence pair representation. We then uti- lize t-SNE (Van der Maaten and Hinton, 2008) reduce the dimensionality of the embeddings and visualize them in a 2D space, with similarity scores serving as the labels. Our analysis reveals that the embeddings the training and test data are closely clustered together in the dimen--innsionall space of EARA, indicating that the model is capable of generalizing well to new data.
5.3 Impact of Entity Number
To investigate the impact of entity information on proposed EARA model based on BlueBERT, we randomly discard certain percentage of entity information from the data and evaluate the per- formance on the N2C2STS dataset using only the entity alignment matrix. Our experimental results reveal that when we randomly mask 0%, 20%, 40%, 60%, 80%, and 100% of entities, the correspond- ing PCC results are 0.8699, 0.8637, 0.8632, 0.8570, 0.8630, respectively. Notably, when significant proportion of entity types removed (i.e., 80% discarded), the entity information may have intro- duced noise, potentially leading a decline in the model's performance.
Conclusion
In this work, we present an entity-aligned and re- trieval augmentation PLM for the biomedical se- mantic textual similarity tasks. We propose an entity alignment matrix and an auxiliary loss to reg- ularize the attention matrix of PLMs and introduce retrieval augmentation to improve the generaliza- tion ability of models further. The comprehensive experiments show that the EARA achieves state-of- the-art performance on the biomedical STS data. Besides, EARA also improves the performance out-of-domain datasets.
5.4 Generalization
To assess the generalization ability of our proposed EARA model, we conduct visualization analy- sis of the representations of the training and test sets, as depicted in Figure 6. To generate embed- dings, we feed the BIOSSES dataset through the model and extract the '[CLS]' token representation

---

7 Limitations
Emanuele Bugliarello and Naoaki Okazaki. 2020. En- hancing machine translation with dependency-awee aware self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1618-1627, Online. Association for Computational Linguistics.
The limitations of our work mainly lie in two as- pects. One is that when there few entities (less than 3) in sentence, our proposed EARA may hurt the representation of the sentence, leading to a slight increase in prediction error probability, as shown in Figure 5. The other is that the number of retrieved examples is limited by the computation and memory resources. Recent works (Borgeaud et al., 2022a) have demonstrated that introducing more retrievals improve model's performance.
David Chang, Eric Lin, Cynthia Brandt, Richard An- drew Taylor, et al. 2021. Incorporating domain knowledge into language models by using graph convolutional networks for assessing semantic tex- tual similarity: Model development and perfor- mance comparison. JMIR Medical Informatics. 9(1 1):e23101.
Qingyu Chen, Jingcheng Du, Sun Kim, W John Wilbur, and Zhiyong Lu. 2020. Deep learning with sentence embeddings pre-trained on biomedical corpora im- proves the performance of finding similar sentences in electronic medical records. BMC Medical Infor- maticy and Decision Making, 20( 1001)
8 Acknowledgement
This study is partially supported by National Key RD Program of China (2021ZD01 13402 ), National Natural Science Foundations of China (62276082) and Major Key Project of PCL (PCL2021A06).
Qingyu Chen, Alex Rankine, Yifan Peng, Elaheh Aghaarabi, Zhiyong Lu, al. 2021. Benchmarking effectiveness and efficiency of deep learning mod- els for semantic textual similarity in the clinical do- main: Validation study. JMIR Medical Informatics, 9(12)e27386.
References
Kathrin Blagec, Hong Xu, Asan Agibetov, and Matthias Samwald. 2019. Neural sentence embedding models for semantic similarity estimation in the biomedical domain. BMC Bioinform., 20(1):178:1-1788
Xiang Chen, Lei Li, Ningyu Zhang, Xiaozhuan Liang, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Decoupling knowledge from memorization: Retrieval- auggeneed prompt learn- ing. In Advances in Neural Information Processing Systems.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Bapien Ban-tattttte Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones. Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals. Simon Osindero, Karen Si- monyan, Jack W. Rac, Erich Elsen, and Laurent Sifre. 2022a. Improving language models by retrieving from trillions of tokens. In International Confer- ence on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of bert's attention. In Proceed- ings of the 2019 ACL Workshop BlackboxNLP. An- alyzing and Interpreting Neural Networks for NLP, BlackboxNI.PGG
2019, Florence, Italy, August I, 2079, pages 276-286. Association for Computational Linguistics.
Raphael Cohen, Michael Elhadad, and Noémie EI- hadad. 2013. Redundancy in electronic health record corpora: analysis, impact on text mining perfor- mance and mitigation strategies. BMC bioinformat- ics, 14(1):1-15
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, cean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Mag- giore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rac, Erich Elsen, and Laurent Sifre. 2022b. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240 PMLR.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre- me-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACI-IHLLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguisties.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics

---

and the 11th International Joint Conference on Natu- val Language Processing (Volume I: Long Papers), pages 38 16-3830, Online. Association for Computa- tional Linguisties.
Wei Liu, Xiyan Fu, Yue Zhang, and Wenming Xiao. 2021. Lexicon enhanced Chinese sequence labeling using BERT adapter. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 5847-5858, Online. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee. Zora Tung, Panupong Pasupat. and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
Diwakar Mahajan. Ananya Poddar, Jennifer J Liang. Yen-Ting Lin, John M Prager. Parthasarathy Surya- narayanan, Preethi Raghavan, Ching- Tsou, et al. 2020. Identification of semantically similar sentences in clinical notes: Iterative intermediate training us- ing multi-task learning. JMIR medical informatics, 8(1 1):222008.
Hamed Hassanzadeh, Anthony N. Nguyen, and Karin Verspoor. 2019. Quantifying semantic similarity of clinical evidence in the biomedical literature to facili- tate related evidence synthesis. J. Biomed. Informat- ics, 100.
Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similar- ity. In Proceedings of the Thirtieth AAAI Confer- ence Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2786-2792. AAAI Press.
Chen Jia, Yuefeng Shi, Qinrong Yang, and Yue Zhang. 2020. Entity enhanced BERT pre-training for Chi- nese NER. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6384-6396, Online, Association for Computational Linguistics.
Andriy Mulyar, Öxlem Uzuner, and Bridget T. McInnes. 2021. Mt-clinical BERT: scaling clinical information extraction with multitask learning. J. Am. Medical Informatics Assoc., 28(10):210--2) 2115.
Alistair FW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessi- ble critical care database. Scientific data, 3(1):1-9.
Mark Ormerod, Jesús Martinez Del Rincón, Barry Dev- creux, et al. 2021. Predicting semantic similarity be- tween clinical sentence pairs using transformer mod- els: Evaluation and representational analysis. JMIR Medical Informatics, 9S):e2309,
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemayer, and Mike Lewis. 2021. Nearest neigh- bor machine translation. International Conference on Learning Representations.
Heather C O'Donnnel, Rainu Kaushal, Yolanda Barrón, Mark A Callahan, Ronald D Adelman, and Eugenia L Siegler. 2009. Physicians' attitudes towards copy and pasting in electronic note writing. Journal of general internal medicine, 24(1)663-63
Sawan Kumar and Partha Talukdar. 2021. Reorder- ing examples helps during priming- based few-shot learning. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Trans- fer learning in biomedical natural language process- ing: An evaluation of BERT and elmo ten bench- marking datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, SSNNLP@ACL 2019, Flo- rence, Italy, August 1, 2019, pages 58-65. Associa- tion for Computational Linguistics.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinform., 36(4):1234 1240.
Guergana K. Savova, James J. Masanz, Philip V.
Ogren, Jiaping Zheng, Sunghwan Sohn, Karin Kip- per Schuler, and Christopher G. Chute. 2010. Mayo clinical text analysis and knowledge extraction sys- tem (ctakes): architecture, component evaluation and applications. Journal of the American Medical Infor- matics Association, 17(5):507-513
Junyi Li, Xuejie Zhang. Xiaobing Zhou, al. 2021a. Albert-based self-ensemble model with semisuper- vised learning and data augmentation for clini- cal semantic textual similarity calculation: algo- rithm validation study. JMIR Medical Informatics, 9(1 13:22606
Aviv Slobodkin, Leshem Choshen, and Omri Abend. 2021. Semantics-aware attention improves neural machine translation. CoRR, abs/21 10.06920.
Zhongli Li, Qingyu Zhou, Chao Li, Ke Xu, and Yunbo Cao. 2021b. Improving BERT with syntax-awar local attention. In Findings of the Association for Computational Linguistics: ACI/I/CNLP 2021, On- line Event, August 1-6, 2021, volume ACL/IJCNPP 2021 of Findings of ACL. pages 645-653. Associa- tion for Computational Linguistics.
Gizem Sogancioglu, Hakime Özüürk, and Arzucan Özgür. 2017. BIOSSES: semantic sentence simi- larity estimation system for the biomedical domain. Bioinform., 33(14)::44)) 4):44--558.

---

Joe Stacey, Yonatan Belinkov, and Marek Rei. 2022. Su- pervising model attention with human explanations for robust natural language inference. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Shirty-Fourth Conference on Innovative Ap-
Ying Xiong, Shuai Chen, Haoming Qin, He Cao. Yedan Shen, Xiaolong Wang. Qingcai Chen, Jun Yan, and Buzhou Tang. 2020b. Distributed representation and one-hot representation fusion with gated network for clinical semantic textual similarity. BMC medical informatics and decision making, 20(1):1-7.
plications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium Edlucational Advances in Ar- tificial Intelligene, EAAI 2022 Virtual Event, Febru- ary 22- March 1, 2022. pages 11349-1191100 11357. AAAI Press.
Xi Yang, Xing He, Hansi Zhang, Yinghan Ma, Jiang Bian, Yonghui Wu, et al. 2020. Measurement of semantic textual similarity in clinical texts: compar- ison of transformer-baee based models. JMIR medical informatics, 1111133
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-mee. Journal of machine learning research, 9(1)).
Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. 2016. ABCNN: attention-based convolutional neural network for modeling sentence pairs. Trans. Assoc. Comput. Linguistics, 4:259 272.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill. Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations. ICIR 2079, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Rui Zhang. Serguei Pakhomov, Bridget T McInnes, and Genevieve B Melton. 2011. Evaluating measures of redundancy in clinical texts. In AMIA annual symposium proceedings, volume 2011, page 1612. American Medical Informatics Association.
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022a. Training data is more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 31 3170-3179, Dublin, Ireland. Association for Computational Lin- guistics.
Zhuosheng Zhang, Yuwci Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020a. Semantics-aware BERT for language understanding. In The Shtirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- live Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9628-9635. AAAT Press.
Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2020a. Medsts: resource for clinical semantic textual similarity. Lang. Resour. Evalua- tion, 5-(1)577-7.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020b. Sg-net: Syntax-guide guided machine reading comprehension. The l'hirty-Fourt AAAI Conference on Artificial In- telligence, AAAI 2020, The Phirty-Secone Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9636-9643 AAAI Press.
Yanshan Wang. Sunyang Fu, F Shen, S Henry, O Uzuner, and H Liu. 2020b. Overview of the 2019 n2c2/ohnlp track on clinical semantic textual similarity. JMIR Medical Informatics.
Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. 2022b. Uncertainty estimation and reduc- tion of pre-trained models for text regression. Trans- actions of the Association for Computational Linguis- tics, 10:680-696.
Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuail- iang Zhang. 2020. LIMIT-BERE Linguistics in- formed multi-task BERT. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pages 4450 4461, Online, Association for Computa- tional Linguistics.
Yuxia Wang, Fei Liu, Karin Verspoor, and Timothy Baldwin. 2020c. Evaluating the utility of model con- figurations and data augmentation clinical seman- tic textual similarity. In Proceedings of the 19th SIG- BioMed Workshop on Biomedical Language Process- ing. pages 101, Online. Association for Compu- tational Linguistics.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba. and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEÉE Interna- rional Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19-27. IEEE Computer Society.
Ying Xiong, Shuai Chen, Qingcai Chen, Jun Yan, Buzhou Tang, et al. 2020a. Using character-level and entity-level representations to enhance bidirectional encoder representation from transformers-basee clin- ical semantic textual similarity model: Clinical- sts modeling study. JMIR Medical Informatics, 8( (11))7233

---

Avg
Vole
87
86
0
PCC 85
65
1011
Layer
Heads
(a) PCC of each transformer (b) PCC of supervising differ-
layer N2C2STS.
ent attention heads.
Figure 7: Impact on different transformer layers and different attention heads. L6=Supervising the last 6 heads; F6=Supervising the first 6 heads; A=Supervising heads; N=Supervising no heads; Avg=Average PCC results, Vote=Voting results.
A Choosing Which Layer to Supervise
We evaluate the performance of each layer in Blue- BERT on N2C2STS. We use the '[CLS]' token representation of each layer to predict the similar- ity score. As shown in Figure 7 (a), we find that the last layer obtains the best PCC performance. Therefore, we choose to supervise self-attention heads the last layer in PLMs with multi-type fine-grained entity information.
B Choosing Which Heads to Supervise
The last layer of PLMs has 12 attention heads, which play different roles (Clark et al., 2019). We investigate which single or multiple heads we should supervise to improve the performance. We evaluate BlueBERT on the N2C2STS and train with 5-fold cross-validation We obtain five test results, then we ensemble the results as the voting results to compute the PCC. Besides, we also cal- culate each test PCC and average the PCC results. When determining which head regularize, consider both voting PCC and average PCC.
As shown in Figure (b), we depict the voting PCC and average PCC of five models for different choices of heads as well as the error bar. From the experimental results, we find that supervising the 8th head, the 12th head or all heads can give better PCC performance. As a result, the results in Table and Table 4 the results based the best- pest-performin cases on the validation among these three scenarios.


Reviews:
[{"review_id": "HoNRrLuwLA", "reviewer": "Reviewer_yReV", "summary": "", "questions": "The author needs to show some theoretical interpretation of the mechanism of the proposed method. Also, need supplemental experiments such as the significance test with experimental validity.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**yReV-Q1:  Theoretical interpretation.**\n\nSorry for confusing you. The reason why we introduce entity-alignment information to guide the semantic similarity computation is that semantic information of texts in the biomedical domain is primarily expressed through entities rather than language structures. Therefore, the proposed method in this paper mainly focuses on how to capture the entity-to-entity relationships across sentences in the semantic similarity task, that is, incorporating entity alignment information across sentences as well as retrieving examples similar to the input samples to expand the scope of entity pairs, which can enhance the method's robustness and preventing overfitting. Specifically, in the N2C2 dataset, sentences with more than 5 entity types account for over 42% of the total. If the alignment relationships between similar entities are not constrained, the baseline model exhibits a high error probability on sentences with multiple entity types (as shown in Figure 1(a)). Additionally, we observed that in the encoding of sentence pairs, most examples have word attention focused only on the sentence they belong to, with little attention paid to the content of the other sentence (as shown in Figure 1(b)). When using entity alignment information to constrain the attention mechanism of pre-trained models, the error probability on sentences with more than 5 entity types decreased from 45% to 25% (Refer to **tAyj-Q2**). Furthermore, visualizing the attention matrices revealed that incorporating entity alignment information allows for greater attention to be paid to similar information within sentence pairs (as shown in Figure 4). \n\n**yReV-Q2: Significance test with experimental validity.**\n\nThanks for your suggestion. Following the paired student’s t-test in [1], we do the t-test experiments on our best results for each dataset. We train each best model on the training set using a 5-fold cross-validation and evaluate our model on the test set. As a result, we obtain five results on the N2C2, BIOSSES, and EBMSASS data sets, separately. The significance test results are computed as follows:\n\nEBMSASS_BlueBERT p-value = 0.0325986636218; \n\nBioSSES_BioBERT p-value=0.0453868190038; \n\nN2C2_BlueBERT p-value=0.0337792965136.\n\nFrom the significance test results, our best model is significantly better than the baseline model (p-value<0.05).\n\n[1] Dror, R., Baumer, G., Shlomov, S. and Reichart, R., 2018, July. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 1383-1392).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper shows that this method makes PLMs pay more attention to important entities by aligning the same type entities in sentence pairs.\n2. EARA uses a retriever module and feeds retrievals into transformers, which improves the model’s generalization and eases the overfitting problem for the biomedical STS datasets.\n3. The authors visualize their results well for their evaluation metrics and perform a useful sensitivity analysis. Results show that their model can achieve promising performance both on in-domain and out-of-domain 102 datasets.\n", "reasons_to_reject": "The experimental results lack a significance test; hence, the demonstration of experimental validity is insufficient.", "questions_to_authors": "The author needs to show some theoretical interpretation of the mechanism of the proposed method. Also, need supplemental experiments such as the significance test with experimental validity.", "paper_topic_and_main_contributions": "In this paper, the authors propose a new model named EARA, an entity-aligned attention-based and retrieval-augmented Pre-trained Language Models. EARA firstly aligns the same type of fine-grained information in each sentence pair with an entity alignment matrix. Then, EARA regularizes the attention mechanism with an entity alignment matrix with an auxiliary loss. Finally, a retrieval module is added which retrieves similar instances to improve the model’s generalization. The experiments reflect that EARA can achieve state-of-the-art performance on three datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "There is a double ‘with in line 125"}, "extra_text": {}}, {"review_id": "mieIdyQxSu", "reviewer": "Reviewer_tAyj", "summary": "", "questions": "* Figure 2 tries to differentiate \"newly added or modified modules\" -- but added/modified compared to what? In keeping with that, Figure 2 is quite busy and hard to digest (FAISS, Frozen kNN Retriever, etc). Perhaps it would be better to put these portions in a \"black box\" if they can be referenced elsewhere.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**tAyj-Q1: The writing and presentation needs some polish.**\n \nThanks for your suggestion. We will improve the writing and polish the paper with careful proofreading.\n\n**tAyj-Q2: The error probability of EARA.**\n\nThanks for your good suggestion, which will make our work more comprehensive. The error probability for BlueBERT and EARA on the N2C2 dataset are shown as follows. \n\nTable 1. Error probability of BlueBERT and our EARA on N2C2 dataset.\n\n| # Entities |\tSentence ratio |\tBlueBERT\t| EARA (BlueBERT)|\n| ----------- | ----------- |----------- | ----------- |\n|0\t|9.51%\t|0.093023\t|0.250000|\n|1\t|12.44%\t|0.174419\t|0.193181|\n|2\t|13.17%\t|0.127907\t|**0.125000**|\n|3\t|9.02%\t|0.058140\t|0.113636|\n|4\t|13.66%\t|0.093023\t|**0.068181**|\n|5\t|42.20%\t|0.453488\t|**0.250000**|\n|Overall\t|100%\t|0.256713\t|**0.189333**|\n\nFrom the results, when the number of entities in a sentence is greater than 5, compared to BlueBERT, our proposed EARA can significantly reduce the error probability by about 20%. Moreover, the overall error probability is also reduced from 25.7% to 18.9%. Unfortunately, when there are few entities (less than 3) in the sentence, the introduced entity alignment information can slightly increase the error probability. This inspires us to customize the learning strategy based on the numbers of entities in the future, determining when to introduce entity alignment information.\n\n**tAyj-Q3: Evaluations lack a progression of thought.**\n\nThanks for your suggestion. Experiments in sections 4.3.5 and 4.3.6 are included in the main content, which may appear somewhat unrelated to the overall content. These experiments are conducted for parameter analysis. We will revise the description in our paper and move these experiments into the appendix.\n\n**tAyj-Q4: Limitations section doesn't discuss limitations.**\n\nThanks for your suggestion. We will rewrite this section. The limitations of our work mainly lie in two aspects. One is that when there are few entities (less than 3) in a sentence, our proposed EARA may hurt the representation of the sentence, leading to a slight increase in prediction error probability (refer to **tAyj-Q2**). The other is that the number of retrieved examples is limited by the computation and memory resources. Recent works [1] have demonstrated that introducing more retrievals can improve model’s performance.\n\n[1] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G.B., Lespiau, J.B., Damoc, B., Clark, A. and de Las Casas, D., 2022, June. Improving language models by retrieving from trillions of tokens. In International conference on machine learning (pp. 2206-2240). PMLR. \n\n**tAyj-Q5: Newly added or modified modules.**\n\nSorry for the confusion. The newly added or modified modules are in the context of the baseline pre-trained language model. Specifically, the entity aligned MHA module is modified from the multi-head self-attention module in the pre-trained language model transformer shown on the right side of Figure 2. The logits fusion module is a newly added module that fuses features with retrievals (refer to the formula 7 in the paper).\n\n**tAyj-Q6: Figure 2 is quite busy and hard to digest.**\n\nSorry for the confusion. We will revise the Figure 2. The overall architecture of our EARA is shown on the left side of Figure 2. Our EARA consists of two components: the retriever module and the entity-aligned backbone model. For the retriever module, we use the MIMIC-III instances to build the external database and adopt a frozen kNN retriever to efficiently retrieve similar examples from the database. The external database and the frozen kNN retriever are built before the training process. In the frozen kNN retriever, we use an efficient index to support similarity search, e.g., FAISS, a widely used vector similarity search library proposed by Facebook AI, and a frozen pre-trained language model (BlueBERT in this paper) to encode the input examples for similarity search. Notably, the retriever index is not specified to FAISS, there are also other famous indexes that can also be an option, such as ScaNN [1]. For the entity-aligned backbone model, we modified the multi-head self-attention module for the entity alignment and added a logits fusion module to fuse features with retrievals (refer to **tAyj-Q5**).\n\n[1] Guo, R., Sun, P., Lindgren, E., Geng, Q., Simcha, D., Chern, F. and Kumar, S., 2020, November. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning (pp. 3887-3896). PMLR.\n\n**tAyj-Q7: Put retriever portions in a \"black box\" & Figure 2 problem in typo.**\n\nThanks for your suggestion. We will revise the Figure 2. Specifically, we will replace the upper-left part of Figure 2 with a black box named the frozen retriever module and remove the right-side part in Figure 2, keeping Figure 2 concise and clear. Then, we will create a new figure demonstrating the details of the frozen retriever module and the entity-aligned multi-head attention module, corresponding to sub-figure (a) and sub-figure (b). We believe such modifications can better explain our proposed EARA and make readers understand our techniques easily.\n\n**tAyj-Q8: Missing reference and presentation improvements.**\n\n Thanks for your suggestion. We will revise all typos and further improve the presentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an appropriate methods paper. It would provide a new SotA in Medical STS.", "reasons_to_reject": "The writing and presentation needs some polish (see other notes), and the discussion needs some depth -- loose ends remain untied. For example, Figure 1a motivated this work; can you show the EARA-BlueBERT's characteristic with Error Probability? (Nicely supplied in the rebuttal.) There are plenty of evaluations, but they don't seem to all follow a progression of thought. Another example: the Limitations section doesn't discuss limitations.", "questions_to_authors": "* Figure 2 tries to differentiate \"newly added or modified modules\" -- but added/modified compared to what? In keeping with that, Figure 2 is quite busy and hard to digest (FAISS, Frozen kNN Retriever, etc). Perhaps it would be better to put these portions in a \"black box\" if they can be referenced elsewhere.", "paper_topic_and_main_contributions": "This work introduces a new method for biomedical semantic textual similarity (STS). Namely, the authors utilize named entities (obtained as a preprocess) and their alignment (computed on a pair of texts) in multi-head attention. Subsequently, given a query instance, they retrieve the most similar instance from a knowledge store and use it as a backoff model. Their model is able to obtain SotA results on the N2C2STS dataset.\n\nThere is an innovative methodological contribution here with an adequate evaluation showing excellent results. Enthusiasm is dampened somewhat by the lack of clarity in presentation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "* cTAKES should cite Savova et al 2010 instead of the website"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* L009: \"the Pre-trained Language Models\" -> \"Pre-trained Language Models\"\n* L010: \"an entity-aligned attention-based and\" -> \"entity-aligned, attention-based, and\"\n* L012: \"firstly\" -> \"first\"\n* L023: \"available at 1.\" -> \"available.1\"\n* L066: \"approach 1\" -> \"approach 1.0\"\n* L223: \"a entity\" -> \"an entity\"\n* Equation 1: x_i and x_j are not defined\n* Figure 2: not directly tied to Section 3's subsection numbers; hard to follow, strange to have the retrieval portion on the top. Dotted black line in the middle not helpful; suggestion: split into Figure 2 (overall architecture) and 3a (entity-aligned MHA) and 3b (alignment matrix)\n* L202: \"CTakes\" -> \"cTAKES\"\n* L308-309: \"totally has 1,642 sentence pairs,\" -> \"has 1,642 sentence pairs in all,\"\n* Table 1&2: spell out \"Pearson Correlation Coefficient\" in one of the captions.\n* Table 1: caption needs to be more descriptive; e.g., this is a standard test; describe the delta% number\n* Table 2: indicate this table is about out-of-domain training"}}, {"review_id": "VE8tFdycFl", "reviewer": "Reviewer_pgrB", "summary": "", "questions": "Question A：Please supplement results of more strong backbones with EARA method\nQuestion B:  Considering weakness2, please elaborate more on the motivation for introducing EA.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**pgrB-Q1: Low novelty due to the application of FAISS.**\n\nThanks for your comments and we think there might be some misunderstandings due to improper presentations in the original manuscript. The contribution of the frozen retriever module is to fuse features with retrievals, expanding the scope of similar entity pairs and enhancing the robustness of the model. We mainly focus on how to incorporate retrievals into the model, rather than how to apply FAISS library. Besides, the index in the frozen retriever module is not specified to FAISS and can be replaced with any other vector similarity search library, such as ScaNN [1]. The reason we chose FAISS is that it is a widely used vector similarity search library and can provide high retrieval efficiency. We think Figure 2 might misled readers to believe that we only apply FAISS to retrieve similar examples, so we will revise Figure 2 following the suggestions of **tAyj-Q7**.\n\n[1] Guo, R., Sun, P., Lindgren, E., Geng, Q., Simcha, D., Chern, F. and Kumar, S., 2020, November. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning (pp. 3887-3896). PMLR.\n\n**pgrB-Q2: Regularizing the attention mechanism & motivation for introducing EA.**\n\n Thanks for your comments. Indeed, regularizing attention mechanisms is one of the focal points in current NLP research, as we also mentioned in lines 214-215 of the paper, where we were inspired by previous work. However, determining what information to use for regularizing and how to regularize it for a specific task is currently a research paradigm. Thus, we introduce biomedical entity information and use the entity alignment matrix to regularize multi-head attention in the biomedical semantic textual similarity tasks. As explained in **yReV-Q1**, we provide a detailed explanation of why we introduce the entity alignment matrix. \n\n**pgrB-Q3: The entity-aligned multi-head attention.**\n\nThanks for your comments. Whether it's the way we introduced the entity alignment matrix or the retrieval of similar examples, our goal is to focus more on the entity pairs within sentence pairs to simplify the prediction of semantic similarity between sentence pairs. Retrieving similar examples effectively expands the scope of entity pairs, enhancing the model's robustness. From our ablation experiments, the two strategies we proposed complement each other. Following **yReV-Q2**, we conduct significance tests, and our proposed method shows a significant improvement compared to the baseline model.\n\n**pgrB-Q4: More backbone.**\n\nThanks for your suggestion. We supplement the results based on ELECTRA [1], which is the state-of-the-art encoder after BERT, RoBERTa. The experimental results are shown in Table 1.\n\nTable 1. Results on N2C2STS, BIOSSES, EBMSASS of ELECTRA and our EARA.\n\n|                |N2C2STS|\tBIOSSES|\tEBMSASS|\n| ----------- | ----------- |----------- | ----------- |\n|ELECTRA\t|0.8682\t|0.7166\t|0.8695|\n|EARA (ELECTRA)\t|0.8732\t|0.7677\t|0.8826|\n\nResults show that our EARA can also significantly improve model’s performance on different datasets.\n\n[1] Clark, K., Luong, M.T., Le, Q.V. and Manning, C.D., 2020. Electra: Pre-training text encoders as discriminators rather than generators. In the 8th International Conference on Learning Representations (ICLR).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "● Two innovative points show good transferability on three BERT-based backbones.\n● EARA achieve state-of-the-art performance on both in-domain and out-of-domain datasets.", "reasons_to_reject": "1. FAISS as an unsupervised retrieval technique has already been quite mature. In this paper, the authors applied FAISS to the task of biomedical semantic text similarity and conducted retrieval of similar instances. Although this application is not particularly innovative.\n2.  The method of regularizing the attention mechanism is not novel, and the author did not explain in depth why introducing the entity alignment matrix brings significant effects to this method, especially for Bio data.\n3. Entity-Aligned Multi-Head Attention（EA）did not bring significant improvement, as evidenced by the results of the ablation experiments. Compared to the supervision signals brought by retrieving additional knowledge, the effect of EA itself is not significant. (solved by introducing significant test)\n4. The optimized backbone models used in the study are all methods before 2020,  This makes the credibility of the EARA insufficient. (have more backbones now)", "questions_to_authors": "Question A：Please supplement results of more strong backbones with EARA method\nQuestion B:  Considering weakness2, please elaborate more on the motivation for introducing EA.", "paper_topic_and_main_contributions": "The paper proposes EARA, a model for improving biomedical semantic textual similarity (STS). EARA has two main components. Entity-Aligned Attention is constructed to align same-type entities between sentence pairs. Retrieval Augmentation retrieves similar instances from the MIMIC-III database by FAISS. Experiments show EARA achieves state-of-the-art results on biomedical STS datasets including N2C2STS, BIOSSES, and EBMSASS.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gDKo257LE3", "reviewer": "Reviewer_zskt", "summary": "", "questions": "**A)** How many epochs did you train the models for? (this information is missing from the hyperparameter description at line 365-374)\n\n**B)** Section 4.3.5. When you say you evaluate the performance of each layer in BlueBERT, do you mean the baseline BlueBERT or the EARA one?\n\n*Section 4.3.5 and 4.3.6 are extremely confusing and feel disconnected from the rest of the paper… Here are some questions about them.*\n\n**C)** Section 4.3.5, how do you evaluate the performance of the various layers?\n\n**D)** Section 4.3.6, what is the “voting PCC”? What about the average one reported in Figure 3b?\n\n**E)** End of Section 4.3.6, “As a result… validation set”. What do you mean with this? Where are the results of these experiments?\n\n*In general, were these two sections “preliminary tests” to see in which layers/heads to apply the EA part of the EARA? If that’s so, they should probably be moved before… Or this process of selecting which layers/heads to modify should be mentioned somewhere in Section 3 (?). It really feels like a separated set of unrelated experiments right now.*\n\n**F)** Line 528-536. Is the experiment performed using only EA or EARA? The text states “using only the entity alignment matrix” (so without RA).\n\nHowever, the result masking 0% of the entities is 0.8872, the same of EARA(blue). It should be 0.8699 if you are using EARA(blue) w/o RA.\n\nAt the same time, the results masking 100% of entities are 0.8630, which is the same or EARA(blue) w/o All, confirming which seems to confirm that you didn’t use RA… Therefore, the reported numbers right now seem to be a mix of using RA and not using RA.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**zskt-Q1: Training epochs.**\n\nThanks for your suggestion. We run 8 epochs on each dataset. We will revise the section 4.2.\n\n**zskt-Q2: Performance of each layer in BlueBERT.**\n\n Thanks for your comments. We evaluate the performance of each layer in baseline BlueBERT.\n\n**zskt-Q3: Evaluation of the various layers.**\n\n Thanks for your comments. We use the first token (‘[CLS]’ token) representation of each layer to predict the similarity score.\n\n**zskt-Q4: Voting PCC & average PCC.**\n\nThanks for your comments. We use 5-fold cross-validation and obtain five test results, then we ensemble the results as the voting results to compute the PCC. Besides, we also calculate each test PCC and average the PCC results. When determining which head to regularize, we consider both voting PCC and average PCC.\n\n**zskt-Q5: Question E.**\n\nSorry for the confusion. We will revise our descriptions. As you concluded, this should be a preliminary experiment. To determine which attention head should be regularized by the entity alignment matrix, we conducted experiments on different heads, and the results evaluated on N2C2 with BlueBERT are shown in Figure 3b. We found that regularizing the 8th head, the 12th head, or all heads obtain better results than others. Therefore, the results in Table 1 and Table 4 are the results based on the best-performing cases on the validation set among these three scenarios.\n\n\n**zskt-Q6: Question F.**\n\n Thank you for kindly pointing out our mistakes. When masking 0% of the entities, the result should be 0.8699 rather than 0.8872. When evaluating the impact of the entity alignment, we only use the entity alignment without retrieval augmentation. Therefore, the result masking 100% of the entities is 0.8630, which is the same as EARA(blue) w/o All.\n\n**zskt-Q7: Missing a legend/explanation.**\n\nThanks for your suggestion. We will add a legend to Figure 1 and revise the corresponding descriptions. Specifically, the blue bars represent the proportion of sentences containing the number of entities k (k=0,1,2,3,4,>5) in the sentence. \n\n**zskt-Q8: Footnote 3.**\n\nThanks for your suggestion. We will change the footnote.\n\n**zskt-Q9: “Unsupervised PLMs”.**\n\n Sorry for the confusion. Yes, the unsupervised PLMs are not-finetuned PLMs. We will revise the corresponding descriptions.\n\n**zskt-Q10: Line 480.**\n\nMany thanks for pointing out our typos. We will re-number heads starting from 1 and revise Figure 3b. \n\n**zskt-Q11: Improper figure legend (replacing “A, B, C” with “Align, PLM, EARA”).**\n\n Thanks for your suggestion, we will follow your suggestions and revise Figure 4.\n\n**zskt-Q12: The plot of t-SNE.**\n\n Thanks for your suggestion. In fact, Figure 5 is drawn on the training set and the test set, where the training set does include the validation set. We are sorry for confusions in the text description, and we will correct it in line 553.\n\n**zskt-Q13: Wrong order of the Limitations section.**\n\nThanks for your suggestion. We will change the order of the Limitation section. And for the section 4.3.5 & 4.3.6, we will revise them and move them to the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is effective, model-agnostic, and could be applied to different fields where entity-alignment is important\n- The authors proved the relevant code to reproduce the results\n", "reasons_to_reject": "- Some parts are confusing and feel disconnected from the rest of the paper (see questions C-E)\n", "questions_to_authors": "**A)** How many epochs did you train the models for? (this information is missing from the hyperparameter description at line 365-374)\n\n**B)** Section 4.3.5. When you say you evaluate the performance of each layer in BlueBERT, do you mean the baseline BlueBERT or the EARA one?\n\n*Section 4.3.5 and 4.3.6 are extremely confusing and feel disconnected from the rest of the paper… Here are some questions about them.*\n\n**C)** Section 4.3.5, how do you evaluate the performance of the various layers?\n\n**D)** Section 4.3.6, what is the “voting PCC”? What about the average one reported in Figure 3b?\n\n**E)** End of Section 4.3.6, “As a result… validation set”. What do you mean with this? Where are the results of these experiments?\n\n*In general, were these two sections “preliminary tests” to see in which layers/heads to apply the EA part of the EARA? If that’s so, they should probably be moved before… Or this process of selecting which layers/heads to modify should be mentioned somewhere in Section 3 (?). It really feels like a separated set of unrelated experiments right now.*\n\n**F)** Line 528-536. Is the experiment performed using only EA or EARA? The text states “using only the entity alignment matrix” (so without RA).\n\nHowever, the result masking 0% of the entities is 0.8872, the same of EARA(blue). It should be 0.8699 if you are using EARA(blue) w/o RA.\n\nAt the same time, the results masking 100% of entities are 0.8630, which is the same or EARA(blue) w/o All, confirming which seems to confirm that you didn’t use RA… Therefore, the reported numbers right now seem to be a mix of using RA and not using RA.\n", "paper_topic_and_main_contributions": "The paper focuses on the task of Semantic Textual Similarity.\nThe authors propose EARA, a finetuning technique that combines Entity Alignment and Retrieval Augmentation to better finetune language models for STS.\nRA is used to create a larger dataset, preventing overfitting, while EA is useful to guide the model’s attention.\n\nThe authors show that this training technique can be applied to different backbone models (BERT, BioBERT, and BlueBERT) and it leads to increased performances on three different datasets.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Figure 1a, please add a legend / explanation for the blue bars\n2) Footnote 3, the link should be https://pubmed.ncbi.nlm.nih.gov/\n3) Line 408 and 413, “unsupervised PLMs”, did you mean not-finetuned PLMs?\n4) Line 480, I believe that the text should read “7th head, the 11th head” or “8th head, the 12th head”, looking at the peaks in Figure 3b. In any case, it would be better to re-number the heads from 1 to 12 in Figure 3a and 3b to avoid confusion.\n5) Figure 4, it would be easier to read the Figure/text if you replaced “A, B, C” with “Align, PLM, EARA” or something similar.\n6) Line 552-553, any chance you can include the plot of t-SNE with training and validation data to visualize this claim?\n7) The “Limitations” section should be un-numbered and come *after* the Conclusions. It doesn’t count towards the page limit, so you have more space in the paper to better explain the unclear parts (e.g., section 4.3.5, 4.3.6).\n"}}]

Author rebuttal:

Meta-review: As a reviewer indicates the authors propose a new model named EARA, an entity-aligned attention-based and retrieval-augmented Pre-trained Language Models. EARA firstly aligns the same type of fine-grained information in each sentence pair with an entity alignment matrix. Then, EARA regularizes the attention mechanism with an entity alignment matrix with an auxiliary loss. Finally, a retrieval module is added which retrieves similar instances to improve the model’s generalization. The experiments reflect that EARA can achieve state-of-the-art performance on three datasets.


The main reasons to accept the paper are the following ones:

	This is an appropriate methods paper. It would provide a new SotA in Medical STS.

	There is an innovative methodological contribution here with an adequate evaluation showing excellent results. 

	This paper shows that this method makes PLMs pay more attention to important entities by aligning the same type entities in sentence pairs.

 	EARA uses a retriever module and feeds retrievals into transformers, which improves the model’s generalization and eases the overfitting problem for the biomedical STS datasets.

	The authors visualize their results well for their evaluation metrics and perform a useful sensitivity analysis. Results show that their model can achieve promising performance both on in-domain and out-of-domain 102 datasets.

 	Two innovative points show good transferability on three BERT-based backbones. ● EARA achieve state-of-the-art performance on both in-domain and out-of-domain datasets.
	The proposed method is effective, model-agnostic, and could be applied to different fields where entity-alignment is important.

   	 The authors proved the relevant code to reproduce the results


The main reasons to reject the paper are the following ones:

	Enthusiasm is dampened somewhat by the lack of clarity in presentation. The writing and presentation needs some polish (see other notes), and the discussion needs some depth -- loose ends remain untied. 

	The experimental results lack a significance test; hence, the demonstration of experimental validity is insufficient.

	There are plenty of evaluations, but they don't seem to all follow a progression of thought. Another example: the Limitations section doesn't discuss limitations

	FAISS as an unsupervised retrieval technique has already been quite mature. In this paper, the authors applied FAISS to the task of biomedical semantic text similarity and conducted retrieval of similar instances. Although this application is not particularly innovative.

   	 The method of regularizing the attention mechanism is not novel, and the author did not explain in depth why introducing the entity alignment matrix brings significant effects to this method, especially for Bio data.

	Entity-Aligned Multi-Head Attention（EA）did not bring significant improvement, as evidenced by the results of the ablation experiments. Compared to the supervision signals brought by retrieving additional knowledge, the effect of EA itself is not significant. (solved by introducing significant test)

   	 The optimized backbone models used in the study are all methods before 2020, This makes the credibility of the EARA insufficient. (have more backbones now)


In sum, although the paper can be methodologically interesting, reviewers agree that the way it is written hampers the understanding of the paper. More work is needed for it to become a good paper.
Decision: Accept-Findings