# Full-Abstract Biomedical Relation Extraction with Keyword-Attentive Domain Knowledge Infusion
## 3 Materials and Methods
### Network Architecture  
A network architecture (as shown in Figure 2) similar to [44] is adopted. However, our version has two differences compared to the original design, including the input form and keyword manipulation. The input is a sequence pair \((seq_{A},seq_{B})\), where \(seq_{A}\) starts with a [cls] token, ends with a [sep] token, and has a tokenized full abstract \(A\) in the middle; \(seq_{B}\) specifies the head and tail entity mentions. For the CDR dataset, \(seq_{B}\) consists of a chemical entity mention \(C\) (the head) and a disease entity mention \(D\) (the tail), both appearing in \(A\). The model's job is to understand the semantic relation between \(C\) and \(D\)  
Figure 1: System framework. BioBERT is fine-tuned on the CDR training set. Then, the keyword extraction algorithm is applied to the tuned BioBERT model to generate a set of seed keywords, expanded to form the final keyword set. The BioBERT is changed to Kw-BioBERT and further tuned on the training set with the keyword attention mechanism enabled. Finally, the tuned Kw-BioBERT is evaluated on the CDR test set.  
given \(A\) as a context. The neural architecture is modified from BioBERT (with the same neural architecture with BERT), adding a keyword attentive layer side by side with the last transformer encoder layer in BERT.  
The keyword attentive layer differs from a transformer encoder in two aspects. First, the attention masks, in the transformer encoder, are used to mask the padding tokens so that they do not participate attention; in the keyword attentive layer, however, the attention masks are also manipulated to allow tokens in \(seq_{A}\) to only attend the two entity mentions in \(seq_{B}\) and allow the two tokens in \(seq_{B}\) to only attend the keywords in \(seq_{A}\). In other words, for each token in \(seq_{A}\), we only care about its attention (or impact) on the two entity mentions in \(seq_{B}\); also, for each token in \(seq_{B}\), we only consider its attention on the keywords in \(seq_{A}\).  
With the manipulation attention masks, the model learns how the entity mentions and the keywords interact and jointly determine the relation. The output of the keyword attentive layer is a vector of hidden states \([h_{cls},h_{A1},h_{A2},...,h_{B1},h_{B2},h_{sep}]\), which has the same size as the input. The hidden state vector can be divided to two sections corresponding to \(seq_{A}\) and \(seq_{B}\). Then, a pooling operation is applied to each section individually, producing \(h_{kw,A}\) and \(h_{kw,B}\), which represent the aggregated and keyword-attentive embeddings for \(seq_{A}\) and \(seq_{B}\), respectively. Now, the semantic difference between \(h_{kw,A}\) and \(h_{kw,B}\) is denoted as \(h_{diff}\), which is defined as  
\[h_{diff}=[h_{kw,A}-h_{kw,B};h_{kw,B}-h_{kw,A}] \tag{1}\]  
Figure 2: Neural architecture of Kw-BioBERT. A keyword-attentive layer is added in parallel with the last transformer encoder to represent the semantic interaction between the relation-suggestive tokens and entity mentions.  
where \([;]\) is the concatenation operation. Next, the four pieces of information are concatenated, including the \(h_{cls}\) from the last transformer encoder, \(h_{kw,A}\), \(h_{kw,B}\), and \(h_{diff}\), and feed the resulting vector into the detection head, which consists of a dense layer and a softmax function.