# Syntax-Informed Self-Attention Network for Span-Based Joint Entity and Relation Extraction
## 4 Experiments  
### Datasets and Experimental Settings  
We conduct experiments on two relation extraction datasets and achieve significant improvements on both corpora.  
The first dataset is Conll04 [51], which contains sentences with annotated name entities and relations from news articles. We follow the processing method of the dataset in [40] and split the data into training set, validation set, and test set. The test set consists of 288 sentences. The training set and validation set contain 1153 sentences in total, and we use 20% of them for validation. For Conll04, it defines four entity types (_Person_, _Organization_, _Location_, and _Other_) and five relation categories (_Live-in_, _Work-for_, _OrgBased-in_, _Kill_, and _Located-in_).  
The second dataset is SciERC [52], which contains 500 scientific abstracts that come from 12 AI conference/workshop proceedings in four AI communities. We process the dataset according to [52]. There are 551 sentences for the test set, and 1861 and 275 sentences for the training and validation sets, respectively. For SciERC, it defines six entity types (_Method_, _Metric_, _Task_, _Material_, _Generic_, and _Other-ScientificTerm_) and seven relation types (_Feature-of_, _Used-for_, _Compare_, _Part-of_, _Hyponym-Of_, _Conjunction_, and _Evaluate-for_).  
Some statistical data of the two datasets are shown in Table 1.  
In our experiments, we measure NER and RE performance by computing precision, recall, and F-measure score on the test set. To evaluate the experimental results, we adopted the standard formula to calculate the scores as (17)-(19):  
\[Precision=\frac{correct\ positive\ predict}{all\ samples\ predicted\ as\ positive} \tag{17}\]  
\[Recall=\frac{correct\ positive\ predict}{all\ positive\ samples} \tag{18}\]  
\[F1=\frac{2\times Precision\times Recall}{Precision+Recall} \tag{19}\]  
Following previous works, for Conll04, we report both micro average and macro average F1 score. For micro-F1, we calculate the metric globally by counting the total true positives, false negatives, and false positives. For macro-F1, we calculate the metric for each label, and get their unweighted mean. A relation is correct if its type and corresponding two entities are all correct and an entity is considered correct if its span and type are both right. For SciERC, we report micro-F1 score as previous works and, in the RE task, we do not consider whether the entity type is correct.  
In this work, we employ BERT as our pre-trained model, and its parameters will be fine-tuned during training. For the Conll04 dataset, we will adopt BERT-base-cased from HuggingFace's Transformers [53]. In addition, for SciERC, the SciBERT [54] model will be adopted, which is trained on papers from the corpus of _semanticallynorg_ specifically. For the acquisition of syntactic features, we use the Stanford dependency parser of the Stanford CoreNLP toolkit [55] to generate dependency structures. We train the model with a learning rate of \(5\times 10^{-5}\), and adopt the Adam optimizer with weight decay as 0.01, using a linear warmup schedule. The dimension of width embedding is 25, and the batch size is set to 2. The contextual embedding size from BERT is 768, and the threshold for filtering relations is 0.55. For all layers with multi-head self-attention, the number of heads is 8, and we use half of the heads to focus on syntactic features. In addition, for span pruning, we limit spans to a max length of 10. All of the hyper-parameters are tuned on the validation set.