## 5 Experiments and Results  
### Methodology  
In the experiments, we pursued the following objectives:  
* To find the optimal model for mention detection (in Section 5.2). In Sections 5.2.1 and 5.2.2, respectively, we choose the optimal language model and combination of input features for Model A. In Section 5.2.3, we compare several variants of neural network topology for Model A. Then, we evaluate the XLM-RoBERTa model separately, and combine it with the optimal features found for Model A, resulting in the creation of Model B;
* To compare the ADR mention extraction accuracy on our corpus against the available data of a similar type for the Russian language (see Section 6);
* To show how the following characteristics of the corpus affect the ADR extraction accuracy: the proportion of phrases containing ADR, the proportion of ADR and Indication mentions, the corpus size, etc. (in Section 5.3);
* To evaluate the influence of the strictness of ADR labeling on the ADR identification precision (in Section 5.4).  
The reason for the focus on ADR when calibrating models is that this entity type is practically important while at the same time the most difficult for automated identification because it is strongly dependent on its context.  
The performance of the entity detection models is estimated with the help of the chunking metric that was introduced at the Conll-2000 shared task and has been used to compare named entity recognition systems since then. The script ([[https://www.clips.ununwrepen.be/conll2000/chunking/](https://www.clips.ununwrepen.be/conll2000/chunking/)]([https://www.clips.ununwrepen.be/conll2000/chunking/](https://www.clips.ununwrepen.be/conll2000/chunking/)), accessed on 12 December 2021) receives a file as its input, where each line contains a token, its true tag and its predicted tag. Tags could be \"O\" if the token does not belong to any mentions, \"B-X\" if the token starts a mention of some type X or \"I-X\" if it continues a mention of type X. If a tag \"I-X\" appears after \"O\" or \"I-Y\" (mention of some other type), it is treated as \"B-X\" and starts a new mention. We use the F1-exact score that estimates the accuracy of full entity matching. The script calculates F1-exact as the \(F_{1}\) score based on the percentage of detected mentions that are correct (precision) and the percentage of correct mentions that were detected (recall):  
\[F_{1}=\frac{2\cdot\text{precision}\cdot\text{recall}}{\text{precision}+\text{ recall}}.\]The coreference resolution performance is estimated with three commonly used metrics [63]: MUC, which is based on counting coreference relations added or missing in the generated markup compared to the ground truth; \(B^{3}\), where recall and precision are calculated for every mention as the fractions of correct mentions in the coreference chain to which this mention belongs in the generated markup; CEAFe, which is calculated by finding an optimal mapping between coreference chains from the ground truth markup to coreference chains in the generated markup, and then using another similarity metric to compare mentions in the obtained pairs of chains.  
Estimation of the Influence of Language Models, Input Features and Network Topology on the ADR Detection Accuracy