# Biomedical Flat and Nested Named Entity Recognition: Methods, Challenges, and Advances
## 5 Analysis of Entity Types and Span Representation
### Strategies for Obtaining Span Representations  
The span representation strategy is a key factor in span-based models and significantly impacts model performance. This section evaluated and compared four span representation strategies, such as concatenation, max-pooling, mean-pooling, and PL-marker. Because previous studies have performed comparisons with subsets of these strategies, this experiment aims to provide a comprehensive evaluation of all four strategies.  
The concatenation, max-pooling, and mean-pooling strategies generate span representations based on token representations. Formally, given a sentence of \(n\) tokens, an input is denoted as \(X=\{x_{1},x_{2},\ \ldots,x_{n}\}\), and their latent embeddings are denoted as \(H=\left\{\hat{h}_{1},\hat{h}_{2},\ \ldots,\ \hat{h}_{n}\right\}\). The candidate spans are defined based on the enumerate method as \(span(X)=\left\{s_{x_{1},x_{1}},\ \ldots,\ s_{x_{1},x_{1}},\ \ldots,\ s_{x_{n-1},x_{n}},\ \ldots,\ s_{x_{n},x_{n}}\right\}\), where \(l\) is the maximum span length. Given \(s_{x_{i},x_{j}}\), the concatenation [85], max-pooling, and mean-pooling [98] methods are as follows:  
\[Concat\Big{(}s_{x_{i},x_{j}}\Big{)}=\left\{\hat{h}_{i};\hat{h}_{j}\right\}, \tag{7}\]  
\[Max\text{-}pooling\left(s_{x_{i},x_{j}}\right)=\left\{\hat{h}_{i};max\left( \hat{h}_{i},\ldots,\ \hat{h}_{j}\right);\hat{h}_{j}\right\}, \tag{8}\]  
\[Mean\text{-}pooling\left(s_{x_{i},x_{j}}\right)=\left\{\hat{h}_{i};mean\left( \hat{h}_{i},\ldots,\hat{h}_{j}\right);\hat{h}_{j}\right\}. \tag{9}\]  
Here, ; denotes the concatenation operation.  
The PL-markers refer to individual spans and learn their contextual information [87]. Specifically, the span candidate \(s_{x_{i},x_{j}}\) is assigned a pair of markers \(m^{(s)}_{x_{i},x_{j}}\) and \(m^{(e)}_{x_{i},x_{j}}\), respectively referring to the start and end tokens of the span. A set of these markers is provided as input to the model along with a sentence. The latent embeddings of the marker are then employed as the representation of the span. Consequently, the PL-marker of span \(s_{x_{i},x_{j}}\) is defined as follows [87]:  
\[PL\text{-}Marker\Big{(}s_{x_{i},x_{j}}\Big{)}=\left\{\hat{h}^{(s)}_{m_{x_{i},x _{j}}};\hat{h}^{(e)}_{m_{x_{i},x_{j}}}\right\}. \tag{10}\]  
The experiment was performed on three datasets: GENIA [69], BB19 [70], and SciERC [99]. SciERC is not a biomedical dataset, but it is based on scientific corpora and contains nested NEs. The experiments were conducted on a single NVIDIA Tesla V100 DGXS 32 GB GPU (produced by NVIDIA Corporation in Santa Clara, CA, USA) using bioformers/bioformer-16L model. A batch size of 8 and a learning rate of \(3\times 10^{-5}\) were applied. The training was performed for 20 epochs for the GENIA dataset, while the BB19 and SciERC datasets were trained for 50 epochs. To ensure robustness, each experiment was repeated with five different seeds, and the average precision, recall, and F1-scores were used for performance comparison (Table 4). The concatenation strategy achieved an overall lower F1-score. Specifically, the differences from best strategies were substantial: \(-0.69\) in GENIA, \(-1.96\) in BB19, and \(-2.08\) in SciERC. These results demonstrate the effectiveness of pooling and marker strategies. The strategy that demonstrated consistently high performance was mean-pooling, which achieved the highest F1-score in the BB19 datasetand the second highest in both the GENIA and SciERC datasets. On the other hand, the PL-Marker strategy showed the best performance in the GENIA and SciERC datasets but ranked third in the BB19 dataset. These results indicate that the choice of strategy should consider the characteristics of the data. Additionally, although the PL-Marker strategy requires additional computational resources because it allocates two markers per span, it proved its worth by outperforming the second-best strategies by 0.58 in GENIA and 0.88 in SciERC. Therefore, future work should balance performance and computational cost when selecting a span representation strategy.