SlowGuess/ACL_2020_OCR huggingface dataset:
We implemented a CRF baseline model using CRFsuite to extract the software entities. This model uses standard orthographic, contextual and gazetteer features. It also includes the code markdown tags (§3.1.3) and a set of regular expression features. The regular expressions are developed to recognize specific categories of code-related entities. Feature ablation experiments on this CRF model are presented in Table 9. One noticeable distinction from the named entity recognizer in many other domains is that the contextual features are not as helpful in feature-based CRFs for classifying software entities. This is because, in the StackOverflow NER corpus a significant number of neighbouring words are shared among different software entities. As an example, the bigram 'in the' frequently appears as the left context of the following types: APPLICATION, CLASS, FUNCTION, FILE TYPE, UI ELEMENT, LIBRARY, DATA STRUCTURE and LANGUAGE.
P	R	F1
Feature-based CRF	66.85	46.19	54.64
- Context Features	68.91	43.58	53.39
- Markdown Feature	70.64	40.15	51.20
- Rule and Gazetteer Features	69.71	40.66	51.36
Table 9: Feature based CRF performance with varying input features on dev data.
B Attentive BiLSTM CRF with ELMoVerflow
We propose a baseline Attentive NER model that utilizes a BiLSTM-CRF network to predict the entity type of each word from its weighted representations. The weighted word representations are extracted by a multi-level attention network, similar to Yang et al.(2016), that combines the contextualized ELMo embeddings with the code recognizer (§3.1.2) and segmenter vector (§C). These three input embeddings are merged together in the first attention layer and then their corresponding weights are calculated using the second layer. Although such multi-level attention is not commonly used in NER, we found it empirically helpful for the software domain (see Table 10).
Embedding-Level Attention uses three embeddings, ELMo  $(w_{i1})$ , Code Recognizer  $(w_{i2})$ , and Entity Segmenter  $(w_{i3})$ , for each word  $w_i$  in
P	R	F1
Attentive BiLSTM-CRF	79.43	80.00	79.72
- Multi-level Attention	77.68	78.08	77.88
- Code Recognizer	77.18	77.76	77.47
- Entity Segmenter	74.82	75.32	75.07
Table 10: Ablation study of Attentive-NER on the dev set of StackOverflow NER corpus.
the input sentence. The embedding-level attention  $\alpha_{it}$  ( $t\in \{1,2,3\}$ ) to captures each embedding's contribution towards the meaning of the word. To compute  $\alpha_{it}$ , it passes the input embeddings through a bidirectional GRU and generate their corresponding hidden representations  $h_i = \overleftarrow{GRU} (w_i)$ . These vectors are then passed through a non-linear layer, which outputs  $u_{it} = \tanh (W_e h_{it} + b_e)$ . It uses an embedding-level context vector,  $u_{e}$ , which is learned during the training process. This context vector is combined with the hidden embedding representation using a softmax function to extract weight of the embeddings,  $\alpha_{it} = \frac{\exp(u_{it}^T u_e)}{\sum_t\exp(u_{it}^T u_e)}$ . Finally, the word vector is created by a weighted sum of all the information from different embeddings as  $word_i = \sum_t\alpha_{it}h_{it}$ .
Weighted Word Representation uses a word-level weighting factor  $\alpha_{i}$  to emphasize the importance of each word  $w_{i}$  for the NER task. Similar to the embedding-level attention, it calculates  $\alpha_{i}$  from the weighted word vectors  $word_{i}$ . A bidirectional GRU is used to encode the summarized information from neighbouring words and thus it gets  $h_{i} = \overleftarrow{GRU}(word_{i})$ . This is then passed through a hidden layer which outputs  $u_{i} = \tanh(W_{w}h_{i} + b_{w})$ . Then the normalized weight for each word vector is extracted by  $\alpha_{i} = \frac{\exp(u_{i}^{T}u_{w})}{\sum_{t}\exp(u_{i}^{T}u_{w})}$ , where  $u_{w}$  is another word-level context vector that is learned during training. The final weighted word representation is computed by  $word_{i}' = \alpha_{i}h_{i}$ .
Subsequently, the aggregated word vector  $\text{word}_i'$  is fed into a BiLSTM-CRF network, which predicts the entity category for each word. The complete architecture of the Attentive BiLSTM-CRF model is illustrated in Figure 4. Compared to BiLSTM-CRF, our proposed Attentive BiLSTM-CRF demonstrates a 9.7 increase in  $\mathbf{F}_1$  on the test set (Table 2) and reduces the segmentation errors and entity type errors by  $80.33\%$ $23.34\%$  respectively.
<sup>8</sup>http://www.chokkan.org/software/crfsuite/
Figure 4: Our SoftNER model. It utilizes an attention network to combine the contextual word embeddings (ELMo) with the domain-specific embeddings (Code Recognizer and Entity Segmenter). The detailed structure of the attention network is depicted on the right.
C Entity Segmentation with ELMoVerflow
The Attentive-NER tagger utilizes the outputs from an auxiliary segmentation module which consists of a BiLSTM encoder and a CRF decoder. This model concatenates ELMo embeddings with two hand-crafted features- word frequency and code markdown (§3.1.3).
The segmentation model follows the same architecture and training setup as the Attentive-NER model except for the input, where ELMo embeddings are concatenated with 100-dimensional code markdown and 10-dimensional word frequency features. The binary output from this entity segmenter model is later passed as through an embedding layer and used as one of the auxiliary inputs of the Attentive NER model.
Table 11 shows the performance of this segmentation model with ELMoVerflow on the dev set. This model achieves an  $\mathrm{F}_1$  score of 84.3 and an accuracy of  $97.4\%$ . The ablation study in Table 11 depicts the importance of the hand-crafted frequency and markdown features for this segmenter model by providing an increment of 1.2 and 2.1 points in the  $\mathrm{F}_1$  score respectively.
P	R	F1
Entity Segmentation (ELMoVerflow)	86.80	81.86	84.26
- Word Frequency	84.61	81.53	83.04
- Code Markdown	82.49	81.83	82.16
Table 11: Ablation study of our segmentation model with ELMoVerflow on the dev set of the StackOverflow NER corpus.