SlowGuess/ACL_2020_OCR huggingface dataset:
Jeniya Tabassum, Mounica Maddela, Wei Xu, Alan Ritter
{bintejafar.1, maddela.4, xu.1265, ritter.1492}@osu.edu
There is an increasing interest in studying natural language and computer code together, as large corpora of programming texts become readily available on the Internet. For example, StackOverflow currently has over 15 million programming related questions written by 8.5 million users. Meanwhile, there is still a lack of fundamental NLP techniques for identifying code tokens or software-related named entities that appear within natural language sentences. In this paper, we introduce a new named entity recognition (NER) corpus for the computer programming domain, consisting of 15,372 sentences annotated with 20 fine-grained entity types. We trained indomain BERT representations (BERTOverflow) on 152 million sentences from Stack-Overflow, which lead to an absolute increase of  $+10\mathrm{F}_1$  score over off-the-shelf BERT. We also present the SoftNER model which achieves an overall  $79.10\mathrm{F}_1$  score for code and named entity recognition on StackOverflow data. Our SoftNER model incorporates a context-independent code token classifier with corpus-level features to improve the BERT-based tagging model. $^1$
Recently there has been significant interest in modeling human language together with computer code (Quirk et al., 2015; Iyer et al., 2016; Yin and Neubig, 2018), as more data becomes available on websites such as StackOverflow and GitHub. This is an ambitious yet promising direction for scaling up language understanding to richer domains. Access to domain-specific NLP tools could help a wide range of downstream applications. For example, extracting software knowledge bases from
Figure 1: Examples of software-related named entities in a StackOverflow post.
text (Movshovitz-Attias and Cohen, 2015), developing better quality measurements of StackOverflow posts (Ravi et al., 2014), finding similar questions (Amirreza Shirani, 2019) and more. However, there is a lack of NLP resources and techniques for identifying software-related named entities (e.g., variable names or application names) within natural language texts.
In this paper, we present a comprehensive study that investigates the unique challenges of named entity recognition in the social computer programming domain. These named entities are often ambiguous and have implicit reliance on the accompanied code snippets. For example, the word 'list' commonly refers to a data structure, but can also be used as a variable name (Figure 1). In order to recognize these entities, we propose a software-related named entity recognizer (SoftNER) that utilizes an attention network to combine the local sentence-level context with corpus-level information extracted from the code snippets. Using our newly annotated corpus of 15,372 sentences in StackOverflow, we rigorously test our proposed SoftNER model, which outperforms BiLSTM-CRF model and fine-tuned BERT model for identifying 20 types of software-related named entities. Our key contributions are the following:
- A new StackOverflow NER corpus manually annotated with 20 types of named en
1Our code and data are available at: https://github.com/jeniyat/StackOverflowNER/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4913-4926 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tities, including all in-line code within natural language sentences (§2). We demonstrate that NER in the software domain is an ideal benchmark task for testing effectiveness of contextual word representations, such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), due to its inherent polysemy and salient reliance on context.
- An in-domain trained neural SoftNER tagger for StackOveflow (§3) that can recognize 20 fine-grained named entity types related to software developing. We also tested its performance on GitHub data ofREADME files and issue reports.
- A code token recognizer (§3.1) that utilizes StackOverflow code snippets to capture the spelling patterns of code-related tokens, and consistently improves the NER tagger.
- In-domain pretrained ELMo and BERT representations (§3.3) on 152 million sentences from StackOverflow that significantly outperforms off-the-shelf ELMo and leads to more than 21 points increase in  $\mathrm{F}_1$  score over off-the-shelf BERT.
Overall, our named entity tagger (SoftNER) achieves a  $79.10\%$ $\mathrm{F_1}$  score on StackOverflow and  $61.08\%$ $\mathrm{F_1}$  score on GitHub data for extracting the 20 software related named entity types. We believe this performance is sufficiently strong to be practically useful. We have released our data and code, including the named entity tagger, our annotated corpus, annotation guideline, a specially designed tokenizer, and pre-trained StackOverflow BERT and ELMo embeddings.
2 Annotated StackOverflow Corpus
In this section, we describe the construction of our StackOverflow NER corpus. We randomly selected 1,237 question-answer threads from Stack-Overflow 10-year archive (from September 2008 to March 2018) and manually annotated them with 20 types of entities. For each question, four answers were annotated, including the accepted answer, the most upvoted answer, as well as two randomly selected answers (if they exist). Table 1 shows the statistics of our corpus.  $40\%$  of the question-answer threads were double-annotated, which are used as the development and test sets in our experiments  $(\S 4)$ . We also annotated 6,501 sentences from GitHub README files and issue reports as additional evaluation data.
Train	Dev	Test	Total
#questions	741	247	249	1,237
#answers	897	289	315	1,501
#sentences	9,315	2,942	3,115	15,372
#tokens	136,996	43,296	45,541	225,833
#entities	11,440	3,949	3,733	19,122
per Question	per Answer
avg. #sentences	6.84	4.60
avg. #tokens	98.46	69.37
avg. #entities	7.62	5.11
avg. #tokens per sent.	14.38	15.08
Table 1: Statistics of our StackOverflow NER corpus. These counts exclude all the code blocks and output blocks (i.e., lines that appear within  $\langle$ code\rangle and  $\langle$ blockquote\rangle tags).
2.1 Annotation Schema
We defined and annotated 20 types of fine-grained entities, including 8 code-related entities and 12 natural language entities. The code entities include mentions of CLASS, VARIABLE, IN LINE CODE, FUNCTION, LIBRARY, VALUE, DATA TYPE, and HTML XML TAG. Whereas the natural language entities include mentions of APPLICATION, UI ELEMENT, LANGUAGE, DATA STRUCTURE, ALGORITHM, FILE TYPE, FILE NAME, VERSION, DEVICE, OS, WEBSITE, and USER NAME.
Our annotation guideline was developed through several pilots and further updated with notes to resolve difficult cases as the annotation progressed. Each entity type was defined to encourage maximum span length (e.g., 'SGML parser' instead of 'SGML'). We annotated noun phrases without including modifiers (e.g., 'C' instead of 'Plain C'), except a few special cases (e.g., 'rich text' as a common FILE TYPE). On average, an entity contains about 1.5 tokens. While VARIABLE, FUNCTION and CLASS names mostly consist of only a single token, our annotators found that some are written as multiple tokens when mentioned in natural language text (e.g., 'array list' for 'ArrayList' in Figure 1). The annotators were asked to read relevant code blocks or software repositories to make a decision, if needed. Annotators also searched Google or Wikipedia to categorize unfamiliar cases.
The annotators were asked to update, correct, or add annotations from the user provided <code> markdown tags. StackOverflow users can utilize <code> markdowns to highlight the code entities
2Our annotation guideline is available at: https://github.com/jeniyat/StackOverflowNER/.
within the natural language sentences. However, in reality, many users do not enclose the code snippets within the <code> tags; and sometimes use them to highlight non-code elements, such as email addresses, user names, or natural language words. While creating the StackOverflow NER corpurs, we found that  $59.73\%$  of code-related entities are not marked by the StackOverflow users. Moreover, only  $75.54\%$  of the <code> enclosed texts are actually code-related, while  $10.12\%$  used to are highlighting natural language texts. The rest of cases are referring to non-code entities, such as SOFTWARE NAMES and VERSIONS. While markdown tag could be a useful feature for entity segmentation (§3.1.3), we emphasize the importance of having a human annotated corpus for training and evaluating NLP tools in the software domain.
2.2 Annotation Agreement
Our corpus was annotated by four annotators who are college students majored in computer science. We used a web-based annotation tool, BRAT (Stenetorp et al., 2012), and provided annotators with links to the original post on StackOverflow. For every iteration, each annotator was given 50 question-answer threads to annotate, 20 of which were double-annotated. An adjudicator then discussed disagreements with annotators, who also cross-checked the 30 single-annotated questions in each batch. The inter-annotator agreement is 0.62 before adjudication, measured by span-level Cohen's Kappa (Cohen, 1960).
2.3 Additional GitHub Data
To better understand the domain adaptability of our work, we further annotated theREADME files and issue reports from 143 randomly sampled repositories in the GitHub dump (Gousios and Spinellis, 2012) (from October 29, 2007 to December 31, 2017). We removed all the code blocks from the issue reports and README files collected from these 143 repositories. The resulting GitHub NER dataset consists of 6,510 sentences and 10,963 entities of 20 types labeled by two in-house annotators. The inter-annotator agreement of this dataset is 0.68, measured by span-level Cohen's Kappa.
2.4 StackOverflow/GitHub Tokenization
We designed a new tokenizer, SOTOKENIZER, specifically for the social computer programming domain. StackOverflow and GitHub posts exhibit
common features of web texts, including abbreviations, emoticons, URLs, ungrammatical sentences and spelling errors. We found that tokenization is non-trivial as many code-related tokens are mistakenly split by the existing web-text tokenizers, including the CMU Twokenizer (Gimpel et al., 2011), Stanford TweetTokenizer (Manning et al., 2014), and NLTK Twitter Tokenizer (Bird et al., 2009):
txScope.Complete()	[ 'txScope' ': ' Complete' ('')']
std::condition_variable	[ 'std' ': ': ' condition_variable']
math.h	[ 'math' ': ' h']
< span>	[ '(' 'span' ')'
a==b	[ 'a' ':=' ': 'b']
Therefore, we implemented a new tokenizer, using Twokenizer<sup>3</sup> as the starting point and added additional regular expression rules to avoid splitting code-related tokens.
3 Named Entity Recognition Models
The extraction of software-related named entities imposes significant challenges as it requires resolving a significant amount of unseen tokens, inherent polysemy, and salient reliance on context. Unlike news or biomedical data, spelling patterns and long-distance dependencies are more crucial in the software domain to resolve ambiguities and categorize unseen words. Taken in isolation, many tokens are highly ambiguous and can refer to either programming concepts or common English words, such as: 'go', 'react', 'spring', 'while', 'if', 'select'. To address these challenges, we design the SoftNER model that leverages sentential context to disambiguate and domain-specific character representations to handle rare words. Figure 2 shows the architecture of our model, which consists of primarily three components:
- An input embedding layer (§3.1) that extracts contextualized embeddings from the BERT<sub>base</sub> model and two new domain-specific embeddings for each word in the input sentence.
- A embedding attention layer (§3.2) that combines the three word embeddings using an attention network.
- A linear-CRF layer that predicts the entity type of each word using the attentive word representations from the previous layer.
3https://github.com/myleott/ark-twokenize-py
Figure 2: Our SoftNER model. It utilizes an attention network to combine the contextual word embeddings  $(\mathrm{BERT}_{\mathrm{base}})$  with the domain-specific embeddings (Code Recognizer and Entity Segmenter). The detailed structure of the attention network is depicted on the right.
3.1 Input Embeddings
For each word in the input sentence, we extract in-domain BERT (Devlin et al., 2019) representations and two new domain-specific embeddings produced by (i) a Code Recognizer, which represents if a word can be part of a code entity regardless of context; and (ii) an Entity Segmenter, that predicts whether a word is part of any named entity in the given sentence. Each domain-specific embedding is created by passing a binary value, predicted by a network independent from the SoftNER model. We describe the two standalone auxiliary models that generate these domain-based vectors below.
3.1.1 In-domain Word Embeddings
Texts in the software engineering domain contain programming language tokens, such as variable names or code segments, interspersed with natural language words. This makes input representations pre-trained on general book or Wikipedia texts unsuitable for software domain. We pre-trained different in-domain word embeddings, including BERT (BERTOverflow), ELMo (ELMoVerflow), and GloVe (GloVerflow) vectors on the Stack-Overflow 10-year archive of 152 million sentences and 2.3 billion tokens (\$3.3).
3.1.2 Context-independent Code Recognition
Humans with prior programming knowledge can easily recognize that 'list') is code, 'list' can be either code or a common English word, whereas listing' is more likely a non-code natural language token. We thus introduce a code recognition module to capture such prior probability of how
likely a word can be a code token without considering any contextual information. It is worth noting that this standalone code recognition model is also useful for language-and-code research, such as retrieving code snippets based on natural language queries (Iyer et al., 2016; Giorgi and Bader, 2018; Yao et al., 2019)
Our code recognition model (Code Recognizer) is a binary classifier. It utilizes language model features and spelling patterns to predict whether a word is a code entity. The input features include unigram word and 6-gram character probabilities from two language models (LMs) that are trained on the Gigaword corpus (Napoles et al., 2012) and all the code-snippets in the Stack-Overflow 10-year archive respectively. We also pre-trained FastText (Joulin et al., 2016) word embeddings using these code-snippets, where a word vector is represented as a sum of its character ngrams. We first transform each ngram probability into a  $k$ -dimensional vector using Gaussian binning (Maddela and Xu, 2018), which has shown to improve the performance of neural models using numeric features (Sil et al., 2017; Liu et al., 2016; Maddela and Xu, 2018). We then feed the vectorized features into a linear layer, concatenate the output with FastText character-level embeddings, and pass them through another hidden layer with sigmoid activation. We predict the token as a code-entity if the output probability is greater than 0.5. This binary prediction is then converted into a vector and used as an input to the SoftNER model.
3.1.3 Entity Segmentation
The segmentation task refers to identifying entity spans without assigning entity category. Entity segmentation is simpler and less error-prone
4https://archive.org/details/ stackexchange
than entity recognition as it does not require a fine-grained classification of the entity types. In fact, a segmentation model (Entity Segmenter) trained on our annotated StackOverflow corpus can achieve  $90.41\%$  precision on the dev set (details in  $\S 4.5$ ), predicting whether each token is a part of entity in the given sentence. Our segmentation model fine-tunes the in-domain BERT after concatenating it with two hand-crafted features:
- Word Frequency represents the word occurrence count in the training set. As many code tokens are defined by individual users, they occur much less frequently than normal English words. In fact, code and non-code tokens have an average frequency of 1.47 and 7.41 respectively in our corpus. Moreover, ambiguous token that can be either code or non-code entities, such as 'windows', have a much higher average frequency of 92.57. To leverage this observation, we include word frequency as a feature, converting the scalar value into a  $k$ -dimensional vector by Gaussian binning (Maddela and Xu, 2018).
- Code Markdown indicates whether the given token appears inside a <code> markdown tag in the StackOverflow post. It is worth noting that <code> tags are noisy as users do not always enclose inline code in a <code> tag or sometimes use the tag to highlight non-code texts (details in §2.1). Nevertheless, we find it helpful to include the markdown information as a feature as it improves the performance of our segmentation model.
The inclusion of hand-crafted features is influenced by Wu et al. (2018), where word-shapes and POS tags were shown to improve the performance of sequence tagging models.
3.2 Embedding-Level Attention
For each input word  $w_{i}$  in the input sentence, we have three embeddings: BERT  $(w_{i1})$ , Code Recognizer  $(w_{i2})$ , and Entity Segmenter  $(w_{i3})$ . We introduce the embedding-level attention  $\alpha_{it}$  ( $t \in \{1, 2, 3\}$ ), which captures each embedding's contribution towards the meaning of the word, to combine them together. To compute  $\alpha_{it}$ , we pass the input embeddings through a bidirectional GRU and generate their corresponding hidden representations  $h_{it} = \overleftarrow{\overrightarrow{GRU}}(w_{it})$ . These vectors are then passed through a non-linear layer, which outputs  $u_{it} = \tanh(W_{e}h_{it} + b_{e})$ . We introduce an
embedding-level context vector  $u_{e}$ , which is randomly initialized and updated during the training process. This context vector is combined with the hidden embedding representation using a softmax function to extract weight of the embeddings:  $\alpha_{it} = \frac{\exp(u_{it}^T u_e)}{\sum_t \exp(u_{it}^T u_e)}$ . Finally, we create the word vector by a weighted sum of all the information from different embeddings as  $word_{i} = \sum_{t} \alpha_{it} h_{it}$ . The aggregated word vector  $word_{i}$  is then fed into a linear-CRF layer, which predicts the entity category for each word based the BIO tagging schema.
We use PyTorch framework to implement our proposed SoftNER model and its two auxiliary components, namely code recognition and entity segmentation. The input to the SoftNER model include 850-dimensional vectors extracted from both the code recognizer and the entity segmenter.
We pre-trained  $\mathrm{BERT}_{\mathrm{base}}$ , ELMo and GloVe vectors on 152 million sentences from the Stack-Overflow, excluding sentences from the 1,237 posts in our annotated corpus. The pretraining of the 768-dimensional  $\mathrm{BERT}_{\mathrm{base}}$  model with 64,000 WordPiece vocabulary took 7 days on a Google TPU. The pre-training of 1024-dimensional ELMo vectors took 46 days on 3 NVIDIA Titan X Pascal GPUs. The pre-training of 300-dimensional GloVe embeddings (Pennington et al., 2014) with a frequency cut-off of 5 took 8 hours on a server with 32 CPU cores and 386 GB memory.
We train the SoftNER model and the two auxiliary models separately. Our segmentation model follows the simple BERT fine-tuning architecture except for the input, where BERT embeddings are concatenated with 100-dimensional code markdown and 10-dimensional word frequency features. We set the number of bins  $k$  to 10 for Gaussian vectorization. Our code recognition model is a feedforward network with two hidden layers and a single output node with sigmoid activation.
In this section, we show that our SoftNER model outperforms all the previous NER approaches on the StackOverflow and GitHub data. We also discuss the factors pivotal to the performance of our model, namely pre-trained in-domain BERT embeddings and two domain-specific auxiliary tasks.
P	R	F1
Test set
Feature-based CRF	71.77	39.70	51.12
BiLSTM-CRF (ELMoVerflow)	73.03	64.82	68.68
Attentive BiLSTM-CRF (ELMoVerflow)	78.22	78.59	78.41
Fine-tuned BERT	77.02	45.92	57.54
Fine-tuned BERTOverflow	68.77	67.47	68.12
SoftNER (BERTOverflow)	78.42	79.79	79.10
Dev set
Feature-based CRF	66.85	46.19	54.64
BiLSTM-CRF (ELMoVerflow)	74.44	68.71	71.46
Attentive BiLSTM-CRF (ELMoVerflow)	79.43	80.00	79.72
Fine-tuned BERT	79.57	46.42	58.64
Fine-tuned BERTOverflow	72.11	70.51	71.30
SoftNER (BERTOverflow)	78.81	81.72	80.24
Table 2: Evaluation on the dev and test sets of the StackOverflow NER corpus. Our SoftNER model outperforms the existing approaches.
We train and evaluate our SoftNER model on the StackOverflow NER corpus of 9,352 train, 2,942 development and 3,115 test sentences we constructed in §2. We use the same data for our segmentation model but replace all the entity tags with an I-ENTITY tag. For the code recognition model, we created a new lexicon of 6,000 unique tokens randomly sampled from the training set of the StackOverflow NER corpus. Each token was labelled independently without context as CODE, AMBIGUOUS or NON-CODE by two annotators with computer science background. The inter-annotator agreement was 0.89, measured by Cohen's Kappa. After discarding disagreements, we divided the remaining 5,312 tokens into 4,312 train and 1,000 test instances. Then, we merged AMBIGUOUS and NON-CODE categories to facilitate binary classification. We name this dataset of 5312 individual tokens as SOLEXICON.
We compare our model with the following baseline and state-of-the-art approaches:
- A Feature-based Linear CRF model which uses the standard orthographic, context and gazetteer features, along with the code markdown tags and handcrafted regular expressions to recognize code entities (details in Appendix A).
- A BiLSTM-CRF model with in-domain ELMo embeddings (ELMoVerflow; details in §3.3). This architecture is used as the state-of-the-art baseline named-entity recognition
models in various domains (Lample et al., 2016; Kulkarni et al., 2018; Dai et al., 2019).
- An Attentive BiLSTM-CRF model with in-domain ELMo embeddings as well as domain-specific embeddings from the code recognizer and the entity segmenter. This model combines these three word embeddings using an attention network and then utilizes a BiLSTM-CRF layer to predict the entity type of each input word (details in Appendix B).
- A Fine-tuned out-of-domain BERT model where we fine-tune the original  $\mathrm{BERT}_{\text {base }}$  based on our annotated corpus.
- A Fine-tuned in-domain BERT model where we fine-tune the in-domain pre-trained  $\mathrm{BERT}_{\text {base }}$  (BERTOverflow; details in §3.3) cased checkpoint on our annotated corpus.
Table 2 shows the precision (P), recall (R) and  $\mathbf{F}_1$  score comparison of different models evaluated on the StackOverflow NER corpus. Our SoftNER model outperforms the existing NER approaches in all the three metrics. Fine-tuning over in-domain trained BERT (BERTOverflow), in particular, improves  $\mathrm{F_1}$  score by more than 10 points in comparison to using the original BERT.
4.4 In-domain vs. out-of-domain Word Embeddings
Table 3 shows the performance comparison between in-domain and out-of-domain word embeddings. We consider off-the-shelf BERT (Devlin et al., 2019), ELMo (Peters et al., 2018) and GloVe (Pennington et al., 2014) vectors trained on newswire and web texts as out-of-domain embeddings. When using the BiLSTM-CRF model (Lample et al., 2016; Kulkarni et al., 2018; Dai et al., 2019), we observe a large increase of  $13.64\mathrm{F_1}$  score when employing in-domain ELMo (ELMoVerflow) representations over indomain GloVe (GloVeOverflow), and an increase of  $15.71\mathrm{F_1}$  score over out-of-domain ELMo. We found that fine-tuning out-of-domain BERT (Devlin et al., 2019) outperforms the out-of-domain
$^{5}$ https://github.com/google-research/BERT
$^{6}$ https://github.com/lanwuwei/BERTOverflow/