Venue: NAACL 2019

# Train One Get One Free: Partially Supervised Neural Network for

Bug Report Duplicate Detection and Clustering

Lahari Poddar\({}^{1,2}\)

Luis Marujo\({}^{2}\)&Leonardo Neves\({}^{2}\)

Sergey Tulyakov\({}^{2}\) &William Brendel\({}^{2}\)

**Pradeep Karuturi\({}^{2}\)**

###### Abstract

Tracking user reported bugs requires considerable engineering effort in going through many repetitive reports and assigning them to the correct teams. This paper proposes a neural architecture that can jointly (1) detect if two bug reports are duplicates, and (2) aggregate them into latent topics. Leveraging the assumption that learning the topic of a bug is a sub-task for detecting duplicates, we design a loss function that can jointly perform both tasks but needs supervision for only duplicate classification, achieving topic clustering in an unsupervised fashion. We use a two-step attention module that uses self-attention for topic clustering and conditional attention for duplicate detection. We study the characteristics of two types of real world datasets that have been marked for duplicate bugs by engineers and by non-technical annotators. The results demonstrate that our model not only can outperform state-of-the-art methods for duplicate classification on both cases, but can also learn meaningful latent clusters without additional supervision.

School of Computing, National University of Singapore

lahari@comp.nus.edu.sg

Snap Research

{lneves, willb, luis.marujo}@snap.com

{stulyakov, pradeep.karuturi}@snap.com

## 1 Introduction

User feedback is a key part of the development and improvement of software products. Each piece of feedback needs to be manually reviewed and assigned to the correct engineer responsible for maintaining the feature mentioned in the report. On online platforms with millions of users, different users tend to report the same issue, yielding a large number of duplicate reports. Sorting through these massive volumes of bug reports incur a significant amount of engineering time and cost. Additionally, these services are constantly releasing new product features. Therefore, we cannot rely on static annotated data for product feature classification because they rapidly become outdated. This motivates us to develop a framework that can automatically identify duplicate bug reports and cluster them without requiring additional labels.

Previous research in the software engineering domain has addressed duplicates detection and product feature identification as two separate problems framed as independent fully-supervised classification tasks Nguyen et al. (2012); Budhiraja et al. (2018); Jonsson et al. (2016); Mani et al. (2018). However, we observe that users generally report issues when accessing certain features of a product, e.g., '_app crashed when opening camera_', '_that won't load_' and so on (here, camera and chat are the product features, respectively). Hence, two reports should _at least_ discuss the same feature to be considered as duplicates. Therefore, we hypothesize that determining the feature discussed in a report is a _sub-task_ of detecting whether or not a report is a duplicate to another one.

Inspired by the effectiveness of Siamese architectures for modeling pairs of texts Tan et al. (2015); Bowman et al. (2015), we use a shared Recurrent Neural Network (RNN) to encode the two reports. We note that the latent vectors, learned by an RNN for a sequence of words, encode a multitude of semantic information; all of which may not be necessary to understand the topic of the report. We decompose the latent semantic vectors in order to distill only the topical information in a few designated dimensions. This allows us to perform the sub-task of feature-based clustering using only a subset of dimensions, and use the complete vector for the duplicate classification task. We propose a partially supervised learning framework that uses the label for duplicity through a similarity loss on the designated topic dimensions of the latent representation to learn topic clusters.

We use a two-step attention module, since the same words are often not crucial for both tasks. We first learn a self-attention for topic similarity

modeling and learn a conditional attention using a memory vector for duplicate classification.

To summarize, we present a systematic study of a classic problem in the software industry. The work has three major contributions.

* We propose a neural model for multi-task learning that requires supervision for only one of the tasks.
* The model uses semantic space decomposition and a hierarchical-conditional attention module to perform the tasks of duplicate detection and topic based clustering.
* We present the challenges we faced during our experience obtaining labels from non-technical annotators, and conduct extensive experiments on both engineer labeled and non-technical labeled datasets.

## 2 Related Work

The software engineering community has conducted some research work for detecting duplicate bugs. In Minh (2014), a combination of n-gram features and cluster shrinkage algorithm is used for duplicate classification, whereas, in Sun et al. (2011) the BM-25 based scoring has been used akin to information retrieval engines. A combination of tf-idf and topics learned by LDA have also been used Nguyen et al. (2012); Budhiraja et al. (2018b). Recently, word embeddings have been used to compute similarity of two reportsYang et al. (2016); Budhiraja et al. (2018a). However, in these approaches, the sequence information of a natural language sentence is not captured.

Among the NLP community, the closest line of applicable work are the generic approaches of textual similarity Cer et al. (2017); Wang et al. (2017); Neculou et al. (2016); Yin et al. (2016). They predominantly employ Siamese architecture Mueller and Thyagarajan (2016); Severyn and Moschitti (2015) and more recently, attention mechanism Wang et al. (2017); Shen et al. (2018); Wang et al. (2018); Tran and Niederee (2018). Inspired by these approaches, we propose a solution for duplicate detection while utilizing the partial supervision to achieve a sub-task of clustering _for free_.

Solving owner attribution of a report has been approached using feature-based methods Jonsson et al. (2016); Xuan et al. (2012) and very recently using deep learning solutions Mani et al. (2018). However, the connection between these two problems have not been explored and they require supervision for product feature identification.

## 3 Annotation Challenges

We consider a dataset consisting of user reported bugs collected from the Snapchat app. The bugs have been submitted by beta-testers using a bugtracking functionality named Shake2Report(S2R) within the app. In S2R a user can submit a small textual description of the bug and attach a screenshot of the app while experiencing the issue. In this work, we only consider the textual descriptions of the reports.

Following previous work of duplicate bug tracking Lazar et al. (2014), we first studied \(500\) pairs of reports marked as duplicates by engineers. However, these pose several challenges. Firstly, we realized that engineers often needed additional meta-data such as stack-traces and timestamps to accurately determine whether or not a pair of reports referred to the same problem. On the other hand, such additional meta-data cannot be made available to external crowd-sourced annotators due to legal and user privacy restrictions. Finally, the total amount of reports was far beyond what an engineering team could tackle, hence the need for our classification system, which helps scaling the bug report load while maintaining the annotation quality.

Our task is to automatically classify report pairs, and to add engineers to the loop only when pairs are textually ambiguous. We asked non-expert annotators to label the pairs only according to their semantic similarity (no metadata). The annotators had a high agreement among themselves with a Krippendorff's alpha Krippendorff (1970) score of \(0.78\). After adding engineers to the annotation pool, the score dropped to \(0.58\). To understand this disparity, consider the pairs:

* _"Crashed when posting story from memories"_ and _"Crashed while exporting from memories"_
* _"Stuck on typing notification"_ and _"Did not get notification for this group chat"_ Although not marked as duplicates by non-experts, they referred to the same underlying problem from the engineering side. While out-of-scope for this paper, we plan to conduct a dedicated study to explore these differences in future work.

As mentioned in the introduction, bug \(\blacktriangleright\)team assignments become rapidly outdated as the app 

product features (and therefore the teams working on them) are constantly evolving. Also, non-expert annotators cannot be assumed to have any knowledge of these assignments, so we treat bug\({}^{*}\) team assignments as an unsupervised task.

## 4 Proposed Method

We now describe the proposed learning framework as shown in Figure 1. Given a pair of reports \(P\) and \(Q\), the system uses a single binary label \(r(P,Q)\) (1 if the reports are duplicate, 0 otherwise) to supervise both tasks: duplicate classification and topic similarity modeling.

### Text Encoder

This component takes a bug report represented as a sequence of words \(\{w_{1},w_{2},\cdots,w_{n}\}\) as input and encodes it to latent vectors \(\{\mathbf{h}_{1},\mathbf{h}_{2},\cdots,\mathbf{h}_{n}\}\).

We first use word embeddings to transform all words in a text into finite \(d\)-dimensional vectors \(\{\mathbf{v}_{1},\mathbf{v}_{2},\cdots,\mathbf{v}_{n}\}\). The vectors are then fed to a Gated Recurrent Unit (GRU) layer Chung et al. (2014). We use bi-directional GRU units to encode the context information around a word. For a word \(w_{i}\), outputs from the forward and backward GRUs are latent vectors of dimension \(g\) and denoted as \(\mathbf{h}_{i}^{f}\) and \(\mathbf{h}_{i}^{b}\), respectively. We concatenate these two vectors to form a single latent representation of the word as, \(\mathbf{h}_{i}=\mathbf{h}_{i}^{f}\oplus\mathbf{h}_{i}^{b}\) .

### Semantic Space Decomposition

A word's encoded representation \(\mathbf{h}_{i}\) has various information intertwined in an abstract way within the vector dimensions. It is difficult to interpret individual dimensions and meaningfully use a subset of dimensions for a different sub-task.

We aim to distill the coarser topical information into designated dimensions within the vector, storing other finer pieces of information in remaining dimensions. Such disentanglement allows us to do the topic similarity modeling using only the designated dimensions and ignore the rest.

To this end, we force the network to learn topical information about each word in the first \(k\) dimensions of its latent vector space. For a bi-directional GRU, we have two encoded representations for the \(i^{th}\) word from the forward and backward passes i.e., \(\mathbf{h}_{i}^{f}\), and \(\mathbf{h}_{i}^{b}\). We define the topic vector of a word \(i\) as,

\[\boldsymbol{\theta}_{i}=\mathbf{h}_{i}^{f}[1:k]\oplus\mathbf{h}_{i}^{b}[1:k] \tag{1}\]

where \(\mathbf{h}_{i}^{f}[1:k]\) and \(\mathbf{h}_{i}^{b}[1:k]\) denote the first \(k\) dimensions in the GRU encoded latent vectors from the forward and backward passes respectively.

### Topic Similarity Modeling

We aggregate the topic vectors of the constituent words to represent the topic of a report, and use a self-attention layer to learn the weights of the words important in determining the topic:

\[\mathbf{z}_{i}=\tanh(\textbf{W}\cdot\mathbf{h}_{i}+\mathbf{b}_{ i}) \tag{2}\] \[\boldsymbol{\alpha}_{i}=\frac{\exp(\mathbf{z}_{i})}{\sum_{i}\exp (\mathbf{z}_{i})},\] (3) \[\boldsymbol{\theta}=\sum_{i=1}^{n}\boldsymbol{\alpha}_{i}\cdot \boldsymbol{\theta}_{i}. \tag{4}\]

where \(\boldsymbol{\alpha}_{i}\) is the weight for word \(w_{i}\) and \(\boldsymbol{\theta}\) is the topic vector of the report. Note that only the designated topical dimensions of the words contribute to representing the topic vector of the report.

In order to learn the semantic space of topic vectors, we need to constrain them in such a way that if two reports (\(P\) and \(Q\)) have similar topics (e.g., both are talking about _camera_), their \(\boldsymbol{\theta}^{P}\) and \(\boldsymbol{\theta}^{Q}\) values should be closer and vice-versa. Since only the ground-truth label for duplicity is available, we consider this signal as _partially_ observed for the topic modeling task. Following our hypothesis that determining the topic of a report is a sub-task for duplicate detection, there can be three possible cases for a pair of reports:

**Case 1:** Duplicates from the same topic;

**Case 2:** Non-duplicates from different topics;

**Case 3:** Non-duplicates from the same topic.

Since for duplicate tickets (\(P,Q\)) their topics are bound to be same, we reduce the distance between \(\boldsymbol{\theta}^{P}\), and \(\boldsymbol{\theta}^{Q}\) in our loss. However, non-duplicate pairs may or may not be from the same topic (Case 2, and 3).

We do not include Case 3 in the loss, as inferring it is difficult without explicit labels. We assume that if a pair of non-duplicate reports have no word overlap (apart from stopwords), then only they belong to different topics i.e., Case 2. For such pairs we wish to increase the distances between their topic vectors. We imbuic this principle in the network through the following loss function:

\[\mathcal{L}_{\mathrm{sim}} =r(P,Q)\cdot S_{\mathrm{C}}(\boldsymbol{\theta}^{P},\boldsymbol{ \theta}^{Q})\] (5) \[-(1-r(P,Q))\cdot S_{\mathrm{C}}(\boldsymbol{\theta}^{P}, \boldsymbol{\theta}^{Q})\] 

where \(r(P,Q)\) is the ground-truth duplicity label (0 or 1), \(S_{\mathrm{C}}\) is cosine similarity. A careful reader might observe that the function minimizes distance of duplicates pairs (\(r(P,Q)=1\)), while exhibiting the exact opposite behaviour for non-duplicates (\(r(P,Q)=0\)). To account for the skewed distribution of duplicate vs. non-duplicates, we normalize the loss using proportional class weights.

### Duplicate Classification

The final component of the network considers the complete latent vectors of the words for the task of duplicate classification. We use another attention layer for this component to allow the network to focus on important words. However, it is not necessary that the same words will be important for both the tasks. For example, consider the following pair of tickets: _"Can't import these pics from camera roll to memories"_ and _"No pics in memory"_. In order to determine the product feature, the self attention module needs to focus on the words _memories_ and _memory_ in the reports, respectively. However, to decide whether they are duplicates or not, the second attention module needs to focus on the specific error, i.e. _Can't import pics_ and _No pics_ in the two reports respectively. For report \(P\), we create a memory vector \(\mathbf{\phi}^{P}\) to guide this attention layer using the hidden representations of the words as well the previously learned self-attention weights for topic modeling.

\[\mathbf{\phi}^{P}=\sum_{i}\mathbf{\alpha}_{i}^{P}\cdot\mathbf{h}_{i}^{P}+\sum_{i} \mathbf{h}_{i}^{P} \tag{6}\]

We also note that unlike the self-attention used for topic similarity modeling, for duplicate classification we need to use a form of conditional attention. For determining whether a report (\(P\)) is duplicate to another one (\(Q\)), the words that are important in \(P\) are dependent on the words in \(Q\). Therefore, for each word position \(i\) in \(P\) we compute its relevance to the report \(Q\) as,

\[\mathbf{\gamma}_{i}^{P}=\mathbf{h}_{i}^{P}\odot\mathbf{h}^{Q} \tag{7}\]

where \(\mathbf{h}_{i}^{P}\) is the hidden state of the \(i^{th}\) word in report \(P\), \(\odot\) denotes element-wise multiplication, and \(\mathbf{h}^{Q}\) is the representation of report \(Q\) obtained using averaging the hidden states of all words in \(Q\) i.e. \(\mathbf{h}^{Q}=\operatorname*{Avg}(\mathbf{h}_{1},\mathbf{h}_{2},\cdots, \mathbf{h}_{m})\), \(m\) is the number of words in \(Q\).

We now use this conditional representation \(\mathbf{\gamma}_{i}^{P}\) and the memory vector \(\mathbf{\phi}^{P}\) to learn the conditional attention weights (\(\mathbf{\beta}\)) and compute the weighted representation \(\mathbf{c}^{P}\) as,

\[\mathbf{c}_{i}^{P} =\tanh(\mathbf{W}_{1}\cdot\mathbf{\gamma}_{i}^{P})\cdot\tanh( \mathbf{W}_{2}\cdot\mathbf{\phi}^{P}) \tag{8}\] \[\mathbf{\beta}_{i}^{P} =\frac{\exp(\mathbf{c}_{i}^{P})}{\sum_{i}\exp(\mathbf{c}_{i}^{ P})}\] (9) \[\mathbf{c}^{P} =\sum_{i=1}^{n}\mathbf{\beta}_{i}^{P}\cdot\mathbf{c}_{i}^{P} \tag{10}\]

Figure 1: **The proposed partially supervised learning framework. The framework takes two user reports \(P\) and \(Q\) as inputs. The left branch performs topic clustering with self-attention mechanism using topic dimensions. The right branch performs duplicates detection with conditional attention taking a pair of encoded reports as inputs.**



We concatenate the weighted representations of \(P\) and \(Q\) as \({\bf c}^{PQ}={\bf c}^{P}\oplus{\bf c}^{Q}\). We pass this concatenated vector through a Multi-Layer Perceptron, the final layer of which outputs a prediction \(\hat{r}(P,Q)\) whether they are duplicates or not. We use binary cross-entropy loss to train this objective.

\[\mathcal{L}_{\rm dup}=H(r(PQ),\hat{r}(PQ)), \tag{11}\]

where \(H\) is binary cross-entropy. The overall loss for the whole network is a weighted combination of the two losses.

\[\mathcal{L}=\lambda\mathcal{L}_{\rm sim}+(1-\lambda)\mathcal{L}_{\rmdup} \tag{12}\]

In our experiments, we use \(\lambda=0.5\) but it can be varied depending on the importance of the tasks. We optimize the network using Adam optimizer and train in an end-to-end fashion through back-propagation.

## 5 Evaluation

### Dataset

We experiment on four real-world datasets.

**(1) Snap S2R**: Composed of bugs reported through an in-app tool. The data was labeled as duplicate or not by non-technical annotators (described in section 3).

**(2-4). Open Source Projects**1 : Repository of bugs submitted for the open-source projects of **Eclipse Platform**, **Mozilla Firefox** and **Eclipse JDT**. Each report consists of a short title and a longer description which often describes the steps to reproduce the issue. We only consider the title of the report in our experiments. The reported bugs have been marked for duplicates by engineers while resolving them.

Footnote 1: [https://github.com/logpai/bugrepo](https://github.com/logpai/bugrepo)

We augment the S2R data with randomly sampled negative pairs to resemble a positive to negative ratio that is estimated in production. For the Eclipse and Firefox datasets, we randomly sample negative pairs, keeping the positive to negative ratio close to what was previously mentioned Lazar et al. (2014). Table 1 shows the statistics of the four datasets. These datasets were selected to capture different training sizes and variations of vocabulary used.

### Experimental Settings

We initialize the words using pre-trained Glove embeddings Pennington et al. (2014) of 300 dimensions but tune it during training to capture the intrinsic features of the specific task and dataset at hand. For the GRU layers we use 150 dimensions and the first 20 dimensions of them are used for topic representation. For the MLP in duplicate classification we use \(2\) layers of fully connected layers of \(100\) hidden neurons each, with \(relu\) activation and \(20\%\) dropout rate. For fair comparison we use the same number of parameters in all baselines. The learning rate is set to \(0.003\) and a batch size of \(128\) samples is used while training all the models.

### Evaluating Duplicate Classification

We first start evaluating the proposed approach for detecting duplicate bug reports. We compare with the following six baselines:

* **Logistic Regression** uses a bag of \(n\)-grams representation with \(n\) ranging from one to three words. Tf-idf scores are used for feature weighting.
* **Siamese CNN**Severyn and Moschitti (2015) encodes the texts with a shared convolution network followed by an MLP for classification.
* **Siamese Bi-GRU** uses a shared bidirectional GRU for encoding text.
* **Siamese Bi-GRU with Attention** uses an attention layer on top of the GRU outputs to encode the texts
* **DWEN**Budhiraja et al. (2018a) is the state-of-the-art deep learning approach for duplicate bug detection using word embeddings.
* **BiMPM**Wang et al. (2017) is the state-of-the-art sentence similarity modeling that uses multi-perspective symmetric matching for a pair of texts.

We report results after averaging five independent trials using \(80\%\) data for training, \(10\%\) for

\begin{table}
\begin{tabular}{l|c c c c c} \hline \multirow{2}{*}{Dataset} & \multirow{2}{*}{\#pairs} & \multirow{2}{*}{\#vocab} & \multirow{2}{*}{\#dops} & avg \#words \\  & & & & \(k\)report \\ \hline Snap S2R & 66,945 & 7763 & 9\% & 17,255 & 14.6 \\ Eclipse Platform & 170,312 & 29,702 & 12\% & 83,608 & 7.9 \\ Eclipse JDT & 90,592 & 17,228 & 14\% & 44,670 & 8.1 \\ Firefox & 231,628 & 34,590 & 26\% & 113,262 & 10.0 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of the datasets used 

validation, and \(10\%\) for testing. Due to the imbalanced class distribution, we use precision, recall and F1-score of the positive class as evaluation metrics.

From the results in Table 2 we see that our proposed method largely outperforms all other models in terms of recall and in most cases on F1 score as well. Using a sequence encoder like GRU boosts the performance significantly compared to using n-grams as in Logistic Regression or CNN. We note that the overall scores on Eclipse and Firefox datasets are much higher than the scores on Snap S2R data. One of the reasons could be the difference in data size where the Snap S2R dataset is much smaller in size compared to others. Noisy labels from non-technical annotators could also be a potential reason. Manually examining the posts we also note that the bug reports submitted for the open projects are often by engineers, who use concrete technical terms to describe the problem. In contrast, for the Snap S2R dataset, the reports are submitted by end-users using free text with non-technical terms that make it harder for a machine learning model to disambiguate.

### Evaluating Clusters

A major advantage of our model is its ability to perform clustering of the bug reports to aid in ownership attribution. In this set of experiments, we show that our model is able to learn a semantic space so that nearby reports in the space indeed belong to the same product feature.

We use the topic vector (\(\theta\)) of a report learned by our model and run an off-the-shelf clustering algorithm K-means2. Ideally, there should be a one-to-one mapping between the obtained clusters and the actual project features. For comparison, we use one of the most popular unsupervised topic model, LDA [4] to learn latent topics from the bug reports. For implementation of LDA we use the Gensim library3. We perform standard pre-processing steps, and remove stopwords before training the LDA model. We evaluate the learned latent clusters by comparing them to the ground truth project features, as have been marked by users while submitting reports for S2R dataset. For fair comparison, we use the same (\(20\)) number of clusters for both methods, which is close to the actual number of product features.

Footnote 3: [https://radimrehurek.com/gensim/models/ldamodel.html](https://radimrehurek.com/gensim/models/ldamodel.html)

Figure 2: Mapping between the learned latent clusters and top-\(5\) product features for Snap S2R dataset.

 

Figure 2 shows the mapping between learned latent clusters and top-\(5\) ground truth product features for the proposed model and LDA. To reduce the effect of noisy labels, we consider a cluster-product feature mapping only if more than \(10\) reports about a product feature are assigned to a cluster. The width of the connecting edge is proportional to the number of times a cluster-product feature association is observed.

Firstly, we observe that LDA requires more clusters to represent the same number of features. This demonstrates that LDA over-clusters, failing to identify reports that talk about the same product feature and assigns them to different clusters. Secondly, some clusters in LDA (e.g., \(c.13\)) are connected to multiple product features denoting impurity of the cluster. Finally, for most features, their mappings to a cluster are much stronger in our model as demonstrated by the width of the edges in the graph. This shows that with partial supervision from duplicity labels, we are able to learn clusters that better correspond to ground truth product features.

Table 3 shows top words according to their tf-idf scores appearing in bug reports assigned to a cluster. The learned clusters are coherent and represent specific product features. For example, _Cluster 1_ talks about issues regarding video or audio, while _Cluster 2_ is mostly about issues related to loading of a story (a feature in the app). In _Cluster 3_ reports related to various lenses and filters are placed, while _Cluster 4_ seems to be about maps and navigation features. It is also interesting to note that generic issues like _'crashed'_ appear in some of the clusters. Although one could identify most of the bugs as crashes, this indicates that users have a specific vocabulary when referring to a particular product feature.

### Analyzing Attention Weights

Finally, we present case studies of the attention weights learned by our proposed model. As we have two-steps of attention layers optimized by the two objectives, they can learn to focus on words that are important for the specific tasks.

Figure 3 shows the attention weights for two sample bug reports from Snap S2R dataset. We observe that the two attention modules tend to focus on different parts of the text. The self-attention gives more weight to the words that determine the overall topic of the report (_tile_ in Figure 2(a), _specs_ in Figure 2(b)). In contrast, the conditional-attention focuses on the set of words describing the specific issue to aid in detecting duplicates.

## 6 Conclusion

In this paper we have studied bug-tracking, which is a widespread problem in the software industry. We develop a neural architecture that can learn to classify duplicates and cluster them into meaningful latent topics without additional supervision. The architecture decomposes the latent semantic space of a word to only distill the topical information into a few designated dimensions, and uses a two-step attention module to focus on different textual parts for the two tasks. We share the challenges of annotating a user reported bug dataset with non-technical annotators, as opposed to using annotations from engineers. This is a direction we plan to further explore in future work. Experimental results on different types of datasets indicate that the proposed approach is promising compared to existing techniques for both tasks. Most importantly, our model's construction is generic and presents new possibilities in various domains for modeling sub-tasks for free, with partial supervision from another task.

Figure 3: Visualization of the attention weights learned by the two attention modules on few sample bug reports

\begin{table}
\begin{tabular}{|l|p{79.8pt}|p{79.8pt}|p{79.8pt}|p{79.8pt}|} \hline Cluster 1 & Cluster 2 & Cluster 3 & Cluster 4 & Cluster 5 \\ \hline video & load & lens & app & specs \\ audio & long & lenses & map & spec \\ videos & story & carousel & viewing & importing \\ playback sync & time & preview & crashed & v2 \\ sync & taking & applied & navigating & unpaired \\ \hline \end{tabular}
\end{table}
Table 3: Top Words according to their tf-idf scores from few clusters learned by our model.

 



## References

* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _Journal of machine Learning research_, 3(Jan):993-1022.
* Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. _Conference on Empirical Methods in Natural Language Processing_.
* Budhiraja et al. (2018a) Amar Budhiraja, Kartik Dutta, Raghu Reddy, and Manish Shrivastava. 2018a. Dwen: deep word embedding network for duplicate bug report detection in software repositories. In _Proceedings of the 40th International Conference on Software Engineering: Companion Proceedings_, pages 193-194. ACM.
* Budhiraja et al. (2018b) Amar Budhiraja, Raghu Reddy, and Manish Shrivastava. 2018b. Lwse: Lda refined word embeddings for duplicate bug report detection. In _Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings_, pages 165-166. ACM.
* Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. _arXiv preprint arXiv:1708.00055_.
* Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In _NIPS Workshop on Deep Learning and Representation Learningp, December_.
* Jonsson et al. (2016) Leif Jonsson, Markus Borg, David Broman, Kristian Sandahl, Sigrid Eldh, and Per Runeson. 2016. Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts. _Empirical Software Engineering_, 21(4):1533-1578.
* Krippendorff (1970) Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. _Educational and Psychological Measurement_, 30(1):61-70.
* Lazar et al. (2014) Alina Lazar, Sarah Ritchey, and Bonita Sharif. 2014. Generating duplicate bug datasets. In _Proceedings of the 11th working conference on mining software repositories_, pages 392-395. ACM.
* Mani et al. (2018) Senthil Mani, Anush Sankaran, and Rahul Aralikatte. 2018. Deeptrage: Exploring the effectiveness of deep learning for bug triaging. _arXiv preprint arXiv:1801.01275_.
* Minh (2014) Phuc Nhan Minh. 2014. An approach to detecting duplicate bug reports using n-gram features and cluster shrinkage technique. _Int. J. Sci. Res. Publ.(IJSRP)_, 4(5):89-100.
* Mueller and Thyagarajan (2016) Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In _AAAI_, volume 16, pages 2786-2792.
* Neculoiu et al. (2016) Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru. 2016. Learning text similarity with siamese recurrent networks. In _Proceedings of the 1st Workshop on Representation Learning for NLP_, pages 148-157.
* Nguyen et al. (2012) Anh Tuan Nguyen, Tung Thanh Nguyen, Tien N Nguyen, David Lo, and Chengnian Sun. 2012. Duplicate bug report detection with a combination of information retrieval and topic modeling. In _Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering_, pages 70-79. ACM.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Severyn and Moschitti (2015) Aliaksein Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In _Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval_, pages 373-382. ACM.
* Shen et al. (2018) Chenlin Shen, Changlong Sun, Jingjing Wang, Yangyang Kang, Shoushan Li, Xiaozhong Liu, Luo Si, Min Zhang, and Guodong Zhou. 2018. Sentiment classification towards question-answering with hierarchical matching network. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3654-3663.
* Sun et al. (2011) Chengnan Sun, David Lo, Siau-Cheng Kroo, and Jing Jiang. 2011. Towards more accurate retrieval of duplicate bug reports. In _Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering_, pages 253-262. IEEE Computer Society.
* Tan et al. (2015) Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Lstm-based deep learning models for non-factoid answer selection. _arXiv preprint arXiv:1511.04108_.
* Tran and Niederee (2018) Nam Khanh Tran and Claudia Niederee. 2018. Multihop attention networks for question answer matching. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 325-334. ACM.
* Wang et al. (2018) Lu Wang, Shoushan Li, Changlong Sun, Luo Si, Xiaozhong Liu, Min Zhang, and Guodong Zhou. 2018. One vs. many qa matching with both word-level and sentence-level attention network. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2540-2550.
* Wang et al. (2017) Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. _IJCAI_

Jifeng Xuan, He Jiang, Zhilei Ren, and Weiqin Zou. 2012. Developer prioritization in bug repositories. In _Software Engineering (ICSE), 2012 34th International Conference on_, pages 25-35. IEEE.
* Yang et al. [2016] Xinli Yang, David Lo, Xin Xia, Lingfeng Bao, and Jianling Sun. 2016. Combining word embedding with information retrieval to recommend similar bug reports. In _2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)_, pages 127-137. IEEE.
* Yin et al. [2016] Wenpeng Yin, Hinrich Schutze, Bing Xiang, and Bowen Zhou. 2016. Abcon: Attention-based convolutional neural network for modeling sentence pairs. _Transactions of the Association of Computational Linguistics_, 4(1):259-272.