# ML-Augmented Automation for Recovering Links Between Pull-Requests and Issues on GitHub

Zakarea Alshara, Hamzeh Eyal-Salman, Anas Shatnawi, Abdelhak-Djamel

Seriai

► To cite this version:

Zakarea Alshara, Hamzeh Eyal-Salman, Anas Shatnawi, Abdelhak-Djamel Seriai. ML-Augmented Automation for Recovering Links Between Pull-Requests and Issues on GitHub. IEEE Access, 2023, 11, pp.5596-5608. 10.1109/ACCESS.2023.3236392. hal-03980366

HAL Id: hal-03980366

https://hal.science/hal-03980366

Submitted on 9 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# ML-Augmented Automation for Recovering Links between Pull-Requests and Issues on GitHub

ZAKAREA ALSHARA<fnref n="1" />, HAMZEH EYAL SALMAN<fnref n="2" />, ANAS SHATNAWI<fnref n="3" />, ABDELHAK-DJAMEL SERIAI<fnref n="4" />

Corresponding author: Zakarea Alshara (e-mail: zmalshara@just.edu.jo).

**ABSTRACT** GitHub provides a distributed and collaborative platform to develop and maintain open-source projects. This social coding platform achieves this collaborative development, with or without coordination, using pull requests and issues artefacts. When the number of daily submitted issues rapidly grows up, especially in popular repositories, managing issues becomes more complicated. To help the repository's developers in issues processing, there are external contributors who fix issues by submitting pull-requests. On GitHub, a pull-request is frequently linked with a submitted issue to show that a solution is in progress. Unfortunately, contributors might forget or be lazy to link the Pull-Requests with their corresponding Issues. Only a tiny share of these links are established, but a large portion of links are missed in the development history. However, manually recovering the links between Pull-Request and Issues from evolutionary development history is a challenging, time-consuming, and error-prone task, even for senior developers. In this article, we propose to build ML models to recover links between pull-requests and issues using two Machine Learning algorithms (KMeans and BIRCH) based on lexical and semantic weighting measurements. These models are evaluated using *PI-Link* ground-truth dataset. The obtained results show that pull-request and issue links can be recovered with an accuracy of 91.5% using BIRCH clustering algorithm.

**INDEX TERMS** clustering, deep learning, GitHub, issue, link, machine learning, pull-request, recovering, similarity.

## I. INTRODUCTION

In social coding platforms, such as GitHub, when a contributor finds an Issue, he informs other contributors of the hosted repository on GitHub via the issue tracking system [1]. An Issue is either a bug, feature enhancement, new feature, or any maintenance task. On GitHub, the integrated Issues tracking system allows the Issue submitter to provide textual information about the detected Issue containing a title and description. After reviewing and assigning Issues to internal developers (responsible developers), they are required to promptly respond to Issues identified in Issue trackers and resolve such Issues with the least amount of work. When the number of daily submitted Issues rapidly grows up, especially in popular repositories, managing Issues becomes more complicated, adding to developers heavier workloads [2].

To help the internal developers in Issues processing, there are external contributors or volunteers who contribute to fix Issues by submitting pull-requests (PRs). Contributors clone (fork) a hosted repository on GitHub and make their changes locally on the cloned repository [3]. The changes include development tasks (such as adding new functional features), resolving Issues, or making enhancements (in terms of usability, reliability, performance, and so on). Then, they create PRs to notify the repository owners about these changes and merge them into the original repository after reviewing the submitted PRs [4]. Each PR has a title and body (description) to explicitly clarify the goal of the PR submitter.

On GitHub, a PR should be linked with a submitted open Issue to show that a solution is in progress [5]. When the PR is ready to merge, the open Issue is closed automatically.

<footnote n="1">¹Department of Software Engineering, Jordan University of Science and Technology, P.O.Box 3030, Irbid 22110, Jordan (e-mail: zmalshara@just.edu.jo)</footnote>

<footnote n="2">²Department of Software Engineering, Mutah University, Al-Karak 61710, Jordan (e-mail: hamzehmu@mutah.edu.jo)</footnote>

<footnote n="3">³DRIT, Berger-Levrault, Montpellier, France (e-mail: anas.shatnawi@berger-levrault.com)</footnote>

<footnote n="4">⁴LIRMM Lab, University of Montpellier, 34000 Montpellier, France (e-mail: seriai@lirmm.fr)</footnote>
The nature of the parallel and distributed process of PR mechanism allows many contributors to submit separate PRs to tackle the same Issue. As a result, the same Issue may be linked to one or more PRs where each submitted PR represents a proposed solution. Therefore, many PRs include in their titles and descriptions the linked Issue identifier (Issue number).

The established links between Issues and their corresponding PRs are valuable, as such links can be exploited to learn from previous Issues/PRs to predict potential future solutions. Also, such links are valuable assets from the development and maintenance point of view where they have different semantics. For example, these are used to link features with their corresponding implementations at different levels of abstraction [6]–[8]. In bug localization, they are used for training to build prediction models [9]–[11]. In Issues assignments, they are used to build a recommender system to propose relevant reviewers for upcoming Issues [2], [12], [13].

In GitHub, developers (or contributors) typically create PR-Issue links in a manual way. They should specify in PR title or description the Issue identifier, especially in large projects. Unfortunately, in such a way, contributors may forget or sometimes are busy to link PR and Issue as mentioned above. As a result, a significant number of links are missed while only a small fraction of PR-Issue links are established. At the same time, manually recovering such links between PR-Issues is a time-consuming, error-prone, and challenging task, even for senior developers.

Nowadays, machine learning (ML) and deep learning (DL) achieve promising results in different tasks of natural language processing (NLP), such as clustering, prediction, sentiment analysis, etc. [15]. Many ML and DL algorithms can build models fed with datasets to discover the representations needed for detection or prediction automatically. To explore the possibilities of automatically recovering the missing PR-Issue links using ML, we conducted an empirical study to identify the characteristics of explicit PR-Issue links based on their textual representations. We found that the links between PR and Issue exhibit certain features. For example, the titles, body descriptions, and comments for Issues and their corresponding PRs share varieties textual similarities.

In this article, we propose to build ML models to recover PR-Issue links using two ML algorithms (KMeans and BIRCH) based on lexical and semantic weighting measures. We selected these algorithms rather than other ones for two reasons: The first reason is their capability to predict the clusters for new datasets. The second reason is that the number of clusters is known, which is the number of PR-Issue links in our case. Therefore, overcome the problem to determine the best number of clusters faced by the other clustering algorithms (the local optima problem [14]). The effectiveness of these algorithms is evaluated using a ground-truth dataset on the subject called PI-Link [15]. The experimental results show that the BIRCH algorithm outperforms the KMeans algorithm in terms of the most widely used

metrics in this domain: homogeneity, completeness, and V-measure. Moreover, three features of PRs and Issues are used to build clustering models: Title, Body, and Comment. According to the experimental results, the Title and Body features are more important in this recovering task than the comment feature.

The rest of this article is divided into the following sections. In section II, we provide the necessary background to easily understand the proposed approach. Section III provides a detailed explanation of how the clustering models are built. In section IV, the clustering models are experimentally evaluated with a ground-truth dataset. Related works are discussed in section V. Finally, the article's conclusions and future work are drawn in section VI.

## II. BACKGROUND

In this section, we provide the necessary background to understand the contribution presented in this article. This background includes GitHub flow, PR-Issue links, and clustering algorithms.

### A. GITHUB FLOW

GitHub is the most popular social coding platform to host open source projects, with over 100 million developers working on over 352 million repositories (more than 41 million public repositories [16]) by November 2022 [17] [18]. It is built on top of a version control system called Git [19]. Therefore, GitHub provides a distributed version control system in addition to other services, such as Issue tracking, parallel development, incremental development, integration, etc.

GitHub is a branch-based development. This helps developers to work on development activities independently without affecting other branches of other developers in the same repository. In fact, each repository has one default main branch, called Master, and other branches created by developers. Figure 1 shows the GitHub flow. The flow starts by creating a new branch for adding new features, fixing bugs, or doing some enhancements. Then, developers made their changes to new the branch, which included (adding, deleting and editing files) as commits. To help the owner or future developers easily understand the committed changes (e.g., performance enhancement, fixing a bug, etc.), each commit includes a message describing it. Next, the developers create PR when their changes (commits) on that branch are completed. Also, developers should provide a summary of each PR (including title and description body) to describe their accomplished task. Additionally, it maintains a record of code changes. If the submitted PR is a solution to an open Issue, it is linked to that Issues to show that a solution is in progress. After several rounds of reviewing and discussion among developers and reviewers (e.g., leave questions, comments, and suggestion on PR timeline), a decision is made to (approve, enhance, or reject) the PR. Finally, when the PR is approved and tested, it is merged back to the Master branch
<figure><img src="image_2.png" /><figcaption>FIGURE 1: Branch-based workflow in GitHub.</figcaption></figure>

to save changes permanently. Also, the state of the merged PR is changed from open to closed PR.

## B. PR-ISSUE LINKS IN GITHUB

In GitHub, Issues and PRs are the main important assets for collaborative distributed development and maintenance activities. On the one hand, Issues are used for different purposes, such as asking questions, signalling bugs, and other maintenance tasks. They are typically stored in the Issue tracking system. On the other hand, PRs are used to propose code changes by other collaborators and reviewers. They are managed by the version control system. Also, both Issues and PRs represent artefacts of different domain spaces, as Issues represent problem space (requirements, features, bugs, etc.) while PRs represent solution space (fixing bugs, implementing a new feature, etc.). In fact, linking PRs and Issues is also a link between solution and problem spaces. As a result, such links have different semantics. It can be a documentation link between features or requirements and their corresponding code [20]. Also, it can be a maintenance link between bugs and their solutions [21], [22]. The links between PRs and Issues may be established in the same repository or different repositories. When PRs are approved and merged into the main branch, the state of their linked Issues is changed to closed (solved).

<figure><img src="image_3.png" /><figcaption>FIGURE 2: A PR linked to an Issue implicitly.<fnref n="1" /></figcaption></figure>

<figure><img src="image_4.png" /><figcaption>FIGURE 3: A PR linked to an Issue explicitly.<fnref n="2" /></figcaption></figure>

There are two ways to link PRs and Issues: implicitly and explicitly [23]. In the implicit way (as shown in Figure 2), GitHub provides keywords to link PRs and Issues (e.g., resolve(s,d), close(s,d), and fix(es,ed)). These keywords are placed in the PR text (either description or the commit message). This way has two main concerns. Firstly, removing or changing PRs-Issues links is a time-consuming and boring process as the owner requires manually modifying the PR's description to remove or modify the link's keywords. Secondly, a PR may include many commits with link keywords only placed in the message of some of these commits. When a PR is merged to the branch, the linked Issue is closed despite there being some commits that are not linked to that Issue. This leads to missing the main link between solution space

<footnote n="1">¹https://github.com/AdAway/AdAway/pull/784</footnote>

<footnote n="2">²https://github.com/AdAway/AdAway/issues/781</footnote>
and problem space. In the explicit way, these limitations are eliminated where PRs and Issues are manually linked by PRs sidebar or Issue sidebar as shown in Figure 3. In this way, contributors and reviewers can monitor the progress of development and maintenance activities on the linked Issues and their corresponding PRs. Therefore, any authorized contributor can easily change or remove the PR-Issue links.

### C. CLUSTERING ALGORITHM

Clustering is the process of grouping similar items (based on patterns or structural properties) as a cluster. Clustering algorithms are unsupervised learning algorithms where the datasets are unlabeled. In the literature, there are many clustering algorithms such as, KMeans, BIRCH, DBSCAN, Spectral clustering, and hierarchical clustering [24]. Each one has its own considerations. One of the main considerations depends on the number of clusters provided in advance. For example, the KMeans and BIRCH algorithms require a predefined number of clusters to provide, whereas the DBSCAN and hierarchical clustering algorithms do not need this predefined number of clusters. Another important consideration is whether the algorithm is able to predict clusters for an untrained (unseen) dataset. For example, the KMeans and BIRCH algorithms can be used to predict the clusters for new datasets, whereas the DBSCAN algorithm can not be used for this purpose. In anyway, the ground-truth dataset is important for unsupervised learning techniques to compare the prediction accuracy of different machine-learning (ML) models and specify the appropriate one. In this article, we rely on KMeans and BIRCH algorithm to recover (cluster) PR-Issue links that are best fit to the requirements of the selected ground-truth dataset.

The KMeans is an unsupervised ML algorithm. It is used to group data points into a predetermined number of clusters (K). It starts with K randomly chosen items acting as the clusters' centroids. After that, the algorithm iteratively calculates the total of the squared distances between the data points and the centroids of the established clusters. A data point is assigned to the cluster which its centroid is the closest to that data point. After that, it computes a new centroid for each cluster until the best centroid is discovered [25]. Like KMeans, BIRCH is driven by a predefined number of clusters. BIRCH stands for Balanced Iterative Reducing and Clustering using Hierarchies [26]. It converts data points into a tree data representation with the centroids. These centroids can be the final cluster centroid or the input for other iterative clustering round. BIRCH starts by clustering a given dataset into small summaries in one-time scan. Then, the summary can also be clustered by other clustering algorithms, such as agglomerative hierarchical clustering.

## III. CLUSTERING MODEL FOR PR-ISSUE LINKS

In this study, the recovering process of PRs-Issues links is driven by three features which are always available in PRs and Issues contents. These features are: Title, Body, and Comment. Each of these features works as a clue to

recover the links between PRs and their corresponding Issues. Similar Titles give a strong indicator that these PR and Issue addressing the same problem. Similar Bodies and Comments give details about the type of Issue the current PR is addressing and how it is being addressed. However, other features like PR and Issue labels are overlooked where they are missed and repetitive in many PRs and Issues.

In this section, we build the clustering models to link PRs with their relevant Issues. The clustering models take the raw data of Issues and PRs without their links as inputs. Then, the textual content of PRs and Issues are parsed to extract only the features of interest. Next, PRs and Issues go through preprocessing steps for cleaning. Then, lexical and semantic similarities are computed for PRs and Issues vectors. Finally, these vectors are clustered using KMeans and BIRCH algorithms to recover PR-Issue links by building clustering models. The recovered links are evaluated with ground PR-Issue links that existed in the ground-truth dataset. Figure 4 gives an overview of this recovery process. Below, we detail the steps required to build clustering models using KMeans and BIRCH algorithms.

<figure><img src="image_5.png" /><figcaption>FIGURE 4: Recovering process overview.</figcaption></figure>

### A. FEATURE EXTRACTION

Feature extraction is the process of eliminating irrelevant and noisy data from the textual content of input PRs and
Issues. Therefore, only relevant features should remain after this process to be ready to build the clustering models. As mentioned above, we are interested in three features as clues to link PRs with their corresponding Issues: Title, Body, and Comment of both PRs and Issues.

## B. DATA PRE-PROCESSING

We apply the following standard pre-processing steps to clean the dataset.

### 1) Removing Punctuation marks and special characters

In this task, we remove punctuation marks, numbers, parentheses, and any special characters. A suitable regular expression has been used to carry out this cleaning task.

### 2) Tokenization

We tokenize (split) the text of input PRs and Issues into words. This text includes the Title, Body, and Comment of both PRs and Issues. The tokenization strategy considers space as a separator and discards punctuation marks and unnecessary symbols. To do so, we use the "spaCy" library [27] to end up with useful words.

### 3) Removing stop words

In this step, we remove all stop-words that frequently occur in the tokens list (obtained in the previous) but do not carry any meaning and can not be used to compute the similarity between two PR and Issue (e.g., the, on, between, etc.). We use "NLTK" (Natural Language Toolkit [28]) to remove all stop words.

### 4) Stemming

Stemming is a process of returning back tokens to their base forms (for example, makes to make, raising to raise). We use one of the most popular stemming algorithms, "Porter stemmer", provided by the NLTK library. Moreover, we convert the entire text into lowercase characters.

## C. COMPUTING SIMILARITY FOR EACH FEATURE

In this step, we compute the similarity between two PR and Issue by using each feature, individually. This similarity is computed using lexical and semantic weighting measures.

For the lexical measure, we rely on a well-known weighting measure called TF-IDF (Term Frequency Inverse Document Frequency) [29]. In this measure, a weighting score (TF-IDF score) is given to each token in a feature's corpus. This score increases when the frequency of a token in a given PR/Issue vector increases but it is reduced by the frequency of the token in the other PR/Issue vector. Also, this score indicates the importance of a token in a given PR/Issue vector; the content of PR/Issue is better represented by tokens with higher TF-IDF scores. In this step, three TF-IDF matrices are created; a matrix for each feature (Title, Body, Comment of both PRs and Issues in each time). In each TF-IDF matrix, rows represent PRs and Issues vectors, columns represent

all tokens in the feature's corpus, and the content of this matrix represents TF-IDF weights. This is performed using "TfIdfVectorizer" method provided by "scikit-learn" library. The similarity between PRs and Issues vectors is computed using the cosine similarity according to the equation 1. In this equation, V1 and V2 refer to vectors of PRs and Issues.

$$Cosine(V_1, V_2) = cos(\theta) = \frac{\vec{V_1} \cdot \vec{V_2}}{||\vec{V_1}|| \cdot ||\vec{V_2}||} \quad (1)$$

For the semantic weighting measure, we use the most recent semantic measure in natural language processing (NLP) called BERT (Bidirectional Encoder Representations from Transformers). It is a pre-trained language-based transformer model that is developed by Jacob et al. in Google [30]. BERT is a semantic model representation based on deep learning that learns contextual relations between words in a text. Therefore, it can be used to embed the semantics of texts via vector representations. In this study, we exploit the BERT transformer that works based on "bert-base-nli-mean-tokens" pre-trained model in combination with a similarity measure to compute the semantic similarity score between PRs and issues. In our case, BERT is applied three times; one time for each feature. The BERT transformer converts

<table><caption>TABLE 1: Statistical information about the ground-truth dataset.</caption><thead><tr><th></th><th>Total</th><th>Min</th><th>Max</th><th>Mean</th><th>Std</th></tr></thead><tbody><tr><td>Issue title</td><td>34732</td><td>1</td><td>974</td><td>6.04</td><td>30.13</td></tr><tr><td>PR title</td><td>50369</td><td>1</td><td>1570</td><td>8.77</td><td>44.78</td></tr><tr><td>Issue body</td><td>34591</td><td>1</td><td>974</td><td>6.04</td><td>30.13</td></tr><tr><td>PR body</td><td>50240</td><td>1</td><td>1570</td><td>8.77</td><td>44.78</td></tr><tr><td>Issue comment</td><td>96366</td><td>0</td><td>7185</td><td>16.78</td><td>109.51</td></tr><tr><td>PR comment</td><td>133865</td><td>0</td><td>3206</td><td>23.31</td><td>101.9</td></tr><tr><td>Issue label</td><td>22235</td><td>0</td><td>1087</td><td>3.87</td><td>26.96</td></tr><tr><td>PR label</td><td>8882</td><td>0</td><td>1192</td><td>1.54</td><td>23.01</td></tr></tbody></table>

textual contents of PR and Issue into vectors $\vec{V_1}$ and $\vec{V_2}$, respectively. Then, the similarity score is obtained using the cosine similarity in Equation 1.

## D. CLUSTERING ALGORITHMS

In this step, we apply clustering algorithms which are fit to our case. We find KMeans and BIRCH algorithms the best fit for our case (refer to background section II for further details about these algorithms). We adapted these clustering algorithms as follow. In both algorithms, the number of clusters is predefined, which is the number of linked issues. Also, the data points in both algorithms are issues and PRs vectors which are built in the previous step. For each issue vector, the clustering algorithms find PRs' vectors that are the closest to that issue according to the considered features (i.e. Title, Body, and Comment). The quality of predicted clusters can be compared with the ground-truth ones. The obtained Issue-PR clusters are compared with their corresponding ground-
truth Issue-PR clusters (I.e. Issue-PR links). We relied on *sklearn.cluster* to implement KMeans [31] and BIRCH [32] algorithms.

## IV. EVALUATION OF CLUSTERING MODEL

In this section, we aim to evaluate the effectiveness of the KMeans and BIRCH algorithms for recovering PR-Issue links.

### A. GROUND-TRUTH DATASET

The considered clustering algorithms are applied to a ground-truth dataset in the subject (available on: *Kaggle* <fnref n="3" />). This dataset includes 5742 Android projects that have explicit PR-Issue links. Table 1 provides statistical information about the ground-truth dataset in terms of the total number of PRs and Issues across all projects in the dataset, min, max, mean, and standard deviation (Std) at the level of projects. For example, the dataset has 34732 Issue titles across all projects. Some of these projects have just one title (i.e. contains just one Issue), whereas a project has 974 titles (i.e. contains 974 Issues). The average and Std for the number of Issues overall projects are 6.04 and 30.13, respectively. The table shows that the dataset has 34732 Issues linked to 50369 PRs. The difference in the numbers of PRs and Issues refers to the cardinality ratio relationships between PRs and Issues, where a PR may be linked to many Issues and an Issue may be linked to many PRs.

Table 2 represents the number of empty data found in the ground-truth dataset. We observe that each PR and Issue has a title, but a few do not have a body (141 out of 34732 in Issues, and 129 out of 50369 in PRs). Besides, 1296 Issue and 8 PR do not have a comment. The reason is that the

<table><caption>TABLE 2: Empty data found in the ground-truth dataset</caption><thead><tr><th></th><th># Φ Title</th><th># Φ Body</th><th># Φ comment</th><th># Φ lable</th></tr></thead><tbody><tr><td>Issue</td><td>0</td><td>141</td><td>1296</td><td>2830</td></tr><tr><td>PR</td><td>0</td><td>129</td><td>8</td><td>4636</td></tr></tbody></table>

PRs usually go through a review and discussion process using PR comments, but this process is not mandatory for Issues. Finally, we can notice that around 8% and 9% of Issues and PRs without labels, respectively.

### B. RESEARCH QUESTIONS AND EVALUATION METRICS

To quantify the accuracy of the recovered links, we designed experiments to answer the following research questions:

1) **RQ1:** How far are the recovered PR-Issue links correct?

The aim of this research question is to measure the correctness of recovered links (clusters) between PRs and their associated Issues. To do so, we apply these clustering algorithms

on our dataset after removing all links between PRs and Issues. This means that the aim of the clustering algorithms is to recover these removed links. We measure the correctness using the Homogeneity metric. The *Homogeneity* measures the percentage of how far the resulting clusters include similar members. The homogeneity score is calculated based on Equation 2 [33].

$$homogeneity = 1 - \frac{H(C|K)}{H(C)} \qquad (2)$$

where $H(C|K)$ is the conditional entropy of the classes (i.e., PR-Issue links from the ground truth), and it is obtained by Equation 3.

$$H(C|K) = - \sum_{k=1}^{|K|} \sum_{c=1}^{|C|} \frac{n_{c,k}}{N} \log \left( \frac{n_{c,k}}{n_c} \right) \qquad (3)$$

and $H(C)$ is the entropy of the classes, and it is obtained based on Equation 4.

$$H(C) = - \sum_{c=1}^{|C|} \frac{n_c}{N} \log \left( \frac{n_c}{N} \right) \qquad (4)$$

where $N$ is the total number of items, $n_c$ is the number of items belonging to class $c$, and $n_{c,k}$ refers to the number of items from class $c$ that belong to cluster $k$.

The homogeneity score will be 100% when all items grouped in cluster $k$ have the same class $c$ (see Figure 5). In our case, it measures if all PRs and Issues belonging to a recovered cluster are associated to the same link or not. For example, the homogeneity will be 100% if all PRs and Issues in a cluster are members of the same PR-Issue link.

<figure><img src="image_6.png" /><figcaption>FIGURE 5: Homogeneity = 100, $C \cup K = C$.</figcaption></figure>

2) **RQ2:** How far are the recovered PR-Issue links complete?

This RQ aims to measure the completeness of recovered link members (PRs and Issues). We measure the completeness using the completeness metric. The *completeness* metric is a complementary measure to the homogeneity metric. It measures the percentage of the true positive members in the recovered clusters to all members in ground-truth clusters. The completeness scores are formally obtained based on Equation 5 [33]:

<footnote n="3">³https://www.kaggle.com/datasets/zakareaalshara/android-closed-issues-20110101-20210101-clean</footnote>
$$completeness = 1 - \frac{H(K|C)}{H(K)} \qquad (5)$$

where $H(K|C)$ is the conditional entropy of the clusters, and it is obtained by Equation 6.

$$H(K|C) = - \sum_{c=1}^{|C|} \sum_{k=1}^{|K|} \frac{n_{c,k}}{N} \cdot \log \left( \frac{n_{c,k}}{n_k} \right) \qquad (6)$$

and $H(K)$ is the entropy of the cluster, and it is obtained based on Equation 7.

$$H(K) = - \sum_{k=1}^{|K|} \frac{n_k}{N} \cdot \log \left( \frac{n_k}{N} \right) \qquad (7)$$

where $N$ is the total number of items, $n_k$ is the number of items belonging to cluster $k$, and $n_{c,k}$ refers to the number of items from class $c$ they belong to cluster $k$.

The completeness score equals 100% when all items of class $c$ have been assigned to the same cluster $k$ (see Figure 6). In our case, the completeness will be 100% if all PRs and Issues that are members of the same PR-Issue link are recovered in the same cluster.

<figure><img src="image_7.png" /><figcaption>FIGURE 6: Completeness = 100, $C \cap K = C$.</figcaption></figure>

### 3) RQ3: How far are the recovered PR-Issue links accurate?

We measure the accuracy based on the homogeneity and completeness of recovered links. For this aim, we use v-measure metric. The V-measure metric is the harmonic mean of homogeneity and completeness scores [33] and it is formally given by Equation 8.

$$V-measure = 2 \cdot \frac{homogeneity \cdot completeness}{homogeneity + completeness} \qquad (8)$$

The V-measure equals 100% when both homogeneity and completeness are equal 100% (see Figure 7). On the other hand, if a cluster does not satisfy the homogeneity or the completeness (i.e., the homogeneity or the completeness equals zero), then its V-measure will be zero. For example, the V-measure of a recovered link will be 100% if the whole and only its PRs and Issues members are from the same PR-Issue link.

<figure><img src="image_8.png" /><figcaption>FIGURE 7: V-measure = 100, $C \cup K = C \cap K$.</figcaption></figure>

## C. CLUSTERING CONFIGURATION

To answer the RQs, we applied KMeans and BIRCH algorithms to the ground-truth dataset seven times. The first time, we trained one-sevenths of the dataset, and each time the dataset is doubled (i.e., the second time is 2/7, the third time is 3/7, etc.). This method aims to monitor the clustering models' performances over diverse dataset sizes. Moreover, it helps to overcome biased datasets and overfitting. For each experiment, we compute homogeneity, completeness, and v-measure scores for the selected features (the text of PRs and Issues titles, the text of PRs and Issues bodies, and the text of PRs and Issues comments). These scores are measured by using the `sklearn.metrics` module [34]–[36].

Table 3 demonstrates the parameters that we used for KMeans and BIRCH configurations.

<table><caption>TABLE 3: Configurations of clustering algorithms.</caption><thead><tr><th></th><th>KMeans</th><th>BIRCH</th></tr></thead><tbody><tr><th>Stop-words</th><td>English</td><td>English</td></tr><tr><th>Max features</th><td>None</td><td>None</td></tr><tr><th># of clusters</th><td># of PR-Issues links</td><td># of PR-Issues links</td></tr><tr><th>Random state</th><td>Zero</td><td>Zero</td></tr><tr><th>Threshold</th><td>-</td><td>0.03</td></tr></tbody></table>

We used the English dictionary provided by `sklearn` library [37] to remove the stop-words. Furthermore, we build a vocabulary that only considers the top max-features (determined by `TfidfVectorizer`) ordered by term frequency across the dataset. As we know the number of PR-Issue links regardless of the cardinality among them, then the number of clusters will be the number of PR-Issue links. For example, if an Issue is linked to many PRs, these links are considered a cluster consisting of that issue and their corresponding PRs (and vice versa). Moreover, we fixed the randomness of the initial cluster centers to obtain consistent results for comparisons. Finally, the threshold used by the BIRCH algorithm denotes the maximum distance between the cluster's members from the cluster's center. After evaluating many thresholds on BIRCH, we found that the 0.03 threshold value satisfied our goal.
## D. RESULTS

### 1) RQ1: How far are the recovered PR-Issue links correct?

Figure 8 illustrates the homogeneity scores for KMeans and BIRCH using the lexical weighting (TF-IDF) and based on the selected features on different sizes of the dataset. We can observe that the homogeneity scores of KMeans are strongly incremental with the size of the dataset, while BIRCH is almost consistent (roughly flat). Moreover, the homogeneity scores of trained titles and bodies are good by the two algorithms (between 73% and 90%) rather than the homogeneity scores of trained comments (31% - 50%). Nevertheless, the highest homogeneity score is 90.4%, obtained by the BIRCH algorithm.

<figure><img src="image_9.png" /><figcaption>FIGURE 8: The obtained homogeneity scores of KMeans and BIRCH based on lexical similarity using different features.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>FIGURE 9: The obtained homogeneity scores of KMeans and BIRCH based on semantic similarity using different features.</figcaption></figure>

Figure 9 shows the homogeneity scores when KMeans and BIRCH are applied to the dataset using the semantic weighting (BERT) based on the selected features. Obviously, KMeans and BIRCH have roughly flat curves regardless of

the data size and considered features. Also, Title and Body features in both algorithms have higher homogeneity scores than Comment score. The homogeneity scores of Title and Body take a range of (76% - 89%) while Comment features scores take a low range of (30% - 32%). Moreover, when we compare the homogeneity results of KMeans and BIRCH across all features, we deduce that BIRCH is also the best with maximum homogeneity score of 89% in the case of using semantic weighting.

### 2) RQ2: How far are the recovered PR-Issue links complete?

Figure 10 illustrates the completeness scores for KMeans and BIRCH for all selected features on different sizes of the dataset using the semantic weighting. We can see that all selected features have high completeness scores (83.3% - 92.7%). The highest completeness score is 92.7% obtained by the BIRCH algorithm.

<figure><img src="image_11.png" /><figcaption>FIGURE 10: The obtained completeness scores of KMeans and BIRCH based on lexical similarity using different features.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>FIGURE 11: The obtained completeness scores of KMeans and BIRCH based on semantic similarity using different features.</figcaption></figure>
Figure 11 displays the completeness results in case of using the semantic weighting in both KMeans and BIRCH based on the selected features across all dataset sizes. In this figure, there is a minor incremental growth of KMeans and BIRCH curves against the dataset size. Also, KMeans and BIRCH have high completeness scores in all features as these scores take a high range in (84% - 92%). Moreover, BIRCH outperforms KMeans with a maximum completeness score 92% while the maximum completeness score in KMeans equals to 91%.

### 3) RQ3: How far are the recovered PR-Issue links accurate?
Figure 12 illustrates the V-measure scores for KMeans and BIRCH for all selected features on different sizes of the dataset. We can observe that both Title and Body features give high scores (76.8% - 91.5%), but the Comment feature gives a low one. The highest v-score obtained by the BIRCH algorithm is 91.5%.

In Figure 13, V-measure scores are presented of both KMeans and BIRCH using the semantic weighting across all selected features and dataset sizes. As shown in this figure, BIRCH curves across all features and dataset sizes are flatter than KMeans curves. For selected features, we note that Title and Body features give high V-measure results, (81% - 90%) respectively. Also, this figure confirms that BIRCH as usual outperforms KMeans in terms of V-measure with a maximum value of 90% while KMeans has a V-measure maximum value of 88%.

<figure><img src="image_13.png" /><figcaption>FIGURE 12: The obtained v-measure scores of KMeans and BIRCH based on lexical similarity using different features.</figcaption></figure>

Regardless of the weighting measure used (semantic or lexical), the Title and the Body features achieve promising and good homogeneity, completeness, and V-measure scores over different sizes of datasets in KMeans and BIRCH. Firstly, the homogeneity scores, respectively obtained by KMeans and BIRCH, are (72.9%-89.5%) and (88%-90.4%) for the Title, and 69%-85% and (81.1%-82.6%) for the Body. Secondly, the completeness scores, respectively obtained by

KMeans and BIRCH, are (88.3%-91.3%) and (90%-92.3%) for the Title, and (86.4%-89.6%) and (89%-87.8%) for the Body. Finally, The V-measure scores, respectively obtained by KMeans and BIRCH, are in a range of (79.9%-90.4%) and (89%-91.5%) for the Title, and (76.8%-85%) and (84.9%-85.5%) for the Body.

The Comment feature performs poorly in homogeneity either using the semantic weighting or the lexical weighting over different sizes of datasets in KMeans and BIRCH as shown in Figure 8. The homogeneity scores take a range of (30%-50%) and (30%-32.7%), obtained by KMeans and BIRCH, respectively. This means that the PRs and Issues of the recovered links using their comments have come from different PR-Issue links. On the other hand, the completeness scores of the Comment feature are not bad. In Figure 10, the completeness scores obtained by KMeans and BIRCH are (83.3%-86.9%) and (83.3%-86%), respectively. This indicates that most of the PRs and their associated Issues are placed together in the same cluster. The low homogeneity and high completeness scores indicate that a recovered link has many PR-Issue links. However, the V-measure scores reveal a bad performance of the Comment feature, as shown in Figure 12. They are (44.9%-63.5%) and (44.9%-45.2%), obtained by KMeans and BIRCH, respectively.

As a summary, we can observe that BIRCH outperforms KMeans in terms of homogeneity, completeness, and V-measures in both the semantic and lexical weighting. Also, BIRCH outperforms KMeans by using Title and Body features but Comment feature does not. However, these scores of both algorithms using the Comment features are not promising. Moreover, BIRCH has demonstrated its effectiveness on different sizes of the dataset using Title feature where maximum homogeneity, completeness, and V-measure scores are (90.4%, 92.3%, and 91.5%), whereas KMeans has varied scores depending on the size of the dataset for the same feature (89.5%, 91.3%, and 90.4%). Consequently, the BIRCH algorithm is a better fit in our case.

<figure><img src="image_14.png" /><figcaption>FIGURE 13: The obtained v-measure scores of KMeans and BIRCH based on semantic similarity using different features.</figcaption></figure>
## E. DISCUSSION

The evaluation revealed some PR-Issue links that the clustering models are not able to recover (8%) based on the Title feature. During the evaluation, we inspected the links that had been mispredicted by the model. As a result, we found three categories of mispredicted PR-Issue links; common topics, different levels of abstraction, and short titles.

### 1) Common Topics

We found some instances of Issues that have common descriptions (similar text). Where the dataset is extracted from Android projects, most of these instances are related to the Android platform. For example, some of these Issues arise when a new level of Android API is released. The new API might provide new features that should be implemented by Android developers. Accordingly, these Issues are raised due to the implementation of new features by Android developers in their projects. While these Issues represent the same topic, then the probability of having similar titles will increase. Consequently, this will group these similar Issues in the same cluster and leads the model to misprediction.

### 2) Different Levels of Abstraction

We found some instances where the Title of an Issue has a different level of abstraction than its associated PRs. In most of these instances, when the Issue of a new feature or a new bug is described by using texts close to the feature requirement or bug use cases (conceptual level), whereas the PR associated with that Issue is described by using texts close to technical implementation (implementation level). Consequently, this will place this kind of Issue and its linked PRs in different clusters due to its textual dissimilarity, which leads the model to make a misprediction.

### 3) Short Titles

We found some instances of PRs and Issues with too short titles (i.e., only one or two words). Whenever the inputs are sufficient for comparisons, the prediction of the clustering model will be more accurate. The short text makes insufficient inputs to the clustering model, which leads to misprediction.

As shown in Figures 8, 10, and 12, the Comment feature shows poor performance in recovering PR-Issue links. This is due to two reasons. Firstly, many Issues do not have any Comment (1296 Issues). Another reason is that comments are usually inserted by different types of contributors with different points of view, including reviewers, developers, and other involved people. Therefore, each Comment represents a different level of matter and abstraction. This leads to have comments with a variant textual similarity which provides misleading information of ML prediction model.

## F. THREATS TO VALIDITY

We identified a set of issues as potential threats to the validity of this research work. These threats are as follows.

As an internal threat, the used clustering algorithms are sensitive to the vocabulary used in Titles, Bodies, Comments of linked PRs and Issues. In the case of using different vocabularies, this may degrade the prediction model's accuracy. However, this threat is shared among all research works employing textual matching to find the similarity.

As an external threat, the effectiveness of KMeans and BIRCH algorithms is evaluated using Android-based ground-truth dataset. This gives a first glance impression that these clustering algorithms can not be applied to other types of applications. In fact, these algorithms work with any type of application hosted on GitHub.

Another external activity regarding the extensibility of our approach is to recover PR-Issue links from projects hosted on other social coding platforms. We can apply our approach to other social coding platforms that provide PR-Issue linking features like GitLab and integrated Bitbucket/JIRA. However, there are some slightly different changes between GitHub and the aforementioned platforms. These changes come from different nomenclature features and metadata supported by social coding platforms. For example, we can assume that GitLab's "merge-request" feature (Git merge) is equivalent to GitHub's "pull-request" feature (Git pull). As same GitHub, GitLab and Bitbucket/JIRA support the linking feature between Issues and pull-request/merge-request. However, GitLab supports just implicit links by mentioning issues in merge request comments, whereas Bitbucket/JIRA supports only explicit links by using integrated features provided by JIRA issue tracking system. Moreover, the metadata names are different between these social coding platforms. For example, we can assume that Issue "Body" in Github is equivalent to GitLab and JIRA "Description". In this paper, we focused on GitHub because we have a ground-truth dataset for explicit pull-request/issue links extracted from GitHub projects.

## V. RELATED WORK

To the best of our knowledge, there is no research work in the literature on this subject for recovering the links between PRs and Issues in GitHub and other social coding platforms. The closest works in the literature aim to recover Issue-Commit links but not PR-Issue links. These works are evaluated using different and non-benchmark datasets. In this section, we present the most recent and relevant works related to our work. These research works are organized into two categories based on the type of linked software artifacts: recovering links between Issue and Commit, and recovering links between other software artefacts.

### A. RECOVERING LINKS BETWEEN ISSUE-COMMIT

There are many proposed approaches for recovering Issue-Commit links that are based on Issue tracking (such as Bugzilla) and version control (such as Git) systems. The traditional approach for recovering such links involves searching for Issues identifiers in version control systems logs [38] [39] [40] [41]. Many researchers, however, discovered that
these traditional approaches are ineffective because they can
only retrieve a limited number of Issue-Commit relationships
while missing a significant number. This is because devel-
opers may not always provide Issue IDs in version control
system logs.

Heuristic-based approaches are proposed to recover Issue-
Commit link. Bachmann et al. [42] and Bird et al. [43] intro-
duced a simi-automation tool called LINKSTER to facilitate
recovering Issue-Commit links. Also, Wu et al. [44] proposed
a tool called ReLink to recover such links taking into ac-
count three heuristics: Issue and commit reports should be
similar, their resolving times should be the same or close
to each other, and their contributors should be the same.
ReLink recovered Issue-Comit links with better accuracy
than traditional heuristics approaches, with 89% precision
and 78% recall compared with 91% precision and 64% recall
for the traditional heuristics ones. In [45], Bissyande et al.
proposed an improvement for ReLink. In [46], Nguyen et
al. proposed another heuristic-based approach called MLink.
This approach is based on code heuristics to recover Issue-
Commit links based on the code snippets' similarity in Issue
report and Commits. Their empirical evaluation with related
approaches showed that MLink improved the Issue-Commit
recovery links by 17%, 13-17%, and 11-18% in precision,
recall, and F-score, respectively.

In the literature, there is another direction to recover Issue-
Commits links using machine learning algorithms. In [47],
Le et al. implement a clustering tool (called RCLinker)
based on textual similarity and metadata of Issues and Com-
mits to predict Issue-Commit links. The accuracy results of
RCLinker reported from six software projects outperform
MLink in F-measure by 138.66%. Similarly, Rath et al. [48]
employ textual similarity and metadata to recover such links.
In [49], Sun et al. suggested two approaches (called FRLink
and PULink) to recover Issue-commit links. FRLink was
evaluated on the same software projects used by RCLinker,
and the results showed that FRLink outperforms RCLinker in
F-Measure by 40.75%.

B. RECOVERING LINKS BETWEEN OTHER SOFTWARE
ARTIFACTS

The research in this category focuses on recovering links
with different semantics between different software artifacts
like source code, documentation, requirements, design com-
ponents, etc.

In [50], Antoniol et al. introduced a probabilistic approach
for recovering Code-Documentation links. In [51], Hayes et
al. proposed vector space model (VSM) based approach to
recover Code-Requirement links. In [52], Eyal Salman et al.
proposed an approach to link functional features with their
implementing source code elements using a combination
of information retrieval (IR) and formal concept analysis
(FCA). In [53], Ali et al. introduced an approach, called
COPARVO, for recovering links between source code and
requirements by partitioning the source code into different
sources of information (class names, comments, methods

names, and class variables) using VSM. In [54], Poshyvanyk
et al. suggested an approach to combine the latent semantic
indexing technique and a dynamic technique called scenario-
based probabilistic ranking (SPR) to link features with their
respective source code methods.

VI. CONCLUSION AND FUTURE WORK

In this paper, we proposed an approach to recover links be-
tween PRs and Issues in collaborative software development
repositories on GitHub, based on the textual representation
made in issues and PRs. The proposed approach investi-
gates the results of two clustering algorithms (KMeans and
BIRCH) to recover these links based on semantic and lexical
measures. To do so, we trained KMeans and Birch using
different features on the ground-truth dataset in the subject.
The results show that PRs-Issues links can be recovered with
an accuracy of 91.5% using Birch clustering algorithm.

We plan to employ deep-learning techniques to recognize
the rules and patterns between similar Issues and similar PRs
that reside in different GitHub repositories. Therefore, when
a developer has a new Issue, the PR associated with this Issue
could be found in another repository. Therefore, the devel-
oper can gain benefits from other developers’ experiences.
Consequently, it helps to create a recommendation system for
software engineering.

REFERENCES

[1] R. Kallis, A. Di Sorbo, G. Canfora, and S. Panicella, “Ticket tagger: Machine learning driven issue classification,” in *2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)*, 2019, pp. 406–409.

[2] T. F. Bissyandé, D. Lo, L. Jiang, L. Réveillère, J. Klein, and Y. L. Traon, “Got issues? who cares about it? a large scale investigation of issue trackers from github,” *2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE)*, pp. 188–197, 2013.

[3] G. Gousios, M. Pinzger, and A. van Deursen, “An exploratory study of the pull-based software development model,” *Proceedings of the 36th International Conference on Software Engineering*, 2014.

[4] Y. Yu, H. Wang, G. Yin, and C. X. Ling, “Reviewer recommender of pull-requests in github,” in *2014 IEEE International Conference on Software Maintenance and Evolution*, 2014, pp. 609–612.

[5] I. C. Irsan, T. Zhang, F. Thung, D. Lo, and L. Jiang, “Autoprtitle: A tool for automatic pull request title generation,” 2022. [Online]. Available: https://arxiv.org/abs/2206.11619

[6] B. Andam, A. Burger, T. Berger, and M. R. V. Chaudron, "Florida: Feature location dashboard for extracting and visualizing feature traces," in *Proceedings of the Eleventh International Workshop on Variability Modelling of Software-Intensive Systems*, ser. VAMOS '17. New York, NY, USA: Association for Computing Machinery, 2017, p. 100–107. [Online]. Available: https://doi.org/10.1145/3023956.3023967

[7] F. Beck, B. Dit, J. Velasco-Madden, D. Weiskopf, and D. Poshyvanyk, “Rethinking user interfaces for feature location,” in *2015 IEEE 23rd International Conference on Program Comprehension*, 2015, pp. 151–162.

[8] B. Bassett and N. A. Kraft, “Structural information based term weighting in text retrieval for feature location,” in *2013 21st International Conference on Program Comprehension (ICPC)*, 2013, pp. 133–141.

[9] M. Borg, O. Svensson, K. Berg, and D. Hansson, "Szz unleashed: An open implementation of the szz algorithm - featuring example usage in a study of just-in-time bug prediction for the jenkins project," in *Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation*, ser. MaLTEsQuE 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 7–12. [Online]. Available: https://doi.org/10.1145/3340482.3342742
[10] K. Herzig, S. Just, and A. Zeller, "It's not a bug, it's a feature: How misclassification impacts bug prediction," in *2013 35th International Conference on Software Engineering (ICSE)*, 2013, pp. 392-401.

[11] Y. Zhou and A. Sharma, "Automated identification of security issues from commit messages and bug reports," in *Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering*, ser. ESEC/FSE 2017. New York, NY, USA: Association for Computing Machinery, 2017, p. 914-919. [Online]. Available: https://doi.org/10.1145/3106237.3117771

[12] A. Sajedi-Badashian and E. Stroulia, "Guidelines for evaluating bug-assignment research," *Journal of Software: Evolution and Process*, vol. 32, no. 9, p. e2250, 2020.

[13] P. Bhattacharya, I. Neamtiu, and C. R. Shelton, "Automated, highly-accurate, bug assignment using machine learning and tossing graphs," *Journal of Systems and Software*, vol. 85, no. 10, pp. 2275-2292, 2012. automated Software Evolution. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121212001240

[14] E. Trauwaert, L. Kaufman, and P. Rousseeuw, "Fuzzy clustering algorithms based on the maximum likelihood principle," *Fuzzy Sets and Systems*, vol. 42, no. 2, pp. 213-227, 1991.

[15] Z. Alshara, A. Shatnawi, S. A.-D. Eyal-Salman, Hamzeh, and M. Shatnawi, "Pi-link: A ground-truth dataset of links between pull-requests and issues in github," in *Submitted to IEEE access*, 2022, p. TBD.

[16] Github Number of Public Repository Search, "https://github.com/search?q=is:public," online, accessed 01/11/2022, November 2022, search result shows more than 41M public repositories.

[17] User search, "GitHub," online, accessed 01/11/2022, November 2022, search result shows more than 100M users.

[18] Github Number of Repository Search, "https://github.com/search," online, accessed 01/11/2022, November 2022, search result shows more than 352M repositories.

[19] L. Dabbish, C. Stuart, J. Tsay, and J. Herbsleb, "Social coding in github: Transparency and collaboration in an open software repository," in *Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work*, ser. CSCW '12. New York, NY, USA: Association for Computing Machinery, 2012, p. 1277-1286. [Online]. Available: https://doi.org/10.1145/2145204.2145396

[20] B. Dit, M. Revelle, M. Gethers, and D. Poshyvanyk, "Feature location in source code: a taxonomy and survey," *Journal of Software: Evolution and Process*, vol. 25, no. 1, pp. 53-95, 2013. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/smr.567

[21] R. Saha, Y. Lyu, W. Lam, H. Yoshida, and M. Prasad, "Bugs.jar: A large-scale, diverse dataset of real-world java bugs," in *2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR)*, 2018, pp. 10-13.

[22] Z. Tóth, P. Gyimesi, and R. Ferenc, "A public bug database of github projects and its application in bug prediction," in *Computational Science and Its Applications - ICCSA 2016*, O. Gervasi, B. Murgante, S. Misra, A. M. A. Rocha, C. M. Torre, D. Taniar, B. O. Apduhan, E. Stankova, and S. Wang, Eds. Cham: Springer International Publishing, 2016, pp. 625-638.

[23] GitHub Inc., "Linking a pull request to an issue," online, accessed 1/10/22, October 2022, https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue.

[24] A. E. Ezugwu, A. M. Ikotun, O. O. Oyelade, L. Abualigah, J. O. Agushaka, C. I. Eke, and A. A. Akinyelu, "A comprehensive survey of clustering algorithms: State-of-the-art machine learning applications, taxonomy, challenges, and future research prospects," *Engineering Applications of Artificial Intelligence*, vol. 110, p. 104743, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S095219762200046X

[25] P. Pandey and I. Singh, "Comparison between standard k-mean clustering and improved k-mean clustering," *International Journal of Computer Applications*, vol. 146, no. 13, pp. 39-42, Jul 2016. [Online]. Available: http://www.ijcaonline.org/archives/volume146/number13/25462-2016910868

[26] T. Zhang, R. Ramakrishnan, and M. Livny, "Birch: An efficient data clustering method for very large databases," ser. SIGMOD '96. New York, NY, USA: Association for Computing Machinery, 1996, p. 103-114. [Online]. Available: https://doi.org/10.1145/233269.233324

[27] M. Honnibal and I. Montani, "spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing," 2017, to appear.

[28] S. Bird, E. Klein, and E. Loper, *Natural language processing with Python: analyzing text with the natural language toolkit*. "O'Reilly Media, Inc.", 2009.

[29] G. Salton and C. Buckley, "Term-weighting approaches in automatic text retrieval," *Information Processing Management*, vol. 24, no. 5, pp. 513-523, 1988. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0306457388900210

[30] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," *arXiv preprint arXiv:1810.04805*, 2018.

[31] scikit-learn.org, "sklearn.cluster.KMeans," online, accessed 12/21/22, December 2022, https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html.

[32] ___, "sklearn.cluster.Birch," online, accessed 12/21/22, December 2022, https://scikit-learn.org/stable/modules/generated/sklearn.cluster.Birch.html.

[33] A. Rosenberg and J. Hirschberg, "V-measure: A conditional entropy-based external cluster evaluation measure," in *Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL)*, 2007, pp. 410-420.

[34] scikit-learn.org, "sklearn.metrics.homogeneity_score," online, accessed 12/21/22, December 2022, https://scikit-learn.org/stable/modules/generated/sklearn.metrics.homogeneity_score.htmlsklearn.metrics.homogeneity_score.

[35] ___, "sklearn.metrics.completeness_score," online, accessed 12/21/22, December 2022, https://scikit-learn.org/stable/modules/generated/sklearn.metrics.completeness_score.htmlsklearn.metrics.completeness_score.

[36] scikitlearn.org, "sklearn.metrics.vMeasure_score," online, accessed 12/21/22, December 2022, https://scikitlearn.org/stable/modules/generated/sklearn.metrics-vMeasure_score.htmlsklearn.metrics.vMeasure_score.

[37] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., "Scikit-learn: Machine learning in python," *Journal of machine learning research*, vol. 12, no. Oct, pp. 2825-2830, 2011.

[38] M. Fischer, M. Pinzger, and H. Gall, "Analyzing and relating bug report data for feature tracking," in *Proceedings of the 10th Working Conference on Reverse Engineering*, ser. WCRE '03. USA: IEEE Computer Society, 2003, p. 90.

[39] D. Cubranic and G. Murphy, "Hipikat: recommending pertinent software development artifacts," in *25th International Conference on Software Engineering, 2003. Proceedings.*, 2003, pp. 408-418.

[40] J. undefinedliwerski, T. Zimmermann, and A. Zeller, "When do changes induce fixes?" in *Proceedings of the 2005 International Workshop on Mining Software Repositories*, ser. MSR '05. New York, NY, USA: Association for Computing Machinery, 2005, p. 1-5. [Online]. Available: https://doi.org/10.1145/1083142.1083147

[41] A. Schröter, T. Zimmermann, R. Premraj, and Zeller, "If your bug database could talk," *Empirical Software Engineering - ESE*, 01 2009.

[42] A. Bachmann, C. Bird, F. Rahman, P. Devanbu, and A. Bernstein, "The missing links: Bugs and bug-fix commits," in *Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering*, ser. FSE '10. New York, NY, USA: Association for Computing Machinery, 2010, p. 97-106. [Online]. Available: https://doi.org/10.1145/1882291.1882308

[43] C. Bird, A. Bachmann, F. Rahman, and A. Bernstein, "Linkster: Enabling efficient manual inspection and annotation of mined data," in *Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering*, ser. FSE '10. New York, NY, USA: Association for Computing Machinery, 2010, p. 369-370. [Online]. Available: https://doi.org/10.1145/1882291.1882352

[44] R. Wu, H. Zhang, S. Kim, and S.-C. Cheung, "ReLink: Recovering links between bugs and changes," in *Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering*, ser. ESEC/FSE '11. New York, NY, USA: Association for Computing Machinery, 2011, p. 15-25. [Online]. Available: https://doi.org/10.1145/2025113.2025120

[45] T. F. Bissyandé, F. Thung, S. Wang, D. Lo, L. Jiang, and L. Réveillère, "Empirical evaluation of bug linking," in *2013 17th European Conference on Software Maintenance and Reengineering*, 2013, pp. 89-98.

[46] A. T. Nguyen, T. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, "Multi-layered approach for recovering links between bug reports and fixes," in *Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering*, 2012, pp. 1-11.
[47] T.-D. B. Le, M. Linares-Vasquez, D. Lo, and D. Poshyvanyk, "Rclinker: Automated linking of issue reports and commits leveraging rich contextual information," in *2015 IEEE 23rd International Conference on Program Comprehension*, 2015, pp. 36-47.

[48] M. Rath, J. Rendall, J. L. C. Guo, J. Cleland-Huang, and P. Mäder, "Traceability in the wild: Automatically augmenting incomplete trace links," in *Proceedings of the 40th International Conference on Software Engineering*, ser. ICSE '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 834-845. [Online]. Available: https://doi.org/10.1145/3180155.3180207

[49] Y. Sun, Q. Wang, and Y. Yang, "Frlink: Improving the recovery of missing issue-commit links by revisiting file relevance," *Information and Software Technology*, vol. 84, pp. 33-47, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584916303792

[50] G. Antoniol, G. Canfora, G. Casazza, A. De Lucia, and E. Merlo, "Recovering traceability links between code and documentation," *IEEE Transactions on Software Engineering*, vol. 28, no. 10, pp. 970-983, 2002.

[51] J. Hayes, A. Dekhtyar, and S. Sundaram, "Advancing candidate link generation for requirements tracing: the study of methods," *IEEE Transactions on Software Engineering*, vol. 32, no. 1, pp. 4-19, 2006.

[52] H. Eyal-Salman, A.-D. Seriai, and C. Dony, "Feature-to-code traceability in a collection of software variants: Combining formal concept analysis and information retrieval," ser. IRI'13. California, USA: IEEE Computer Society, 2013, pp. 209-216.

[53] N. Ali, Y.-G. a. Gueheneuc, and G. Antoniol, "Requirements traceability for object oriented systems by partitioning source code," in *18th Working Conf. on Reverse Engineering*. IEEE Computer Society, 2011, pp. 45-54.

[54] D. Poshyvanyk, Y.-G. Gueheneuc, A. Marcus, G. Antoniol, and V. Rajlich, "Feature location using probabilistic ranking of methods based on execution scenarios and information retrieval," *IEEE Trans. Softw. Eng.*, vol. 33, no. 6, pp. 420-432, June 2007.

<figure><img src="image_15.png" /></figure>

HAMZEH EYAL-SALMAN received the Ph.D. degree in software engineering from LIRMM Laboratory and the University of Montpellier, France, in 2014. He is currently an associate professor in Software Engineering department at Mutah University. His current research interests include software product line engineering, software reuse, software maintenance, and feature location. He has published many papers in various international journals, conferences on these topics.

<figure><img src="image_16.png" /></figure>

ZAKAREA AL SHARA received the Ph.D. degree in computer and software engineering from LIRMM and the University of Montpellier, France, in 2016. He is currently an assistant professor in Software Engineering department at Jordan University of Science and Technology. Moreover, he is the team leader of collaboration with the Compact Muon Solenoid (CMS) and the European Organisation for Nuclear Research (CERN) in Geneva, Switzerland. His current research interests include Software maintenance, Software evaluation, Software architecture, and Software modeling. He has published many papers in various international journals, conferences and workshops on these topics.

<figure><img src="image_17.png" /></figure>

DJAMEL SERIAI He is an associate professor at University of Montpellier and a member of the MaREL team of the LIRMM Laboratory. He is co-head of the Software Engineering Master. SERIAI is a senior software architect with more than 25 years of experience as an engineer, architect, and project manager in software development and maintenance field. and more than 20 years of experience in innovation and related to applied research projects.

...

<figure><img src="image_18.png" /></figure>

ANAS SHATNAWI is a senior research engineer at Berger-Levrault, France. He obtained his Ph.D. degree in Computer Science from Laboratory of Computer Science, Robotics, and Microelectronics (LIRMM) of University of Montpellier, France. His research interest is in software reuse, reengineering, reverse engineering and empirical software engineering. He has published many papers in various international journals, conferences and workshops on these topics. Contact him at

anas.shatnawi@berger-levrault.com.