dataesr/scientific-paragraphs-categorization huggingface dataset:
Cross-Validation Comparing different cross-validation algorithms revealed that K-Fold is more sensitive to imbalanced data than Stratified K-Fold (Fig. 3e ). While both procedures led to similar Balanced Accuracy scores with balanced classes, K-Fold cross-validation without stratification showed an increase in Balanced Accuracy towards extremely imbalanced datasets. This likely stems from validation splits containing only a single class, which Balanced Accuracy does not account for. While a classifier voting for the majority class has a Balanced Accuracy of 0. 5 on data containing two classes, its score will be 1 on validation splits that, by chance, only contain one class. The likelihood of this is higher for low sample sizes and is completely resolved by using Stratified K-Fold cross-validation.
Results on EEG data We performed two experiments using the EEG dataset with classification between eyes-open versus eyes-closed during resting (Section 2. 4. 1). We herein first describe the results of the multi-feature classification using parieto-occipital electrodes and then present the results of the channel-wise classification. As previously shown in the simulated data, Acc increased with increasing data imbalance, reflecting the proportion of the majority class. In contrast to this, BAcc approached chance level with increased imbalance and reached its maximum with maximally balanced data. In line with the simulated data, the F1 score increased abruptly at optimal class balance. AUC was stable over a wide range of class imbalance. While SVM, LR, and LDA showed similar behavior (i. e. being equally sensitive to data imbalance), the performance metrics using RF exhibited more stability over different levels of imbalance (Fig.
Results on MEG data As described in Section 2. 4. 2, classification was performed between auditory and visual stimuli, using channel-wise low-gamma spectral power averaged across the two MEG gradiometers. Figure  5  shows the results of the selected channel, whose spatial location is highlighted in Figure  5b. While AUC shows best overall classification performance of LR and LDA on the balanced dataset, the other metrics (Acc, BAcc, and F1) show similar scores across LR, LDA, and SVM. The Random Forest did not perform as well in this single-feature classification analysis (Fig. 5 ). In line with the simulated and EEG datasets, BAcc approached chance level with increased imbalance and reached its maximum with perfectly balanced data. F1 increased abruptly at optimal data balance. While AUC was stable across a wide range of imbalance ratios for LR, LDA, and RF, we observed comparable behavior to BAcc for SVM (i.
e. approaching chance level towards more data imbalance). While SVM, LR, and LDA show similar behavior (being equally sensitive to data imbalance), the performance metrics in RF exhibit more stability over different levels of imbalance. While the behavior of Accuracy is generally the same as in the experiments with simulated data, it is notable that, in LR and LDA, Accuracy is high when the majority of samples consists of visual stimuli and is decreased for a majority of auditory stimuli. This effect is most pronounced for light imbalance around 0. 5 and likely stems from a difference in distribution between the two classes. Samples belonging to the auditory class come from a more narrow distribution (Fig. 5a ) and are therefore prone to misclassification when the classifier blindly votes for the majority class.
Discussion The present work shows that the implementation of classification on imbalanced data is feasible, though it demands certain important considera-Our approach demonstrates how one needs to be mindful of class imbalance when choosing a classifier, an evaluation metric and a cross-validation scheme  [13]. Here, we sought to provide a didactic technical note on this ques-21 tion using a combination of simulated data and electrophysiological brain signals. Concretely, we quantified the behavior of commonly used classifiers, performance metrics, and cross-validation approaches across varying levels of data imbalance. An exhaustive exploration of all available techniques that have been proposed to tackle data imbalance is beyond the scope of this study. Instead, we chose to focus on machine learning tools and metrics that are often used within the neuroscience community.
In line with this, the methods we address-and the open-source pipelines and notebooks we provide-all use the scikit-learn library. Taken together, our observations support the idea that classification on moderately imbalanced data is feasible, as long as appropriate classifiers and performance metrics are employed. More specifically, by systematically manipulating the degree of data imbalance, we illustrated and quantified several key effects. First, we confirmed the tendency of classifiers to resort to blindly voting for the majority class as data imbalance was accentuated. When assessed with the widely used Accuracy measure, this behavior was associated with an artificial improvement in the model's classification performance. AUC and BAcc were more robust to the increase in imbalance, and are therefore more appropriate under these circumstances.
Moreover, our data confirms that Random Forest is more robust when handling imbalanced data, compared to other commonly used algorithms, such as LR, LDA, and SVM-especially when using class-weighting hyperparameter optimization. This result is expected, but our analyses quantify this for a wide range of imbalances and illustrates the effect with simulated data as well as EEG and MEG recordings. We also found that the balancing hyperparameter can be used to improve LR's robustness to data imbalance. Our study also highlights an important caveat concerning the use of permutation tests on imbalanced data. tests allow data-driven computation of the chance level, and from this chance level, they provide an estimate of statistical significance. However, because chance levels can be much greater than 50% in imbalanced data (for binary classification problems; Fig.
By contrast to some of the previous studies, the present work focuses on insights that are directly relevant to researchers in neuroimaging using standard tools, such as those available through the scikit-learn library. In addition, the code and Jupyter notebooks (https://github. com/thecocolab/ data-imbalance) were designed so the figures could be easily replicable, and to extend the investigations to an even wider range of metrics and methods, in a collaborative and open science perspective. Recommendations In discussing recommendations and best practices for handling data imbalance, it is important to note that the utility or suitability of a metric should always be determined in relation to the specific problem being tackled. The various types of machine learning problems differ among other things in the type of error one seeks to minimize-this will in turn determine the appropriate classification paradigm in which to operate.
Use Balanced Accurary (BAcc). BAcc has been largely underexploited in neuroscience research. Given (i) its superior robustness to imbalance, (ii) the fact that it simply reduces to Accuracy for balanced datasets and (iii) its applicability to both binary and multi-class  [61]  datasets, we recommend the routine use of BAcc, rather than the commonly used Acc, as a default for neuroscience machine learning applications where overall classification error should be minimized. Use ensemble methods. If data imbalance cannot be avoided, we recommend the use of classifier families that provide additional robustness. In line with previous recommendations, ensemble methods such as Random Forest and decision trees are less sensitive to data imbalance and provide a set of hyperparameters that can be optimized for the classification of imbalanced data.
In this study, we focused on families of models and performance metrics that are easily accessible and widely used in the neuroscience community, in particular through the scikitlearn library. Specifically, we focused on four popular classifiers and four standard evaluation metrics, as well as two cross-validation schemes. Reviewing or comparing all existing tools is beyond the intended goal of this paper, but the Python code we provide is open-source, which allows users to extend these investigations. Second, the data we used consisted of simulated Gaussian data distributions, as well as open-access electrophysiological brain 25 signals (MEG and EEG). We did not examine the impact of noise, though it could have been interesting to consider it, as it has been shown to interact with the performance of the classifier  [62].
In addition to its robustness to class imbalance, Balanced Accuracy collapses to standard Accuracy for balanced datasets and is readily extendable to multiclass problems. More generally, we hope that the recommendations and red flags reported here-using simulations and real brain data-will strengthen good practices for the application of supervised ML in neuroscience, and increase awareness, especially among new-comers to the field. Lastly, by providing open-source code and well-documented pipelines, we hope that others will further explore this question with a wider variety of parameters, classifiers, and different types of classification problems.
Acknowledgements KJ was supported by funding from the  Canada Research Chairs program  and a  Discovery Grant  from the  Natural Sciences and Engineering Research Council of Canada (NSERC) , and  IVADO-Apogée  fundamental research project grant. YH was supported by the  Courtois-Neuromod  Project. CM was supported by the  FRQNT Strategic Clusters Program  ( 2020-RS4-265502  - Centre UNIQUE -Union Neurosciences and Artificial Intelligence -Quebec  and the  Canada First Research Excellence Fund and Fonds de recherche du Quebec , awarded to the  Healthy Brains, Healthy Lives initiative at McGill University. HA was supported  Mitacs Globalink Research Award. JOB was supported by the  Canadian Institutes of Health Research (CIHR). AD was supported by  Mila. AK was supported by  Mitacs Graduate Fellowship and Courtois foundation.
In spite of that, even for senior developers, manually recovering the links between Pull-Request and Issues from evolutionary development history is a time-consuming, challenging, and errorprone task. In this article, we propose to build ML models to recover links between pull-requests and their issues using two Machine Learning algorithms (KMeans and BIRCH) based on lexical and semantic weighting measurements. These models are evaluated using PI-Link ground-truth dataset. The obtained results show that pull-request and issue links can be recovered with an accuracy of 91. 5% using BIRCH clustering algorithm.
I. INTRODUCTION In social coding platforms, such as GitHub, when a contributor finds an Issue, he informs other contributors of the hosted repository on GitHub via the issue tracking system  [1]. An Issue is either a bug, feature enhancement, new feature, or any maintenance task. On GitHub, the integrated Issues tracking system allows the Issue submitter to provide textual information about the detected Issue containing a title and description. After reviewing and assigning Issues to internal developers (responsible developers), they are required to promptly respond to Issues identified in Issue trackers and The associate editor coordinating the review of this manuscript and approving it for publication was Ines Domingues. resolve such Issues with the least amount of work. When the number of daily submitted Issues rapidly grows up, especially in popular repositories, managing Issues becomes more complicated, adding to developers heavier workloads  [2].
Nowadays, machine learning (ML) and deep learning (DL) achieve promising results in different tasks of natural language processing (NLP), such as clustering, prediction, sentiment analysis, etc. [15]. Many ML and DL algorithms can build models fed with datasets to discover the representations needed for detection or prediction automatically. To explore the possibilities of automatically recovering the missing PR-Issue links using ML, we conducted an empirical study to identify the characteristics of explicit PR-Issue links based on their textual representations. We found that the links between PR and Issue exhibit certain features. For example, the titles, body descriptions, and comments for Issues and their corresponding PRs share varieties textual similarities. In this article, we propose to build ML models to recover PR-Issue links using two ML algorithms (KMeans and BIRCH) based on lexical and semantic weighting measures.
We selected these algorithms rather than other ones for two reasons: The first reason is their capability to predict the clusters for new datasets. The second reason is that the number of clusters is known, which is the number of PR-Issue links in our case. Therefore, overcome the problem to determine the best number of clusters faced by the other clustering algorithms (the local optima problem  [14] ). The effectiveness of these algorithms is evaluated using a ground-truth dataset on the subject called PI-Link  [15]. The experimental results show that the BIRCH algorithm outperforms the KMeans algorithm in terms of the most widely used metrics in this domain: homogeneity, completeness, and V-measure. Moreover, three features of PRs and Issues are used to build clustering models: Title, Body, and Comment.
According to the experimental results, the Title and Body features are more important in this recovering task than the comment feature. The rest of this paper includes the following sections. In section II, we provide the necessary background to easily understand the proposed approach. Section III provides a detailed explanation of how the clustering models are built. In section IV, the clustering models are experimentally evaluated with a ground-truth dataset. Related works are discussed in section V. Finally, the article's conclusions and future work are drawn in section VI.
B. PR-ISSUE LINKS IN GitHub PRs and Issues in GitHub are the main important assets for collaborative distributed development and maintenance activities. On the one hand, Issues are used for different purposes, such as asking questions, signalling bugs, and other maintenance tasks. They are typically stored in the Issue tracking system. On the other hand, PRs are used as a way to enable other collaborators and reviewers to submit their code changes. They are managed by the version control system. Also, both Issues and PRs represent artefacts of different domain spaces, as Issues represent problem space (requirements, features, bugs, etc. ) while PRs represent solution space (fixing bugs, implementing a new feature, etc. ). In fact, linking PRs and Issues is also a link between solution and problem spaces.
As a result, such links have different semantics. It can be a documentation link between features or requirements and their corresponding code  [20]. Also, it can be a maintenance link between bugs and their solutions  [21] ,  [22]. The links between PRs and Issues may be established in the same repository or across different repositories. When PRs are approved and merged into the main branch, the state of their linked Issues is changed to closed (solved). There are two ways to link PRs and Issues: implicitly and explicitly  [23]. In the implicit way (as shown in Figure  2 ), GitHub provides keywords to link PRs and Issues (e. g. , resolve(s,d), close(s,d), and fix(es,ed)). These keywords are placed in the PR text (either description or the commit message). This way has two main concerns. Firstly, removing or changing PRs-Issues links is a time-consuming and boring process as the owner requires manually modifying the PR's description to remove or modify the link's keywords.
C. CLUSTERING ALGORITHM Clustering is the process of grouping similar items (based on patterns or structural properties) as a cluster. Clustering algorithms are unsupervised learning algorithms where the datasets are unlabeled. In the literature, there are many clustering algorithms such as, KMeans, BIRCH, DBSCAN, Spectral clustering, and hierarchical clustering  [24]. Each one has its own considerations. One of the main considerations depends on the number of clusters provided in advance. For example, the KMeans and BIRCH algorithms require a predefined number of clusters to provide, whereas the DBSCAN and hierarchical clustering algorithms do not need this predefined number of clusters. Another important consideration is whether the algorithm is able to predict clusters for an untrained (unseen) dataset.
For example, the KMeans and BIRCH algorithms can be used to predict the clusters for new datasets, whereas the DBSCAN algorithm can not be used for this purpose. In anyway, the ground-truth dataset is important for unsupervised learning techniques to compare the prediction accuracy of different machine-learning (ML) models and specify the appropriate one. In this article, we rely on KMeans and BIRCH algorithm to recover (cluster) PR-Issue links that are best fit to the requirements of the selected ground-truth dataset. The KMeans is an unsupervised ML algorithm. It is used to group data points into a predetermined number of clusters (K). It starts with K randomly chosen items acting as the clusters' centroids. After that, the algorithm iteratively calculates the total of the squared distance values between the data points and the centroids of the established clusters.
A data point is assigned to the cluster whose centroid is the closest to that data point. After that, it computes a new centroid for each cluster until the best centroid is discovered  [25]. Like KMeans, BIRCH is driven by a predefined number of clusters. BIRCH stands for Balanced Iterative Reducing and Clustering using Hierarchies  [26]. It converts data points into a tree data representation with the centroids. These centroids can be the final cluster centroid or the input for other iterative clustering round. BIRCH starts by clustering a given dataset into small summaries in one-time scan. Then, the summary can also be clustered by other clustering algorithms, such as agglomerative hierarchical clustering.
Then, the textual content of PRs and Issues are parsed to extract only the features of interest. Next, PRs and Issues go through preprocessing steps for cleaning. Then, lexical and semantic similarities are computed for PRs and Issues vectors. Finally, these vectors are clustered using KMeans and BIRCH algorithms to recover PR-Issue links by building clustering models. The recovered links are evaluated with ground PR-Issue links that existed in the ground-truth dataset. Figure  4  gives an overview of this recovery process. Below, we detail the steps required to build clustering models using KMeans and BIRCH algorithms.
In each TF-IDF matrix, rows represent PRs and Issues vectors, columns represent all tokens in the feature's corpus, and the content of this matrix represents TF-IDF weights. This is performed using ''Tfid-fVectorizer'' method provided by ''scikit-learn'' library. The similarity between PRs and Issues vectors is computed using the cosine similarity according to the equation 1. In this equation, V1 and V2 refer to vectors of PRs and Issues. Cosine(V 1 , V 2 ) = cos(θ) = -→ V 1. -→ V 2 ∥ -→ V 1 ∥∥ -→ V 2 ∥ (1) For the semantic weighting measure, we use the most recent semantic measure in natural language processing (NLP) named BERT (Bidirectional Encoder Representations from Transformers). It is a pre-trained language-based transformer model that is developed by Jacob et al. in Google  [30]. BERT is a semantic model representation based on deep learning that learns contextual associations between words placed in the same text.
D. CLUSTERING ALGORITHMS In this step, we apply clustering algorithms which are fit to our case. We find KMeans and BIRCH algorithms the best fit for our case (refer to background section II for further details about these algorithms). We adapted these clustering algorithms as follow. In both algorithms, the number of clusters is predefined, which is the number of linked issues. Also, the data points in both algorithms are issues and PRs vectors which are build in the previous step. For each issue vector, the clustering algorithms find PRs' vectors that are the closest to that issue according to the considered features (i. e. Title, Body, and Comment). The quality of predicted clusters can be compared with the ground-truth ones. The obtained Issue-PR clusters are compared with their corresponding ground-truth Issue-PR clusters (I.
A. GROUND-TRUTH DATASET The considered clustering algorithms are applied to a ground-truth dataset in the subject (available on: Kaggle 3  ). This dataset includes 5742 Android projects which have PR-Issue explicit links. Table  1  illustrates some of the statistical information about the ground-truth dataset in the subject of the total number of Issues and PRs across all projects in the dataset. The rest statistical information is min, max, mean, and standard deviation (Std) but at the level of projects. For example, the dataset has 34732 Issue titles across all projects. Some of these projects have just one title (i. e. contains just one Issue), whereas a project has 974 titles (i. e. contains 974 Issues). The average and Std for the number of Issues overall projects are 6. 04 and 30. 13, respectively.
The table also shows that the dataset has 34732 Issues linked to 50369 PRs. The difference in the counts of PRs and Issues is due to the cardinality ratio relationships between linked PRs and Issues, where a PR may be linked to many Issues and an Issue may be linked to many PRs. Table  2  represents the number of empty data found in the ground-truth dataset. We observe that each PR and Issue has a title, but a few do not have a body (141 out of 34732 in Issues, and 129 out of 50369 in PRs). Besides, 1296 Issue and 8 PR do not have a comment. The reason behind this is that the PRs usually go via a review and discussion process utilising PR comments, whereas this process is not mandatory in case of Issues. Finally, we can notice that around 8% and 9% of Issues and PRs without labels, respectively.
B. RESEARCH QUESTIONS AND EVALUATION METRICS To quantify the accuracy of the recovered links, we designed experiments for answering the following research questions: The goal of this research question is to measure the correctness of recovered links (clusters) between PRs and their associated Issues. To do so, we apply these clustering algorithms on our dataset after removing all links between PRs and Issues. This means that the aim of the clustering algorithms is to recover these removed links. We measure the correctness using the Homogeneity metric. The Homogeneity measures the percentage of how far the resulting clusters include similar members. The homogeneity score is calculated based on Equation 2  [33]. homogeneity = 1 - H (C|K ) H (C) (2) where H (C|K ) is the conditional entropy of the classes (i.
C. CLUSTERING CONFIGURATION To answer the RQs, we applied KMeans and BIRCH algorithms to the ground-truth dataset seven times. The first time, we trained one-sevenths of the dataset, and each time the dataset is doubled (i. e. , the second time is 2/7, the third time is 3/7, etc. ). This method aims to monitor the clustering models' performances over diverse dataset sizes. Moreover, it helps to overcome biased datasets and overfitting. For each experiment, we compute homogeneity, completeness, and v-measure scores for the selected features (the text of PRs and Issues titles, the text of PRs and Issues bodies, and the text of PRs and Issues comments). These scores are measured by using the sklearn. metrics module  [34] ,  [35] ,  [36]. Table  3  demonstrates the parameters that we used for KMeans and BIRCH configurations.