dataesr/scientific-paragraphs-categorization huggingface dataset:
Indeed, projects in GitHub produce software from various application domains, from web applets, gaming software, to operating systems. These projects are conducted by teams of developers of varying sizes-between 1 and several thousands, and include source code from a myriad of programming languages. While our study focuses on the issue trackers managed by the different projects, there is a need to correlate statistical findings on issues with different characteristics of the projects. Thus, we are required to (1) collect a huge amount of data from GitHub and (2) infer important information, such as a project's number of lines of code, that is not directly available through the API, making a large study with GitHub challenging. Other project hosting platforms, including SourceForge, GoogleCode and Freecode, also provide issue trackers to hosted projects.
III. METHODOLOGY For our empirical study, we consider the first 100,000 projects returned by the GitHub API following a search request on available project repositories. These projects were retrieved with GitHub API v2 which returned a random list of projects  4. We collect our dataset based on information from those projects. The obtained dataset is furthermore curated to distinguish projects that can introduce bias in our analysis from others: e. g. , some projects are hosted in GitHub but maintain their issue tracking system outside of GitHub. Last but not least, we statistically characterize the popularity and usage of issue trackers in several dimensions, relying on well-known metrics to assess the statistical significance of our findings.
A. Collecting the dataset For each project in our large set of projects from GitHub, we collect data related to a number of important software artefacts that commonly occur in development processes and are leveraged in this study. a) Project information: The GitHub social coding site provides a number of features that enable developers to monitor and track activities in a project repository. In this study, we focus on two concepts, namely watchers and forks, that can be used as metrics for evaluating the popularity and thus the success of a project. The "watchers" metric gives an indication of the amount of attention that is given to a project by the developer community. Developers who subscribe as watchers to a project typically use, report bugs and incidentally promote the project in their developer social network.
Because GitHub uses the git 5 software configuration management system (SCM) to store software revisions, we download the actual git repository for the project, and rely on the SLOCCount 6 utility to compute the actual lines of code in the latest revision of the project, ignoring code comments and blank lines. d) Developer contributions: Finally, we consider for our study the development team for each project. Because git records code contributors' names and email addresses with each revision in the repository, we are able to accurately identify all contributors to a project code base, whether they are registered to GitHub or not. Indeed, the git SCM distinguishes between revision authors, who are the end-contributors to the code base, and committers, who have access to the mainline repository and can forward contributions from revision authors.
B. Curating the dataset GitHub is a hosting site that is used by different project teams to get exposure in the developer community. Thus longlived projects whose active development has started before the era of GitHub keep managing their projects on their own development sites while mirroring their source code repository in GitHub. Examples of such projects include the numerous projects developed under the auspices of the Apache Software Foundation and the Linux kernel mainline tree. These projects disable the issue tracker provided by GitHub to avoid confusion on which tracker is maintained by the project development team. Thus, based on this information we remove from our study the 3,801 projects (out of the 100,000) for which the issue trackers were disabled. We further investigate the list of projects, which include popular projects such as the Linux kernel or the Ruby on Rails framework, to ensure that most of the projects are non toy projects of substantial sizes.
To this end we count the total 5 http://git-scm. com 6 http://dwheeler. com/sloccount lines of source code (LOC) in each project. Over 70% of the projects contain more than 1,000 LOC. Around 35% of the projects include more than 5,000 LOC, while more than 20% contain more than 10,000 LOC. Finally, over 600 projects contain more than 1,000,000 LOC. This distribution suggests that a significant number of the projects in the dataset are of substantial sizes. To further ensure that most of the projects are not toy projects typically created by GitHub new users, we have filtered the data to projects with at least 1,000 LOC.
C. Research questions Research and practice of software development is performed under various assumptions about issue tracking that have not been validated through extensive empirical studies. Thus several hypotheses can be made, and accepted, that we will attempt to (in)validate in this work: H1 -Open source projects receive large numbers, e. g. , over 1000, of issue reports. H2 -During a software project lifecyle, developers write code for which a distinct community of users report issues. Thus we expect less than 50% of issue reporters to be among project developers, and vice versa. H3 -Collaborative coding environment has a positive impact on issue reporting. H4 -The more people are interested in a project and submit issue reports, the quicker project developers handle them. Given the extent of our dataset, we investigate in this work the following research questions to explore and assess the relationship between various aspects of software projects and the activities in their issue trackers: RQ1.
RQ5. What is the relationship between utilization of issue tracking system and project success (i. e. , number of forks and watchers)? We would like to establish and assess the correlation between the success of a project and the number of issue reports that it receives. RQ6. Does the size of user communities impact the timeto-fix rates of bugs? In this research question, we investigate on a larger dataset of projects a result of previous studies based on a unique development project, namely Firefox.
D. Statistical measurements Our empirical study is based on a sample of projects. Though, to the best of our knowledge, no related study involving issue trackers has ever exploited that many projects, there is a need to ensure that, statistically, our findings are significant. To this end, we resort to common metrics in statistical analysis for assessing the statistical significance of our figures, and to confirm the existence of a correlation among data from compared artefacts. a) The Mann-Whitney-Wilcoxon (MWW) test: The MWW test is a non-parametric statistical hypothesis test that assesses the statistical significance of the difference between the distributions in two datasets  [18]. We adopt this test as it does not assume any specific distribution, a suitable property for our experimental setting.
Once the Mann-Whitney U value is computed it is used to determine the p-value. Given a significance level α = 0. 001, if p-value < α, then the test rejects the null hypothesis, implying that the two datasets have different distributions at the significance level of α=0. 001: there is one chance in a thousand that this is due to a coincidence. b) Spearman's rho: Spearman's rho (ρ), also known as Spearman's rank correlation coefficient, is used in statistics as a non-parametric measure of statistical dependence between two variables X and Y. This measure is used without any assumption that the data is normally distributed, making it a good fit for the datasets that we investigate in this study. The values of ρ are limited to the interval [-1; 1], and a perfect Spearman correlation of -1 or +1 occurs when each variable is a perfect monotone function of the other.
A. RQ1: Adoption of Issue Trackers In the first research question we investigate the distributions of issue trackers across projects. By default, a developer creating a project in GitHub, is provided with a source control repository and a corresponding issue tracker for the project. Nonetheless, the developer can disable the issue tracking making it inaccessible to users. Issue trackers are disabled in only 3. 8% of the projects. About 30% of the projects have issue trackers containing some issues, while 66% of the projects have completely unused issue trackers. In the rest of our study, we dismiss projects with issue trackers disabled from our dataset in order to emphasize on the differences between projects that have no issue reports (albeit their issue trackers are enabled) and projects having issues.
Thus, 2 projects out of 3 do not have issue reports though their issue trackers are enabled. We thus investigate why. To this end, we correlate the presence/absence of issues, in enabled issue trackers, with different properties of a project, including its age, the size of its code base, and the number of people in the development team. We also investigate the influence of the project's owner (in general the leader) over the presence of issues. a) Project age: We measure the age of a project solely based on its date of creation on GitHub. For each project, we count the number of weeks that have passed since it has been hosted in GitHub until June 2012. Our goal is to investigate the relationship between the duration of a project's exposure in GitHub and the presence/absence of issue reports.
05 weeks, with a median value of 49. 57 weeks, while, on average, projects with issues had been in GitHub for 81. 57 weeks with a median value of 70. 04 weeks. Using the Mann-Whitney-Wilcoxon test, we have established that the difference between the two datasets is statistically significant at 0. 001 significance level. This experiment reveals that projects with issues have been hosted in GitHub, in median over half a year longer than projects that do not have issue reports. b) Lines of code: Figure  2  shows the distribution of LOCs in project code bases. Projects with issues have a median value of 1,820 LOC, while projects without issues have a median value of 1,027 LOC. We have tested and confirmed that the two distributions are significantly different using the MWW test at 0. 001 significance level.
32. Nevertheless, the open source software projects in our dataset deal with a variety of applications domains with different complexity which can account for different bug rates. The projects also use different programming languages with disparate verbosity and a wide range of technical difficulties. We therefore investigate whether we can actually establish a correlation between the numbers of LOC and the numbers of issues. In Figure  6  we provide the scatter plot of the numbers of issues and the numbers of LOC from the projects in our dataset. We compute Spearman's rho which yielded a value of 0. 341. According to Hopkins  [12] , a Spearman's rho of between 0. 3 and 0. 4 indicates a moderate correlation.
C. RQ3: Tagging in Issue Trackers We discuss, in the third research question, the categorization of issue reports. GitHub issue trackers provide a flexible feature for tagging issues to categorize them in order to ease their management by developers. This flexibility, however, comes with a cost since issue reporters may categorize the issues with typographical mistakes, or using various idiosyncrasies. Among the 803,840 issue reports in our dataset, 218,467 (27. 17%) contain tags. Issue reporters have used in total 6,951 distinct tags to categorize their reports. There are two widespread tags, namely bug and feature, which are observed in 18. 36% and 10. 29% of the tagged issues respectively. Issue trackers are then essentially used in software projects to report bugs and request new features.
We then attempt to establish whether projects written in common languages for web programming, such as Ruby, Python, JavaScript or PHP, contain more issues than the rest. We use David A. Wheeler's SLOCCount  [29]  utility to count actual physical source lines of code and their associated language. We classify the languages according to the number of issues that are filed in all the projects that are written in each language. Figure  7  shows the distribution of the number of issues for the top-10 languages that correspond to the highest number of issues. We observe that PHP is hidden in the middle with a median value comparable to that of system-level languages such as C and C++. Ruby has a slightly higher median value, while JavaScript does not appear in the top-ranked languages. Thus, we dismiss the first phenomenon.
We proceed to investigate the correlation between the numbers of watchers and the numbers of issue reporters. The scatter plot of Figure  10  shows the correlation between the two datasets. Spearman's rho amounts to 0. 789. This suggests that there is a very large correlation between the numbers of issue reporters and the numbers of watchers. Figure  11  shows the scatter plot of the numbers of issue reporters and the numbers of forks from the projects in our dataset. We have computed Spearman's rho which yielded the value of 0. 829, suggesting a very large correlation between the number of forks and the number of issue reporters. We also compute Spearman's rho to assess the strength of the dependence between the number of forks and the number of issues as in the previous experiment. In this case, the coefficient drops to 0.
Forks vs. Issue Reporters There is a large correlation between the number of issues and the number of watchers and forks. There is also a very large correlation between the number of issue reporters and the number of watchers and forks. Thus, distributed programming positively impacts the amount of feedback (i. e. , issue reports) for a project, suggesting that this software development style should be promoted more. Our hypothesis H3 is thus validated. F. RQ6: Does the number of reporters impact the time-toclose? Finally, we investigate the relationship between the number of reporters and the effort made by developers to quickly address issues. In previous work based on issue reports, Van Liere had shown that the large number of Firefox issue reporters has lead to the reduction of the time-to-close interval for software defects  [27].
V. THREATS TO VALIDITY We have identified the following threats to validity to our study. a) External validity: Our sample set of projects, though sizeable with tens of thousands projects, may not represent the universe of all real-world projects. Furthermore, we have b) Internal validity: Second, although we have curated our datasets to remove projects whose development, in particular issue management, occurs outside GitHub, some project owners may not disable the issue trackers even though they only deal with issues reported from other channels. This can lead to bias in our survey. Nonetheless, we have manually checked that for popularly known projects, such as Linux, that maintain issue trackers in well-known locations outside GitHub, have disabled their issue trackers on GitHub. Finally, we believe that, with a sizeable sample of 100,000 projects, the number of outliers is very small compared to the rest.
In addition, we have filtered our dataset to only consider projects with more than 10 kLOC. A different threshold may yield different findings. c) Construct validity: Finally, we have relied on heuristics for estimating the popularity of a project leader, the success of a project, and the application domain of a project. As we have described for each, those are simple proxies to the ground truth. However, they provide useful metrics and good indications on the popularity of each project, the popularity of each project owner, and the domain of a project in GitHub.
VI. RELATED WORK GitHub, as a social coding site that hosts millions of software projects, contains a wealth of information about the practice of software development. In previous works, data from GitHub was leveraged to conduct large-scale studies on the popularity of programming languages  [6]  and the adoption of software testing  [20]. In this paper, we exploit this data to investigate issue reporting in open source projects. To stress the importance of issue reporting we highlight in the following a number of studies related to issue reports in software development projects.
A. Acknowledging user communities A significant number of studies have discussed the importance of the feedback provided by user communities in the life cycle of a software development project. Bagozzi et al. [3] , Iivari  [14] , Hendry  [10]  and Singh et al. [22]  have investigated the role of users in open source communities. In these studies they make the point that users actually drive the software project and its evolution. Furthermore, they discuss how users influence the project towards their needs and how developers acknowledge users' input. Unfortunately, while the importance of user communities is widely accepted, the participation of users in issue tracking is more controversial in the literature. For example, considering the case of the Firefox web browser, Van Liere has concluded that large number of bug reporters reduces the time-to-fix interval for software defects  [27] , while Ko and Chalina reported for the same system that users file "non-issues that devolved into technical support, redundant reports with little new information, or narrow, expert feature requests"  [15].
C. Improving issue reporting Issue reporters may file reports that are incomplete or even invalid, adding to the challenges that developers face in software projects. A number of studies have discussed and proposed approaches to enhance issue reports to make them more useful  [23]. Hooimeijer and Weimer have proposed a descriptive model for measuring the quality of a bug report  [11]. To build the model, they assume that the "time until resolved" is a good indicator for the quality of a bug report. Bettenburg et al. have later relied on actual developer feedback to train CUEZILLA  [4] , their utility for providing bug reporters with tips to improve the quality of their reports.
VII. CONCLUSION AND FUTURE WORK Issue reports are important artefacts in software development. Various issue tracking systems have been developed and widely used, among which, Bugzilla and Jira are well-known. Hosting platforms such as Freecode, Google Code, and GitHub implement in-house issue trackers to collect user feedback. The assumed prevalence of issue reports in software development projects has led to the development of approaches for various purposes, such as improving issue reporting, enhancing issue management, and leveraging issue information for bug fixing. In this paper, we have investigated the actual adoption of issue trackers in software projects to shed lights to many research questions on the involvement of users and developers in reporting issues. We describe the findings of our empirical study on tens of thousands of open source projects in GitHub.
, a L ) (4) f ij (a i , a j ) = {a k |k =i,j} P (a 1 ,. , a L ) , (5) i. e. marginal distributions of P (a 1 ,. , a L ) are required to coincide with the empirical counts up to the level of position pairs. Beyond this coherence, we aim at the least constrained statistical description. The maximum-entropy principle  [23, 24]  stipulates that P is found by maximizing the entropy H[P ] = - a1,. ,aL P (a 1 ,. , a L ) log P (a 1 ,. , a L ) , (6) subject to the constraints Eqs. (  4 ) and  (5). We readily find the analytical form P (a 1 ,. , a L ) = 1 Z({e ij (a, b), h i (a)}) exp    1 2 i,j e ij (a i , a j ) + i h i (a i )    , ( 7 ) where Z is a normalization constant. The MaxEnt model thus takes the form of a (generalized) qstates Potts model, a celebrated model in statistical physics  [44] , or a Markov random field in a more mathematical language.
Features of the Hopfield-Potts patterns We have tested the above inference framework in great detail using three protein families, with variable values of protein length L and sequence number M : • The Kunitz/Bovine pancreatic trypsin inhibitor domain (PFAM ID PF00014) is a relatively short (L = 53) and not very frequent (M = 2, 143) domain, after reweighting the effective number of diverged sequences is M ef f = 1, 024 (cf. Eq. (  28 ) in Methods for the definition). Results are compared to the exemplary X-ray crystal structure with PDB ID 5pti  [46]. • The bacterial Response regulator domain (PF00072) is of medium length (L = 112) and very frequent (M = 62, 074). The effective sequence number is M ef f = 29, 408. The PDB structure used for verification has ID 1nxw  [47]. • The eukaryotic signaling domain Ras (PF00071) is the longest (L = 161) and has an intermediate size MSA (M = 9, 474), leading to M ef f = 2, 717.
Results are compared to PDB entry 5p21  [48]. In a second step, we have used the 15 protein families studied in  [33]  to verify that our findings are not specific to the three above families, but generalize to other families. A list of the 15 proteins together with the considered PDB structures is provided in Text S1, Section 4. To interpret the Hopfield patterns in terms of amino-acid sequences, we first report some empirical observations made for the patterns corresponding to the largest and smallest eigenvalues, i. e. to the most likely attractive and repulsive patterns. We concentrate our discussion in the main text on one protein family, the Trypsin inhibitor (PF00014). Analogous properties in the other two protein families are reported in Text S1. The upper panel of Fig. 2  shows the spectral density.
Repulsive patterns In the upper row of Fig. 3  we display the three most localized repulsive patterns (smallest, 3rd and 4th smallest eigenvalues) for the trypsin inhibitor protein (PF00014). All three patterns have two very pronounced peaks, corresponding to, say, amino-acid a in position i and amino-acid b in position j, and some smaller minor peaks, resulting in IPR values above 0. 3. For each pattern, the two major peaks are of opposite sign: ξ i (a) ≃ -ξ j (b). As a consequence, amino-acid sequences carrying amino-acid a in position i, but not b in position j (as well as sequences carrying b in j but not a in i) show large log-scores S ≃ [ξ i (a)] 2 , cf. Eq. (  16 ). Their probability in the Hopfield-Potts model, given by  (17) , will be strongly reduced as compared to the probability of sequences carrying either both amino-acids a and b in, respectively, positions i and j, or none of the two (scores S close to zero).
The eigenspace is spanned by vectors, which are localized over single sites. For a realistic MSA, i. e. without perfect conservation, degeneracies will disappear, but the features found above remain qualitatively correct. In particular, we find in real data a pronounced peak of eigenvalues around 1, corresponding to localized eigenmodes (Fig. 2 ). In addition, low-eigenvalue modes are found to be strongly localized, and the the order of magnitude of λ -≃ 0. 09 is in good agreement with the smallest eigenvalues, ≃ 0. 1, reported for the three analyzed domain families. Finally, the largest eigenmodes are largely extended, as found in the limit case above. Note that the eigenvalues found in the protein spectra, e. g. λ 1 ≃ 23 for PF00014, are however smaller than in the limit case, λ + ≃ 48, due to only partial conservation in the real MSA.