dataesr/scientific-paragraphs-categorization huggingface dataset:
e. , whether developers still improve the products or whether it is no longer maintained. Project maintainers often setup issue trackers to record and manage issues submitted by software users. Development context: Finally, a software project is an evolving ecosystem. Besides the creation date, important information on the activities in a project include the last commit date, i. e. , the date at which a code change was last introduced. The size of the development team is also an important criterion for differentiating projects, as the number of authors involved in the project may have an impact on the quality of the code base, both in terms of functionalities and of bug management. Information on the development context can be retrieved from version control systems such as git  3  or subversion  4  which are used to setup project code repositories.
Indeed, once all information related to a software project are integrated, a project seeker should be empowered to construct rich queries that exploit the diversity of information types to accurately identify projects that are relevant to his needs. Where traditional search engines cannot uncover the "intended meaning" of flat queries, a dedicated project search engine should capture the semantics of structured and flexible queries. To meet these requirements, we have designed and implemented the Orion integrated search engine. Orion includes an extensible and maintainable knowledge database of software projects with information retrieved from a variety of sources, and is complemented with a declarative language for allowing project seekers to compose expressive queries to easily search across the database.
III. ORION Figure  2  illustrates the overall approach of Orion for addressing the challenges in searching software projects. The main objective of Orion is to reduce the gap between user's search intent and the interpretation that search engines associate to search queries. To this end, we propose to use a domain-specific language to formulate search specifications that describe the criteria for selecting relevant projects. The queries written with the high-level Orion DSL are translated into low-level advanced database queries where several information tables are joined to select, from a knowledge database, a set of projects. This knowledge database of projects is constructed and maintained by crawling the web for project code and development artifacts. In the following, we provide implementation details of these aspects of Orion.
Ideally, project seekers should be enabled to enter complex queries in natural language which would be processed to extract the semantics of search requests in order to assess whether these can be served by the underlying database. However, the value-added of such elaborate engines is disproportionate compared to the implementation efforts that are required to deliver such services for the very narrow domain of project search. Nonetheless, allowing expressive and intuitive queries remain essential for improving the user-experience in the quest for software projects. Towards supporting semantic query for searching software project instances, we propose a domain-specific language (DSL) which is built to capture the characteristics of software projects, the variety of search criteria, and the readiness at which users can navigate across project data.
1) Harvesting the web: The World Wide Web has endless amount of information on software development projects. With the momentum on open-source development, a number of platforms have emerged to offer project hosting services. Sourceforge  5  and Google Code  6  are popular examples of such platforms that mainly are built around source code repositories while providing other tools such as bug tracking systems. Recently, the concept of social coding has lead to the success of a new kind of development platforms with developerfriendly interfaces  [23]. Instances of this concept, including Atlassian Bitbucket 7  and GitHub 8  , have increasingly outperformed pioneering platforms. For example, in June 2012, after only 4 years of existence, GitHub is hosting over 3,000,000 repositories, while Sourceforge, launched 13 years ago, contains only about 360,000 projects.
Nevertheless, traditional hosting platforms are still used for many projects, and provide various useful features, including extensive project labelling, that are exploited by developers. Strong organizations, such as the Apache Software Foundation  9  or the Linux community  10  , host the development of their projects on their own portals where large amounts of data are available for download. Nonetheless, automatically identifying the homepages of these projects on the web and building tools to parse these web pages to extract project information is challenging. Fortunately, the popularity of hosting platforms and their practical aspects have incited many project managers to register their projects and provide mirrors on GitHub-like platforms, making them easily identifiable. In this work, we use data collected from GitHub to build the main corpus of software projects for our database.
a) GitHub -a software project corpus A decade ago, the task of identifying and collecting a huge number of projects would have been extremingly tedious, due to the scarcity of repositories containing an inventory of projects in development. But, today there are rich assets available that can be harnessed for automatic project information harvesting. Information integration on these platforms is however lacking on such assets, a gap that Orion aims at filling. GitHub is very convenient for repository mining as it provides extensive REST API  11  for accessing its internal data stores. We have used the API to retrieve general information 100,000 repositories hosted by GitHub. Fig. 7  summarizes the distribution of the GitHub projects in different segments of number of LOC. Over 70% of projects contain more than 1,000 LOC.
Around 35% of projects include more than 5,000 projects while more than 20% contain more than 10,000 LOC. Finally, over 600 projects contain more than 1,000,000 LOC. This distribution suggests that a significant number of the projects in the dataset are actually real-world projects. % of Projects 0 20 40 60 > 1,000 > 5,000 > 10,000 > 50,000 > 100,000 > 500,000 > 1,000,000 Fig. 7. Distribution of projects in the datasets in terms of total Lines of code The information collected from GitHub for each project includes a description of the project, the organization/company developing the project, the GitHub user who owns the project, the project homepage, and its name and URL. GitHub also implements bug trackers for each repository and exposes information from these trackers through its APIs. Fundamentally, GitHub bases its operation on git  12  , a distributed revision control and source code management system, which keeps track of all contributors by differentiating between revision authors and committers-an important feature in a hierarchical development scheme, such as in Linux, where contributors do not have direct access to the mainline repository.
Nonetheless, the software project corpus thus constituted presents two limitations: (1) it does not contain data on all possible information types for all selected projects. GitHub is indeed developer-oriented and its features may not favour a categorization style where users could identify and select project following the content. Other platforms, which are more user-oriented, may provide better facilities for this particular setting. (2) For some of the project data that are available in hosting platforms, further steps, including downloading raw data, processing and counting, are required to expose relevant information types.
b) Reaching out to other platforms Aside from projects whose development essentially occurs on GitHub, many others are simply mirrored on GitHub to benefit from broader community exposure. For such projects, the corresponding GitHub repositories lack essential information. For example, the description of the project is often summarized (e. g. , "Mirror of Apache CouchDB"  13  ) and bug trackers are often disabled as the development occurs outside of GitHub. As GitHub is meant for collaborative development, singleman projects for delivering small utilities are often hosted on other platforms such as Google Code and Freecode 14. Google Code furthermore provides extensive tagging facilities to developers for labelling and categorizing their projects. These tags can be relied upon to improve query results.
Due to these considerations, we also collect data for 50,000 projects from Google Code and for 35,000 projects from Freecode. All these project sources provide common information types, such as name and source code, which are already available in the corpus in the case of projects that are also hosted in GitHub. In such cases, the corpus is augmented with information types that are not available in GitHub. However, when a project is only available from one source, the queryable information types are restricted to those available in the dataset. Nevertheless, common queries, including two among the three search scenarios illustrated in this paper, require information types that are common to all platforms.
c) Crawling the web and beyond While popular software hosting platforms provide bug tracking systems with common features, most of them cannot link bug reports with patches submitted in software revision control systems. GitHub provides a pull request feature that can be harnessed to infer those links in cases where the requests are actually accepted by the repository owner. However, the results are often limited to the small set of patches by developers who have no direct access to the main repository. To compensate this deficiency, JIRA, a commercial bug/issue tracking system, provides add-ons for connecting issues to revision control systems and is used by hundreds of projects from the Apache Software Foundation 15. To collect bug links, we crawl the web based on the specific URL format of JIRA.
Finally, we note that other important information for the success of queries, as illustrated by our motivating search scenarios, are not apparent on the hosting platforms. These information include the number of lines of code which can only be inferred after an offline processing of source code. The real distribution of programming languages used in the projects also requires a refined investigation of the source code of each project to differentiate the main language from appearing languages. The source code tree structure from which we deduce e. g. , the extent of test cases is also not available directly from project page. Instead, we download every source code repository ourselves in order to explore these properties. 2) Constructing the knowledge database: For the construction of our prototype knowledge database, we have harvested from the collected software projects different types of information and artifacts.
This step has required processing project description data, project labels data, code commits, issue/bug reports, source code (for computing the LOC in all languages, and surveying the source tree structrues), etc. In cases where we merge data from several hosting platforms, since project names are unreliable, we infer the correlation among datasets based on the project's URLs. For example, GitHub repository API provides a homepage field where developers can indicate e. g. , the Google Code URL from which the project is mirrored. Though we already support a 14 freecode. com 15 issues. apache. org dozen information types  16  , extending the database to include additional information should be readily possible. Finally, the knowledge database contain raw information to ease the maintenance of the update of the knowledge database.
Search scnearios that request mean values or other combined information are handled on-the-fly by the underlying engine using virtual tables, such as SQL views. Resource-consuming tasks that were performed to fill the database can now be performed on a small scale for each evolved project. For example, the change of source lines of code can directly be inferred from the new commits without any need for recomputation in the entire source code. After 1 week of continuous data collection, we have processed about 1. 5 terabytes of raw data to fill our MySQL database with 2 gigabytes of harvested information data. To ensure query performance, we have referred to the best practices in MySQL database tuning, and created lookup indices in all tables.
DB request generation: In the current implementation of the database, the specifications are directly translated to generate appropriate SQL statements which look up multiple tables to serve search requests. Each attribute of the Orion DSL representing an information type that is mapped to a specific table, the Orion compiler iteratively joins the relevant tables that are necessary to answer the requests. A few informations types such as the url are used as keys in the predicates of the JOIN statements. User search preferences are also translated into GROUP BY or ORDER BY statements depending on the specification. Advanced queries that lead to nested SQL statements are also supported. Figure  8  provides an illustrative example of the generation outcome for an Orion basic search specification.
The specification outlined in Figure  4  was previously used to request researcher datasets in Section II-A where the query relates to information on project organization, on the reported bugs, and on the test cases in the source code. The generated SQL request, in Figure  8 , shows that 3 tables (repos info, issues, test cases), corresponding to different information sources, are joined to serve the request. The use of the Orion DSL achieves two aims: (1) firstly, the Orion DSL abstracts the internal structures of the knowledge database which can become challenging to comprehend; (2) secondly, the DSL allows users who may ignore the underlying requirements of database management systems to easily tune the search criteria, sort the output, and limit the extent of output processing. In our case, for common search scenarios, querying the knowledge database does not require any knowledge on SQL-based relational databases.
Nonetheless, project seekers that are knowledgeable in SQL, can survey the database schemas and construct their own SQL requests to query the knowledge database. SQL request generated by the Orion compiler for research query (see Orion query in Fig. 4 ) Table  I  shows the result of the researcher query (cf. Fig. 4 ) that were returned by the Orion search engine. Table  II  describes the output of the developer request (cf. Fig. 5 ). The user submitted an advanced query whose execution ran across all records of projects and their contributors to identify small-sized projects where some of the developers are also involved in much larger teams. The output provided following the specification of the requester does not make apparent the complex combination (cf. Fig. 9 ) of data sources that was required to resolve the query.
17  Fig. 9. SQL request corresponding to a relatively-advanced search query (cf. Orion query in Fig. 5 ) 17 Due to space limitations, we leave detailed descriptions on the project page In Table  III  we show the results of the programmer request. The results are comprised of projects with the "opengl" label with code licenses that are neither GPLv1 nor GPLv2. The query was answered by aggregating project data from GitHub, Google code and source code raw data. The engine exploited the license annotations and the tags provided in Google code by the developer teams. The generated SQL request for this query involves the joining of several tables and an extensive checking of multiple conditions. IV. ASSESSMENT In this section, we conduct an evaluation of our approach and of the prototype implementation of Orion.
To do so, we perform a user study using the software projects used to build the Orion knowledge database. We explore two aspects: (1) need, assessing the opportunity of the Orion approach compared to traditional search engines, and (2) productivity, assessing users satisfaction with the results provided by Orion, in terms of query delay and result conformance. We study these aspects through the following research questions: RQ1. Need: Is Orion needed despite the existence of numerous search engines and project hosting platforms? RQ2. Productivity: Does Orion improve productivity, i. e. do project seekers find the results quickly and accurately? We perform our measurements based on the attitudes of service requesters towards it. To this end, we rely on a psychometric scale, namely the Likert-type scale  [18] , which specifies the degree if a user agrees or disagrees with a particular issue.
e. , 25% of the data points are below this line, and the UPPER QUARTILE (25% of the data points are above this line). The bold line in the box plots indicate the MEDIAN (the middle of the dataset). The isolated points (small circles) below the Orion box represent outliers. The horizontal line above the "traditional search engines" box indicate the MAXIMUM (the greatest value). With Orion, users have found some or all the projects requested (scales 4 and 5) while, on average, they could not find any (scale 2) using traditional search engines. We furthermore use the Mann-Whitney-Wilcoxon test  [16] , a non-parametric test, to assess the statistical difference in the distributions of user responses for Orion and traditional search engines. We find that the distributions of user responses are different at a significant level of 0.
q q 2 3 4 5 Fig. 11. Assessing the productivity of Orion Project seekers find, with Orion, accurate results in a relatively limited period of time. V. RELATED WORK We discuss in the following a number of studies related to the collection of project data, to project and code search, and to semantic search implementations. Collection of software project data. In  [21] , Nagappan has reported on the discussions of a working group that describes the opportunities and challenges in using open source software repositories for empirical studies. In the past years, significant effort has been spent into collecting, curating, and analyzing data from open source projects around the world. The FLOSSMole project 19 , initially collecting Sourceforge projects, includes datasets from various sources in various formats.
The Flossmetrics  [7]  and Sourcerer  [1]  projects furthermore provide statistics on their collected data. These global statistics are not suitable for selecting a subset or identifying a unique project based on desired properties. Bird et al. have provided insightful discussions on the opportunities and challenges of mining Git repositories  [2] , while Dabbish et al. have presented a study that investigates the impact of transparency in GitHub through a series of interviews  [5]. More recently, the GHTorrent project aims at bringing GitHub's rich product and process data to the hands of research community  [9]. Unfortunately, they do not improve this collection for the many projects whose real development 19 flossmole. org activities occur outside of GitHub. The FRASR framework for analyzing software repositories has mostly focused on addressing the challenges for data collection and curation  [22].
They do not exploit the large datasets in GitHub and its clean APIs. The TA-RE project aims to facilitate sharing of benchmark datasets among researchers  [15] , through an exchange language. Overall, our work improves over previous efforts by merging various artifacts from various sources, and provides the capabilities for executing complex queries that can yield more relevant search results for different types of project seekers. The SeCold  [13]  platform links data sources and creates facts, which are analysis results published by developers who gathered them with their own tools. Similar to SeCold, Orion integrates and links a variety of information. The BOA  [6]  language and infrastructure also aim at improving MSR studies by reducing the workload and increasing the efficiency in computing statistics (e.
g. average size of commits) across repositories. Orion, on the other hand, is targeted at the selection of specific projects which for search purposes are considered as entities with rankable properties. These two approaches thus target different usage scenarios. Additionally, different from BOA, Orion integrates different sources of information including: GitHub, Google Code, JIRA, etc. Projects, applications and code search. Project hosting platforms such as GitHub offer search facilities in their user interfaces and through their APIs. The capabilities of these search engines are however limited to a few information types (e. g. , search by language), and do not allow advanced queries across on multiple criteria. Finally, search results are returned as unstructured text  [20]. CodeFinder  [11] , CodeGenie  [17] , CodeBroker  [24] , Examplar  [10]  and PortFolio  [20]  identify and return relevant projects or functions by mining source code repositories.
Orion on the hand is focused at searching for project entities based on different search criteria for diverse development artifacts. Similar to our project discovery infrastructure, the Ohloh 20 graphical interface, provides users with updated ranking of projects (e. g. , popular projects, most active projects, etc. ). Orion is more powerful as it considers many more development artifacts. Thus, a much wider variety of search queries is supported. Semantic search. Real world implementations of semantic search have been deployed. Example such as Wolfram Alpha, focus on a restricted domains for more accuracy. The Semantic MediaWiki 21 framework includes a simple query language for semantic search, so that users can directly request certain information from the wiki database. Orion follows the same approach by providing a user-friendly language for specifying project search requests.
VI. CONCLUSION & FUTURE WORK In this paper, we have introduced Orion, a unified platform for searching relevant software projects leveraging various data sources including source code and revision control information, bug reports and developer activity. We have downloaded and processed software development artefacts for tens of thousands of software projects on popular hosting platforms 20 http://ohloh. net 21 semantic-mediawiki. org and crawled development artifacts data on the web to build our prototype knowledge database. Orion enables the execution of various complex queries that are not supported by traditional web search engines. Orion furthermore comes with a language target at project search. We have demonstrated the need for Orion as well as its productivity by conducting a user-study.
The results indeed show with strong statistical significance that users find relevant software projects faster and more accurately with Orion. This work is being expanded in various directions for (1) enriching the knowledge database with more software projects, (2) exposing more information types for queries and (3) improving the engine towards the implementation of a transparent semantic query. Availability: Materials for this paper, including information on the collected projects, the database, the user study, etc. can be found at http://momentum. labri. fr/orion.