dataesr/scientific-paragraphs-categorization huggingface dataset:
Datasets for Machine Learning on Code Since machine learning requires considerable amounts of data, multiple datasets have been produced, usually as a means of validating a specific machine learning method, rather than as a principled standalone effort. This has resulted in datasets that contain input data either not far from raw text, or that contain a lossy view of the underlying analyzed software systems. Allamanis and Sutton (2013)  collected a set of over 1 billion Java code tokens and provide the code text per file for training n-gram models. Later,  Karampatsis et al. (2020)  extended this with additional datasets for C and Python; and a different extension of the dataset was provided by  Alon et al. (2018). Raychev et al. (2016)  released Py150 and JS150, two datasets of 150,000 Python and Javascript functions parsed into ASTs.
Code Datasets Several datasets focus on specific tasks, such as the BigCloneBench Svajlenko and Roy (2015) dataset for large-scale clone detection in Java. ManyTypes4Py  Mir et al. (   2021 ) is a Python dataset aimed at evaluating type inference in Python, and Devign  (Zhou et al. 2019) , provides labeled code with coarse-grained source code vulnerability detection in mind. JEMMA, on the other hand, is not task-specific and supports multiple tasks out of the box. Datasets with specific representations of code have been common. CoCoGum  (Wang et al. 2020 ) use class context represented as abstracted UML diagrams, for code summarization, at the file-level. Allamanis et al. (2017)  and  Allamanis et al. (2020)  extract control, data flow graphs, along with syntax within a single file. Representation-specific datasets are useful but they limit cross-representational and cross-architectural analyses for tasks.
JEMMA supports several representations including raw source code, tokens, ASTs, and graphs, at both method-level and class-level, building up to even coarser granularities. Datasets of code from student assignments, programming competitions, and other smaller programs, have also been created. Among them, Google Code Jam 2  and POJ-104  (Mou et al. 2016 ) are clone detection tasks (clone detection in this case is formulated as a program classification task). COSET  (Wang and Christodorescu 2019) , and CodeNet  (Puri et al. 2021 ) also feature smaller programs, but complement them with additional metrics and labels. Although these datasets have many desirable properties, they do not represent source code used in real-life software systems and thus it is unclear if learning on these datasets can generalize to general-purpose software.
Semi-synthetic datasets, such as NAPS  (Zavershynskyi et al. 2018) , also fall into the same category. JEMMA balances the preceding concerns by building upon organic projects coming from a diverse set of domains (e. g. , games, websites, standalone applications, etc) and of development standards (ranging from student projects to industry-grade open-source projects) which add a healthy factor of generalization for source code modeling. Code Datasets with Natural Language Natural language presents an interesting, yet separate, modality from source code and is central to the NLP task of semantic parsing (i. e. , text-to-code). A few datasets have focused on this: CodeNN  (Iyer et al. 2016) , CoNaLa  (Yin et al. 2018), and StaQC (Yao et al. 2018). Datasets such as NL2Bash  (Lin et al. 2018)  provide data for semantic parsing from natural language to Bash commands, while  Spider (Yu et al.
2018 ) is a dataset for the text-to-SQL semantic parsing task. Finally,  (Barone and Sennrich 2017) , CodeSearchNet  (Husain et al. 2019), and (LeClair and McMillan 2019)  pair natural language documentation comments with code, targeting code search and code summarization applications. All these datasets provide dumps of source code snippets per file, and while it is possible to parse the code text and perform some intra-procedural analyses for the few file-level datasets, information about external dependencies is commonly lost rendering it impossible to extract accurate semantic data. JEMMA successfully mitigates such issues by providing a dataset of inter-related code entities across granularities, along with comprehensive intra-and inter-procedural relationship information coming from data-flow, control-flow, call graphs, etc.
Code Datasets with Higher-Level Representations While the above datasets focus on code snippets or files, some work has extracted datasets aiming for representations that capture information beyond a single file. However, commonly these datasets opt for an application-specific representation that loses information that could be useful for other research. For example,  (DeFreez et al. 2018 ) extract path embeddings over functions in Linux. LambdaNet  (Wei et al. 2020 ) extracts type dependency graphs in TypeScript code but removes most code text information; their dataset is also limited to 300 projects, which range from 500 to 10,000 lines of code. The dataset by  Bansal et al. was also refined and used in a source code summarization approach that defined a project-level encoder, that considers functions in up to 10 source code files in a project  (Bansal et al.
Code Datasets with Changes Given the importance of software maintenance in the development lifecycle, a few datasets have focused on edit operations in code. The goal of these datasets is to foster research in Neural Program Repair. ManySStubs4J (Karampatsis and Sutton 2020) and Bugs2Fix  (Tufano et al. 2019)  both fall in this category: they are corpora of small bug fixes extracted from GitHub commits. These datasets often focus on local changes (e. g. , diffs) and ignore the broader context. Although our dataset does not come with changes, it provides increased modeling opportunities for users as it comes with inter-procedural relationship information among code entities for all 50K projects. Neural Program Repair workflows could benefit at the dataset creation stage by leveraging bug-related properties in our dataset.
The Qualitas Corpus and its Descendants The Qualitas corpus  (Tempero E et al. 2010) , is an influential corpus of 111 large-scale Java systems that was used for a large number of empirical studies of Java and the characteristics of the systems implemented in it. While this dataset was source code only, it was post-processed in various ways, producing several derived datasets. The Qualitas. class corpus,  (Terra et al. 2013 ) is a version of the Qualitas corpus that is also compiled in. class files. The QUAATLAS corpus  (De Roover et al. 2013) , is a post-processed version of the Qualitas corpus that allows better support for API usage analysis. XCorpus  (Dietrich et al. 2017) , is a subset of the Qualitas corpus (70 programs) complemented by 6 additional programs, that can all be automatically executed via test cases (natural, or generated).
(Lämmel et al. 2011 ) gathered a dataset of Java software from Sourceforge, that had 1,000 projects that were parsed into ASTs. The BOA dataset and infrastructure, by  (Dyer et al. 2013) , provides an API for pre-processing software repositories, such as providing and analyzing ASTs, for 32,000 Java projects. The 50K-C dataset of  (Martins et al. 2018 ) contains 50,000 Java projects that were selected because they could be automatically compiled. A follow-up effort is the Normalized Java Resource (Palsberg and Lopes 2018) (NJR). A first release, NJR-1, provides 293 programs on which 12 different static analyzers can run  (Utture et al. 2020 ), but has a stated goal of gathering 100,000 runnable Java projects, but is still a work in progress.
Java Datasets Other Datasets (Spinellis 2017) released a dataset that contains the entire history of Unix as a single Git repository. (Geiger et al. 2018 ) present a graph-based dataset of commit history of real-world android apps. The entire Maven software ecosystem was released as a dataset with higher-level metrics, such as changes and dependencies  (Raemaekers et al. 2013). The Maven Dependency Graph by  (Benelallam et al. 2019 ) provides a snapshot of the Maven Central as a graph, modeling all its dependencies. Fine-GRAPE is a dataset of fine-grained API usage across the Maven software ecosystem  (Sawant and Bacchelli 2017). Finally, both Software Heritage  (Pietri et al. 2019)  and World of Code  (Ma et al. 2021 ) are very large-scale efforts that aim to gather the entirety of open-source software as complete and up-to-date datasets.
The main goal of World of Code is to enable analytics, while the main goal of Software Heritage is preservation (although it also supports analytics). The Perceval tool  (Gonzalez-Barahona et al. 2018 ) also promises automatic and incremental data gathering from almost any tool related to contributing to open source development, among other sources. We find that although there are similarities between JEMMA and Perceval, the differences highlight that both tools can complement each other well. Perceval can be used to fetch raw project data from a wide variety of data sources. On the other hand JEMMA focuses on source code, and can be used to take care of the analysis of data, pre-processing of data, task definition, and training of models out of the box.
The 50K-C Dataset Having surveyed the landscape of existing datasets, we conclude that most machine learning datasets focus on small-scale entities such as functions, methods, or single classes. The ones that offer higher-level representations are specific and too small in scale. The corpora of systems used for empirical studies provide a better starting point, as they can be preprocessed to extract additional information. Of the existing datasets, the most suitable option that is large enough and that allows the most pre-processing is the 50K-C dataset of 50,000 compilable projects. Since JEMMA builds upon 50K-C, we provide detailed background information on it in this section. The 50K-C dataset is a collection of 50,000 compilable Java projects, with a total of almost 1. 2m Java class files, its compiled bytecode, dependent jar files, and build scripts.
Beyond the size of the dataset, the fact that the projects are compilable is the main reason we chose to build upon 50K-C. The extensive pre-processing that we perform on top of 50K-C requires the use of static analysis tools, to do things such as call graph extraction, and to extract valuable metrics about the systems. Since the vast majority of static analysis tools operate on bytecode, 50K-C was the most suitable option that combines both scale and the ability to automate the analysis at such scale.
Selection Criteria The dataset authors downloaded close to 500k Java projects, attempted to compile all of them, and selected 50k projects among the ones that could be compiled. Two filters were applied: projects that were Android applications were excluded, and projects that were clones were also excluded-using the DéjàVu clone repository  (Lopes et al. 2017) , and the Sourcerer CC tool  (Sajnani et al. 2016). We find that the projects have a diverse set of domains (e. g. , games, websites, standalone apps, etc), and development levels (ranging from student projects to industry-grade open-source projects). The dataset consists of both large-scale projects with as many as 5k classes, and smaller projects with as low as 5 classes. While the larger projects are good representatives of realworld projects, the smaller projects are valuable too, since machine learning models of code still need to make significant headway in code understanding which necessitates reasoning on projects across all sizes.
The JEMMA Dataset Our goal with the JEMMA project is to provide the research community with a large, comprehensive, and extensible dataset for Java that can be used to advance the field of source code modeling. The JEMMA datasets consist of a large collection of code samples in varying granularities, with wide-ranging and diverse metadata, a range of supported source code representations, and several properties. In addition, it also includes source code information related to code structure, data-flow, control-flow, caller-callee relationships etc. For every project in the JEMMA Dataset, we gather data at the project-level, and provide information on all the packages and classes. Furthermore, for every class, we parse and provide data on all the methods-including respective metadata, several representations and properties.
The detail of data provided for every method entity is comprehensive, with data at the level of AST with data-flow, control-flow, lexical-usage, and call-graph edges among others. In addition to necessary information, such as line numbers and position numbers of code tokens, supplementary information such as token types, node types, etc, are also provided. More details are presented in the following sections. JEMMA also comes equipped with Workbench tools that allow users to perform a variety of tasks out of the box, such as: transforming code samples into intermediate source code representations, making tailored selections of entities to define tasks and forming custom datasets, or to run supported models, among others (Section 4 provides more details).
Statistics The original 50K-C dataset contains a total of 50,000 projects. It has 85 projects with over 500 classes (with a maximum of 5549 classes in a project), 1264 projects with 101-500 classes, 2751 projects with 51-100 classes, 10693 projects with 21-50 classes, 14322 projects with 11-20 classes, and 20885 projects with 10 or fewer classes (with a minimum of 5 classes per project). We have collected metadata for all of these projects. Overall, the data consists of 1. 2 million Java classes, which define over 8 million unique method entities. Granularity JEMMA supports multiple granularities. We have processed and catalogued data starting from the project-level descending to smaller entities, which means a spectrum of granularites of code can be accessed. However, since a method is the most basic unit of behavior in an object-oriented program, and it is a self-contained program segment that carries out some specific, well-defined logical task, we collect all the properties at the (finer) method level.
For instance, method entities can be sampled from the datasets and used independently to run code-analysis tools. In this sense, methods can be considered as our primary entity-of-interest. Starting from the method-level, larger contextual entities at the class-, package-, and project-level can then be created by building upon the smaller entities. And since most prevalent models of source code accept input samples at the method-level, we do provide all source code properties and representations at this (finer) method-level by default; properties and representations of larger entities-of-interest can always be built upon smaller entities within it. Compilability Successful compilation ensures that the source code snippets from the projects have been type-checked and parsed successfully, and are valid Java code.
Storage and Sharing Considerations The amount of data produced is considerable. To maximize accessibility, we provide it as a set of Comma Separated Values (CSV) files, so that users can choose and download the data that they need. Note that only the metadata and the original source data are absolutely necessary; other data can be downloaded on a perneed basis. The JEMMA Workbench allows the recomputation of the other properties, if, for some properties, it is more efficient to recompute them than to download them. The data is uploaded on Zenodo; due to its size, it is provided as multiple artifacts. We present the components of the dataset, along with their DOIs (links to the download page), and sizes later on. Interacting with the Data Most of the data from the JEMMA are organized in Comma-Separated-Values (CSV) files, consequently basic analyses can be run with tools such as csvstat.
Furthermore, our Workbench APIs can be used to gather extensive statistics of the projects, classes, methods, bytecode, and data and control-flow information. The JEMMA datasets are grouped into three major parts: data at the metadata level (Section 3. 1), data at the property level (Section 3. 2), and at the representation level (Section 3. 3). In addition, we also provide project-wide callgraph information for the 50,000 projects, uniquely identifying and associating source and destination nodes in the callgraph with the help of the metadata defined by JEMMA (Section 3. 4). This allows for accessing project-wide data on the whole, for different granularities of code entities. Figure  1  gives a glimpse of the extent and detail of the data contribution made by JEMMA. The top-left corner represents the raw data from 50K-C, which we catalog by adding UUIDs (symbolized by colored squares).
JEMMA: Metadata In this section we present the metadata for the JEMMA datasets. The metadata is made available in CSV (comma-separated values) files. This allows for easy processing, even with simple command-line tools. The metadata is organized in four parts, from the largest units of interest to the smallest: projects, packages, classes, and methods. The units of interest can then be inter-related systematically. The metadata serves two major purposes: 1. Uniquely identify a source code entity with an UUID. 2. Gather basic and often-used information on each source code entity. Taken together, these two purposes allow users to extend the dataset with additional properties. The UUID allows us to uniquely identify an entity in the dataset, and the supplementary metadata helps disambiguate entities (file paths, parent relationships, location information in the file, etc).
Packages For the package-level metadata, a single CSV file lists all the packages present in the projects. The metadata comprises of the UUID of the parent project as project id, the UUID assigned to the package as package id, the relative path of the package as package path, and the name of the package directory as package name. Classes For the class-level metadata, we provide a single CSV file that lists all the classes in the 50K-C dataset along with their corresponding metadata: project id, package id, class id, class path, class name. Similarly to the projects, the class path is a relative path starting from the 50K-C dataset's root directory, that allows to access the raw source code of the class.
JEMMA: Properties JEMMA leverages the UUIDs assigned to projects, classes, and methods as a way to attach additional properties to these entities. Thus, a property can be an arbitrary value that is associated to an entity, such as a metric. Even though we have gathered several properties associated with code entities, it should be noted that a particular property may not be available or may not apply for a given code entity. Users can add new properties associated with code entities as contributions to the dataset, where the property should be given a unique name and be stored in the correct location for it to be visible to JEMMA Workbench APIs. (Section 4 provides more details). Next, we list the tools used to obtain the properties: • The Infer static analyser  (Calcagno et al. 2015 ) is a tool that provides advanced abstract interpretation-based analyses for several languages, including Java.
• PMD is a static code analysis tool 5  that can compute a variety of static analysis warnings and metrics, such as the npath complexity metric, among many, many others. We used PMD because it is inexpensive while reviewing large codebases; and it is trusted by industry practitioners and researchers. PMD can also be used to identify defects and problems in code entities which can be useful for future works. • The java-callgraph 6  extractor is a tool for generating call graphs in Java. We used this tool to extract project-wide call graphs, from which callers and callees were identified and linked to their respective UUIDs at the post-processing stage. The java-callgraph generator tool was used since it was capable of generating both static and dynamic call-graphs suitable for our dataset of compilable code entities.
Tokens (TKNA, TKNB) Each Java method snippet is tokenized with an ANTLR4 grammar  (Parr 2013 ) and made available to the user preprocessed. The tokenized code includes method annotations, if any, but does not include natural language comments. However, with the entire raw text of method snippets made available by default, users are free to include comments in their custom tokenizations. For every method snippet in our dataset, we provide the corresponding string of tokens. In fact, we provide two types of tokenized strings. First, a simple space-separated string of tokens. This representation is meant to be directly passed to an existing ML model that has its own tokenizer, without any further processing. The downside is that some literals that include spaces may be treated as more than one token, or symbols and special characters may be ignored while using certain tokenizers (e.
• LastLexicalUse edges link lexical usage of variables (independent of data flow). • GuardedBy and GuardedByNegation edges connecting a variable used in a block to conditions of a control flow. • ReturnTo edges link return tokens to the method declaration. • FormalArgName edges connect arguments in method calls to the formal parameters. This representation is significantly feature-rich, as it includes, for instance, all the source code tokens and their precise locations in the original source code, the signatures of all the methods called in the class, the source code comments, if any, including a variety of dataflow, control-flow, lexical-usage, and hierarchical edges. Derivative representations such as AST with dataflow information, a subset of the feature graph representation, which corresponds to the data used by models such as GraphCodeBERT can also be produced from the feature graph representations.
Extending and Using JEMMA Table  2  presents the links to the actual datasets with JEMMA: meta-data, properties, representations, and callgraphs. These are standalone CSV files that can be used on their own, but to make it easy for users to access and use them in common usage scenarios we have added a Workbench component to JEMMA. When building JEMMA, we intended it to be large-scale, yet extensible, flexible, and most importantly, easy to use. We have implemented several tools to help with this, and as a result, researchers can readily use JEMMA as a Workbench to experiment with variants of datasets, models, and tasks while minimizing the processing that is involved. The JEMMA Workbench tools and implementations, written in Python, are accessible through a set of APIs, which helps developers interface with it when writing machinelearning code and take advantage of several pre-implemented functionalities from viewing, creating, retrieving from, and appending to datasets, defining task labels, generating custom/variant code representations, to training and evaluating supported models on such datasets.
Potential Uses for the Data In the following paragraphs, we describe some of the potential ways in which data can be utilized directly with the help of the Workbench tools. Create a multitude of new datasets: Since JEMMA is built on top of the 50K-C Dataset, we have catalogued all of the 50,000 projects and their child code entities, made them uniquely-identifiable, and provided a range of properties associated with them, along information on inter-relationships. Using this information a multitude of datasets can be prepared, not just specifically for ML4Code but also for other purposes, e. g. creating a dataset of projects based on the project size, or creating a dataset of method snippets with complexities based on a criterion, and so on, for a diverse range of use-cases. Define ML task datasets: Users can decide on a modelling objective and retrieve the training data from JEMMA.