dataesr/scientific-paragraphs-categorization huggingface dataset:
This may sound straightforward but preparing a sound dataset for model training is one of the most important steps, and it is often time-consuming given the amount of data cleaning and transformations involved before model training. The JEMMA Workbench tools help users choose from a range of pre-processed source code representations across 8M samples, and filter them based on a range of properties, and even use the properties as prediction labels. The representations and properties can be used, either singularly, or in combination, to generate thousands of combinations of clean and balanced ML task datasets ready to be trained. Section 4. 2. 1 demonstrates a similar example. Retrieve code information: Within JEMMA we catalogue code at the project, package, class, and method-level. Furthermore, we process source code into feature graphs yielding feature-rich code information at the AST node-level.
Adding a New Property The simplest way to extend JEMMA is to add a new property. This could be any property of interest that can be computed for a source code entity. Examples include defining a new source code metric, or the result of a static analysis tool indicating the presence (or absence) of a specific source code characteristic. To extend JEMMA with a new property, the workflow has three main steps: a) accessing a set of source code entities, b) generating associated property values, and c) merging the associated property values to the dataset. JEMMA facilitates accessing the correct code input by providing the location and metadata for code entities, and several initial representations (raw text, ASTs, etc. ). An associated property could then be obtained either directly (e. g. method name) or by means of a code analysis tool (e.
Adding a New Representation Different machine learning models of code require different source code representations as input. Some models reason over tokenized source code, while other models reason over more complex structures such as ASTs. Each representation comes with its own set of advantages and drawbacks, while one is extremely feature-rich the other is simple, scalable, and practical. Therefore, the work on representations is still an active area of research-as researchers are continuing to develop new source code representations, or improving the present ones, e. g. , by augmenting them with further information. JEMMA makes is quite simple to do both: create new representations, and modify existing ones. There are three main steps to extend JEMMA with a representation: a) accessing a set source code entities, b) generating associated representations, and c) merging the representations to JEMMA.
The raw source text, or even representations such as the AST, could be accessed directly to produce new representations for associated code entities. And with an array of source code representations readily made available for over 8 million code enities, simplifying or augmenting such representations to create others would save a lot of pre-processing The feature graph representation which we include in our dataset (see Section 3. 3. 5) is built upon the abstract syntax tree (AST) of the code, and is extended with a number of additional edges, depicting various inter-relationships between the AST nodes (e. g. , dataflow, control-flow, lexical-usage, call-edges among others). In addition to other necessary information such as line numbers and position numbers of every source code token, supplementary information such as token types, node types, are also provided.
Adding a New Project With the evolution of source code over time, and the inclusion of new libraries and modern technologies -source code datasets (especially the ones built for modeling code) must also keep at pace if they are to remain relevant. We built JEMMA on top of the 50K-C Dataset of 50,000 compilable Java projects, however, we want it to be extensible. So, we have provided mechanisms to include additional projects into the fold of JEMMA. Adding a new project to JEMMA involves three main steps: 1) forking the jemma repository, 2) generating the meta-data, representations, properties, call-graphs -by running the relevant scripts, and 3) making a pull request to added the new-generated data. We provide a simple bash script that helps users generate all the relevant data in one go-generating metadata and cataloging code entities within the project, generating representations, generating properties, and generating project-level call-graphs.
Defining Tasks Based on Properties Once a property is defined in JEMMA, it can be used in a variety of ways. One such way is to use them as prediction labels for a prediction task. A good example of such a prediction task complexity prediction, i. e. , given a snippet of code as input, a source code model must predict its cyclomatic complexity (property) as output. While this may appear trivial (taking a random sample of entities that have that property defined, and splitting it into training, validation, and test sets), in practice it is often more complex. This is because care must be taken that the data does not contain biases that provide an inaccurate estimate of model performance. In this context, there are several groups of issues that JEMMA helps contend with while defining the task datasets.
Rare Data The first is that some property values may be very rare, making them hard to learn at all. Examples of this would be uncommon bugs and errors such as, e. g. , resource leaks. Since JEMMA is large to start with (over 8 million method entities), the scale of data makes it much more likely that there is enough data to learn in the first place, compared to other alternatives. Defining Task Labels Once a property is defined, the Workbench tools provide flexibility in the use of property values as task prediction labels. For instance, for classification tasks, the Workbench API endpoints allow users to query and obtain a balanced set of prediction labels, ready for training. Furthermore, the Workbench tools allow the selection of property values as prediction labels that satisfy some criteria based on other properties in the dataset.
A subset of the data can also be selected, if one wants to define a task for which data is more scarce, in order to incentivize sample-efficient models. Similar to a Database Workbench, the JEMMA Workbench allows several such operations in the context of defining prediction labels for a machine learning task, and in managing and retrieving large amounts of specific information. Creating datasets for training machine learning models can often be time-consuming and frustrating. Since we start with a lot of data, it might be necessary to filter it down. With the tools included as a part of our Workbench, users can query JEMMA and obtain clean, complete, and balanced datasets.
In the case of the code complexity example, Fig. 5  shows the relationship between the size of methods (SLOC) and their complexity as a hexbin plot. We observe that there is an overall tendency for shorter methods to be less complex, and longer methods to be more complex. On the other hand, there also methods that are very long, but have very low complexity (along the bottom axis). This information can be used to properly balance the data, for instance, by making sure that examples that are short and complex, and examples that are long and simple, are also included in the training and evaluation datasets.
Empirical Study I: On the Extent of Non-Localness of Software Since software is made of a lot of interacting entities across files, packages, and projectsmodeling source code by learning on smaller entities of code (e. g. , methods) can at best provide a localized understanding of source code for a given task. We hypothesize that in order to improve our source code models comprehensively beyond localized understanding, we must include non-local contextual features while modeling source code. As a result, we must study the extent of non-localness of software and whether non-local context is useful. In this section, we demonstrate the utility of JEMMA Dataset and Workbench through an empirical investigation. We study the extent to which software is made up of interacting functions and methods in a sample of projects contained in JEMMA by analysing their call graphs.
Task definition We define the method call completion task as a masking task: for each method snippet, we mask one single method call in the code snippet at random. These methods can be present in the same class (18% of the dataset), in another class in the same package (10%), in another package in the system (26%), or imported from a dependency (46%). The goal of the task is for a source code model to predict the exact method name that was masked. We sample 100K methods from the JEMMA Datasets, splitting 80K samples as training data, 5K as validation data, and 15K as test data for training and evaluation.
It is important to note that both CodeBERT and Codex are not models trained from scratch on source code: given the amount of time needed to train such a model from scratch, previous models trained on English (RoBERTa for CodeBERT, a version of GPT-3 for Codex) were fine-tuned on source code instead. This means that both CodeBERT and Codex use a subword tokenizer that was not learned for source code, but for English, which might lead to sub-optimal tokenization. To estimate the number of tokens that a method will take in the model's input window, we first selected a sample of 200,000 Java methods from JEMMA, and used several subword tokenizers to estimate the ratio of subtokens that each subword tokenizer will produce. We first noticed that the choice of subword tokenizer has a significant impact on the produced tokenization, and consequently the amount of code that can fit in a model's input window.
For a window size of 512, a Javaspecific tokenizer will, on average, be able to effectively fit 602 actual tokens, while the English-specific tokenizer-used by both CodeBERT and Codex-will be able to fit only 409 actual tokens. For example, for tokens such as ArrayIndexOutOfBoundsException, efficient language-specific code tokenizers will tokenize it as a single token, rather than six separate tokens. This establishes that language-specific code tokenizers are more efficient in tokenizing source code compared to their English-language counterparts. And since almost all model architectures a maximum input size limit, the tasks that specifically rely on additional context information can benefit from efficient tokenizers, whereby input source code snippets can be represented in less number of tokens leaving space for additional context information.
Furthermore, Graph Neural Networks can also struggle with long-distance relationships in the graph  (Alon and Yahav 2020). Clearly, significant work is needed to find architectures that can fit contexts at the project-level, especially if the model size is to be kept small enough to be manageable. On the other hand, we see promise in an approach that is able to select the input relevant to the task. Of note, recent work has started to go in this direction for code summarization, both at the file level  (Clement CB et al. 2021 ) and multiple files  (Bansal et al. 2021). Significant work lies ahead in devising techniques that truly take into account a larger global context, thus addressing the "Out-Of-Window" (OOW) problem; at a minimum, JEMMA provides the data at scale, and tools to investigate this.
Limitations Stemming from the Use of 50K-C Monolingual JEMMA is comprised of projects in the Java programming language only. This poses issues as to whether models that work well for Java would also work well for other languages. The reason for this limitation is twofold: 1) adding other languages at a similar scale would drastically increase the already extremely significant time we invested in pre-processing data, and 2) restricting to one language frees us from tooling issues: we don't need to settle on a "common denominator" in tool support (e. g. , Infer supports few programming languages, and many of its analyses are limited to a single programming language). Monoversion JEMMA is comprised of snapshots of projects, rather than multiple project versions. This prevents us from using it for tasks that would rely on multiple versions, or commit data, such as some program repair tasks.
On the other hand, this frees us from issues related to the evolution of software systems, such as performing origin analysis  (Godfrey and Zou 2005) , which is essential as refactorings are very common in software evolution, and can lead to discontinuities in the history of entities, particularly for the most changed ones  (Hora et al. 2018). Omitting versions also considerably reduces the size of the dataset, which is already rather large as it is. Static Data Only While the projects included in 50K-C were selected because they could be compiled, 50K-C provide no guarantees that they can be run. Indeed, it is hard to know if a project can run, even if it can be compiled. In case it can run, the project likely expects some input of some sort. This leaves running test cases as the only option to reliably gather runtime data.
In our previous work in Smalltalk, where we performed an empirical study of 1,000 Smalltalk projects, we could run tests for only 16% of them  (Callaú et al. 2014 ). Thus, JEMMA makes no attempt at gathering properties that comes from dynamic analysis tools at this time. In the future, JEMMA's property mechanism could be used to document whether a project has runnable test cases, as a first step towards gathering runtime information. We could also expand the dataset with the 76 projects coming from XCorpus, which were selected because they are runnable  (Dietrich et al. 2017).
Conclusion In this article, we presented JEMMA, a dataset and workbench to support research in the design and evaluation of source code machine learning models. Seen as a dataset, JEMMA is built upon the 50K-C dataset of 50,000 compilable Java projects, which we extend in several ways. We add multiple source code representations at the method level, to allow researchers to experiment on the effectiveness of these, and their variations. We add a project-level call graph, so the researchers can experiment with models that consider multiple methods, rather than a single method or a single file. Finally, we add multiple source code properties, obtained by running source code static analyzers-ranging from basic metrics to advanced analyses characteristics based on abstract interpretation. JEMMA Workbench, its toolchain and corresponding APIs, help achieve a variety of objectives.
Finally, we have shown that taking into account this global context will be challenging, by studying its size. While state-of-the-art transformer models such as CodeBERT can fit most methods in the dataset, fitting package-level or higher context is much more challenging, even for the largest models such as OpenAI's Codex model. This indicates that significant effort lies ahead in defining models able to process this amount of data, a task that we hope JEMMA will support the community in achieving. Fig. 13  Step-wise inclusion of greater context information, based on caller-callee relationships, from 1-hop, 2-hop, and 3-hop neighborhoods (003, 005, 007) of the direct callee methods provide indirect context information from a further neighborhood (2-hop or level-2 neighborhood) and so on. Based on the call graph, a given method can have multiple sources of context information from different neighborhoods (level-n) of callers and callers.
This approach enabled substantial progress over the previous iteration. Its reliance on geolocation data to anchor augmented objects relative to the user's position poses a set of challenges: On mobile devices, GNSS accuracy typically lies between 1 m and 30 m. Due to its impact on the anchoring, this lack of accuracy can have deleterious effects on usability. We conducted a comparative user test using the application in combination with two different geolocation data types (GNSS versus RTK). While the test's results are undergoing analysis, we hereby present a methodology for the assessment of our system's usability based on the use of eye-tracking devices, geolocated traces and events, and usability questionnaires.
Introduction 1. Goals and Structure of the Paper The goal of this paper is to describe the development of the location-based augmented reality (AR) web application BiodivAR and associated user-centered design (UCD) methods  [1]  for geolocation data enhancement. It also proposes a methodology to further evaluate the usability of the system. As such, it does not present new research results, but merely a new tool and the methods used to develop it. In 2017, the first iteration in the form of a proof of concept in educational technologies (edTech) saw light under the name of BioSentiers. This paper aims to present the output of the second iteration of this project and the methods behind it. The first iteration both highlighted some benefits of using locationbased AR for nature exploration and identified key challenges that needed to be addressed to make this pairing more meaningful.
Design a location-based AR interface that supports nature exploration The application features an AR immersive mode for visualizing the text, photography, graphics, 3D animation, and sound in virtual environments. When these spatial media contain information about the geographical space they're anchored to, they have the potential to engage users in a singular way  [3] [4] [5] [6] [7] [8]. We aim to leverage this particular type of immersive interface's attractiveness and efficiency to engage users to actively explore outdoors. Try to better the usability of location-based AR interfaces Because location-based AR depends on geolocation data, the accuracy of that data is of direct consequence on the interface's stability  [4, 5, [8] [9] [10] [11]. We propose the use of an external module for satellite navigation with real-time kinematics (RTK) that achieves centimeter-accuracy.
We also lay down the concept for a data fusion method using simultaneous localization and mapping (SLAM) data and geolocation data to further improve heading data, which has an equally important impact on location-based AR interfaces. Evaluate the tool's usability As a UCD project, its successes and failures are to be assessed primarily on the basis of its usability, acceptance and user satisfaction as measured by self-reported user data, and on the improvements observed with respect to the prior iteration  [1]. We have conducted a comparative user test (n = 54) to assess the tool's usability in combination with two geolocation data sources. The results have not been fully processed and analyzed yet and will be presented in a subsequent paper.
Concept The BiodivAR mobile application aims to support nature exploration for pupils, students, and concerned citizens and allow them to visualize and interact with geolocated POIs containing media in AR. Additionally, it aims to enable teachers to author location-based AR experiences. Our concept thus assumes the two following main types of utilization: media visualization in AR on location/outdoors ; and the authoring of AR environments that can be carried out remotely/indoors. As such, the application comprises two main modules: a mobile component, which mainly allows AR visualization of geographic data, and a desktop cartographic authoring geoportal that allows users to create and manage environments and visualize data on a 2D interactive map. This two-part design is shown in Figure  6.
Tools, Frameworks and Standards After the conceptual grounds were set, the actual development of the application began and the adoption of suitable frameworks came into question. After reviewing various options (Unity, Ionic, Wikitude, WebXR) we opted for a web-based application for the numerous advantages it offered. It can be accessed through an URL in a WebXR-enabled Web Browser without downloading any file. A single deployment allows the targeting of handheld and head-mounted AR devices. As an applied research project, continuity is important, and this choice meant our code would run on new devices without the need for important updates. However, it also came with limitations. Because the code and assets are stored on a distant server, web-based applications tend to be slower and less responsive than native apps, which can be critical for AR usability.
js, which also features an authoring tool  [47]. However, AR. js covers a wide range of AR types that we would not use in our system, and its geolocation components had not been updated or maintained in a while. Consequently, we created LBAR. js https://github. com/MediaComem/LBAR. js/ (accessed on 28 December 2022), a minimalist A-Frame library for creating WebXR locationbased anchors. LBAR. js places entities in a virtual 3D space based on geolocation data and inertial sensor data. It includes one system (gps-position) and three components (facesnorth; gps-position; pitch-roll-look-controls)  [48]. The gps-position system defines the accuracy threshold over which (after a movement of n meters) the geolocated entities' anchoring is refreshed. The faces-north component forces an entity to always be facing north (as measured by the compass bearing, or absolute device orientation).
It lets users create a parent entity aligned with reality under which all geolocated entities are anchored as children. The gps-position component allows the position of an entity (a geolocated POI) to be expressed in WGS84 coordinates, by converting those into local coordinates so that it may be anchored in the A-Frame scene. It needs to be a child entity of an entity with the faces-north component in order for it to work. The pitch-roll-look-controls component disables the default yaw control of the AR camera, since the faces-north entity (and its children entities) rotates on that axis instead. Other open-source dependencies that were used for the client-server structure include SQLite (database), PrismaJS (ORM), hapi. js (API), vue. js (frontend), and leaflet. js (2D interactive map). The full dependencies' diagram is visible in Figure  7.
BiodivAR structure and dependencies. The user interfaces (desktop and mobile) are managed by the Vue. js framework, which sends a request to a Rest API built with hapi. js. The Rest API gets the requested data from an SQLite database through a Prisma. js object-relational mapping (ORM). The mobile and desktop 2D interactive maps are powered by the Leaflet. js library and the AR cameras and objects are managed by the WebXR API, the three. js 3D library and the A-Frame framework, with the LBAR. js [48]  additional custom library we designed to power geolocated POIs. All of theses dependencies are open-source.
Database Structure The data are organized around the concept of bioverse, consistently with the application principle of managing this learning environment on an interactive 2D map and visualizing them in AR. A bioverse is stored in a GeoJSON object with the type "FeatureCollection"  [49]. Each POI is a feature that contains a geometry type (point), geographic coordinates, and a bunch of user-defined properties related to the POI's style, contents, media, positions, and events. As visible in Figure  11 , eight tables structure all the data in the database:.
Location-Based AR Bottleneck One of location-based AR's main advantages over other types of AR is the possibility to create and anchor augmented objects in a global spatial referential. They can thus be anchored remotely, as opposed to having to be physically on site to anchor content. It can leverage existing geodatabases, and it is possible to enrich them for further visualization or analysis of other geoportals. However, as observed during the BioSentiers field test  [9]  as well as by other researchers who have tried to use this technology for education purposes,  [4, 5, 8, 10]  location-based AR's main drawback is the instability of the anchoring of augmented objects which results in a degraded user experience. Indeed, a typical mobile device-embedded GNSS sensor has limited accuracy  [50].