dataesr/scientific-paragraphs-categorization huggingface dataset:
I. INTRODUCTION Code readability is the first requirement for program comprehension: one cannot comprehend what one cannot easily read. To improve code readability, most developers agree on using coding conventions, so the code is clear and uniformly consistent across a given code base or organization  [23] ,  [16]. A major challenge of using coding conventions is to keep all source code files consistent with the agreed conventions. The first step towards that is the detection of coding convention violations (or errors). This can be automatically performed using linters, which are static analysis tools that warn software developers about possible violations of coding conventions  [36]. The usage of linters also brings challenges because the developers need to create a configuration according to their adopted conventions so that the linter detects the right violations (not more and not less), and then to repair eventually violations.
In this paper, we focus on the later task, automatically repairing linter violations, which is a little researched problem, and we focus on formatting errors  1. To repair a formatting error detected by a format checker, developers can either perform the fix manually or use a code formatter. Both alternatives are not satisfactory. Manually fixing formatting errors is a waste of valuable developer time. With code formatters, the key problem is that they do not take into account the project-specific convention rules, those that are configured by the developers for the used format checker. Inspired by the problem statement of program repair  [24] , we state in this paper the problem of automatically repairing formatting errors: given a program, its format checker rules, and one rule violation, the goal is to modify the source code formatting so that no violation is raised by the format checker.
In this paper, we explore this problem in the context of  [8] , a popular format checker for the Java language. We present STYLER, a repair tool dedicated to fix Checkstyle formatting errors in Java source code. The uniqueness of STYLER is to be applicable to any formatting coding convention, because its approach is not based on rules to repair specific Checkstyle errors. The key idea of STYLER is the usage of machine learning to learn the coding conventions that are used in a software project. Once trained, STYLER predicts changes on formatting characters (e. g. whitespaces, new lines, indentation) to fix a formatting convention violation happening in the wild. Technically, STYLER uses a sequence-to-sequence machine learning model based on a long short-term memory neural network (LSTM). We conduct a large scale experiment to evaluate STYLER using a curated dataset of 11,220 real Checkstyle errors mined from 70 GitHub projects.
Based on our research questions, we find that STYLER repairs many errors (38%), and repairs errors from more different Checkstyle formatting rules compared to the state-of-the-art of machine learning formatters  [3] ,  [26]  and the tailored, human engineered IntelliJ plugin CHECKSTYLE-IDEA  [9]. Moreover, STYLER produces small repairs and its performance is fast enough for developers. To sum up, our contributions are: • A novel approach to fix violations of code formatting conventions, based on machine learning. The approach is able to learn project-specific formatting rules without manual setup; • A tool, called STYLER, which implements our approach in the context of Java and Checkstyle, to repair Checkstyle formatting violations. The tool is made publicly available  [21] ; • A curated dataset of real-world formatting Checkstyle errors, which contains 11,220 errors mined from 70 GitHub repositories; To our knowledge, this is the largest dataset of this kind, made publicly available for future research; • A comparative experiment of the performance of STYLER against the state-of-the-art of automatic code formatting  [9] ,  [3] ,  [26] , showing that STYLER outperforms it.
II. BACKGROUND A. Coding Conventions Coding conventions (also known as coding style or coding standards) are rules that developers agree on for writing code. The usage of coding conventions improves code readability but it does not change the program behavior. There are several coding convention classes: e. g. naming, control flow style, and formatting. In this paper, we focus on the latter: formatting coding conventions. Formatting here refers to the appearance or the presentation of the source code. One can change the formatting by using non-printable characters such as spaces, tabulations, and line breaks. In free-format languages such as Java and C++, the formatting does not change the abstract syntax tree. In non-free-format languages such as Haskell or Python, formatting is even related to behavior: correcting formatting issues can fix a bug  [7].
Linters can be usually integrated in IDEs and build tools. When integrated in IDEs, the developer manually runs the linter before she commits her changes. If she does not do it, she might face a lot of errors raised by the linter after the end of the building step for a release or for shipping the program. On the other hand, when a linter is integrated in build tools, it is automatically executed in Continuous Integration (CI) environments. The important coding conventions might be configured to make CI builds break when they are violated. This way, developers are forced to repair coding convention violations early in the software development process. Several linters have been developed depending on the programming language: e. g. ESLint  [13]  for JavaScript, Pylint  [29]  for Python, StyleCop  [34]  for C#, and RuboCop  [31]  for Ruby.
For Java, which is our target language in this paper, the most commonly used linter is Checkstyle  [8]. Checkstyle supports predefined well-known coding conventions, such as the Google Java Style Guide  [16]  and the Sun Code Conventions  [35]. It also allows developers to configure a specific ruleset to match their own preferences. Checkstyle is a flexible linter that can be integrated in both an IDE (e. g. IntelliJ, Eclipse, and NetBeans) and in a build tool (e. g. Maven and Gradle). In the Java ecosystem, Checkstyle is often executed in Continuous Integration environments such as Travis and Circle CI. [39] ,  [38] ,  [22]. However, there is little empirical knowledge of the extent of what Checkstyle, one popular static analyzer, is used in the wild. To ground our work with a solid empirical basis, we then investigate the usage of Checkstyle and its rules in open source projects.
III. STUDY OF CHECKSTYLE USAGE IN THE WILD Static analysis tools have been subject of investigation in recent research Checkstyle can be executed on a project in different ways. The straightforward ways are 1) by directly invoking Checkstyle on the command line, 2) by a build tool, or 3) by a continuous integration service. Independently of the way Checkstyle is executed, there must exist a configuration file with the Checkstyle rules defined by the developers: we refer to this file as Checkstyle ruleset. In this section, we report on our large-scale study on the usage of Checkstyle on GitHub.
A. Checkstyle Usage in Practice Method. To measure the usage of Checkstyle on GitHub, we queried GitHub 2  to only retrieve Java projects with at least five stars, because stars have been shown meaningful to sample projects from GitHub  [6] : we found 148,127 Java projects. Then, we searched each of them for finding a Checkstyle ruleset file. A Checkstyle ruleset file can have any name, but we followed a conservative approach towards identifying true positives: we used a set of commonly used names  3. For simplicity, in the rest of this paper we refer to a Checkstyle ruleset file as checkstyle. xml. Results. We found 3,793 Java projects containing a checkstyle. xml file, which is 2. 56% of all Java projects  with at least five stars on GitHub. Table  I  shows the proportion of those projects with their build tools and CI services if any.
A. Targeted Error Types STYLER is about learning how to repair errors related to formatting coding conventions (see Section II-A). For instance, consider that a developer specified that her preference on the left curly token "{" in a conditional block must always be placed in a new line (as shown in Figure  1a ). If this rule is not satisfied (e. g. such as in Figure  1b ), Checkstyle triggers a formatting-related error (see Figure  4a ). In order to fix this violation, a new line break should be inserted in the program before the token "{". In Checkstyle, there are different classes of checks: e. g. formatting, naming, and lightweight linting checks. In STYLER, we exclusively focus on formatting checks, such as indentation and whitespace before and after punctuation. We ignore Checkstyle checks that are not related to formatting, e.
Repaired Java code Repaired Java code de-tokenized (Figure  4f ) Java code (Figure  4b ) tokenized (Figure  4c ) Styler Training (learning)  The component 'STYLER training' is responsible for learning how to repair Checkstyle errors on the given project according to its project-specific Checkstyle ruleset. It creates the training data by injecting Checkstyle formatting errors on source code files in the project (step A). Then, it translates the training data into abstract token sequences (step B) in order to train LSTM neural networks (step C). The learned LSTM models are eventually used to predict repairs. The component 'STYLER prediction' is responsible for predicting fixes for real Checkstyle errors. It first localizes Checkstyle errors by running Checkstyle on the project (step D). Then, STYLER encodes the error line into an abstract token sequence (step E), which is given as input to the LSTM models (step F) previously learned.
The models predict fixes for the given Checkstyle error: these fixes are in the format of abstract token sequences, so they must be translated back to Java code (step G). STYLER then runs Checkstyle on the new Java codes containing the predicted fixes (step H). Finally, among the predicted fixes where no Checkstyle error is raised, STYLER selects one formatting repair to give as output (step I). As STYLER only impacts the formatting of the code, its repairs do not change the behavior of the program under consideration.
C. STYLER in Action Consider the Checkstyle error presented in Figure  4a. This error is raised by a violation of the Checkstyle LeftCurly rule: the left curly should be on a new line. Checkstyle provides, for a given error, the location (line and column) where the Checkstyle rule is violated. The Java source code that caused such an error is presented in Figure  4b. STYLER encodes the incorrectly formatted lines (Figure  4b ) into the abstract token sequence shown in Figure  4c. Then, this abstract token sequence is given as input to LSTM models, which predict the formatting token sequence shown in Figure  4d. This predicted formatting token sequence is then used to modify the formatting tokens from the buggy abstract token sequence. It results in a predicted abstract token sequence, as shown in Figure  4e , that may fix the current Checkstyle error.
D. Java Source Code Encoding STYLER encodes the Java source code into an abstract token sequence that is required to predict formatting changes. First, STYLER translates each Java token to an abstract token by keeping the value of the Java keywords, separators, and operators (e. g. + → +), and by replacing the other token kinds such as literals, comments, and identifiers by their types (e. g. x → Identifier). Second, for each pair of subsequent Java tokens, STYLER creates an abstract formatting token that depends on the presence of a new line. If there is no new line, STYLER counts the number of whitespaces, and then represents it like n_SP, where n is the number of whitespaces (e. g. → 1_SP). If there is no whitespace between two Java tokens (e. g. x=), STYLER adds 0_SP between the tokens.
E. Training Data Generation STYLER does not use predefined templates for repairing formatting errors. STYLER uses machine learning for inferring a model to repair formatting errors and, consequently, it needs training data. One option is to mine past commits from the project under consideration to collect training data. However, there might not exist enough data in the history of the project to cover all Checkstyle formatting rules. So in order to have enough data for training, our key insight is to generate the training data. The idea is to modify errorfree Java source code files in the project in order to trigger Checkstyle formatting rule violations. Then, one obtains a pair of files (α orig , α err ): α orig is the file without the formatting error, and α err is the file with the formatting error.
α orig is a repaired version of α err , and we can use supervised machine learning to predict α orig given α err. We experiment that idea in two different ways (called protocols in this paper) to generate training data: we name them as Styler random and Styler 3grams , which we present as follows. The Styler random protocol for injecting Checkstyle errors in a project consists of automated insertion or deletion of a single formatting character (space, tabulation, or new line) in Java source files. These modifications require a careful procedure so that 1) the project still compiles and 2) its behavior is not changed. For this, we specify the locations in the source code files that are suitable to perform the modifications. For insertions, the suitable locations are before or after any token.
For deletions, the suitable locations are 1) before or after any punctuation (". ", ",", "(", ")", "[", "]", "{", "}", and ";"), 2) before or after any operator (e. g. "+", "-", "*", "=", "+="), and 3) in any token sequence longer than one indentation character. The Styler 3grams protocol is meant to produce likely errors. It performs modifications at the abstract token level instead of directly changing the Java source code as Styler random. The idea is to replace formatting tokens by the ones used by developers in a similar context (i. e. the same surrounding Java tokens). For that, we use 3-grams, where 3gram = {Java_token, f ormatting_ token, Java_token}. So given an error-free Java file, the task of Styler 3grams is the following. First, the Java file is tokenized (see Section IV-D), and a random formatting token is picked and used to form a 3-gram, which is 3gram orig.
Then, given a corpus of 3grams previously mined from a project, Styler 3grams finds a 3gram i-corpus that matches the surrounding Java tokens of 3gram orig. Several matches can be found, but the selection of a 3gram i-corpus is random according to its frequency in the corpus. Then, 3gram orig is replaced by 3gram i-corpus : since the Java tokens match, only the formatting token is actually replaced. Finally, Styler 3grams performs a de-tokenization so that an error version of the original error-free Java file is created. Algorithm 1 presents the algorithm that STYLER uses to generate one training dataset per protocol (Styler random and Styler 3grams ). The input of the algorithm is the Checkstyle ruleset of the project, a corpus of error-free Java files taken from the project, the number of errored files to be generated, and the injection protocol to be used.
F. Error Encoding In order to repair formatting errors, the Java source code encoding using an abstract token sequence (see Section IV-D) must capture both the error in the code and the context surrounding the error. Therefore, STYLER considers a token window of k lines before and after the error. f ile ← selectRandom(f iles) 7: f ile ← changeF ormatting(f ile, protocol) dataset. append(erroredF iles) 13: end while 14: return dataset Once the context surrounding a formatting error is tokenized, STYLER places two tags around the error, so that its location and its violation type can be further identified. The tags consist of the name of the Checkstyle rule that was violated and raised the error. For instance, the error presented in Figure  4a  is about the Checkstyle LeftCurly rule, so the tags around the error are <LeftCurly> and </LeftCurly> as shown in Figure  4c.
G. Machine Learning Model Learning (Figure  3-step C ). STYLER aims to translate a buggy token sequence (input sequence) to a new token sequence with no Checkstyle errors (output sequence). STYLER uses a sequence-to-sequence translation based on a recurrent neural network LSTM (Long Short-Term Memory), similar to what is used for natural language translation. Thanks to the token abstraction employed by STYLER to encode Java source code    (see Section IV-D and Section IV-F), the input and output vocabularies are small (respectively ∼150 and ∼50), hence are well handled by LSTM models. We use LSTM with bidirectional encoding, which means that the embedding is able to catch information around the formatting error in the two directions: for instance, an error triggered by the Checkstyle WhitespaceAround rule, which checks that a token is surrounded by whitespaces, requires the contexts before and after the token.
H. Repair Verification and Selection STYLER performs x predictions per training data generation protocol (i. e. Styler random and Styler 3grams ), so in the end STYLER generates x × 2 predictions to repair a single error. After the translation of those predictions back to Java source code (Figure  3 -step G), STYLER performs a verification (Figure  3 -step H), where Checkstyle is executed on the resulting Java source code files. From the correctly repaired files (i. e. the ones that do not result in Checkstyle errors), STYLER selects the best one to give as output, where the best prediction is the one that has the smallest source code diff (Figure  3 -step I).
RQ #1 [Accuracy]: To what extent does STYLER repair realworld Checkstyle errors, compared to other systems? Overall accuracy is an important metric to measure the value of tools. We investigate the accuracy of STYLER on real Checkstyle errors, which allows us to understand to what extent STYLER repairs formatting errors that have occurred in practice. Moreover, we compare the accuracy of STYLER to the accuracy of three code formatters, by using the same dataset of errors, to investigate if, and to what extent, STYLER outperforms the competing systems.
B. Data Collection To answer our research questions, we create a dataset of real Checkstyle formatting errors by mining open source projects. For that, we first build a list of projects to collect errors from by filtering projects from our study presented in Section III. We select the projects that 1) use Checkstyle, 2) have only one Checkstyle ruleset file, 3) contain at least one Checkstyle formatting rule in the Checkstyle ruleset, and 4) use Maven. This results in 1,791 projects. For each project, we try to reproduce Checkstyle errors with the following procedure. We first clone the remote repository from GitHub  6. Then, we search in the history of the project for the last commit (c n ) that contains modifications in the checkstyle. xml file: this commit is used as a starting point for the reproduction of real errors.
C. Systems Under Comparison We selected three systems to be compared with STYLER: one is an IDE-based code formatter plugin for Checkstyle, and the other two are the state-of-the-art of machine learning formatters that aim to assist developers to fix code formattingrelated issues without any prior or ad-hoc formatting rules. 1) CHECKSTYLE-IDEA: CHECKSTYLE-IDEA  [9] , also referred as CS-IDEA in this paper, is a plugin for the IntelliJ IDE. It provides IDE integrated feedback against a given Checkstyle ruleset and suggests fixes for Checkstyle errors. 2) NATURALIZE: NATURALIZE  [3]  is a tool dedicated to assist developers on fixing coding conventions related to naming and formatting in Java programs. It learns coding conventions from a codebase and suggests fixes to developers such as formatting modifications, based on the n-gram model.
To run CODEBUFF, we give it the required configuration, including the number of spaces for indentation. This number is based on the most common indentation used in the considered projects (usually two or four spaces). 3) Training tools: We trained STYLER for each project in our real error dataset. The training process includes a step for creating the training data (see Figure  3-step A ), where we create 10,000 errors per project. To conduct a fair evaluation, we ensure that STYLER learns repairs based on the same Checkstyle ruleset that is used for the real errors in the evaluation. Therefore, for each project from the real error dataset, we select as training seeds all error-free Java files from the last commit that modified the checkstyle. xml file used to collect the real errors. We take special care of consistency in the observed results: all three machine learningbased systems, STYLER, NATURALIZE, and CODEBUFF, are trained using the same Java files.
STYLER repairs 38% of the errors while CS-IDEA repairs 63%, which is the greatest overall accuracy among the four considered tools. NATURALIZE and CODEBUFF repair less errors (13% and 15%, respectively). To check if there is a significant difference between STYLER and the other tools, we used McNemar test and we considered α = 0. 05: we found p-value=0. 000 for all three tests. This means that STYLER and any other tool have a different proportion of errors. We note that STYLER and CS-IDEA are the most reliable tools in the sense of delivering to an end-user either a repaired source code or, in the worst case scenario, the code with the same error. It is not the same case of NATURALIZE and CODEBUFF, which have higher rates of delivering source code with new errors or broken. They were, however, designed for a different goal, and do not take into account the Checkstyle ruleset of the project like STYLER and CS-IDEA do.
The reason for CS-IDEA's overall good performance is that it works well in the three most frequent rules in our dataset: LineLength, EmptyLineSeparator, and FileTabCharacter. STYLER, on the other hand, has a perfect success rate in rules that are not that frequent: EmptyForIteratorPad, GenericWhitespace, NoLineWrap, and ParenPad. Moreover, STYLER is the only tool that fixes (two) rules exclusively: NewlineAtEndOfFile and NoLineWrap. This shows that Styler is able to repair error types for which one would not need to put engineering effort to write the repair code (as opposed to common rules). STYLER is reasonably good for fixing LineLength. This is interesting because LineLength is a configurable rule, where the developers specify the maximum length allowed for lines. Fixing a LineLength error is not straightforward: it requires the repair tool to localize a place for breaking the line such that 1) the maximum length is not reached and 2) no compilation error is introduced.
C. Size of the Repairs (RQ #3) One dimension of repair quality is the size of the diff (added + deleted lines) between the source code with a Checkstyle error and the repaired source code. Among all repairs that pass all Checkstyle rules, the diff should be as small as possible for being the least disrupting for the developers. In the context of a pull request on GitHub, a smaller diff is usually considered as easier to review and merge  [12]. We calculate the size in lines of the diff from the errors that STYLER, CS-IDEA, NATURALIZE, and CODEBUFF repaired. Figure  7  shows the results: the x axis presents the size distribution of the diffs, and each boxplot represents one tool. STYLER (in green) and NATURALIZE (in yellow) have the smallest medians of diff size, which are both equal to five changed lines.