dataesr/scientific-paragraphs-categorization huggingface dataset:
| WHAT S HOULD CODE RE VIE W E VA LUATE ? Code review is the process of either formally (as part of the peerreview process) or informally (as co-authors or colleagues) checking and evaluating each other's code. It is critical to help avoid conceptual, programmatic, and syntactic errors in code and can take place at any stage of the research cycle; pre-submission, during formal peer review, or post-publication. Although the manner and scope in which code review occurs may vary depending on the position in the research cycle, the core priorities remain the same: to ensure code is as reported in the methods section, is able to successfully run, is reliable, and is able to reproduce stated results. Below we describe these key priorities as the four Rs of code review (Figures  1  and 2 ):.
| Is the code as reported? Code is a key research output and a critical component of scientific methodology. As such, open code accompanying written methods sections is becoming more common, following similar pushes for Open and FAIR data  (Lamprecht et al. , 2020). Therefore, it is imperative that code is checked for consistency when presented with the corresponding manuscript. These questions help us avoid concep- appropriate version numbers) are stated somewhere in the manuscript. In general, it is good practice to, at the very least, list the packages (with version numbers) that are integral to the analysis or to visualization in the manuscript. These can be obtained by using the 'citation()' function in R or using the {setuptools} package in Python. A full list of all packages used (and versions), for instance those involved with cleaning and tidying of data, could be given elsewhere such as in an associated.
R or. py file. Importantly, this will allow for any package or module with versions that are found to contain bugs or coding errors to be identified at a later stage. Packages such as {renv}  (Ushey & Wickham, 2023; which replaces {packrat}, Ushey et al. , 2022) , {groundhog}  (Simonsohn & Gruson, 2023) , or {poetry}  (Eustace, 2023)  and {pipenv}  (Pipenv Maintainer Team, 2023)  in Python can help with ensuring a reproducible environment and allow for specific loading of desired package versions. Another option is containerization through the use of Docker  (Boettiger, 2015; N. B. detailed tutorials already exist which highlight the use of this reproducible method in far more detail than we will discuss here).
| Does the code run? Even if code matches the methodology reported in a paper, this does not mean the code is executable (i. e. , can 'Run'). Programmatic and syntactic errors can make code fail to rerun. For example, code will not be able to be run if it includes calls to libraries (or modules) that are not installed in the current computing environment or if there are spelling mistakes (Figure  1 , Box 1 in Data S1). Data sharing, where possible, should accompany code sharing, so that code can be fully rerun with the original data. If data sharing is not possible, simulated data or a data snippet should be provided so that the code can be rerun. In cases where it would take a long period of time to rerun code (for instance with some forms of Bayesian modelling), the code should be accompanied with appropriate model outputs (readily provided by the author, see below 'Output reproducibility').
| Is the code reliable? Errors can still propagate through code that runs and produces an output, because code can produce incorrect results in a reproducible manner (i. e. , every time the code is run). For example, if code selects or modifies the wrong column in a dataset, the code will still run, but produce a reproducible yet inaccurate result (i. e. , the code is not 'Reliable'; Figure  1 , Box 1 in Data S1). This type of error could easily be conceptual, arising from a misunderstanding of the dataset, or programmatic, such as from indexing by number and producing a mistaken column order or from user-defined functions. Although some coding techniques, such as explicitly indexing by column name or by performing unit testing of any user-defined function (see  Cooper, 2017) ; relevant packages include {testthat},  (Wickham, 2011)  in R or {pytest} in Python  (Okken, 2022) , can help avoid many of these mistakes, this type of error is common and also extremely difficult to pick up by anyone without deep familiarity with the dataset and code.
In particular, these errors are thought to scale with the number of lines and complexity of code  (Lipow, 1982). Although intrinsically linked to evaluating whether code can be run (the second 'R'), evaluating code reliability means not only ensuring that the code runs to completion without error, but examining intermediate outputs of the code to ensure there are no mistakes. The functions 'identical()' in R and 'numpy. array_equal()' in Python can be useful at this stage of code review to compare object similarity between newly generated and previously saved intermediate outputs.
| Are the results reproducible? The last 'R' of code review builds on the previous code review stages, and is perhaps the most fundamental: can the code produce the output, and thus support the conclusions, given in the paper  (Goodman et al. , 2016 ; Figure  1 , Box 1 in Data S1)? As several recent papers have highlighted  (Archmiller et al. , 2020; Errington et al. , 2021; Minocher et al. , 2021; Obels et al. , 2020; Tiwari et al. , 2021) , reproducibility in research results is often very low. Therefore, the final step of code review is ensuring that final outputs when code is rerun match those reported in the analysis and results sections (including any relevant figures and narrative text contained within these sections). With that said, at times obtaining the exact same result is not possible.
Some level of tolerance must therefore be applied especially when dealing with stochastic methods in which parameter estimates will change between subsequent runs or with techniques that are computationally demanding and slow. This can occur for example if the 'set. seed() function in R or 'random. seed()' function in Python F I G U R E 2 A basic workflow for reviewable code that can be adopted from the onset of a project. See Data S1 for a printable checklist of the points listed here. has not been used prior to stochastic sampling. Providing model outputs can go some way in helping with this (see above); however, it does not allow for the code to be explicitly run to see if you can obtain similar results as stated in the paper (regardless of potential time taken). In this case, newly generated results should be compared to the originally reported results.
| S E T TING UP YOUR CODE FOR EFFEC TIVE CODE RE VIE W Code review should evaluate if code matches reported methods, whether code runs and is reliable, and lastly, if results can be reproduced. But in order for these questions to be addressed, code must be written and shared in a way that it is possible for someone else to rerun an analysis; both to allow for code to be reviewed and to be reused in the future when properly maintained and contained  (Boettiger, 2015). For this to happen, all necessary scripts must be shared along with appropriate metadata indicating how the scripts interact with one another, along with describing all other necessary software and appropriate versions. Often, researchers lack formal training in coding, and learn to code in an ad hoc fashion that excludes training on general styling, appropriate use of workflows, and project organization.
| Project organization Every project needs some form of directory organization and folder structure. This is likely to be largely driven by the function and form that your research takes, but an efficient and transparent folder structure that keeps raw data separate from code and intermediate outputs should be created. This helps to ensure that raw data is not accidentally modified or overwritten if any data cleaning or wrangling techniques are applied. A simple folder and file structure such as this will go a long way to help researchers from all coding skill levels understand the order and flow of the data analysis, particularly when the user creates sequentially labelled subfolders and scripts where someone following the code knows which order things must be run (e. g. , files beginning with '01…') in addition to dividing and naming folders to fit their purpose (e.
g. , data, scripts, function). Several incredibly useful examples already exist  (Alston & Rick, 2021; Chure, 2022; Cooper, 2017 ; see also https://coder efine ry. github. io/ repro ducib le-resea rch/ and https://lakens. github. io/stati stical_infer ences/ 14-compu tatio nalre produ cibil ity. html). Project code should be stored and available on any data or code repository. Another option for organizing a project is to use pipeline or workflow tools (for instance see https://github. com/pdito mmaso/ aweso me-pipeline), such as the {targets}  (Landau, 2021)  and {workflowr} R packages  (Blischak et al. , 2019)  or the {luigi} package  (The Luigi Authors, 2023)  in Python (see https://www. marti nalar con. org/2018-12-31-a-repro ducib le-scien ce/). These tools allow users to automate the process of data analysis, taking a raw dataset through the steps necessary to produce data analysis and visualization.
The advantage to the user is that the code is compartmentalized into logical steps (e. g. , import raw data, data cleaning, data wrangling, data analysis, data visualization) and any changes to the code only affects the downstream steps. For example, if we change the type of analysis we do, we do not need to re-import the data or clean it again. This saves time in computation (especially important for complex, long-running pipelines) but is also advantageous for reproducibility, and sharing and reuse of code. Reviewers can effectively rerun the steps needed to produce a data analysis or figure without having to rerun timeconsuming pre-processing steps.
| Project and input metadata Projects will instantly have better organization and increased reproducibility when users know how they should work through the various folders and subfolders. A README text file and additional metadata gives users the signposts required to facilitate rerunning of code. This can contain information on the packages used (e. g. , the package name and version number), along with a detailed description of the various data files, project aim, contact information of the authors, and any relevant licences in place for code or for data (see https:/choos ealic ense. com/licen ses/ for more information). Furthermore, key information about source data is critical for reproducing analysis code. If sharing data are inappropriate to your study (for example when dealing with sensitive confidential data) or if data are so large it cannot be easily shared, then a user can provide a sample of simulated data or a primer so that the code can be checked and read  (Hennessy et al.
, 2022; Quintana, 2020). However, if data are readily available, then providing detailed information about what the data are (preferably in an associated README) and where the data are (e. g. , stored on a free data repository such as The Open Science Framework (OSF), Zenodo, or for ecology data, the Knowledge Network for Biocomplexity) should be provided. Metadata should include information, such as where the data come from, who the owners are, as well as what each column header entails, and any relevant acronyms or shorthand notation (ideally following FAIR principles, so data are Findable, Accessible, Interoperable, and Reusable; see  Lamprecht et al. , 2020). This is particularly useful when controlled vocabulary is used throughout, and R packages such as {codemeta}  (Boettiger, 2017)  and {dataReporter}  (Petersen & Ekstrøm, 2019)  or Python packages such as {CodeMetaPy} (van Gompel, 2023) and {cookiecutter}  (Feldroy, 2022)  can help with this.
| Code readability Good readability of code is extremely important in enabling effective code review. Several quick solutions exist to provide increased clarity: (1) explicitly calling packages (via a package's namespace, e. g. , package::function() in R or package. module. function() in Python); (2) using relative file paths (for instance using the {here} package (Müller, 2020) and preferably with an associated R project file, if using R and RStudio or in a virtual environment if using Python); (3) removing redundant packages; and (4) writing analysis code with clear subheadings, intuitive coding comments, and easy-to-understand object names. Best practice coding tips can be implemented by R packages such as {styler}  (Müller & Walthert, 2020)  or {pycodestyle} in Python  (Rocholl, 2022)  and can format code in a number of standardized styles (e.
g. , Google, tidyverse in R, or PeP8 in Python) with a single line of code or a click of a button. Furthermore, the use of R/ Python Markdown, Quarto or the open-source integrated development environment, Jupyter Notebook (or its extension, JupyterLab) enables users to present code in chunks which, along with suitably descriptive comments, allows for far easier readability. In addition, several recent guides and primers have been written that focus on increasing coding cleanliness  (Filazzola & Lortie, 2022; Hunter-Zinck et al. , 2021; Sweigart, 2020) , so we urge the reader to consult these guidelines for tips and advice on improving code readability.
| Output reproducibility One of the key principles and requirements of code is the ability to correctly reproduce published graphs, statistics, and results. In order to do so, a user's code needs to provide a clear link between each section of the code and the various reported graphs and outputs to enable comparison of code to paper and to results. This should then facilitate checking that the results produced by the code match the stated results in the publication. In some cases, reproducing analysis from models can take considerable time to complete, for instance, when re-running complicated Bayesian models or other techniques involving long computational time. In this case the 'exact' reproducibility of results is not always possible if code must simulate a stochastic process (e. g. , Monte Carlo sampling methods).
| PRE-PUB LI C ATI ON: S E T TING UP A CODE RE VIE W G ROUP Informal training coupled with insufficient time and incentives  (Touchon & McCoy, 2016)  means that coding and subsequent analysis are often the responsibility of a single member of a team throughout a project's entire lifetime. This is in stark contrast to the research-team wide collaboration typical when developing methodology and experimental design. The individual nature of writing research code is part of what makes pre-publication code review so unlikely, but even more critical. Although code review has a place in the formal peer-review process and post-publication, one of the most important places for code review to take place is before publication. To achieve this, there must be a culture of peer code review among research teams.
One of the most effective methods by which researchers can establish a culture of peer code review in research labs or among colleagues is by setting up a code review group. Here, we draw on our experience building a code review club (which we set up in collaboration with the Society for Open, Reliable, and Transparent Ecology and Evolutionary Biology, SORTEE) to present tips for establishing this type of community. In particular, we focus on advice for removing the barriers people have towards sharing their code and receiving feedback; be these due to a lack of time and incentive, a lack of technical knowledge and unclear workflows, or due to social pressures and the fear of being judged by peers  (Gomes et al. , 2022).
| Encourage collaboration from the start of a project Code review can begin as early as the first initiation of a project and play a role beyond publication; it is useful to keep continuous code collaboration at all stages of a manuscript. Collaboration can be facilitated through various code-sharing platforms, such as GitHub where users can submit and comment on pull requests  (Braga et al. , 2023). At SORTEE, we established a peer-review group and used GitHub issues to summarize discussion of an individual's code during an interactive zoom session (see https://github. com/SORTE E/peer-code-revie w/issue s/8 for an example including a summary). However, it is important to find a method of facilitating code review that works for your group.
| Set clear goals for the review Setting out what you want to achieve with each code review session is particularly important when it comes to organizing peer-review meetings. Is the focus on general learning and improving readability or is it to error-check and scrutinize the reproducibility of your code? Having a clear structure and goal for each peer-review session is important in order to focus comments and advice to address the precise reason for review. Similarly, unless the aim of a code review is to evaluate different analytical options, it would be better to leave methodological questions aside to ensure code review is streamlined.
| Normalize coding errors and establish a judgement-free environment Code review volunteers often feel very anxious about showing code that may have errors. It is therefore vital to normalize the existence of errors and highlight that perfection is never possible. It is also useful to stress that there is no such thing as bad code  (Barnes, 2010)  and there are usually multiple ways to approach the same problem  (Botvinik-Nezer et al. , 2020; Silberzahn et al. , 2018). One of the most important statements for peer code review is that there is no single way to code. It is important for code review not to get bogged down by modifying or homogenizing style; as long as code is readable, then coding diversity should be encouraged. It is important to create a relaxed environment where people can learn and correct mistakes without judgement or fear of failure and everyone in the peer-review group should have a chance to contribute and speak.
| Carefully consider group size Usually, a smaller group is a friendly starting point for peer code review because it allows people to feel more comfortable speaking up and participating. Small peer-review groups (potentially even one-to-one) can better facilitate peer-to-peer learning and a more focused review of code. However, there are also times when larger groups are more effective, such as having wider discussions on general themes and tips. It is worth considering the aims in establishing the group to help guide the ideal size. For instance, if your goal is to facilitate more general discussions, then a big group size is more likely to enable this. However, if your goal is to enable more focused review of code, then perhaps it is better to reduce the size of the peer-review group for this purpose.
| DURING PUB LIC ATION: FORMAL CODE RE VIE W One of the most crucial aspects of code review can take place during the formal peer-review process. This is where reviewers are able to carefully follow and understand the logic of analyses, much like the flow of writing from the introduction to the discussion of a paper  (Powers & Hampton, 2019). In some journals, such as The Royal Society ('Data sharing and mining | Royal Society', 2023), Behavioural Ecology and Sociobiology  (Bakker & Traniello, 2020) , and The American Naturalist  (Bolnick, 2022)  both code and data are requested for review at from the submission stage (although, based on our experience, this is not enforced). In some cases, such as in Journal of Open Source Software, the entire process of formal peer review, including that of code and manuscript is hosted on GitHub and implemented via GitHub issues (see https://github.
com/openj ourna ls/ joss-revie ws/issues for several useful examples). This, as Fernández-Juricic (2021) points out, has several benefits. For authors, providing code during peer review could lead to an increase in the quality of the manuscript, and for reviewers, available code allows for a far deeper insight into the manuscript as there is a clearer link between experimental methodology and statistical analysis (the First R; code as 'Reported'). These benefits are substantial and could ultimately contribute to the adoption of code review during the publication process by journals. However, beyond the availability of code during submission, there are numerous other hurdles before effective and in-depth code review can be reasonably formalized as part of the peer-review process. One of the most pressing issues is finding suitable individuals to review code given there is already a lack of willing reviewers in the current system.
It is reasonable to expect reviewers to check that code is as reported, but anything more in depth could take up the time of already overworked academics, who may not necessarily have the exact expertise needed to check other people's code. Reviewers could be asked to state if they are competent to review the code is as reported, and the journal could ensure that at least one reviewer has checked the code. Another vital issue is how to provide code and data during peer review within a double-blind peer-review system (which has been shown to significantly reduce peer-review bias; see  Fox et al. , 2023). There are several solutions to this. Anonymized data and code could be submitted directly to the journal during review (e. g. , as a zip file). Alternatively, the anonymized data and code could be uploaded to repositories such as figshare, the Open Science Framework (OSF), GitHub or Dryad (although we note the latter may not be freely available).
These repositories allow the authors to provide an anonymized link for peer review (see also https://metho dsblog. com/2023/08/23/doubl e-anony mous-peer-revie w-frequ ently -asked -quest ions/ for useful links to repositories that enable doubleblinding). While there are still issues that need to be fully considered before any kind of extended code review becomes a standard part of the peer-review process, the mandatory provision of code for peer review alongside the explicit expectation that at least one reviewer checks the code matches the reported methodology (i. e. , Is the code as reported?) would make a strong start in shifting publication culture and increasing the reliability of published research.
As Figure  2  from  Culina et al. (2020)  shows, although the number of journals that possesses a mandatory code rule is increasing (from 15% in 2015 to 79% in 2020) the number of articles that actually provide open code is still around 27% (although this number varies considerably among journals). This suggests that not many authors are adhering to this policy, which is an impediment to computational reproducibility  (Culina et al. , 2020). However, there is hope to be found here. As  Culina et al. (2020)  have shown, journals requiring code to be shared are increasing in number yearly, and as a field, we already have improved substantially  (Jenkins et al. , 2023; Mislan et al. , 2016). In some cases, journals have implemented far stricter (and rightly so) data and code requirements along with assigning corresponding data editors  (Bolnick, 2022).
However, the first necessary step is for all journals to make it a requirement for both code and data to be present from the very start of the submission stage  (Fernández-Juricic, 2021; Powers & Hampton, 2019). But what happens if the code is not available? In this case, the main option is to reach out to the corresponding author (or perhaps the journal itself) and ask if the code could be made available; similar to data being made available 'upon reasonable request'. The next part is relevant to the previous section above ('What should code review evaluate?'). If you find that the code associated with a manuscript does not adhere to any of the 'R's listed above, then the first step is to contact the corresponding author (or if the paper uses the MeRIT system  (Nakagawa et al. , 2023) , the person who actually conducted the analysis).