dataesr/scientific-paragraphs-categorization huggingface dataset:
Static tests Since static analysis tools are limited for Fortran, we have developed small scripts to test two simple aspects of the code. First, a naive dead-code detection is made at the procedure level by identifying procedures defined but never called. Second, a scripts checks that the coding rules of the project concerning things such as case consistency between declaration and use or indentation are correctly applied. Each of these scripts returns a status whether a problem has been found in the code and in such a case, a list of the problems found. Since the code is not even compiled to apply these tests, they do not depend on any of the previously identified parameters: execution platform, compilation options or execution parameters. Each test only takes a few seconds to run. Another static test applied is a compilation test that returns a status whether the code can be fully compiled and the list of errors and warnings generated by the compiler.
The compilation parameter space is slightly reduced however as the code must be compiled in deterministic mode for the tests to be possible. This still leaves an extremely large range of parameters that can not be exhaustively explored and choices have to be made regarding the configurations to test. The process we have adopted is to design a small basic test and derive new tests from it whenever a defect is discovered that was not covered by the existing tests. The base test uses the default compilation options and has been designed to run on all execution machines including the VMs. Since the VMs only have two cores, this is the maximum number of threads that can be used, but they can either be used as two OpenMP threads inside a single MPI process or inside distinct MPI processes. We have chosen this second option which in in Gysela has more impact on the executed code path than the number of OpenMP threads.
The bitwise comparison approach enables to used a very small domain size, we use 1 M points (N r = 32, N θ = 64, N ϕ = 16, N v = 16, N µ = 2). Similarly, the number of iterations can be reduced to a minimum as long as all parts of the code are executed at least once such as the computation and output to disk of derived values in Gysela, known as "diagnostics". We use 2 iterations and have set-up the configuration so that diagnostic are executed every two iterations. Additional tests have been designed to catch defects that this base test did not detect. These tests typically differ from the base by changing the value of a parameter in the namelist used to choose between alternative algorithm in the code. This type of change does indeed have a much larger impact on code coverage than the values used to initialize the data manipulated by the code.
Executing the code requires interfacing with batch schedulers but this aspect was already handled by a script provided for the use of Gysela by humans in production, the only change that was required was the support of a synchronous mode as described in Section 3. 2. This leaves two technical questions open: the approach used for comparing results and the storage of the reference data. We have chosen to use the HDF5 checkpoint files generated at the end of Gysela runs for comparison since by definition they contain the whole state of the simulation. In addition to small scalar describing the iteration number and such, the most important field stored in the Gysela checkpoint files is the ion distribution function. HDF5 provides the h5diff command that compares arrays inside HDF5 files at the precision specified as parameter.
Unit tests Like many HPC scientific simulation software, Gysela has been designed in a very monolithic way and extracting unit tests is often difficult. The unit tests available are therefore limited to tests of: 1) the deterministic MPI reduction implementation, 2) the gyroaverage operator implementation and 3) the checkpoint-restart mechanism. The deterministic MPI reduction implementation has been described is Section 2. 3. 3 and reproduces the pre-defined MPI interface. It is therefore as independent of Gysela as MPI and can be unit tested. The test compares the results of reductions using the deterministic implementation with those of the underlying implementation and returns an error in case the difference between the two crosses a threshold. It tries to use a large range of features to validate the API as much as possible, including using the INPLACE parameter, various kind of Fortran arrays (normal, allocatable, pointers) and different lower and upper bounds.
The development process starts with a reference implementation of the gyroaverage based on Hermite interpolation. So every optimized implementations must reproduce the same results of the reference implementation on the unit test. Between 15 and 20 seconds are required to test the 12 gyroaverage methods available in Gysela including compilation. The last test is not strictly speaking a unit test because is executes the complete code due to the lack of modularity of Gysela. It does however specifically validate the checkpoint-restart mechanism of the code. The approach we have implemented consist in comparing the results of two executions of the code with similar parameters, except one stops after a checkpoint in the middle and restarts for the second half of the run while the other executes the complete run without any restart.
The comparison of the two executions uses the same approach as for bitwise comparison. The namelist file used is the same as that used for bitwise comparison tests except with a restart inserted. The time required for their execution is therefore very close at about 2 minutes. Unlike the bitwise comparison tests, this one does not require any data to be stored since the results of two executions are compared. Since they do not test the actual code, the domain of unit tests is not impacted by Gysela compilation and execution options. Only the execution platform impacts their result. They are also very fast to execute and are therefore integrated in the tests executed for any merge request. Since their number is very small currently and their execution very fast, we have not implemented any tool to limit the execution of unit tests to the parts of the code that did actually change.
Applying this approach requires a non negligible investment that might prevent other codes from doing it. The return on investment is however very positive in multiple aspects. The amount of computation time lost in production due to an error is greatly reduced. Developers can focus on implementing new features rather than correcting defects. The confidence in the results is increased and physicists spend less time figuring out if a given result is an error or is actually physically relevant. There are two directions we are currently exploring to extend this work. The first direction is to extend the quality and quantity of tests executed in the Gysela automated testing procedure. We for example plan to add a code coverage analysis to asses the amount of code actually tested. The more interesting tests we now target do however rely on more invasive approaches.
Other parts on the other hand can be reused from one code to the other. This is the case for multiple elements we intend to make available to the community, including the Cktrace library, the deterministic MPI reductions or the deterministic compilation framework. In addition, we are in the process of making an automated testing platform available in collaboration with the computing centers so as to handle the technical aspects once for all code and to enable the use of the same supercomputers as those used in production for testing.
Background: The CONSORT Statement is an evidence-informed guideline for reporting randomised controlled trials. A number of extensions have been developed that specify additional information to report for more complex trials. The aim of this study was to evaluate the impact of using a simple web-based tool (WebCONSORT, which incorporates a number of different CONSORT extensions) on the completeness of reporting of randomised trials published in biomedical publications. Methods: We conducted a parallel group randomised trial. Journals which endorsed the CONSORT Statement (i. e. referred to it in the Instruction to Authors) but do not actively implement it (i. e. require authors to submit a completed CONSORT checklist) were invited to participate. Authors of randomised trials were requested by the editor to use the web-based tool at the manuscript revision stage.
Authors registering to use the tool were randomised (centralised computer generated) to WebCONSORT or control. In the WebCONSORT group, they had access to a tool allowing them to combine the different CONSORT extensions relevant to their trial and generate a customised checklist and flow diagram that they must submit to the editor. In the control group, authors had only access to a CONSORT flow diagram generator. Authors, journal editors, and outcome assessors were blinded to the allocation. The primary outcome was the proportion of CONSORT items (main and extensions) reported in each article post revision. Results: A total of 46 journals actively recruited authors into the trial (25 March 2013 to 22 September 2015); 324 author manuscripts were randomised (WebCONSORT n = 166; control n = 158), of which 197 were reports of randomised trials (n = 94; n = 103).
Over a third (39%; n = 127) of registered manuscripts were excluded from the analysis, mainly because the reported study was not a randomised trial. Of those included in the analysis, the most common CONSORT extensions selected were non-pharmacologic (n = 43; n = 50), pragmatic (n = 20; n = 16) and cluster (n = 10; n = 9). In a quarter of manuscripts, authors either wrongly selected an extension or failed to select the right extension when registering their manuscript on the WebCONSORT study site. Overall, there was no important difference in the overall mean score between WebCONSORT (mean score 0. 51) and control (0. 47) in the proportion of CONSORT and CONSORT extension items reported pertaining to a given study (mean difference, 0. 04; 95% CI -0. 02 to 0. 10).
Design We conducted a multi-journal, two-arm parallel group, randomised trial to assess the impact of the WebCON-SORT tool compared to a control intervention on the completeness of reporting of randomised trials submitted to biomedical journals. The study obtained ethics approval from the University of Oxford Central Research Ethics Committee, Oxford, UK (MSD-IDREC-C1-2012-89) and is registered on ClinicalTrials. gov (NCT01891448).
Journal participants To be eligible for inclusion, journals were required to (1) endorse the CONSORT Statement (assessed via journal Instruction to Authors and as listed on the CONSORT website: www. consort-statement. org); (2) not actively implement the CONSORT Statement (defined as requiring authors to submit a completed CONSORT checklist alongside their manuscript at the time of article submission); and (3) publish reports of randomised trials (criteria assessed February 2013). All journals that met the above inclusion criteria were sent an email (February 2013) from the WebCONSORT study scientific committee inviting them to participate in the study. The description of requirements for participation were included in the email and study information sheet (Appendix 1) and editors were asked to verify that they complied with these criteria and that, while they endorsed the CONSORT Statement, they do not actively implement it.
If a journal agreed to participate, and confirmed they met the eligibility criteria, then the journal editor was asked (Appendix 2) to include an electronic web address to the WebCONSORT study website in their request for revision letter to authors for any manuscript identified by the journal as reporting the results of a randomised trial. We did this by asking the journal to include this standard sentence in their revision letter to authors: "As part of the process of revising your manuscript we would like you to use the WebCONSORT tool which is designed to help you improve the reporting of your randomised trial. You can access the tool by clicking on the following link: [link to WebCONSORT study site]. Please be aware that by submitting your manuscript to our journal it may be part of a research study, any participation will not impact on any future acceptance or rejection of your manuscript".
Random assignment Authors registering on the WebCONSORT study website were asked to provide some basic information about their randomised trial. This included the name of the journal where the manuscript was submitted, the manuscript number and title, name of submitting author, trial design (e. g. parallel, cluster, non-inferiority, pragmatic), type of intervention (e. g. non-pharmacologic, herbal, acupuncture), and number of study groups (arms). Registered manuscripts were then randomised into two groups (i. e. WebCONSORT tool or control). The sequence of randomisation was computer generated and stratified by whether or not a CONSORT extension was relevant. The assignment was centralised using a web-based system. Authors and journal editors were blinded to allocation of the intervention.
The tool automatically generated a unique list of items customised to a specific trial combining the list of items from the main CONSORT and the items from all relevant extensions (e. g. for a pragmatic trial evaluating a non-pharmacological treatment with cluster randomisation, the main CONSORT checklist was combined with three extensions: pragmatic trial, cluster trial, and non-pharmacological extensions). This list was generated based on the description of the trial made by the author (i. e. type of design and interventions). A website (Appendix 3: Figure  6 ) was created where authors were able to log on and register. Using a drop-down menu, they could select their precise type of trial, taking into account the methodological characteristics. Authors were unaware that they were randomised by the software to the WebCONSORT or control intervention.
Experimental intervention Authors randomised to the WebCONSORT arm were directed to a list of CONSORT items specific to their trial which they could print out. They could also obtain an automatic flowchart adapted to the design of their trial. Authors were told that the items generated by the WebCONSORT tool should be reported in the revised manuscript and that the completed checklist and flow diagram should be submitted to the editor. The content of the WebCONSORT tool was validated by members of the study team; this was done by performing a number of "dummy" randomisations to ensure the correctly formatted customised checklist was generated based on different numbers and types of CONSORT extensions being selected. The WebCONSORT tool website was also tested by the scientific committee of the study and by external experts with experience in designing and conducting clinical trials to ensure the website was clear and well understood.
The secondary outcomes were the mean proportion of adequately reported items from the main CONSORT Statement (based on the 10 items for the primary outcome above), and the mean proportion of adequately reported items for each of the relevant CONSORT extensions (based on the five items for the primary outcome above). We also collected data on the rejection rate of studies. We had planned to assess the compliance rate of authors submitting a CONSORT checklist to the journal and to obtain feedback from authors and journal editors on the review process; however, these proved difficult to implement in practice and hence were not assessed. The evaluation of revised manuscripts was conducted by a team of 10 reviewers (based at the Centre for Statistics in Medicine, University of Oxford), with statistical expertise in the design and reporting of clinical trials, working in pairs who were blinded to the nature of the study and allocation of the interventions.
Statistical analysis The main population for analysis were all manuscripts resubmitted to journals after the intervention occurred, which was during the revision process of the manuscript. Statistical analysis was undertaken using STATA IC (version 13). All outcomes were quantitative and described using proportions, mean, standard deviation, and minimum and maximum values. Quantitative variables with asymmetric distributions were presented as medians and interquartile ranges. For the primary and secondary outcomes, we estimated the difference between means in the two groups with 95% confidence intervals. The analysis was also stratified according to those articles which required the inclusion of one or more CON-SORT extensions and those which did not. Due to the much larger than anticipated incorrectly specified extensions, we also performed a post-hoc sensitivity analysis for both primary and secondary outcomes to exclude an extension from the analysis of a manuscript if it was wrongly selected by the authors.
Results Between 25 March 2013 and 22 September 2015, 357 manuscripts were registered on the WebCONSORT study site from 46 general medical and specialty journals with an impact factor ranging from 11. 34 to 0. 65 as of 2014 (Appendix 5: Table  4 ). Two journals (n = 33 manuscripts) subsequently withdrew and were therefore excluded as we were not able to obtain the revised manuscripts. Of the remaining 324 registered manuscripts, 166 were randomised to the WebCONSORT tool and 158 to the control intervention; of these, 197 were reports of randomised trials (and we were able to obtained the revised manuscript) and were included in the analysis (WebCONSORT n = 94; Control n = 103). Over a third (39%; n = 127) of registered manuscripts were excluded from the analysis. Reasons for exclusion were similar across study arms, the most common reason being that the study was not in fact a report of a randomised trial (Fig.
Characteristics of manuscripts of randomised trials Of those included in the analysis (n = 197), the most common CONSORT extensions selected were nonpharmacologic (WebCONSORT n = 43; control n = 50), pragmatic (n = 20; n = 16), cluster (n = 10; n = 9), and then non-inferiority (n = 9; n = 8), herbal (n = 2; n = 13), and acupuncture (n = 2; n = 0). Over two-thirds (64%; 72%) of manuscripts were registered as requiring one or more CONSORT extension. However, for almost a quarter (23%; 21%) of the manuscripts authors either wrongly selected an extension or failed to select the right extension when registering their manuscript on the WebCONSORT study site (Table  1 ). Most of the 197 trials were two-arm (WebCON-SORT 86%; control 82. 5%), about half were multicentre (45%; 46. 5%), half non-industry funded (50%; 53%), and the median sample size was 98 (IQR, 51 to 180).
Around one-third of interventions were drugs (42. 5%; 32%), a third were rehabilitation, psychological or educational interventions (30%; 36%), and just under a quarter were surgical or device interventions (23%; 19%). A CONSORT flow diagram was included in 85% and 86% of WebCONSORT and control manuscripts, respectively. Most manuscripts were Non-inferiority extension 3 5 Non-pharmacological extension There may be more than one reason for a miss match between the extension selected by the author and the extension which should have been selected subsequently published (81%; 84%) in the journal requesting the revision (Table  2 ).
Impact of the WebCONSORT tool on reporting of the revised manuscript There was no important difference in the overall mean score (primary outcome) between the WebCONSORT (mean score 0. 51; SD 0. 2) and control (mean score 0. 47; SD 0. 2) interventions in the proportion of CONSORT and CONSORT extension items reported pertaining to a given study (mean difference (MD) 0. 04; 95% CI -0. 02 to 0. 10) (Fig. 3 ). There was no difference between groups when the analysis was stratified according to those articles which were registered as requiring the inclusion of one or more CONSORT extensions (MD 0. 03; 95% CI -0. 03 to 0. 09) and those which did not (MD 0. 03; 95% CI -0. 07 to 0. 13) (Fig. 4 ), excluding manuscripts for which an extension wrongly selected by the author had little impact on the results (MD 0.
Principal findings and implications Our study is the first to evaluate the impact of a simple web-based intervention (WebCONSORT) that incorporates the original CONSORT checklist and different CONSORT extensions, on the completeness of reporting of randomised trials published in biomedical journals. Over 40 journals took part in this study, all of which endorsed the CONSORT Statement in their journal 'Instruction to Authors'. Our study found no overall  difference between WebCONSORT and the control intervention in the completeness of reporting of revised manuscripts. This finding suggests that creating a customised CONSORT checklist specific to an individual trial, for use at the revision stage of manuscript submission, may not optimise the use of CONSORT and its extensions. There are several potential reasons why we did not see an effect of the WebCONSORT intervention.
Firstly, all journals included in our sample already endorsed CONSORT, thus authors may have felt they complied with CONSORT guidelines as part of their original submission; although the low level of reporting seen in our study suggest this may not be the case. Secondly, it is possible that the combined customised WebCONSORT checklist had too many items and was overwhelming for authors to comply with. There were also no instructions on how to implement each item in the checklist, suggesting that the checklist alone may not provide sufficient information for most authors. It might be more effective to focus on a core set of CONSORT items with more detail about how to implement each item. Thirdly, it is possible that implementation of a WebCONSORT tool to improve reporting at the revision stage of a manuscript once submitted to a journal may be too late.
The process of conducting our study produced some other interesting findings with important implications. More than one third (39%) of registered manuscripts were excluded from the analysis as they were not reports of randomised trials. This was despite clear instructions provided to journal editorial staff, and included in the revision letter to authors, that only manuscripts reporting the results of randomised trials were eligible for inclusion. Clearly, the journal editorial staff at some journals were unable to correctly identify a randomised trial based on what was reported in the submitted manuscript. Another important finding is that in a quarter (23%) of manuscripts authors either selected an inappropriate CONSORT extension or failed to select the right extension applicable to their trial when registering their manuscript on the WebCONSORT study site.
Limitations Our study has several limitations. Firstly, we had to exclude a number of inappropriately registered studies from the analysis and, as such, we had less precision than anticipated to detect a difference between the WebCONSORT intervention and control. Secondly, we do not have information on the number of manuscripts, at each journal, which were eligible for inclusion in the study but where the author chose not to register their manuscript on the WebCON-SORT study website (and therefore be randomised). Finally, we do not understand the reasons why authors who registered their manuscript on the WebCONSORT study website and were randomised to the WebCONSORT intervention arm did not then address the recommended checklist items pertaining to their study in their revised manuscript. Future qualitative research to understand the potential barriers and facilitators to better implementation of reporting guidelines would be beneficial.