dataesr/scientific-paragraphs-categorization huggingface dataset:
However, in one of our preliminary experiments, the vitreous and SRF from three patients presenting with RRD were analyzed using the same proteomics strategy (unpublished data). We identified 70% of proteins that were common to the SRF and vitreous from these patients (257 of 367 proteins). Among these proteins, several known to be enriched in the vitreous such as opticin, ostoepontin, versican, retinoschisin, transthyretin, or hyaluronan-binding protein 2 (related to hyaluronic acid), or collagen alpha-2 were found. These data are consistent with the notion that RRD involves the creation of a retinal tear, leading to passage of liquefied vitreous humor through the tear into the subretinal space. Unfortunately, the proteomic strategy used in the present study does not allow us to separate the raw protein lists specific to CSCR or RRD SRF.
Acknowledgments Authors thank  Cindy Salvisberg  for technical assistance. Supported by grants from the  Swiss National Science Foundation  (NT: # PMPDP3_158370 ; AT: # 310030-156771 ; and FBC: # 320030_156401 ), from the  Agence Nationale de la Recherche  (FBC:  NR-15-CE18-0032  '' ROCK SUR MER ''), and from the  Faculty of Biology and Medicine Research Commission Fund, University of Lausanne  (AM). Disclosure:  L. Kowalczuk , None;  A. Matet , None;  M. Dor , None;  N. Bararpour , None;  A. Daruich , None;  A. Dirani , None;  F. Behar-Cohen , None;  A. Thomas , None;  N. Turck ,  None *LK  and AM contributed equally to this work †AT and NT contributed equally to this work.
Data collection The data was collected with an online survey among student of a Finnish university. The respondents were randomly selected from a sample that in turn was randomly selected from student database. The survey was sent to altogether 1,500 respondents. Altogether 289 usable responses were received. This yielded to a response rate of 19. 27 per cent. 68% of the respondents were female. To investigate the possible non-respondent bias, we ran a series of independent samples t-tests. The respondents were first divided into two groups based on the median date of completion of the survey and then compared their responses on subjective vitality, addiction, habit, social enhancement, voyeurism, exhibitionism and enjoyment  (Armstrong & Overton, 1977). The test did not reveal any statistically significant differences between the early and late responses, supporting the notion that non-response bias was not a major concern in our study.
Density Functional Theory calculations Energy minimization of the nanotubes and slabs were performed with Density Functional Theory (DFT) calculations with periodic boundary conditions and the use of symmetry to reduce computational cost. We used these calculations with the CRYSTAL14 21 software package. The exchange-correlation functional used was the solid-adapted generalized gradient approximation (GGA) PBEsol, and the basis set for all atoms (Si, Al, O, H) was double- valence polarized (DZVP). We also performed DFT-based molecular dynamics simulations, aka first principles or ab initio molecular dynamics (MD), as a way to generate reference data for comparison and validation of classical force fields. These simulations were run with CP2K. 22 Temperature was kept constant at 300 K with a CSVR (canonical sampling through velocity rescaling) thermostat,  23  with a time constant of 200 fs.
Reproducibility has been shown to be limited in many scientific fields. This question is a fundamental tenet of scientific activity, but the related issues of reusability of scientific data are poorly documented. Here, we present a case study of our difficulties in reproducing a published bioinformatics method even though code and data were available. First, we tried to re-run the analysis with the code and data provided by the authors. Second, we reimplemented the whole method in a Python package to avoid dependency on a MATLAB license and ease the execution of the code on a high-performance computing cluster. Third, we assessed reusability of our reimplementation and the quality of our documentation, testing how easy it would be to start from our implementation to reproduce the results. In a second section, we propose solutions from this case study and other observations to improve reproducibility and research efficiency at the individual and collective levels.
Background The collective endeavor of science depends on researchers being able to replicate the work of others. In a recent survey of 1,576 researchers, 70% of them admitted having difficulty in reproducing experiments proposed by other scientists  [1]. For 50%, this reproducibility issue even concerns their own experiments. Despite the growing attention on the replication crisis in science  [2, 3] , this controversial subject is far from being new. Even in the 17th century, scientists criticized the air pump invented by physicist Robert Boyle because it was too complicated and expensive to build  [4]. Several concepts for reproducibility in computational science are closely associated  [5, 6]. Here, we define them as mentioned by Whitaker  [6]. First, obtaining the same results using the same data and same code is "reproducibility".
If code is dif- In reality, articles do not take into account adjustment and configuration for significant replication in most cases. Therefore, there is a significant gap between apparent executable work (i. e. , above water portion of iceberg) and necessary effort in practice (i. e. , the full iceberg). ferent, it is "robustness". If we used different data but with the same code, it is "replicability". Last, using different data and different code is referred as "generalizability". Here, we primarily elaborate on reproducibility and robustness and acknowledge that new datasets or hardware environments introduce additional hurdles  [7]. Reproducibility is a key first step. For example, among the 400 algorithms published during the major artificial intelligence conferences, only 6% offered the code  [8].
Even when authors provide data and code, the outcome can vary either marginally or fundamentally  [9]. Tackling irreproducibility in bioinformatics thus requires considerable effort beyond code and data availability, an effort that is still poorly recognized in the current publication-based research community. In most cases, there is a significant gap between apparent executable work (Fig. 1 -i. e. , the above-water portion of iceberg) and necessary effort in practice (Fig. 1 -i. e. , the full iceberg). Such effort is necessary to increase the consistency of the literature and the efficiency of the scientific research process by making research products easily reusable.
We are very grateful to the main authors who kindly provided all the related data and code online and gave us invaluable input upon request. The authors of this study did much more to help reproduce their results than is generally done. Despite their help, we experienced a number of difficulties that we document here, hoping that this report will help future researchers improve the reproducibility of results and reusability of research products. Our first step was to execute the original method code with the given data: reproducibility (Table  1 ). To improve execution speed, the original authors used a library for MATLAB on a Linux platform, using executable compiled code MEX file  [13] , i. e. , MTIMESX  [14] , a library allowing acceleration of large matrix multiplication. MEX files, however, are specific to the architecture and have to be recompiled for each operating system (OS).
Since our lab was using Mac OS X Sierra, the compilation of this MEX file into a mac64 binary required a new version of MTIMESX. It was also necessary to install and configure properly OpenMP  [15] , a development library for parallel computing. After this, the original MATLAB code was successfully run in our environment. These issues are classic but may not be overcome by researchers with little experience in compilation or installation issues. For these reasons alone, many individuals may turn down the opportunity of reusing code and therefore the method. In the next section, we focus on code re-implementation, a procedure that can increase understanding of the method but is even more time consuming.
Metadata and file formats Even if the original code could be run, we had to handle several file formats to check and understand the structure of the original data. For instance, the data were provided by the Cancer Genome Atlas (TCGA)  [22]  and made available in a MATLAB. mat file format v7. 2 as compressed data (sparse matrices). Thanks to SciPy, Python can load the MATLAB files version. We wanted to use the open format HDF5 to save the results; however, Scipy's sparse matrices could not be stored in HDF5 format (Table  1 ). We thus decided to continue saving in. mat format. Moreover, the original authors had denoted download dates of patients' data of TCGA, thereby clarifying the data provenance. However, in the absence of structural metadata, we had to explore the hierarchical structure of the variables (e.
Codes and parameters Beyond documentation and file formats, code initialization and parameter settings are also keys for reproducibility. Upon execution of the code, "unexpected" results were obtained. One cause was application of the hierarchical clustering step for which we used the clustering tools of SciPy. Both SciPy and MATLAB (Math-Works) functions offer seven linkage methods. However, SciPy's default option (single method)  [23]  differs from MATLAB's default option (Unweighted Pair Group Method with Arithmetic Mean: UPGMA)  [24] , which was used in the original study (Table  1 ). Another cause for the variation in results is the value of one of the most important parameters of the method, the graph regulator factor, which was not clarified in the original article. From the article, we believed that this factor had a constant value of 1.
0 until we found that in the original code its value varies across iterations and converges to an optimal value around 1,800. Therefore, we initially obtained very different results from the original NBS (Fig. 2 -a ) with heterogeneous subgroups. Once the optimal value was set up, we finally observed homogenous clusters (Fig. 2 -b ). Moreover, during our attempts to run the original code in order to understand the causes of the errors, we realized that some parts of the code were not run any longer (e. g. , discarded work, remaining traces of debugging), which made understanding the implementation harder. To allow others to reproduce our results, we wrote some documentation and tutorials for the Python package StratiPy  [16]. Readers are invited to experiment with reproducibility by generating the two confusion matrices of Fig.
Documentation and examples During the recoding process, we used an enhanced Python interpreter, IPython, which is an interactive shell supporting both Python 2 and 3, to debug. Since the dataset is large and the execution takes a significant amount of time, we used IPython to re-run interactively some subsections of the script, which is one of the most helpful features. IPython can be integrated in the web interface Jupyter Notebook, offering an advanced structure for mixing code and documentation. While the Jupyter/IPython notebook was initially convenient, it does not scale well to large programs and is not well adapted to versioning. However, the ability to mix code with document text is very useful for tutorials. A user of the code can read documentation (docstring) and text explanations and see how to run the code, explore parameters, and visualize results in the browser.
Our work on NBS, as related here, can be reproduced with a Jupyter/IPython notebook available via our GitHub repository  [16]. One can find more examples and several helpful links via this "gallery of interesting Jupyter Notebooks"  [25] , which contains a section about "Reproducible academic publications. " To conclude, we were able to test the robustness of the method with our Python implementation. However, this took approximately two months for a senior researcher and six months for a master student. Fig. 3  illustrates this work through an analogy between robustness issues and road transport. Driving in a different environment (e. g. , OS), we attempt to obtain identical results (i. e. , to reach the same location) using the same input data (i. e. , gasoline), but with a different computational environment (i.
Collaborative code development and best practices Throughout the project we used the version control system (VCS) Git to document the development of our Python package. Git is arguably one of the most powerful VCSs, allowing easy development of branches and allowing the distributed team (Paris, Berkeley, Montreal) to work collaboratively on the project. This project, StratiPy, is hosted on GitHub, a web-based Git repository hosting service  [16]. While the original code was not available on GitHub, the main authors shared their code on a website. This should be sufficient for reproducibility and replicability for our purposes but makes it less easy to collaborate on the code. While working on our GitHub repository, researchers from the USA, India, China, and Europe contacted us about our robustness experiment.
Reproducibility of robustness: from Python to Python Knowing how difficult it can be to re-run someone else's code, we then attempted to start the analysis from scratch and to reproduce the results on another platform from our newly developed Python package. While the Python code was developed under Mac OS X Sierra (10. 12), we used an Ubuntu 16. 04. 1 (Xenial) computer to test the Python implementation. Some additional issues emerged (Table  1 ). First, our initial documentation did not include the list of the required packages and instructions to launch the code. Second, the code was very slow to the extent that it was impractical to run it on a laptop because the Numpy package had not been compiled with a basic linear algebra subprogram that speeds up low-level routines that perform basic vector and matrix operations.
Publish software and their environment Increased reproducibility and replicability can be obtained by following Buckheit and Donoho's long-standing motto: "When we publish articles containing figures which were generated by computer, we also publish the complete software environment which generates the figures" by offering a complete and free package (WaveLab) to reproduce the published output  [29]. Container and virtual machine technologies such as Docker  [30] , Vagrant  [31] , and Singularity  [32, 33]  (easily works in cluster environments) are becoming a standard solution to mitigate installation issues. These rely, however, on competencies that we think too few biologists possess today. While a container might encapsulate everything needed for a software execution, it can be hard to develop in a container.
Write readable code We draw the following conclusions from our experience in working with others code. First, the structure of the program should be clear and easily accessible. Second, good, concise code documentation and naming convention will help readability. Third, the code should not contain leftovers of previously tested solutions. When a solution takes a long time to compute, an option to store it locally can be proposed. Use of standard coding and documentation conventions (e. g. , PEP 8 and PEP 257 in Python  [36, 37] ) with detailed comments and references of articles makes the code more accessible. When an algorithm is used, any modification from the original reference should be explained and discussed in the article as well as in the code. We advocate for researchers to write code "for their colleagues," hence, ask for the opinion and review of co-working or partner laboratories.
Test the code To check if the code is yielding a correct answer, software developers associate test suites (unit tests or integration tests) with their software. While we developed only a few tests in this project, we realized that this practice has a number of advantages, such as checking if the software installation seems correct and checking if updates in the code or in the operating system impact the results. In our case, we propose to check for the integrity of the data and for the results of some key processing.
Training the new generation of scientists on digital tools and practices The training for coding and software development is still too limited for biologists. Often, it is limited to self-training from searching answers on Stack Overflow or equivalent. Despite efforts by organizations such as Software [39] or Data Carpentry  [40]  and the growing demand for "data scientists" in the life sciences, university training and assessment on coding practices is still not generalized. The difficulty in accessing and understanding code may lead to applying code blindly without checking the validity of the results. Often, scientists prefer to believe that results are correct because checking the validity of the results may require significant time. Mastering a package such that results are truly understood can take a long time, as was the case in our experiment.
Academia could, and we argue should, instruct young scientists in best practices for reproducibility. For instance, Hothorn and Leisch organized a reproducibility workshop, gathering mostly PhD students and young post-docs specialized in bioinformatics and biostatistics. Then, they evaluated 100 random sample papers from Bioinformatics  [3]. Their study revealed how such a workshop can raise young scientists' awareness about "what makes reproduction easy or hard at first hand. " Indeed, they found out that only a third of the original papers and twothirds of applications notes had given access to the source code of the software used.
Standard consensus dataset and testing ecosystem Here, we propose that publications related to bioinformatics methods are systematically accompanied with a test dataset, code source, and some basic tests (given ethical and legal constraints). As the method is tested on new datasets, the number of tests and range of applications would expand. We give a first example with our NBS re-implementation. A schematic overview of a possible testing ecosystem generalizing our test study is shown in Fig. 4. The core of this system would be a set of standard consensus datasets used to validate methods. For instance, in the field of machine learning, standard image databases are widely used for training and testing (e. g. , MNIST for handwritten digits  [41] ). In the case of our proposal, data could be from different categories such as binary, text, image (shown as folders in different colors, Fig.
4 -b ), and subcategories to introduce criteria such as size, quantitative/qualitative, and discrete/continuous, using a tagging system. Datasets could be issued from simulations or from acquisition and would validate a method on a particular component. This testing ecosystem would help scientists who cannot release their data because of privacy issues (Fig. 4  -a. 1), although this can often be overcome, but also give access to data and tests to a wide community including establishments with limited financial resources. We divide those who interact with scientific software or analysis code into two broad categories. First, the authors who propose a method and need to verify its validity and usefulness with open and/or private data. Second, the users (e. g. , developers, engineers, bioinformaticians) who need to test and evaluate the proposed methods with other data.
When authors propose a new method, we propose that authors and users progressively build its reproducibility profile ( Fig. 4 -b. 3, b. 4)  to document what method works best with what data. During the optimization of a project, the software code and associated documentation should be accessible to foster collaboration on additional use cases and data. When the work reaches some level of maturity, a complete article can be posted on a preprint server such as bioRxiv  [42, 43]  and be associated with a GitHub/GitLab repository with a DOI. With considerable effort, Stodden et al. conducted a reproducibility study on 204 random articles that appeared in Science. Despite some availability of the code, it had often been changed after publication, causing difficulties in replication  [44]. In our proposed testing ecosystem, users will be able to launch reproducibility projects more easily thanks to code and article versioning.
Conclusions and Perspective In the 19th century, Pasteur introduced a detailed "Methods" section in his report. This advanced approach was necessary to reproduce his experiments and became the norm in the practice of science  [45]. Today, with the advent of computational science, the reproducibility issue is seen as a growing concern. To sum-  r Develop an open-source continuous testing ecosystem with community standards, well-identified datasets to validate tools across versions and datasets, and go beyond the publication of a PDF file. Verifying a previously published method can be very time consuming and is often poorly acknowledged. Some top-down initiatives already provide some incentives for such a process, e. g. , the Horizon 2020 (H2020)  [47]  project of the European Commission (EC) that mandates open access of research data while respecting security and liability.
H2020 supports OpenAIRE  [48] , a technical infrastructure of the open access, that allows the interconnection among projects, publications, datasets, and author information across Europe. Thanks to common guidelines, OpenAIRE interoperates with other web-based generalist scientific data repositories such as Zenodo, hosted by CERN, which allows the combination of data and GitHub repository via DOIs. The Open Science Framework also hosts data and software for a given project  [49]. Respecting standard guidelines to transparently communicate the scientific work is a key step toward tackling irreproducibility and ensures a robust scientific endeavor.
Funding This work was supported by the following:  Institut Pasteur  (http://dx. doi. org/ 10. 13039/501100003762 ); H2020 Societal Challenges  (http://dx. doi. org/ 10. 13039/100010676 ); Centre National de la Recherche Scientifique  (http://dx. doi. org/ 10. 13039/501100004794 );  Universit é Paris Diderot  (http://dx. doi. org/ 10. 13039/501100005736 );  Conny-Maeva Charitable Foundation ;  Cognacq-Jay Foundation ; Orange (http://dx. doi. org/ 10. 13039/501100003951 ); Fondation pour la Recherche M édicale  (http://dx. doi. org/ 10. 13039/501100002915 ); GenMed Labex; and  BioPsy Labex. J. -B. P. was partially funded by  NIH-NIBIB   P41 EB019936  (ReproNim) NIH-.
Results and Discussion An optical mapping approach allowed improving pseudomolecules. To improve the melon genome assembly v3. 5. 1, an optical mapping approach  [19] [20] [21]  was undertaken in collaboration with the company OpGen. OpGen collected 21 high density MapCards from all Argus-generated MapCards with the NcoI enzyme (see Materials and Methods), to obtain single molecule restriction maps (SMRMs) for the melon genome. We obtained 350,646 DNA molecules with a total size of 146. 46 Gbp. The average size of the molecules was 417. 69 Kbp and the corresponding average size of the NcoI-digested fragments was 11. 95 Kbp. An in silico NcoI-digested profile of the v3. 5. 1 assembly and SMRMs were later used as input data for the Genome-Builder TM assembler (OpGen Inc. ), which generated a new assembly with 27 superscaffolds (hereafter called "optical assembly"; Supplementary Table  S1 ).