dataesr/scientific-paragraphs-categorization huggingface dataset:
It is possible to manually execute only one cell, but the previous/next cells that may depend on it, also have to be manually run (a common source of human error, and frustration for complex operations). Integration of directional graph features (dependencies between the cells) into Jupyter has been discussed, but as of this publication, there is no plan to implement it (see Jupyter's GitHub issue 1175  19  ). The fact that the. ipynb format stores narrative text, code, and multi-media visualization of the outputs in one file, is another major hurdle and against the modularity criteria proposed here. The files can easily become very large (in volume/bytes) and hard to read when the Jupyter web-interface is not accessible. Both are critical for scientific processing, especially the latter: when a web browser with proper JavaScript features is not available (can happen in a few years).
This is further exacerbated by the fact that binary data (for example images) are not directly supported in JSON and have to be converted into a much less memory-efficient textual encoding. Finally, Jupyter has an extremely complex dependency graph: on a clean Debian 10 system, Pip (a Python package manager that is necessary for installing Jupyter) required 19 dependencies to install, and installing Jupyter within Pip needed 41 dependencies. [36]  reported such conflicts when building Jupyter into the Active Papers framework (see Appendix B-K). However, the dependencies above are only on the server-side. Since Jupyter is a web-based system, it requires many dependencies on the viewing/running browser also (for example special JavaScript or HTML5 features, which evolve very fast). As discussed in Appendix A-F having so many dependencies is a major caveat for any system regarding scientific/long-term reproducibility.
Since a domain scientist does not always have the resources/knowledge to modify the conflicting part(s), many are forced to create complex environments with different versions of Python and pass the data between them (for example just to use the work of a previous PhD student in the team). This greatly increases the complexity of the project, even for the principal author. A well-designed reproducible workflow like Maneage that has no dependencies beyond a C compiler in a Unix-like operating system can account for this. However, when the actual workflow system (not the analysis software) is written in a high-level language like the examples above. Another relevant example of the dependency hell is mentioned here: merely installing the Python installer (pip) on a Debian system (with apt install pip2 for Python 2 packages), required 32 other packages as dependencies.
For each solution, we summarize its methodology and discuss how it relates to the criteria proposed in this paper. Freedom of the software/method is a core concept behind scientific reproducibility, as opposed to industrial reproducibility where a black box is acceptable/desirable. Therefore proprietary solutions like Code Ocean 20  or Nextjournal  21  will not be reviewed here. Other studies have also attempted to review existing reproducible solutions, for example, see  [40]. We have tried our best to test and read through the documentation of almost all reviewed solutions to a sufficient level. However, due to time constraints, it is inevitable that we may have missed some aspects the solutions, or incorrectly interpreted their behavior and outputs. In this case, please let us know and we will correct it in the text on the paper's Git repository and publish the updated PDF on zenodo.
A. Suggested rules, checklists, or criteria Before going into the various implementations, it is useful to review some existing suggested rules, checklists, or criteria for computationally reproducible research. Sandve et al. [41]  propose "ten simple rules for reproducible computational research" that can be applied in any project. Generally, these are very similar to the criteria proposed here and follow a similar spirit, but they do not provide any actual research papers following up all those points, nor do they provide a proof of concept. The Popper convention  [33]  also provides a set of principles that are indeed generally useful, among which some are common to the criteria here (for example, automatic validation, and, as in Maneage, the authors suggest providing a template for new users), but the authors do not include completeness as a criterion nor pay attention to longevity: Popper has already changed its core workflow language once and is written in Python with many dependencies that evolve fast, see A-F.
For more on Popper, please see Section B-W. For improved reproducibility Jupyter notebooks,  [42]  propose ten rules and also provide links to example implementations. These can be very useful for users of Jupyter but are not generic for non-Jupyter-based computational projects. Some criteria (which are indeed very good in a more general context) do not directly relate to reproducibility, for example their Rule 1: "Tell a Story for an Audience". Generally, as reviewed in Section IIand Section A-E3 (below), Jupyter itself has many issues regarding reproducibility. To create Docker images, Nüst et al. propose "ten simple rules" in  [43]. They recommend some issues that can indeed help increase the quality of Docker images and their production/usage, such as their rule 7 to "mount datasets [only] at run time" to separate the computational environment from the data.
I. Image Processing On Line journal, IPOL (2010) The IPOL journal 32   [57]  (first published article in July 2010) publishes papers on image processing algorithms as well as the the full code of the proposed algorithm. An IPOL paper is a traditional research paper, but with a focus on implementation. The published narrative description of the algorithm must be detailed to a level that any specialist can implement it in their own programming language (extremely detailed). The author's own implementation of the algorithm is also published with the paper (in C, C++, or MATLAB), the code must be commented well enough and link each part of it with the relevant part of the paper. The authors must also submit several example of datasets that show the applicability of their proposed algorithm. The referee is expected to inspect the code and narrative, confirming that they match with each other, and with the stated conclusions of the published paper.
Furthermore, since all reading and writing is done in the HDF5 file, it can easily bloat the file to very large sizes due to temporary files. These files can later be removed as part of the analysis, but this makes the code more complicated and hard to read/maintain. For example the Active Papers HDF5 file of  [61, in zenodo. 2549987 ] is 1. 8 giga-bytes. In many scenarios, peers just want to inspect the processing by reading the code and checking a very special part of it (one or two lines, just to see the option values to one step for example). They do not necessarily need to run it or obtain the output the datasets (which may be published elsewhere). Hence the extra volume for data and obscure HDF5 format that needs special tools for reading its plain-text internals is an issue.
VCR also has special L A T E X macros for loading the respective VRI into the generated PDF. In effect this is very similar to what have done at the end of the caption of Figure  1 , where you can click on the given Zenodo link and be taken to the raw data that created the plot. However, instead of a long and hard to read hash, we simply point to the plotted file's source as a Zenodo DOI (which has long term funding for logevity). Unfortunately, most parts of the web page are not complete as of January 2021. The VCR web page contains an example PDF  38  that is generated with this system, but the linked VCR repository  39  did not exist (again, as of January 2021). Finally, the date of the files in the MATLAB extension tarball is set to May 2011, hinting that probably VCR has been abandoned soon after the publication of  [63].
P. Sumatra (2012) Sumatra 40   [66]  attempts to capture the environment information of a running project. It is written in Python and is a command-line wrapper over the analysis script. By controlling a project at running-time, Sumatra is able to capture the environment it was run in. The captured environment can be viewed in plain text or a web interface. Sumatra also provides L A T E X/Sphinx features, which will link the paper with the project's Sumatra database. This enables researchers to use a fixed version of a project's figures in the paper, even at later times (while the project is being developed). The actual code that Sumatra wraps around, must itself be under version control, and it does not run if there are noncommitted changes (although it is not clear what happens if a commit is amended).
R. Sciunit (2015) Sciunit 42   [69]  defines "sciunit"s that keep the executed commands for an analysis and all the necessary programs and libraries that are used in those commands. It automatically parses all the executable files in the script and copies them, and their dependency libraries (down to the C library), into the sciunit. Because the sciunit contains all the programs and necessary libraries, it is possible to run it readily on other systems that have a similar CPU architecture. Sciunit was originally written in Python 2 (which reached its endof-life on January 1st, 2020). Therefore Sciunit2 is a new implementation in Python 3. The main issue with Sciunit's approach is that the copied binaries are just black boxes: it is not possible to see how the used binaries from the initial system were built.
S. Umbrella (2015) Umbrella  [70]  is a high-level wrapper script for isolating the environment of the analysis. The user specifies the necessary operating system, and necessary packages for the analysis steps in various JSON files. Umbrella will then study the host operating system and the various necessary inputs (including data and software) through a process similar to Sciunits mentioned above to find the best environment isolator (maybe using Linux containerization, containers, or VMs). We could not find a URL to the source software of Umbrella (no source code repository is mentioned in the papers we reviewed above), but from the descriptions in  [17] , it is written in Python 2. 6 (which is now deprecated).
T. ReproZip (2016) ReproZip  43   [71]  is a Python package that is designed to automatically track all the necessary data files, libraries, and environment variables into a single bundle. The tracking is done at the kernel system-call level, so any file that is accessed during the running of the project is identified. The tracked files can be packaged into a. rpz bundle that can then be unpacked into another system. ReproZip is therefore very good to take a "snapshot" of the running environment into a single file. However, the bundle can become very large when many/large datasets are involved, or if the software environment is complex (many dependencies). Since it copies the binary software libraries, it can only be run on systems with a similar CPU architecture to the original. Furthermore, ReproZip just copies the binary/compiled files used in a project, it has no way to know how the software was built.
Y. Occam (2018) Occam 49  [18]  is a web-based application to preserve software and its execution. To achieve long-term reproducibility, Occam includes its own package manager (instructions to build software and their dependencies) to be in full control of the software build instructions, similar to Maneage. Besides Nix or Guix (which are primarily a package manager that can also do job management), Occam has been the only solution in our survey here that attempts to be complete in this aspect. However it is incomplete from the perspective of requirements: it works within a Docker image (that requires root permissions) and currently only runs on Debian-based, Red Hat based, and Arch-based GNU/Linux operating systems that respectively use the apt, pacman or yum package managers. It is also itself written in Python (version 3.
4 or above). Furthermore, it does not account for the minimal complexity criteria because the instructions to build the software and their versions are not immediately viewable or modifiable by the user. Occam contains its own JSON database that should be parsed with its own custom program. The analysis phase of Occam is also through a drag-and-drop interface (similar to Taverna, Appendix B-C) that is a web-based graphic user interface. All the connections between various phases of the analysis need to be pre-defined in a JSON file and manually linked in the GUI. Hence for complex data analysis operations that involve thousands of steps, it is not scalable. 47 https://wholetale. org 48 https://github. com/whole-tale/wt-design-docs/issues/113 49 https://occam. cs. pitt. edu.
ACKNOWLEDGMENT The authors wish to thank (sorted alphabetically)  Julia Aguilar-Cabello ,  Dylan Aïssi ,  Marjan Akbari ,  Alice Allen ,  Pedram Ashofteh Ardakani ,  Roland Bacon ,  Michael R. Crusoe ,  Antonio Díaz Díaz ,  Surena Fatemi ,  Fabrizio Gagliardi ,  Konrad Hinsen ,  Marios Karouzos ,  Johan Knapen ,  Tamara Kovazh ,  Terry Mahoney ,  Ryan O'Connor ,  Mervyn O'Luing ,  Simon Portegies Zwart ,  Idafen Santana-Pérez ,  Elham Saremi ,  Yahya Sefidbakht ,  Zahra Sharbaf ,  Nadia Tonello ,  Ignacio Trujillo  and the AMIGA team at the  Instituto de Astrofísica de Andalucía  for their useful help, suggestions, and feedback on Maneage and this paper. The five referees and editors of CiSE ( Lorena Barba  and George Thiruvathukal) provided many points that greatly helped to clarify this paper.
This project (commit eeff5de) is maintained in Maneage (Managing data lineage). The latest merged Maneage branch commit was a1a966a ( 4 Jan 2021 ). This project was built on an x86 64 machine with Little Endian byte-order and address sizes 39 bits physical, 48 bits virtual. Work on Maneage, and this paper, has been partially funded/supported by the following institutions: The Japanese MEXT PhD scholarship to M. A and its  Grant-in-Aid for Scientific Research  ( 21244012 ,  24253003 ). The  European Research Council (ERC)  advanced grant 339659-MUSICOS. The  European Union (EU) Horizon 2020  ( H2020 ) research and innovation programmes No  777388  under  RDA EU 4. 0  project, and  Marie Skłodowska-Curie  grant agreement No  721463  to the SUNDIAL ITN. The  State Research Agency (AEI)  of the  Spanish Ministry of Science, Innovation and Universities (MCIU)  and the  European Regional Development Fund (ERDF)  under the grant  AYA2016-76219-P.
tex (in the source directory; written manually). Other software is used in other steps. The solid arrows and full-opacity built boxes correspond to the lineage of this paper. The dotted arrows and built boxes show the scalability of Maneage (ease of adding hypothetical steps to the project as it evolves). The underlying data of the left plot is available at zenodo. 4291207/tools-per-year. txt. INPUTS. conf demo-year. conf param. conf top-make. mk initialize. mk download. mk format. mk demo-plot. mk next-step. mk out-a. dat Frac. papers with tools (green) 0 % 20 % 40 % 60 % 80 % 100 % 1990 2000 2010 10 1 10 2 10 3 10 4 10 5 Num. papers (red, log-scale) initialize. tex Basic project info Also defines project structure (for *. mk files). (e. g. , Git commit). download. tex menke20. xlsx table-3.
Core-periphery networks are typical economic and social networks which exhibit a core-periphery structure, a well-known concept in the analysis of social networks in sociology and economics  [BE00, DJ10] , which defines a bipartition of the agents into core and periphery, such that certain key features are identified. We consider an axiomatic framework that has been introduced in previous work in computer science [ALP + 14], which is based on two parameters only, dominance and robustness. The ranges for these parameters in the theorems we obtain include the values used in the the experimental part of this work, in which our results are validated on important datasets of real-world networks. Intuitively, the core is a set of agents that dominates the rest of the network. In order to do so, it maintains a large amount of external influence on the periphery, higher than or at least comparable to the internal influence that the periphery has on itself.
Experimental Results In the previous Section 3 we formally studied the 2-Choices dynamics on Core-Periphery networks, observing a phase transition phenomenon that appears on a dominance threshold c = √ 2-1 2. Here, we report the results of the empirical data obtained by simulating the 2-Choices dynamics on real-world networks. Furthermore, we discuss our results and compare them with our theoretical analysis. The source code of the experiments is freely available. 5 e simulated the 2-Choices dynamics on 70 real-world networks, out of which 25 taken from KONECT  [Kun13]  and 45 from SNAP  [LK14]. Detailed information regarding the networks and the results of the experiments are reported in Table  1. The networks chosen for the experiments are drawn from a variety of domains including social networks, communication networks, road networks, and web graphs; moreover, they range in size from thousands of nodes and thousands of edges up to roughly one million of nodes and tens of millions of edges.
We apply the densest-core method to the networks and, as expected, we obtain higher robustness and lower dominance values compared to the k-rich-club method. The data reported in Table  1  shows Algorithm 1 Densest-Core Extraction. return C that the robustness of the core extracted by our method is higher in all the considered datasets but one. Indeed, we finally obtain dominance values below the theoretical threshold c. We proceed as follows: We initialize all the agents in C with blue and all the agents in P with red. Then, we simulate the 2-Choices dynamics on each network, keeping track of the volumes of blue and red agents in each iteration. We declare an almost-consensus on the majority's color if within |V | iterations either the red or the blue agents reach a volume greater than 95% of the network's volume.
Otherwise we consider the simulation metastable -waiting for a superpolynomial number of rounds would be infeasible. The experiments were repeated 50 times for each network. Table 1: Experimental data. Source reports the source of the dataset, i. e. SNAP (S) or KONECT (K). The values c r and c d are the robustness and dominance obtained using the densest-core method; the values cr and cd are the robustness and dominance obtained using the k-rich-club method. C and P are the fraction of experiments in which the core's and the periphery's color respectively spread to reach an almost-consensus, while M is the fraction of experiments in which there is metastability, all having the core extracted with the densest-core method. K stands for thousand, M for million. As can be observed in Figure  1 , there exists an empirical threshold σ = 1 2 which is different from the theoretical one.
In fact, in 81% of the datasets with a dominance above the threshold the 2-Choices dynamics converges to an almost-consensus while in 86% of the datasets with a dominance below the threshold, the 2-Choices dynamics ends up in a metastable phase. The empirical threshold is greater than the theoretical threshold because of several factors: (i) in the experiments the core actually changes color to a small extent (while in the theoretical part we ignored such small perturbations), and it consequently lowers the probability for an agent in the periphery to pick the core's color; (ii) the real-world network we used in the experiments do not have the regularity assumptions of the networks that we consider in the analysis; (iii) in the experiments we declare metastability only after |V | iterations and this increases the likelihood of false positive metastable runs.
In addition, harmonised test guidelines and templates for toxicity, physico-chemical and fate and behaviour properties have been developed at OECD level, and form the basis for consistent exposure and hazard assessment. With the International Uniform Chemical Information Database (IUCLID), an internationally agreed data exchange format has been developed; however, mostly focussing on hazard data. Achieving the objectives of the EU Green Deal and related policy strategies requiring the EU to become climate neutral, waste-free and non-toxic by 2050, relies on the excellence of exposure science applied within current and future regulatory domains  [6, 9, 18, 19]. At the time of writing of this paper, also other geographical areas around the globe adopted or are in the phase of adopting sustainability targets (e.
g. ,  [29] ). However, despite several efforts to improve processes and information quality  [30, 31] , a common EU scientific framework on exposure assessment is still lacking  [32, 33]. The European Commission has recently completed a series of policy evaluations, including the REACH review  [9] , the fitness checks of chemical legislation (excluding REACH)  [34] , the Water legislation  [35]  and the General Food Law 178/2002. Following the revision of the General Food Law, the Transparency Regulation 2019/1381  [36]  resulted in increasing the transparency and sustainability of the EU risk assessment in the food chain. These evaluations have revealed some of the gaps and inefficiencies with respect to the generation and use of exposure information. These include the need to enable the use of exposure datasets and exposure modelling tools across policy areas, to consider open data policies and simplification of their use for private actors with limited resources (e.
g. , missing reporting of limit of detection/ quantification). In addition, there is a lack of analytical standards to quantify detected exposure, and there is a lack of data and/or methods for traceability  [3]. In other cases, data and assessments are accepted based on a more flexible case-by-case evaluation. For human occupational exposure modelling, for example, various competing tools exist. Depending on the tool, exposure predictions for the same use-case scenario can differ significantly and are frequently not in line with measured datasets  [67] [68] [69]. Exposure assessments are hence frequently inconsistent across sectors and sometimes even among registrants of the same chemical substance  [70]. A specific case concerns physiologically based kinetic models, which are playing an increasing role in the estimation of internal exposure, among others in conjunction with in vitro-to-in vivo extrapolation in emerging alternative methods to animal testing (e.
This includes, for example, a harmonised structured format for the transfer of safety data through the supply chain. • REACH is based on the principle of 'One Substance -One Registration', aiming to make all companies placing the same substance on the market to come up with one common dataset in harmonised electronic format (IUCLID), characterising the properties and hazards of the substance. However, there is no requirement yet to conduct and maintain one common safety assessment per substance, and hence suppliers provide.
Box 1. Interventions on phthalates under REACH and the Food Contact Material Regulation Low molecular weight phthalates were assessed for their risk to human health under REACH by ECHA in 2017, and subsequently by EFSA in 2019, specifically for concerns related to their use in food contact materials  [115]. The two assessments were carried out largely independently, without alignment with respect to the definition of chemical grouping, in the exposure scenarios considered and, consequently, in the methods and exposure data used (e. g. , population age grouping, food intake estimates). By using HBM data for the exposure assessment, ECHA pointed to evidence of risk to human health from combined exposure to four phthalates and of a large contribution of exposure to di(2-ethylhexyl) phthalate from the diet based on modelling estimates  [116].