dataesr/scientific-paragraphs-categorization huggingface dataset:
(the exact pattern is documented as part of the dataset replication package). The dataset consists of 6. 5 million unique license files that can be used to conduct empirical studies on open-source licensing, training of automated license classifiers, natural language processing (NLP) analyses of legal texts, as well as historical and phylogenetic studies on FOSS licensing. Additional metadata about shipped license files are also provided, making the dataset ready to use in various empirical software engineering contexts. Metadata include file length measures, detected MIME type, 8 See https://www. softwareheritage. org/mirrors/ for details, including storage requirements. At the time of writing, a full mirror of the archive requires about 1 PiB of raw storage. Note that individual files are gzip-compressed to further reduce storage size.
The general empirical analysis workflow involves three simple steps: identify the source code files of interest using the metadata available in the graph dataset, obtain their checksum identifiers, and then retrieve them in batch and in parallel from public cloud providers. This process scales well up to many million files to be analyzed. For even larger-scale experiments, e. g. , analyzing all source code files archived at Software Heritage, research institutions may consider setting up a local mirror of the archive. detected SPDX  [45]  license (using ScanCode  [35] , a state-of-the-art tool for license detection), example origin (e. g. , GitHub repository), and oldest public commit in which the license appeared. The dataset is released as open data as an archive file containing all deduplicated license files, plus several portable CSV files for metadata, referencing files via cryptographic checksums.
Enabling Artifact Access and (Large-Scale) Analysis Applied research in various fields has been conducted to ease access to such a huge amount of data as the Software Heritage archive for empirical researchers. This kind of research is not, strictly speaking, research enabled by the availability of the archive to solve software engineering problems but rather research motivated by the practical need of empowering fellow scholars to do so empirically. As a first example, SwhFS (Software Heritage File System)  [2]  is a virtual filesystem developed using the Linux FUSE (Filesystem in User SpacE) framework that can "mount," in the UNIX tradition, selected parts of the archive as if they were available locally as part of your filesystem. For example, starting from a known SWHID, one can, for instance:  In the second example, we are grepping through the code of Apollo 11 guidance computer code, searching for reference to antennas.
The price to pay for this is that (1) the compressed graph representation loaded in memory is derived from the main archive and not incremental (it should periodically be recreated) and (2) only the graph structure and selected metadata fit in RAM; others reside on disk (although using compressed representations as well  [37] ) and need to be memory mapped for efficient access to frequently accessed information. Finally, the archive also provides interesting use cases for database research. Recently, Wellenzohn et al. [48]  has used it to develop a novel type of content-andstructure (CAS) index, capable of indexing over time the evolution of properties associated to specific graph nodes, e. g. , a file content residing at a given place in a repository changing over time together with its metadata (last modified timestamp, author, etc.
Software Forks The same characteristics that enable studying the evolution and provenance of public code artifacts can be leveraged to study the global ecosystem of software forks. In particular, the fact that commits are fully deduplicated allows to detect forks -both collaborative ones, such as those created on social coding platforms to submit pull requests, and hostile ones used to bring the project in a different direction-even when they are not created on the same platform. It is possible to detect the fork of a project originally created on GitHub and living on GitLab. com, or vice versa, based on the fact that the respective repositories share a common commit history. This is important as a methodological point for empirical researchers, because by relying only on platform metadata (e.
g. , the fact that a repository has been created by clicking on a "fork" button on the GitHub user interface), researchers risk overlooking other relevant forks. In previous work, Zacchiroli  [51]  provided a classification of the type of forks based on whether they are explicitly tracked as being forks of one another on a coding platform (Type 1 forks), they share at least one commit (Type 2), or they share a common root directory at some point in their histories (Type 3). He empirically verified that between. 3. 8% and. 16%, forks could be overlooked by considering only type 1 forks, possibly inducing a significant threat to validity for empirical analyses of forks that strive to be comprehensive. Along the same lines, Bhattacharjee et al. [6]  (participants in the MSR 2020 mining challenge) focus their analyses on "cross-platform" forks between GitHub and GitLab.
Diversity, Equity, and Inclusion Diversity, equity, and inclusion studies (DE&I) are hot research topics in the area of human aspects of software engineering. Free/open-source software artifacts, as archived by Software Heritage, provides a wealth of data for analyzing evolutionary DE&I trends, in particular in the very long term and at the largest scale attempted thus far. A recent study by Zacchiroli  [50]  has used Software Heritage to explore the trend of gender diversity over a time period of 50 years. He conducted a longitudinal study of the population of contributors to publicly available software source code, analyzing 1. 6 billion commits corresponding to the development history of 120 million projects, contributed by 33 million distinct authors over a period of 50 years. At this scale, authors cannot be interviewed to ask their gender, nor cross-checking with large-enough complementary dataset was possible.
Building the Software Pillar of Open Science Software plays a key role in scientific research, and it can be a tool, a result, and a research object. [. ] France will support the development and preservation of source codeinseparable from the support of humanity's technical and scientific knowledge -and it will, from this position, continue its support for the Software Heritage universal archive. So as to create an ecosystem that connects code, data and publications, the collaboration between the national open archive HAL, the national research data platform Recherche Data Gouv, the scientific publishing sector and Software Heritage will be strengthened. Second french national plan for open science, July 2021  [22]  Software is an essential research output, and its source code implements and describes data generation and collection, data visualization, data analysis, data transformation, and data processing with a level of precision that is not met by scholarly articles alone.
These are not only different concerns but also separate ones. Establishing proper credit for contributors via citations or providing proper metadata to describe the artifacts requires a curation process  [3, 8, 14]  and is way more complex than simply providing stable, intrinsic identifiers to reference a precise version of a software source code for reproducibility purposes  [4, 15, 26]. Also, as remarked in  [4, 25] , research software is often a thin layer on top of a large number of software dependencies that are developed and maintained outside of academia, so the usual approach based on institutional archives is not sufficient to cover all the software that is relevant for reproducibility of research. In this section, we focus on the first two concerns, archival and reference, that can be addressed fully by leveraging the Software Heritage archive, but we also describe how Software Heritage contributes through its ecosystem to the two other concerns.
com/, http:// www. replicabilitystamp. org/, and the Code and data links recently introduced in ArXiv. org. deposits in academic repositories take snapshots of a given state of the source code, usually in the form of a. zip or. tar file, and store it in the repository exactly like an article or a dataset, with an associated publisher identifier; a typical example in computer science is the ACM Digital Library, but there are a number of general academic repositories where software artefacts have been deposited, like FigShare and Zenodo. The approaches in the first category rely on code hosting platforms that do not guarantee persistence of the software artifact: the author of a project may alter, rename, or remove it, and we have seen that code hosting platforms can be discontinued or decide to remove large amount of projects.
Growing Technical and Policy Support In order to take advantage of the services provide by Software Heritage in this setting, a broad spectrum of actions have been started and are ongoing. We briefly survey here the ones that are most relevant at the time of writing. At the national level, France has developed a multi-annual plan on Open Science that includes research software  [21, 22]  and consistently implemented this plan through a series of steps that range from technical development to policy measures. On the technical side, the French national open-access repository HAL [14] (analogous to the popular arXiv service  11  ) has been integrated with the Software Heritage archive. The integration allows researchers to have their software projects archived and referenced in Software Heritage, while curated rich metadata and citation information are made available on HAL  [14] , with a streamlined process depicted in Fig.
2. 7. On the policy side, the second French national plan for open science  [22] , published in July 2021, prescribes the use of Software Heritage and HAL for all the research software produced in France, and Software Heritage is now listed in the official national roadmap of research infrastructures published in February 2022  [23]. This approach is now being pushed forward at the European level, through funding for consortia that will build the needed connectors between Software Heritage and several infrastructures and technologies used in academia, using the French experience as a reference. Most notably, the FAIRCORE4EOSC  [19]  European project includes plans to build connectors with scholarly repository systems like Dataverse  [47]  and InvenioRDM  [28]  (the white-label variant of Zenodo), publishers like Dagstuhl  [43]  and Episcience  [18] , and aggregators like swMath  [20]  and OpenAire  [36].
Integration with Web hooks is available for a variety of code hosting platforms, including Bitbucket, GitHub, GitLab. com, and SourceForge, as well as for instances of GitLab and Gitea, which enable owners of projects hosted on those platforms to trigger archival automatically on any new release, reducing the burden on researchers even more. Software Heritage will try to detect and parse intrinsic metadata present in software projects independently of the format chosen, but we see the value of standardizing on a common format. This is why, with all academic platforms, we are working with, we are advocating the use of codemeta. json, a machine readable file based on the CodeMeta extension of schema. org, to retrieve automatically metadata associated to software artifact, in order to avoid the need for researchers to fill forms when declaring software artifacts in academic catalogs, following the schema put in place with the HAL national open-access portal.
Conclusions and Perspectives In conclusion, the Software Heritage ecosystem is a useful resource for both software engineering studies and for Open Science. As an infrastructure for research on software engineering, the archive provides numerous benefits. The SWHID intrinsic identifiers make it easier for researchers to identify and track software artifacts across different repositories and systems. The uniform data structure used by the archive abstracts away all the details of software forges and package managers, providing a standardized representation of software code that is easy to use and analyze. The availability of the open datasets makes it possible to tailor experiments to one's needs and improves their reproducibility. An obvious direction at the time of writing is to leverage Software Heritage's extensive source code corpus for pre-training large language models.
• reduces the opportunity cost of conducting large-scale experiments by offering at regular intervals as open datasets full dumps of the archive content • contributes to mitigate selection bias and the associated external validity threats by providing a corpus that strives to be comprehensive for researchers conducting empirical software engineering experiments targeting large project populations. • the persistence offered by an independent digital archive, run by a nonprofit open organization, eases the process of ensuring the reproducibility of large-scale experiments, avoiding the need to re-archive the same open-source code artifacts in multiple papers, a wasteful practice that should be avoided if possible. Using Software Heritage is enough to thoroughly document in replication packages the SWHIDs (see Sect.
Rule 5: Determine the method of data access When data cannot be directly available, the organization that has generated it may have assigned a data access committee or similar body to review the appropriateness of requests for reuse (see  Rule 7). After checking the relevance of the request, the method of data access is usually chosen by the institution on the basis of the security and cost-effectiveness of implementation. For example, to best secure data-sharing, remote access to an infrastructure that incorporates a login option should be used. Remote access to the data also enables the user's rights to be restricted (e. g. , copy and paste data). To achieve this high level of security, it is often necessary to use an external service provider that specializes in providing remote servers.
In this spirit, many repositories have been implemented in the USA, such as Vivli and the Yale Open Data Access (YODA) program  [19]  that already have their own requirements to control and secure datasharing. Most data from trials that is shareable to date is stored by the industry in these repositories and the rules are already fixed. Researchers can ask their institutions which repository is recommended or use the TRUST principles to ensure that they are reliable  [20]. Sometimes it is possible to download the data directly locally onto the re-user's server; this method has the disadvantage of lowering the level of traceability of the actions carried out on the dataset after sharing. Some tools such as Datashield  [21]  propose the running of analyses remotely without physically accessing the dataset and only seeing the results (this is called querying the data).
Rule 8: Deploy optimal data management to ensure that the data shared is useful Preparing data for sharing is an essential step that should not be neglected. It requires technical skills and knowledge of regulations to ensure that the data corresponds to the requests of the re-users while respecting regulations. Data management should always aim for compliance with the FAIR principles  [26] , which ensure that data is Findable  [27] , Accessible, Interoperable, and Reusable. Similarly, for clinical trial data that includes indigenous participants, the CARE principles  [28]  should be consulted. The database should use a universal computerized format to enable new research teams to use it easily. Data and metadata should be standardized and preferably in English language. In other words, clinical trial data cannot be kept in an ivory tower nor in the Tower of Babel  [29].
Metadata should always be published to enable reusers to easily identify studies of interest. The statistical codes should be annotated to make them understandable by any another statistician  [23]. To help the citation of reused datasets, we advise data generators to obtain a DOI name (digital object identifier) for each dataset. Often the entire dataset does not need to be shared. In this case, researchers should create and share a dataset containing only the data needed for reuse. They should also ask re-users to comply with the data citation principles  [30]  and require them to cite the relevant DOIs in all publications of that reused data.
Rule 9: Minimize risks Data-sharing is a risky endeavor from a data protection perspective, because of the risk of reidentification. However, safeguards exist to mitigate this. If the data is publicly available, it should be anonymized. However, anonymizing data does not only raise the issue of potential loss of information from the dataset. Researchers should also expect to undertake a large-scale task, since the rules ensuring the anonymization of data are numerous and not necessarily concordant between countries. For example, one will not obtain the same set of data by following the Health Insurance Portability and Accountability Act (HIPAA)  [31]  in the US or the data protection working group recommendations  [32]  in Europe. When anonymization is not possible, researchers can use pseudonymized data, i.
e. , datasets with no directly identifying data. As re-identification of individuals is possible from these datasets, it is necessary to ensure that the recipients are specified by the protocol and that they are not able to perform analyses that were not agreed on beforehand. Current measures to secure the process often involve (i) establishing a data-sharing agreement that defines the data use rights and obligations; (ii) minimizing the data transmitted, i. e. , giving access only to the variables that are strictly necessary for the data reuse; and (iii) choosing the method of data access wisely. However, these security measures sometimes complicate sharing and can therefore appear restrictive but are a guarantee of confidence.
Acknowledgments We thank  Audrey Acloque ,  Pierre-Henri Bertoye ,  Clemence Bolla ,  Charlotte Douard ,  Jean-Christophe Hebert ,  Marie Lang ,  Frederique Lesaulnier ,  Bernard Nordlinger , and  Tabassome Simon , who are part of the clinical trial data-sharing statement working group [37], for bringing together knowledge about data-sharing in a complementary way. We thank  Angela Swaine Verdier  for the linguistic proofreading.
, 2019)  while being a hard problem  (Golubchik et al. , 2009)  Self-managed data must then be able to consider both fields in order to provide good services. In this paper, we present SKYDATA, the first proposal to design an environment for self-managed data. We present the state of the art and the context in which this paradigm came to existence (Section 2), then we define the SKYDATA Environment structure and environment in (Section 3). We then discuss of capabilities and goals of SKD (Section 4). We then conclude and discuss the SKYDATA paradigm (Section 5).
RELATED WORK We can define four major eras for the data management in IT (Information Technology). The early era was the digitalization of information  (Sendov, 1997). The second era was the building of Database Management Systems (DBMS). While many of them were based on SQL, nowadays, DBMS continue to grow with the amount of data, and solutions based on NoSQL approaches have appeared  (Leavitt, 2010; Moniruzzaman and Hossain, 2013). Next, the emergence of distributed management systems  (Moysiadis et al. , 2018)  has arisen and evolved through Grid, P2P and Cloud architectures: the third era around distributed data management was born. Technical comparison of NoSQL data management systems can be found in  (Bunch et al. , 2010; Han et al. , 2011). Those studies compare the different approach chosen for data management systems, as well as a study on their performances.
Short Definitions • SKD: SkyData, noted SKD, are autonomous agents that carry the data itself and metadata. Among the metadata, some code allows the SKD to plan for its actions. SKDs are endowed with goals and capabilities making it autonomous, proactive and reactive within the system. • Family (of SKDs): As in many distributed system, stored data is replicated. When a SKD is replicated, a new SKD containing a replica is created. The set of SKDs that hold the same piece of data is called a family. • SKW: SkyWorkers, noted SKW, are another entity of a SKYDATA Environment. They can be viewed as special SKDs that are entrusted with a system task rather than application data. They are capable of moving within the system and communicating with SKDs to accomplish their task. • Harbour: Harbours are the physical host that hold SKDs and SKWs.
Indexable Our fourth and last contribution is to provide an open-source implementation of our algorithm in Python with Numba, and to present an empirical evaluation of the performance of our implementation. Our results show that our algorithm is very efficient in computing Whittle index and testing indexability. Moreover, our simulations indicate that the subcubic version of our algorithm not only has an asymptotically small complexity but is also faster in practice than our original (2/3)n 3 algorithm. Testing the indexability and computing indices takes less than one second for n = 1000 states and less than 10 minutes for n = 15000 states. This is 15 to 20 times faster than the original computation times reported in  [29]  (for a Matlab implementation), and about 5 times faster than an optimized implementation of  [29]  (for a Julia implementation).
In order to compute Whittle index and test indexability, our algorithm needs that all policies π k constructed by the algorithm to be unichain. It does not require the arm to be unichain. This leads to Algorithm 1, that we write in pseudo-code. This algorithm relies on two subroutines: on Line 7, to compute the next index and on Line 8 to test if a policy is Bellman optimal. We will describe later in the paper how to implement these functions in an efficient manner. Note that in all the paper, we use the superscript k (e. g. , π k , µ k , σ k ) to refer to the quantities computed at iteration k. We use the subscripts i or j (e. g. , π i , λ i , µ i , π j ) to refer to the quantities related to states i or j. Note that when µ k min = +∞, the quantity α π k i (µ k min ) defined in Line 7 of Algorithm 1 should be understood as lim λ→∞ α π k i (λ) ∈ R∪{-∞, +∞}.
The subcubic algorithm 5. 1 Main idea: recomputing X k from A π k periodically The main computational burden of Algorithm 2 is concentrated on two lines: on Line 5 where we compute X 1 by solving a linear system, and on Line 10 where we compute the column vector X k :σ k-1 from X 1 :σ k-1. The remainder of the code runs in O(n 2 ) operations and is therefore negligible for large matrices. In fact, the computation of X 1 is done by solving a linear system, which can be computed by using a subcubic algorithm (like Strassen  [39] ). In this section, we show how to optimize our algorithm by reducing the complexity of the update X() function, at the price of recomputing the full matrix X k0 from A π k 0 periodically. At iteration k of Algorithm 2, the quantities X k iσ k-1 are used at Line 11 to obtain the values y k i.
To verify this, we launched a detailed profiling of the code of Algorithm 2 with the non-optimized Subroutine 3. It shows that for a problem of dimensions 5000, the update of Line 10 takes more than 90% of the computation time, the initialization of X 1 on Line 5 takes about 5% of the time and the rest of the code takes less than 1% of the running time. Now, if inverting the full matrix takes about 5% of the execution time, and updating the gray zone takes 95%, then by doing 5 updates, one can hope to obtain an algorithm whose running time is roughly 5 × 5 + 95/5 ≈ 43% the one of the original implementation. As we observe in Section 6, this is close to the gain that we obtain in practice. A general way to choose the best number of updates is used in the numerical section. It is based on the following reasoning.