dataesr/scientific-paragraphs-categorization huggingface dataset:
Gabon We collected 1,056 mosquitoes by human landing catch from different locations in Lopé village in Gabon. All individuals were morphologically identified as members of the An. gambiae complex. 11 mosquitoes were identified as An. fontenillei by a species diagnostic PCR  (Fanello, Santolamazza, and della Torre 2002)  and those clustered with the An. fontenillei and An. bwambae samples in the VAE projection as expected. Those samples were used to construct the convex hulls. For 1002 (95. 9%) of the unidentified samples we obtain at least 10 targets and these are classified by the NN step of NNoVAE. The vast majority (993 samples; 99. 1%) is assigned to the An. gambiae complex. The other species we find in this dataset are An. funestus (2 individuals) and An. coustani complex (2 individuals), plus 5 individuals that could not be assigned at any level.
Conclusion In this paper we presented NNoVAE, a method for robust species identification for the entire Anopheles genus from multilocus targeted amplicon sequencing data. This integrated approach removes the need for sorting the specimens into species groups or complexes based on morphology, which is labour intensive and error prone, particularly in the case of damaged mosquitoes, for instance when collected with CDC light traps. The NN step can distinguish between most species in our reference database. But equally importantly, it gives an indication of the uncertainty of the assignment, using the same thresholds for the entire genus, enabling us to quantify confidence in assignment in a meaningful way allowing for comparison between all species groups. For individuals from species not yet represented in the reference dataset, we can often assign to a species-group at the intermediate level (corresponding to taxonomic species groups or complexes) or at the coarse level (corresponding to taxonomic series or subgenera).
A few samples did not meet the threshold to be assigned at the coarse level, but the assignment results do give an indication of the position of these samples in the phylogeny. Initial explorations of the mitochondrial sequences for those samples do not indicate a close match to publicly available mitochondrial data of any Anopheles species. We hope to resolve the species identity of these samples by extending our reference database and collaborating with morphological experts, but until then we retain the groups of unresolved species to compare future samples against them. The NN step alone struggles to differentiate closely related species within species complexes. For the An. gambiae complex we developed a high resolution species identification method based on a variational autoencoder.
arabiensis in Gabon, but entomologists can still decide to exclude certain species labels if they are sure that they are not appropriate for their collection location and time, for instance the An. tengrela label in Gabon. The VAE projection of the An. gambiae complex also shows some population structure within species clusters. Some of the structure reflects the geography of the collection locations, e. g. Madagascar stands out as a separate subcluster both for An. gambiae and An. arabiensis. However, the sample sets from Burkina Faso and Gabon show that samples from the same location can be projected to different positions in latent space and it would be interesting to investigate what is driving this observed diversity. NNoVAE relies on a reference database and therefore the accuracy of the species assignments also depends on the quality and completeness of the reference database.
We expect that version 2 of the reference database will contain approximately fifty additional species, as well as more individuals of species that are underrepresented in NNv1 and we are seeking further well-characterised samples. NNoVAE can characterise vector populations in a uniform way across the Anopheles genus and as such contributes to our understanding of Anopheles species composition, population structure, species ranges, and transmission potential. The combination of ANOSPP and NNoVAE offers a cheaper, more robust, more informative, and more reliable way to carry out malaria vector surveillance that we hope will be embraced over the coming years by the medical entomology community and National Malaria Control Programs.
The bars at the bottom show the overall assignment proportions for the displayed species-groups, only assignment proportions of at least 1% are shown. From top to bottom the assignment proportions are for the fine, intermediate and coarse level. For the three-letter code abbreviations of species-groups, see Supplementary Table1. The numeric abbreviations stand for 1: An. marshallii complex sp1, 2: Myzomyia sp1, 3: An. gabonensis, 4: An. culicifacies, 5: An. maculatus B; none of these species-groups represent more than 1% of the assignment. Right panel: The heatmap showing the nearest neighbours of sample Agam-35, an An. gambiae specimen. Not all samples in the reference database are displayed, only those in the Pyretophorus series (except Agam-35) as well as five samples from the Neocellia series.
Acknowledgments We would like to thank  Vickie Brooks  for general logistic support for ANOSPP. We thank  Petra Korlević  for support in wet lab work, valuable discussions and conducting the mitochondrial analysis. For field sampling in Burkina Faso, we would like to thank  Franck Adama Yao ,  Patric Stéphane Epopa Ngomé ,  Nouhoun Traoré , and the  Target Malaria Team. We thank  Samuel Anifowose ,  Akinkunle Adeniyi ,  Joshua Oduwu ,  Idris Otun ,  Juwon Adeniji ,  Wale Enisemo ,  Cynthia Umunnakwe  and  Adedotun Bayegun  for sample collection and processing in Nigeria. In Gabon, we thank the  ESV team at the CIRMF. We also thank the  Institut Pasteur de Madagascar  for sample collection in this country. We also thank  Sanger's Scientific Operation Teams  for carrying out all PCRs, library generation, and sequencing on data presented here and  Catherine McCarthy  for her support in ensuring all samples are compliant with Access and  Benefit Sharing  of sequence data as laid out by the  Nagoya Protocol.
A secondary objective is to estimate these probabilities of being latent status positive, using dierent denitions for the latent status, from surveillance data collected as part of a CP against the infection by the BVDV in Loire-Atlantique, France. The challenges of dening prior distributions and the implications of using dierent prior distributions are discussed. R functions to perform the analyses described in this paper are gathered in an R package which is available from GitHub (https://github. com/AurMad/ STOCfree).
At each iteration of the MCMC algorithm, given the data and priors, a herd status (0 or 1) and the coecients for the associations between risk factors, latent status and test results are drawn from their posterior distribution. In the next 3 sections, the parameters for which prior distributions are required, i. e. test characteristics, status dynamics and risk factor parameters, are described. The outputs of Bayesian models are posterior distributions for all model parameters. Specically, in our model, the quantities of interest are the herd level probabilities of being latent status positive on the last test month in the dataset as well as test sensitivity, test specicity, infection dynamic parameters and parameters for the strengths of association between risk factors and the probability of new infection.
Selection of risk factors A diculty in the evaluation of putative risk factors was that Bayesian models usually take time to run, especially with large datasets as used here. It was therefore not possible to perform this selection with our Bayesian model. To circumvent this problem, logistic models as implemented in the R glm function (R Core Team, 2019) were used 4. The outcome of these models was seroconversion dened as a binary event, and covariates of interest were risk factors for becoming status positive as dened through the τ 1 variable. All herds with 2 consecutive test results whose rst result was negative (ODR below the chosen threshold) were capable of seroconverting. Of these herds, the ones that had a positive result (ODR above the chosen threshold) on the second test were considered as having seroconverted.
Bayesian models Four dierent Bayesian models were considered. For all models, historical data were used for parameter estimation and the probability of infection on the last month in the dataset was predicted. Model 1 -Perfect routine test: in order to evaluate the monthly dy- Model 4 -Imperfect routine test, conrmatory testing and risk factors: the objective of this model was to assess the impact of conrmatory testing. The same prior distributions as in scenario 3 were used. In this case however, every time a positive test result was recorded, a new conrmatory test was randomly generated in the following month so that 85% of these tests were positive and 15% were negative. The conrmatory test was assumed to have both a sensitivity and a specicity close to 1. For each model, 4 chains were run in parallel.
In our dataset, herds purchasing cattle were more likely to have seroconverted 8 months after the introduction. In the Bayesian model, cattle introduction was modelled as aecting the probability of becoming status positive 8 months after the introduction. It can be argued that infection is present but not detected during this period, as the expression delayed detection suggests, and that the probability of infection should increase as soon as risk factor occurrence is recorded. Modelling this phenomenon would be possible by decreasing the test sensitivity for a period corresponding to the lag used in the current version of the model. This would imply that for this duration, any negative BTM test result would not provide any information about the true status regarding infection and that the herd would have an increased predicted probability of infection.
Background & Summary Open Source Software (OSS) has recently been described as the "infrastructure" of the digital society  1. OSS is an excellent example of open collaboration among many individuals that has a significant impact on the economy  [2] [3] [4] [5]. Within specific OSS ecosystems -collections of software programs or libraries are in many cases, but not always delineated by the use of a particular programming language like Rust, Python, or PHP -developers contribute software that depends on software already in the ecosystem, often created by strangers. For instance, a library that generates data from probability distributions may use a random number generator from another library rather than writing a new one. The outsourcing of core functions leads to a rich structure of technical dependencies, often represented as a network  6.
These libraries are usually hosted on collaborative coding platforms like GitHub or GitLab. The nature of OSS contributions is such that the traces of activity of individuals are observable, i. e. , what they contributed to which libraries and when. The cumulative efforts of thousands of developers can reveal a great deal about the nature of collaborative projects and work  7. Information on the use, visibility, and popular success of individual libraries can be tracked over time  8  , along with the co-evolution of technical dependencies and social collaboration  9. Such data can give insight into the dynamics of massive and decentralized collaborations  6  and how these digital ecosystems evolve. Here, we present a comprehensive dataset on one such ecosystem built around the Rust programming language.
Rust, a relatively young language, has recently seen a sharp increase in popularity. Besides its significant connections with Mozilla 10 , it is, as of December 2021, the second approved language of the Linux kernel besides C. For several years now, it has been voted the "most loved" language in the Stack Overflow Developer Survey. We have collected and curated temporal data on the technical dependencies, developer contributions, and the use and success of individual libraries. Specifically, we can observe when a developer made an elemental contribution of code to a specific library, what other libraries that library depends on, and how widely used and popular the library is. We record over five million distinct contributions of over 72 thousand developers, contributing to over 74 thousand libraries over eight years.
Our data processing pipeline, available as open-source software, combines data from Cargo (the Rust ecosystem library manager) and the code hosting platforms GitHub and GitLab. It considers and handles multiple issues common to the study of collaborative software development data  11  : contributor disambiguation  12, 13  , bot detection  14  , and the identification of nested projects and merged work. The result is a database tracking the evolution of a large, interconnected software ecosystem at a fine scale. In contrast to other data sources on collaborative software development, our dataset contains more accurate and complete data for the Rust software ecosystem. Focusing on Rust allows us to integrate developer contributions with data on software dependencies and usage. In this way, our data is richer and more focused than what can be found in more extensive databases such as GHTorrent  15  , GHArchive, Software Heritage  16  , or World of Code  17.
Moreover, as we highlight in the Technical Validation section, we achieve a broader coverage by focusing on the Rust ecosystem: 15% of the packages in our dataset are not in the GHTorrent database. Our dataset also requires significantly less storage space than the sources mentioned above and can be directly analysed by researchers with minimal computing infrastructure requirements. At the same time, Rust is a large ecosystem that has evolved in a decentralized manner with contributions from thousands of developers hosted on multiple platforms, differentiating it from data sourced from single projects like the Linux Kernel or Apache projects  18. We plan to update the dataset annually provided that the primary upstream sources (crates. io, GitHub & GitLab) remain stable. Researchers can also use our pipeline to reconstruct the dataset, a process that requires some data storage space (around 300 GB, though this volume will likely increase over time as the dataset is updated) and several days to query the data sources and process the results.
Our code can also be adapted to collect data from other ecosystems, such as the Julia programming language's ecosystem. However, we note that not all ecosystems offer the same scope of data as Rust. For example: Rust is relatively young compared to Python, Ruby, or Javascript -as a result a much larger share of the Rust ecosystem's history is accessible on GitHub. The Rust ecosystem is also relatively small: estimates of the NPM (Node) ecosystem's size suggest its metadata alone are greater than 100 GB, while the repos themselves would take up multiple terabytes as described here: https:// socket. dev/blog/inside-node-modules. We proceed as follows: first, we describe our data collection and wrangling process and the resulting database. We compare our data coverage against GHTorrent, a widely used database of OSS contributions, finding that our data is more complete.
Methods We describe the data sources, and how we combine and curate data from various sources to create a comprehensive overview of the Rust ecosystem. We provide a visual overview of the established data processing pipeline in Fig. 1. Data sources and collection. Cargo: Libraries and dependencies. Our first source of data is the Cargo package (which are called crates in the Rust community) registry. Registries, often called package managers, play an important role in nearly all OSS ecosystems. They allow users to download and update different libraries while resolving dependencies and managing conflicts. Other examples of registries around different programming languages include PyPI for Python, CRAN for R, Rubygems for Ruby, and NPM for Node. We use Cargo as a source of technical dependencies and downloads for Rust.
These are available as part of a daily dump from crates. io, accessible via: https://crates. io/data-access. The data can be directly imported in a PostgreSQL database, and contains package names and creation dates, their versions, a list of dependencies for each version with the semantic versioning (semver) syntax associated to them, and the daily downloads per version of each package. For a relevant discussion of the importance of semantic versioning in OSS ecosystems, see recent work by Decan and Mens  25. Packages are also often associated to a repository URL and a documentation URL, but those are not always provided and depend on maintainer input. Code repositories: github, gitlab and other git platforms. To understand who contributes to which library, we turn to the social coding platforms on which these packages are hosted.
In the case of Rust, nearly all packages in Cargo are hosted on either GitHub or GitLab. Specifically, 74,829 packages had links to either platform. Of these links, 51,657 were unique and 46,895 of them could be cloned from either GitHub (44,893), GitLab (1,498)  or other git platforms (504). The inclusion of data from GitLab represents an important extension over the most widely used databases in OSS research, GHTorrent and GH Archive, which only use data from GitHub. Both of these platforms use Git version control, making projects hosted on either alternative comparable. The elemental code contributions to OSS projects using the Git version control system are called commits and are associated to email addresses belonging to the author and contributor (in practice these are usually the same).
GitHub and GitLab both host Git projects (called repositories or repos, for short), which we downloaded and used to extract information about activity and collaboration. The mapping between repos and the libraries hosted on Cargo is not one-to-one and requires additional processing, described below. The Git version control history of a project allows us to examine in detail the contribution histories of all developers working on a project. Indeed, previous work has shown how this highly granular data can be exploited to study collaboration and interactions among developers  26, 27. To do so, we "clone" (download) each repo locally. We also make use of the GitHub and GitLab GraphQL APIs to disambiguate contributors. Measuring use, visibility, and success: downloads and stars. We quantify two dimensions of the use of libraries: the number of times they are downloaded and the number of times they received positive social feedback (stars) on GitHub.
Data processing. Several additional curation steps need to be taken to insure the data collected is useful for the purposes of research into digital collaboration. We describe three specific steps here. Repositories, packages, forks. One would easily assume individual packages of the Rust ecosystem tracked by Cargo correspond one-to-one with repositories on GitHub or GitLab, and this is generally the case. However, some repositories host several packages. For example modules or plugins that extend a core package are often hosted together in the same repo, but are distinguished in Cargo. This presents a data challenge: dependencies are recorded between packages, while contributions are recorded at the repo level. For each package listed in the registry, one or several URLs are typically provided.
They correspond to a link to the code and/or the documentation. Here, we take the URL corresponding to the code, and if empty, coalesce it with the documentation URL. We parse the individual URLs by recognizing the prefixes synonyms of github. com and GitLab. com and the pattern github. com/<owner>/<name> corresponding to a repository. We also try to identify other git platforms by matching the pattern <platform_root_url>/<owner>/ <name>, and using git ls-remote on a maximum of 5 repos per candidate to check if it is indeed a git platform. Repositories are sometimes renamed, and both URLs can be present in distinct packages. The old URL typically redirects to the new one. To resolve this and be able to merge repositories under one entity, we use the Fig. 1  Data processing pipeline. We collect data from Cargo, the package registry of the Rust programming language, and complement it with data from the code hosting platforms GitHub and GitLab.
One remaining repository, although corresponding to numerous downloads, has been pruned from all dependencies to it because of the high number of cycles of the dependency network involving it. We included these pruned dependencies in the dataset for the sake of completeness, but we flagged them for easy removal, or for letting the possibility to investigate other link removal policies. We guarantee absence of cycles for the state of the dependency network at the end of the dataset (2022-09-07) and at the end of the preceding year (2021-12-31) in both spaces, but not at any arbitrary timestamp.
Merging of developer identities. There are many potential ways to disambiguate the identities of contributors  12, 13  , each presenting tradeoffs. In general, Git commits are signed by an email address, not a platform-specific username. Developers often commit code from different computers or environments with different email addresses in their configuration files, and this can result in a significant disambiguation problem. Rather than attempting to infer which email addresses potentially refer to the same person, we query the GitHub API for the GitHub account linked to each commit  35. While this ignores the potential that developers use multiple accounts, we argue that it makes a larger amount of highly justifiable merges among commit author identities than an email address based approach.
Email address-based author identity disambiguation would scale better in larger systems at the cost of accuracy. For each email address, we carry out this process for the most recent commit registered for that email, and if this fails to return an account, we try again with a randomly sampled commit among all those corresponding to this email. After doing the same for GitLab, we also merge matching GitHub and GitLab logins (finding 87 such cases). An additional step is to parse the emails and spot those obviously belonging to GitHub, following the patterns <login>@users. noreply. github. com or <randomint>+<login>@users. noreply. github. com.
Bot accounts. Bots play an important role in modern software development  36, 37  , but need to be handled with care in any study of software systems, as they can make orders of magnitude more contributions than any human developer. Pooling them and their activity with that of human developers would skew any analysis of social cooperation and collaboration  11, 14. While bots have interesting effects on project evolution  36  , we chose to detect and mark bots and more generally invalid accounts in our dataset with a view to excluding them as we are primarily interested in the patterns of contributions of developers. To identify bots, we used a two-step filtering process. First, we extracted all bots on a curated list used as ground truth for bot detection in the software engineering community  14.
We then filtered remaining GitHub accounts with the substrings "*[bot]", "*-bot", "*-bors", "bors-*", "dependabot-*" in their usernames, and finally inspecting manually each individual account with pattern "*bot*". After filtering out bot accounts labelled this way, a manual inspection of the 100 most active remaining accounts was conducted, as well as of all accounts containing the substring "bot". The manual inspection took the following steps: checking the GitHub webpage of a user for a clear name, looking at the description, looking at the commit/PR comments. The 100 most active accounts by number of commits across the full timespan of the dataset were considered, as well the 100 most active accounts in the year 2020. For users that could not be associated to a GitHub account, their emails are filtered when the last part of their prefix (separated by.