dataesr/scientific-paragraphs-categorization huggingface dataset:
Generation of multiple organ atlases A total of 38 scRNA-seq and snRNA-seq datasets were collected. Detailed information of these datasets can be found in Table  S1. Each tissue or organ consisted of four datasets. Because some datasets contained multiple organs, these datasets were repeatedly used in different tissue atlases. For each dataset, raw count expression data was downloaded, accompanied by the meta-information of original annotations, experimental assays, donor identities, ages, genders, and suspension types. These metadata were next unified across datasets. Using the procedure in the Methods section "Cell type harmonization", we performed automatic cell type harmonization for cells from each organ that comprised four datasets. After harmonization, we injected the cell reannotation information into the combined dataset for supervised data integration.
For data integration, we used the annotation-aware procedure as detailed in the Methods section "Annotation-aware data integration", with the batch key being the dataset of origin (for organs of heart, lung, and pancreas) or the donor ID (for organs of blood, bone marrow, hippocampus, intestine, kidney, liver, lymph node, skeletal muscle and spleen) depending on how these batches manifested in the organ studied. We next renamed the harmonized cell types into biologically meaningful labels, followed by further manual curations of cell annotations including missed cell subtypes. These cell types were also linked with the cell type labels from the Cell Ontology 51 for future community-wide query. Violin plot to the right displays marker gene expression across the two goblet subtypes. (C) Upper: as in (A), but for all collagen-producing cell types.
Lower: as in (A), but for the IPF "Fibroblast" in the second dataset that is harmonized into myofibroblasts based on the first and third datasets. Dot plot to the right shows expression of marker genes for the "Fibroblast" identified (bottom) as compared to those from the other two datasets (top). (D) Upper: as in (A), but for macrophage-related populations in the four datasets. Lower: as in (B), but for the pathological "Macrophages" in the third dataset that are harmonized into two macrophage subtypes. Dot plot to the right shows expression of marker genes for the two subtypes identified (bottom) as compared to those from the other two datasets (top). (F) As in (A), but for the three mast cell-related populations, including a "Mast cell" population highly abundant in ambient RNAs (and also with more differentially expressed genes) and failing to match other mast cells.
Acknowledgements We thank  D. Sumanaweera  and  K. Polanski  for feedback on the CellTypist algorithm;  K. Meyer ,  A. Cujba ,  E. Madissoon , and other members of the  Teichmann lab  for helpful discussions;  A. Maartens  and  R. Vento-Tormo  for editing and commenting on the manuscript. This work was funded in part, by the  Wellcome Trust  (grants  206194  and  108413/A/15/D  to S. A. T. ). This work is also funded by  Technology Vision Fund  under  Turing  project code  R-TEC-001. This publication is part of the  Human Cell Atlas  (www. humancellatlas. org/publications/).
aim of this stage is to combine D1 and D2 into a new D1, allowing addition of a new D2 in stage III. As a first step, cells from D1 and D2 are combined, and the combined set of cells are called a new dataset D1. Next, CellTypist reannotates the identities of these cells into cell types in the harmonization table 𝐻𝑇 obtained from stage I. Specifically, based on what cell.
2. Form the 𝑀21 matrix 3. For step 1 , 31 Perform the procedure in stage I with 𝑀12 and 𝑀21, and obtain a new 𝐻𝑇 4. Disentangle 𝐻𝑇 to have multiple columns (one column per original dataset) 5. Perform the procedure in stage II with the new 𝐻𝑇, and form a new D1 6. Incorporate an additional dataset D2, and repeat stage III with the new D1 𝑀12 is formed by examination of the predicted cell types of D1 cells in D2, on the basis of the information from the assignment matrix 𝐴. For step 2, 𝑀21 illustrates what cell types D2 cells are predicted to be in D1. Notably, since cells in D2 are predicted into cell types in D1 which correspond to rows in 𝐻𝑇, not all cells in D2 can be annotated and are thus assigned "UNASSIGNED", which are not used when generating 𝑀21. After step 3, a new 𝐻𝑇 with three columns (D1, relation, and D2) is formed.
Machine learning-based CellTypist models were next built for each tissue and each dataset from the given tissue. The function celltypist. train in CellTypist v1. 0 was used to train logistic regression-based classifiers, with the manual annotation being the label key and feature selection turned on (tag-value pairs: feature_selection = True, n_top_genes = 300). Ultimately, five models were trained for each tissue (four dataset-specific models and one combined model), resulting in a total of 60 models that could be used for automatic cell type annotation for cells querying our database in different tissue contexts. The integrated datasets with harmonized and curated cell annotations in 12 organs, as well as the CellTypist models, are all available at https://www.
Color of the dot represents normalized gene expression, and size represents the percentage of cells expressing a given gene. (D) Upper: as in (C), but for the five memory B cell-related populations. Lower: UMAP visualizations of memory B cells in two datasets that contain a hidden population of ageassociated B cells (ABCs). Dot plots show expression of marker genes for memory B cells and ABCs, with the color and size representing normalized gene expression and the percentage of cells expressing a given gene, respectively.
Color and shape of the dot represent the degree of PF signature and dataset of origin, respectively. (D) As in (C), but for distribution of the extracellular matrix (ECM) signature in collagenproducing cell types. (E) Dot plot showing expression of differentially expressed genes among alveolar epithelial cell types and diseases, with the color and size representing normalized gene expression and the percentage of cells expressing a given gene, respectively. Cell types and diseases are.
Software and data availability PAVICS-Hydro is a set of software packages developed to add hydrological modelling capabilities to PAVICS, an existing platform for climate data analysis available at https://pavics. ouranos. ca/. Users may register a PAVICS account for free with the understanding that the platform be used for scientific purposes. Development of PAVICS started in 2015 and provides climate model data and processing capabilities to users. PAVICS-Hydro, which began services in 2019, created or made significant contributions to the packages shown in Table  1. RavenPy is a Python wrapper for the Raven hydrological framework, meant to facilitate the configuration of hydrological models, the launch of simulations and retrieval of the results. RavenWPS is a server based on the Open Geospatial Consortium (OGC) Web Processing Service (WPS) Application Programming Interface (API), offering processes to extract watershed properties.
Birdy is a Python client providing a user-friendly interface to remote WPS services. Jupyter notebooks  (Fangohr et al. , 2021)  included with the RavenPy documentation show how to use these software libraries together to run typical hydrological modeling workflows. A support forum  (Craig, 2023)  is available for Raven, while for the other libraries, GitHub Issues and Pull requests are the best way to ask questions and propose enhancements. Raven, RavenPy and Birdy can be run on a laptop or workstation running Linux, Windows or OSX. RavenWPS is a server component that has been developed and tested on Linux only. Various installation methods are available for these components, including compiling from source, installation with pip or conda. For those who cannot install the package locally for IT or administrative reasons, the PAVICS platform provides a JupyterLab environment where all these and more are available.
Introduction There has been a growing interest over the last years in developing computational environmental modeling frameworks, with the emergence of multiple hydrology-oriented platforms or systems (e. g. , among others  Alfieri et al. , 2013; Arsenault et al. , 2020; Horsburgh et al. , 2016; Tague and Band, 2004; Taylor et al. , 2021; Werner et al. , 2013; Zhang et al. , 2016). For instance, specific developments in this field include the provision of web-based applications and services, online geospatial databases, model sharing technology, data analysis platforms, and software as a service including hydrological modeling  (Choi et al. , 2021; Gan et al. , 2020; Khattar et al. , 2021; Morsy et al. , 2017; Souffront Alcantara et al. , 2019; Werner et al. , 2013). All these developments have the potential to support hydrological research by facilitating the use of high-end hardware architectures, easing access to large volumes of data, making information retrieval more interactive, and for ensuring the development, repeatability, and reproducibility of experiments while advancing hydrology in a more coherent way  (Ceola et al.
, 2015). Several hydrology-oriented platforms have recently been developed to support the Findable, Accessible, Interoperable, and Reproducible (FAIR) principles  (Wilkinson et al. , 2016). Among others, we can cite the HydroShare  (Horsburgh et al. , 2016) , HydroDS  (Gichamo et al. , 2020) , and eWaterCycle  (Hut et al. , 2022)  platforms. All these platforms rely on the data, models, and codes sharing with the aim of enhancing the trustworthiness of research findings, while increasing the reusability of research results. HydroShare provides open access datasets and codes in an easily accessible and organized repository, also providing access through Jupyter notebook environments (e. g. ,  Bandaragoda et al. , 2019; Leonard et al. , 2019). HydroDS is a web service that facilitates downloading data in a suitable format for hydrological models; the combination of both provides users a gateway to all required data for running hydrological models and performing hydrological analyses.
The eWa-terCycle platform gives open access to hydrological models of various complexity (conceptual to distributed structures) through a special interface that allows users to add their own models to the platform, written in different programming languages that interact from the Jupyter notebook environment. Also, eWaterCycle has the possibility of incorporating any hydrological model in any programming language through the Basic Modeling Interface (BMI), a modeling framework that allows interoperability between models and data through a common programming interface. The eWaterCycle platform currently includes the PCR-GLOBWB 2. 0  (Sutanudjaja et al. , 2018) , wflow  (Schellekens et al. , 2020) , Hype  (Lindström et al. , 2010) , LISFLOOD  (Van Der Knijff et al. , 2010) , TopoFlex HBV  (Gao et al.
, 2014) , MARRMoT  (Knoben et al. , 2019)  and WALRUS  (Brauer et al. , 2014)  models, with upcoming support for GlobWat  (Hoogeveen et al. , 2015) , SUMMA  (Clark et al. , 2015)  and mHM  (Samaniego et al)  models and modelling frameworks. These platforms provide users with a variety of methods and tools for performing hydrological research and applications, making hydrological sciences more accessible and reproducible. However, some applications and workflows are not currently trivial to implement, such as streamflow forecasting, climate change impact studies or regionalization at ungauged sites, due to the necessity of implementing specific processes and accessing particularly large datasets. For example, HydroShare is a repository for data and codes sharing, and is not in itself a modeling platform, and HydroDS allows generating model inputs but does not provide flexible modeling capabilities.
eWaterCycle provides a comprehensive package by including flexibility in hydrological model structure and data inputs in an online environment where users can tailor workflows according to their needs. Choi et al. (2021)  showed that it was possible to combine several existing platforms and tools for data access, geospatial processing, and hydrological modelling to provide reliable and reproducible studies. In their work, some example applications are provided, and tools include the ability to simulate streamflow and forecast flows. However, tools pertaining to climate change impact assessment are not included de facto and would have to be integrated from another source. In Canada, a consortium initiative aims to create an innovative research software platform to streamline climate science research.
The Power Analytics and Visualization for Climate Science (PAVICS) platform (https://pavics. ouranos. ca/) is a virtual laboratory first launched in 2015 facilitating the analysis of climate data over North America. It was originally developed jointly by Ouranos, a regional climate adaptation consortium, and the Centre de Recherche Informatique de Montréal, an applied research center on information technology and artificial intelligence. PAVICS provides access to various data collections ranging from observations, reanalyses, numerical weather forecasts and climate projections, in addition to ensuring quality control on the accompanying metadata. It also hosts a JupyterLab Python programming environment with the most up-to-date scientific libraries preinstalled. Acting as a climate data repository and processing suite, it allows researchers to access, process, analyze, and transform data into useable climate information to improve the quality and reproducibility of scientific research studies.
Overview of the PAVICS-hydro platform PAVICS-Hydro is an online hydrological modeling and forecasting service that is free of charge, open-source, and open-access. It is developed in the Python programming language and offers users the capacity to perform geospatial analysis, build hydrological models, apply various parameter estimation algorithms, estimate and predict streamflows, and/or evaluate the impacts of climate change on future streamflows via open access to data and PAVICS-Hydro software libraries. Fig. 1  presents a high-level overview of the PAVICS architecture, which includes a THREDDS data server for hydroclimatic data stored in the netCDF format, a GeoServer to store watershed polygons and land-use raster maps, a JupyterLab server, Open Geospatial Consortium (OGC) Web Processing Services (WPS) servers, and middleware responsible for authentication and authorization.
The JupyterLab environment comes installed with a large collection of scientific libraries operating on xarray data structures. Novel components or substantial contributions to existing components made through the PAVICS-Hydro project are indicated by a color code. The contributions of PAVICS-Hydro to the PAVICS architecture allow users to perform hydrological studies within PAVICS. The main contribution of PAVICS-Hydro is the RavenPy Python package, which wraps the Raven hydrological model framework  (Craig et al. , 2020). RavenPy provides a programmatic mechanism to configure models (setting parameters, options, and hydrological processes), run simulations, and retrieve results. Together, these capabilities allow hydrologists to design multiple experiments (e. g. , ensemble forecasting, regionalization), run them in parallel and analyze results, something that would be time consuming to do manually.
Actual simulations are performed by the Raven executable, RavenPy playing the role of an interface. Raven and RavenPy can be installed and run on a laptop. However, setting up a programming environment with multiple scientific libraries can be difficult in some professional settings with IT restrictions. The PAVICS platform provides an online ready-to-use Python programming environment where RavenPy is pre-installed, along with a host of other packages that can be used to perform hydrological modelling workflows, such as bias-correction and time-series analysis, high bandwidth access to large data collection (e. g. , ERA5 and CMIP6), hindcast and forecast databases (ECCC real-time and archived forecasts), HydroShare interoperability, and compute services such watershed delineation and spatial subsetting.
Initial design choices When the project team first proposed the idea of an online hydrological modeling service, a few requirements were obvious. The design of the platform would provide: • geospatial analysis capabilities to evaluate catchment drainage areas, boundaries, slopes, and other relevant physiographic properties; • at least one hydrological model, with the possibility to add others over time; • easy access to meteorological databases and future climate scenarios to feed the hydrological model(s). These three aspects are the basis of the platform on which other functionalities would eventually build upon. The main design choices are presented here, along with the reasons why these solutions were preferred to alternatives.
, 2023) , this functionality was contributed to xESMF  (Zhuang et al. , 2023) , a package compatible with the xarray stack that exposes regridding algorithms implemented in the Earth System Modeling Framework (ESMF) Fortran library. The approach leverages a conservative regridding algorithm, which preserves the integral over the surface area between input and output grids. This xESMF. SpatialAverager is now used to convert gridded meteorological datasets into time series of subbasin averages, which simplifies hydrological model setup and speeds up computations.
Another solution for the standardization of model configurations, inputs and outputs is to adopt a general purpose hydrological framework such as SUMMA  (Clark et al. , 2015) , MARRMoT  (Knoben et al. , 2019) , wflow  (Schellekens et al. , 2020)  and Raven  (Craig et al. , 2020). Raven is a flexible framework that can emulate a large suite of existing models or create new models by combining hydrological process (e. g. , users can select from a series of snow accumulation and melt modules, infiltration methods, evapotranspiration formulas, routing schemes. etc. ). Watershed configuration and model options can be shared across different emulators, facilitating multi-model experiments. Furthermore, Raven has a powerful and flexible data processing engine that supports both lumped and semi-distributed hydrological model structures.
In Raven, processes, simulation parameters and input files are specified in text-based configuration files parsed by the executable, and simulation outputs are written to an output file on disk. In situations where multiple simulations are run using the same input data, such as model calibration, this unnecessary input/output (I/O) hampers performance. Work is underway to expose Raven components through the BMI interface, which could eventually allow on-demand execution of individual model components, in addition to making Raven compatible with the eWa-terCycle platform. To integrate the Raven modeling framework into an existing climate analytic programming environment written in Python, a Python wrapper for Raven was developed, called RavenPy (see Table  1 ). Rav-enPy offers an interface to a subset of Raven's configuration options and writes text-based Raven configuration files ready to be executed.
Meteorological and climate data access Users can access meteorological and climate data in one of three ways on the PAVICS-Hydro platform: 1) Use one of the datasets stored on the platform; 2) Upload their own data to the platform in their personal account; or 3) Stream data from external online databases. As for the basic set of data provided with the platform, we provide access to ERA5 reanalysis data, which is a high-quality dataset with a complete series of climate data over the entire world (0. 25 • × 0. 25 • spatial grid) from 1950 until today, in quasi-real time  (Hersbach et al. , 2020). The netCDF files distributed by ECMWF are however not designed for typical hydrological applications. Indeed, the chunking Direct access to the python xarray package to process netcdf files and datasets allows users to tailor the data to the specific needs for their use-case.
Third, data were stored in zarr format and hosted on a public Amazon S3 bucket to take advantage of distributed object storage technology. An intake catalog indexes individual files and quickly and efficiently retrieves the data. Together, these measures make it such that the speed at which data is delivered is limited by internet bandwidth rather than computing power. This data can be accessed through the Jupyter Notebook titled "03_Extracting_forcing_data. ipynb" demonstrating ERA5 data access on the PAVICS-Hydro platform as well as in the supplementary information. The same data request now takes 2 s rather than 2 h. This was a breakthrough that was solved by community members that were outside the project team, highlighting the advantages of collaboration in open-source software. Note that this database currently only contains hourly ERA5 precipitation and temperatures covering the globe.
There is an ongoing effort to also include ERA5-Land hourly precipitation and temperatures as well as snow-water equivalent (SWE), which should be made available later in 2023. More data could be added in time if an alternate funding source can cover the long-term data hosting costs. These datasets are continuously updated once per week with the available data as they become available from the official ERA5 database. It is important to note that running a very large (e. g. , continentalsized) watershed in semi-distributed mode over long periods with distributed ERA5 meteorological data could overwhelm the PAVICS system memory in certain cases. In those cases, users should consider running their model locally using the RavenPy package, but can also gather the ERA5 data they need from the platform independently.
For example, Raven can run in distributed mode over a 275,000 km 2 watershed and be calibrated with 10,000 model evaluations without any problems on a standard computer  (Brown and Craig, 2020) , but larger catchments or models remain untested. The ERA5 database is also accessible to users outside of PAVICS-Hydro and is used by many research groups throughout the world to ensure quick and efficient access to this data. Another option that has been included in PAVICS-Hydro is the possibility of getting data directly from the HydroShare database. This allows for a greater level of interoperability and allows users to build workflows with archived data that makes workflows more repeatable. An example workflow for HydroShare access using Notebooks is available in the "HydroShare_Integration. ipynb" notebook.
Given a finite budget, investing in a front-end necessarily reduces the number of scientific applications that can be developed and maintained. On the opposite end of the spectrum, some platforms use APIs to gather requests and return results directly, without any GUI. Users must learn all the options through the documentation. This makes the learning curve very steep, but gives complete control over the operations to users, including batch-processing and duplicating results easily. For PAVICS-Hydro, a compromise between user-friendliness and the ability to customize workflows, reproduce methodologies and maximize efficiency was found in the adoption of Jupyter Notebooks, which can be seen as an intermediate solution between a GUI and pure API or code access. They can contain widgets that make certain graphical interactions possible, such as displaying and interacting with figures, maps, and simple menu items.