dataesr/scientific-paragraphs-categorization huggingface dataset:
Data Collection There are many openly accessible repositories of health data  (14) (15) (16). One of the key challenges of building a data analysis platform onto a broad set of data sources, which operate under distinct governance systems and are funded by a variety of different sources, is the longterm availability and immutability of these datasets. Lin et al. (17)  defines the TRUST principles for open datasets: Transparency, Responsibility, User focus, Sustainability and Technology. Another set of principles, known as the FAIR data principles  (18)  mention that all open data repositories should be Findable, Accessible, Interoperable, and Reusable. The platform follows all these principles, but also guarantees that all the data used for a particular analysis remains fixed and accessible as it was at the time the analysis was done.
In order to allow for that, we keep a copy of the original data in our servers, whenever its licensing terms permit. In the implementation of this replication, we make a distinction between static and dynamic datasets. Static datasets are the ones not subject to updating and revisionthese datasets are imported only once. Dynamic datasets, on the other hand, must be periodically updated or extended. A good example of these are disease surveillance databases such as COVID-19 case counts. For these datasets, we define a periodicity for their updating, which is automatically triggered by the platform. All the workflow related to the data collection and integration is implemented using Apache airflow. Apache airflow is a distributed task scheduler, that allow for computational tasks defined as parts of a workflow DAG to be efficiently scheduled and monitored.
All datasets, raw or transformed, are stored into a PostgreSQL relational database. On this database server, data is organized according to access level, wherein public and restricted access datasets are kept on entirely separate databases (figure  4 ). This guarantees that we can expose the full contents of the public datasets on our web-based visual analytics tool, without compromising the security of the non-public datasets. One key aspect of the platform is its ability to connect and pull data from widely used health information management systems (HIMS) such as DHIS2, GoData and others. Governmental and also private HIMS are typically closed platforms, where only authorized personnel have access to the data. Therefore, EpiGraphHub aims to make available open source software to make it as easy as possible to pull data from these platforms, given the required authorization from data owners.
Data Transformation As an integral part of the Data integration pipeline, some datasets may require some transformations applied before they are inserted into the database for storage. Examples of such transformation can go from simple normalization of dates to a standard, Addition of ISO geocodes to facilitate the construction of maps, etc. These transformations are designed to not alter in any way the semantics of the data, or its scope. They focus rather on enriching it, or making it consistent to data representation standards. As data collection and transformation scripts document the source of the original data and the transformations applied, all transformations are clear to users, who are able to modify them for their personal use.
Data Exploration and Visualization A web interface for interactive querying and visualization of the platform's datasets is available 3 , to help users without programming skills to create visualizations from data queries. The results from the queries can then be published or shared or be integrated into live dashboards that get updated whenever the underlying data changes. This interface is based on the open source visual analytics platform Apache Superset. Figure  5  shows a dashboard created on the platform. Sharing and publishing of the visualizations built inside the platform can be done in multiple ways. All charts, SQL Queries, and Dashboards created and saved by a user are given a permanent URL which can be shared with users outside the platform. Dashboards can be flagged as published, meaning they can be seen by all users.
This web platform allows for fine-grained access control to datasets via user permissions that can be customized for users that create an account on the platform. Another advantage of the EpiGraphHub Web interface is that it offers an accessible environment for a gentle introduction to SQL (Structured Query Language) through the 3 https://sedac. ciesin. columbia. edu/data/collection/gpw-v4 possibility of translating point-and-click based queries performed on the platform into the equivalent SQL code which can be saved, shared and further modified by users.
Data Analyses Different levels of analyses are possible within the EpiGraphHub platform. One level is the data cleaning and transformation procedures which are applied between the data collection and its storage into the platform's database. Then we have the analyses which are done within the database through the web interface either through the graphical interface or by writing arbitrary SQL code that runs on the server. Finally, we have the more advanced, analytical API, exposed through the EpiGraphHub Python and R Libraries. These Software libraries expose a number of analytical methods as well as data access functions that help users to create their own analytical applications based on EpiGraphHub's available data. One example of a data analysis workflow within the EpiGraphHub platform was the support to the African regional office of the World Health Organization (WHO AFRO) during the first year of the COVID-19 pandemic  (19, 20).
Individual case data from member states was received as national linelists stored as Excel® files. Data quality checks were performed for each country linelist, and datasets were then harmonized to have standardized variables. The following step was the integration of all datasets into the EpigraphHub database using our libraries. These datasets contained individual level information for all countries. All variables related to the geographical localization of the cases (such as residence, local of infection, local of reporting, among others) were then standardized using ISO 3166-1 alpha-3 codes for country, district and province levels. These harmonization procedures were then followed by integration with data provided by the GADM project 4 for all three administrative levels. Population data was obtained via raster files from the Gridded Population of the World (GPW v4 5 ) collection, which provided fine resolution for population counts inside districts and provinces of the analysed countries.
Application Hosting The EpiGraphHub also provides a workflow for quick deployment of Web or mobile applications based on the platform. This workflow allows for continuous integration and deployment from a GitHub repository. It currently supports Python-based Streamlit® and R-based Shiny® applications. Other application frameworks may be supported in the future. 4 Can create "data preview" charts 5 Can federate across instances Applications are encapsulated within docker containers hosted on our server. This allows for low latency in accessing Hub-hosted datasets. Docker templates are available for app developers to test and deploy their apps locally, without having to bother with deployment details. Applications in production are updated via a continuous deployment workflow that allows for a rebuild of the containers whenever a new release is created in its GitHub repository.
Discussion A number of health data aggregation platforms have been developed in recent years, particularly in response to the COVID-19 pandemic. The problem is not new. In Brazil, the Infodengue and Infogripe projects have been monitoring arboviruses and Influenza respectively and making data and epidemiological analyses open to the general public for many years  (21, 22). Hürliman et al. (23) , in 2011, talks about the need for a global database to monitor neglected tropical diseases, but more than ten years later no such database exists. During the COVID-19 pandemic a number of new data repositories for disease surveillance have been developed. For example, Our World in Data (OWID), spearheaded by Oxford University, not only focused on disease surveillance, has played an essential role during the pandemic to guarantee an openly accessible global perspective of the global spread of SARS-CoV2 and its variants  (24).
OWID also maintains live dashboards for interactive exploration of their datasets. Google also entered this arena, through its public datasets program with the construction of a fine-grained collection of COVID19 data  (15). Google's datasets excel on ease of use, and built-in functionality for quick visual explorations of the data. The COVID19 pandemic has also spurred many countries to make their own COVID19 data available on the web. However, we suspect that this attitude will not survive the end of the pandemic. The best indication of this is the fact that the openness observed for COVID data has not yet been extended to other communicable diseases. The need for more widespread transparency towards data that is of interest to public health is clear. Such transparency cannot come only eventually, in response to visible health emergencies.
Rather, it must be an integral part of governmental infrastructure for health data management. The data Integration aspect of EpiGraphHub goes further than helping with openness, but it also brings together all the essential pieces of the data puzzle of effective public health policies  (25). Moreover, being a completely open source package that can easily be deployed in a country, it fills a gap with an easy to deploy and low-cost solution for health data analysis. The EpiGraphHub platform, although still in development, is already being used to help in the response to the COVID-19 pandemic in many countries  (19, 20). As the pressure for urgent responses to the current pandemic finally wanes, we hope our platform will be ready to continue to provide value to disease surveillance programs across the globe and serve as a prime example of the benefit of openly accessible public health data.
Figure2: Platform architecture is based on docker containers. A container is a standard unit of software that packages up code and all its dependencies, so the application can be deployed quickly and runs reliably across different computing environments. In the diagram above, two of the key components of the platform are represented as the boxes with green lids. Containers are connected into a virtual network within the host environment and can exchange data. External web apps can connect directly to the containers to request data via the exposed APIs.
Feature EpiGraphHub Our World in Data CKAN(12) Socrata Google Data Studio Automated, user-defined data collection Connects to external databases 1 ¹ Dashboard creation ² 2 Codeless use User-specific data processing and analysis Open Source Cloud or local deployment Integrated SQL IDE Epidemiological analysis libraries 1 https://epigraphhub. org 2 https://gadm. org/ Methods System Architecture Figure 1: Overall organization of the EpiGraphHub platform, with its four main modules.
Li et al. developed a novel scoring model focusing on the interaction of immune infiltration and ferroptosis to predict the overall survival of ovarian cancer patients. Finally, Sun et al. focused on endometrial cancer and correlated a messenger RNA alternative splicing gene signature to the patient's prognosis and the immune-tumor microenvironment. Additional genetics analyses were the focus of 6 contributions that used, for example, datasets from genomewide association studies or bulk and single-cell RNA-seq to investigate different aspects of women's health, minority populations' health, or human development. He et al. used summary datasets from genome-wide association studies in a Mendelian randomization analysis to estimate the causal relationship between birth weight and female-related phenotypes and diseases.
They demonstrated that birth weight may play a role in women's body mass index, menarche, decreased levels of adult sex hormone-binding globulin, and increased levels of bioavailable testosterone. Liu et al. used bulk RNA-seq ovary datasets and single-cell RNA-seq follicles datasets to determine molecular mechanisms underlying the ovarian reserve, and identified a gene expression signature that was highly correlated to the ovarian reserve of the primordial follicle pool. This gene signature has the potential to be used as clinical biomarkers for the prediction of women's ovarian reserve and in the development of future fertility targeted interventions. Yin et al. focused on competitive endogenous RNAs, including long non-coding RNAs, circular RNAs, and messenger RNAs, and their potential role in the pathogenesis of endometriosis.
As a result, six different queries for information retrieval were written, one for each research axis in both of the chosen sources. For English documents, information was retrieved from the Web of Science database. Items written in French were accessed from the HAL-SHS collection using the API made available by the platform (CCSD, 2021). HAL-SHS is a mandatory repository for French researchers that has been poorly explored and remains highly relevant since regional databases are an essential source to study social sciences' research output  (Engels et al. , 2012). In both cases, the fields we retrieved were the titles, abstracts, keywords, authors, and publication years. In WoS, the search strategy was based on the documents' topics. In HAL-SHS, full texts were available and searched; we excluded doctoral thesis and other teaching-related documents.
We refined and improved our search strategy by consecutive iterations after analyzing the results in a continuous dialog with the researchers, experts in the field. As a result, we established the following query lines for each language, as shown in "Appendix" (Table  1 ). In this process, we gathered 1798 English documents from the Web of Science (WoS) and 4545 French documents from the HAL-SHS, both between 1980 and 2020. The query aimed at articles, books, book chapters, and research communications in both sources. We processed the two resulting collections with the help of the CorText platform using NLP techniques to obtain a fine-grain meaningful set of 750 terms describing the main semantic features of both collections. These lists of terms were manually curated 1  to eliminate noise and shallow terms in both French and English.
Different objects are addressed at different scales in English, establishing a particular set of connections towards specialization rather than comprehensive approaches, as seen in French. Regarding our data sources, some limitations are worth being mentioned. We have resorted to a novel data source such as HAL-SHS for the French part of our analysis. Although this is a rapid-evolving open science initiative, many normalization tasks are still lacking. These restrictions limit the possibilities of running traditional bibliometric analysis based on journals, authors, or institutions, as this information is not yet available or complete for records within HAL-SHS. Further research should investigate these publication practices for non-English speakers; the particular case of French is of specific interest since it goes well beyond France (e.
, τ n , λ 1 1 , λ 1 2 ,. , λ n 1 , λ n 2 ) ∈ R ∑ i=1 n|A i |+2n , (20) and G(ζ) =                               -P 1 (τ -1 ) -C 1 Q 1 (τ -1 )τ 1 ∥Q 1 2 1 (τ -1 )τ 1 ∥ -λ 1 1 1 |e 1 | + λ 1 2 1 |e 1 | -P 2 (τ -2 ) -C 2 Q 2 (τ -2 )τ 2 ∥Q 1 2 2 (τ -2 )τ 2 ∥ -λ 2 1 1 |e 2 | + λ 2 2 1 |e 2 |. -P n (τ -n ) -C n Q n (τ -n )τ n ∥Q 1 2 n (τ -n )τ n ∥ -λ n 1 1 |e n | + λ n 2 1 |e n | ∑ |A 1 | j=1 τ 1 j -1 1 -∑ |A 1 | j=1 τ 1 j ∑ |A 2 | j=1 τ 2 j -1 1 -∑ |A 2 | j=1 τ 2 j. ∑ |A n | j=1 τ n j -1 1 -∑ In this section, we generate random instances in Matlab and we use PATH Solver to come up with Nash equilibrium. PATH Solver is an implementation of a stabilized Newton method for the solution of the Mixed Complementarity Problem (MCP)  (Dirkse and Ferris, 1995). For our concern, once the analytic form of G(x) and its Jacobian is known and coded, we can directly use PATH to solve our NCP.
Numerical Results for Large Size Game Instances Here we solve large size instances with up to (5 × 5 × 5 × 5 × 5) games. The NCPs are implemented in Matlab and solved by PATH on Intel Core i72,6 GHz with 32GB RAM. We randomly generated 100 tests of several groups of different game sizes and confidence levels for both distributions, then we computed the average running time and the success rate in relation of solved instances by PATH solver. Table  3  summarizes the numerical results for different sizes of the chanceconstrained games. Column 1 presents the size of the game instances. Columns 2-4 show the confidence level α, success rate and average CPU time for problems under Cauchy distribution, respectively. Columns 5-7 provide the same information as Columns 2-4 for problems under normal distribution.
A Case of Study of Workflow Deconstruction: SurfCut The ImageJ macro "SurfCut" was chosen as a study case for workflow deconstruction during the NEUBIAS training school TS15  (Bordeaux, March 2020). Interestingly, this led to numerous new ideas and ways to implement SurfCut. Some trainees added GPU processing capability with CLIJ  (Haase et al. , 2020) , while others completely rewrote the workflow in Python 2  and Matlab 3  and benchmarked the different versions (SurfCut, GPU-SurfCut, Python-SurfCut and Matlab-SurfCut). Furthermore, this deconstruction session, along with the writing of this book chapter, also prompted us to develop a new version of the SurfCut macro, SurfCut2, including a complete refactoring of the code (as described in this book chapter), bug-fixing, and addition of new functions.
What Is SurfCut? SurfCut is an ImageJ macro that allows the numerical extraction of a thin, curved, layer of signal in a 3D confocal stack by taking as reference the surface of a 3D biological object present in the volume of the stack  (Erguvan et al. , 2019). The macro is written in the ImageJ1. x(IJ1) macro language, and runs on the Fiji platform  (Schindelin et al. , 2012). Using built-in ImageJ functions, the biological object in the image is blurred, segmented, filled, shifted in the Z-axis at two different depths and used as a mask to erase unwanted raw signals at a chosen distance from the surface of the detected object (. Fig. 6. 1, and detailed description in 7 Sect. 6. 4). The whole workflow can be viewed as a sort of ''object surface''-guided signal filtering method. This allows the removal of unwanted signals relative to the surface of the biological object and the extraction of specific structures from the 3D stack, such as the cell contours (.
Given the lack of available user-friendly tools to perform a proper extraction of 2D cell contours from 3D confocal stacks, we developed SurfCut  (Erguvan et al. , 2019). Although the SurfCut macro was written in the context of a biological project and could have ended (somewhat hidden) in the "Material and method"-section of a larger biological publication (still being finalized at the time of writing this chapter but available as a preprint  (Malivert et al. , 2021 )), we decided to publish it separately  (Erguvan et al. , 2019) , to assign a DOI to the code and provide image data, also identified with a DOI  (Erguvan and Verger, 2019) , to enable testing of the macro. We think that the publishing of this type of macro gives more visibility to the bioimage analysis workflows and, by giving all the space needed to the description of the workflow, ensures a greater reproducibility.
Other Similar Tools Before developing SurfCut, we had identified in our bibliographical searches other workflows performing apparently similar outputs, but none of them fitted exactly our needs. As described in  Erguvan et al. (2019) , we were originally using the software MorphoGraphX (MGX) (Barbier de  Reuille et al. , 2015)  that provides a very accurate solution to our problem  (Erguvan et al. , 2019; Verger et al. , 2018) , but requires too many manual steps and does not easily allow batch processing. In addition, Merryproj (Barbier de  Reuille et al. , 2005) , SurfaceProject  (Band et al. , 2014) , LSM-W2  (Zubairova et al. , 2019)  and Smooth 2D manifold  (Shihavuddin et al. , 2017)  were discussed in  Erguvan et al. (2019)  and were found inadequate for our purpose. After the independent publication of the SurfCut macro, we discovered other workflows that our first search had not revealed, such as the ImageJ macro identifyuppersurfacev2  (Galea et al.
Tools 5 Fiji: Download and install Fiji on your computer (7 https://imagej. net/Fiji/ Downloads) 5 ImageJ macro SurfCut: Download the "SurfCut. ijm" macro file to your computer (7 https://github. com/sverger/SurfCut). To run the macro in Fiji either click on Plugins>Macro>Run and select "SurfCut. ijm", or drag and drop "SurfCut. ijm" into the Fiji window and click run. 5 ImageJ macro SurfCut2: Download the "SurfCut2. ijm" macro file to your computer (7 https://github. com/VergerLab/SurfCut2). Follow the same instructions as for the ImageJ macro SurfCut. 5 ImageJ macro used for exercises in this chapter can be found at: 7 https://github. com/NEUBIAS/neubias-springer-book-2021.
Step 1. Identification of Components in the Textual Description When working with a published workflow, the first step is to identify the components in the text of the publication and the order in which they are used. Nowadays, published workflows are often accompanied by a detailed user manual and/or a "readme" if the code is released on GitHub or GitLab. This text can also contain additional information on the components and on the links between them. ? Exercise 1 1. Read  Erguvan et al. (2019)  and underline in the text all elements describing the components of the SurfCut workflow. Then summarize the result as an ordered list of components. 2. Which additional useful information relative to the components can you find on the GitHub repository of the SurfCut macro 8  ? v Solution to Exercise 1 1.
Step 2. Drawing a Workflow Scheme We identified above the workflow components from the text. Let us now draw a scheme of the workflow. A workflow scheme summarizes and links all the components of a workflow. This scheme will serve as a guide to get an overview of the workflow, and identify those components in the code that can be refined after Step 4, if needed. Figure  6. 3 is a graphical scheme of a general bioimage analysis workflow. ? Exercise 2 Utilizing information found in Exercise 1, draw the scheme of the SurfCut workflow: start drawing one box per component following the guidelines in Step 1. Then identify each component by a short informative name and link components with each other, so that the input of a component is an output of the previous component. The SurfCut macro has two modes: (1) "calibrate", where the components are executed one-by-one and only once, and (2) "batch", where the complete workflow is repeated on several images.
For other more advanced or less known languages, as well as more exotic software, another sequence of preliminary checks would be: 5 Language: In which language is this workflow written? 5 Platform: On which platform can I execute it? 5 Inter-operability: How complex will it be to link this workflow to my other tools written in another language or executed on another platform? 5 Code migration capability: In case I need to make some modifications to the workflow, do I have other options than fully rewriting it in my favorite language? ? Exercise 3 1. Install the SurfCut macro and execute it on the associated data. 2. Identify, based on the text of the reference publication, the "readme" in the Github repository and the user guide, all elements restricting the datasets of certain type to be used with SurfCut.