dataesr/scientific-paragraphs-categorization huggingface dataset:
These challenges typically focus on a single dataset and task and thus do not reflect the diversity (e. g. , multi-modal and multi-institutional) and complexity (e. g. , different clinical and technical workflows) of real-world use cases. Model training and evaluation on non-diverse datasets carries increased risk of overfitting and the chance that even top-performing models will not generalize in real-world use cases, where clinical data reside in multi-institutional, geographically distributed organizations with significant differences across domains (i. e. , domain shifts)  14. 3 Proposed Solution: An Open Benchmarking.
Importantly, by limiting the goal to model evaluation, and by aggregating only evaluation metrics, federated evaluation poses significantly lower risk to patient privacy than collaborative model training, while also minimizing the risk  37, 38  of intellectual property theft and data misuse. More specifically, our open platform for federated evaluation will provide a common, open-source infrastructure for defining medical AI benchmarks and coordinating federated evaluation of models against such benchmarks. We are building the infrastructure with best practices to help align AI model owner/developers with data owners, through an active community with a neutral organization at its core. We intend for our approach to be compatible with, and to build upon, existing federated learning frameworks, rather than to compete with them.
Risks are Mitigated by Focusing on Model Evaluation and Trusted Groups MedPerf addresses regulatory, liability, and public perception risks using a three-pronged approach. First, because the initial focus is on model evaluation instead of training, our federated evaluation approach maximizes value without data leaving the possession of data owners, either directly or accidentally leaked through results. We only need data owners to share agreed-upon evaluation metrics (e. g. , specificity), which are aggregated across participating institutions before disseminating. This mitigates most regulatory, public perception, and legal risk. Second, Medperf retains human evaluators  39  as a critical part of the proces: the MedPerf client software requires a data-owner's system administrator to approve all model evaluations and result uploads, and automatically records transactions to support auditing.
Moreover, to protect against malicious or erroneous implementations, MedPerf requires that (a) all novel code has no network access and restricted local file-system access, (b) evaluation algorithm implementations are well-vetteed and common among benchmarks, and (c) all output (i. e. , statistics) must be explicitly approved by data owners before it is uploaded to the MedPerf platform. Third, we leverage social trust: we enable benchmarks to be specified, developed, and deployed publicly or within closed groups, such as provider networks with pre-existing trusted relationships and business and legal contracts, and these closed-group benchmarks will be prioritized during the pilot phase of deployment. We are developing the MedPerf infrastructure through MLCommons, a non-profit with diverse membership and open-source practices, backed by dozens of high-profile companies and institutions (https://mlcommons.
Practices We aim to reduce the costs of data sharing by developing open-source infrastructure and best practices that enable infrastructure vendors, model owners, and data owners to collaboratively build within a fast-growing ecosystem. First, we provide community best practices for sharing models and data. For instance, we are using the MLCube container for model sharing (see (https://mlcommons. org/en/mlcube/) for concept introduction and practical examples, and (https://github. com/mlcommons/mlcube) for the code repository). MLCube extends common container standards, such as Docker and Singularity, to offer a simple and consistent file system-based interface for other infrastructure to train or make inferences using AI models (e. g. , for testing harnesses or federated learning). Additionally, deployment tools like Docker and Singularity enable hospital information technology groups to evaluate the AI model code for security concerns using common methods and tools.
Second, we are developing an open-source hub for medical AI benchmarks and a consistent methodology for benchmarking. The hub will offer coordination among benchmark groups, model developers, and data owners by providing a central model and data registry and by storing results, but will not directly handle proprietary models or data, ensuring that these assets remain in the hands of their owners. Instead, model and data owners will register hashes to enable checking the integrity of their assets without exposing them to the platform. This method will ensure that benchmark results can be compared to better establish promising technical approaches.
Return on Investment: Increasing Certainty through Better Model Evaluation Our approach decreases the uncertainty of deploying AI models by enabling easy evaluation against data held by multiple data owners. We enable model developers to indirectly interact with data owners' datasets and thus tap into a large, virtual test set. In doing so, we increase the size of the test set and thereby reduce uncertainty of the evaluation -even if all the data are from a single provider. More importantly, by enabling evaluation against data from multiple providers, we can more effectively evaluate how the model will perform when deployed at different facilities with diverse patient populations. And by providing multi-site performance feedback to model developers, we increase the odds of successful model deployment.
6. Registered Models: a list of registered models to run in this benchmark. 7. Documentation: documents for understanding and using the benchmark. Our platform uses the MLCube container for components such as Dataset Preparation, Evaluation, and the Registered Models. MLCube containers are software containers (e. g. , Docker and Singularity) with standard metadata and a consistent file-system level interface. By using MLCube, the infrastructure software can easily interact with models implemented using different approaches and/or frameworks, running on different hardware platforms, as well as leverage common software tools for validating proper secure implementation practices (e. g. , CIS Docker Benchmarks).
2  for details of all workflow steps, 1 through 5. • Once the benchmark, dataset, and models are registered to the benchmarking platform, the platform notifies the data owners that models are available for benchmarking • The data owner runs a benchmarking client that downloads available models, reviews and approves models for safety, then approves execution • Once execution completes, the data owner reviews and approves upload of the results to the benchmark platform 5 Release results • Benchmark results are aggregated by the benchmarking platform and shared per the policy specified by the benchmark group.
Many benchmarking efforts may initially focus on data providers with existing agreements Prepare and • In coordination with the benchmark group, dataset owners are responsible for register data data preparation (i. e. , extraction, preprocessing, labelling, reviewing for sets legal/ethical compliance) • Once the data is prepared and approved by the data owner, the dataset can be registered with the benchmarking platform 3 Recruit • Model owners modify the benchmark reference implementation. To enable model consistent execution on data owner systems, the solutions are packaged owners inside MLCube containers • Model owners must conduct appropriate legal and ethical review prior to submission of a solution for evaluation Prepare and register models • 4 Execute benchmarks.
Introduction: Hernia Basecamp is an online learning platform hosted within the WebSurg website. One of the drivers of its development was to cover the syllabus of the UEMS AWS examination, but it is a learning resource in its own right. There are currently 205 video lectures, with a number of them selected to create 10 modules of 3 h each with UEMS CME accreditation. The aim of this study was to review the Hernia Basecamp usage since launch in June 2021.
Methods: The Hernia Basecamp WebSurg platform was interrogated using Matomo Analytics in January 2023 (19 month period since launch). Data on the number of visits, pages looked at and time spent on the platform per visit, along with the number of CME modules taken and passed were collected. Results: Users from 146 countries visited the Hernia Basecamp site 17,171 times (6,586 times, 38. 4% in first 9 months). The top 5 countries by visitors were the United Kingdom, Mexico, Spain, United States and Germany (accounting for 29. 4% of the visits). The average time spent per visit was 11 min 37 s (range: 47 s-49 min 4 s), and the number of pages/videos viewed per visit was 8. 1 (range: 2-21). The number of UEMS CME modules taken was 675, and 326 (48%) of these tests were passed.
Further examinations are planned in Prague (Czech Republic), Paris (France), and Porto (Portugal) in 2024, 2025, and 2026, respectively. In the midst of the COVID-19 pandemic, and on the back of a successful online learning platform for gynaecological surgery hosted on WebSurg, 2  the online learning university portal of the French Strasbourg-based IRCAD Institute, a number of the authors came up with the plan to set up an online, free to use, video lecture-based learning platform to cover the UEMS knowledge syllabus. It was called "Hernia Basecamp". 3  At its launch in June 2021, over 200 video lectures, and recently published EHS guidelines were grouped into eight topic areas. A number of these lectures and guidelines were selected to create 10 e-learning modules (ELMs) of 3 h maximum for each, with a 10-question multiple choice questionnaire (MCQ) at the end of each module (8 out of 10 correct questions pass rate).
METHODS The Hernia Basecamp WebSurg platform 3 was interrogated using Matomo Analytics in January 2023 (over a 19-month period since launch). Data on the number of visits, pages looked at, and time spent on the platform per visit, along with the number of CMEaccredited ELMs were collected. Data was also collected on the countries and the world regions accessing the site. Data on the number of CME module tests taken, the pass rate, and the country of the examinee were collected.
A module on this area is necessary and is one of the expansion projects planned. The strategic partnership between the EHS and the British Journal of Surgery Society is noted, and the work on the BJS Academy website 5  in this area is area is welcomed to help educate surgeons in this important topic. Readers of this article, and users of the Hernia Basecamp platform are encouraged to offer suggestions for improvement, revision, and indeed contribute new lectures and CME modules as they see necessary. The major sponsors of WebSurg, the online university of IRCAD, are industry partners. And Medtronic, in particular, have funded the Hernia Basecamp set-up and running cost through their unrestricted educational sponsorship. Funding is provided free of any attempt to influence the lectures, subjects for discussion, content, and choice of faculty members.
g. ,  De Vrese & Hagemann, 2018). Alongside all these applications sits also research towards using novel technology instruments (e. g. ,  Doutriaux-Boucher et al. , 1998)  or to revisit early satellite data records (e. g. ,  Poli et al. , 2017). However, climate research presents several distinct challenges when it comes to observation data simulators. First, the time-series covered by climate model and by related satellite-based CDR products are necessarily long. This makes running a full data assimilation system (with underpinning Earth-system models and covering many observation types) an overly computationally-expensive and inadequate venture. This is also partly unnecessary in the face of the efforts already deployed by large modelling centres to create model-gridded global decadal datasets, such as reanalyses, which gradually widen their remits to exploit (and hence simulate) an increasing diversity of satellite-based data records.
, 2018). It must be recalled that both elements, RADSIM and RTTOV, benefit from a long-term support of the EUMETSAT climate services and development plan, with activities distributed between the EUMETSAT central facility and its Satellite Applications Facility (SAF) network, including the NWP-SAF, for these simulators. The results shown in this manuscript build on an implementation of RADSIM and RTTOV in the EUMETSAT infrastructure, with massively parallel computations carried out on a multi-node cluster computing system. In the present study, we use RADSIM interfaced with RTTOV version 13. 0, except for simulating data from the Medium-Resolution Infrared Radiometer (MRIR) where we used RTTOV version 12. 2. Additional details about the radiance simulation configuration are given in Supplement Text S1.
, 2017) , • Special Sensor Microwave Water Vapor Profiler (SSM/T-2), flown on U. S. DMSP satellites  F-11, -12, -14, and -15 (EUMETSAT, 2021). The first 2 instruments are visible and infrared imagers on geostationary satellites, the next 4 are visible and/or infrared imagers or infrared sounders on polar-orbiting satellites, and the last 2 are microwave radiometers on polar-orbiting satellites. Several instruments are historical sensors, given their early data record. While it would take too long to expand all details of these instruments, as well as their detailed configurations, Table  1  provides a summary of some of their key characteristics. Other references, such as the WMO Observing Systems Capability Analysis and Review tool (OSCAR) Space database (https://space. oscar. wmo. int), provide further information for these instruments.
Additional instrument information is given later, as relevant, when presenting the simulation applications. Table  1  indicates if the data records have been used in one way or another in global reanalysis, indicating here the situation only for the data sources assimilated in ERA5  (Hersbach et al. , 2020) , because it is the only reanalysis used for all comparisons. There are several cases of indirect data use in ERA5, as indicated in Table  1. There are only three cases of direct assimilation of the radiance data considered in the present study into ERA5  (Hersbach et al. , 2020)  Taking note of this inter-relation between reanalysis and the radiance data records, the following remediation steps are taken. (  1 ) For MVIRI we only show results before the date when MVIRI radiances started being assimilated in ERA5.
(2) For SEVIRI we do not simply consider departures (differences between observed radiances and simulations), but consider how they vary by changing the simulation setup. (3) For HIRS we do not consider the departures alone but along with AVHRR, and we also exploit the departures at a time-scale for which we believe there is independence between the satellite data record and the reanalysis. Observations with missing geolocation, brightness temperatures, or reflectances (in the case of AVHRR visible and near-infrared channels) are excluded from further analysis. In addition, specific quality controls are applied to the data records of each instrument, using the information available. For completeness, the details are reported in Supplement Text S1.
Class-I applications: Validating an assumption When developing a dataset or an application, it is common to be faced with the issue of validating an assumption used in the methodology. The assumption could, for example, relate to the data themselves, or how to use them. However, a common difficulty is the impracticality of proving the assumption. One can then revert to demonstrating that the assumption is not violated, based on the evidence available. If the results obtained violate the assumptions, then the assumption is proven wrong. If they do not, then the assumption cannot be rejected, and is hence considered to remain valid.
9 K to 1. 4 K, and a mean reduced to near-zero. In summary, the radiative transfer simulations help us validate the assumption that an advanced image quality control should improve exploitation of the MVIRI data record. For this reason, cloud mask validation is important. Such activities are already performed routinely by cloud mask data producers. We show an example of how radiative transfer simulations can further assist in this fashion. To this end, we consider the infra-red and visible data records of two polar-orbiting instruments, the AVHRR and HIRS instruments, operated both on NOAA and EUMETSAT polar-orbiting satellites, and compare with clear-sky radiative transfer simulations. The effects of clouds and aerosols are not included in the radiative transfer simulations considered here.
Considering the maximum observed in reflectance departures by the near-infrared channel, the effect of the eruption was most pronounced over the Tropical Indian Ocean, and was felt in southern latitudes 2 to 3 months later, or in the Mediterranean 5 to 6 months later. The effects of this eruption were analyzed in detail previously  (Stenchikov et al. , 1998). However, the results shown here quantify the relevance of this episode with respect to the HIRS and AVHRR data records. To summarize, this example validates the hypothesis that the CLARA-A3 cloud mask can help to a) filter out cloudy scenes, and b) quantify the radiative effects of the Mount Pinatubo eruption in the HIRS and AVHRR data records, with a separation between regional and temporal variations, at the wavelengths covered by the channels selected here.
This also suggests that further studies of instrument-related departures should provide useful insights into the state of the instrument calibration. In the case of the MRIR it is difficult to go beyond the bias correction models shown in Figure  8  as we lack the low level telemetry data (Level 0 data) that are needed to correct for instrument calibration errors at source. In summary, this example shows that improvements may be made for MRIR to the bias correction models generally used in data assimilation, by considering likely instrument sources of uncertainties. (from July 1987). This particular time period is often looked at to enable inter-calibration of the two instruments' data records (e. g. ,  Dai et al. , 2015). The SMMR instrument collected measurements at five microwave frequencies and two polarizations (vertical and horizontal).
Explained observation variability, example with MSG The SEVIRI instruments on-board MSG satellites extend the data records started by MVIRI instruments on-board MFG satellites for the 3 heritage channels, i. e. , the water vapor, infrared, and visible channels. Furthermore, SEVIRI includes 6 additional channels in the infrared region as compared to MVIRI. When the MFG and MSG satellites are positioned near 0-degree longitude, the field of view of the instruments covers Africa and Europe. Thus, the observed radiances of these satellites allow patterns of variability to be inferred over areas with important societal applications (e. g. ,  Barbosa et al. , 2019; Harrison et al. , 2019). Satellites in geostationary orbit are subject to small displacements around their nominal positions around the Equator.
These satellites are affected by gravity pulls from the Earth and the Moon. This so called three-body system, or Lissajous track, results in figure-of-eight displacements (e. g. ,  Hubert & Swale, 1984). In addition, geostationary satellites may see displacements during their lifetime when the nominal longitude changes. This section shows the importance for climate applications of these displacements (even if seemingly small), through an analysis of subsequent satellite data records that appear as originating from a single longitude position at the Equator. The 15-minute MSG All-Sky Radiances (ASR) products are simulated here only for the two observation times closest to the hour (i. e. , two images per hour are simulated, and two images are not). The quality controls applied selects only pixels that are believed to be free of clouds (so-called Clear-Sky Radiances, CSR), and for which the radiances are computed from an average of at least 10 pixels.
An alternative explanation for these offsets could be varying amounts of humidity biases (over time) in the ERA5 reanalysis. Such small inter-satellite differences are not believed to be a problem for applications of the SSM/T-2 data into reanalysis, which generally applies bias corrections to such data when assimilating them. However, this does require further attention, to enable, for example, direct use of the data to retrieve humidity information, unless applying a priori approaches such as harmonization (e. g. ,  Giering et al. , 2019). Finally, Figure  13  shows that the increase in observation horizontal local variability is associated with a slow but steady decrease of the bias towards negative values in the departures. This effect is most pronounced for the lowest-peaking channel in Figure  13 (c), and consistent with the findings of  Calbet et al.
Discussion There are several factors that could explain the departures between instrument data records and simulations from reanalyses reported and analyzed in this paper. First, there is the issue of data independence. One needs to assess, for each comparison, if the observational data record was assimilated in the reanalysis that is used for the simulations. The data of several data records used in this paper were independent (SI-1, SSM/T-2, MRIR). Other data were partly or indirectly used in the reanalysis. As for example is the case for the MVIRI radiances, which were indirectly assimilated as another variable or derived product (such as atmospheric motion vector, or to construct the sea-ice or sea-surface temperature that was used as forcing in reanalysis). The HIRS data, on the other hand, were fully assimilated.
Third, even if different reanalyses (such as ERA5 and ERA-Interim) are from different generations, they often used very similar observations input (especially in the early years). This limits the degree of independence between comparisons to several reanalyses. For this reason, global reanalyses from a wider diversity of producers should be selected in future work. Fourth, there are instrument-induced effects that are not all understood or simulated. A few of these effects are listed by  Fennig et al. (2017) , for example, for the Nimbus-7 SMMR data record. These effects include unknown variations in the satellite zenith angle, errors in the satellite attitude control, potential errors in the underlying level 1B processing and, more generally, insufficient correction of instrument-induced effects (such as calibration, spill-over, and polarization mixing).
These are all effects that are best addressed at the source, and for which the simulations can help quantifying the overall cumulated effects. Even in cases where instrument errors cannot be corrected at the source, such as the case of the MRIR data record, improvements in bias correction predictors will help in including early satellite data into either data assimilation systems, or at least in its use as a check on aspects of a reanalysis for periods with limited or no satellite data. Fifth, the quality of radiance simulations to reproduce the variability in the observations is not equal for all channels/instruments. This originates in the spatio-temporal scale and magnitude of natural phenomena responsible for the variability, differing by instrument and channel, as compared to instrument and simulation resolutions and uncertainties.