dataesr/scientific-paragraphs-categorization huggingface dataset:
1: ENSURE EASE OF USE Avoid software that requires local installation or maintenance Minimize need for technical resources or personnel Enable users to work in desired data environments (SAS, Stata, CSV/Excel, R, SPSS) Provide intuitive, easy-to-use interface Generate automatic, customizable reports REQUEST CATEGORY 2: SIMPLIFY DATA HARMONIZATION Flag variables, formats, and codes that are not consistent with the data model Check for plausible numeric values and dates, date order logic, and agreement between related variables (e. g. , an end date and a reason for ending) Offer data quality results interactively and in downloadable spreadsheets Provide an error spreadsheet that includes all details necessary to locate records with data quality issues Keep data quality checks up to date with the current version of the common data model REQUEST CATEGORY 3: PROVIDE WORKFLOW SUPPORT Remind users of details of tables and variables requested for a selected study Prioritize dataset security and privacy (no long-term data storage) Transfer files submitted for multiregional studies using a secure approach Generate dataset summaries and data quality reports automatically when datasets are transferred, to provide a history of dataset submissions.
Funding The authors received no specific funding for this research. A small amount of discretionary funding provided by the PI's institution paid for pilot study participants and for honoraria to organizations that assisted with data collection in several locations. These were provided by  Columbia University Undergraduate Global Engagement  and the  Department of Health Policy and Management. No other funding was provided; all collaborators did so in a voluntary capacity. Thiago  Otto Columbia University, New York, United States Riinu Pae University College London, London, United Kingdom Tsvetelina Panchelieva IPHS -Bulgarian Academy of Sciences, Sofia, Bulgaria Sun Young Park Columbia University, New York, United States Daria Stefania Pascu University of Padua, Padua, Italy Irena Pavlović University of Belgrade, Belgrade, Serbia Marija B.
Analysis of Example Dataset The example data set was created for this special issue by simulation. The details of the simulations were unknown to the analyst for the current study. The dataset contains 2000 subjects' responses to 12 hypothetical items from the Simulated Disease Questionnaire (SDQ-12). Each item has 4 response categories. Items were originally scored such that high scores indicate poor health; we reversecoded the original scores to be consistent with our discussions, so high scores indicate good health. Scores are available for four time points: baseline and follow-up 1, follow-up 2, and follow-up 3. Additionally, responses to a seven category TR item are available at follow-up 1, 2, and 3. Our analyses established an MWIC threshold for changes from baseline to follow-up 1, 2, and 3, respectively.
I. The Experiment The data were collected in a behavioral experiment in which participants were presented with a series of mini stock markets. Each participant observed the graph of a market price as it gradually changed in time in 0. 8-second intervals and had to decide when to sell an "asset" (see Figure  1 ). For the first 15 periods, participants could only observe the price. 1 Then, in period 15, they were forced to buy an asset at the current price. The point of entry was marked with a vertical red line. The market price kept changing until participants decided to sell the asset (marked with a blue line on the graph). In case no selling decision was made, the market continued until its closure in period 50, at which point participants were forced to sell. The profit was equal to the selling price minus the entry price (price in period 15) so that participants could actually lose money (each participant received a € 10 fee that covered her in the case of a loss).
A. Estimation of the Conditional Choice Probabilities The conditional probability of selling the asset (or continuing) at period t is computed directly from the data. We exclude periods 15 and 50 since no one sold the asset in the former (first choice period) and the choice is forced in the latter (last period). Participants sell their asset in different periods, resulting in a highly unbalanced dataset. The CCPs are constructed using a logit estimator of the choices of the active participants in each period t ∈ {16, …, 49} as a function of the realized state variables. It is important to stress that there are two policy functions to be estimated for each period since the experiment has two conditions. The CCP for either the No Info or Info condition in period t can be represented as follows: (5) Pr { d t = 0 | x t } = Λ ( β 1t y t + β 2t s p,t + β 3t s f,t ) , ∀ t ∈ { 16, …, 49 } , where Λ( ⋅ ) stands for the logistic distribution.
was funded by the  European Research Council  ( ERC Consolidator Grant   617629 ). Michele Fioretti  gratefully acknowledges  USC Dornsife INET  for funding and support during the first stage of this project. Computation for the work described in this paper was supported by the  University of Southern California Center for High-Performance Computing  (hpc. usc. edu). All mistakes are our own. The IRB approval for the data collection was provided by  University of Trento (Department of Economics).
We further ran cross-species analyses for the proportion of optimal responses in conditions with skewed reward distributions. We used the baboon data from the first two blocks of Experiment 3 and the human data from Experiment 1B (which contains 480 trials, and allows us to straightforwardly compare two blocks of 240 trials across-species). We ran a logistic mixed-effects model with number of alternatives (binary or ternary tasks), testing block (centered), and species (humans or baboons) as fixed effects with interaction terms; as random effects, we included by-participant random intercepts and by-participant slopes for the effects of block and number of alternatives. The categorical effect of Species was simple coded so levels are compared to each other directly (humans vs baboons, where the reference level were baboons) but the intercept is the grand mean across levels.
Acknowledgements This work was funded by the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation programme  (Grant agreement  681942 , held by KS). Experiments with humans were further aided by the  NCCR Evolving Language ,  Swiss National Science Foundation  (Agreement Nr. 51NF40_180888 ). Experiments with baboons were also supported by the  Agence Nationale de la Recherche grants LabEx Brain & Language Research Institute  ( ANR-11-LABX-0036 ) and  LabEx Institute of Language, Communication  and the  Brain  ( ANR-16-CONV-0002 ). We would like to thank the developers of the opensource software used for this study:  Josh de Leeuw  and collaborators for the development of jsPsych 63 , and  Vanessa Sochat  for the curation of The Experiment Factory 66.
Data accessibility All data reported are available at osf. io/ qnm57/, as well as the preregistered designs and analysis plans  53. The preregistered design and analysis plan for Experiment 1 and 2 is accessible at osf. io/ b3nke; the preregistered design and analysis plan for Experiment 1B is accessible at osf. io/ ky7z8; and the preregistered design and analysis plan for Experiment 3 and 4 is accessible at osf. io/ evxk4. The analysis scripts used for the analyses and figures reported are also available at osf. io/ qnm57/, as well as all the materials used for the human experiments. Human experiments were coded using the jsPsych JavaScript library  63. Experiments with baboons were coded using a custom software developed by Joël Fagot. All data were analysed using lme4 and the tidyverse collection of R packages  64, 65.
Obtaining standardized benchmark of computational methods is a major issue in data science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here we introduce Codabench, a meta-benchmark platform that is open-sourced and community driven for benchmarking algorithms or software agents versus datasets or tasks. A public instance of Codabench https://www. codabench. org/ is open to everyone, free of charge, and allows benchmark organizers to compare fairly submissions, under the same setting (software, hardware, data, algorithms), with custom protocols and data formats. Codabench has unique features facilitating easy organization of flexible and reproducible benchmarks, such as the possibility of re-using templates of benchmarks, and supplying compute resources on-demand.
Introduction The methodology of unbiased algorithm evaluation is crucial for machine learning, and has recently received renewed attention in all data science scientific communities. Often, researchers have difficulties understanding which dataset to choose for a fair evaluation, with which metrics, under which software/hardware configurations, and on which platforms. The concept of benchmark itself is not well standardized and includes many settings. For instance, the following may be referred to as a benchmark: a set of datasets; a set of artificial tasks; a set of algorithms; one or several dataset(s) coupled with reference baseline algorithms; a package for fast prototyping algorithms for a specific task; a hub for compilation of related algorithm implementations. In addition, many benchmarks often integrate new progresses by manual verification instead of automatic submission and execution, which delays the benchmark update and requires extra human efforts.
Typical examples of existing frameworks addressing such needs are inventoried in Table  1 , including competition platforms, repository hubs and domain specific benchmarks. Firstly, competition platforms focus on the participants and A meta-benchmark platform has three types of contributors: platform developers (in yellow), benchmark organizers (in green) and benchmark participants (in red). Codabench is at the meta level to support diverse benchmarks. Each benchmark is implemented by a benchmark bundle that contains one or more tasks. provide limited support for organizing general tasks. Famous platforms like Kaggle 2  , Tianchi  3  , CodaLab  4  organize many data science challenges attracting a large number of participants. However, the platform providers retain some control: the organizers do not have full flexibility and control over their competitions.
Thus, the experience for organizers are not enjoyable. A comparison between competitions and benchmarks is given in Table  S1  in the supplementary. Secondly, repository hubs such as UCI repository  5  , OpenML  19  and PapersWithCode  6  , also play an important role for benchmarks and research. They collect large amount of datasets, methods, and results from academic papers, but reproducibility by running code in given containers (or similar ways) is not guaranteed. Besides the above-mentioned platforms, many domain specific benchmarks exist, e. g. DAWNBench  5  , KITTI Benchmark Suite  8. These benchmarks usually focus on a couple of closely related tasks, but are not designed to host general benchmarks. In addition, they require repetitive efforts to develop and maintain, which is not always affordable by data science teams.
Thus, to facilitate benchmarking, we need a platform to allow users to flexibly and easily create benchmarks with custom evaluation protocols and custom data formats, and execution in a controlled reproducible environment, which is totally free and open-sourced. To answer these unmet needs, we developed Codabench, a meta-benchmark platform (Figure  1 ). A meta-benchmark platform is designed to support general purpose benchmarks and to facilitate the organization and usage of benchmarks. (3) and public information. An intermediate block also exists for information exchange of time budget, scoring, input data, ground truth data and predictions. Red bottom right block: participant prepares a submission "z" uploaded to the platform. The submission is then executed by the ingestion program. The role of the scoring program is to produce scores that are then displayed on the leaderboard.
Method: Design of Codabench Codabench is a meta-benchmark platform to provide flexible, easy-to-use and reproducible benchmarking service that is publicly and freely available for everyone. In Codabench, benchmarks are implemented by benchmark bundles which contain one or several tasks. The concept of a task is newly introduced, which is the minimal unit for composing a benchmark (bundle). A task consists of an "ingestion module" (including an ingestion program and input data), a "scoring module" (including a scoring program and reference data, invisible to the participant's submission), a baseline solution with sample data, and meta-data information if needed. Tasks in Codabench may be programmed in any programming language in any custom way, which are run in a docker specified by organizers.
The reader is referred to Codabench official repository  8  , where the code and complete documentation are found. In the supplementary, we also include instructions and references to get started. To use the public instance of Codabench, please visit the Codabench website. To test and install locally, the instructions are given in the README file of the official repository. The Codabench code is released under an Apache 2. 0 License. ✘ ✔ ✘ ✔ ✘ ✔ ✔ ✔ Tianchi ✘ ✔ ✘ ✔ ✘ ✘ ✔ ✔ CodaLab ✔ ✔ ✘ ✔ ✔ ✘ ✔ ✔ UCI ✘ ✘ ✔ ✘ ✔ ✘ ✘ ✔ OpenML ✘ ✔ ✔ ✔ ✔ ✔ ✘ ✔ PapersWithCode ✘ ✔ ✘ ✔ ✔ ✘ ✘ ✔ DAWNBench ✘ ✔ ✘ ✘ ✔ ✘ ✘ ✔ Codabench ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 3 Results.
Key features of Codabench Codabench is task-oriented. Using tasks, the organizers have the flexibility of implementing any benchmark protocol, with any dataset format and API, or even using data generating models, allowing them to organize reinforcement learning challenges. In this section, we introduce the key features of Codabench contributing to the flexibility, easiness and reproducibility, as shown in Figure  1. Codabench also supports custom leaderboards and has full documentation of usage.
Flexibility Codabench supports flexible benchmark types including results submission, code submission and even dataset submission. Benchmarks on Codabench are organized by bundles containing all the information of a benchmark. Bundle (hosting a benchmark). A benchmark bundle is a zip file containing all necessary constituents of a benchmark: tasks, documentation, and configuration settings (such as leaderboard settings). A Codabench bundle may include single or multiple tasks. Classical benchmark is usually single-task while AutoML 11 , Transfer Learning 14 , Meta Learning 20 benchmarks are multi-task. Results or code submission. "Classic" Codabench benchmarks are either with result or code submission. On one hand, result submissions are used when organizers wish that participants use they own computational resources.
In supervised learning competitions, participants would supply e. g. predictions of output values on some test datasets. Other types of results may be supplied, for instance high resolution images in a hyper-resolution challenge for which inputs are low-resolution images. On the other hand, if the organizers wish to run all algorithms in a uniform manner on the platform, Codabench allows the participants to make code submissions. The submitted software is run in a docker supplied by the organizers, either on the default compute worker, or on compute workers supplied by the organizers. This code submission design allows organizers to provide suitable computational resources (e. g. GPUs), and improve reproducibility. Dataset submission. The role of dataset and algorithm can be transposed with Codabench to facilitate Data-Centric AI  9  which is a trending research topic that cares about the quality and usage of data.
In a "classic" benchmark, organizers provide datasets and participants submit algorithms. In a transposed benchmark, participants submit datasets and organizers provide reference algorithms. A "classic" benchmark will have a leaderboard with datasets in columns, growing by adding more lines as algorithm submissions are made. In a transposed dataset submission benchmark, the leaderboard will have algorithms in columns and lines are added as more datasets are submitted. With Codabench's transposed benchmark, it is easier to try different data augmentation and processing methods with fixed algorithms as test cases.
Reproducibility Codabench makes extensive use of Dockers 10  to guarantee reproducibility. Benchmark organizers specify the Docker image by providing its Docker Hub name and tag. Docker wraps all the software dependencies into a lightweight virtual image. Once a docker is provided by the benchmark organizer, the program can be run inside a docker which contains exactly the same installed packages. This docker will be pulled every time a benchmark's program is executed. Different benchmarks could use different dockers, which are usually provided by organizers. We also provide a default docker for more general benchmarks' usage or people who are not familiar with dockers.
Use case 1: AutoGraph benchmark In this section, we introduce Automated Graph Machine Learning (AutoGraph) benchmark, which targets at automated node classification methods on diverse dataset scenarios. With this use case, we show a set of fundamental features of Codabench: (1) the code submission mode (2) reproducibility guaranteed by docker (3) flexible benchmark bundle configuration with multiple tasks, and (4) customizable computational resources. Background. Graph Machine Learning has been a very hot topic due to ubiquity of graph-structured data, e. g. social network  9  , molecule graph  13  , knowledge graph 3 , etc. Typical tasks of graph data include node-level (node classification), edge-level (link prediction) and graph-level (graph regression/classification). The task of our benchmark here is node  classification, i.
e. given a graph where some nodes are labeled and the rest are unlabeled, we want to predict the classes of the unlabeled nodes. In addition, we require the algorithm to perform well on a set of datasets instead of just one dataset. This leads to Automated Graph Machine Learning problem which we call AutoGraph. Implementation. The AutoGraph benchmark is a typical code submission use case. It focuses on Automated Machine Learning (AutoML) methods  11  which requires more than one dataset to be evaluated together. Codabench bundle is by design flexible with multiple tasks each of which contains a seperate dataset. We also provide a docker hosted on DockerHub, which could be pulled automatically by Codabench platform to run each algorithm submission and be used for researchers' local development.
Every time a new method is uploaded, a new docker container instance will be called to independently run the method for each dataset. In this way, we make sure every algorithm is fairly run under the same setting and the whole process can be fully reproduced on other machines. Codabench is designed to adapt to any Docker-enabled computational resource (local machine, cluster server, cloud machines, etc. ). We currently host the AutoGraph benchmark on Codabench with free computational resources thanks to Google's sponsorship, encouraging everyone to contribute. Besides, the datasets are also available to the public for local usage and further benchmarking on GitHub  12  and Kaggle. We uploaded the solutions of the winners of the challenge as baselines. Since the benchmark datasets are already released, users can also run complementary experiments on their local computers and debug mode easily, thus more rapidly making progress.
Use case 2: DECONbench benchmark In this section, we introduce the DECONbench benchmark  7. DECONbench aims at benchmarking algorithms inferring tumor cellular composition from molecular data. Here we highlight two features of Codabench: (1) flexibility of benchmark bundle (in this use case, another task and programming language R supported) (2) reusability and portability of benchmark bundles Background. Successful treatment of cancer is still a challenge and this is partly due to a wide heterogeneity of tumor cellular composition across patient population. Tumors are made up of cells with different identities and origins. Cancer cells evolving in a dynamic environment consist of aberrant non-cancerous cells, such as blood vessels or immune cells. Tumor cellular composition is difficult to observe and quantify, as it is hidden inside the bulk molecular profiles of the samples, with millions of cells present in the tumor (and not only cancer cells) contributing to the bulk recorded signals.
Taking advantage of the large amount of molecular data publicly available, a wide number of supervised and unsupervised algorithms have been recently developed to estimate tumor cellular composition 4;1;6. DECONbench is a series of benchmarks dedicated to the quantification of tumor composition, focusing on estimating cell-types and proportion in biological samples using multimodal molecular data. Participants have to identify an estimation of the tumor composition, i. e. a matrix of estimated proportion of each deconvoluted cell-type (rows) for each sample (columns). The discriminating metric is mean absolute error (MAE) between prediction and ground truth matrix. Note that DECONbench series is optimized to run methods developed in the statistical programming language R. Implementation. Using the Codabench platform, the COMETH consortium firstly developed a benchmark for continuous evaluation of computational methods based on epigenomic data  13.
Use case 3: COMETH benchmark In this section, we introduce the COMETH benchmark, motivated by real clinical application and it is an exciting step towards Data-Centric AI. With this use case, we show that (1) Codabench supports a transposed benchmark consolidating Data-Centric AI (2) the provided API interaction opens a window for other customization scenarios. Background. When it comes to clinical application, it is often necessary for health data scientists and clinicians to identify the most suitable existing method to be applied on a given dataset. In this case, we focus more on the data instead of algorithmic development, which aligns with Data-Centric AI. Usually, the clinicians do not (need to) know much about the algorithm details. Instead, they have access to newly available data and want to apply the most relevant algorithms on their new data.
There is thus a need to provide an effective tool displaying the evaluation of state-of-the-art algorithms on reference dataset, and enabling their application on new dataset. This will guide and facilitate the appropriate use of these algorithms by non-expert clinicians. Note that these algorithms are usually provided by benchmark organizers who are domain experts on certain tasks. Implementation. To solve this question, the COMETH consortium developed the COMETH benchmark  17  , a transposed challenge in which datasets should be submitted to be evaluated against existing algorithms (i. e. tasks in the Codabench design). For instance, the COMETH benchmark provides a series of recent deconvolution algorithms that are able to quantify tumor heterogeneity 4;1;6. Clinicians aiming to quantify tumor heterogeneity from molecular data can submit their dataset of interest to the COMETH benchmark and retrieve the corresponding outputs, in a fully reproducible environment.