dataesr/scientific-paragraphs-categorization huggingface dataset:
The TILES app uploaded data directly to the research server while the RealizD app uploaded data to the RealizD server and that data was later pulled to the research server. The research server (code available at https://github. com/usc-sail/tiles-data-collection-pipeline/) consists of a RESTful API hosing a series of endpoints to collect push-type data streams (e. g. Owl-in-One, TILES app) in addition to a suite of tasks to fetch pull-type data streams (e. g. Fitbit, OMsignal).
g. if 2 out of 5 annotators use a category, that category has frequency 2/5 = 0. 4). We refer the reader to the README file in the dataset for further details on those categories and how they are reported in the data. Data for the enrollment session baseline survey (part I), take-home baseline survey (part II) and study-completion survey (part II) were stored in single files each. Variables were renamed to correspond to what each question measured. After the above steps were taken, total scores for each psychological measurement were calculated (scored folder in Table  9 ).
A non-zero value corresponds then to a row with detected foreground speech. For the current data release, we further curated the data by only including a subset of the features collected, and omitting filterbank features such as MFCCs and PLPs, as well as LPC features. We believe filterbanks should be released with some form of information obfuscation or encryption, as it contains potentially recoverable language information and poses privacy concerns. We intend to release privacy preserving embeddings on the filterbanks at a later stage. For information on features included in the release, refer to Section Audio. Inference of days at work. For convenience, we also provide an estimate of working days for all participants. This was obtained using the EMAs, as well as the data collected from the OMsignal garments and a combination of the Jelly phones and Owl-in-One data.
Technical Validation Sensor validation. In this section, we give an overview of works in the literature as well as work that we have conducted with the sensor data to validate the results. OMsignal. We have run two studies to validate the data obtained from the OMsignal garments. In  39  , we studied the differences between the heart rate data between the Fitbit and OM garments in the TILES-2018 dataset, where we observed higher correlation between Fitbit and OM garments than previous studies. In  40  , we also compared the accuracy of the Fitbit's PPG-based heart rate measurements against the OMsignal garments' ECG-based heart rate measurements. We extracted several heart rate variability (HRV) features and studied correlations with stress and anxiety. Since the discontinuation of OMsignal garments, several white papers comparing their performance with medical-grade devices were removed from the public domain.
TILES Audio Recorder. We presented an analysis of the audio recorder in  32. TAR primarily extracts the audio features using openSMILE  42. OpenSMILE is a widely used tool for extracting a wide range of features from audio signals. To test the feature distortion from the recording device, a recording setup was proposed in  32  to allow TAR to record speech amplified through a speaker. In this feature degradation experiment, 1000 gender-balanced utterances from the TIMIT  43  database were randomly sampled and concatenated into one file. The audio files were then played through a loud-speaker. Multiple TARs set at distances 15cm, 20cm, 25cm, and 30cm from the speaker extracted the audio features simultaneously. This experiment quantified the feature distortion by measuring the root-mean-squared error (RMSE) and cosine distance between the features extracted from the source file and recorded features.
This can be easily done in Python and R (examples follow in Box 1). Note that. csv. gz files can also be opened directly in LibreOffice Calc (free software alternative to Microsoft Excel: https:// libreoffice. org) without decompression. If you need to decompress the files, we recommend using the command line utility gzip. Data records: Use cases. This data set was initially developed to model and predict self-report mental states from wearable sensors. However, we are devising more uses for it, and hope that researchers will find other uses for various aspects of the data. # In R, using data. table library(data. table) df = fread("file. csv. gz") # or using tidyverse library(tidyverse) read_csv("file. csv. gz") -> df Multimodal signal processing. This data set proposes several problems in core (multimodal) signal processing.
Acknowledgements The research is based upon work supported by the  Office of the Director of National Intelligence (ODNI) ,  Intelligence Advanced Research Projects Activity (IARPA) , via  IARPA  Contract No  2017-17042800005. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the  ODNI ,  IARPA , or the  U. S. Government. The U. S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. We thank  Felipe Osorno  and  Brooke Baldwin-RodrÃguez  from  USC Keck Hospital  for their assistance, and all the volunteers that decided to be part of the study. We thank  Luca Foschini  from  Evidation Health  for his helpful insights during the study design.
INTRODUCTION Researchers in medical imaging now have access to large amounts of data via open source databases  [The Cancer Imaging Archive (TCIA; Dataset, 2020b), Center Traumatic Brain Injury (CTBI; Dataset, 2020a) ]. In parallel of the quantity of images available, the complexity of the medical images post-processing is increasing. Where data scientists applied a single process to their images to obtain valuable results  (Lemasson et al. , 2016) , state of the art analysis requires the execution of very complex interdependent processes called pipelines involving the execution of many operations, referred to as modules, on large databases  (Funck et al. , 2018). Among the modules used (e. g. , bias removal, brain extraction, or image registration) some of them are part of toolboxes well-recognized by the community [such as  SPM (Software, 2020)  or FSL  (Jenkinson et al.
This includes BrainCAT  (Marques et al. , 2013)  which generates diffusion tensor images (DTI) from MRI diffusion weighted images, MRtrix  (Tournier et al. , 2019) , which allows visualization and specific processes on diffusion MRI data, Pydpiper  (Friedel et al. , 2014)  which offers registration algorithms, or FuNP  (Park et al. , 2019)  which gathers some of the most widely used functions available for MRI data in a single post-processing pipeline. In those software, either the type of input data or the processing applied is unalterable. The second category contains software that can apply many different processes on a large type of data. To our knowledge, only few software available in free access belong to this category. One can quote Vaa3D  (Peng et al. , 2010) , able to handle large dataset of images converted in Tagged Image File Format (TIFF) in a few seconds, to display them in 3D, and to define and execute homemade pipelines.
One can also refer to GraphMIC  (Zehner et al. , 2015) , which provides a node based interface for famous image processing libraries and allows to link the modules of these libraries to create complex image processing pipelines without programming, simply by connecting existing modules. This powerful software makes the design of complex pipeline user-friendly as well as the integration of home-made modules. Those powerful software lack of functionalities that we believe will become essential to develop new imaging biomarkers in order to handle and process large heterogeneous databases. Instead of processing one file or subject at a time, one will need a tool able to handle large cohorts, containing heterogeneous (multi-format) and multiparametric data, and to process them with specific complex pipelines and with respect to the intra-subject time dependencies.
Overview MP3, a MATLAB (Software, 2017) toolbox, intends to assist an end-to-end research study, from the loading and the converting of raw images to the statistical analysis through the creation of a database containing metadata and the design and execution of complex analysis pipelines. MP3 is composed of three linked graphical user interfaces (GUI) that stands as its backbone: the converter, the viewer, and the pipeline manager (Figure  1A ). Briefly, those GUI enable the conversion, display, and processing of different medical image formats and architectures (Bruker, DICOM, PAR/REC, NIfTI, BIDS;  Gorgolewski et al. , 2016). The imported data is summarized in a database able to be homogenized, filtered, or improved with metadata such as the name of the patient or the day of the acquisition.
The viewer can display up to four 5D images simultaneously and to draw ROI. Eventually, a graphical interface called Pipeline Manager creates, manages, and executes complex pipelines using editable modules. A pipeline execution system named PSOM (v2. 2. 2)  (Bellec et al. , 2012)  allows to judiciously parallelize the execution of the modules. In this section, we detail the aims and functionalities specific to each of these GUI and present the concept of an MP3 project and its architecture. Detailed information as well as videos presenting latest developments of MP3 are available online (Github: https://github. com/nifm-gin/MP3, YouTube: https://www. youtube. com/playlist?list=PL-Tj6Wc9aE9x7i6s-RLetvNE0isnEsFm7).
It is also possible to import data organized in the Brain Imaging Data Structure (BIDS) convention. Each user can also customize the metadata to be collected from the raw data (Bruker, DICOM, etc. ) that will be stored in the JSON file by editing the provided YAML file. Full details about the customization of the image conversion is available online (https://populse. github. io/mri_conv/Home/index. html). The converter interface (Figure  1B ) can be launched from the main MP3 window. To start a new project, one defines a folder to store the project. Then, thanks to the different parts of the converter (data browser, parameters, images list, etc. ), one can easily navigate through complex and often human unreadable medical image architecture. When the desired images are found, they will be automatically converted to NIfTI/JSON and metadata such as timepoints and subjects names may be modified.
Project Architecture On the hard drive, the main directory of each project, sorted in multiple sub-folders, contains all the data needed to open the project via MP3. We defined several types of files, among which Scans, classical medical images, described as a NIfTI and a JSON file, and ROI, described as a NIfTI file (containing a binary matrix). Each imported scan file is stored in the "Raw_data" sub-folder, while the processed files, written after a pipeline execution, are saved in the "Derived_data" sub-folder and the ROI in "ROI_data. " Other sub-folders can be part of the project, such as for instance "Tmp, " where lie temporary files, or "Saved_Pipelines, " that stores the. mat files that summarize the designed pipelines (see section 2. 5). To manage these subfolders, files, and associated metadata, MP3 relies on a database containing, for each entry, information referred to as tags, as its Type-Tag (Scan or ROI), Subject-Tag, Timepoint-Tag, or Group-Tag associated, as well as its Path-Tag or Filename-Tag.
Since MP3 is developed on MATLAB, we decided not to use a real database, as we could have developed in SQL language, but to use a MATLAB table object. The power of these variables relies on their ability to be quickly and easily filtered, which is a key operation in MP3. Each entry is linked to the corresponding files through this table. It is therefore this database that undergoes operations as renaming, sorting, or filtering to select part of our available data or to homogenize the database to make it more consistent. Since a project is completely described by its database, MP3 offers the possibility of easily transferring a whole project to another user. To save space and to easily share any project, MP3 handles the compressed NIfTI format. nii. gz.
Viewer The Viewer is a MATLAB GUI that is divided into three parts (Figure  1C ): a database manager, to display and manage the database, the image display, that displays up to four 5D images from the database and supports the drawing of ROIs, and the statistics display, which displays scatter plots, curves, histograms, or first order statistics, like mean or standard deviation within a ROI (Video 2: https://www. youtube. com/watch?v=X26RV7VmXTA&list=PL-Tj6Wc9aE9x7i6s-RLetvNE0isnEsFm7).
Database Manager The database (section 2. 3) is presented as four listboxes each displaying a tag (Subject-Tag, Timepoint-Tag, Sequence_Name-Tag, and Filename-Tag) and working as filters that reduce the database. Thereby, when the Subject-Tag list displays all the different subjects of the database, selecting one of them filters the database and reduces the files displayed in the other lists. This way of presenting the database is inspired by the BIDS architecture and is a simple and efficient way to quickly access a specific scan. One can switch from the Scans list to the ROIs list by just a click on the specific button above the Sequence_Name-Tag list. These lists allow renaming the value of a tag, copying, or deleting an entry in the database thanks to specific rightclick menus associated with each listbox.
Image Display One can load and display up to four 5D scans simultaneously. One of the main advantages of the NIfTI files is their transformation matrix stored in the header (section 2. 2). This matrix locates the center of the scanner and its geometry. It is therefore possible, thanks to a light interpolation provided by functions of the SPM toolbox (Software, 2020), to display different orientations and different resolutions. This ability is used to open the selected scans in a selected referential. One can change the selected referential thanks to a popup menu displayed on the top of the figures. Open different scans in the same referential is particularly useful when displaying two scans acquired with a different field of view or a different voxel size. Three push buttons above the images can be used to change the referential orientation among axial, sagittal, or coronal.
Principal of the Pipeline Manager Thanks to our structured project-data conversion (section 2. 2) and database (section 2. 3), the pipeline manager is prepared to manipulate our data. The pipeline manager allows iterations of a given process over the whole database (or a sub-part of the database). For instance, we are able to create a process that computes an image B from one specific image A and to apply it on each image A of the database. This process is called pipeline, because it can be a simple computation or a complex sequence of independent steps.
Pipeline Definition As illustrated on Figure  2A , a pipeline is composed of one or several modules. A module is a function, more or less complex, that is designed to compute one or more NIfTI files thanks to one or more other NIfTI files, and some parameters. For instance, a module called Module_Smooth takes as input a Sequence_Name-Tag and 3 parameters (the dimension of the filter-2D or 3D Gaussian, the size of the filter-the variance of the Gaussian, and the Extension string of the output files), and generates a smoothed image. A module shall contain a basic operation, although nothing prevent a module from containing a whole process. A module applied to a database creates a certain number of jobs, i. e. , a certain number of occurrences of that module. Therefore, since a pipeline is simply a sequence of modules linked together, applying a pipeline to a database creates a certain number of jobs for each module.
Procedure to Create a Pipeline Now that the concept of pipeline is introduced, we share the procedure to create one (Figure  2B ). First, one needs to select the part of the data on which to execute the pipeline. Indeed, MP3 allows the user to apply a pipeline to the entire database or only on a sub-part of it (i. e. , the 10 first subjects, or on a specific timepoint). This operation is managed by the "database filter" on the upper left corner (Figure  1D ). This part of the GUI contains manual or predefined filters as well as a table that shows the filtered database. After the data selection, one can go to the module creator (Figure  1D ) and choose a module among all the ones currently available (section 2. 6. 1). Selecting a module displays all its required parameters. For instance, selecting the Module_Smooth shows a line aimed to select a Sequence_Name-Tag value, another aimed to set a filter size, etc.
After the modification of all parameters (which can also be left to their default value), one can add the module to the pipeline by clicking on the button add module to a pipeline. When one adds a module to a pipeline, the module automatically creates as many jobs as necessary (depending on the database). Each output Scan or ROI of each job is then added to the filtered database displayed on the upper left. Since the pipeline has not been executed yet, all those files do not exist yet but are already accessible for the user in the displayed database. This offers the possibility to parameterize another module with the tags of those virtual entries, and therefore link some modules together. The operation of filtering the database, selecting a module, parameterizing it, and adding it to the pipeline, which creates jobs (Figure  2B ), can be reproduced an unlimited number of times.
Saving and Sharing In order to share a pipeline between several users, computers or projects, MP3 offers the option to save and load pipelines. Since a pipeline is a list of modules, we just store in a. mat file the sequence of modules with their parameters, but we do not store the jobs, since they are specific to a database. Loading a pipeline consists of applying each module to the new database. If the tag values of the pipeline to load are not consistent with the database, a module cannot generate the jobs. The module name is then displayed in red in the pipeline explorer and one just needs to edit the affected module, select the adapted tag value and save it to make it compliant, which turns the entry from red to green. This color convention intends to make clear the content of a pipeline as well as the consequences of its execution.
Execution of a Pipeline Once a pipeline is well-designed, there are two execution ways in MP3. The first way is the Single Processing. Each job is executed in the module order: all the jobs of the first module, then all the jobs of the second, etc. This quick execution, without strong dependencies between the modules and a low computing time, is especially useful when testing or developing a module. The second way is to use a pipeline execution system called "PSOM" (v2. 2. 2)  (Bellec et al. , 2012)  by selecting the option Multi Processing. This powerful system enables the execution of a pipeline on several cores. The dependencies between jobs are taken into account and the jobs are then distributed upon each core. This system also offers a garbage collector in order to save Random Access Memory and a way to monitor the number of jobs launched in parallel.
As it has an influence on the computer's behavior during the execution of the pipeline, one can set this number of Workers before the execution. During the execution, all output files are written in a temporary folder. At the end, each output file of a successful job is saved in its data folder according to its type (Derived_data or ROI_data). In order to help navigating in the database (cf. Database filter section), we defined a color code based on each scan type. The blue represents raw data, derived data are in green, while the pink means ROI data, and the yellow colorizes the virtual files (output files of not executed pipeline yet).
Modules The power of the pipeline manager lies in its modules. They are the basic operations needed to design complex pipelines. We adapted the way to define modules exposed in PSOM (v2. 2. 2)  (Bellec et al. , 2012). A module is then a MATLAB function, stored in a. m file whose name begins by the string "Module_. " All those functions lie in a folder called Modules of the MP3 source code and are sorted by area, with all the modules of an area in a folder. On launch, the pipeline manager reads the Module folder and its sub folders to list every available module. Thereby, the addition or deletion of a file in the subfolders of this repository updates the module list when launching the pipeline manager.
Provided Modules MP3 is provided along with 12 modules, performing basic operations, such as smooth, threshold, or mask an image with a ROI, compute a brain extraction module  (Chou et al. , 2011) , export a. csv file containing first order statistics or an HTML report of images and ROI. We also provide modules used in the preclinical pipeline exposed section 3. 1, which are MRI oriented, and a module able to delete files of the project's database. Finally, a module interfacing the famous toolbox SPM (Software, 2020) allows reslicing images to match the referential of a reference image.
Template Other users must be able to create their own modules. To facilitate the development of a new module, a module named Module_Template is available. It is extensively commented and explains how a module works. One then has to complete and adapt it to the wanted behavior and to save the new. m file in the Module folder to rapidly apply a new module on a database. To create a new module, even if it is simply an interface with another language, one need to know the basics of MATLAB programming.
Module Repository The homemade modules are aimed to gather the laboratory knowledge and know how. It is also a way to avoid loss of skill due to the departure of a team member. For example, the development team wrote more than 60 modules that match our needs of image processing. All those modules are available at https://github. com/nifm-gin/MP3_User_Modules_Repository. One can find MRI oriented modules, interfaces to others toolboxes functions, such as the co-registration modules from Software (2020) or  Jenkinson et al. (2012) , or more advanced processing, such as clustering, texture analysis, MR fingerprinting analysis, quantitative MR relaxometry, MR perfusion analysis (cerebral blood flow, cerebral blood volume, vessels size, vessels permeability, etc. ).
Preclinical Application A research study published in 2017 compared the blood-barrier permeability changes induced by synchrotron microbeam or uniform radiation therapy. Eighteen rats bearing intracranial tumors were treated and imaged by multi-modal MRI using the Grenoble MRI facility IRMaGE  (Bouchet et al. , 2017). All procedures related to animal care conformed to the Guidelines of the French Government with licenses 380325 and 380321 (authorized lab A3818510002 and A3851610004). Part of these data are available online as an example of a MP3 project  (Dataset, 2019). This project contains data of two rats among the 18 ones each imaged at three timepoints. Each timepoint contains 10 scans corresponding to the data described in  Bouchet et al. (2017) , a ROI delineating the brain, and some masked scans obtained thanks to the former ROI.