dataesr/scientific-paragraphs-categorization huggingface dataset:
Data managers who log in to the IeDEA Hub are able to transfer datasets in response to data requests. Submitted datasets are stored in Amazon S3 buckets, to be retrieved by specified data downloaders after multi-factor authentication. All data transactions, including downloads, uploads, and deletions, are logged in the IeDEA Hub. Uploaded datasets are automatically deleted from Amazon S3 buckets after 30 days. Our Amazon Web Services contract includes a Business Associate Addendum to ensure that protected health information is safeguarded. IeDEA regional data centers ensure compliance with all data sharing and data de-identification regulations. The Harmonist team operated under established institutional data use agreements as the Harmonist project was developed by investigators in the IeDEA Latin America region.
Quantifying Toolkit usage and dataset errors In order to ascertain Toolkit acceptance by users, learn which features of the Toolkit were used most often, and track the number and types of errors in user datasets over time, we created a REDCap project that creates a new record for specific actions performed by Toolkit users. Each record documents the type of action chosen by the user and the timestamp of the action. For example, when a user uploads a dataset, the names of the tables and variables are stored in the REDCap record. When data quality checks have been completed, the REDCap record includes a summary of the types of errors found in the dataset. For users who accessed the Toolkit directly by visiting the URL, the records are anonymous and are not linked to a user or region. When users have entered the Toolkit by logging in to the IeDEA Data Hub and selecting a data request, the tracking records identify the user, their region, and the data request number.
Determining Toolkit impact on dataset quality In many cases, users uploaded and revised datasets multiple times for a single data request before ultimately submitting the files. This provided an opportunity to compare errors before and after a user viewed Toolkit data quality results, since the inclusion/exclusion criteria, tables, and variables were constant. To observe changes in data quality after repeated sessions with the Toolkit, we selected only those sets of multiple uploads by a single user for a specific data request. We tracked the number and types of errors in each upload/data quality check cycle for each of these groups of uploads, from the initial to the final data quality check before dataset transfer. Using the error summaries on these sets of datasets that were uploaded and revised, we wrote an additional R script to determine which types of errors occur most frequently in datasets and which types of errors are most commonly corrected.
Software requirements/constraints In their responses to the initial Needs Assessment survey, IeDEA data managers who prepare and send datasets for IeDEA multiregional studies (n = 9) identified multiple challenges that they encounter when responding to IeDEA multiregional data requests: mapping datasets to the IeDEA common data model; cleaning data; finding a secure way to send datasets, confirm receipt, and later track what was sent; responding to questions from the investigator or data manager who received the dataset; and managing resubmissions of datasets due to feedback from the requesting investigator. IeDEA data managers who also receive and merge datasets for IeDEA multiregional studies (n = 6) reported the following challenges in their survey responses: receiving datasets with nonstandard variable names and codes; datasets with missing data; late data submissions; implausible dates in datasets; and the lack of a consistent, secure method of receiving datasets from other regions.
Data quality checks in Toolkit The data quality checks are conducted on each dataset, table, and variable, according to the details stored in REDCap, as described in Section 2. 2. As soon as a new table or variable is added to the data model and defined in REDCap, any uploaded dataset that includes that table or variable will automatically be checked for duplicate records, correct variable format, completeness, valid codes if applicable, plausible values for numeric and date variables, and correct date order logic. Similarly, when a new code is added to the data model, it is automatically included as a permissible value for all variables associated in REDCap with that code list. When a data quality issue is detected in the dataset, a new entry is added to the Toolkit's list of dataset errors.
Each entry in the list includes the table name(s) and primary key values needed for a data manager to locate the record(s) related to the data quality issue. Each error is tagged with a severity level: Critical, Error, or Warning. Critical errors are those that present significant problems for dataset analysis and integrity, such as missing or duplicate patient IDs in tables with patient ID as the sole primary key. Examples of IeDEA data quality checks and corresponding errors are shown in Table  2. (See the Appendix for further descriptions of Harmonist data quality checks. ). Additional data quality information is presented in the form of visualizations in the Toolkit reports. Data managers and investigators can assess temporal plausibility and completeness issues by reviewing histograms of the number of observations by date (Fig.
Redcap Testing/Training survey results In November 2018, following the release of Version 1 of the Toolkit, the nine primary IeDEA regional data managers representing all seven IeDEA regions were invited to complete a series of assigned tasks and submit responses to Toolkit Survey #1. All nine data managers  participated and identified themselves in the survey. The ability to download error detail spreadsheets was ranked as the most useful Toolkit feature, followed by interactive review of data quality results and automatically generated reports. In response to open-ended questions about the Toolkit, users suggested additional features, including customizing report names and enabling users to download separate error spreadsheets for each patient group and/or data table. These features were incorporated into Version 2 of the Toolkit.
All respondents indicated that they envisioned the Toolkit would save time and improve data quality. To introduce Version 2 of the Toolkit, test its behavior, and encourage its adoption, we conducted a comprehensive Testing/ Training Activity followed by Toolkit Survey #2 in June 2019. Seventeen data-related personnel from all IeDEA regional data centers were emailed step-by-step instructions, a sample dataset, and a REDCap survey link. All 17 participated in the training activity and 14 of the 17 (82%) completed the anonymous feedback survey. When asked to report their comfort on a scale of 0-100 (100 = "Completely comfortable") with using the Toolkit to upload, check, and share datasets for IeDEA multiregional studies, the mean response was 92 (min = 75, max = 100, n = 14). In addition, 93% (13) of respondents answered "No" to the question: "Were any of the [Toolkit] steps confusing?" In this REDCap survey, we also asked users to submit questions that remained unanswered after the exercise and to list features of the Toolkit they would like to understand better.
Application workflow We listened to feedback from users throughout the design process and modified the user interface and functionality as requested. This included implementing the Harmonist Data Toolkit as a web-based application, eliminating the need for local installation or maintenance of software. The resulting application consists of 4 main steps (Fig. 6 ): (1) Upload files, in the user's preferred data format, (2) Check data, using the common data model definition in REDCap as an up-to-date guide, (3) Create reports, to download and share, and (4) Submit data to the investigator requesting the data. Step 4 is available to users who have logged in to the IeDEA Hub and selected an active data request prior to Step 1. Step 1: Upload data. The Toolkit prompts users to browse and select the file(s) containing the dataset.
If a user has accessed the Toolkit through the IeDEA Hub in response to a multiregional data request, the Toolkit reminds the user which tables are requested for this study. A link to the complete data request is available for more details about study inclusion and exclusion criteria. Data managers may upload a single ZIP file or select multiple individual files, whose names must match the table names of the IeDEA Data Exchange Standard. The files can be in the user's preferred format (SAS, Stata, CSV, or SPSS). At a minimum, the dataset must include the patient index table with one record per patient and each table in the dataset must include the primary key variables. Each patient ID in the patient  index table must be unique. If these conditions are met, the user can review and confirm the list of IeDEA tables and variables that were detected in the selected files.
Non-IeDEA files may be uploaded, such as text or PDF files that convey important information about the dataset to the investigator. These files will be ignored by the data quality checks but will be available to authenticated designated data recipients if the dataset is ultimately submitted to secure cloud storage for a multiregional data request. Before users continue to Step 2, they are notified of any missing requested tables and variables (if logged in to a specific data request) and encouraged to revise and upload again. Step 2: Check data. As the dataset is checked for conformance, plausibility, and completeness, the application displays the progress through the data quality checks. Once all data quality checks are complete, a summary of the results for each IeDEA table are displayed in an interactive webpage (Fig.
The overall report includes descriptive statistics of the dataset, a summary of data quality checks, histograms of important dates, and heat maps summarizing data completeness (Fig. 5 ). Users can tailor report content and choose to include all patients, a single patient group, or create individual reports for each patient group. The quality metrics report, created in response to feedback from IeDEA investigators, displays the completeness and quality of data for each variable within a table, for each site included the dataset. Step 4: Submit data. For authenticated users who access the Toolkit through the IeDEA Hub in response to a specific data request, an additional step is available. This final step allows users to transfer the uploaded dataset to secure cloud storage. Clicking the "Submit Data" button zips the files shared by the user, and communicates via the AWS API to store the dataset in a new secured AWS S3 bucket.
In addition, the "Submit Data" process triggers generation of a standard overall report to be stored in a REDCap Data Uploads project along with the name of the corresponding S3 bucket. The report is available to authorized data downloaders on the IeDEA Hub. Datasets are automatically deleted after 30 days. Optional step: Dataset visualization. We added a visualization option to allow users to explore their dataset and download publication-ready plots. Consistent user interface design. Throughout the application, we maintained a cohesive user interface with consistent color choices. We also followed a convention of presenting the user with two boxes at the bottom of each page: a green box highlighting the recommended next step and a gray box providing an alternative option. These colors are chosen dynamically depending on the severity of errors found in the dataset.
Tutorials and help page. Users who enter the Toolkit directly through the URL land on a welcome page which explains the workflow and includes links to the IeDEA data model browser, the IeDEA Hub, a sample data to use in testing the Toolkit, and a YouTube video tutorial. The Toolkit also includes a Help page with an exhaustive Frequently Asked Questions section, informed by questions submitted by data managers in the Toolkit Testing REDCap Survey. These questions and answers are retrieved from a REDCap project which allows for easy updates to the FAQ section.
User Toolkit evaluation after one year of use In July 2020, one year after initial adoption of the Toolkit, we invited 14 regional data managers representing all seven IeDEA regions to respond to a 1-Year Reflection Survey in REDCap. Of these, 13 (93%) completed the anonymous survey and shared their experiences with the Toolkit, including their perceptions of its impact on workflow and data quality. Survey respondent roles within IeDEA included the following: prepare datasets to submit for IeDEA multiregional projects (n = 13), receive or merge datasets for multiregional projects (n = 9), and analyze datasets (n = 7). All respondents had used the Toolkit for the following activities: testing and training activities (n = 11), checking and correcting data (n = 11), submitting datasets for multiregional projects (n = 11), and generating reports (n = 9).
Survey participants indicated that Toolkit use has made IeDEA data management tasks easier (Fig. 8 ). Notably, 100% of respondents reported that Toolkit use had improved the quality of IeDEA datasets. When asked about the Toolkit's automatically generated reports, all users responded that the reports were useful in multiple ways, including checking for data gaps and revising datasets. Among the 11 data managers who used the Toolkit to submit datasets for multiregional requests, 91% (n = 10) reported having revised datasets based on Toolkit data quality reports before transferring data to investigators.
Toolkit adoption and usage in first 18 months Analysis of the first 18 months of Toolkit usage as tracked in REDCap revealed that the Toolkit was used by data managers from all seven IeDEA regions to check, summarize, and transfer datasets that included patients from every clinic in their region. Datasets were uploaded to the Toolkit 507 times for quality checking, revealing a total of over 269 million data errors and warnings across these datasets. Approximately 35% of data quality check sessions were initiated through the IeDEA Hub in response to an IeDEA multiregional data request, whereas 65% were anonymous sessions initiated by accessing the URL directly (without access to dataset transfer). The number of patients in a single dataset ranged from 457 to 986,089, with a mean of 123,769 patients per dataset.
A total of 260 dataset reports were downloaded by users, not including the reports automatically generated and stored on the Hub to document dataset transfer. Of all uploads, 41 final datasets and linked reports were transferred to secure cloud storage through the Toolkit in response to multiregional data requests. Use of the Toolkit for dataset transfer varied across the seven regional data centers: One region transferred 13 datasets through the Toolkit, two regions transferred 7 datasets each, three regions transferred 4 datasets each, and one region transferred 2 datasets during this 18-month window of time.
Impact of Toolkit use on data quality Of the 41 datasets that were shared for multiregional studies in the first 18 months of Toolkit use, 20 were uploaded, checked, and revised before final dataset transfer. In all cases, the datasets were transferred to the requesting investigator with fewer errors than were present in the initial upload. In 6 cases, 100% of errors were corrected. The mean percent decrease in the number of errors between the first data quality session with the Toolkit and the final dataset submission was 61% (max = 100%, min = 0. 003%). Fig. 9  shows an example of the decrease in the number of dataset records that triggered errors with each cycle of checking a dataset and analyzing data quality feedback. (See the Appendix for error reduction visualizations for all twenty datasets.
). Among these 20 datasets, the most common type of error found in the initial upload was the existence of duplicate records (n = 16 datasets). After Toolkit data quality feedback and subsequent dataset revision, 6 datasets contained duplicate records. Invalid codes for coded variables (e. g. , medication code) were found in 15 of the 20 datasets initially, but were present in 5 of the final, transferred datasets. The variable most commonly coded incorrectly was ART_ID, the ATC code for patient antiretroviral medication. Fig. 10  summarizes how many datasets included errors of each type on initial upload and which types of errors users were able to correct after receiving data quality feedback from the Toolkit.
Discussion Incorporating the Harmonist Data Toolkit into the IeDEA data Fig. 10. Error types found in initial uploads as compared with final uploads among datasets that were checked with the Toolkit multiple times and revised before submission. workflow has improved the quality of datasets submitted for studies and increased transparency around data quality. The Toolkit has simplified the task of linking data quality checks to a data model and, unlike many other data quality solutions  [5, [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] , it does not require locally installed and maintained software. Use of the Toolkit is popular among IeDEA data managers; they have observed new efficiencies from the streamlined workflow, automated data quality feedback, sharable reports, and the way that the Toolkit tracks the history of final dataset submissions.
Lessons learned Frequent contact between the application development team and IeDEA data managers not only resulted in an application well-suited to users' needs but also increased user comfort with and adoption of the tool. A few data managers were initially slow to learn the new process, but usage tracking data showed that each time a testing and training activity or a demonstration was conducted, more data managers began using the Toolkit and the frequency of sessions increased. Adoption of the Toolkit by every region required working to gain the trust of the investigators and data managers in each region. By taking steps to ensure data privacy (allowing each region to review data requests and choose whether or not to participate, deleting datasets from Toolkit server memory as soon as a Toolkit session ends, limiting transferred dataset access to previously designated data downloaders, requiring multifactor authentication of data downloaders, logging all data uploads, downloads, and deletions in the IeDEA Data Hub, and automatically deleting datasets from cloud storage after 30 days), we demonstrated respect for the sensitive data the Toolkit processes and transfers.
Although using a cloud service provider delivered a robust server infrastructure for our users, it may not be appropriate for all installations of the Toolkit. Experienced network and server managers are needed to set up a secure cloud environment. If using the Toolkit to process datasets subject to heightened data privacy regulations (e. g. , European General Data Protection Regulation [GDPR]), additional planning and documentation with legal counsel is needed. Our implementation used the free version of Shiny Server, which does not support multithreading. Since our users were distributed across time zones and the Toolkit processed datasets quickly, we were able to handle all requests sequentially. If parallel processes were needed, however, to conduct data quality checks in other networks with simultaneous user sessions involving large datasets, it would require additional servers or a paid license for Shiny Server Pro for multithreading support.
Finally, we recognize that the Toolkit is not an ideal solution for all types of research networks. Large research networks with established data quality tools and processes, such as PCORnet or OHDSI, would not benefit from adoption of the Toolkit. Similarly, networks processing large EHR-extracted datasets should use software designed for data in OMOP or HL7 Fast Healthcare Interoperability Resources (FHIR) formats  [32]. The Toolkit fills the gap of available tools for less highlyresourced research networks facing data harmonization challenges who use disease-specific data models and have limited technical resources available for data quality.
Future work Our next priority is to adapt the Toolkit for use beyond IeDEA while also continuing our active collaboration with the IeDEA Data Harmonization Working Group to improve the Toolkit's data quality assessments and reports. To facilitate use of the Toolkit by other research networks, we are currently in the process of generalizing the Toolkit code to remove IeDEA-dependent elements (e. g. , logos, consortium details, variable naming conventions) and store those details in REDCap. The resulting application will accommodate any data model that has been defined in standardized REDCap templates. After testing the generalized Toolkit with a different data model and dataset, we plan to create documentation, including custom data quality check templates, for our public GitHub repository to guide other groups in Toolkit implementation.
Conclusion The importance of data quality in research networks is well established. For federated research networks similar to IeDEA, with no central database and limited availability of regional technical personnel and data resources, a web-based software solution for data quality checking, report generation, and dataset transfer like the Harmonist Data Toolkit can improve data quality and increase research throughput. Furthermore, by abstracting the data quality checking algorithms based on a common data model definition in REDCap, we designed data quality checks that automatically adapt to an evolving data model. Because software tools can only be effective if they are embraced by users, our collaboration with IeDEA data managers throughout the design process and implementation, coupled with structured testing and training activities, were key to the Toolkit's impact and success.
software that requires local installation or maintenance Minimize need for technical resources or personnel Enable users to work in desired data environments (SAS, Stata, CSV/Excel, R, SPSS) Provide intuitive, easy-to-use interface Generate automatic, customizable reports REQUEST CATEGORY 2: SIMPLIFY DATA HARMONIZATION Flag variables, formats, and codes that are not consistent with the data model Check for plausible numeric values and dates, date order logic, and agreement between related variables (e. g. , an end date and a reason for ending) Offer data quality results interactively and in downloadable spreadsheets Provide an error spreadsheet that includes all details necessary to locate records with data quality issues Keep data quality checks up to date with the current version of the common data model REQUEST CATEGORY 3: PROVIDE WORKFLOW SUPPORT Remind users of details of tables and variables requested for a selected study Prioritize dataset security and privacy (no long-term data storage) Transfer files submitted for multiregional studies using a secure approach Generate dataset summaries and data quality reports automatically when datasets are transferred, to provide a history of dataset submissions.