2_use_case_name: ClinicalTrials.gov Protocol Registration and Results System Review Assistant
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: ClinicalTrials.gov is a website and online database of clinical research studies and information about their results. The purpose of ClinicalTrials.gov is to provide information about clinical research studies to the public, researchers, and health care professionals.

This project aims to help ClinicalTrials.gov determine whether the addition of AI could make reviewing study records more efficient and effective.
12_outputs: The classifier takes as input free-text related to a study’s endpoint and returns a prediction of whether one or more issues is present. 
16_dev_stage: Acquisition and/or Development
17_impact_type: Neither
18_date_initiated: 2/1/2021
19_date_acq_dev_began: 11/1/2021
22_dev_method: Developed with contracting resources.
23_contract_piids: GSA Contract # GS-10F-0064W    NLM BPA Task Order: 75N97019A00005/75N97021F00006
31_data_catalog: No
33_agency_data: We utilized agency-owned data for training and evaluating the model's performance. The data consisted of clinical study record submissions from ClinicalTrials.gov's Protocol Registration and Results System (PRS) that were divided into two groups:

Set A: This group included submissions that did not pass quality control review, meaning they were returned to data submitters for revision and, in certain cases, are versions of the record that were never posted on ClinicalTrials.gov (not accessible to the public). These submissions had a relevant issue identified during manual quality control review, providing examples of endpoints known to contain the specific issue.

Set B: This group included submissions that were posted on ClinicalTrials.gov with endpoints that had never been labeled as having the specific issue. These submissions provided examples of endpoints that were considered not to have the specific issue.The input to the classifiers consisted of endpoint attributes, including Measure Titles, Descriptions, Time Frames, Units of Measure, and labels indicating whether the issue was present. This data did not have any personal identifiers. The data was in a structured format in the PRS, but the specific endpoint attributes were provided as free-text.

We processed over 10,000 endpoints from Set A, which included examples with identified issues, and over 40,000 endpoints from Set B, which had not been flagged with issues. It is important to note that the labeling of issues in the dataset is not exhaustive. Some endpoints may present multiple issues combined and labeled as a single issue, and there is a possibility that some issues were missed by reviewers. Consequently, Set B might contain endpoints that actually have the issue despite not being labeled as such. 
34_data_docs: Documentation is complete
40_has_ato: No
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Use of existing data platforms: This use case is being developed on existing enterprise data and analytics platforms within the agency rather than procuring additional platforms or SaaS to operate.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.