2_use_case_name: Using Generative AI for Stance Analysis of Public Comments on CDC’s Proposed Rules
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: CDC
8_topic_area: Mission-Enabling (internal agency support)
10_commercial_ai: None of the above.
11_purpose_benefits: By law, Public comments are required to be manually review for any notice of public rule making. These comments are highly labor intensive due to both the quantity and wide range of potential opinions shared by the public. These comments can range in both length of response and quantity serving as a challenge for properly ensuring the insights from the public are accurately gathered in the review process. By using an LLM based Topic Modeling approach, manual reviewers can organize responses based upon theme to enhance both the insights gained, the speed of manual review, and ensure all topics mentioned can be accurately reported in all review processes. This process can use open source python libraries and open-source LLMs to complete different actions.

 As public feedback on various rules gets collected across the CDC’s divisions, this initiative along with other NLP approaches can potentially save time for CDC’s public policy experts during public comment review. 
12_outputs: Reviewing the public comments currently involves a manual review process that necessitates a high level of time and effort on the part of regulatory analysts. To improve the process of reviewing public comments as part of regulatory analysis, our team implemented topic modeling (Grootendorst et al., 2022) and sentiment analysis (Hartman et al., 2023) of comments published to the new amendment to foreign quarantine regulation published by CDC on July 10, 2023. This solution expands these efforts by using generative AI models  to try for capture stance expressed in each comment. While there is also an option to manually label and train models for stance detection, custom-trained text classification models may not be transferable to future rules due to rule-specific discussions and arguments. In contrast, using generative AI models for stance detection can be transferable to future rules irrespective of the discussion contexts and arguments.
16_dev_stage: Implementation and Assessment
17_impact_type: Neither
18_date_initiated: 7/1/2023
19_date_acq_dev_began: 7/1/2023
20_date_implemented:  
21_date_retired:  
22_dev_method: Developed in-house.
23_contract_piids:  
24_hisp_support:  
25_hisp_name:  
26_public_service:  
27_public_info: No
28_iqa_compliance:  
29_contains_pii: No
30_saop_review: No
31_data_catalog: No
34_data_docs: Documentation is missing or not available
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
39_code_link:  
40_has_ato: No
41_system_name:  
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Use of existing data platforms: This use case is being developed on existing enterprise data and analytics platforms within the agency rather than procuring additional platforms or SaaS to operate.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.