GreatNorthCollective/greatnorth-us-federal-ai-inventory-2024 huggingface dataset:
2_use_case_name: Improving Metadata Retrieval and Transformation for Metadata Management
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: The Dataset Catalog (catalog) is a catalog of biomedical datasets from various repositories for users to search, discover, retrieve, and connect with datasets to accelerate scientific research. Metadata from included repositories and their datasets are needed to appropriately index them in the catalog. This manual process can be labor intense, and thus NLM is pilot testing the development of a working interface for internal users to retrieve the metadata information from external repositories' websites and then transform the data to a specific format used in internal metadata management for the Dataset Catalog. This is currently being developed behind NIH's firewall for a select internal user base. The artificial intelligence (AI) will automate the retrieval and transformation of metadata but will not replace human oversight in critical metadata curation tasks. Additional safeguards would make sense in a production environment. The AI uses information from publicly available biomedical data repository websites and internal metadata management systems. The AI uses metadata from biomedical repositories and will be deployed within data management and IT departments, particularly those involved in cataloging and metadata management.

Primary Objectives: Streamline metadata retrieval and transformation to improve efficiency and accuracy.

Anticipated Positive Outcomes: Resource savings, enhanced data quality, and improved management of the Data Set Catalog.
12_outputs: Inputs: Metadata from various biomedical repositories, internal schemas.

Outputs: Transformed metadata, Python scripts, user-friendly tools for metadata retrieval.

Frequency: Continuous retrieval and transformation as new metadata is ingested.
16_dev_stage: Acquisition and/or Development
17_impact_type: Neither
18_date_initiated: 11/1/2023
19_date_acq_dev_began: 5/1/2023
22_dev_method: Developed with contracting resources.
23_contract_piids: 75N97023A00004/75N97023F00007
31_data_catalog: Yes
33_agency_data: Website data (HTML, CSS, JavaScript), accessibility standards. Measures taken to ensure data accuracy and reliability: grounding, testing, evaluation, NIST framework.
34_data_docs: Documentation is complete
40_has_ato: No
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Re-use production level code from a different use-case: This use case re-uses production level code and/or internally developed code libraries for model development from existing AI systems or repositories inside the agency.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.
2_use_case_name: Internal Referral Module 
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: The NIH Extramural Research Program uses the eRA system to efficiently award and manage grants.

The AI system is designed for use by IC ‘internal referral’ officers who manage a pool of applications and are responsible for assigning the applications to program officials. It creates efficiencies by automating the ‘internal referral’ assignments, currently done manually on spreadsheets by the ICs. The inputs are grant applications, the outputs are referrals to Program Officers, Program Class Codes, Organizational units - Divisions and Branches and Scientific Research Clusters.
12_outputs: The Internal Referral initiative automates a manual process by using AI and NLP capabilities to help assign grant applications to NIH Institute and Center Program Officers to make informed decisions.

The inputs are grant applications, the outputs are referrals to Program Officers, Program Class Codes, Organizational units - Divisions and Branches and Scientific Research Clusters.
16_dev_stage: Operation and Maintenance
17_impact_type: Neither
18_date_initiated: 1/1/2018
19_date_acq_dev_began: 1/1/2019
20_date_implemented: 2/1/2023
22_dev_method: Developed with contracting resources.
23_contract_piids: N/A, The work is performed across multiple eRA contracts.
27_public_info: No
29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: We used eRA grant application data for all fine tuning and optimization for the models. Specially, we extract the title, abstract, specific aims and public health narrative to train our models for prediction. 
34_data_docs: Documentation is complete
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: Yes
41_system_name: eRA
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Use of existing data platforms: This use case is being developed on existing enterprise data and analytics platforms within the agency rather than procuring additional platforms or SaaS to operate.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.
2_use_case_name: IT System Outage Alerts
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: Use Generative AI to create maintenance and IT system outage alerts that will be sent to NIDCD staff.

General communication to the NIDCD staff. The GenAI tool will reduce staff time to create and distribute e-mail alerts.
12_outputs: Communicates alerts for IT System outages. Output: System, building, day, time.
16_dev_stage: Initiated
17_impact_type: Neither
18_date_initiated: 10/1/2023
2_use_case_name: JIT Automated Calculator (JAC)
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: NIGMS supports basic research that increases our understanding of biological processes and lays the foundation for advances in disease diagnosis, treatment, and prevention. NIGMS-funded scientists investigate how living systems work at a range of levels from molecules and cells to tissues and organs, in research organisms, humans, and populations.

NIGMS needs to know how much total support an investigator has to ensure that it is not funding Principal Investigators who are already adequately resourced. The tool can determine how much funding a Principal Investigators has been awarded.
12_outputs: The JIT Automated Calculator (JAC) uses NLP to parse Just-In-Time (JIT) Other Support forms and determine how much outside support PIs are receiving from sources other than the pending application.
16_dev_stage: Operation and Maintenance
17_impact_type: Neither
18_date_initiated: 3/1/2024
19_date_acq_dev_began: 3/1/2023
20_date_implemented: 5/1/2023
22_dev_method: Developed in-house.
27_public_info: No
29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: All data come from the internal NIH IMPAC II database.
34_data_docs: Documentation is missing or not available
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: No
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Use of existing data platforms: This use case is being developed on existing enterprise data and analytics platforms within the agency rather than procuring additional platforms or SaaS to operate.
50_internal_review: No documentation is available: No documentation detailing model performance metrics, model architecture, features and intended use of the models have been created or are currently accessible to other data science teams within the agency.
2_use_case_name: Large Language Automation
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: Automate use cases as indicated below.

1. Automate BOT to locate and extract Resource Sharing Plan from post awarded grant application and feed into Microsoft Azure AI to teach it to determine if grant application as written has met compliance. The RFA provides guidelines for grantees.
2. Generate summary for published papers.
3. Generate summary for papers for concept development, research data analysis or other internal work.
4. Generate summary as part of a preliminary review of documents.
5. Generate summary of public meetings.
6. Utilize existing MS Excel worksheets to automate templated emails.
7. Generate FAQs.
8. Produce training materials. 

The RFA: 
1) provides guidelines for grantees,
2) generates summary for published papers,
3) generates summary for papers for concept development, research data analysis or other internal work,
4) generates summary as part of a preliminary review of documents,
5) generates summary of public meetings,
6) utilizes existing MS Excel worksheets to automate templated emails,
7) generates FAQs, and
8) produces training materials. 
12_outputs: Input: Push data into Microsoft Azure AI from QVR post awarded grant application data.

Output: Integration of Microsoft Azure AI with UiPath. 
16_dev_stage: Initiated
17_impact_type: Neither
18_date_initiated: 10/1/2023
2_use_case_name: Leveraging User-Generated Content for Digital Behavioral Interventions
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Health & Medical
10_commercial_ai: None of the above.
11_purpose_benefits: The project uses AI-based automated text classification technology to build a predictive model to categorize smoking-related tweets into six distinct categories (e.g., quitting strategies). Specifically, the project uses the AI-based BERT pre-trained language model, which will be further fine-tuned using neural network models to capture the contextual nuances of our use case. AI will also be used to select data for the training dataset that will form the basis of the predictive model. Then, we use ChatGPT to classify extracted and labelled tweets according to its sentiment, presence of sarcasm, presence of irony, presence of emojis. 

The primary objective of the project is to classify thousands of tweets to six categories (e.g., strategies for quitting, quitting benefits) and according to their sentiment and presence of irony, sarcasm, and emojis. Tweets that are positive, not ironic, nor sarcastic would then be tested with target audiences for message effectiveness and possible integration in a smoking cessation intervention.
12_outputs: Input: Publicly available tweets.

Output: sentiment (very positive, positive, neutral, negative, very negative), presence of sarcasm (yes, no), presence of irony (yes, no), presence of emojis (yes, no - and if yes, a description of the emojis present in the tweet).
16_dev_stage: Implementation and Assessment
17_impact_type: Neither
18_date_initiated: 1/1/2024
19_date_acq_dev_began: 1/1/2024
22_dev_method: Developed with contracting resources.
27_public_info: No
29_contains_pii: No
30_saop_review: No
31_data_catalog: No
33_agency_data: Corpus of 5 million tweets extracted for the purposes of the research by a colleague who had access to twitter archives.
34_data_docs: Documentation is missing or not available
37_custom_code: No
38_code_access: No – agency does not have access to source code.
40_has_ato: No
42_dev_tools_wait: More than 12 months
43_infra_provisioned: No
45_compute_request: No
47_timely_resources: No
49_existing_reuse: None: This use case does not re-use any internally developed tooling or managed infrastructure from any other AI development efforts within the agency.
50_internal_review: No documentation is available: No documentation detailing model performance metrics, model architecture, features and intended use of the models have been created or are currently accessible to other data science teams within the agency.
2_use_case_name: LLM Support for Admin Services
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: This project aims to explore the integration of large language models (LLMs) into administrative operations, aiming to enhance efficiency, accuracy, and productivity. By leveraging LLMs, repetitive tasks can be automated including data analysis, documentation drafting, and customer support, thereby freeing up valuable time for administrative personnel to focus on more strategic initiatives. Additionally, LLMs can facilitate natural language processing (NLP) for documentation, ensuring consistency and accuracy in communication across various channels. Through process automation, policy monitoring, and virtual assistance, LLMs offer the potential to streamline our administrative workflow, reduce manual workload, and mitigate the risk of errors. This assessment will provide valuable insights into the feasibility and effectiveness of integrating LLMs into our administrative framework, ultimately optimizing our operations and enhancing organizational performance.

It will reduce the burden of administrative tasks, faster response to administrative inquiries, fewer resources required to support administrative operations.
12_outputs: Uses NLM administrative and operational data to output procedures, forms, schedules, etc.
16_dev_stage: Initiated
17_impact_type: Neither
18_date_initiated: 5/1/2024
2_use_case_name: Mapping Sequence Data to Research Outcomes
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: A proof-of-principle analysis is being conducted to identify downstream research products related to sequence records included in NLM-managed repositories.

To identify sequence records in NLM-NCBI-managed data repositories and related research products to inform operations and prioritize resources.
12_outputs: Sequence records included in NLM-NCBI-managed data repositories and abstracts for citations records indexed in PubMed; outputs are text summaries identifying in a yes/no fashion if research products were identified
16_dev_stage: Initiated
17_impact_type: Neither
18_date_initiated: 12/1/2023
2_use_case_name: Medical Text Indexer-NeXt Generation (MTIX) MEDLINE Indexing
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: Machine learning-based system for the automated indexing of MEDLINE citation records using Medical Subject Heading (MeSH) terms. Automated indexing is achieved using a multi-stage neural text ranking approach.

MeSH indexing has been shown to improve information retrieval in PubMed, but manual indexing was not sustainable due to the increasing volume of published biomedical literature. Automated indexing allows for cost-effective and timely indexing of MEDLINE citation records.
12_outputs: The input is PubMed citation data including the article title and abstract, and the output is a set of MeSH terms describing the topic of the article.
16_dev_stage: Operation and Maintenance
17_impact_type: Neither
18_date_initiated: 1/1/2021
19_date_acq_dev_began: 1/1/2022
20_date_implemented: 5/1/2023
22_dev_method: Developed with contracting resources.
23_contract_piids: Black Canyon Consulting LLC 47QTCA18D00JA/75N97023F00098
27_public_info: No
29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: The MTIX dataset is approximately 10 million PubMed MEDLINE citations published after 2006. It is publicly available data, used for training and evaluation of the MeSH terms predicted by the algorithm. 
34_data_docs: Documentation is complete
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: No
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Re-use production level code from a different use-case: This use case re-uses production level code and/or internally developed code libraries for model development from existing AI systems or repositories inside the agency.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.
2_use_case_name: MetaMap
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: NLM is the world’s largest biomedical library. MetaMap is a program providing access from biomedical text to the concepts in the Unified Medical Language System Metathesaurus.

MetaMap uses NLP to provide a link between the text of biomedical literature and the knowledge.
12_outputs: Metamap is used to map biomedical text to the Unified Medical Language System Metathesaurus or, equivalently, to discover Metathesaurus concepts referred to in text. 
16_dev_stage: Operation and Maintenance
17_impact_type: Neither
18_date_initiated: 1/1/1996
19_date_acq_dev_began: 1/1/1996
20_date_implemented: 1/1/1998
22_dev_method: Developed with contracting resources.
23_contract_piids: GSA Contract Number:  47QTCA18D00JA
27_public_info: No
29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: UMLS Metathesaurus is used by MetaMap to map the incoming text to concepts that exist in the UMLS Metathesaurus.  MEDLINE citations from the MEDLINE baseline were used in the development, training, and testing of MetaMap.   
34_data_docs: Documentation is missing or not available
37_custom_code: Yes
38_code_access: Yes – source code is publicly available.
39_code_link: https://lhncbc.nlm.nih.gov/ii/tools/MetaMap/run-locally/Sources.html
40_has_ato: No
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: No
47_timely_resources: Yes
49_existing_reuse: Re-use production level code and/or data products: This use case re-uses production level code as well as data products that have been published or made widely available for data science teams for within the agency.
50_internal_review: No documentation is available: No documentation detailing model performance metrics, model architecture, features and intended use of the models have been created or are currently accessible to other data science teams within the agency.
2_use_case_name: MicroStrategy Evaluation
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: The software to be evaluated should provide a business intelligence/reporting/visualization interface for possible use for 2 systems (Azure DevOps and OneStream). This software should provide robust development capabilities for manually-generated reports and AI capabilities for end-users to generate additional reports without the need for development. 

This software should provide robust development capabilities for manually-generated reports and AI capabilities for end-users to generate additional reports without the need for development. 
12_outputs: Input: Financial data from OneStream and Project Management data from Azure DevOps.

Output: Data driven business intelligence dashboards. 
16_dev_stage: Initiated
17_impact_type: Neither
18_date_initiated: 10/1/2023
2_use_case_name: MTIX
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: NLM is the world’s largest biomedical library. MEDLINE is NLM's bibliographic database that contains references to journal articles in life sciences, with a concentration on biomedicine.

This is a machine learning-based system for the automated indexing of MEDLINE articles with Medical Subject Headings (MeSH) terms. Automated indexing is achieved using a multi-stage neural text ranking approach. 
12_outputs: Automated indexing allows for cost-effective and timely indexing of MEDLINE articles.
16_dev_stage: Implementation and Assessment
17_impact_type: Neither
18_date_initiated: 1/1/2021
19_date_acq_dev_began: 1/1/2022
22_dev_method: Developed with contracting resources.
23_contract_piids: Black Canyon Consulting LLC 47QTCA18D00JA/75N97023F00098
27_public_info: No
29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: The MTIX dataset is approximately 10 million PubMed MEDLINE citations published after 2006. It is publicly available data, used for training and evaluation of the MeSH terms predicted by the algorithm. 
34_data_docs: Documentation is complete
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: No
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Re-use production level code from a different use-case: This use case re-uses production level code and/or internally developed code libraries for model development from existing AI systems or repositories inside the agency.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.
2_use_case_name: NanCI: Connecting Scientists
3_agency: Department of Health and Human Services
3_abr: HHS
4_bureau: NIH
8_topic_area: Government Services (includes Benefits and Service Delivery)
10_commercial_ai: None of the above.
11_purpose_benefits: NCI is helping to connect scientists.

NanCI is the app for biomedical scientists that helps discover scientific papers, build a career network, and explore events. 
12_outputs: NanCI is a new mobile application that uses machine learning algorithms to match users’ interests and provide a unique experience by recommending tailored content.
16_dev_stage: Operation and Maintenance
17_impact_type: Neither
18_date_initiated: 1/1/2021
19_date_acq_dev_began: 1/1/2022
20_date_implemented: 1/1/2024
22_dev_method: Developed with contracting resources.
23_contract_piids: Google & Barnacle.ai via STRIDES mechanism
27_public_info: No
29_contains_pii: No
31_data_catalog: No
34_data_docs: Documentation is missing or not available
37_custom_code: No
38_code_access: No – agency does not have access to source code.
40_has_ato: No
42_dev_tools_wait: 6-12 months
43_infra_provisioned: No
45_compute_request: No
47_timely_resources: No
49_existing_reuse: None: This use case does not re-use any internally developed tooling or managed infrastructure from any other AI development efforts within the agency.
50_internal_review: No documentation is available: No documentation detailing model performance metrics, model architecture, features and intended use of the models have been created or are currently accessible to other data science teams within the agency.