2_use_case_name: AskPRIMR
3_agency: Department of Energy
3_abr: DOE
4_bureau: NREL - National Renewable Energy Laboratory (EE)
8_topic_area: Mission-Enabling (internal agency support) 
10_commercial_ai: Collaborating in real-time using AI-assisted tools in word processors.
11_purpose_benefits: Making data more accessible and user-friendly for the public. 

12_outputs: The systems leverages Retrieval-Augmented Generation to find semantically relevent content which the AI (LLM) summarizes for the end user as a method to describe relevent content within the data catalog. 

16_dev_stage: Operation and Maintenance
17_impact_type: Neither
18_date_initiated: 9/30/2024
19_date_acq_dev_began: 10/1/2024
20_date_implemented: 10/30/2024
22_dev_method: Developed in-house.
23_contract_piids: NA

24_hisp_support: No
27_public_info: Yes
28_iqa_compliance: "To demonstrate compliance with the DOE Information Quality Act (IQA) guidelines, the approach with RAG and LLM-based applications ensures the quality, objectivity, utility, and integrity of disseminated information. 

Quality and Objectivity: The approach uses a locked-down Retrieval-Augmented Generation (RAG) system that minimizes inaccuracies (hallucinations) and excludes memory components, ensuring accurate, clear, and reliable information presentation. This satisfies the IQA's requirement for objective, unbiased dissemination and maintains transparency.

Utility: Integrating metadata-driven semantic search in the vector database enhances the utility of DOE's data catalog for public users, meeting the IQA's standard for usefulness. This utility-focused design ensures that data is accessible, relevant, and usable for public insights, supporting transparency and public knowledge.

Transparency and Reproducibility: The approach supports data reproducibility by basing responses on catalog metadata and summary information, ensuring consistency and enabling independent reanalysis by qualified users. Transparency is further upheld through documentation of the RAG interactions, aligning with the IQA's standards for open access and transparency in information dissemination.

Response to Public Needs: This design incorporates continuous feedback to improve relevance and usability, engaging public and stakeholder needs effectively. It aligns with the IQA's goal of considering public feedback in information quality and ensuring accountability through documented, high-level review processes for data corrections if requested.

By adhering to these DOE IQA guidelines, this approach ensures public trust in DOE's open data, demonstrating alignment with the standards of quality, utility, objectivity, and integrity as required under OMB M-19-15."

29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: As this is summarizing data from the PRIMRE data repository the related datasets are described by the catalog which is also used to validate the AI responses:  https://openei.org/wiki/PRIMRE

34_data_docs: Documentation is widely available: Documentation is not only complete, but is widely accessible within the agency, and has an owner and a regular update cadence.
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: Yes
41_system_name: Amazon Web Services, NREL Azure System
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Re-use production level code from a different use-case: This use case re-uses production level code and/or internally developed code libraries for model development from existing AI systems or repositories inside the agency.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.
51_extension_request: No – Agency did not request an extension for this use case.
52_impact_assessment: Planned or in-progress.
53_real_world_testing: Performance evaluation in operational environment: The AI use case has been tested in an operational environment before being fully implemented as a solution, or has been tested in simulated or controlled environment using operational or synthetic data.
54_key_risks: As the use case for the AI has been limited to summarization of existing metadata the key risks were identified as LLM hallucinations or LLM generated text which would cause reputational harm to DOE.   Techniques such as red-teaming, adversarial testing, and prompt-injection testing were used to evaluate constraint of hallucinations and LLM.  

55_independent_eval: Planned or in-progress
56_monitor_postdeploy: Intermittent and Manually Updated: A plan for monitoring the AI use case is in place, and requires data science teams to work with DevOps engineers to manually update models at scheduled intervals, and create metrics to detect data distribution shifts between the operational environment and the training data for the model. 
57_autonomous_impact: No - Some individual decisions or actions require direct human oversight.
59_ai_notice: Online – in the terms or instruction for the service
61_adverse_impact: No
62_disparity_mitigation: Process to detect bias and fairness are in testing and being regulary evaluated by model provideers. There are built-in guardrails in place to prevent abuse

65_appeal_process: No – it is not operationally practical to offer this.
66_no_appeal_reason: As the action of the system is to summarize and communicate existing information returned for a query, the ability of the AI to make impact individuals or groups is significantly limited. 

67_opt_out: Other
67_question_type: Other methods of searching content using common search methodlogies are also available to the catalog users.