2_use_case_name: Energy Wizard
3_agency: Department of Energy
3_abr: DOE
4_bureau: NREL - National Renewable Energy Laboratory (EE)
8_topic_area: Mission-Enabling (internal agency support) 
10_commercial_ai: Collaborating in real-time using AI-assisted tools in word processors.
11_purpose_benefits: Making NREL data more accessible for researchers and accelerating research. 
12_outputs: The systems leverages Retrieval-Augmented Generation to find semantically relevent content which the AI (LLM) summarizes for the end user as a method to describe relevent content within the selected publications and research profiles. 
16_dev_stage: Operation and Maintenance
17_impact_type: Neither
18_date_initiated: 6/21/2024
19_date_acq_dev_began: 7/1/2024
20_date_implemented: 9/23/2024
22_dev_method: Developed in-house.
24_hisp_support: No
27_public_info: Yes
28_iqa_compliance: "To demonstrate compliance with the DOE Information Quality Act (IQA) guidelines, the approach with RAG and LLM-based applications ensures the quality, objectivity, utility, and integrity of disseminated information. 

Quality and Objectivity: The approach uses a locked-down Retrieval-Augmented Generation (RAG) system that minimizes inaccuracies (hallucinations) and excludes memory components, ensuring accurate, clear, and reliable information presentation. This satisfies the IQA's requirement for objective, unbiased dissemination and maintains transparency.

Utility: Integrating metadata-driven semantic search in the vector database enhances the utility of DOE's data catalog for public users, meeting the IQA's standard for usefulness. This utility-focused design ensures that data is accessible, relevant, and usable for public insights, supporting transparency and public knowledge.

Transparency and Reproducibility: The approach supports data reproducibility by basing responses on catalog metadata and summary information, ensuring consistency and enabling independent reanalysis by qualified users. Transparency is further upheld through documentation of the RAG interactions, aligning with the IQA's standards for open access and transparency in information dissemination.

Response to Public Needs: This design incorporates continuous feedback to improve relevance and usability, engaging public and stakeholder needs effectively. It aligns with the IQA's goal of considering public feedback in information quality and ensuring accountability through documented, high-level review processes for data corrections if requested.

By adhering to these DOE IQA guidelines, this approach ensures public trust in DOE's open data, demonstrating alignment with the standards of quality, utility, objectivity, and integrity as required under OMB M-19-15."

29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
33_agency_data: As this is summarizing data from the OEDI data repository the related datasets are described by the catalog which is also used to validate the AI responses:  https://data.openei.org/
34_data_docs: Documentation has been partially completed: Some documentation exists (detailing the composition and any statistical bias or measurement skew for training and evaluation purposes), but documentation took place within this use case's development.
37_custom_code: Yes
38_code_access: No – agency does not have access to source code.
40_has_ato: Yes
41_system_name: NREL Stratus AWS Compute Environment, NREL Azure System, and NREL Google Cloud Platform
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Re-use production level code and/or data products: This use case re-uses production level code as well as data products that have been published or made widely available for data science teams for within the agency.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.
51_extension_request: No – Agency did not request an extension for this use case.
52_impact_assessment: Planned or in-progress.
53_real_world_testing: Benchmark evaluation: Testing of the AI model has involved the use of benchmarks (either publicly available or internally created) to estimate performance in real-world settings, but has not been tested in an operational environment. 
54_key_risks:  Intellectual Property and Copyright Concerns. Bias and Fairness
55_independent_eval: Planned or in-progress
56_monitor_postdeploy: No monitoring protocols have been established: Necessary infrastructure needed to perform monitoring of AI systems in production is not available and/or a plan to monitor models has not yet been established.
57_autonomous_impact: No - Some individual decisions or actions require direct human oversight.
59_ai_notice: Online – in the terms or instruction for the service
61_adverse_impact: No
62_disparity_mitigation: Process to detect bias and fairness are in testing and being regulary evaluated by model provideers. There are built-in guardrails in place to prevent abuse

63_stakeholder_consult: None of the above
65_appeal_process: Yes
67_opt_out: Other
67_question_type: Process to detect bias and fairness are in testing and being regulary evaluated by model provideers. There are built-in guardrails in place to prevent abuse