2_use_case_name: Stratification Tool for Opioid Risk Mitigation (STORM) predictive model 
3_agency: Department of Veterans Affairs
3_abr: VA
4_bureau: VHA: Veterans Health Administration
8_topic_area: Health & Medical
10_commercial_ai: None of the above.
11_purpose_benefits: The STORM model is a predictive model to estimate risk of a suicide or overdose-related health care event or death in the next year.  The model uses patient level predictors extracted from the patient’s medical records (into the VA’s Corporate Data Warehouse), such as diagnoses, prescriptions, and health care utilization, etc.  
This predictive model is used for three purposes. (1) Primary use case is to identify patients at elevated risk of suicide or overdose-related events for referral to an interdisciplinary team for clinical case review.  This team makes treatment recommendations to share with treating providers and documents them in the medical record. The STORM algorithm is used to identify patients to refer to this program at each facility.  Assignment to this case management program has been shown to reduce all-cause mortality in patients estimated at “very high” risk on the STORM predictive model by 22% in the next 4 months. (2) Secondary use case is to help risk stratify patients based on prior health care interactions to support augmentation of risk mitigation interventions and treatments for patients with patterns of care associated with risk of suicide or overdose events. Risk estimates are provided as a sorting/filtering and informational feature on the STORM clinical decision support system.  This decision support system enables population management of patients prescribed opioids or with opioid use disorders, as well as providing curated information to support treatment planning for individual patients. (3) The last use case is to meet the requirements of the Comprehensive Addiction Recovery Act (CARA) mandating that clinicians use VA decision support tools to review patient risks prior to initiating opioid therapy for pain.  Direct synchronized links from the VISTA medical record to the STORM report integrate this decision support summary information into clinician workflows as a support for risk assessment before writing and initial opioid prescription.  Each of these use cases is built into existing clinical workflows completed by overlapping, but different health care providers. For use case 1, designated interdisciplinary risk review teams at each health care system use the risk estimates to determine which patient require risk review.  For use case 2, VA health care providers (e.g., opioid prescribers) and teams (e.g., primary care and general mental health teams) use the risk estimates in combination with decision support recommendations of interventions to consider facilitate and augment risk assessment and treatment planning for their patients on opioid analgesics or being treated for opioid use disorder.  For use case 3, VA medication prescribers or their delegated team members use the categorized risk scores in the context of curated descriptive data from the patient’s medical record to review risk factors and consider risk and benefits of treatment options for pain management before prescribing opioids.  Risk considerations are documented in a clinical note for tracking.   Use of STORM is expected to improve decision-making about opioid prescribing and treatment of patients with opioid use disorders, as well as consistency of use of clinical practice guideline recommended treatment practices for safe management of patients exposed to opioids.  STORM also focuses on ensuring recognition of patient risks and facilitating care coordination across treatment providers for complex patients with multiple interacting disorders. By improving use of risk mitigation practices and treatment interventions, we expect STORM will reduce adverse outcomes for patients exposed to opioids.  Program evaluations support this expectation, finding that implementation of the targeted interdisciplinary review requirements associated with use case 1 significantly reduced all cause mortality in the targeted patient population. 

Links: https://psycnet.apa.org/buy/2017-03732-004
https://www.tandfonline.com/doi/10.1080/08897077.2018.1540376
https://link.springer.com/article/10.1007/s11606-022-07622-1
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10698768/
12_outputs: The STORM predictive model estimates the risk of having an overdose or suicide-related event in next year for all patients currently receiving a VA opioid analgesic prescription with nightly updates.  For all other active VHA patients, the STORM predictive model estimates the hypothetical risk of having an overdose or suicide-related event in next year, if the patient was prescribed or otherwise had access to low, medium, or high dose opioid analgesics.  These risk estimates are provided to clinicians on a family of dashboards, both as percentage likelihood of an event in the next year, and as a categorical grouping of estimated low, medium, high or very high risk.   Paired with this risk estimate is a summary of all risk factors identified and considered in generating the risk estimate for that patient, and recommendations for risk mitigation and treatment options to consider.  These recommendations are tailored based on patient characteristics identified in the medical record (e.g. Medications for opioid use disorder are recommended for patients seen for an opioid use disorder diagnosis) and patient receipt of these interventions is tracked and displayed on the dashboard.  The STORM dashboards are used by clinicians as an informational tool to summarize and organize medical record information to support risk reviews during treatment planning (e.g. when considering an opioid prescription), for management of patients prescribed opioids or treated for opioid use disorder, and for interdisciplinary case reviews of very high risk patients to ensure safe and effective care coordination and treatment planning across health care providers.
16_dev_stage: Operation and Maintenance
17_impact_type: Both
18_date_initiated: 6/1/2012
19_date_acq_dev_began: 6/1/2012
20_date_implemented: 6/1/2015
22_dev_method: Developed in-house.
24_hisp_support: No
27_public_info: No
29_contains_pii: Yes
30_saop_review: Yes
31_data_catalog: Yes
33_agency_data: Clinical data came from the VHA National Corporate Data Warehouse, an archival data set that contains comprehensive patient-by-patient, encounter-by-encounter clinical and administrative data derived from the Department of Veterans Affairs’ electronic medical records. Data regarding vital status and cause of death were from the National Death Index, a centralized database maintained by the National Center for Health Statistics of death record information on file in state vital statistics offices.
34_data_docs: Documentation is widely available: Documentation is not only complete, but is widely accessible within the agency, and has an owner and a regular update cadence.
35_demo_features: Marital Status, Sex/Gender, Age
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: Yes
41_system_name: VA Corporate Data Warehouse - 1152
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Re-use production level code from a different use-case: This use case re-uses production level code and/or internally developed code libraries for model development from existing AI systems or repositories inside the agency.
50_internal_review: Limited documentation for review: Some information regarding the model’s performance on specific benchmarks are available or have been shared with specific stakeholders within the agency.
51_extension_request: No – Agency did not request an extension for this use case.
52_impact_assessment: Yes
53_real_world_testing: Impact evaluation in operational environment: The AI use case has been tested in an operational environment before being fully implemented as a solution and has utilized randomized experiments with a control group or other counterfactual, social systems analysis, or other rigorous research methodologies to evaluate impact and identify potential harm to users as well as broader groups of people.
54_key_risks: (1) Providers might over-rely on statistical risk assessment. This risk was a concern based on known tendencies of some providers to over-interpret recommendations as requirements.  This tendency was observed in a minority of providers and addressed explicitly in training and training materials moving forward.  (2) The program could be clinically inefficient and use provider time that might be more effectively focused elsewhere.  This is a concern for all VHA clinical programs, and VHA management consistently seeks to optimize clinical time and attention towards achieving the greatest patient benefit.  (3) The program could lead to unfair allocation of clinical resources if the model is biased and other clinical programming does not compensate for areas of model under-performance. Here, a targeted case review is required for patients identified as "very high" risk per the STORM predictive model, though all clinical services are available and recommended for all patients based on clinical indication in the STORM decision support system.   After review of the common risks template, the following risks were identified: Fair & Equitable (FE) - Algorithm target not reflective of real-world outcome of interest Response: There is a risk that changes in population exposures and behaviors related to opioids may alter patterns of risk away from those the model was trained on.    Transparent & Explainable (TE) - Degradation of End-User Trust and Ineffective Challenge or Remedy Processes Response: This is a risk, there has been public misunderstanding of how VA predictive models are developed and used. VA has responded and clarified misconceptions where concerns have been raised.  Accountable & Monitored (AM) - Performance Efficacy and Fairness (e.g. Model/Data Drift, Model Degradation, or inappropriate application)  Response: This is a risk, model performance is degrading over time and an update is in process. However all identified patients are clinically complex and appropriate for clinical attention.  Accountable & Monitored (AM) - User-Introduced Errors Response: There is a low risk, there is a community of practice and regular monthly education to support proper implementation of the VA opioid safety program, including the STORM decision support system and predictive model.  Accountable & Monitored (AM) - System Performance Not as Intended Response: There is a low risk, team has validation processes to review all steps.
55_independent_eval: TRUE
56_monitor_postdeploy: Intermittent and Manually Updated: A plan for monitoring the AI use case is in place, and requires data science teams to work with DevOps engineers to manually update models at scheduled intervals, and create metrics to detect data distribution shifts between the operational environment and the training data for the model. 
57_autonomous_impact: No - Some individual decisions or actions require direct human oversight.
59_ai_notice: Online - in the terms or instructions for the service.
61_adverse_impact: Yes
62_disparity_mitigation: VA has examined performance of the STORM model for estimating overdose and suicide -related risk among Veterans in VHA care, stratified by patient sex, race and ethnicity.  The first iteration of model performance evaluation is described in List et al., 2023, available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10698768/.   The main finding from this evaluation was that the model tended to underestimate risk in non-white populations in years subsequent to the year the model was trained on.  This underestimation tracked population level changes in opioid overdose rates across this period, as there was a significant increase in observed overdose rates in non-white populations between the model training year and later years.  This finding emphasized the importance of regular updating of the predictive model to address changes in risk contributions in the U.S. population over time.  Additional efforts to optimize bias evaluation methods were undertaken utilizing the STORM model as an example (Meerwijk et al., 2024).  While this evaluation emphasized the complexity of bias evaluation, it also suggested opportunities for improvement of model performance, particularly for older adults, who tend to utilize VHA mental health services less than younger VHA patients. Guided by these findings, VHA is actively working on an update to the STORM predictive model, incorporating a significantly expanded set of candidate predictor variables in hopes of better capturing indications of risk across subpopulations and optimizing the model to reflect evolving trends in opioid exposure and outcomes. As part of this effort, VHA has built a decision support infrastructure to enable remodeling, model evaluation, and model deployment into VHA clinical decision support systems on an annual or as-needed cycle.    References:  Meerwijk EL, McElfresh DC, Martins S, Tamang SR. Evaluating accuracy and fairness of clinical decision support algorithms when health care resources are limited. J Biomed Inform. 2024 Aug;156:104664. doi: 10.1016/j.jbi.2024.104664. Epub 2024 Jun 6. PMID: 38851413.   List JM, Palevsky P, Tamang S, Crowley S, Au D, Yarbrough WC, Navathe AS, Kreisler C, Parikh RB, Wang-Rodriguez J, Klutts JS, Conlin P, Pogach L, Meerwijk E, Moy E. Eliminating Algorithmic Racial Bias in Clinical Decision Support Algorithms: Use Cases from the Veterans Health Administration. Health Equity. 2023 Nov 30;7(1):809-816. doi: 10.1089/heq.2023.0037. PMID: 38076213; PMCID: PMC10698768.
63_stakeholder_consult: Direct user testing
65_appeal_process: No – it is not operationally practical to offer this.
66_no_appeal_reason: While we do have a process via which a patient can request to not have risk estimates created, the output of the system only consists of information to guide clinical decision-making.  All decisions and outcomes are driven by the clinician treating the patient.
67_opt_out: Yes