2_use_case_name: Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment (REACH VET) 1.0 predictive model
3_agency: Department of Veterans Affairs
3_abr: VA
4_bureau: VHA: Veterans Health Administration
8_topic_area: Health & Medical
10_commercial_ai: None of the above.
11_purpose_benefits: The REACH VET 1.0 model is a predictive model using predictors extracted from the patient’s medical records (into the VA’s Corporate Data Warehouse), such as diagnoses, prescriptions, and health care utilization, to estimate statistical risk of a suicide death in the next month.  
This predictive model is used for two purposes. (1) Primary use case is to support identification of patients at elevated statistical risk of suicide in the next month for referral to a clinical case review and outreach program. Staffing guidance to ensure each health care system is staffed to conduct the case review and outreach protocol is established per clinical policy. To date, the population recommended to the clinical case review program has been limited to a number equivalent to 0.1% of the total treated patient population at each facility each month; the number of patients in the program may be modified based on guidance from the Office of Suicide Prevention. The REACH VET 1.0 statistical risk estimates contribute to an algorithm used to identify patients to refer to this program at each facility. (2) Secondary use case is to help risk stratify patients based on prior health care interactions to support augmentation of clinical attention and interventions for patients with patterns of care associated with high suicide risk. This is used, for example, to support clinical review of relevant health care history during triage of Veteran Crisis Line calls and during risk assessment of patients in mental health care.
For use case 1, REACH VET coordinators and REACH VET providers use the list and supporting information to guide execution of the targeted prevention program. For use case 2, Veterans Crisis Line responders and VHA health care providers use the categorized risk scores in the context of curated descriptive data from the patient’s medical record to review risk factors towards optimizing assessment and clinical follow-up.   
The REACH VET 1.0 model is expected to identify a population of patients at elevated risk of suicide and other negative outcomes for review by clinicians; clinicians conduct telephone outreach to identified patients when they deem such outreach appropriate.  This model and targeted prevention in which it is used (i.e. the REACH VET program) is used to augment VHA's extensive clinical suicide prevention program. While VHA conducts universal screening for suicide risk by asking patients structured questions regarding suicidality, this screening process does not identify all patients who go on to die of  suicide. The REACH VET model is intended to identify patients who might be missed in clinical assessment or whose risk may have changed since their last assessment. Here, the REACH VET model identifies patients at statistical risk of suicide, that is they share similarities in their health care data to patients who did die of suicide in the past.  The patients at highest statistical risk are highlighted to mental health providers for additional review. This targeted prevention program has been nationally implemented in VHA health care systems since 2017 and an evaluation of the effects of the REACHVET program was conducted. This evaluation found that REACH VET implementation was associated with greater treatment engagement and new safety plan documentation and fewer mental health admissions, emergency department visits, and suicide attempts. Full findings of this program evaluation are available at:  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8524305/
12_outputs: The REACH VET predictive model estimates the likelihood of dying of suicide in the next month for each active patient in VHA. This estimate is used as part of an algorithm that considers factors such as the size of the health care system, current patient treatment engagement, and whether the patient has had previous engagement with the REACH VET program to generate a dashboard with patients at high statistical risk of suicide for review by a REACH VET coordinator and identified clinicians at each health care system. The REACH VET dashboard provides information about identified risk factors for the included patients and supports a coordinated case review and, where applicable, clinical outreach process to support identification of care gaps and offer additional services as appropriate.
16_dev_stage: Operation and Maintenance
17_impact_type: Both
18_date_initiated: 1/1/2014
19_date_acq_dev_began: 1/1/2014
20_date_implemented: 11/1/2016
22_dev_method: Developed in-house.
24_hisp_support: No
27_public_info: No
29_contains_pii: Yes
30_saop_review: Yes
31_data_catalog: Yes
33_agency_data: As quoted in publication 'Clinical data came from the VHA National Patient Care Database, an archival data set that contains comprehensive patient-by-patient, encounter-by-encounter clinical and administrative data derived from the Department of Veterans Affairs’ electronic medical records. Data regarding vital status and cause of death were from the National Death Index, a centralized database maintained by the National Center for Health Statistics of death record information on file in state vital statistics offices. We conducted searches with established procedures.24 We constructed 3 samples: (1) model development, (2) model validation, and (3) prediction. The development and validation samples contained data for patient-months from all patients who died from suicide from October 1, 2008, to September 30, 2011, and had received VHA services in the previous 2 years (case patients) and a random 1% of patients who survived the month and received VHA services in the previous 2 years (control patients). We randomly assigned half of the case patients and half of the control patients to the development sample and half to the validation sample. Each contained patient-months from 3180 case patients and 1 056 004 control patients.' (McCarthy et al, 2015).  Data from the VA Corporate Data Warehouse was used to evaluate performance of the models.
34_data_docs: Documentation has been partially completed: Some documentation exists (detailing the composition and any statistical bias or measurement skew for training and evaluation purposes), but documentation took place within this use case’s development.
35_demo_features: Age, Sex/Gender, Marital Status, Race/Ethnicity, Ability Status
35_question_type: Note that "Ability status" is service connected disability status in this model.
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: Yes
41_system_name: Corporate Data Warehouse - 1152
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Use of existing data platforms: This use case is being developed on existing enterprise data and analytics platforms within the agency rather than procuring additional platforms or SaaS to operate.
50_internal_review: Documentation has been developed: Complete documentation detailing model performance across a range of benchmarks, architecture, relevant features and information regarding the appropriate use of the model for predictive tasks has been created.
51_extension_request: No – Agency did not request an extension for this use case.
52_impact_assessment: Yes
53_real_world_testing: Impact evaluation in operational environment: The AI use case has been tested in an operational environment before being fully implemented as a solution and has utilized randomized experiments with a control group or other counterfactual, social systems analysis, or other rigorous research methodologies to evaluate impact and identify potential harm to users as well as broader groups of people.
54_key_risks: Our main concerns regarding risks are as follows:  (1) Providers might over-rely on statistical risk estimate. To reduce this risk, we provide information regarding which risk factors were identified by the predictive model to encourage transparent understanding of the basis for the model estimate.  Moreover, the clinical prevention program provides only general guidelines to clinicians indicating that identified patients should be reviewed.  This forces clinicians to rely on their clinical skills to plan intervention responses. (2) The program could be clinically inefficient and use provider time that might be more effectively focused elsewhere.  This is a real risk which may evolve over time as other components of the overall VA suicide prevention program improve, and potentially reduce the need for this targeted prevention program. VHA conducts ongoing evaluations to assess for potential inefficiencies in the model and test methods to improve clinical efficiency.  For example, VA is currently piloting a process to expand focus of the REACH VET program to patients not currently engaged in mental health care.  (3) The program could lead to unfair allocation of clinical resources if the model is biased and other clinical programming does not compensate for areas of model underperformance. This is also a risk and thus VHA evaluates model bias across key demographic populations and actively develops methods to reduce model bias or address bias by adjusting other components of the treatment system to compensate.   After review of the common risks template, the following additional risks were identified: Fair & Equitable (FE) - Algorithm target not reflective of real-world outcome of interest Response: There is a minor risk that the county coroners who assign cause of death might have miscategorized some Veteran deaths, reducing the validity of the outcome variable used.   Transparent & Explainable (TE) - Degradation of End-User Trust and Ineffective Challenge or Remedy Processes Response: This is a risk. There has been public misunderstanding of how the model performs and is used. VA has responded and clarified misconceptions in the public forums in which misunderstanding has arisen.   Accountable & Monitored (AM) - Performance Efficacy and Fairness (e.g. Model/Data Drift, Model Degradation, or inappropriate application)  Response: This is a risk, model performance is degrading over time and an update is in process. However all identified patients are clinically complex and appropriate for clinical attention.  Accountable & Monitored (AM) - User-Introduced Errors Response: There is a low risk, there is a community of practice to support proper implementation of the REACH VET Program, which is supervised and trained by national program leads.   Accountable & Monitored (AM) - System Performance Not as Intended Response: There is a low risk, the internal VA team that manages the model and risk estimates based on it has validation processes in place to review all steps of the process.
55_independent_eval: TRUE
56_monitor_postdeploy: Intermittent and Manually Updated: A plan for monitoring the AI use case is in place, and requires data science teams to work with DevOps engineers to manually update models at scheduled intervals, and create metrics to detect data distribution shifts between the operational environment and the training data for the model. 
57_autonomous_impact: No - Some individual decisions or actions require direct human oversight.
59_ai_notice: Telephone
61_adverse_impact: Yes
62_disparity_mitigation: VA has examined suicide risk concentration among Veterans in VHA care, stratified by patient sex, race and ethnicity.  Model performance was reasonably similar across demographic subgroups. Because the REACH VET clinical program is just one augmentative component of the overall suicide prevention program, weaknesses in the ability of the REACH VET algorithm to identify high risk patients for intervention have been considered in design of other components of the overall suicide prevention program and mitigated where possible.  For example, the REACH VET model lacks information about patients who have had minimal engagement with the health care system for suicide risk factors they may have experienced.  As such, the model will inherently underestimate risk for new patients and patients experiencing new episodes of symptoms or stressors that could elevate risk.  To combat this weakness, VA implemented a universal suicide risk screening process, including at least annual screening of all patients in primary care and mandatory suicide screening in VA emergency departments which are likely to be the entry point for patients with new challenges that are not yet documented in medical records. This screening triggers mandatory follow-up for positive cases by a clinician with mental health care training, who can offer services (as would be done in REACH VET triggered outreach).  As another example, some known risk factors for suicide were not well captured in medical records at the time of model training.  VHA has implemented clinical screening programs for some of these risk factors (e.g. military sexual trauma, food insecurity) and trained clinicians regarding their contribution to risk and the importance of offering services to address challenges.  Likewise, VA decision support systems that provide drill-in summaries of suicide related risks for REACH VET and other VHA patients (i.e. CRISTAL) summarize and display information from these clinical screeners with other information about suicide risk factors to facilitate clinical evaluation and response to suicide risks.  While it is impossible to build a REACH VET model that is not affected by biases in detection of risk factors due to variation in access to/utilization of VA health care, VHA has taken steps to address expected/known gaps in model performance through implementation of other suicide prevention programming.
63_stakeholder_consult: Post-transaction customer feedback collections, Other, Public hearings or meetings
65_appeal_process: Yes
67_opt_out: Yes