2_use_case_name: Using Artificial Intelligence to Predict Suicide
3_agency: Department of Veterans Affairs
3_abr: VA
4_bureau: VHA: Veterans Health Administration
8_topic_area: Health & Medical
10_commercial_ai: Searching for information using AI.
11_purpose_benefits: Prevention of Veteran suicide is VHA’s top clinical priority. We use AI to identify patients with two key risk factors for death by suicide: 1) having access to a gun, and 2) having an active opioid use disorder. This project is funded by and developed with input from the Office of Suicide Prevention.

THREE REASONS KNOWING ABOUT ACCESS TO GUNS IS CRITICAL TO SUICIDE PREVENTION
First, when a person attempts suicide using a gun, the death rate is about 85%. For the most common method of attempting suicide—drug overdose—the death rate is less than 5%. Second, Veterans are more likely to own a gun and know better how to use it than other patients, so Veterans’ death rate when attempting suicide using a gun is even higher. Third, there are things clinicians can do to protect Veterans from their guns while they are suicidal, such as counselling them to give the guns or the ammunition to friends or family until they feel better.  This is always voluntary, but Veterans very often agree to it, because they see and fear the risk. 

WHY KNOWING WHICH PATIENTS HAVE ACTIVE OPIOID USE IS CRITICAL TO SUICIDE PREVENTION
Patients who die by suicide almost always have a prior history of emotional and mental health difficulties. When they die by suicide, it’s usually not due to that history, but rather because some bad life event (such as breaking up with their spouse) happens and then they have an overwhelming impulse to end their lives. 

Opioid use disorder (OUD) contributes to suicide risk in two ways. First, it makes bad life events much more common. It’s hard to keep a job or have good relationships with OUD. Second, OUD also increases impulsivity. That is, among patients who have just lost their jobs, those with OUD are more likely to have the impulse to end their lives. (Among substance use disorders, we focus on OUD because it is very common and there are good treatments available.)


WHY USE AI TO DETERMINE ACCESS TO GUNS?
The need for AI arise from how information is stored in an electronic health record. For historical reasons, when a clinician sees a patient, there are codes they can use to record what diseases (e.g., diabetes) the patient has, but there is no code for having access to a gun. VHA has considered asking all patients during intake for each visit whether they have access to a gun, but this is time consuming and can cause concern among patients about why they’re always being asked about guns. 

AI, specifically natural language processing (NLP) of clinical notes, offers another way to know which patients have guns. We have developed NLP algorithms that are very good at assessing from notes whether a patient owns or has access to a gun. But we use only notes already written by clinicians, any mention of gun access came voluntarily from the patient. However, to find this information, we need a method that can go through notes quickly, and NLP provides that method. 

WHY USE AI TO ASSESS WHETHER PATIENTS HAVE SUBSTANCE USE DISORDERS?
OUD is a disease, and there are disease codes available to record it. However, because there are many different states you can be in with OUD (e.g., actively using vs in rehab vs sober for years), there are over 100 possible ways to code OUD. Most clinical settings (like outpatient clinics and ERs) have short list of commonly used codes for clinicians to pick from, but that may not offer up much specificity. So most clinics will only put on their list a single code, “opioid use, unspecified” and leave off all the more specific codes. Thus, typical coding practice does not differentiate between a patient who is currently using opioids or one who is 10 years sober (and, if you live along time with OUD, it is usually because you spend much of that time sober). That differentiation is important because the increase in impulsivity with OUD is only during active opiate use.

NLP can again be useful, because descriptions of OUD in clinical notes are likely to be much more specific than the codes used
12_outputs: OUR INPUT, OUTPUT, AND INTENDED USER In the “Please add anything else you would like to share” box below we describe in more detail why it is necessary to use AI to find information about gun access or OUD (versus either not having the information or getting that information without using AI). We also describe how we build and test AI models to determine whether a patient has gun access or is actively using opioids. First, however, we think it is important to describe how human clinicians in VHA will use our AI.   The INPUT to our AI models is simply the notes that have already been written on patients. These only contain information volunteered by the patient. The OUTPUT of our models will be simply indicators that a patient **may** have access to a gun or **may** have an active substance use disorder, along with a presentation of the text that leads the AI algorithm to reach its conclusion.   However, the INTENDED USE of the output is that a human mental health clinician be notified about patients at risk, then read the note or notes that led to the determination that the patient was at risk. The clinician may conclude that the AI was incorrect (this occurs about 12-16% of the time with our current models) and drop the issue. Alternatively, the clinician may decide that the patient does have the risk factor and that the risk factor needs to be addressed.   There is no planned interaction between the AI and the patient, nor does the AI make any clinical decisions. Rather, it is a tool to help point clinicians to potentially critical information that they might not otherwise have been aware of.   OUR AI METHOD We begin by creating a list of common terms centered around gun access or OUD. We then pull thousands of pieces of clinical text with these terms in them, and have physicians determine whether the risk factor is present or not. (A term can be present, but the patient can still not have the risk factor in a variety of ways. It could be that “the patient’s brother has a gun” or “the patient has NO gun” or “the patient used to use Fentanyl, but quit”.) The text that has been assessed by physicians is called “labeled data”. We then give our NLP models some of the labeled data, so they can learn what positive and negative examples look like.   For each risk factor, we develop 4 non-neural AI text classification models: random forest, bagging, gradient boosting, and logistic regression with ridge penalization. We also implement two large language models, BioBERT and Bio-ClinicalBERT.   Subsequently, we test each algorithm on other data for which we have (but don’t give the algorithm) the label and ask each algorithm to produce its own labels. We then compare each algorithm’s conclusions to the physicians’ conclusions. That is how we know that our algorithms so far, will be incorrect about 12-16% of the time. Over time, we hope to improve the performance of these algorithms. For example, the large language models are relatively new additions to the AI toolkit and perform better than the non-neural models. We expect further advances over time. However, the current level of accuracy is already higher than many commonly used diagnostic tests, such as mammograms or erythrocyte sedimentation rates.   WHERE DO WE DO THIS WORK? Our team currently uses the Summit Data Platform (SDP) – formerly called the Health Data and Analytics Platform (HDAP) – maintained by the VA Office of Information Technology. This is an Azure Machine Learning environment.
16_dev_stage: Acquisition and/or Development
17_impact_type: Both
18_date_initiated: 10/1/2021
19_date_acq_dev_began: 10/1/2021