Automated classification of psychotherapy note text: implications for quality assessment in PTSD care

In recent years, studies have attempted to use various methods to characterize the quality of care for post-traumatic stress disorder (PTSD) delivered in United States Veterans Administration (VA) outpatient clinics. Dieperink et al. used manual chart review to characterize care for 150 veterans at three VA medical centres during the 2001 fiscal year [1]. They found wide variation in the types of social services, psychotherapy and pharmacotherapy received in the 6 months following entry into specialized PTSD programmes. For example, clinics in Minneapolis and Memphis tended to provide pharmacotherapy while clinics in Boston tended to provide psychotherapy. As a result, there was wide variation in the amount of care received; veterans in Minneapolis received an average of seven mental health contacts per year (visits with a psychiatrist, psychologist, social worker or other mental health practitioner) while veterans in Boston received an average of 16 mental health contacts per year. Although it was not clear which approach was superior, it was clear that PTSD care was not standardized among the three VA facilities. In order to include more sites and be able to generalize across sites, further studies on the quality of care for PTSD in the VA have attempted to use national administrative data rather than chart review to capture information about the process of care. This work has relied on a combination of Current Procedural Technology (CPT) codes, International Classification of Diseases, Ninth Revision (ICD-9) codes, and pharmacy data. Cully et al. used administrative data to examine the receipt of psychotherapy in the VA nationally for the 12 months following initial diagnoses of PTSD, anxiety and depression in the 2004 fiscal year [2]. While the 77 743 veterans with new diagnoses of PTSD had a higher chance of receiving psychotherapy than veterans with anxiety or depression, the amount of psychotherapy was still very low; only 10% received an adequate number of sessions (defined as eight in this study), and the median wait to start psychotherapy was 50 days. Two studies using similar methods were published in 2010. Using stricter inclusion criteria, Spoont et al. examined care for 20 284 veterans with a new diagnosis of PTSD from the 2004 mid-fiscal year through the 2005 mid-fiscal year [3]. They evaluated whether veterans received a large enough medication supply that they could have gotten an adequate trial of pharmacotherapy or whether they received enough psychotherapy visits that they could have received an adequate trial of psychotherapy (defined, again, as eight visits). Based on this resource utilization, they concluded that, at most 33%, of veterans could have received an adequate trial of evidence-based treatment for PTSD. Seal et al. examined all mental health visits over the year following a new PTSD diagnosis in veterans returning from Iraq and Afghanistan [4]. The study included 49 425 veterans enrolling in VA care from the 2002 mid-fiscal year through the 2008 mid-fiscal year. They asserted that delivery of evidence-based psychotherapies endorsed in the VA Mental Health Uniformed Services Package [5] (prolonged exposure [6] and cognitive processing therapy [7]) required at least nine sessions over 15 weeks and found that only 9.5% received this level of service. A key limitation in these three studies is that they tell us only the best possible scenario about the amount of psychotherapy that could have been delivered based on the number of visits – we do not know whether veterans actually received psychotherapy during these visits. Therefore, it is possible that these studies overestimate the amount of psychotherapy actually delivered to veterans with PTSD. Researchers and policy makers wishing to understand care delivery for PTSD are left with a dilemma. Manual medical record review can generate detailed information about clinical processes, including psychotherapists’ reports of the specific techniques they used in a session. However, the method is time-consuming and difficult to apply on a large scale. Administrative review techniques are applicable on a large scale, but are limited in the granularity of the information they provide. We learn how much of a given service practitioners report providing, but because we do not read the notes, we have little information about the content of those services. Automated text-based information retrieval technologies, such as natural language processing (NLP) have the potential to bridge this gap by extracting detailed information found in a medical record review on the larger scale permitted by administrative review. NLP is an effort to have computers draw specific information from free text. The application of NLP has traditionally been limited by the need to customize programming for each new application. However, modern NLP applications can use a technique known as machine learning to ‘teach’ a computer to recognize patterns in documents [8,9]. Through the recognition of language patterns within a document the NLP application can help users make inferences about the content of the text. We sought to understand whether using administrative data to determine the number of psychotherapy sessions veterans receive is equivalent to manual medical records review. We thought it was possible that psychotherapy billing codes might sometimes be misapplied to other services delivered by psychotherapy-oriented practitioners, such as psychologists and social workers. These might include intakes, psychological testing and case management. Alternatively, administrative data review might be accurate, making manual or automated review of note text an unnecessary method. Our primary hypothesis was that administrative data overestimates the number of psychotherapy sessions delivered to veterans when compared to manual chart review (as some sessions administratively coded as psychotherapy are actually used for other purposes). Our secondary hypothesis was that if administrative data review was inaccurate, our manual medical record review could be approximated using an automated NLP programme, creating the potential for a more accurate method to be efficiently applied to large-scale treatment studies.