Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool
Authors: Levis Brooke , Bhandari Parash Mani , Neupane Dipika , Fan Suiqiong , Sun Ying , He Chen , Wu Yin , Krishnan Ankur , Negeri Zelalem , Imran Mahrukh , Rice Danielle B. , Riehm Kira E. , Azar Marleine , Levis Alexander W. , Boruff Jill , Cuijpers Pim , Gilbody Simon , Ioannidis John P. A. , Kloda Lorie A. , Patten Scott B. , Ziegelstein Roy C. , Harel Daphna , Takwoingi Yemisi , Markham Sarah , Alamri Sultan H. , Amtmann Dagmar , Arroll Bruce , Ayalon Liat , Baradaran Hamid R. , Beraldi Anna , Bernstein Charles N. , Bhana Arvin , Bombardier Charles H. , Buji Ryna Imma , Butterworth Peter , Carter Gregory , Chagas Marcos H. , Chan Juliana C. N. , Chan Lai Fong , Chibanda Dixon , Clover Kerrie , Conway Aaron , Conwell Yeates , Daray Federico M. , de Man-van Ginkel Janneke M. , Fann Jesse R. , Fischer Felix H. , Field Sally , Fisher Jane R. W. , Fung Daniel S. S. , Gelaye Bizu , Gholizadeh Leila , Goodyear-Smith Felicity , Green Eric P. , Greeno Catherine G. , Hall Brian J. , Hantsoo Liisa , Härter Martin , Hides Leanne , Hobfoll Stevan E. , Honikman Simone , Hyphantis Thomas , Inagaki Masatoshi , Iglesias-Gonzalez Maria , Jeon Hong Jin , Jetté Nathalie , Khamseh Mohammad E. , Kiely Kim M. , Kohrt Brandon A. , Kwan Yunxin , Lara Maria Asunción , Levin-Aspenson Holly F. , Liu Shen-Ing , Lotrakul Manote , Loureiro Sonia R. , Löwe Bernd , Luitel Nagendra P. , Lund Crick , Marrie Ruth Ann , Marsh Laura , Marx Brian P. , McGuire Anthony , Mohd Sidik Sherina , Munhoz Tiago N. , Muramatsu Kumiko , Nakku Juliet E. M. , Navarrete Laura , Osório Flávia L. , Pence Brian W. , Persoons Philippe , Petersen Inge , Picardi Angelo , Pugh Stephanie L. , Quinn Terence J. , Rancans Elmars , Rathod Sujit D. , Reuter Katrin , Rooney Alasdair G. , Santos Iná S. , Schram Miranda T. , Shaaban Juwita , Shinn Eileen H. , Sidebottom Abbey , Simning Adam , Spangenberg Lena , Stafford Lesley , Sung Sharon C. , Suzuki Keiko , Tan Pei Lin Lynnette , Taylor-Rowan Martin , Tran Thach D. , Turner Alyna , van der Feltz-Cornelis Christina M. , van Heyningen Thandi , Vöhringer Paul A. , Wagner Lynne I. , Wang Jian Li , Watson David , White Jennifer , Whooley Mary A. , Winkley Kirsty , Wynter Karen , Yamada Mitsuhiko , Zeng Qing Zhi , Zhang Yuying , Thombs Brett D. , Benedetti Andrea ,   Depression Screening Data (DEPRESSD) PHQ Group
Published: November 2024

Importance

Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates.

Objective

To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates.

Design, Setting, and Participants

This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled.

Main Outcomes and Measures

For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population.

Results

The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes.

Conclusions and Relevance

This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.