Evaluation of an artificial intelligence model for detection of pneumothorax and tension pneumothorax on chest radiograph
Authors: Hillis James M , Bizzo Bernardo C , Mercaldo Sarah , Chin John K , Newbury-Chaet Isabella , Digumarthy Subba R , Gilman Matthew D , Muse Victorine V , Bottrell Georgie , Seah Jarrel CY , Jones Catherine M , Kalra Mannudeep K , Dreyer Keith J 
Published: July 2022

Importance

Early detection of pneumothorax, most often on chest radiograph (CXR), can help determine need for emergent clinical intervention. The ability to accurately detect and rapidly triage pneumothorax with an artificial intelligence (AI) model could assist with earlier identification and improve care.

Objective

This study aimed to compare the accuracy of an AI model (Annalise Enterprise) to consensus thoracic radiologist interpretations in detecting (1) pneumothorax (incorporating both non-tension and tension pneumothorax) and (2) tension pneumothorax.

Design

A retrospective standalone performance assessment was conducted on a dataset of 1,000 CXR cases.

Setting

The cases were obtained from four hospitals in the United States.

Participants

The cases were obtained from patients aged 18 years or older. They were selected using two strategies from all CXRs performed at the hospitals including inpatients and outpatients. The first strategy identified consecutive pneumothorax cases through a manual review of radiology reports and the second strategy identified consecutive tension pneumothorax cases using natural language processing. For both strategies, negative cases were selected by taking the next negative case acquired from the same x-ray machine. The final dataset was an amalgamation of these processes.

Methods

Each case was interpreted independently by up to three radiologists to establish consensus ground truth interpretations. Each case was then interpreted by the AI model for the presence of pneumothorax and tension pneumothorax.

Main Outcome

The primary endpoints were the areas under the receiver operating characteristic curves (AUCs) for the detection of pneumothorax and tension pneumothorax. The secondary endpoints were the sensitivities and specificities for the detection of pneumothorax and tension pneumothorax at predefined operating points.

Results

Model inference was successfully performed in 307 non-tension pneumothorax, 128 tension pneumothorax and 550 negative cases. The AI model detected pneumothorax with AUC of 0.979 (94.3% sensitivity, 92.0% specificity) and tension pneumothorax with AUC of 0.987 (94.5% sensitivity, 95.3% specificity).

Conclusions and Relevance

The assessed AI model accurately detected pneumothorax and tension pneumothorax on this CXR dataset. Its use in the clinical workflow could lead to earlier identification and improved care for patients with pneumothorax.

Key Points

Question

Does a commercial artificial intelligence model accurately detect simple and tension pneumothorax on chest x-ray?

Findings

This retrospective study used 1,000 chest x-rays from four hospitals in the United States to compare artificial intelligence model outputs to consensus thoracic radiologist interpretations. The model detected pneumothorax (incorporating both simple and tension pneumothorax) with area under the curve (AUC) of 0.979 and tension pneumothorax with AUC of 0.987. The sensitivity and specificity were 94.3% and 92.0% respectively for pneumothorax, and 94.5% and 95.3% for tension pneumothorax.

Meaning

This artificial intelligence model could assist radiologists through its accurate detection of pneumothorax.