Venue: EMNLP 2020

# CheXbert: Combining Automatic Labelers and Expert Annotations for

Accurate Radiology Report Labeling Using BERT

 Akshay Smit 1

Stanford University

akshaysm@stanford.edu

&Saahil Jain *

Stanford University

saahl.jain@stanford.edu

&Pranav Rajpurkar *

Stanford University

pranavsr@cs.stanford.edu

Anuj Pareek

Stanford University &Andrew Y. Ng

Stanford University &Matthew P. Lungren

Stanford University

Equal contribution

###### Abstract

The extraction of labels from radiology text reports enables large-scale training of medical imaging models. Existing approaches to report labeling typically rely either on sophisticated feature engineering based on medical domain knowledge or manual annotations by experts. In this work, we introduce a BERT-based approach to medical image report labeling that exploits both the scale of available rule-based systems and the quality of expert annotations. We demonstrate superior performance of a biomedically pretrained BERT model first trained on annotations of a rule-based labeler and then fine-tuned on a small set of expert annotations augmented with automated backtranslation. We find that our final model, CheXbert, is able to outperform the previous best rule-based labeler with statistical significance, setting a new SOTA for report labeling on one of the largest datasets of chest x-rays.

## 1 Introduction

The extraction of labels from radiology text reports enables important clinical applications, including large-scale training of medical imaging models Wang et al. (2017). Many natural language processing systems have been designed to label reports using sophisticated feature engineering of medical domain knowledge Pons et al. (2016). On chest x-rays, the most common radiological exam, rule-based methods have been engineered to label some of the largest available datasets Johnson et al. (2019). While these methods have generated considerable advances, they have been unable to capture the full diversity of complexity, ambiguity and subtlety of natural language in the context of radiology reporting.

More recently, Transformers have demonstrated success in end-to-end radiology report labeling Drozdov et al. (2020); Wood et al. (2020). However, these methods have shifted the burden from feature engineering to manual annotation, requiring considerable time and expertise for high quality. Moreover, these methods do not take advantage of existing feature-engineered labelers, which represent state-of-the-art on many medical tasks.

We introduce a simple method for gaining the benefits of both existing radiology report labelers and expert annotations to achieve highly accurate automated radiology report labeling. This approach begins with a biomectically pretrained BERT model Devlin et al. (2019); Peng et al. (2019) trained on the outputs of an existing labeler, and performs further fine-tuning on a small corpus of expert annotations augmented with automated backtranslation. We apply this approach, shown in Figure 1, to the task of radiology report labeling of chest x-rays, and call our resulting model _CheXpert_.

CheXbert outperforms the previous best reported labeler Irvin et al. (2019) on an external dataset,

Figure 1: We introduce a method for radiology report labeling, in which a biomedically pretrained BERT model is first trained on annotations of a rule-based labeler, and then fine-tuned on a small set of expert annotations augmented with automated backtranslation.



MIMIC-CXR (Johnson et al., 2019), with an improvement of 0.055 (95% CI 0.039, 0.070) on the F1 metric, and is only 0.007 F1 away from a radiologist performance benchmark. We expect this method of training medical report labelers is broadly useful for natural language processing within the medical domain, where collection of expert labels is expensive, and feature engineered labelers already exist for many tasks.

## 2 Related Work

Many natural language processing systems have been developed to extract structured labels from free-text radiology reports (Pons et al., 2016; Yadav et al., 2016; Hassanpour et al., 2017; Annarumma et al., 2019; Savova et al., 2010; Wang et al., 2018; Chen et al., 2018; Bozkurt et al., 2019). In many cases, these methods have relied on heavy feature engineering that include controlled vocabulary and grammatical rules to find and classify properties of radiological findings. NegEx Chapman et al. (2001), a popular component of rule-based methods, uses simple regular expressions for detecting negation of findings and is often used in combination with ontologies such as the Unified Medical Language System (UMLS) (Bodenreider, 2004). NegBio Peng et al. (2017), an extension to NegEx, utilizes universal dependencies for pattern definition and subgraph matching for graph traversal search, includes uncertainty detection in addition to negation detection for multiple pathologies in chest x-ray reports, and is used to generate labels for the ChestX-Ray14 dataset (Wang et al., 2017).

The CheXpert labeler Irvin et al. (2019) improves upon NegBio on chest x-ray report classification through more controlled extraction of mentions and an improved NLP pipeline and rule set for uncertainty and negation extraction. The CheXpert labeler has been applied to generate labels for the CheXpert dataset and MIMIC-CXR (Johnson et al., 2019), which are amongst the largest chest x-ray datasets publicly available.

Deep learning approaches have also been trained using expert-annotated sets of radiology reports Xue et al. (2019). In these cases, training set size, often driving the performance of deep learning approaches, is limited by radiologist time and expertise. Chen et al. (2017) trained CNNs with GloVe embeddings Pennington et al. (2014) on 1000 radiologist-labeled reports for classification of pulmonary embolism in chest CT reports and improved upon the previous rule-based SOTA, peFinder Chapman et al. (2011). Bustos et al. (2019) trained both recurrent and convolutional networks in combination with attention mechanisms on 27,593 physician-labeled radiology reports and apply their labeler to generate labels. More recently, Transformer-based models have also been applied to the task of radiology report labeling. Drozdov et al. (2020) trained classifiers using BERT Devlin et al. (2019) and XLNet Yang et al. (2020) on 3,856 radiologist labeled reports to detect normal and abnormal labels. Wood et al. (2020) developed ALARM, an MRI head report classifier on head MRI data using BioBERT Lee et al. (2019) models trained on 1500 radiologist-labeled reports, and demonstrate improvement over simpler fixed embedding and word2vec-based Mikolov et al. (2013) models Zech et al. (2018).

Our work is closely related to approaches to reduce the number of expert annotations required for training medical report labelers Callahan et al. (2019); Ratner et al. (2020); Banerjee et al. (2018). A method of weak supervision known as data programming Ratner et al. (2018) has seen successful application to medical report labeling: in this method, users write heuristic labelling functions that programmatically label training data. Saab et al. (2019) used data programming to incorporate labeling functions consisting of regular expressions that look for phrases in radiology reports, developed with the help of a clinical expert in a limited time window, to label for intracranial hemorrhage in head CTs. Dunnmon et al. (2019) demonstrated that in under 8 hours of cumulative clinician time, a data programming method can approach the efficacy of large hand-labeled training sets annotated over months or years for training medical imaging models, including chest x-ray classifiers on the task of normal / abnormal detection. Beyond data programming approaches, Drozdov et al. (2020) developed a fully unsupervised approach utilizing Siamese Neural Network and Gaussian Mixture Models, reporting performance similar to the CheXpert labeler without requiring any radiologist-labeled reports on the simplified task of normal / abnormal detection. Concurrently developed to our work is the CheXpert++ labeler McDermott et al. (2020), which was trained on the outputs of the rule-based CheXpert labeler and showed improved performance after a single additional epoch of fine-tuning on expert-labeled report sentences 



## 3 Methods

### Task

The report labeling task is to extract the presence of one or more clinically important observations (e.g. consolidation, edema) from a free-text radiology report. More formally, a labeler takes in as inputs sentences from a radiology report and outputs for 13 observations one of the following classes: blank, positive, negative, and uncertain. For the 14th observation corresponding to _No Finding_, the labeler only outputs one of the two following classes: blank or positive.

### Data

Two existing large datasets of chest x-rays, CheXpert Irvin et al. (2019) (consisting of 224,316 images), and MIMIC-CXR Johnson et al. (2019) (consisting of 377,110 images) are used in this study. Both datasets have corresponding radiology reports that have been labeled for the same set of 14 observations using the CheXpert labeler Irvin et al. (2019), from the _Impression_ section, or other parts of the radiology report.

A subset of both datasets also contain manual annotations by expert radiologists. On CheXpert, a total of 1000 reports (_CheXpert manual set_) were reviewed by 2 board certified radiologists with disagreement resolution through consensus. On MIMIC-CXR, a total of 687 reports (_MIMIC-CXR test set_) were reviewed by 2 board certified radiologists and manually labeled for the same 14 medical observations as in CheXpert. In this study, CheXpert is used for the development of models, and the MIMIC-CXR test set is used for evaluation.

Some reports from the same patient appear multiple times in the CheXpert dataset. Removing duplicate reports as well as the CheXpert manual set from the CheXpert dataset results in 190,460 reports, the class prevalences for which are shown in Table B1 of the Appendix. We remove excess spaces and newlines from all reports.

### Model Architecture

All models use a modification of the BERT-base architecture Devlin et al. (2019) with 14 linear heads (as shown in Figure 2): 12 heads correspond to various medical abnormalities, 1 to medical support devices, and 1 to 'No Finding'. Each radiology report text is tokenized, and the maximum number of tokens in each input sequence is capped at 512. The final-layer's hidden state corresponding to the CLS token is then fed as input to each of the linear heads.

### Training Details

For all our models, unless otherwise specified, we fine-tune all layers of the BERT model, including the embeddings, and feed the CLS token into the 14 linear heads to generate class scores for each medical observation. BERT-Base contains \(\sim 110\) million parameters, and the linear heads contain \(\sim 40,000\) parameters.

All models are trained using cross-entropy loss and Adam optimization with a learning rate of \(2\times 10^{-5}\), as used in Devlin et al. (2019) for fine-tuning tasks. The cross-entropy losses for each of the 14 observations are added to produce the final loss. During training, we periodically evaluate our model on the dev set and save the checkpoint with the highest performance averaged over all 14 observations. All models are trained using 3 TITAN-XP GPUs with a batch size of 18.

### Evaluation

Models are evaluated on their average performance on three retrieval tasks: positive extraction, negative extraction, and uncertainty extraction. For each of the tasks, the class of interest (e.g. negative for the negative extraction and uncertain for the uncertainty extraction) is treated as the positive class, and the other classes are considered negative. For each of the 14 observations, we compute a weighted average of the F1 scores on each of the above three tasks, weighted by the support for each

Figure 2: Model architecture. The model contains 14 linear heads, one for each medical observation, but only 3 heads are shown here.



class of interest, which we call the _weighted-F1_ metric, henceforth simply abbreviated to F1.

We report the simple average of the F1 across all of the observations. We include the 95% two-sided confidence intervals of the F1 using the non-parametric percentile bootstrap method with 1000 bootstrap replicates Efron and Tibshirani (1986).

## 4 Experiments

### Supervision Strategies

We investigate models trained using three strategies: trained only on radiologist-labeled reports, trained only on labels generated automatically by the CheXpert labeler Irvin et al. (2019), and trained on a combination of the two.

Radiologist LabelsT-rad is obtained by training the model on the CheXpert manual set, fine-tuning all weights. As baselines, we also train models that freeze all weights in the BERT layers, and only update the weights in the linear heads: **T.cls-rad** is identical to T-rad in architecture, while **T.token-rad** averages the non-padding output tokens as the input into the linear heads rather than using the CLS token output. All models are trained using a random 75%-25% train-dev split on the CheXpert manual set, and are trained until convergence.

Automatic LabelsT-auto is obtained using labels generated by the rule-based CheXpert labeler, described in Irvin et al. (2019). T-auto is trained using a random 85%-15% train-dev split of the CheXpert dataset, different from the models trained on radiologist labels. T-auto is trained for 8 epochs, since slightly higher dev performance is observed compared to the typical 2-4 epochs for BERT fine-tuning tasks.

Hybrid LabelsT-hybrid is obtained by initializing a model with the weights of T-auto, and then fine-tuning it on radiologist-labeled reports, as for T-rad.

ResultsAs shown in Table 1, T-rad achieves an F1 of \(0.705~{}(0.680,0.725)\), significantly higher than the performance of the baselines with T.cls-rad at \(0.286~{}(0.265,0.305)\), and T.token-rad at \(0.396~{}(0.374,0.416)\). T-auto achieves a higher F1 of \(0.755~{}(0.731,0.774)\). Superior performance is obtained by T-hybrid, with an F1 of \(0.775~{}(0.753,0.795)\).

### Biomedical Language Representations

We investigate the effect of having models pretrained on biomedical data. For the following models, we use an identical training procedure to T-rad, but initialize the weights differently. **Thoirad** is obtained by using BioBERT weight initializations Lee et al. (2019). BioBERT was obtained by further pretraining the BERT weights on a large biomedical corpus comprising PubMed abstracts (4.5 billion words) and PMC full-text articles (13.5 billion words). **Tclinical-rad** is obtained by using Clinical BioBERT weight initializations Alsentzer et al. (2019), which were obtained by further pretraining the BioBERT weights on 2 million clinical notes from the MIMIC-III database. Finally, **Table-rad** is obtained by using BlueBERT, a BERT model pretrained on PubMed abstracts and clinical notes (MIMIC-III) Peng et al. (2019).

ResultsAs shown in Table 1, Tbio-rad achieves an F1 of \(0.616~{}(0.587,0.639)\) and Tclinical-rad achieves an F1 of \(0.677~{}(0.651,0.699)\), lower than T-rad. However, Tblue-rad achieves an F1 of \(0.741~{}(0.714,0.763)\), higher than T-rad. The drop in performance with Tbio-rad and Tclinical-rad may possibly be attributed to using different vocabulary, sequence length, and other configurations (stopping procedure, embedding dimensions) than those used by Tblue-rad, which uses the configurations provided in Devlin et al. (2019).

### Data Augmentation using Backtranslation

We investigate the use of backtranslation to improve the performance of the models. Backtranslation is designed to generate alternate formulations of sentences by translating them to another language and back. Although backtranslation has been successfully used to augment text data in a variety of NLP tasks Yu et al. (2018); Poncelas et al. (2018), to our knowledge, the technique is yet to be applied to a medical report extraction task. In this experiment, we augment the CheXpert manual set using Facebook-FAIR's winning submission to the WMT'19 news translation task Ng et al. (2019) to generate backtranslations. Although this submission includes models that produce German/English and Russian/English translations, initial experiments with Russian did not demonstrate semantically correct translations, so we only continued experiments with German. We use beam search with a beam size of 1 to select the single 

most likely translation. We perform this experiment using our best models: **Tblue-rad-bt** is obtained by using an identical training procedure to Tblue-rad on the augmented dataset (which is twice the size of the CheXpert manual set). **Tblue-hybrid-bt** is obtained by first training a BlueBERT-based labeler on automatically generated CheXpert labels, and then fine-tuning on radiologist-labeled reports of the CheXpert manual set, augmented by backtranslation. We also report the performance of T-rad-bt and T-hybrid-bt.

ResultsAs shown in Table 1, T-rad-bt achieves an F1 score of \(0.729~{}(0.702,0.749)\), higher than that of T-rad. Similarly, T-hybrid-bt achieves an F1 of \(0.795~{}(0.772,0.815)\). Tblue-rad-bt achieves an F1 of \(0.770~{}(0.747,0.790)\), higher than that of the CheXpert labeler. Tblue-hybrid-bt achieves a superior F1 score of \(0.798~{}(0.775,0.816)\).

### Comparison to previous SOTA and radiologist benchmark

We compare the performance of our best model to the previous best reported labeler, the CheXpert labeler Irvin et al. (2019), and to a radiologist benchmark. CheXpert is an automated rule-based labeler that extracts mentions of conditions like pneumonia by searching against a large manually curated list of words associated with the condition and then classifies mentions as uncertain, negative, or positive using rules on a universal dependency

\begin{table}
\begin{tabular}{l l c}  & Model & F1 (95\% CI) \\ \hline \multirow{3}{*}{Training Strategy} & T-rad & 0.705 (0.680, 0.725) \\  & T.cls-rad & 0.286 (0.265, 0.305) \\  & T.token-rad & 0.396 (0.374, 0.416) \\  & T-auto & 0.755 (0.731, 0.774) \\  & T-hybrid & 0.775 (0.753, 0.795) \\ \hline \multirow{3}{*}{Biomedical Representations} & Tbio-rad & 0.616 (0.587, 0.639) \\  & Tclinical-rad & 0.677 (0.651, 0.699) \\  & Tblue-rad & 0.741 (0.714, 0.763) \\ \hline \multirow{3}{*}{With Backtranslation Augmentation} & T-rad-bt & 0.729 (0.702, 0.749) \\  & T-hybrid-bt & 0.795 (0.772, 0.815) \\ \cline{1-1}  & Tblue-rad-bt & 0.770 (0.747, 0.790) \\ \cline{1-1}  & **Tblue-hybrid-bt (CheXbert)** & 0.798 (0.775, 0.816) \\ \hline Previous SOTA & CheXpert & 0.743 (0.719, 0.764) \\ \hline Benchmark & Radiologist & 0.805 (0.784, 0.823) \\ \end{tabular}
\end{table}
Table 1: Average F1 score with 95% confidence intervals for all our models, with comparisons to CheXpert labeler and radiologist benchmark.

\begin{table}
\begin{tabular}{l l l} Category & CheXbert & Improvement over CheXpert \\ \hline Pneumonia & 0.835 (0.789, 0.881) & 0.151 (0.093, 0.206) \\ Fracture & 0.791 (0.665, 0.895) & 0.120 (0.019, 0.236) \\ Consolidation & 0.877 (0.810, 0.935) & 0.105 (0.029, 0.192) \\ Enlarged Cardiom. & 0.713 (0.623, 0.783) & 0.100 (0.038, 0.166) \\ No Finding & 0.640 (0.482, 0.759) & 0.097 (0.007, 0.182) \\ Pleural Other & 0.534 (0.372, 0.671) & 0.056 (0.008, 0.124) \\ Cardiomegaly & 0.815 (0.759, 0.860) & 0.051 (0.018, 0.086) \\ Pneumothorax & 0.928 (0.892, 0.960) & 0.046 (0.015, 0.076) \\ Atelectasis & 0.940 (0.910, 0.971) & 0.023 (-0.001, 0.051) \\ Support Devices & 0.888 (0.856, 0.919) & 0.021 (0.004, 0.040) \\ Edema & 0.881 (0.843, 0.916) & 0.017 (-0.007, 0.042) \\ Pleural Effusion & 0.919 (0.892, 0.947) & 0.014 (-0.005, 0.034) \\ Lung Lesion & 0.664 (0.550, 0.771) & -0.019 (-0.098, 0.056) \\ Lung Opacity & 0.741 (0.684, 0.792) & -0.021 (-0.056, 0.006) \\ \hline Average & 0.798 (0.775, 0.816) & 0.055 (0.039, 0.070) \\ \end{tabular}
\end{table}
Table 2: The F1 scores for CheXbert as well as improvements over the CheXpert labeler on the MIMIC-CXR test set, in descending order of improvement, and reported with 95% confidence intervals.

 

parse of the report. For the radiologist benchmark, the annotations by one of the 2 radiologists on the MIMIC-CXR test set is used, while the other is used as ground truth. We report the improvement of our best model, Tblue-hybrid-bt, which we also call **CheXbert**, over the CheXpert labeler by computing the paired differences in F1 scores on 1000 bootstrap replicates and provide the mean difference along with a 95% two-sided confidence interval.

ResultsWe observe that CheXbert has a statistically significant improvement \((p<0.001)\) over the existing SOTA, CheXpert, which achieves a score of \(0.743~{}(0.719,0.764)\). Notably, we also find that Tblue-rad-bt, the best model trained only on manually labeled radiology reports, performs at least as well as the CheXpert labeler.

Table 2 shows the F1 per class (along with 95% confidence intervals) for CheXbert and for the improvements over CheXpert. CheXbert records an improvement in all but 2 medical conditions, and a statistically significant improvement in 9 of the 14 conditions. The largest improvements are observed for Pneumonia [\(0.151~{}(0.093,0.206)\)], Fracture [\(0.120~{}(0.019,0.236)\)], Consolidation [\(0.105~{}(0.029,0.192)\)], Enlarged Cardiomediastinum [\(0.100~{}(0.038,0.166)\)], and No Finding [\(0.097~{}(0.007,0.182)\)]. Further significant improvements are observed for Pleural Other [\(0.056~{}(0.008,0.124)\)], Cardiomegaly [\(0.051~{}(0.018,0.086)\)], Pneumothorax [\(0.046~{}(0.015,0.076)\)] and Support Devices [\(0.021~{}(0.004,0.040)\)]. Overall, CheXbert achieves a statistically significant improvement on F1 of \(0.055~{}(0.039,0.070)\). The board-certified radiologist achieves an F1 of \(0.805~{}(0.784,0.823)\), which is \(0.007\) F1 points higher than the performance of CheXbert.

Training timesFor all our models except the baselines, training on radiologist-labeled reports takes \(\sim 30\) minutes, training on the radiologist-labeled reports augmented via backtranslation takes \(\sim 50\) minutes. Training on the larger automatically labeled report set takes \(\sim 7\) hours.

Inference timesWe benchmark the time taken by CheXbert and CheXpert to label all 190,460 report impressions in the CheXpert dataset. On a system with 32GB RAM and 1 CPU core, the CheXbert model takes \(\sim 3.7\) hours. This is an order of magnitude faster than the \(36\) hours required for CheXpert. With a single TITAN-XP GPU, the CheXbert model's inference time reduces to \(\sim 18\) minutes.

## 5 Analysis

### T-auto versus CheXpert

We analyze whether T-auto, which is trained exclusively on labels from CheXpert (a rules-based labeler), can generalize beyond those rules.

We analyze specific examples in the CheXpert manual test set which T-auto correctly labels but CheXpert mislabels. On one example, T-auto is able to correctly detect uncertainty expressed in the phrase "cannot be entirely excluded," which CheXpert is not able to detect because the phrase does not match any rule in its ruleset. Similarly, on another example containing "no evidence of pneumothorax or bony fracture," T-auto correctly labels fracture as negative, while CheXpert labels fracture as positive since the phrasing does match any negation construct part of its ruleset. T-auto, in contrast to CheXpert, also recognizes conditions with misspellings in the report like "carionegaly" in place of "cardiomegaly" and "mediastnium" in place of "mediastinum". Examples of T-auto correctly labeling conditions mislabeled by CheXpert are provided in Table B4 of the Appendix. Table B5 of the Appendix contains examples of CheXpert correctly labeling conditions mislabeled by T-auto. An example of each case is shown in the top two rows of Table 3.

### CheXbert versus T-auto and CheXpert

We analyze how CheXbert improves on T-auto and CheXpert using examples which CheXbert correctly labels but T-auto and CheXpert incorrectly label.

CheXbert is able to correctly detect conditions that CheXpert and T-auto are not able to. On one example, T-auto and CheXpert both mislabel a "mildly enlarged heart" as blank for cardiomegaly, while CheXbert correctly labels it positive. On another containing "Right hilum appears slightly more prominent" (an indicator for enlarged cardiodiestampin), CheXpert correctly classifies enlarged cardiomediatsintium as positive, while T-auto and CheXpert do not detect the condition.

Furthermore, CheXbert correctly labels nuanced expressions of negation that both CheXpert and T-auto mislabel. On the example containing "heart size is slightly larger but still within nor 

mal range," CheXpert and T-auto mistakenly label cardiomegaly as positive, while CheXbert correctly labels cardiomyocyte as negative. On another example containing the phrase "interval removal of PICC lines", CheXpert and T-auto detect "PICC lines" as an indication of a support device but are unable to detect the negation indicated by "removal", which CheXbert correctly does.

Additionally, CheXbert is able to correctly detect expressions of uncertainty that both CheXpert and T-auto mislabel. On an example containing "new bibasilar opacities, which given the clinical history are suspicious for aspiration," CheXpert correctly identifies lung opacity as positive while CheXpert and T-auto incorrectly detect uncertainty (associating "suspicious" as a descriptor of "opacities"). More examples which CheXbert correctly labels but CheXpert and T-auto mislabel can be found in Table B6 of the Appendix. A selected example is shown in the last row of Table 3.

### Report Changes with Backtranslation

We analyze the phrasing and vocabulary changes that backtranslation introduces into the reports. Backtranslation frequently rephrases text. For instance, the sentence "reddemonstration of multiple right-sided rib fractures" is backtranslated to "red demonstration of several rib fractures of the right side". Backtranslation also introduces some error: the phrase "left costophrenic angle" is backtranslated to "left costophrine angle" ("costophrine" is not a word), and the phrase "left anterior chest wall spacer in place" is backtranslated to "pacemaker on the left front of the chest wall", which omits the critical attribute of being in place. In many examples, the backtranslated text paraphrases medical vocabulary into possible semantic equivalents: "cutaneous" becomes "skin", "left clavicle" becomes "left collarbone", "osseous" becomes "bone" or "bony", "anterior" becomes "front", and "rib fracture" becomes "broken ribs". More backtranslations with analyses are provided in Table B7 of the Appendix. Additionally, a physician validated that the backtranslation outputs used correct radiology language and maintained the semantics of the original report. The results are provided in Table A1 of the Appendix.

## 6 Limitations

Our study has several limitations. First, our hybrid/auto approaches require an already-existing labeler. Second, our report labeler has a maximum input token size of 512 tokens, but this may be easily extended to work with longer medical/radiology

\begin{table}
\begin{tabular}{|c|p{140.9pt}|} \hline Report Segment and Labels & Reasoning \\ ...two views of chest demonstrate _carimomegaly_ with no focal consolidation... & T-auto, in contrast to CheXpert, recognizes conditions with misspellings in the report like “carimegaly” in place of “cardiomegaly”. \\  & Cardiomegaly CheXpert: Blank ✗ T-auto: Positive ✓ \\ \hline ..._consistent with acute and/or chronic pulmonary edema...._ & T-auto incorrectly detects uncertainty in the edema label, likely from the “and/or”; CheXpert correctly classifies this example as positive. \\  & Edema CheXpert: Positive ✓ T-auto: Uncertain ✗ \\ \hline ..._Normal heart size, mediastinal and hilar contours are unchanged in appearance..._ & T-auto and CheXpert both incorrectly label this sample as negative for enlarged cardiomediastinum; CheXpert correctly classifies it as uncertain, likely recognizing that “unchanged” is associated with uncertainty of the condition. The condition cannot be labeled positive or negative without more information. \\ \hline \end{tabular}
\end{table}
Table 3: Phrases from reports where CheXpert, T-auto, and CheXbert provide different labels. The correct label is indicated by a checkmark in the first column. The CheXpert versus T-auto comparisons are conducted on the CheXpert manual set. The CheXpert versus T-auto/CheXpert comparison is conducted on the MIMIC-CXR test set.

 

reports. In the CheXpert dataset, we found that only 3 of the 190,460 report impressions were longer than 512 tokens. Third, our task is limited to the 14 observations labeled for, and we do not test for the model's ability to label rarer conditions. However, CheXbert can mark No Finding as blank, which can indicate the presence of another condition if the other 13 conditions are also blank. Fourth, the ground truth labels for the MIMIC-CXR test set were determined by a single board-certified radiologist, and the use of more radiologists could demonstrate a truer comparison to the radiologist benchmark. Fifth, while we do test performance on a dataset from an institution unseen in training, additional datasets across institutions could be useful in further establishing the model's ability to generalize.

## 7 Conclusion

In this study, we propose a simple method for combining existing report labelers with hand-annotations for accurate radiology report labeling. In this method, a biomedically pretrained BERT model is first trained on the outputs of a labeler, and then further fine-tuned on the manual annotations, the set of which is augmented using backtranslation. We report five findings on our resulting model, CheXbert. First, we find that CheXbert outperforms models trained only on radiologist-labeled reports, or only on the existing labeler's outputs. Second, we find that CheXbert outperforms the BERT-based model not pretrained on biomedical data. Third, we find that CheXbert outperforms models which do not use backtranslation. Fourth, we find that CheXbert outperforms the previous best labeler, CheXpert (which was rules-based), with an improvement of 0.055 (95% CI 0.039, 0.070) on the F1 metric; we also find that the best model trained only on manually labeled radiology reports (Tblue-rad-bt) performs at least as well as the CheXpert labeler. Fifth, we find that CheXbert is 0.007 F1 points from the radiologist performance benchmark, suggesting that the gap to ceiling performance is narrow.

We expect this method of training medical report labelers is broadly useful within the medical domain, where collection of expert labels can produce a small set of high quality labels, and existing feature engineered labelers can produce labels at scale. Extracting highly accurate labels from medical reports by taking advantage of both sources can enable many important downstream tasks, including the development of more accurate and robust medical imaging models required for clinical deployment.

## Acknowledgments

We would like to acknowledge the Stanford Machine Learning Group (stanfordmlgroup.github) and the Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI.stanford.edu) for infrastructure support. Thanks to Alistair Johnson for support in the radiologist benchmark, to Jeremy Irvin for support in the CheXpert labeler, Alex Tamkin for helpful comments, and Yifan Peng for helpful feedback.

## References

* A. Bustos, M. Pertusa, J. Salinas, and M. de la Iglesia-Vaya (2019)PadCheST: a large chest x-ray image dataset with multi-label annotated reports. arXiv:1901.07441 [cs, eess] ArXiv: 1901.07441. Cited by: SS1.
* A. Callahan, J. A. Fries, C. Re, J. I. Huddleston, N. J. Giori, S. Delp, 

and Nigam H. Shah. 2019. Medical device surveillance with electronic health records. _npj Digital Medicine_, 2(1):1-10. Number: 1 Publisher: Nature Publishing Group.
* Chapman et al. [2011] Brian E. Chapman, Sean Lee, Hyunseok Peter Kang, and Wendy W. Chapman. 2011. Document-level classification of CT pulmonary angiography reports based on an extension of the ConText algorithm. _Journal of Biomedical Informatics_, 44(5):728-737.
* Chapman et al. [2001] Wendy W. Chapman, Will Bridewell, Paul Hanbury, Gregory F. Cooper, and Bruce G. Buchanan. 2001. A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries. _Journal of Biomedical Informatics_, 34(5):301-310.
* Chen et al. [2017] Matthew C. Chen, Robyn L. Ball, Lingyao Yang, Nathaniel Moradzadeh, Brian E. Chapman, David B. Larson, Curtis P. Langlotz, Timothy J. Amrhein, and Matthew P. Lungren. 2017. Deep Learning to Classify Radiology Free-Text Reports. _Radiology_, 286(3):845-852. Publisher: Radiological Society of North America.
* Chen et al. [2018] Po-Hao Chen, Hanna Zafar, Maya Galperin-Aizenberg, and Tessa Cook. 2018. Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports. _Journal of Digital Imaging_, 31(2):178-184.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. _arXiv:1810.04805 [cs]_. ArXiv: 1810.04805.
* Drozdov et al. [2020] Ignat Drozdov, Daniel Forbes, Benjamin Szubert, Mark Hall, Chris Carlin, and David J. Lowe. 2020. Supervised and unsupervised language modelling in Chest X-Ray radiological reports. _PLOS ONE_, 15(3):e0229963. Publisher: Public Library of Science.
* Dunnmon et al. [2019] Jared Dunnmon, Alexander Ratner, Nishith Khandwala, Khaled Saab, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew Lungren, Daniel Rubin, and Christopher Re. 2019. Cross-Modal Data Programming Enables Rapid Medical Machine Learning. _arXiv:1903.11101 [cs, eess, stat]_. ArXiv: 1903.11101.
* Efron and Tibshirani [1986] B. Efron and R. Tibshirani. 1986. Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy. _Statistical Science_, 1(1):54-75. Publisher: Institute of Mathematical Statistics.
* Hassanpour et al. [2017] Saeed Hassanpour, Curtis P. Langlotz, Timothy J. Amrhein, Nicholas T. Befera, and Matthew P. Lungren. 2017. Performance of a Machine Learning Classifier of Knee MRI Reports in Two Large Academic Radiology Practices: A Tool to Estimate Diagnostic Yield. _American Journal of Roentgenology_, 208(4):750-753. Publisher: American Roentgen Ray Society.
* Irvin et al. [2019] Jeremy Irvin, Franav Rajpurkar, Michael Ko, Yifan Yu, Siliviana Ciurea-Ilcus, Chris Chute, Henrik Marlund, Behaj Haghoo, Robyn Ball, Katie Shpanashkaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. _arXiv:1901.07031 [cs, eess]_. ArXiv: 1901.07031.
* Johnson et al. [2019] Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng. 2019. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. _arXiv:1901.07042 [cs, eess]_. ArXiv: 1901.07042.
* Lee et al. [2019] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics_, page btz682. ArXiv: 1901.08746.
* McDermott et al. [2020] Matthew B. A. McDermott, Tzu Ming Harry Hsu, Wei-Hung Weng, Marzyeh Ghassemi, and Peter Szolovits. 2020. Chexpert++: Approximating the chexpert labeler for speed,differentiability, and probabilistic output.
* Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26_, pages 3111-3119. Curran Associates, Inc.
* Ng et al. [2019] Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 News Translation Task Submission. _arXiv:1907.06616 [cs]_. ArXiv: 1907.06616.
* Peng et al. [2017] Yifan Peng, Xiaosong Wang, Le Lu, Mohammad Bagheri, Ronald Summers, and Zhiyong Lu. 2017. NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. _arXiv:1712.05898 [cs]_. ArXiv: 1712.05898.
* Peng et al. [2019] Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. _arXiv:1906.05474 [cs]_. ArXiv: 1906.05474.
* Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics 

Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating Backtranslation in Neural Machine Translation. _arXiv:1804.06189__[cs]_. ArXiv: 1804.06189.
* Pons et al. (2016) Ewoud Pons, Loes M. M. Braun, M. G. Myriam Hunink, and Jan A. Kors. 2016. Natural Language Processing in Radiology: A Systematic Review. _Radiology_, 279(2):329-343. Publisher: Radiological Society of North America.
* Ratner et al. (2018) Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Re. 2018. Snorkel MeTaL: Weak Supervision for Multi-Task Learning. In _Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning_, DEEM'18, pages 1-4, Houston, TX, USA. Association for Computing Machinery.
* Ratner et al. (2020) Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Re. 2020. Snorkel: rapid training data creation with weak supervision. _The VLDB Journal_, 29(2):709-730.
* MICCAI 2019_, Lecture Notes in Computer Science, pages 811-819, Cham. Springer International Publishing.
* Savova et al. (2010) Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. 2010. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. _Journal of the American Medical Informatics Association : JAMIA_, 17(5):507-513.
* Wang et al. (2017) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. 2017. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 3462-3471. ArXiv: 1705.02315.
* Wang et al. (2018) Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, and Hongfang Liu. 2018. Clinical information extraction applications: A literature review. _Journal of Biomedical Informatics_, 77:34-49.
* Wood et al. (2020) David A. Wood, Jeremy Lynch, Sina Kafiabadi, Emily Guilhem,isha Al Busaidi, Antanas Montvila, Thomas Varasovsky, Juveria Siddiqui, Naveen Gadapa, Matthew Townend, Martin Kiik, Keena Patel, Gareth Barker, Sebastian Ourselin, James H. Cole, and Thomas C. Booth. 2020. Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM). _ArXiv:2002.06588 [cs]_. ArXiv: 2002.06588.
* Xue et al. (2019) Kui Xue, Yangming Zhou, Zhiyuan Ma, Tong Ruan, Huanthuan Zhang, and Ping He. 2019. Fine-tuning BERT for Joint Entity and Relation Extraction in Chinese Medical Text.
* Yadav et al. (2016) Kabir Yadav, Efsunarioglu, Hyeong Ah Choi, Walter B. Cartwright, Pamela S. Hinds, and James M. Chamberlain. 2016. Automated Outcome Classification of Computed Tomography Imaging Reports for Pediatric Traumatic Brain Injury. _Academic Emergency Medicine: Official Journal of the Society for Academic Emergency Medicine_, 23(2):171-178.
* Yang et al. (2020) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. XLNet: Generalized Autoregressive Pretraining for Language Understanding. _arXiv:1906.08237 [cs]_. ArXiv: 1906.08237.
* Yu et al. (2018) Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. _arXiv:1804.09541 [cs]_. ArXiv: 1804.09541.
* Zach et al. (2018) John Zach, Margaret Pain, Joseph Titano, Marcus Badgeley, Javin Schefflein, Andres Su, Anthony Costa, Joshua Bederson, Joseph Lebhar, and Eric Karl Oermann. 2018. Natural Language-based Machine Learning Models for the Annotation of Clinical Radiology Reports. _Radiology_, 287(2):570-580. Publisher: Radiological Society of North America.

 



## Appendix A Physician validation of backtranslation quality

Although the CheXbert model shows empirical improvements using backtranslated reports, backtranslation can introduce additional noise into the reports. A physician validated the quality of the backtranslation outputs. For this experiment, we randomly sampled 100 reports from the CheXpert manual set. The physician read each original report and its backtranslation, and assigned a score for whether the backtranslation a) used valid radiology language, and b) maintained the semantics of the report. For each of tasks a) and b), the expert assigned a score of 1 (worst), 2 or 3 (highest).

For task a), a score of 3 means the backtranslation contained near-perfect radiology language, a 2 means the backtranslation had only minor deviations from valid radiology language, and 1 means the backtranslation had a major deviation from valid radiology language.

For task b), a score of 3 means the backtranslation fully preserved the semantics of the original, a 2 means the backtranslation contained minor semantic errors, and a 1 means the backtranslation had a major change or loss of semantic information compared to the original report.





## Appendix B Additional results

Table 1: After removing duplicate reports for the same patient from the CheXpert dataset (excluding the CheXpert manual set), we are left with a total of 190,460 reports. Labels for these reports are provided by the CheXpert labeler. The class prevalences of this set are displayed for each medical condition.





\begin{table}
\begin{tabular}{|l|p{142.3pt}|p{85.4pt}|p{76.9pt}|} \hline  & **T-auto \(>\) CheXpert** & **CheXbert \(>\) CheXpert** & **CheXbert \(>\) Radiologist** \\ \hline Blank & 0 & 29 & 56 \\ Positive & -22 & 11 & 56 \\ Negative & 14 & 45 & 9 \\ Uncertain & 16 & 46 & -3 \\ \hline Total & 8 & 131 & 118 \\ \hline \end{tabular}
\end{table}
Table B3: The differences in the number of times labels were correctly assigned by one model versus another model. For example, in the first column named “T-auto \(>\) CheXpert,” we report the difference between the number of times T-auto correctly classifies a label and the number of times CheXpert correctly classifies a label. We record the differences between a pair of models by category (blank, positive, negative, uncertain) and by total. These occurrences are obtained on the MIMIC-CXR test set.





\begin{table}
\begin{tabular}{|l|p{142.3pt}|} \hline
**Example & Labels** & **Reasoning** \\ \hline ...2.**mild cardiomegaly**. persistent small bilateral pleural effusions, left greater than right... & T-auto mistakenly labels ” mild cardiomegaly” as uncertain for cardiomegaly. \\ Cardiomegaly CheXpert: Positive T-auto: Uncertain ✗ & \\ \hline ...2.there are diffuse increased interstitial markings and prominence of the central vasculature, **consistent with acute and/or chronic pulmonary edema...** & T-auto may have incorrectly detected uncertainty from “and/or,” which is a conjunction between “acute” and “chronic”. \\  & Edema CreXpert: Positive T-auto: Uncertain ✗ & \\ \hline \end{tabular}
\end{table}
Table B5: Examples where CheXpert correctly assigns a label while T-auto misassigns that label on the CheXpert manual set. We include speculative reasoning for the classifications.





\begin{table}
\begin{tabular}{|p{142.3pt}|p{142.3pt}|} \hline
**Example & Labels** & **Reasoning** \\ \hline
**New bibasilar opacities**, which given the clinical history are suspicious for aspiration, possibly developing pneumonia. & The word “suspicious” does not modify “opacities” in this sentence. Although CheXpert correctly identifies this, CheXpert and T-auto misclassify the “opacities” as uncertain. \\  & Lung Opacity CheXpert: Uncertain X & There is a large number of cases where the model is trained to predict the corrector. In addition, we have used the model to predict the corrector. We also tried to predict the corrector. We found that the incorrector. We did not find any incorrector.





**Example (cont.) & Labels (cont.)**

\begin{tabular}{|p{142.3pt}|p{142.3pt}|} \hline
**Examples (cont.) & **Reasoning (cont.)** \\ \hline As compared to the previous radiograph, there is no relevant change. The reduced volume of the right hemithorax with areas of lateral pleural thickening. The areas of pleural thickening are constant, size and morphology. Unchanged perihilar areas of fibrosis. **Unchanged size and aspect of the cardiac silhouette**, no pathologic changes in the left lung. & CheXbert correctly identifies uncertainty, as the cardiac silhouette is "unchanged," which means that it cannot be labeled positive or negative without additional information regarding the previous state. Both CheXpert and T-auto incorrectly label this example as positive for cardiomegaly. \\ \hline AP chest compared to .....: Small-to-moderate left pleural effusion has increased slightly over the past several days. Moderate enlargement of the cardiac silhouette accompanied by mediastinal vascular engagement is also slightly more pronounced. Pulmonary vasculature is enorgored but there is no edema. Consolidation has been present without appreciable change in the left lower lobe since at least .... **Mediastinum widened** at the thoracic inlet by a combination of tortuous vessels and mediastinal fat deposition. Right jugular introducer ends just above the junction with left brachiodcephalic vein. \\ \hline Moderately enlarged heart size, stable since ... **No findings concerning for pulmonary edema** or pneumonia. & Unlike T-auto and CheXpert, CheXpert correctly labels edema as negative, presumably understanding that the initial phrase "no findings" applies to both edema and pneumonia. \\ \hline AP chest compared to ..... and .....: As far as I can tell, given the severe anatomic distortion of the chest cage and its contents, lungs were clear on .... Small region of opacification may have been developing lateral to the left hilus on ...., and today there is a suggestion of some new opacification at the base of the lung, but these observations are far from certain. I am not even confident that conventional radiographs, should the patient be able to cooperate for them, would clarify the issue. CT scanning, if feasible, would certainly confirm if the lungs are clear, but in the absence of a baseline study **it might be difficult to distinguish atelectasis from pneumonia**. Pleural effusion is minimal if any. Heart is probably not enlarged. Nasogastric tube is looped in the stomach. Right PIC line ends in the mid SVC. No pneumothorax. \\ \hline Atelectasis CheXpert: Positive _X_T-auto: Positive _X_CheXpert: Uncertain \\ \hline \end{tabular}





**Example (cont.) & Labels (cont.)****Reasoning (cont.)****Two frontal views of the chest show new mild interstitial pulmonary edema. Interval increase in mediastinal caliber therefore is probably due to distention of mediastinal veins. Heart size is slightly larger but still within normal range.** Pleural effusions are minimal, if any. No focal pulmonary abnormality. No pneumothorax. ET tube is in standard placement and a nasogastric tube passes below the diaphragm and out of view.

\begin{tabular}{l|c} Cardiomegaly & Although CheXpert and T-auto mistakenly label cardiomegaly as positive given the phrase the "heart is slightly larger," the following phrase "but still within normal range" implies that cardiomegaly is negative. TheCkbert correctly classifies this example as negative for cardiometabolicy. \\ As compared to the previous radiograph, the pre-existing right upper lobe **pneumonia is completely resolved**. The pre-existing signs of mild fluid overload, however, are still present. The pre-existing cardiomegaly is unchanged. Several calcified lung nodules are also unchanged. Unchanged alignment of the sternal wires. No acute pneumonia, no pleural effusions. \\  & DeXbert correctly labels pneumonia as negative, as implied by the phrase " pneumonia is completely resolved," while CheXpert and T-auto both mislabel pneumonia as positive. \\ Subsegmental right lung base atelectasis. Increasing loss of vertebral body height at T11. Stable L1 compression fracture. Right shoulder humeral DJD. **Interval removal of PICC lines**. & CheXbert, presumably using a semantic understanding of the word "removal", correctly labels support devices as negative. CheXpert and T-auto pick up on "PICC lines" but do not detect the negation. Both incorrectly label support devices as positive. \\ AP chest compared to ____: Small-to-moderate left pleural effusion has increased slightly over the past several days. Moderate enlargement of the cardiac silhouette accompanied by mediastinal vascular engagement is also slightly more pronounced. Pulmonary vasculature is engorged but there is no edema. Consolidation has been present without appreciable change in the left lower lobe since at least ____: Mediastinum widened at the thoracic inlet by a combination of tortuous vessels and mediastinal fat deposition. **Right jugular introduced ends just above the junction with left brachioccephalic vein**. & A jugular introducer is a support device that wasn't included in CheXpert's list of mentions for support devices. Consequently CheXpert and T-auto, which trains on CheXpert labels, both incorrectly label support devices as blank. CheXpert, however, correctly labels support devices as positive. \\



**Example (cont.) & Labels (cont.)**

1. Interval removal of the sternal wires with placement of new sternal closure devices, mediastinal staples and tubes. Lungs are well inflated with linear streaky opacities seen at the left base likely representing scarring and/or subsegmental atelectasis. No evidence of pulmonary edema, pneumothorax, pleural effusions or focal airspace consolidation to suggest pneumonia. **Slight lucency at the left apex is felt to be related to underlying emphysema rather than representing a pneumothorax**. **Phenumothorax**. **Preumothorax**

CheXpert: Positive T-auto: Positive CaeXbert: Negative

Table B7: Examples of additional data samples generated using backtranslation on radiologist-annotated reports from the CheXpert manual set. Augmenting our relatively small set of radiologist-annotated reports with back-translation proved useful in improving performance of our labeler on the MIMIC-CXR test set.




2. a dense retrocardiac opacity may represent atelectasis versus consolidation..,small bilateral pleural effusions are present. A convex opacity at the right partracheal region is of uncertain significance; recommend upright pa and lateral for further essential when the patient is able to do so.
3. the cardiomedial silhouette and pulmonary vasculature are unnormarkable. **Backtranslation (cont.)** **Changes (cont.)** **1. a single frontal view of the breast shows a surgical drain extending over the neck, a tracheostolic tube, a feeding tube extending under the diaphragm and over the lower edge of the film. skin clamps protrude over the left collarbone, and surgical clips are visible in the left cervical area. no indication of pneumothorax. **2. dense retrocardiac opacity may represent atelasia versus consolidation..,small bilateral pleural effusions are present. A convex opacity at the right partracheal area is of uncertain importance; recommend upright pa and lateral for further assessment if the patient is able to do so. 3. the cardiacinal silhouette and pulmonary vasculature are unobtrusive. **3. the cardiovascular silhouette and pulmonary vasculature are unobtrusive. However "chest" incorrectly changed to "breast, "tracheostomy tube" incorrectly changed to "traceastolic tube", "cutaneous staples project over the left clavicle" changed to the semantically similar "skin clamps protrude over the left collarbone", but "ski clamps" is suboptimal, "atelectasis" incorrectly changed to "ateltaia", "cardiomedial" is incorrectly changed to "cardiaginal".

 


* [10] A. K. M. C. R. D. J. van der Biel, and S. W. H. G. Wilson, "The role of the effect of a single-level spin-dependent interaction in a quantum dot system," _Phys. Rev. B_**53**, 2469 (1997), no. 8, pp. 103-118.
* [11] P. F. Anderson, "A simple model for the electron-phonon scattering," _Nucl. Phys. B_**465**, 1 (1997), no. 1, p. 103-114.
* [12] T. L. Kane, "An introduction to the theory of electron physics," _JETP Lett._**10**, 1 (1963), no. 1, p. 103-104.

[MISSING_PAGE_POST]