SlowGuess/NAACL_2021_OCR huggingface dataset:
Christy Yuan Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing. 2018. Hybrid retrieval-generation reinforced agent for medical image report generation. In Advances in Neural Information Processing, volume 31, pages 1530-1540.
Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. 2019. Entangled transformer for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8927-8936.
Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Clinically accurate chest x-ray report generation. In Proceedings of the 4th Machine Learning for Healthcare Conference, volume 106, pages 249-269.
Kazuki Matsumaru, Sho Takase, and Naoaki Okazaki. 2020. Improving truthfulness of headline generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1335-1346.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919.
Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. 2020. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10968-10977.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu. 2018. Negbio: a high-performance tool for negation and uncertainty detection in radiology reports. In AMIA 2018 Informatics Summit.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. *Stanza: A python natural language processing toolkit for many human languages*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 101-108.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In International Conference on Learning Representations.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1179-1195.
Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586-1596.
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Parek, Andrew Ng, and Matthew Lungren. 2020. Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1500-1519.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, volume 30, pages 5998-6008.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566-4575.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156-3164.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. 2018. TieNet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9049-9058.
Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen. 2020b. Towards faithful neural table-to-text generation with content-matching constraints. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1072-1086.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language
inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741.
Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280.
Zhaofeng Wu, Yan Song, Sicong Huang, Yuanhe Tian, and Fei Xia. 2019. WTMED at MEDIQA 2019: A hybrid approach to biomedical natural language inference. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 415-426.
Jianbo Yuan, Haofu Liao, Rui Luo, and Jiebo Luo. 2019. Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. In Medical Image Computing and Computer Assisted Intervention, pages 721-729.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations.
Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D. Manning, and Curtis Langlotz. 2020b. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5108-5120.
Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D. Manning, and Curtis P. Langlotz. 2020c. Biomedical and clinical English model packages in the Stanza Python NLP library. arXiv preprint arXiv:2007.14640.
A Detail of Radiology NLI
A.1 Rules & Examples of Weakly-Supervised Radiology NLI
We prepared the 6 rules (E1, N1-N4, and C1) to train the weakly-supervised radiology NLI. The rules are applied against sentence pairs consisting from premises  $(s_1)$  and hypotheses  $(s_2)$  to extract pairs that are in entailment, neutral, or contradiction relation.
Entailment Rule: E1
1.  $s_1$  and  $s_2$  are semantically similar.
2. The named entities (NE) of  $s_2$  is a subset or equal to the named entities of  $s_1$  as  $\mathrm{NE}(\mathrm{s}_2) \subseteq \mathrm{NE}(\mathrm{s}_1)$ .
We used BERTScore (Zhang et al., 2020a) as a similarity metric and set the threshold to  $\text{sim}(\mathrm{s}_1, \mathrm{s}_2) \geq 0.7^6$ . The clinical model of Stanza (Zhang et al., 2020c) is used to extract anatomy entities and observation entities.  $s_1$  and  $s_2$  are conditioned to be both negated or both non-negated. The negation is determined with a negation identifier or the existence of uncertain entity, using NegBio (Peng et al., 2018) as the negation identifier and the clinical model of Stanza is used to extract uncertain entities.  $s_2$  is further restricted to include at least 2 entities as  $|\mathrm{NE}(\mathrm{s}_2)| \geq 2$ . These similarity metric, named entity recognition model, and entity number restriction are used in the latter neutral and contradiction rules. The negation restriction is used in the neutral rules but is not used in the contradiction rule. The following is an example of a sentence pair that matches E1 with entities in bold:
$s_1$  The heart is mildly enlarged.
$s_2$  The heart appears again mild-to-moderately enlarged.
Neutral Rule 1: N1
1.  $s_1$  and  $s_2$  are semantically similar.
2. The named entities of  $s_1$  is a subset of the named entities of  $s_2$  as  $\mathrm{NE}(\mathrm{s}_1) \subsetneq \mathrm{NE}(\mathrm{s}_2)$ .
Since  $s_1$  is a premise, this condition denotes that the counterpart hypothesis has entities that are not
included in the premise. The following is an example of a sentence pair that matches N1 with entities in bold:
$s_1$  There is no pulmonary edema or definite consolidation.
$s_2$  There is no focal consolidation, pleural effusion, or pulmonary edema.
Neutral Rule 2: N2
1. The named entities of  $s_1$  are equal to the named entities of  $s_2$  as  $\mathrm{NE}(\mathrm{s}_1) = \mathrm{NE}(\mathrm{s}_2)$ .
2. The anatomy modifiers  $\mathrm{(NE_{mod})}$  of  $s_1$  include an antonym (ANT) of the anatomy modifier of  $s_2$  as  $\mathrm{NE}_{\mathrm{mod}}(\mathrm{s}_1)\cap \mathrm{ANT}(\mathrm{NE}_{\mathrm{mod}}(\mathrm{s}_2))\neq \emptyset .$
Anatomy modifiers are extracted with the clinical model of Stanza and antonyms are decided using WordNet (Fellbaum, 1998). Antonyms in anatomy modifiers are considered in this rule to differentiate expressions like left vs right and upper vs lower. The following is an example of a sentence pair that matches N2 with antonyms in bold:
$s_1$  Moreover, a small left pleural effusion has newly occurred.
$s_2$  Small right pleural effusion has worsened.
Neutral Rule 3: N3
1. The named entity types  $\mathrm{(NE_{type})}$  of  $s_1$  are equal to the named entity types of  $s_2$  as  $\mathrm{NE}_{\mathrm{type}}(\mathrm{s}_1) = \mathrm{NE}_{\mathrm{type}}(\mathrm{s}_2)$ .
2. The named entities of  $s_1$  is different from the named entities of  $s_2$  as  $\mathrm{NE}(\mathrm{s}_1) \cap \mathrm{NE}(\mathrm{s}_2) = \emptyset$ .
Specific entity types that we used are anatomy and observation. This rule ensures that  $s_1$  and  $s_2$  have related but different entities in same types. The following is an example of a sentence pair that matches N3 with entities in bold:
$s_1$  There is minimal bilateral lower lobe atelectasis.
$s_2$  The cardiac silhouette is moderately enlarged.
Neutral Rule 4: N4
1. The named entities of  $s_1$  are equal to the named entities of  $s_2$  as  $\mathrm{NE}(\mathrm{s}_1) = \mathrm{NE}(\mathrm{s}_2)$ .
${}^{6}$  distilbert-base-uncased with the baseline score is used as the model of BERTScore for a fast comparison and a smooth score scale. We swept the threshold value from  $\{ {0.6},{0.7},{0.8},{0.9}\}$  and set it to 0.7 as a relaxed boundary to balance between accuracy and diversity.
2.  $s_1$  and  $s_2$  include observation keywords (KEY) that belong to different groups as  $\mathrm{KEY}(\mathrm{s}_1)\neq \mathrm{KEY}(\mathrm{s}_2)$
The groups of observation keywords are setup following the observation keywords of CheXpert labeler (Irvin et al., 2019). Specifically,  $\mathrm{G}1 = \{\text{normal}, \text{unremarkable}\}$ ,  $\mathrm{G}2 = \{\text{stable}, \text{unchanged}\}$ , and  $\mathrm{G}3 = \{\text{clear}\}$  are used to determine words included in different groups as neutral relation. The following is an example of a sentence pair that matches N4 with keywords in bold:
$s_1$  Normal cardiomediastinal silhouette.
$s_2$  Cardiomediastinal silhouette is unchanged.
Contradiction Rule: C1
1. The named entities of  $s_1$  is a subset or equal to the named entities of  $s_2$  as  $\mathrm{NE}(\mathrm{s}_2) \subseteq \mathrm{NE}(\mathrm{s}_1)$ .
2.  $s_1$  or  $s_2$  is a negated sentence.
Negation is determined with the same approach as E1. The following is an example of a sentence pair that matches C1 with entities in bold:
$s_1$  There are also small bilateral pleural effusions.
$s_2$  No pleural effusions.
A.2 Validation and Test Datasets of Radiology NLI
We sampled 480 sentence pairs that satisfy the following conditions from the validation section of MIMIC-CXR:
1. Two sentences  $(s_1$  and  $s_2)$  have BERTScore  $(s_1,s_2)\geq 0.5$
2. MedNLI labels are equally distributed over three labels: entailment, neutral, and contradiction<sup>7</sup>.
These conditions are introduced to reduce neutral pairs since most pairs will be neutral with random sampling. The sampled pairs are annotated twice swapping its premise and hypothesis by two experts: one medical expert and one NLP expert. For pairs that the two annotators disagreed, its labels are decided by a discussion with one additional NLP expert. The resulting 960 bidirectional pairs are split in half resulting in 480 pairs for a validation set and 480 pairs for a test set.
<sup>7</sup>We used the baseline BERT model of Wu et al. (2019) to assign MedNLI labels to the pairs.
A.3 Configuration of Radiology NLI Model
We used bert-base-uncased as a pre-trained BERT model and further fine-tuned it on MIMIC-III (Johnson et al., 2016) radiology reports with a masked language modeling loss for 8 epochs. The model is further optimized on the training data with a classification negative log likelihood loss. We used Adam (Kingma and Ba, 2015) as an optimization method with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , batch size of 16, and the gradient clipping norm of 5.0. The learning rate is set to  $lr = 1e^{-5}$  by running a preliminary experiment with  $lr = \{1e^{-5}, 2e^{-5}\}$ . The model is optimized for the maximum of 20 epochs and a validation accuracy is used to decide a model checkpoint that is used to evaluate the test set. We trained the model with a single Nvidia Titan XP taking approximately 2 hours to complete 20 epochs.
B Configurations of Radiology Report Generation Models
B.1  $\mathcal{M}^2$  Trans
We used DenseNet-121 (Huang et al., 2017) as a CNN image feature extractor and pre-trained it on CheXpert dataset with the 14-class classification setting. We used GloVe (Pennington et al., 2014) to pre-train text embeddings and the pre-trainings were done on a training set with the embedding size of 512. The parameters of the model is set up to the dimensionality of 512, the number of heads to 8, and the number of memory vector to 40. We set the number of Transformer layer to  $n_{layer} = 1$  by running a preliminary experiment with  $n_{layer} = \{1,2,3\}$ . The model is first trained against NLL loss using the learning rate scheduler of Transformer (Devlin et al., 2019) with the warm-up steps of 20000 and is further optimized with a joint loss with the fixed learning rate of  $5e^{-6}$ . Adam is used as an optimization method with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ . The batch size is set to 48 for NLL loss and 24 for the joint losses. For  $\lambda_{*}$ , we first swept the optimal value of  $\lambda_{1}$  from  $\{0.03,0.02,0.01,0.001\}$  using the development set. We have restricted  $\lambda_{2}$  and  $\lambda_{3}$  to have equal values in our experiments and constrained that all  $\lambda_{*}$  values sum up to 1.0. The model is trained with NLL loss for 32 epochs and further trained for 32 epochs with a joint loss. Beam search with the beam size of 4 is used to decode texts when evaluating the model against a validation set or a test set. We trained the model with a single Nvidia
Titan XP taking approximately 10 days to complete its optimization.
B.2 TieNet
We used ResNet-50 as a CNN image feature extractor with default ImageNet pre-trained weights. We used GloVe to pre-train text embeddings with the same configuration as  $\mathcal{M}^2$  Trans. The parameters of the model is set up to the LSTM dimension of 256 and the number of global attentions to 5. The combination of NLL loss and the multi-label classification loss is used as its joint loss with the balance parameter  $\alpha = 0.85$ . The model is trained against the joint loss using a linear rate scheduler with the initial learning rate of  $1e^{-4}$  and the multiplication of 0.5 per 8 epochs. The batch size is set to 32 and the model is trained with the joint loss for 32 epochs. Adam is used as an optimization method with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ . Beam search with the beam size of 4 is used to decode texts. We trained the model with a single Nvidia Titan XP taking approximately 2 days to complete its optimization.
B.3 CNN-RNN²
We used DenseNet-121 as a CNN image feature extractor with default ImageNet pre-trained weights. We used GloVe to pre-train text embeddings with the same configuration as  $\mathcal{M}^2$  Trans. The parameters of the model is set up to the LSTM dimension of 256. We modified an information extraction system from CheXpert to CheXbert to improve the training speed of this model. The combination of CIDEr and Clinically Coherent Reward is used as its joint loss with the balance parameter  $\lambda = 10.0$ . The model is first trained against NLL loss using a linear rate scheduler with the initial learning rate of  $1e^{-4}$  and the multiplication of 0.5 per 8 epochs. The model is further optimized with the joint loss with the fixed learning rate of  $5e^{-6}$ . Adam is used as an optimization method with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ . The batch size is set to 32 for the NLL loss and 24 for the joint losses. The model is trained with NLL loss for 32 epochs and further trained for 32 epochs with the joint loss. Beam search with the beam size of 4 is used to decode texts. We trained the model with a single Nvidia Titan XP taking approximately 11 days to complete its optimization.
C Detailed Result of Clinical Metrics
Table 5 shows the detailed results of the clinical metrics for R2Gen,  $\mathcal{M}^2$  Trans w/ BS,  $\mathcal{M}^2$  Trans w/ BS+fcE, and  $\mathcal{M}^2$  Trans w/ BS+fcEN. In most cases, the best  $F_1$  scores are observed in the cases when factENT or factENTNLI is included in the joint losses. Consolidation is one exception where the best precisions, recalls, and  $F_1$  scores vary among the joint losses. We assume this is due to the infrequent appearance of consolidation in both MIMIC-CXR and Open-i. For comparison against some past studies, we show the detailed results when CheXpert is used instead of CheXbert in Table 6. Since CheXbert is more or equally accurate for most observations than CheXpert, the scores in Table 6 follow similar trends against ones in Table 5. Table 7 shows the detailed results for the 9 remaining observations that are defined in CheXpert. Note that many of these observations are infrequent and have relatively weaker and unstable extraction performances compared to the 5 observations in Table 5.
Observation(# MIMIC-CXR / # Open-i)	MIMIC-CXR	Open-i
R2Gen	NLL + BS	NLL + BS + fce	NLL + BS + fcen	R2Gen	NLL + BS	NLL + BS + fce	NLL + BS + fcen
Micro Average(2713 / 654)	P	41.2	46.3	46.3	50.3	27.0	47.7	44.0	48.7
R	29.8	67.5	73.2	65.1	17.3	46.6	53.5	46.9
F1	34.6	54.9	56.7	56.7	21.1	47.2	48.3	47.8
acc.	73.9	74.4	74.2	77.1	94.9	95.9	95.5	96.0
Atelectasis(604 / 216)	P	35.4	39.9	37.9	40.6	35.6	44.0	35.8	39.4
R	27.8	67.4	80.5	76.2	7.4	35.6	47.7	45.4
F1	31.1	50.2	51.6	53.0	12.3	39.4	40.9	42.2
acc.	68.3	65.5	61.1	65.2	93.1	92.9	91.1	91.9
Cardiomegaly(535 / 225)	P	32.4	35.8	34.3	37.5	24.5	56.6	57.3	60.0
R	53.5	73.3	81.3	61.3	33.8	55.6	55.6	46.7
F1	40.4	48.1	48.2	46.6	28.4	56.1	56.4	52.5
acc.	64.0	63.9	60.2	67.9	88.5	94.1	94.2	94.3
Consolidation(157 / 19)	P	14.3	10.5	19.6	19.2	0.0	10.9	15.2	14.3
R	7.0	18.5	5.7	3.2	0.0	26.3	26.3	5.3
F1	9.4	13.4	8.9	5.5	0.0	15.4	19.2	7.7
acc.	91.0	84.0	92.1	92.6	99.0	98.4	98.7	99.3
Edema(645 / 75)	P	55.3	59.7	56.0	65.6	10.0	39.0	30.9	41.4
R	24.3	59.2	69.9	52.7	4.0	30.7	50.7	32.0
F1	33.8	59.5	62.2	58.5	5.7	34.3	38.4	36.1
acc.	73.8	77.8	76.6	79.4	97.0	97.4	96.3	97.5
Pleural Effusion(772 / 119)	P	76.2	67.2	68.2	65.9	85.7	54.3	59.4	56.0
R	24.1	80.6	78.5	82.0	15.1	63.0	66.4	66.4
F1	36.6	73.3	73.0	73.1	25.7	58.4	62.7	60.8
acc.	72.6	80.7	80.9	80.1	96.9	96.8	97.2	96.9
Table 5: The detailed results of R2Gen,  $\mathcal{M}^2$  Trans w/ BS,  $\mathcal{M}^2$  Trans w/ BS+fcE, and  $\mathcal{M}^2$  Trans w/ BS+fcEN for the 5 observations. P is precision, R is recall, and acc. is accuracy. #MIMIC-CXR and #Open-i are the numbers of times that a corresponding observation has appeared as positive in the test set of MIMIC-CXR and Open-i, respectively.
Observation(# MIMIC-CXR / # Open-i)	MIMIC-CXR	Open-i
R2Gen	NLL + BS	NLL + BS + fCE	NLL + BS + fCEN	R2Gen	NLL + BS	NLL + BS + fCE	NLL + BS + fCEN
Micro Average(2713 / 654)	P	37.6	46.0	46.0	49.9	16.7	46.3	42.7	47.8
R	29.1	67.2	72.9	64.6	17.1	45.8	52.5	46.3
F1	32.8	54.6	56.4	56.3	16.9	46.1	47.1	47.0
acc.	74.6	74.2	74.0	76.8	93.4	95.8	95.4	95.9
Atelectasis(604 / 216)	P	35.6	39.9	37.8	40.6	33.3	42.9	34.7	38.2
R	23.9	67.6	80.6	76.4	7.1	35.4	47.2	44.8
F1	28.6	50.2	51.5	53.0	11.7	38.8	40.0	41.2
acc.	72.1	65.6	61.1	65.3	93.2	92.9	91.0	91.9
Cardiomegaly(535 / 225)	P	28.6	36.1	34.6	37.6	20.4	55.2	55.5	60.0
R	49.2	72.8	81.1	60.5	31.3	53.5	53.0	45.7
F1	36.2	48.2	48.5	46.4	24.7	54.3	54.2	51.9
acc.	63.0	63.8	60.0	67.6	86.8	93.8	93.8	94.2
Consolidation(157 / 19)	P	10.7	10.5	19.6	19.2	1.1	10.9	14.7	14.3
R	9.4	17.8	5.5	3.1	10.5	26.3	26.3	5.3
F1	10.0	13.2	8.6	5.3	2.0	15.4	18.9	7.7
acc.	88.4	83.7	91.9	92.4	94.0	98.4	98.7	99.3
Edema(645 / 75)	P	49.0	58.7	54.9	64.3	5.8	35.0	30.1	39.7
R	29.1	59.0	69.5	52.3	4.1	28.8	50.7	31.5
F1	36.5	58.8	61.4	57.7	4.8	31.6	37.8	35.1
acc.	74.5	77.6	76.2	79.2	96.4	97.3	96.3	97.5
Pleural Effusion(772 / 119)	P	75.6	66.5	67.6	65.2	63.3	53.2	57.5	54.6
R	23.4	80.3	78.2	81.6	16.4	63.8	66.4	66.4
F1	35.8	72.7	72.5	72.5	26.0	58.0	61.6	59.9
acc.	75.1	80.3	80.6	79.8	96.8	96.8	97.1	96.9
Table 6: The detailed results of R2Gen,  $\mathcal{M}^2$  Trans w/ BS,  $\mathcal{M}^2$  Trans w/ BS+fcE, and  $\mathcal{M}^2$  Trans w/ BS+fcEN for the 5 observations evaluated with CheXpert instead of CheXbert.
Observation(# MIMIC-CXR / # Open-i)	MIMIC-CXR	Open-i
R2Gen	NLL + BS	NLL + BS + fCE	NLL + BS + fCEN	R2Gen	NLL + BS	NLL + BS + fCE	NLL + BS + fCEN
Enlarged Cardiomediastinum(111 / 24)	P	4.4	5.1	4.8	4.6	3.8	0.7	2.0	4.0
R	19.8	50.5	19.8	47.7	16.7	4.2	4.2	20.8
F1	7.1	9.3	7.7	8.4	6.3	1.2	2.7	6.7
acc.	75.6	53.4	77.5	50.6	96.4	95.1	97.8	95.8
Fracture(56 / 43)	P	0.0	40.0	10.7	26.1	0.0	0.0	0.0	3.1
R	0.0	3.6	5.4	10.7	0.0	0.0	0.0	2.3
F1	0.0	6.6	7.1	15.2	0.0	0.0	0.0	2.7
acc.	97.6	97.6	96.7	97.1	98.7	98.6	98.1	97.8
Lung Lesion(97 / 89)	P	37.5	33.3	22.2	44.4	25.0	0.0	0.0	66.7
R	3.1	1.0	2.1	4.1	1.1	0.0	0.0	4.5
F1	5.7	2.0	3.8	7.5	2.2	0.0	0.0	8.4
acc.	95.8	95.8	95.7	95.8	97.3	97.3	97.3	97.4
Lung Opacity(798 / 344)	P	44.5	48.8	53.5	54.9	43.1	50.5	57.8	41.1
R	29.9	41.6	10.4	26.6	8.1	29.1	7.6	22.1
F1	35.8	44.9	17.4	35.8	13.7	36.9	13.4	28.7
acc.	63.5	65.3	66.5	67.6	89.4	89.7	89.9	88.7
No Finding(396 / 2319)	P	31.4	44.4	49.8	48.8	78.1	80.8	82.1	81.7
R	43.9	35.9	41.7	39.9	84.1	93.4	91.5	88.4
F1	36.6	39.7	45.4	43.9	81.0	86.6	86.5	84.9
acc.	74.4	81.6	83.1	82.8	72.6	80.0	80.2	78.2
Pleural Other(39 / 29)	P	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
R	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
F1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
acc.	98.1	98.3	98.3	98.3	99.1	99.1	99.1	99.1
Pneumonia(424 / 109)	P	42.1	62.7	62.1	0.0	27.6	31.3	38.6	0.0
R	15.8	16.3	17.0	0.0	7.3	19.3	24.8	0.0
F1	23.0	25.8	26.7	0.0	11.6	23.9	30.2	0.0
acc.	80.9	83.1	83.1	81.9	96.3	96.0	96.3	96.7
Pneumothorax(78 / 15)	P	60.0	28.7	37.0	50.0	100.0	40.0	0.0	100.0
R	3.8	34.6	12.8	10.3	6.7	13.3	0.0	13.3
F1	7.2	31.4	19.0	17.0	12.5	20.0	0.0	23.5
acc.	96.7	95.0	96.4	96.7	99.6	99.5	99.6	99.6
Support Devices(624 / 41)	P	52.2	50.8	53.2	49.0	10.0	16.2	19.7	13.1
R	68.9	83.5	78.7	89.7	12.2	43.9	36.6	56.1
F1	59.4	63.2	63.5	63.3	11.0	23.7	25.6	21.3
acc.	75.0	74.1	75.9	72.4	97.6	96.5	97.4	94.9
Table 7: The detailed results of R2Gen,  $\mathcal{M}^2$  Trans w/ BS,  $\mathcal{M}^2$  Trans w/ BS+fcE, and  $\mathcal{M}^2$  Trans w/ BS+fcEN for the remaining 9 observations.