ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning
Year: 2023

Abstract:
This paper explores the task of radiology report generation, which aims at generating free-text descriptions for a set of radiographs. One significant challenge of this task is how to correctly maintain the consistency between the images and the lengthy report. Previous research explored solving this issue through planningbased methods, which generate reports only based on high-level plans. However, these plans usually only contain the major observations from the radiographs (e.g., lung opacity), lacking much necessary information, such as the observation characteristics and preliminary clinical diagnoses. To address this problem, the system should also take the image information into account together with the textual plan and perform stronger reasoning during the generation process. In this paper, we propose an Observation-guided radiology Report GenerAtioN framework (ORGAN). It first produces an observation plan and then feeds both the plan and radiographs for report generation, where an observation graph and a tree reasoning mechanism are adopted to precisely enrich the plan information by capturing the multiformats of each observation. Experimental results demonstrate that our framework outperforms previous state-of-the-art methods regarding text quality and clinical efficacy.1

heading: 1 Introduction
text: Radiology reports, which contain the textual description for a set of radiographs, are critical in the process of medical diagnosis and treatment. Nevertheless, the interpretation of radiographs is very time-consuming, even for experienced radiologists
∗Equal Contribution. †Corresponding authors. 1Our code is available at https://github.com/
wjhou/ORGan.
(5-10 minutes per image). Due to its large potential to alleviate the strain on the healthcare workforce, automated radiology report generation (Anderson et al., 2018; Rennie et al., 2017) has attracted increasing research attention.
One significant challenge of this task is how to correctly maintain the consistency between the image and the lengthy textual report. Many previous works proposed to solve this through planningbased generation by first concluding the major observations identified in the radiographs before the word-level realization (Jing et al., 2018; You et al., 2021; Nooralahzadeh et al., 2021; Nishino et al., 2022). Despite their progress, these methods still struggle to maintain the cross-modal consistency between radiographs and reports. A significant problem within these methods is that, in the stage of word-level generation, the semantic information of observations and radiographs is not fully utilized. They either generate the report only based on the high-level textual plan (i.e., major observations)
8108
or ignore the status of an observation (i.e., positive, negative, and uncertain), which is far from adequate. The observations contained in the highlevel plan are extremely concise (e.g., lung opacity), while the final report needs to include more detailed information, such as the characteristics of the observation (e.g., a subtle but new lung opacity) and requires preliminary diagnosis inference based on the observation (e.g., lung infection must be suspected). In order to identify those detailed descriptions and clinical inferences about the observations, we need to further consider the image information together with the textual plan, and stronger reasoning must be adopted during the generation process.
In this paper, we propose ORGAN, an Observation-guided radiology Report GenerAtioN framework. Our framework mainly involves two stages, i.e., the observation planning and the report generation stages, as depicted in Figure 1. In the first stage, our framework produces the observation plan based on the given images, which includes the major findings from the radiographs and their statuses (i.e., positive, negative, and uncertain). In the second stage, we feed both images and the observation plan into a Transformer model to generate the report. Here, a tree reasoning mechanism is devised to enrich the concise observation plan precisely. Specifically, we construct a three-level observation graph, with the high-level observations as the first level, the observation-aware n-grams as the second level, and the specific tokens as the third level. These observation-aware n-grams capture different common descriptions of the observations and serve as the component of observation mentions. Then, we use the tree reasoning mechanism to capture observation-aware information by dynamically aggregating nodes in the graph.
In conclusion, our main contributions can be summarized as follows:
• We propose an Observation-guided radiology Report GenerAtioN framework (ORGAN) that can maintain the clinical consistency between radiographs and generated free-text reports.
• To achieve better observation realization, we construct a three-level observation graph containing observations, n-grams, and tokens based on the training corpus. Then, we perform tree reasoning over the graph to dynamically select observation-relevant information.
• We conduct extensive experiments on two pub-
licly available benchmarks, and experimental results demonstrate the effectiveness of our model. We also conduct a detailed case analysis to illustrate the benefits of incorporating observation-related information.

heading: 2 Methodology

heading: 2.1 Overview of the Proposed Framework
text: Given an image X , the probability of a report Y = {y1, . . . , yT } is denoted as p(Y |X). Our framework decomposes p(Y |X) into two stages, where the first stage is observation planning, and the second stage is report generation. Specifically, observations of an image Z = {z1, . . . , zL} are firstly produced, modeled as p(Z|X). Then, the report is generated based on the observation plan and the image, modeled as p(Y |X,Z). Finally, our framework maximizes the following probability:
p(Y |X) ∝ p(Z|X)︸ ︷︷ ︸ Stage 1 p(Y |X,Z)︸ ︷︷ ︸ Stage 2 .

heading: 2.2 Observation Plan Extraction and Graph Construction
text: Observation Plan Extraction. There are two available tools for extracting observation labels from reports, which are CheXpert (Irvin et al., 2019) and CheXbert (Smit et al., 2020). We use CheXbert2 instead of CheXpert because the former achieved better performance. To extract the observation plan of a given report, we first adopt the CheXbert to obtain the observation labels within 14 categories C = {C1, . . . , C14} as indicated in Irvin et al. (2019). More details about the distribution of observation can be found in Appendix A.1. The label (or status) of each category belongs to Present, Absent, and Uncertain, except the No Finding category, which only belongs to Present and Absent. To simplify the observation plan and emphasize the abnormalities presented in a report, we regard Present and Uncertain as Positive and Absent as Negative. Then, observations are divided into a positive collection C/POS and a negative collection C/NEG, and each category with its corresponding label is then converted to its unique observation Ci/POS ∈ C/POS or Ci/NEG ∈ C/NEG, resulting in 28 observations. For example, as indicated in Figure 1, the report presents Lung Opacity while Cardiomegaly is absent in it. These categories
2https://github.com/stanfordmlgroup/ CheXbert
are converted to two observations: Lung Opacity/POS and Cardiomegaly/NEG. Then, we locate each observation by matching mentions in the report and order them according to their positions. These mentions are either provided by Irvin et al. (2019)3 or extracted from the training corpus (i.e., n-grams), as will be illustrated in the following part. Finally, we can obtain the image’s observation plan Z = {z1, . . . , zL}. Tree-Structured Observation Graph Construction. Since observations are high-level concepts that are implicitly related to tokens in reports, it could be difficult for a model to realize these concepts in detailed reports without more comprehensive modeling. Thus, we propose to construct an observation graph by extracting observation-related ngrams as the connections between observations and tokens for better observation realization. Specifically, it involves two steps to construct such a graph: (1) n-grams extraction, where n ∈ [1, 4] and (2) <observation, n-gram> association. Following previous research (Diao et al., 2021; Su et al., 2021b), we adopt the pointwise mutual information (PMI) (Church and Hanks, 1990) to fulfill
3https://github.com/stanfordmlgroup/ chexpert-labeler
these two steps, where a higher PMI score implies two units with higher co-occurrence:
PMI(x̄, x̂) = log p(x̄, x̂)
p(x̄)p(x̂) .
For the first step, we extract n-gram units S = {s1, . . . , s|s|} based on the training reports. Given two adjacent units x̄ and x̂ of a text sequence, a high PMI score indicates that they are good collection pairs to form a candidate n-gram s∗, while a low PMI score indicates that these two units should be separated. For the second step, given a predefined observation set O = {z1, . . . , z|O|}, we extract the observation-related n-gram units with PMI(zi, sj), where zi is the i-th observation, sj is the j-th n-gram, and p(zi, sj) is the frequency that an n-gram sj appears in a report with observation zi in the training set. Then, we can obtain a set of observation-related n-grams sz = {sz1, . . . , szk}, where szj = {tzj,1, . . . , tzj,n}, and tokens in n-grams form the token collection T = {t1, . . . , t|T |}. Note that we remove all the stopwords in T , using the vocabulary provided by NLTK4. Finally, for each observation, we extract the top-K n-grams as the candidates to construct the graph, which contains
4https://www.nltk.org/
three types of nodes V = {Z, S, T}. We list part of the n-grams in Appendix A.2. After extracting relevant information from the training reports, we construct an observation graph G =< V,E > by introducing three types of edges E = {E1, E2, E3}:
• E1: This undirected edge connects two adjacent observations in an observation plan (i.e., <zi, zi+1>). • E2: This directed edge connects an observation and an n-gram (i.e., <zi, sj>). • E3: This directed edge connects an n-gram with its tokens (i.e., <sj , tk>).

heading: 2.3 Visual Features Extraction
text: Given an image X , a CNN and an MLP layer are first adopted to extract visual features X:
X = {x1, . . . ,xN} = MLP(CNN(X)), where xi ∈ Rh is the i-th visual feature.

heading: 2.4 Stage 1: Observation Planning
text: The output of observation planning is an observation sequence, which is the high-level summarization of the radiology report, as shown on the left side of Figure 2. While examining a radiograph, a radiologist must report positive observations. However, only part of the negative observations will be reported by the radiologist, depending on the overall conditions of the radiograph (e.g., co-occurrence of observations or the limited length of a report). Thus, it is difficult to plan without considering the observation dependencies (i.e., label dependencies). Here, we regard the planning problem as a generation task and use a Transformer encoder-decoder for observation planning:
hv = {hv1, . . . ,hvN} = Encoderp(X), zl = Decoderp(hv, z<l),
p(zl|X,Z<l) = Softmax(Wzzl + bz), where hvi ∈ Rh is the i-th visual hidden representation, Encoderp is the visual encoder, Decoderp is the observation decoder, z∗ ∈ Rh is the decoder hidden representation, Wz ∈ R|O|×h is the weight matrix, and bz ∈ R|O| is the bias vector. Then the planning loss Lp is formulated as:
Lp = − L∑
l=1
wl log p(zl|X,Z<l)
wl = { 1 + α if zl ∈ C/POS, 1 otherwise.
By increasing α, the planner gives more attention to abnormalities. Note that the plugged α is applied to positive observations and No Finding/NEG instead of No Finding/POS.

heading: 2.5 Stage 2: Observation-Guided Report Generation
text: Observation Graph Encoding. We use a Transformer encoder to encode the observation graph constructed according to Section 2.2. To be specific, given the observation graph G with nodes V = {Z, S, T} and edges E = {E1, E2, E3}, we first construct the adjacency matrix Â = A + I based on E. Then, V and Â are fed into the Transformer for encoding. Now Â serves as the selfattention mask in the Transformer, which only allows nodes in the graph to attend to connected neighbors and itself. To incorporate the node type information, we add a type embedding P ∈ Rh for each node representation:
N = Embed(V ) + P ,
V = {Z,S,T } = Encoderg(N , Â), where Embed(·) is the embedding function, and N ∈ Rh represents node embeddings. For observation nodes, P denotes positional embeddings, and for n-gram and token nodes, P represents type embeddings. Z, S, and T ∈ Rh are encoded representations of observations, n-grams, and tokens, respectively. Vision-Graph Alignment. As an observation graph may contain irrelevant information, it is necessary to align the graph with the visual features. Specifically, we jointly encode visual features X and token-level node representations T so that the node representations can fully interact with the visual features, and we prevent the visual features from attending the node representations by introducing a self-attention mask M:
[hv,TA] = Encoderu([X,T ],M),
where hv,TA ∈ Rh are the visual representation and the aligned token-level node representations, respectively. Observation Graph Pruning. After aligning visual features and the observation graph, we prune the graph by filtering out irrelevant nodes. The probability of keeping a node is denoted as:
p(1|TA) = Sigmoid(WdTA + bd), where Wd ∈ R1×h is the learnable weight and bd ∈ R is the bias. We can optimize the pruning
process with the following loss:
Ld = [−β · d log p(1|TA) − (1− d) log(1− p(1|TA))],
where β is the weight to tackle the class imbalance issue, and d is the label indicating whether a token appears in the referential report. Finally, we prune the observation graph by masking out token-level nodes with p(1|TA) < 0.5 and masked token-level node representations denote as TM = Prune(T ). Tree Reasoning over Observation Graph. We devise a tree reasoning (TrR) mechanism to aggregate observation-relevant information from the graph dynamically. The overall process is shown in Figure 3, where we aggregate node information from the observation level (i.e., first level) to the n-gram level (i.e., second level), then to the token level (i.e., third level). To be specific, given a query ql and node representations at l-th level kl ∈ {Z,S,TM}, the tree reasoning path is q0 Z−→ q1 S−→ q2 T M
−−→ q3, and the overall process, is formulated as below:
vl+1 = MHA(Wqql,Wkkl,Wvkl),
ql+1 = LayerNorm(ql + vl+1),
where MHA and LayerNorm are the multi-head self-attention, and layer normalization modules (Vaswani et al., 2017), respectively. Wq, Wk, and Wv ∈ Rh×h are weight metrics for query, key, and value vector, respectively. Finally, we can obtain the multi-level information q3, containing observation, n-gram, and token information.
Report Generation with Tree Reasoning. As shown in the right side of Figure 2, an observationguided Transformer decoder is devised to incorporate the graph information, including (i) multiple observation-guided decoder blocks (i.e., Decoderg), which aims to align observations with the visual representations, and (ii) a tree-reasoning block (i.e., TrRg), which aims to aggregate observationrelevant information. For Decoderg, we insert an
observation-related cross-attention module before a visually-aware cross-attention module. By doing this, the model can correctly focus on regions closely related to a specific observation. Given the visual representations hv, the node representations V = {Z,S,TM}, and the hidden representation of the prefix hw∗ ∈ Rh, the t-th decoding step is formulated as:
Decoderg =    hst = Self-Attn(h w t ,h w <t,h w <t), hot = Cross-Attn(h s t ,Z,Z),
hpt = Cross-Attn(h o t ,h v,hv),
TrRg =
{ hdt = Self-Attn(h p t ,h p <t,h p <t),
q3t = TrR(h d t , [Z,S,T M ]),
p(yt|X,G, Y<t) = Softmax(Wgq3t + bg), where Self-Attn is the self-attention module, CrossAttn is the cross-attention module, hst ,h o t ,h p t ∈ Rh are self-attended hidden state, observationrelated hidden state, visually-aware hidden state of Decoderg, respectively. hdt ∈ Rh is the selfattended hidden state of TrRg, Wg ∈ R|V |×h is the weight matrix, and bg ∈ R|V | is the bias vector. We omit other modules (i.e., Layer Normalization and Feed-Forward Network) in the standard Transformer for simplicity. Note that we extend the observation plan Z to an observation graph G, so the probability of yt conditions on G instead of Z. Then, we optimize the generation process using the negative log-likelihood loss:
Lr =− T∑
t=1
log p(yt|X,G, Y<t).
Finally, the loss function of the generator is Lg = Lr + Ld.

heading: 3 Experiments

heading: 3.1 Datasets
text: Following previous research (Chen et al., 2020, 2021), we use two publicly available benchmarks to evaluate our method, which are IU X-RAY5 (Demner-Fushman et al., 2016) and MIMICCXR6 (Johnson et al., 2019). Both datasets have been automatically de-identified, and we use the same preprocessing setup of Chen et al. (2020).
• IU X-RAY is collected by Indiana University, containing 3,955 reports with two Xray images per report resulting in 7,470 im-
5https://openi.nlm.nih.gov/ 6https://physionet.org/content/
MIMIC-cxr-jpg/2.0.0/
ages in total. We split the dataset into train/validation/test sets with a ratio of 7:1:2, which is the same data split as in (Chen et al., 2020). • MIMIC-CXR consists of 377,110 chest X-ray images and 227,827 reports from 63,478 patients. We adopt the standard train/validation/test splits.

heading: 3.2 Evaluation Metrics and Baselines
text: We adopt natural language generation metrics (NLG Metrics) and clinical efficacy (CE Metrics) to evaluate the models. BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004) are selected as NLG Metrics, and we use the MS-COCO caption evaluation tool7 to compute the results. For CE Metrics, we adopt CheXpert (Irvin et al., 2019) for MIMIC-CXR dataset to label the generated reports compared with disease labels of the references.
To evaluate the performance of ORGAN, we compare it with the following 10 state-of-the-art (SOTA) baselines: R2GEN (Chen et al., 2020), CA (Liu et al., 2021c), CMCL (Liu et al., 2021a), PPKED (Liu et al., 2021b), R2GENCMN (Chen
7https://github.com/tylin/coco-caption
et al., 2021), ALIGNTRANSFORMER (You et al., 2021), KNOWMAT (Yang et al., 2021), M2TR (Nooralahzadeh et al., 2021), CMM-RL (Qin and Song, 2022), and CMCA (Song et al., 2022).

heading: 3.3 Implementation Details
text: We adopt the ResNet-101 (He et al., 2015) pretrained on ImageNet (Deng et al., 2009) as the visual extractor. For IU X-RAY, we further fine-tune ResNet-101 on CheXpert (Irvin et al., 2019). The layer number of all the encoders and decoders is set to 3 except for Graph Encoder, where the layer number is set to 2. The input dimension and the feed-forward network dimension of a Transformer block are set to 512, and each block contains 8 attention heads. The beam size for decoding is set to 4, and the maximum decoding step is set to 64/104 for IU X-RAY and MIMIC-CXR, respectively.
We use AdamW (Loshchilov and Hutter, 2019) as the optimizer and set the initial learning rate for the visual extractor as 5e-5 and 1e-4 for the rest of the parameters, with a linear schedule decreasing from the initial learning rate to 0. α is set to 0.5, the dropout rate is set to 0.1, and the batch size is set to 32. For IU X-ray, we train the planner/generator for 15/15 epochs, and β is set to 2. For MIMIC-CXR, the training epoch of
the planner/generator is set to 3/5, and β is set to 5. We select the best checkpoints of the planner based on micro F1 of all observations and select the generator based on the BLEU-4 on the validation set. Our model has 65.9M parameters, and the implementations are based on HuggingFace’s Transformers (Wolf et al., 2020). We conduct all the experiments on an NVIDIA-3090 GTX GPU with mixed precision. The NLTK package version is 3.6.2.

heading: 4 Results

heading: 4.1 NLG Results
text: Table 1 shows the experimental results. ORGAN outperforms most of the baselines (except CMCA on IU X-RAY) and achieves state-of-the-art performance. Specifically, our model achieves 0.195 BLEU-4 on the IU X-XRAY dataset, which is the second-best result, and 0.123 BLEU-4 on the MIMIC-CXR dataset, leading to a 5.1% increment of compared to the best baseline (i.e., CMCA). In terms of METEOR, ORGAN achieves competitive performance on both datasets. In addition, our model increases R-L by 0.6% on the MIMICCXR dataset compared to the best baseline and achieves the second-best result on the IU X-RAY dataset. This indicates that by introducing the guidance of observations, ORGAN can generate more coherent text than baselines. However, we notice that on the IU X-RAY dataset, there is still a performance gap between our model and the best baseline (i.e., CMCA). The reason may be that the overall data size of this dataset is small (∼ 2,000 samples for training). It is difficult to train a good planner using a small training set, especially with cross-modal data. As we can see from Table 3, the planner only achieves 0.132 Macro-F1 on the IU X-RAY dataset, which is relatively low compared
to the performance of the planner on the MIMICCXR dataset. Thus, accumulation errors unavoidably propagate to the generator, which leads to lower performance.

heading: 4.2 Clinical Efficacy Results
text: The clinical efficacy results are listed on the right side of Table 1. On the MIMIC-CXR dataset, our model outperforms previous SOTA results. Specifically, our model achieves 0.385 F1, increasing by 1.4% compared to the best baseline. In addition, 0.416 precision and 0.418 recall are achieved by ORGAN, which are competitive results. This indicates that our model can successfully maintain the clinical consistency between the images and the reports.

heading: 4.3 Ablation Results
text: To examine the effect of the observation plan and the TrR mechanism, we perform ablation tests, and the ablation results are listed in Table 2. There are three variants: (1) ORGAN w/o Plan, which does not consider observation information, (2) ORGAN
w/o Graph, which only considers observations but not the observation graph, (3) ORGAN w/o TrR, which select information without using the TrR mechanism. Compared to the full model, the performance of ORGAN w/o Plan drops significantly on both datasets. This indicates that observation information plays a vital role in generating reports. For ORGAN w/o Graph, the performance on NLG metrics decreases significantly, but the performance of clinical efficacy remains nearly the same as the full model. This is reasonable because the observation graph is designed to enrich the observation plan to achieve better word-level realization. On the per-
formance of ORGAN w/o TrR, a similar result of ORGAN w/o Graph is observed. This indicates that TrR can enrich the plan information, and stronger reasoning can help generate high-quality reports.
We also conduct experiments on the impact of the number (K) of selected n-grams, as shown in Table 4. There is a performance gain when increasing K from 10 to 20 and to 30 on both datasets. On the IU X-RAY dataset, B-2 increases by 2.4% and 3.7% and B-4 rises by 1.0% and 1.5%. A similar trend is also observed on the MIMIC-CXR dataset.

heading: 4.4 Qualitative Analysis
text: We conduct a case study and analyze some error cases on the MIMIC-CXR dataset to provide more insights. Case Study. We conduct a case study to show how the observation and the tree reasoning mechanism guide the report generation process, as shown in Figure 4. We show the generated reports of ORGAN, ORGAN w/o TrR, and ORGAN w/o Plan, respectively. All three models successfully generate the first three negative observations and the last positive observation. However, variant w/o plan generates "mild pulmonary vascular congestion without overt pulmonary edema" which is not consistent with the radiograph. In terms of the output of variant w/o TrR, "mediastinal silhouettes are unchanged" is closely related to observation Enlarged Cardiomediastinum instead of Cardiomegaly. Only ORGAN can generate the Cardiomegaly/POS presented in the observation plan with a TrR path. This indicates that observations play a vital role in maintaining clinical consistency. In addition, most of
the tokens in the observation mention mild to moderate cardiomegaly can be found in the observation graph, which demonstrates that the graph can provide useful information in word-level realization. Error Analysis. We depicte error cases generated by ORGAN in Figure 5. The major error is caused by introducing incorrect observation plans. Specifically, the generated plan of the upper case omits one positive observation (i.e., Enlarged Cardiomediastinum/POS), resulting in false negative observations in its corresponding generated report. Another error is false positive observations appearing in the generated reports (e.g., the bottom case). Thus, how to improve the performance of the planner is a potential future work to enhance clinical accuracy.

heading: 5 Related Work

heading: 5.1 Image Captioning and Medical Report Generation
text: Image Captioning (Vinyals et al., 2015; Rennie et al., 2017; Lu et al., 2017; Anderson et al., 2018) has long been an attractive research topic, and there has been a surging interest in developing medical AI applications. Medical Report Generation (Jing et al., 2018; Li et al., 2018) is one of these applications. Chen et al. (2020) proposed a memorydriven Transformer model to generate radiology reports. Chen et al. (2021) further proposed a crossmodal memory network to facilitate report generation. Qin and Song (2022) proposed to utilize reinforcement learning (Williams, 1992) to align the cross-modal information between the image and the corresponding report. In addition to these methods, Liu et al. (2021c) proposed the Contrastive Attention model comparing the given image with normal images to distill information. Yang et al. (2021) proposed to introduce general and specific knowledge extracted from RadGraph (Jain et al., 2021) in report generation. Liu et al. (2021a) proposed a competence-based multimodal curriculum learning to guide the learning process. Liu et al. (2021b) proposed to explore and distill posterior and prior knowledge for report generation.
Several research works focus on improving the clinical accuracy of the generated reports. Liu et al. (2019) proposed a clinically coherent reward for clinically accurate reinforcement learning to improve clinical accuracy. Lovelace and Mortazavi (2020) proposed to use CheXpert (Irvin et al., 2019) as a source of clinical information to generate clini-
cally coherent reports. Miura et al. (2021) proposed to use entity matching score as a reward to encourage the model to generate factually complete and consistent radiology reports. Nishino et al. (2022) proposed a planning-based method and regarded the report generation task as the data-to-text generation task.

heading: 5.2 Planning in Text Generation
text: Another line of research closely related to our work is planning in text generation, which has been applied to multiple tasks (e.g., Data-to-Text Generation, Summarization, and Story Generation). Hua and Wang (2020) propose a global planning and iterative refinement model for long text generation. Kang and Hovy (2020) propose a self-supervised planning framework for paragraph completion. Hu et al. (2022) propose a dynamic planning model for long-form text generation to tackle the issue of incoherence outputs. Moryossef et al. (2019) proposed a neural data-to-text generation by separating planning from realization. Su et al. (2021a) proposed a controlled data-to-text generation framework by planning the order of content in a table.

heading: 6 Conclusion
text: In this paper, we propose ORGAN, an observationguided radiology report generation framework, which first produces an observation plan and then generates the corresponding report based on the radiograph and the plan. To achieve better observation realization, we construct a three-level observation graph containing observations, observationaware n-grams, and tokens, and we propose a tree reasoning mechanism to capture observationrelated information by dynamically aggregating nodes in the graph. Experimental results demonstrate the effectiveness of our proposed framework in terms of maintaining the clinical consistency between radiographs and generated reports.
Limitations
There are several limitations to our framework. Specifically, since observations are introduced as guiding information, our framework requires observation extraction tools to label the training set in advance. Then, the nodes contained in the observation graph are mined from the training data. As a result, the mined n-grams could be biased when the overall size of the training set is small. In addition, our framework is a pipeline, and the report genera-
tion performance highly relies on the performance of observation planning. Thus, errors could accumulate through the pipeline, especially for small datasets. Finally, our framework is designed for radiology report generation targeting Chest X-ray images. However, there are other types of medical images (e.g., Fundus Fluorescein Angiography images) that our framework needs to examine.
Ethics Statement
The IU X-RAY(Demner-Fushman et al., 2016) and MIMIC-CXR (Johnson et al., 2019) datasets have been automatically de-identified to protect patient privacy. The proposed system is intended to generate radiology reports automatically, alleviating the workload of radiologists. However, we notice that the proposed system can generate false positive observations and inaccurate diagnoses due to systematic biases. If the system, as deployed, would learn from further user input (i.e., patients’ radiographs), there are risks of personal information leakage while interacting with the system. This might be mitigated by using anonymous technology to protect privacy. Thus, we urge users to cautiously examine the ethical implications of the generated output in real-world applications.
Acknolwedgments
This work was supported in part by General Program of National Natural Science Foundation of China (Grant No. 82272086, 62076212), Guangdong Provincial Department of Education (Grant No. 2020ZDZX3043), Shenzhen Natural Science Fund (JCYJ20200109140820699 and the Stable Support Plan Program 20200925174052004), and the Research Grants Council of Hong Kong (15207920, 15207821, 15207122).

heading: A Appendices
text: A.1 Observation Statistics There are 14 categories of observations: No Finding, Enlarged Cardiomediastinum, Cardiomegaly, Lung Lesion, Lung Opacity, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other, Fracture, and Support Devices. Table 5 lists the observation distributions annotated by CheXbert(Smit et al., 2020) in the train/valid/test split of two benchmarks.
A.2 Observation-aware N-grams Here are some of the observation-aware n-grams we use in our experiments, as shown in Figure 6. These categories are Enlarged Cardiomediastinum, Consolidation, and Cardiomegaly.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section "Limitations".

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section "Ethical Statement".

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section "Abstract" and Section 1 "Introduction".
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We will include the license and terms of use when releasing our code.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 and Appendix A.1.
C 3 Did you run computational experiments? Section 3.
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Thus, how to improve the performance of the planner is a potential future work to enhance clinical accuracy