Granularity Matters: Pathological Graph-driven Cross-modal Alignment for Brain CT Report Generation
Authors: Yanzhao Shi
Junzhong Ji
Xiaodan Zhang
Liangqiong Qu
Ying Liu
Venue: EMNLP 2023 Main (2023)
Keywords: Medical Report Generation
Contrastive Learning
Knowledge Graph
Brain CT
TL;DR: A novel framework to align fine-grained multimodal features of pathology for boosting Brain CT report generation.

Abstract: The automatic Brain CT reports generation can improve the efficiency and accuracy of diagnosing cranial diseases. However, current methods are limited by 1) coarse-grained supervision: the training data in image-text format lacks detailed supervision for recognizing subtle abnormalities, and 2) coupled cross-modal alignment: visual-textual alignment may be inevitably coupled in a coarse-grained manner, resulting in tangled feature representation for report generation. In this paper, we propose a novel Pathological Graph-driven Cross-modal Alignment (PGCA) model for accurate and robust Brain CT report generation. Our approach effectively decouples the cross-modal alignment by constructing a Pathological Graph to learn fine-grained visual cues and align them with textual words. This graph comprises heterogeneous nodes representing essential pathological attributes (i.e., tissue and lesion) connected by intra- and inter-attribute edges with prior domain knowledge. Through carefully designed graph embedding and updating modules, our model refines the visual features of subtle tissues and lesions and aligns them with textual words using contrastive learning. Extensive experimental results confirm the viability of our method. We believe that our PGCA model holds the potential to greatly enhance the automatic generation of Brain CT reports and ultimately contribute to improved cranial disease diagnosis.

Granularity Matters: Pathological Graph-driven Cross-modal Alignment for Brain CT Report Generation
Yanzhao Shi', Junzhong Ji', Xiaodan Zhang'; Liangqiong Qu Ying Liu² FFaculty of Information Technology, Beijing University of Technology, Beijing, China DDeparmment of Statisties and Actuarial Science, University of Hong Kong, Hong Kong, China Department of Radiology, Peking University Third Hospital, Beijing, China zhangxiaodan@bjut edu. lianggquehku. hk
Abstract
Pathological Graph
Vigh Awres
The automatic Brain CT reports generation can improve the efficiency and accuracy of diagnos- ing cranial discases. However, current methods are limited by 1) coarse grained supervision: the training data image-text format lacks de- tailed supervision for recognizing subtle abnor- malities, and 2) coupled cross-modal alignment: visual-textal alignment may be inevitably cou- pled in coarse- -eeniied manner, resulting in tangled feature representation for report gener- ation. In this paper, we propose a novel Patho- logical Graph-driven Cross-modal Alignment (PGCA) model for accurate and robust Brain CT report generation. Our approach effectively decouples the cross-modal alignment by con- structing a Pathological Graph learn fine- grained visual cues and align them with textual words. This graph comprises heterogeneous nodes representing essential pathological at- tributes (i.e., tissue and lesion) connected by intra- and inter-attribute edges with prior do- main knowledge. Through carefully designed graph embedding and updating modules, our model refines the visual features subtle tis- sues and lesions and aligns them with textual words using contrastive learning. Extensive experimental results confirm the viability of method. We believe that our PGCA model holds the potential to greatly enhance the auto- matic generation of Brain CT reports and ulti- mately contribute to improved cranial disease diagnosis.
Brain CT Sample
lawe dinentr
Repeet: Pars ligitkestt with lor deiid axsee
right endpid elass
Libe
Greph
Thsted Lain
Lusinn Graek
aound is the serpens specktdd mado is paneal varnee. sireunn acrennl
CAUDEC
Figure 1; An example of the pathological graph. Patho- logical elements (nodes) summarized from reports tissue attribute and lesion attribute, which are connected as tissue graph, lesion graph, and tissue-lesion graph with prior relations (edges). These three sub-graphs compose the fine-grained pathological graph.
reporting system aims to automatically generate ac- curate reports, which has the potential to lighten the workload of physicians and economize insufficient clinical resources in populated areas.
With the advent deep neural networks im- age captioning (Vinyals et al., 2015; Anderson et al., 2018), medical report generation (MRG) methods are increasingly ramping up (Jing et al., 2018; Wang et al., 2018; Li et al., 2019; Yang et al., 2021; Yan et al., 2021; Chen et al., 2021; Song al., 2022; Qin and Song, 2022; Yang al., 2023; Li al., 2023). Different from image cap- tioning. MRG task focuses on subtle yet crucial medical terminologies, with the report length typi- cally 4-6 times longer than those of nature object captions. This prompts MRG models to refine the dedicated consistency of salient pathological fea- tures between visual and textual modalities. To achieve this, recent studies have employed vari- ous cross-modal alignment mechanisms (Liu al., 2021; Wang et al., 2022; Li al., 2023), leveraging specific knowledge to effectively improve report generation.
1 Introduction
Writing diagnostic reports for Brain CT imaging with multiple scans is widely applied in medicine, to summarize the findings of cranial diseases. Nonetheless, this traditional clinical practice could be time-consuming and error-prone for radiologists due some subjective factors (e.g. fatigue and distraction) (Brady et al., 2012). A computer-aided
"Corresponding Authors

---

Despite the promising achievements, previous methods are still thwarted by the following two concerns. 1) Coarse-grained supervision: clini- cal practice, brain findings are often characterized by some subtle yet vital pathological elements that belong to different attributes (i.e. tissues and le- (Gris) et al., 2002), see Figure 1. However, medical images and long reports are always treated coarse-grained supervisory signals in the main- stream methods. Although few recent studies have explored the learning of fine-grained signals (Liu et al., 2021; Wang et al., 2022) chest X-ray samples, it is unsuitable for Brain CT which en- compasses more sophisticated pathology patterns. Thus, how to leverage the intrinsic fine-grained cranial knowledge to recognize subtle abnormali- ties remains an open question. 2) Coupled cross- modal alignment: Current cross-modal alignment methods MRG inevitably tend to couple visual- textual representation during roarse-grained align- ment, which is manifested as highly-similar atten- tion maps (Yang et al., 2021) or tangled semantic representations in instance-level contrastive learn- ing (Yan et al., 2021), resulting in inadequate fea- ture learning.
posed reconcile the learned visual features with their corresponding word embeddings in the report, to further boost the visual encoder and textual de- coder in report generation. In sum, our main contributions contain:
1. We propose novel framework to seam- lessly capture detailed domain knowledge from Pathological Graph, and explicitly align fine-grained visual and textual features of pathology, which improves the encoder and decoder of Brain CT report generation by shar- ing feature learning layers.
2.
We, for the first time, introduce the idea of de- coupling cranial tissues and lesions via Patho- logical Graph into the medical report genera- tion area, which is capable of handling fine- grained alignment between long-text report and multiple scans.
3.
We comprehensively validate our model on the BCT-CHE dataset. The experimental re- sults indicate that our method surpasses previ- ous arts in medical report generation.
2 Related Work
In this work, we propose a novel Pathologi- cal Graph-driven Cross-modal Alignment model, named PGCA, which introduces detailed Patho- logical Graph (PG) to seamlessly capture domain knowledge in samples, and explicitly extract fine- grained semantic correspondence between visual regions and diagnostic texts for accurate Brain CT report generation. Specifically, the PG contains heterogeneous nodes tissue and lesion attributes, which are connected by edges with prior knowl- edge. As shown in Figure 1, we decompose PG into two fixed Tissue Graph and Lesion Graph with intra-atrribute knowledge (relations in each attribute), and one dynamic Tissue-Lesion Graph with anterbbte knowledge (correspondences between two attributes for each case). Then, we respectively incorporate Tissue and Lesion Graphs into an attribute-oriented graph convolution net- work for dedicated node features learning, which is jointly updated via an Intra-Attribute Classifica- tion (IAC) loss module. While the Tissue-Lesion graph is updated via an Inter-Correlation Align- ment (ICA) loss module, which facilitates learning the complex tissue-lesion connections, in turn, ben- efits node feature representation. Finally, a cross- modal Contrastive Learning (CL) module is pro-
2.1 Medical Report Generation With Knowledge Graph
To mimic the expert knowledge of radiologists, the using of Knowledge Graph (KG) can endow the MRG model with better capabilities of abnormal recognition and has gained increasing research in- terest, which can be summarized into three types. The first type focuses on stressing clinical termi- nologies (e.g. abnormalities and discases). Li et al. (2019) extracts a set of chest medical terminolo- gies from the MIMIC-CXR dataset, which are re- garded as graph nodes. The edges linking nodes are assigned with attention weights to depict latent relations, which may be affected by error-uddated cases. The second type is to build universal graph that contains lesion nodes with stable edges ac- quired by prior knowledge (Zhang et al., 2020; Liu et al., 2021). Such relationships are stored in the adjacency matrix and learned by graph convolution networks. As extension of this graph, Li et al. (2023) proposes to dynamically add characterized nodes for each sample, but still limits represent another detailed attribute, i.e. pathological tissues. The last type utilizes NLP-rule based methods to extract detailed triplets from the training corpus

---

and build a clinical graph (Li et al., 2022). The sample-related triplcts are restored to learn specific knowledge, though the triplets may be wrong ex- tracted.
ation and Pathological Graph-driven Cross-modal Aligmment (PGCA), which interact by the shared visual and textual embedding layers.
3.1 Brain CT Report Generation
Our pathological graph seamlessly combines the advantages of the last two types. We first separately construct fixed tissue and lesion graphs as the way in the second type for dedicated pathology representing, and then extract structured triplets for each case to depict fine-grained relations between paired attributes.
Given the input scans S {s1,--., 8N), where N denotes the number of scans in each sample, the target of this branch is to generate diag- nostic report Y {yi,.... yar) with M words. The report generation model follows the tradi- tional encoder-decoder pipeline. Firstly, we ap- ply the ResNet101 (He et al., 2016) to extract visual features of S that contain global features F = (fi..... fN] E Nxd (N - 24, d = 2048) and grid features G {91..... .N) E R (H 196). Then, F and G are embedded by visual encoder, resulting in global and spatial vi- sual feature embeddings Vy and Vg. respectively. Finally, visual features V = {[V,, Vg} is used to generate long reports in the decoder, which con- tains a textual embedding layer and a language model with weywords-driven interactive recurrent network (Yang et al., 2021). We train the parame- ters 0 by minimizing the cross-entropy loss, which can be expressed
2.2 Cross-modal Feature Alignment
Learning medical semantics across visual and tex- tual modalities is essential for MRG model to gen- erate logical and accurate reports. Thus, the cross- modal feature alignment has attracted growing in- terest in recent studies, which can be roughly di- vided into three stages. In the early stage, vari- ous attention mechanisms (Jing et al., 2018; Wang et al., 2018; Xue et al., 2018; Yang et al., 2021) are adopted to extract abnormal visual features and guide the generation of diagnostic texts. However, the attention map updated by naive cross-entropy loss could not sufficiently represent the complex cross-modal patterns. For the second stage, Chen al. (2021) introduces the memory vector to re- store multi-modal relations, which is further ex- tended with reinforcement learning (Qin and Song, 2022) and class-related prototypes (Wang et al., 2022). Since the visual and textual features across different samples are highly similar, it is still chal- lenging to capture the essence of abnormal clues. Parallel to the second stage, the incorporation of contrastive learning can effectively distinguish sim- ilar features, which paves the next phase. Yan et al. (2021); Yang et al. (2023) utilize instance-level con- trast (i.e. image-report pairs from same cases are positives, otherwise are negatives) in MRG models to enhance the consistency of multi-modal features. Li et al. (2023) further improves the visual repre- sentation with graph features and benefits the report generation. Our PGCA follows the third stage. Dif- ferent from previous methods, we refine complex visual features into dedicated node features, which are contrasted with related medical word features for fine-grained feature alignment.
Lg Σ log po (1 IV, xim-11,,
(1)
where p( VV, (1mm-1 denotes the predicted prob- ability for the m-th word based on visual features V and previous word embeddings 21, 22, ..., Tm- Tm-1-
3.2 Pathological Graph-driven Cross-modal Alignment
To boost the report generation, we propose to learn fine-grained visual-textual representations to im- prove the encoder and decoder by co-training with PGCA, which contains graph construction, graph embedding and updating, and cross modal con- trastive learning.
3.2.1 Pathological Graph Construction
The knowledge graph is widely adopted to repre- sent the relationship of medical entities. Differ- ent from previous Chest report generation meth- ods (Zhang al., 2020; Liu et al., 2021) with only 1-2 images and fewer pathological entities for di- agnosis, generating Brain CT reports from multi- ple seans meets more challenges. Regularly, ele- ments to be reported from seans contain some key
3 Methodology
As shown in Figure 2, our framework contains two parallel branches, namely Brain CT report gener-

---

Puthelogical Graph
Ieto Aftiidt Koeeinii
Idt Aiiibe Kambeey
Extreied
Tora Goop
Leies Grops
Thee Gregk
Suanc
Lade Riiee
Trath Report
Carelese
Eatradd
Lake
Peus hodk
condiaii Algannen Manii
Fre
Iustndl Eesholding
Commdaaton Nevwek
Nol
Fiatane
pity
Brois CT Scans
Inke-Cerebbtinn
Costradine LLarann Modaly
skand Testral kmbbediter boyrr Tested Boeaker
Choilicaiis Maaaak
Leac AAggeenst Modak,
Grend Touth
Lovkee Nedeck
Tndeeiing
Tostal
Made
Hepurt
Gonentias Lave
Skared Vixual Fander
ThonedLokoo Kdly
(4) Braia CTT Ropart Generation
Shared Tonral Entwddiing
Figure 2: An overview of the proposed model, which comprises (a) Brain CT report generation and (b) Pathological Graph-driven Cross-modal Alignment. Communicated by shared visual and textual embedding layers, both (a) and (b) are simultancously used in the training stage while only (a) is used for testing.
brain tissues and their associated lesions, which are depicted as fundamental brain findings (Grif- fin et al., 2002). Motivated by this, we propose to organize Pathological Graph (PG) to cover clinically important elements in Brain CT sam- ples. We first obtain set of critical tissues and lc- sions, denoted T [41 .... AN, (Nr 19) and L .... ] (Nr 13) respectively, where N₂ and Nj are the numbers of selected tissues and lesions, based on the knowledge of experts and the word frequency in training corpus. These elements serve as the nodes in our PG. Given the heteroge- ncous nature of the nodes (i.e., tissuc and lesion attributes), we proceed to link them using edges derived from various prior knowledge, such as re- lations in tissues, relations in lesions, and relations between tissues and lesions, as shown in Figure 1, This process effectively partitions the PG into three distinct sub-components: the Tissue Graph, Lesion Graph, and Tissue-Lesion Graph. Next, we mainly introduce the building of these three components.
<lissue,lesions pairs from each sentence the training corpus with artificial refining, and thereby relations in lesions can be summarized with tissues as bridge (more details see Appendix A.2). Then, we define the Lesion Graph ((4) (v(*), g(1)
with N₁ lesion nodes and global node, where ((D) ((L) denote the nodes and edges, respectively.
3) Tissue-Lesion Graph: Since Tissue and Le- sion Graphs are fixed to learn stable semantics, Tissue-Lesion Graph is designed to depict detailed attributes' interconnections for each case. We modify the extracted <tissue,lesion> pairs into ritisuu.relation,eesssseee
triplets, where the rela- tion E {0, 1} denotes whether the tissue paired with the lesion. For each tissue node o!?') E G" its associated lesions be denoted as a local Le- sion Graph ((4.) E ((L), and we represent the Tissue-Lesion Graph as ( the inter-attribute correlations for each instance.
to store
As aggregation of three subgraphs, the PG encapsulates wealth of medical knowledge. Next, will explore how represent this knowledge within the model.
1) Tissue Graph: Clinically, physicians often divide Brain CT scans into layers to facilitate the diagnosis of different brain tissues. Follow- ing this diagnostic pattern, we enhance the cor- relation of tissues that existed in the same layer by prior medical knowledge (more details see Ap- pendix A.1), and then define the Tissue Graph ((T) - (,(")) g(T) with N, clinically essential tissue nodes and a global node, where (TT), (TT denote the nodes and edges connecting them.
3.2.2 Pathological Graph Embedding
Different from previous single-attribute learning methods (Zhang et al., 2020; Liu et al., 2021), we adopt an attribute-oriented graph convolution net- work in parallel for both Tissue and Lesion Graphs. Specifically, we first initialize the graph features by spatial visual features V, as (Zhang et al., 2020) to initially map each pathological node 2; to dif- ferent spatial regions, resulting in initialized tissue graph feature T and lesion graph feature L. More
2) Lesion Graph: Following the conclusion that lesions diagnosed in the same tissue are more re- lated (Zhang et al., 2020), we extract adequate

---

details are in Appendix A.4. Then, our graph con- volution network follows Zhang et al. (2020); Liu et al. (2021) to propagate two graph features as:
tissue index. Following this way, can acquire the fine-grained pathology-grounder node features for both tissues and lesions.
While IAC captures the semantics of single at- tribute nodes, learning the inter-attribute relations between tissues and lesions is also essential, as our task is to generate reports with complex cranial findings rather than simple classification. There- fore, we design Inter-Correlation Alignment (ICA) module to match the correct tissue-lesion pairs via BCE loss. Specifically, we first extract the triplets in Tissue-Lesion Graph to depict tissue- lesion relations in a current case. As such, each sample is equipped with an attribute alignment ma- trix Ma E (0, 1) Nex.N as the ground-truth label, where the matrix element Eij denotes the i-th tissue is associated with the j-th lesion in this sam- ple. Afterward, we use dedicated node features (To and Iua) to predict the inter-correlation matrix as:
T 11 f = updatez ( msgez [ AL)), (3)
update (Tý, msger (Ty. Ar)), (2)
where Ty. L', respectively denote the tissue and lesion graph features in the 1-th layer, and Ar,Ar are the normalized Laplacian of adjaceney matrixes according to Tissue and Lesion Graph. msgei, msgc2 are feature aggregation functions based on the adjacency matrix, while update, and update propagate node features in PG.
Afte 2-layer propagation, we obtain the tis- sue graph embedding T3 = [Tii:... Tf Ny. E Nxx and lesion graph embedding Ly (Inn....LLNn)) E RRM where Ty, E Nxd, and Lf; E Nxxd respectively denote the i-th tissue and lesion node embedding. Afterward, we use a global average pooling on T and L to converge the information of N scans and obtain the generalized node features of tissues Ta C Ne and lesions L E IR N,d,
Ma ( W + ba) E R NoN.
(5)
where Wa are parameters. The mutual informa- tion between two attributes finally preserved by minimizing the following ICA loss:
3.2.3 Pathological Graph Updating
N
Since the tissues and lesions are sparse and se- mantically correlated in Brain CT, node features in PG are supposed to be updated with the follow- ing concerns: 1) Concrete semantics of pathology groundings; 2) Tissue-lesion relations. Note that the updating is only the tuning of graph node fea- tures and does not involve any new connections within the tissue and lesion subgraphs.
LICA= Σ)
W WICA log Muj
(6)
+ - (1)-Ma)) log(1 - Maj)l
where WICA is a learnable weight matrix, and Maj.
Ma's denote the (i-th.j-th) value in the ground-truth and predicted attribute matrix, respectively.
To accurately represent the medical semantics of Intra-Attribute classi-
3.2.4 Cross-modal Contrastive Learning
each node, introduce fication (LAC) module dynamically update the node features. Separate IAC modules are employed for tissue and lesion nodes. Here, we take the tissue nodes as an example to illustrate IAC. We design a classification head, which takes Ta as input, to predict the tissue labels O2 E R.,. We then use BCE loss to optimize the IAC module as:
Considering that detailed medical information is significant to find subtle pathological clues that depict critical medical conditions, the goal of our Contrastive Learning (CL) module is represent the alignment of fine- mne-grained pathological seman- tics from visual and textual modalities. To achieve this, we regard the nodes learned by IAC and ICA as fine-grained visual semantics since they compre- hensively considered the intra- and antrr-at knowledge and focused on salient pathological re- gions. Then, we extract textual features of corre- sponding words in the report via the shared textual embedding layers in textual decoder and map each node feature to its matched textual embeddings by contrastive lcarning. Specifically, we use tissuc nodes for example. Given the j-th sample with N
Nr w log Of' -(- -C''),
CIAC. Σ
where C. E N denotes the ground tissue labels extracted from reports, WIAC, is a weight matrix for tissue node updating, and i denotes the current

---

Methods
B1
B2
B3
B4
M
RG
C
P
R
F1
Show-Tell(Vinyake 2015)!
37.2
25.5
18.0
12.9
25.8
35.6
16.6
51.4
53.4
49.0
Soft ATT(Xu al., 2015)!
42.0
28.4
19.7
13.9
27.6
35.1
18.5
51.8
65.5
55.1
HRNN(Krause 2017)
26.3
18.3
13.0
28.8
16.3
37.7
40.6
Up-Down(Anderrson et ul., 2018)
40.7
27.6
19.1
13.5
27.0
35.4
20.1
51.9
54.5
MRMA(Xue et ul., 2018)
39.7
26.9
18.8
13.4
29.0
34.9
15.6
53.0
59.0
52.7
R2Gen(Cheen et al., 2020)1
43.1
27.5
18.3
12.4
28.1
34.1
15.8
52.2
57.3
51.9
WCL, et al., 2021)T
42.6
28.6
19.9
14.1
27.4
34,9
17.5
52.3
60.9
52.7
R2Gen-CMN(CH)) et al., 20
40.1
25.1
16.0
10.3
26.2
32.3
13.9
51.1
50.2
46.5
CMMRL((Qin and Song. 2022)
33.5
19.5
11.7
7.3
22.3
28.5
10.0
49.3
45.4
44.0
XProNer(Waan et al., 2022y
40.1
27.1
18.9
13.2
26.1
35.2
17.9
52.0
61.3
53.8
NGAM(Yamg al., 2021)1
43.6
29.3
20.4
14.5
27.9
18.3
52.9
66.5
55.9
45.0
30.8
21.6
15.5
28.7
36.5
19.9
67.7
57.2
Table I: The performance of our PGCA compared with previous statc-of-the-art models on the Brain CT report generation dataset BCT-CHR. The best results are highlighted in bold. denotes our re-implementation results.
scans and a report, we first project node features and associated word embeddings into latent space,
There are 49,152 CT scans and 2048 Chinese re- ports from 2048 anonymous samples, and each sample includes 24 scans over multiple patholog- ical layers for various abnormal detection and paired patient report. Following Yang et al. (2021), the dataset is split by 7 for training, testing, and validation, respectively. When tokenizing re- ports, the words with less than 2 occurrences are dropped. counting to 798 words in the vocabulary. Notably, the English translations are only for better understanding and are not used in training.
resulting in Vís (v1)) Vij (2)
and (Mg))
Vig
(1) (2) Rij (Ne,) where Vi
Rej [Ri Rí,
(1) E RRd denote the projected visual Rd, Riy
and textual features, and Nt, is the number tis- sues reported in the j-th sample. We then use the loss function similar to temperature-normalieed In- foNCE (van den Oord et al., 2018 8) loss for visual- textual alignment:
N1.
(V( (u) ,16, (1))
Σ log u=1
exp(
4.2 Evaluation Metrics
LCL, =
No
(1 (M) Rn)
We fully evaluate the performance the NLG (Natural Language Generation) and CE (Clini- cal Evaluation) metrics. NLG metrics contains BLEU (Papineni et al., 2002), METEOR (Lavic and Agarwal, 2007), ROUGE-L (Lin, 2004) and CIDEr (Vedantam et al., 2015), which are denoted as B1, B2, B3, B4, M, RG. and C. Then, accord- ing to the pathological knowledge obtained by diologists, use 24 keywords ("basal ganglia", "edema", "lateral ventricle". etc.) to evaluate the CE metrics: (Precision), (Recall), and F1 score.
exp(
(7)
where s(.) measures the cosine similarity between cross-modal vectors, and the temperature hy- perparameter. Following the same process of tissue semantic alignment, we can also obtain the lesion contrastive learning loss LCL--
3.3
Overall objective function
Finally, our model optimized by the total loss regarding the report generation branch and PGCA branch, which is defined as:
4.3 Implementation Details
L C= A1CIAC + AeCICA A3CCL,
We set the scan number N = 24 for each sample and reshape the size of scans to 512 x 512. The scan features are extracted by ResNet101 (He et al., 2016), which is pretrained on ImageNet (Deng et al., 2009) and fine-tuned CQ500 dataset (Chil- amkurthy al., 2018). The hyperparameters are tuned via the validation set. Empirically, the loss coefficients {^1, Az), temperature value т and graph node dimension dg are set to {0.3, 0.001, 0.2). 0.4 and 512, respectively. We
where CIAC and LCL are calculated by the sum of tissue and lesion aspects. Å1, A2, and Aa are the coefficients balance the total loss.
4 Experiments
4.1 Dataset
We evaluate our model on the Brain CT report generation dataset BCT-CHR (Yang et al., 2021).

---

Attributes
Module Loss
Methods
bissue
lesion
AI
LICA
Lea.
Baseline
K
X
K
40.7
27.5
19.2
13.8
26.4
35.3
16.3
x
X
/
41.8
28.4
19.9
14.2
27.1
35.7
18.7
(b)
/
x
/
X
K
42.5
28.9
20.3
14,6
27.3
35.7
18.0
(c)
x
X
v
43.5
29.6
20.7
14.9
27.8
35.9
19.7
(d)
X
43.9
(e)
X
V
44.1
29.8
20.8
14.8
27.8
35.9
19.3
Ours
V
V
45.0
30.8
21.6
15.5
28.7
36.5
19.9
Table 2: Ablation studies of our proposed method. The Baseline model is encoder-decoder framework with attention mechanism that is similar to (Yang et al., 2021). (a), (b), (c). (d) and (e) respectively denote the adding of different pathological graph knowledge (i.e. tissue and lesion) and module losses.
train the model with Adam optimizer (Kingma and Ba, 2015) on an NVIDIA RTX 3090 GPU with a batch size of 4 for 60 epochs. The learning rate is initially set 4e-4 in the first 30 epochs, and decrease it by 0.8 rate per 3 epochs after that.
cal graph and fine-grained cross-modal alignment, our PGCA achieves the best performance in con- trast with previous arts. Lispecially, compared with the only work WGAM for Brain CT report genera- tion, PGCA gains remarkable improvement in B3 (20.4% 21. 21.6%), RG (35.1% + 36.5%), and Fl (55.9% + 57.2%), which justifies that PGCA does not confuse the learning of some essential patho- logical details, but generating sentences with more accurate topics. It is worth noting that PGCA is slightly inferior on CIDEr, which is more sensitive to word frequency. The reason may be that our graph contains some high-frequency terms, leading to more occurrences of them in generated reports and slightly decreasing the CIDEr.
4.4
Results and Discussion
4.4.1
Comparison Study
Besides the only Brain CT report generation method WGAM (Yang et al., 2021), also re- produce some SOTA models in image captioning (Show-Tell (Vinyals et al., 2015). Soft ATT (Xu et al., 2015), HRNN (Krause al., 2017), Up- Down (Anderson et al., 2018)) and chest X-ray Report generation (MRMA (Xue et al., 2018), R2Gen (Chen et al., 2020), WCL (Yan et al., 2021), R2Gen-CMN (Chen et al., 2021), CMMRL (Qin and Song. 2022), XProNet (Wang et al.. 2022)) on BCT-CHR dataset for comprehensive comparisons.
4.4.2 Ablation Study
Table summarizes the results of ablation studies to verify the contributions of graph knowledge in- jection and fine-grained contrastive learning. We remove the supervision of attention in WGAM our Baseline, and then progressively add attribute components (lissue and lesion) and the module losses (CiAC, LICA and LCL), respectively de- noting the incorporation of two types attribute knowledge in pathological graph and the utilization of proposed modules (i.e. IAC, ICA, and CL).
As shown in Table 1, our method outperforms the competitors on almost all evaluation metrics. Specifically, the negative results in Show-Tell in- dicate that complex medical semantics may not be fully captured without an efficient cross-modal in- teraction. Benefit from the cross-modal attention (Soft ATT, Up-Down, and WGAM) and instance- level contrastive learning (WCL), the models gain improving results. Unexpectedly, by simply us- ing memory vectors without domain knowledge, R2Gen-CMN and CMMRL perform poorly on most metrics, while XProNet adds class-related knowledge and gets better results. This reflects the effect of knowledge and the difficulties of our task.
As shown in Table 2, the advantages of adding tissue and lesion attributes can be well reflected by the improvement from Baseline to (b) and further to (d), which justifies our assumption for learn- ing subtle tissues and lesions. The comparison between (b) and (c) also indicates the contributions brought by fine-grained contrastive learning. Note that, (a) denotes the Baseline model with the incor- poration of CLIP loss (Radford al., 2021) that directly uses image-text pairs for coarse grained contrastive lcarning. We can observe that, only with fine- imeinned tissues for detailed feature align-
WGAM benefits from weakly-guide attention to capture visual features of Brain CT and achieves higher BLEU scores. Despite these successes, de- tailed visual and textual features are not sufficiently learned MRG systems, which fails to generate ac- curate Brain CT reports. With the use of pathologi-

---

W/ IAC
w/ [AC+CLL
1.0
1.0
0.8
0.5
0.4
0.2
211
0.0
02
0.4
0.6
D.B
1.0
0.0
0.2
0.4
0.6
D.B
14
Restiet
0.8
0.5
0.4
0.4
Sieederr ARC
Femider ARCC
0.2
Setance ACE
8.0
0.6 0.5
0.8
0.6 0.5
0.8
1.2
Paloe Postive Noto
Palsc Pestave Noto
Figure Comparison of pathological elements classification with different model settings classic ResNet model. The left-top chart shows the ROC curves of TAC module without the effect of other modules, and each curve denotes one pathological element. Based on the first model, CL and ICA modules are gradually added and corresponded the following charts. The last chart shows the classification performance of classic ResNet.
22 0.001 33 0.2
- 33 0.2
X1 0.3 0.001
Bis-44
Bicu
16
10
Vite Aoperpammetee.
Vakke Imperpaaeeeee
Valae orhoperpameeee 13
Figure 4: Parameter sensitivity tests for the loss function. Each hyperparameter is evaluated in a specific range, with the other two kept their optimal values.
ment, (c) has already surpassed (a) by a large mar- gin, and when the lesion attribute is considered, (e) gains further improvements. This indicates the ef- fectiveness of our fine-graince cross-modal feature alignment idea in MRG. Compared with (d), (e) gains inferior results on METEOR, ROUGE and CIDEr, the reason may that pathological seman- ties two attributes are separately learned without a clinically important interaction. With the incor- poration of ICA, our final model achieves the best performance. These results verify the capabilities of our modules to boost report generation.
pathological elements summarized by experienced physicians. Since classification is only performed by our TAC module, the left chart shows the perfor- mance of the model only with IAC and the report generation branch. With the progressive addition of CL and ICA modules, the model gains better performance, which further indicates that CL and ICA can help enhance the node feature represen- tation and boost the recognition of pathological elements. Besides, we also compare the classifica- tion performance with the classic ResNet101 (He et al., 2016) model. It is noticeable that our model outperforms the ResNet by large margin, which not only indicates the effectiveness of our graph learning and cross-modal alignment methods but
To further confirm the module contributions for pathology identification, in Figure 3, we visualize the ROC curves for the classification of 10 essential

---

Ground Trark Sample
Groudd Truth Labe
Predieted Label
Generated Regurts
Lbwe Label
Feiied Lake
X0I Genndall
RE (hig) dewilg)
Vis Midtyll low destiies with purtillyy zker bonke histoul bead gazpli patital vantisko, conpan ceriscaie The whih matorr of the lvorl varnecie deciona broandlyy Bilieral craoral liiionig of maillay sin wasssas))
edees
SPP (eders))
(03 1o 10 th th E
MR 00
MELL [suppecwall
AIN SARER SHE BE Rerusee 2: R0 2 SE 0558818 (Mtoe-ll high ceeett with lon detsly clem in the lell perical and occiital loss with compession adjacet bram parcadtyrs, dulloctn
Fige 2020 Ui sanitthaaaae
WGAM
86, 2200000:17 BIEES ES th
(Misnal poaday hash dunatty with ficctal
1985 mnsii aractan))
sakan, and camoinng ket vontide compossioon millin sructare diiftedd (ggtt)
edenta
(saroo))
The milline serattre is slights riati dilled. aassenn hood geogla is swn with sonvel loe dasilty and de write sudt:: of the iscral sectrdde bizzonlly))
1.80 (krn damidy
Fituaral ventridai
M Osuel pinglal
X01E (decsea)))
98 13 Deedd gangilatt..
the pericul like, compossiion of lell liceal vertiile The midiie strudture
@ a8 fS. (Paady denwar inwdereet
Bottish fols Stedere 82 (orpppoaii 0.88
decrase
cate dentd Spectial low dusitty is secs in the kit haall pengla,, and the wiske metter oh ateol vounice dedecressed bisaeolly))
Figure 5: Example of reports generated by the Baseline, WGAM, and our proposed model. The correct tissues, lesions, and error- pror-predicted pathological elements are marked in green, purple and red, respectively. English translations of the Chinese reports are given for better understanding.
Sentenneo #1SSBRR
knowledge and fine-graine lne-grained cross-modal feature alignment, PGCA can not only recognize complex relations of pathologies and correctly predict la- bels. but also generate high-quality reports based on the understanding of pathological elements. We also visualize the ability to recognize fine-grained pathology semantics in Figure 6, the salient regions grounded by our IAC module are generally consis- tent with human-labeled boxes. These qualitative analyses further demonstrate the superiority of our PGCA model.
Bist Demponl
100)
(elenn)
Graaned Figure 6: Visualization of fine- gre--graied pathology recog- nition. Given sentence, Grad-CAM is used to ground salient visual regions of pathological elements.
Tloass
Hersse-boee Listed PGUA Gronded
Lokis
also hints report generation and classification can boost each other.
To examine the impact of hyperparameters of loss (i.e., \1, X2 and A in Eq. 8) on our model, we provide parameter sensitivity experiments in Figure 4, where we fix two optimal hyperparam- eters and vary the third within a specific range. We can observe that too low or too high values of hyperparameters decrease the performance. The reason may be because the too low score of X1 X2. and \3 respectively lead to inadequate pathologi- cal semantic learning, inter-attribute matching, and visual-textual alignment, which degrades the fea- ture representation. Too high Å₁ 22. and A3 may weaken the training of report generation-related parameters and cause sub-optimal results.
5 Conclusion
We propose a Pathological Graph-driven Cross- modal Alignment model for Brain CT report gener- ation. First, we construct a Pathological Graph that incorporates detailed medical knowledge, allowing for the injection of this knowledge into dedicated node features through graph embedding and updat- ing, thereby achieving fine-grained visual represen- tations. Second, we align the learned node features with the corresponding word embeddings by cross- modal contrastive learning, to further boost report generation. Extensive experiments demonstrate that our model achieves superior performance in generating clinically accurate reports.
4.4.3 Qualitative Analysis
Limitations
Figure 5 visualizes the generated reports from a qualitative perspective, we list the ground truth image-report pairs with pathological labels, pre- dicted labels generated by IAC module, and the reports generated by Baseline, WGAM, and our PGCA. It is observed that Baseline and WGAM suffer from generating suboptimal reports with inaccurate cranial tissues and wrong-matched lc- sions. With the incorporation of our enriched graph
This paper is mainly toward Brain CT medical re- port generation and may not generalize well to other medical imaging, such chest datasets (c.g. MIMIC-CXR and IU-Xray) and ophthalmology datasets (e.g. FFA-IR), without further adaption. Besides, we only focus on mining detailed tissue and lesion elements in brain findings and build- ing fine- grained visual-textuaa alignment based

---

on them, which lacks the consideration of more specific details like tissue orientation or lesion size. the future, explorations of how to incorpo- rate useful medical knowledge with different types and granularity into the medical report generation model will substantially contribute to this field.
critical findings in head et scans. arXiv preprint arXiv:1803.05854
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages -45-
Ethics Statement
Colette M Griffin, Declan T Chard, Geoff JM Parker, Gareth J Barker, Alan Thompson, and David H Miller. 2002. The relationship between lesion and normal appearing brain tissue abnormalities in early relapsing remitting multiple sclerosis. Journal of neurology. 249:193-199
The medical images the BCT-CHR dataset have undergone a thorough process of de-identification to protect participants' privacy. Additionally, we have obtained the required data permissions, ensur- ing our work meets ethical standards.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61906007, 62276010, and 62306253, in part by the R&D Program of Beijing Municipal Education Commission under Grant KMM022 and
Baoyu Jing, Pengtao Xie, and Fric P. Xing. 2018. On the automatic generation of medical imaging reports. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics, 2018, Mel- bourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2577-2586.
K72002100005099.
References
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In 2018 IEÉE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 6077-6086.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hicrarchical approach for gen- erating descriptive image paragraphs. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3337-3345.
Adrian Brady, Risteárd Ó Laoide, Peter McCarthy, and Ronan McDermott. 2012. Discrepancy and error in radiology: concepts, causes and consequences. The Ulster medical journal, 81()):3.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceed- ings of the Second Workshop on Statistical Machine Translation, WMT@ACL 2007, Prague, Czech Re- public, June 23, 2007, pages 228-2311
Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. 2021. Cross modal memory networks for radiology report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5904-5914.
Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2019, Knowledge-dring encode, retrieve, paraphrase for medical image report generation. In The Thirty-Thire AAAI Conference on Artificial Intel- ligence, AAAI 2019, The Thirty-Firsi Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Hon- olula, Hawaii, USA, January 27 February 2019. pages 6666-6673.
Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xi- ang Wan. 2020. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020. pages 1439-1449.
Sasank Chilamkurthy, Rohit Ghosh, Swetha Tanamala, Mustafa Biviji, Norbert G. Campeau, Vasantha Ku- mar Venugopal, Vidur Mahajan, Pooja Rao, and Prashant Warier. 2018 Development and valida- tion of deep learning algorithms for detection of
Mingjie Li. Wenjia Cai, Karin Verspoor, Shirui Pan. Xiaodan Liang, and Xiaojun Chang. 2022. Cross- modal clinical graph transformer for ophthalmic re- port generation. In IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022,

---

New Orleans, LA, USA, June 18-24, 2022, pages 00624-206333
Jun Wang, Abhir Bhalerao, and Yulan He. 2022, Cross- modal prototype driven network for radiology report generation. In Computer Vision- ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23- 27, 2022, Proceedings, Purt XXXV, pages 563-579. Springer.
Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, and Xiaojun Chang. 2023. Dynamic graph enhanced contrastive learning for chest x-ray report generation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1334-3343.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. 2018. Tienet: Text-image em- bedding network for common thorax disease classifi- cation and reporting in chest x-rays. In 2018 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pages 9049-9058.
Chin-Yew Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2021. Exploring and distilling posterior and prior knowledge for radiology report generation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 13762.
Kelvin Xu. Jimmy Ba, Ryan Kiros, Kyunghyun Cho. Aaron C. Courville, Ruslan Salakhutdinov, Richard S. 7emel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, pages 2048-2057.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318.
Yuan Xue, Tao Xu, L. Rodney Long, Zhiyun Xue, Sameer K. Antani, George R. Thoma, and Xiaolei Huang. 2018. Multimodal recurrent model with at- tention for automated radiology report generation. In Medical Image Computing and Computer Assisted Intervention MICCAI 2018 21st International Con- ference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, pages 457-466.
Han Qin and Yan Song. 2022. Reinforced cross-modal alignment for radiology report generation. In Find- ings of the Association for Computational Linguistics: ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 448 458.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- ur, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, pages 8748-8763.
An Yan, 7exue He, Xing Lu, Jiang Du, Eric Y. Chang, Amilcare Gentili, Julian J. McAuley, and Chun-Nan Hsu. 2021. Weakly supervised contrastive learning for chest x-ray report generation. Findings of the Association for Compatational Linguisties: EMNLP 2021, Virtual Event / Punta Cana, Dominican Repub- lic, 16-20 November, 2021. pages 4015-4015.
Shuxin Yang. Xian Wu, Shen Ge, Zhuozhao Zheng, S. Kevin Zhou, and Li Xiao. 2023, Radiology report generation with a learned knowledge base and multi- modal alignment. Medical Image Anal., 86: 102798.
Xiao Song. Xiaodan Zhang, Junzhong Ji, Ying Liu, and Pengxu Wei. 2022. Cross-modal contrastive attention model for medical report generation. In Proceedings of the 29th International Conference on Computa- tional Linguistics, COLING 2022, Gyeongju, Repub- lic of Korea, October 12-17, 2022, pages 2388-2397.
Sisi Yang, Junzhong Ji, Xiaodan Zhang. Ying Liu, and Zheng Wang. 2021. Weakly guided hicrarchical encoder-decoder network for brain report genera- tion. In IEFE International Conference on Bioinfor- matics and Biomedicine, BIBM 2021, Houston. TX, USA, December 9-12, 2021, pages 568-575 IEEE.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 180iv.880.000 0377003748
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus- based image descrip- tion evaluation. In IEÉÉ Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566-4575.
Yixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu, Alan L. Yuille, and Daguang Xu. 2020. When diology report generation meets knowledge graph. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA. February 7-12, 2020, pages 12910-12917.
Oriol Vinyals. Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston. MA, USA June 72 2015, pages 3156-3164

---

A Appendix
ID
Tissue
A.1 Details of Tissue Graph Construction
tu (i ventricle)
fi. Riö (thalamus)
In the Tissue Graph, nodes are defined as Nt tissue terms, which are summarized in two aspects: 1) The knowledge and expertise of experienced radi- ologists: 2) The frequency of tissue words in the training corpus. We present these tissues in Table 3. Then, with the given nodes, a rule-based algorithm is designed to assign the connecting edges via the prior domain knowledge, i.e. tissues in the same scan layer are more related, otherwise not. Inspired by Li et al. (2022), our algorithm contains the fol- lowing three steps:
1 E (basal ganglia)
4
RE T (brainstem)
[[f*(ppriiee lobe)
H0lf (temprral lobe)
(frontalall lobe)
8
(occipita lobe)
(ventricee
10
(midline structure)
3 (third ventricle) 11
12
Ri Ri ventricle)
13
J (sphenoid sinus)
14
Mp (cthmoid sinus)
1) Knowledge Formatting: We format the ex- pert knowledge into <layer, pairs, where layer indicates a layer name (c.g. "canthus carline layer", "suprasellar cistern layer", "upper cerebral cortex layer", etc.), and tissue denotes the tissue that can be observed in this layer.
15
E Aij SE
(maxillary sinus)
Si (temporal occipital lobe)
17
4 R [ [T (cenneruu semiovale)
18
Ris (brain parenchyma)
19
(sullu cerebri)
Table 3: Components of tissue terms in our Tissue Graph, which are summarized by the knowledge of radi- ologists and the word frequency. English translations of the Chinese words are given for better understanding.
2) Dependency Parsing: With the formatted knowledge pairs, for each layer, we collect set of paired tissues and then parse the dependency of these tissues into siissuci,lissuee pairs, where lissue1, lissuez denotes two different tissues paired with the same layer. We traverse all the layers and finally obtain a large set of tissue depen- dency pairs.
Number
Tissue Layer <layer,tissuer 61
19
8
3) Entity Linking: A matrix E. E N.x.N is set store the relation of N₂ tissue nodes, which is initialized as an identity matrix. Then, we traverse the extracted tissue dependency pairs <tissuen,lisssee>>> and add the weight by 0.2 to update the bidirectional edges of tissue; and lissuez for each time. If the edge between two dif- ferent tissues reaches 0.8, we stop to add it. Finally, we add a column and a row with the weight of 1.0 to depict the fully connected edges of global node, resulting in the edge matrix E, E (NN-+)))(NN))) In total, the Tissue Graph contains 133 tissue dependency pairs, and more information are listed in Table 4.
ttissue,,issuee> 133
Statisties of Tissue Graph, contain- ing the number of tissue entities, layer categories. <layer, tissue> pairs, and <tissue,, pairs.
Table 4:
sions diagnosed in the same tissue are more related, otherwise not.
In Knowledge Formatting, we extract ade- quate <tissue,lesion> pairs from each sentence in the training corpus, and manually adjust the pairs to ensure correctness. In Dependency Pars- ing, we parse the dependencies of lesions into <lesson,, where lesion1, lesiong de- notes two different lesions paired with the same tissue. In Entity Linking, we store the relation of Nj lesion nodes in Flg E IR N; which is initial- ized as an identity matrix. We traverse the collected <lesion <lesionn,esion2s pairs and add the weight by 0.1 to update the bidirectional edges of lesion, and lesionz for each time until the score reaches 0.9. Finally, a column and a row with the weight of 1.0 are added to represent the fully connected edges
A.2 Details of Lesion Graph Construction
We comprehensively combine the expert opinions and the word frequency in the training corpus to extract Ni lesion terms our lesion nodes, which are presented in Table 5. Similar to the linking of tissue nodes, we use the same rule-based algorithm to build lesion edges. Different from the knowledge utilized in constructing the Tissue Graph, here we follow the assumption of Zhang et al. (2020): le-

---

Case (I) for Brain CT report generation
Ground Truth Sample
Ground Label
Predicted
Gencrated Reports
ZE EE Seaiiitttaae JE Ha X2 OF ME EL850285 X2M0 R0A SH Bo (Messiile tigh doaiy with low. deauity cEETO asound knt freze laks Damadtee dowey oaer Nikaeee gengla shate mate
Timee Lake
Ledan Labi
Aoseling WGAM
PEL Chigh deastty
2: 11 stateatt RUU SRIES RE 218861
50.3 (conenaned)
5p B 13 IE. (Masllke dendyy with adanor tran
MMavil ligh texily with for lew-
olama sasondorra and shalowong of the sakam ciriin Penn-lkk los donaty sea Is the Hissed besil gaogle, kiti vesticle, and coctress sensonde, and Do lowal
sempona loto. with corpecaiion
deeete are are son the
418 (ralii aundaan
while rette
dedecissed Hastellyy))
The oiffee dighty dgin
& B.... B. REEEIEE saft 10
sitee.)
SW, 19 PHASRBEE SORMAN Ve WI B. P4t ES 559522 1886
Dilaveed liah danny was lon dexiity the let vestide compecued sasoned. The millne souctere slibtty raas- the lnceal
SERR akend ventidez 8.01
sdere. sreand the tad poota
Npe Waigh demabyy 2.58
Pooesprraadt 4.33
Miaarr haod (prgle. white
is darrecs likeralls)
Case (II) for Brain CT report generation
Ground Truth Sample
Ground Truth Label
Predicted Label
Generated Reparts
Those
Lasion Eabel
10 15 9358
13++ (peinal
W desiity)
2E LL R8B BEMMER (Paacly
with low- deadty whera sosel the lelt poinal las Maliple partly los donciis on parially clar bondon bizzzra buxd pengia panstal yartices and connur actionad The mater
Bardio
lateal versido denensei bitmndd
4HG (ridile
11 (ken denutty
B05 FR S00 55... IESSEE (Puddy ligh Jandy was Invodenats ciene in the lell purical lobe. with cxppeciins abaoeed beis
baad-ãkz
METTx thaad
JEB (dteaa)))
shallowing
denety amond
koc mnllne strature sighlyy right diifed speckied, pebday les duedly sees the hadd pmatta und hiteerrl taarnl verrsdo, white desoessd biloorily)
WeRl 10 25 B M 1 22.
panglapa 0.31
cibiett 688 26. 9. 1. 23 12
S5tr (Pancly ligh lesdy
pareet the left toal jandii the white
Janiy
vonilee
AG Ki 0.99
axied. Spuckhal low damaty
master of the kntol werille is deceosed bilrenlly)
Figure 7: More cases of reports generated by the Baseline, WGAM, and our proposed model. English translations of the Chinese reports are given for better understanding.
of one global node, resulting in the edge matrix E/ IR. (N)
Number
Lesion
13
In total, our Lesion Graph includes 652 lesion dependency pairs, and more details are shown in Table 6.
Tissue
19
<tissu,, 159
<lesion, 652
Statistics of our Lesion Graph, contain- ing the number of lesion entities, associated tissues, <lissue, pairs, and <lesion,, pairs.
Table 6:
ID
Lesion
1
[ Eill |9C (high density)
2
(1: [9E (low density)
3
<2 (compressed)
4
* (Widenn)
A.3 More Cases of Brain CT Report Generation
5
J Ai (increase)
6
$ ! (thicken)
7
7 (edemaa)
More cases of Brain CT report generation are pre- sented in Figure 7. The correct tissues, lesions, and error-predicted pathological elements are marked in green, purple and red, respectively.
8
ast (decrease)
9
E (density shadow)
10
t ft (shift)
11
T S (narrow)
12
2: (left shift)
A.4 Details of Graph Features Initialization
13
3 (swell)
To map the graph features into pathology spatial regions, we follow the feature initializa- tion method in Zhang et al. (2020) with some modifications. Given the spatial visual feature Vg E IR (N+H)xdd (N = 24, H 196, dg = 512), where N. H.d, denote the number of scans, grids,
Table 5: Components of lesion terms in our Lesion Graph, which are summarized by the knowledge of physicians and the word frequency. English translations of the Chinese words are given for better understanding,

---

and channels, we duplicate it twice for initializ- ing tissue and lesion graph features separately. We mainly take the tissue graph feature initialization for example.
First, we transform the channel dimension of V into tissue number N₁ as:
Vi91 Relu(V,W7, + bti).
(9)
Vig2 = Relu(Vig1 W + biz);
(10)
Vig3 = Relu(Vigz" Wis + bta); (11)
where Relu(.) denotes the ReLU activation func- tion. W,, E (NN=H)x512 W/, E I2N+H))128
Wts E (N are learnable weights. by beg bts are learnable biases. We use the transformation result Vi03 C (NN+H))x to represent the graph information.
Then, bind the graph information Viga with specific visual regions, we perform the multiplica- tion of matrices Vg and Vig3 along the dimension of spatial grids, resulting in Vig E NxHM, In this way, each row of Vig can depict one specific tissue region, and we represent as one tissue node feature. Afterward, we conduct average pooling on the dimension of node number N₁, to obtain the global node feature Vigiobal N for the tissue graph, which contains general node repre- sentations. Finally, we concat Vig and Viglobat and reshape as the initialized tissue graph feature T E IR IN)+H))
As the same process, the lesion graph feature is initialized as L E (NN+)))) where N₁ is the number of lesion nodes.


Reviews:
[{"review_id": "X3xt3LUhWa", "reviewer": "Reviewer_nJDR", "summary": "", "questions": "1 Is there any result on the sensitivity of the lambda_1, lambda_2, lambda_3, training-validation loss / metric curve for equation (8) since these hyperparameters are vital for the model and training?\n\n2 Is the data available or how does the reader reproduce the results? Is it possible to apply the methodology to more general audience of NLP community? Maybe visual-language, vision grounding, visual captioning related tasks can be tried to validate the effectiveness of proposed method.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Is there any result on the sensitivity of the $\\lambda_1$, $\\lambda_2$, $\\lambda_3$, training-validation loss / metric curve for equation (8) since these hyperparameters are vital for the model and training?**\n\n**A1.** Thanks for your valuable comments. As recommended, we have conducted a thorough analysis to evaluate how different hyperparameters affect our results. Our approach involved conducting several experiments where we fixed two hyperparameters and varied the third within a specific range.\nFor the given optimal parameters ($\\lambda_1$=0.3,$\\lambda_2$=0.001,$\\lambda_3$=0.2), we first fix the value of $\\lambda_2$ and $\\lambda_3$, and tune $\\lambda_1$ in [0.02,0.03,0.2,0.3,0.5]. In the same manner, $\\lambda_3$ is tuned in [0.02,0.03,0.2,0.3,0.5] with the fixed optimal $\\lambda_1$ and $\\lambda_2$. Since ICA loss has a greater order of magnitude compared to other losses, we tune the $\\lambda_2$ in [0.0001,0.001,0.01,0.1] with the other parameters fixed. The results are presented as follows. \n\n| $\\lambda_1$ | 0.02 |0.03 |0.2 |0.3 |0.5 |\n|--------|--------|--------|--------|--------|--------|\n| Bleu-4 |14.1 | 14.0 |15.1 |**15.5** |14.2 |\n| CIDEr | 18.2 | 19.0 |19.8 |**19.9** |17.4 |\n\n| $\\lambda_2$ | 0.0001 |0.001 |0.01 |0.1 |\n|--------|--------|--------|--------|--------|\n| Bleu-4 |14.2 | **15.5** |13.8 |12.9|\n| CIDEr | 17.8 | **19.9** |18.5 |18.0|\n\n| $\\lambda_3$ | 0.02 |0.03 |0.2 |0.3 |0.5 |\n|--------|--------|--------|--------|--------|--------|\n| Bleu-4 |14.5 | 14.7 |**15.5** |14.6 |14.2 |\n| CIDEr | 18.2 | 18.5 |**19.9** |18.9 |18.6 |\n\nWe employed the most representative Bleu-4 and CIDEr metrics for performance evaluation. It can be observed that too low or too high values of $\\lambda_1$,$\\lambda_2$,$\\lambda_3$ decrease the performance. The reason could be that the inadequate pathological graph learning is caused by too low $\\lambda_1$,$\\lambda_2$, and $\\lambda_3$ respectively, which degrades the feature representation. Too high $\\lambda_1$,$\\lambda_2$, and $\\lambda_3$ may weaken the training of report generation task-related model parameters.\nWe will include these sensitivity analysis experiments and discussions in our final version.\n\n**Q2. Is the data available or how does the reader reproduce the results? Is it possible to apply the methodology to more general audience of NLP community?**\n\n**A2.** Thanks for your constructive suggestions. We appreciate your interest in the reproducibility of our results and the potential applicability of our methodology to a broader range of NLP tasks.\n\n(1) Regarding the availability of the dataset, it can be accessed by contacting the authors of paper [1]. For researchers working in related fields, such as Chest X-ray report generation, we recommend constructing a similar graph using our method and applying it to their specific tasks. To further facilitate reproducibility and encourage adoption, we will also open source our code upon publication.\n(2) As for the potential of our method to be applied in other NLP domains, we believe that our core concept of utilizing graph nodes and textual words to enhance feature representations can indeed be beneficial to a wider audience, provided that area-specific graph knowledge is available. To validate this assumption, we have conducted new experiments on the Chest X-ray report generation dataset IU-Xray [2]. For time efficiency, we choose the encoder-decoder model with hierarchical attention as Baseline (consistent with that in our article), and add all the proposed modules as Ours. In the absence of guidance from radiologists, we followed the approach outlined in [3] to select graph nodes, and we allowed the model to automatically learn the edges. The results are presented in the table below:\n\n|  | B1 | B2 | B3 | B4 | M | RG | C |\n|--------|--------|--------|--------|--------|--------|--------|--------|\n| Baseline | 39.5 | 27.3 | 20.0 | 14.6 | 18.4 | 35.6 | 40.5 |\n| Ours | **44.2** | **30.6** | **21.8** | **15.3** | **20.4** | **35.7** | **45.6** |\n\nIt is worth noting that the edges in this experiment were constructed without the input of chest experts, which might have introduced inaccuracies in the pathological semantics due to incorrectly connected edges. This could explain the inferior performance compared to mainstream Chest report generation models. Nonetheless, our model outperforms the baseline on the IU-Xray dataset, highlighting the effectiveness of our knowledge injection and feature alignment mechanisms. \n\nIn the future, we will explore the extensions in more visual-language tasks and contribute to the NLP field.\n\n[1] Yang S, Ji J, Zhang X, et al. Weakly guided hierarchical encoder-decoder network for brain ct report generation[C]//2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2021: 568-573.\n\n[2] Demner-Fushman D, Kohli M D, Rosenman M B, et al. Preparing a collection of radiology examinations for distribution and retrieval[J]. Journal of the American Medical Informatics Association, 2016, 23(2): 304-310.\n\n[3] Zhang Y, Wang X, Xu Z, et al. When radiology report generation meets knowledge graph[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 12910-12917.\n\n**Q3. Ethical Concerns: Yes**\n\n**A3.** We appreciate the concern regarding ethical considerations, and we want to emphasize that this has been a priority throughout our research. The medical images in the BCT-CHR dataset have undergone a thorough process of de-identification to protect participants' privacy. Additionally, we have obtained the required data permissions, ensuring our work meets ethical standards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 The paper is well-written with vivid figures and examples. Readers can get the ideas, what the paper is doing, results, etc., easily.\n\n2 The graph assisted multimodal training is interesting. From the report / paragraph of text, the knowledge graph can be automatically constructed. Then, intra-attribute and inter-attribute sub-graphs can be obtained. Intra-attribute classification, inter-correlation alignment can be enforced leveraging the knowledge graphs. As far as I know, this is the first work leveraging knowledge graph to improve the current visual-language multimodal learning.", "reasons_to_reject": "1 The paper is only validated on one dataset, BCT-CHR, which is a medical report related dataset and cannot be publicly available. This strongly limits the impact and audience of the paper. It would be better if the paper could validate on more datasets and tasks.", "questions_to_authors": "1 Is there any result on the sensitivity of the lambda_1, lambda_2, lambda_3, training-validation loss / metric curve for equation (8) since these hyperparameters are vital for the model and training?\n\n2 Is the data available or how does the reader reproduce the results? Is it possible to apply the methodology to more general audience of NLP community? Maybe visual-language, vision grounding, visual captioning related tasks can be tried to validate the effectiveness of proposed method.", "paper_topic_and_main_contributions": "This paper proposes a pathological graph-driven cross-modal alignment model for brain CT report generation. First, it constructs a pathological graph that incorporates detailed medical knowledge, allowing for the injection of this knowledge into dedicated node features through graph embedding and updating, thereby achieving fine-grained visual representations. Second, it aligns the learned node features with the corresponding word embeddings by cross-modal contrastive learning, to further boost report generation. Extensive experiments demonstrate that the model achieves superior performance in generating clinically accurate reports.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zbjLR6o5AW", "reviewer": "Reviewer_gRk7", "summary": "", "questions": "A. In Section 3.2.3, it is stated that the IAC module dynamically updates node features, for both the tissue nodes, and the lesion nodes. It is also stated that learning inter-attribute relations between tissues and lesions is also essential. This seems to imply that the weights within the tissue subgraph and lesion subgraphs, are also updated/learnt. However, Section 1 stats that the tissue and lesion (sub)graphs are fixed, in contrast to the dynamic tissue-lesion graph. The fixed nature of the tissue and lesion (sub)graphs might thus be clarified - does it mean that the connections are preset, i.e. no new connections can be learnt?\n\nB. Related to the above, it might be clarified as to whether new connections can be learnt for inter-graph weights (i.e. new non-zero values in the sparse attribute alignment matrix)\n\nC. In Section 3.3, a single L_IAC loss is given in Equation 8, while Equation 4 implies two losses (one for each subgraph). This might be clarified. Moreover, if there are indeed two IAC losses, are their weights constrained to be identical?\n\nD. In Section 4.1, it is stated that the reports were in Chinese. It might be clarified as to whether the ensuing training of the model involved the Chinese words/tokens, or English tokens after translation.\n\nE. In Section 4.2, it is stated that 24 keywords were selected to evaluate the CE metrics. It might be discussed as to how these keywords were chosen.\n\nF. In Section 4.3, the data used to tune the loss coefficients and other parameters might be explicitly stated - was it the validation set?\n\nG. In Section 4.4.1, overrepresentation of some high-frequency terms is stated to be a factor possibly affecting CIDEr. It might be briefly discussed as to whether sampling regimes had been considered, to target such token distribution issues.\n\nH. In Section 4.4.2, for the ablation study, it is not immediately clear as to whether any of the configurations represent the proposed model with the initial graph (as designed from prior domain knowledge), but before learning. If the tissue-lesion graph has initial values, it seems that this \"original graph\" might be evaluated too, and the performance reported. This would establish the contribution of the graph training/updating process.\n\nI. For the CE AUROCs presented in Figure 3, it might be strongly considered to also compare against standard image-based deep learning classification models (e.g. ResNet as already used in this study, etc.) for direct label classification performance.\n\nJ. In Section 4.4.3, it is stated that salient regions grounded by the IAC module(s) are generally consistent with human-labeled boxes. The availability of the human annotations appears to admit quantitative analysis, as to performance.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Is there a contradiction between Section 3.2.3 and Section 1 regarding the weights within the tissue and lesion subgraphs are dynamically updated or fixed? The fixed nature of the tissue and lesion (sub)graphs might thus be clarified - does it mean that the connections are preset?**\n\n**A1.** Thanks for your constructive comments. The edges (weights) within the tissue and lesion subgraphs are indeed predetermined based on expert medical knowledge, rendering them fixed and unalterable during training. This fixed nature ensures the injection of precise pathological relationships into the graph convolution network, leading to dynamic updates of node features. \nTo clarify, the \"dynamic update\" mentioned in Section 3.2.3 refers only to the tuning of graph node features and does not involve any new connections within the tissue and lesion subgraphs. We will include these analyses to enhance clarity in the final version.\n\n\n**Q2. Related to the above, it might be clarified as to whether new connections can be learnt for inter-graph weights.**\n\n**A2.** Thank you for your valuable feedback. Indeed, new connections can be learned in the tissue-lesion graph. This is achieved by automatically extracting tissue-lesion relationships from the report in each sample, which contributes to generating a tissue-lesion alignment matrix to store the value of edges for each sample. This matrix plays a crucial role in regulating the learning of node features, with non-zero values precisely denoting the existence of tissue-associated lesions in the report.\n\n**Q3. In Section 3.3, a single $L_{IAC}$ loss is given in Equation 8, while Equation 4 implies two losses (one for each subgraph). This might be clarified. Moreover, if there are indeed two IAC losses, are their weights constrained to be identical?**\n\n**A3.** Thanks for your valuable observation. The $L_{IAC}$ loss indeed encompasses two distinct components: one related to tissue attribute and another to lesion attribute. These two attributes hold equal significance in the context of report generation, and as such, we've amalgamated their losses with equal weights in a 1:1 ratio to construct the ultimate $L_{IAC}$ loss. The tuning of the $L_{IAC}$ loss's importance is governed by the parameter $\\lambda_1$, as denoted in Equation 8. We will clarify these concerns after Equation 4 in our final version.\n\n**Q4. In Section 4.1, it is stated that the reports were in Chinese. It might be clarified as to whether the ensuing training of the model involved the Chinese words/tokens, or English tokens after translation.**\n\n**A4.** Thanks for pointing these out. The reports in the BCT-CHR dataset are in Chinese, so our model training uses Chinese tokens. English translations in the paper are only for readers' understanding and are not used in training. We will clarify this in the final version.\n\n\n**Q5. In Section 4.2, it is stated that 24 keywords were selected to evaluate the CE metrics. It might be discussed as to how these keywords were chosen.**\n\n**A5.** Thanks for your constructive suggestion. The keywords to evaluate the CE metrics are first filtered as high-frequency tokens in the training corpus and further chosen by experienced radiologists, which represent the most common but critical cranial words and hold great significance within the context of cranial pathology assessment.\n\n**Q6. In Section 4.3, the data used to tune the loss coefficients and other parameters might be explicitly stated - was it the validation set?** \n\n**A6.** Thanks for pointing out this issue. It is true that the hyperparameters in our model are tuned via the validation set, and we will add this description in section 4.3.\n\n\n**Q7. In Section 4.4.1, overrepresentation of some high-frequency terms is stated to be a factor possibly affecting CIDEr. It might be briefly discussed as to whether sampling regimes had been considered, to target such token distribution issues.**\n\n**A7.** Thanks for your valuable comments. For fair model comparisons, we followed the widely used random sampling regime for training. Our future research will explore introducing curriculum learning in data sampling, and an easy-to-hard training paradigm will be developed to smooth the representation of pathological words, which has the potential to tackle this token distribution issue.\n\n\n**Q8. In Section 4.4.2, for the ablation study, the contribution of the graph training/updating process might be evaluated too.**\n\n**A8.** Thanks for your suggestions. Note that the graph node features are only updated by IAC and ICA modules, and the case without training the tissue-lesion graph by ICA module is given in Table 2(e). As suggested, to further demonstrate the benefits brought by graph training, we have added two new experiments in Table 2: (f) the case when the initial graph (tissue and lesion graphs) and the tissue-lesion graph are given but not trained; (g) the case when only tissue-lesion graph is trained. The results are as follows:\n\n| Methods | tissue | lesion | $L_{IAC}$ | $L_{ICA}$ | $L_{CL}$ | B1 | B2 | B3 | B4 | M | RG | C |\n|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| Baseline | ✖ | ✖ | ✖ | ✖ | ✖ | 40.7 | 27.5 | 19.2 | 13.8 | 26.4 | 35 3 | 16.3 |\n| (a) | ✖ | ✖ | ✖ | ✖ | ✔ | 41.8 | 28.4 | 19.9 | 14.2 | 27.1 | 35.7 | 18.7 |\n| (b) | ✔ | ✖ | ✔ | ✖ | ✖ | 42.5 | 28.9 | 20.3 | 14.6 | 27.3 | 35.7 | 18.0 |\n| (c) | ✔ | ✖ | ✔ | ✖ | ✔ | 43.5 | 29.6 | 20.7 | 14.9 | 27.8 | 35.9 | 19.7 |\n| (d) | ✔ | ✔ | ✔ | ✖ | ✖ | 43.9 | 29.7 | 20.8 | 14.8 | 28.3 | 36.0 | **21.3** |\n| (e) | ✔ | ✔ | ✔ | ✖ | ✔ | 44.1 | 29.8 | 20.8 | 14.8 | 27.8 | 35.9 | 19.3 |\n| (f) | ✔ | ✔ | ✖ | ✖ | ✔ | 41.5 | 27.8 | 19.3 | 13.6 | 26.3 | 35.2 | 16.7 |\n| (g) | ✔ | ✔ | ✖ | ✔ | ✔ | 42.3 | 28.7 | 20.2 | 14.5 | 27.3 | 35.5 | 18.6 |\n| Ours| ✔ | ✔ | ✔ | ✔ | ✔ | **45.0** | **30.8** | **21.6** | **15.5** | **28.7** | **36.5** | 19.9 |\n\nIt can be seen that without the graph training, (f) performs even worse than coarse-grained CLIP loss in (a). In contrast, with the IAC, ICA, and both the two modules for graph training, (e), (g), and Ours respectively show improved results, affirming the significance of graph training.\n\n\n**Q9. For Figure 3, it might be strongly considered to also compare against standard image-based deep learning classification models (e.g. ResNet) for direct label classification performance.**\n\n**A9.** Thanks for your constructive comments. As recommended, to further evaluate the classification performance, we compare our model with the standard ResNet model for direct classification. The AUC scores are listed as follows, which will be illustrated in ROC curves and added to Figure 3.\n\n|  | brainstem | midline structure | fourth ventricle  | sulcus cerebri | temporal lobe | thalamus | low density | edema | widen | increase |\n|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| ResNet | 0.54 | **0.63** | 0.61 | 0.59 | 0.59 | 0.56 | 0.53 | **0.60** | 0.64 | 0.50 |\n| Ours | **0.65** | 0.60 | **0.64** | **0.63** | **0.62** | **0.57** | **0.55** | 0.55 | **0.69** | **0.56** |\n\nIt is noticeable that our model outperforms the classic ResNet model in pathology classification, which not only indicates the effectiveness of our graph learning and cross-modal alignment methods but also hints report generation and classification can boost each other.\n\n**Q10. In Section 4.4.3, it is stated that salient regions grounded by the IAC module(s) are generally consistent with human-labeled boxes. The availability of the human annotations appears to admit quantitative analysis, as to performance.**\n\n**A10.** We appreciate your constructive suggestion and understand the importance of a quantitative evaluation to provide a more comprehensive understanding of the model's performance. However, the BCT-CHR dataset does not yet contain the salient objective data, and manually collecting this data would be an incredibly time-consuming and costly endeavor. Nevertheless, our qualitative observations, as presented in the manuscript, indicate that the salient regions grounded by the IAC module(s) are generally consistent with expert-labeled boxes. While we acknowledge the limitations of the qualitative evaluation, we believe it still provides valuable insights into the model's performance in grounding salient regions. In the future, we plan to collaborate with domain experts to gather more human-labeled boxes and pave the way for more comprehensive model evaluation.\n\n**Q11. In Line 146, \"chesty terminologies\" might be rephrased. In Line 289, \"with human refines\" might be rephrased.**\n\n**A11.** Thank you for figuring these out. \"chesty terminologies\" is revised as \"chest medical terminologies\", and \"with human refines\" is revised as \"with artificial refining\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - Demonstrates potential of incorporating domain knowledge as a specialized graph, for medical report generation from images\n - Extensive comparison against many state-of-the-art models\n", "reasons_to_reject": " - Contribution of further Pathological Graph updating not very clear\n - Minimal comparisons for element-based classification\n", "questions_to_authors": "A. In Section 3.2.3, it is stated that the IAC module dynamically updates node features, for both the tissue nodes, and the lesion nodes. It is also stated that learning inter-attribute relations between tissues and lesions is also essential. This seems to imply that the weights within the tissue subgraph and lesion subgraphs, are also updated/learnt. However, Section 1 stats that the tissue and lesion (sub)graphs are fixed, in contrast to the dynamic tissue-lesion graph. The fixed nature of the tissue and lesion (sub)graphs might thus be clarified - does it mean that the connections are preset, i.e. no new connections can be learnt?\n\nB. Related to the above, it might be clarified as to whether new connections can be learnt for inter-graph weights (i.e. new non-zero values in the sparse attribute alignment matrix)\n\nC. In Section 3.3, a single L_IAC loss is given in Equation 8, while Equation 4 implies two losses (one for each subgraph). This might be clarified. Moreover, if there are indeed two IAC losses, are their weights constrained to be identical?\n\nD. In Section 4.1, it is stated that the reports were in Chinese. It might be clarified as to whether the ensuing training of the model involved the Chinese words/tokens, or English tokens after translation.\n\nE. In Section 4.2, it is stated that 24 keywords were selected to evaluate the CE metrics. It might be discussed as to how these keywords were chosen.\n\nF. In Section 4.3, the data used to tune the loss coefficients and other parameters might be explicitly stated - was it the validation set?\n\nG. In Section 4.4.1, overrepresentation of some high-frequency terms is stated to be a factor possibly affecting CIDEr. It might be briefly discussed as to whether sampling regimes had been considered, to target such token distribution issues.\n\nH. In Section 4.4.2, for the ablation study, it is not immediately clear as to whether any of the configurations represent the proposed model with the initial graph (as designed from prior domain knowledge), but before learning. If the tissue-lesion graph has initial values, it seems that this \"original graph\" might be evaluated too, and the performance reported. This would establish the contribution of the graph training/updating process.\n\nI. For the CE AUROCs presented in Figure 3, it might be strongly considered to also compare against standard image-based deep learning classification models (e.g. ResNet as already used in this study, etc.) for direct label classification performance.\n\nJ. In Section 4.4.3, it is stated that salient regions grounded by the IAC module(s) are generally consistent with human-labeled boxes. The availability of the human annotations appears to admit quantitative analysis, as to performance.\n", "paper_topic_and_main_contributions": "This paper proposes a Pathological Graph-driven Cross-modal Alignment (PCGA) model, for Brain CT report generation. In particular, a graph model is used to learn fine-grained visual cues, and algin the cues with text. The structure of the graph model comprises a tissue subgraph and a lesion subgraph, with nodes within and between each subgraph connected according to prior domain knowledge. The PCGA is then further trained with contrastive learning, and compared against multiple general and medical image captioning/report models, on the BCT-CHR brain CT scan dataset.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "In Line 146, \"chesty terminologies\" might be rephrased.\nIn Line 289, \"with human refines\" might be rephrased.\n", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "9pq9aYdRKR", "reviewer": "Reviewer_kj9V", "summary": "", "questions": "A) Figure 1, all  graph edges between tissue and lesion are 1.0. Why is it so? Also, why all lesion edges are 0.9? Is it true that they are always the case? It is unclear how you constructed the edge probability. Parts of the Appendix may be needed in the main text for explaining.\nB) Figure 2, the ground truth report should be used for calculating the generative loss. It is weird that it has no connection between the (a) and ground truth repot. \nC) Figure 2, the textual embedding on the right is claimed to be shared with the textual decoding at the bottom. There is no description about how it is shared. The right embedding layer should be encoding layer.  How can it be  shared with the decoding layer?\nD) Figure 3, what does the numbers (e.g.: 0.65) next to the line name means?\nE) Based on the models structure (Figure 2) and the description, it is unclear to me which part is outputting the 'predicted labels' in  Figure 4.\nF) The loss coefficients are set to 0.3, 0.001 and 0.2. Why is it so? Why as low as 0.001?\nG) Table 1,  why there is a line set between up-down and MRMA? Are they belonging to two different groups of models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. There is only one comparison method WGAM (from 2 years ago) relating to brain CT report generation. There are possibly unlisted related works on this task. It is also unclear whether the compared models other than WGAM are from medical domain or not. Considering the fact that brain CT has its unique challenges comparing to other medical images and other natural images, the experiment result may not be as valid as the paper claims.**\n\n**A1.** We appreciate your concern regarding the comparison with only one brain CT report generation method, WGAM. We acknowledge that brain CT report generation is an underexplored area, and WGAM is currently the only available method for this task. Our paper provides comprehensive coverage of this emerging field, and our study contributes by being at the forefront of this evolving domain. \n\nTo ensure the effectiveness of our method, we have incorporated comprehensive comparisons with SOTA Chest X-Ray report generation (medical domain) and image captioning (non-medical domain) models in addition to WGAM. Please refer to L470-480 for more details. All these comparison methods share similar visual-language generation tasks to brain CT report generation and have been commonly used for comparison purposes in the report generation area.\n\nIn experiments, all the comparison models were reproduced and trained on the brain CT dataset with consistency to ours to overcome the unique challenges of this task. Specifically, the comparison models underwent the same visual feature extracting and fine-tuning process as our baseline. We also standardized the training epochs at 60 and selected the best outcomes from 5 runs, maintaining consistency and ensuring fairness and reliability in our comparative analysis. \n\n\n\n\n**Q2. Figure 1, all graph edges between tissue and lesion are 1.0. Why is it so? Also, why all lesion edges are 0.9? Is it true that they are always the case? It is unclear how you constructed the edge probability. Parts of the Appendix may be needed in the main text for explaining.**\n\n**A2.** Thanks for your constructive suggestion. (1) The edges within the tissue-lesion graph are binary (0 or 1), signifying whether a specific tissue corresponds to a particular lesion in the report. In Figure 1, we have displayed only the edges with a value of 1 for conciseness. (2) Notably, lesion edge values range from 0 to 1, with the instances of 0.9 being specific occurrences. To enhance clarity and avoid misinterpretation, we will adjust the cases illustrated in Figure 1 accordingly. (3) The construction of edge probabilities involves a comprehensive approach. Through knowledge formatting and dependency parsing, relation pairs are identified (e.g. 133 <tissue, tissue> pairs for the tissue graph, and 652 <lesion, lesion> pairs for the lesion graph). These pairs guide the addition of weight to update bidirectional edges for corresponding tissues or lesions until a predetermined upper limit is reached. The detailed methodology can be found in Appendix A.1, A.2. For clarity, more information about building the edge probabilities will be added in the main text.\n\n**Q3. Figure 2, the ground truth report should be used for calculating the generative loss. It is weird that it has no connection between the (a) and ground truth report.**\n\n**A3.** Thanks for the comment. We apologize for any confusion caused by the figure. In Figure 2(a), the generated report is indeed compared to the ground truth report by default, and the generative loss is calculated accordingly. To make this clearer, we will modify Figure 2 in the final version of our manuscript to better illustrate the connection between the generated report and the ground truth report.\n\n**Q4. Figure 2, the textual embedding on the right is claimed to be shared with the textual decoding at the bottom. There is no description about how it is shared. The right embedding layer should be encoding layer. How can it be shared with the decoding layer?**\n\n**A4.** Thanks for your valuable comment. Actually, the textual decoder in Figure 2 contains two parts: a \"textual embedding\" layer and a language generation model. The \"textual embedding\" layer embeds words in decoder and is shared with the \"right embedding layer\". In our initial manuscript, we did not divide the textual decoder to keep it concise, which may have caused confusion. We will refine Figure 2 in the final version to provide a clearer visualization of the weight-sharing nature and structure of the textual decoder.\n\n**Q5. Figure 3, what does the numbers (e.g.: 0.65) next to the line name means?**\n\n**A5.** The numbers displayed next to the line names in Figure 3 correspond to the AUC scores of classification. To ensure clarity, we have included explanatory notes in Figure 3.\n\n**Q6. Based on the models structure (Figure 2) and the description, it is unclear to me which part is outputting the 'predicted labels' in Figure 4.** \n\n**A6.** Thanks for pointing out this issue. The 'predicted labels' are outputted by the Intra-Attribute Classification Module in Figure 2, to assess the model's discrimination of pathology. We will revise this description to make this concept clear.\n\n**Q7. The loss coefficients are set to 0.3, 0.001 and 0.2. Why is it so? Why as low as 0.001?**\n\n**A7.** Thanks for raising this question. (1) The choice of loss coefficients is informed by a meticulous tuning process. To transparently elucidate our reasoning, we provide the parameter sensitivity experiments. For the given optimal coefficients ($\\lambda_1$=0.3,$\\lambda_2$=0.001,$\\lambda_3$=0.2), we first fix the value of $\\lambda_2$ and $\\lambda_3$, and tune $\\lambda_1$ in [0.02,0.03,0.2,0.3,0.5]. Similarly, $\\lambda_3$ undergo tuning within the range ([0.02, 0.03, 0.2, 0.3, 0.5]) while keeping optimal $\\lambda_1$ and $\\lambda_2$ constants. Given the larger magnitude of the ICA loss compared to other losses, $\\lambda_2$ was tuned across values ([0.0001, 0.001, 0.01, 0.1]) with the other parameters fixed. The results are listed below, and the metric curves will be added to our final paper.\n\n| $\\lambda_1$ | 0.02 |0.03 |0.2 |0.3 |0.5 |\n|--------|--------|--------|--------|--------|--------|\n| Bleu-4 |14.1 | 14.0 |15.1 |**15.5** |14.2 |\n| CIDEr | 18.2 | 19.0 |19.8 |**19.9** |17.4 |\n\n| $\\lambda_2$ | 0.0001 |0.001 |0.01 |0.1 |\n|--------|--------|--------|--------|--------|\n| Bleu-4 |14.2 | **15.5** |13.8 |12.9|\n| CIDEr | 17.8 | **19.9** |18.5 |18.0|\n\n| $\\lambda_3$ | 0.02 |0.03 |0.2 |0.3 |0.5 |\n|--------|--------|--------|--------|--------|--------|\n| Bleu-4 |14.5 | 14.7 |**15.5** |14.6 |14.2 |\n| CIDEr | 18.2 | 18.5 |**19.9** |18.9 |18.6 |\n\nWe employed the most representative Bleu-4 and CIDEr metrics for performance evaluation. We can observe that too low or too high values of $\\lambda_1$, $\\lambda_2$, $\\lambda_3$ decrease the performance. The reason may be because the too low score of $\\lambda_1$,$\\lambda_2$, and $\\lambda_3$ respectively lead to inadequate pathological semantic learning, inter-attribute matching, and visual-textual alignment, which degrades the feature representation. Too high $\\lambda_1$,$\\lambda_2$, and $\\lambda_3$ may weaken the training of report generation-related parameters and cause sub-optimal results.\n(2) As mentioned above, a low value of $\\lambda_2$ is used to balance the varying orders of magnitude of model losses. This strategic adjustment ensures that our training procedure converges effectively and yields optimal outcomes, striking a delicate equilibrium.\n\n\n**Q8. Table 1, why there is a line set between up-down and MRMA? Are they belonging to two different groups of models?**\n\n**A8.** We appreciate your inquiry. You've rightly observed that \"Up-Down\" and \"MRMA\" originally pertained to distinct tasks. In light of the current limited presence of studies specifically targeting Brain CT report generation - exemplified by the single instance of WGAM - we extended our analysis to encompass a wider scope. We included an array of models from closely related fields such as Chest report generation and image captioning to facilitate a comprehensive assessment, and we separated different groups of models with lines in Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed Pathological Graph is novel as it decouples cranial tissues  and lesions. The experiments well demonstrates the benefits of using such graph. Besides that, it also helps the model interpretability of models. ", "reasons_to_reject": "There is only one comparison method relating to brain CT report generation and this method WGAM is from 2 years ago. There are possibly unlisted related works on this task. In addition, it is uncleared that, models other than WGAM are from medical domain or not.  Considering the fact that brain CT is has its unique challenges comparing to other medical images and other natural images, the experiment result may not be as valid as the paper claims.", "questions_to_authors": "A) Figure 1, all  graph edges between tissue and lesion are 1.0. Why is it so? Also, why all lesion edges are 0.9? Is it true that they are always the case? It is unclear how you constructed the edge probability. Parts of the Appendix may be needed in the main text for explaining.\nB) Figure 2, the ground truth report should be used for calculating the generative loss. It is weird that it has no connection between the (a) and ground truth repot. \nC) Figure 2, the textual embedding on the right is claimed to be shared with the textual decoding at the bottom. There is no description about how it is shared. The right embedding layer should be encoding layer.  How can it be  shared with the decoding layer?\nD) Figure 3, what does the numbers (e.g.: 0.65) next to the line name means?\nE) Based on the models structure (Figure 2) and the description, it is unclear to me which part is outputting the 'predicted labels' in  Figure 4.\nF) The loss coefficients are set to 0.3, 0.001 and 0.2. Why is it so? Why as low as 0.001?\nG) Table 1,  why there is a line set between up-down and MRMA? Are they belonging to two different groups of models?", "paper_topic_and_main_contributions": "This paper proposes a novel Pathological Graph-driven Cross-modal Alignment (PGCA) model for accurate and robust Brain CT report generation. \n\nThe main contributions are as follows:\n1. The paper proposes a novel framework to seamlessly capture detailed domain knowledge from Pathological Graph, and explicitly align fine-grained visual and textual features of pathology.\n2. The paper, for the first time, introduce the idea of decoupling cranial tissues and lesions via Pathological Graph into the medical report generation area, which is capable of handling fine-grained alignment between long-text report and multiple scans. \n3. The paper comprehensively validate the proposed model on the BCT-CHR dataset. The experimental results indicate that the proposed method surpasses previous works in medical report generation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel Pathological Graph-driven Cross-modal Alignment (PCGA) model for the generation of brain CT reports. The authors first construct a pathological graph that encodes detailed medical knowledge, which is then integrated into node features via graph embedding and updating, resulting in fine-grained visual representations. The model aligns these learned node features with corresponding word embeddings through cross-modal contrastive learning to enhance report generation. Extensive experiments on the BCT-CHR dataset show the model's superior performance in generating clinically accurate reports.

Main Contributions:

The authors propose a new framework to capture detailed domain knowledge from a Pathological Graph and align fine-grained visual and textual features of pathology.
The paper introduces the concept of decoupling cranial tissues and lesions via a Pathological Graph in the medical report generation area, a first in the field. This approach can handle the fine-grained alignment between long-text reports and multiple scans.
The authors validate the proposed model comprehensively on the BCT-CHR dataset, demonstrating that their method outperforms previous works in medical report generation.
Reasons for Acceptance:

The paper is well-written, with clear figures and examples that make it easy for readers to understand the ideas, methods, and results.
The use of a graph-assisted multimodal training approach is novel and interesting. The knowledge graph, automatically constructed from the report/paragraph of text, allows for intra-attribute classification and inter-correlation alignment.
The novel pathological graph decouples cranial tissues and lesions, enhancing the model's interpretability while also yielding benefits as demonstrated by the experiments.
The paper demonstrates the potential of incorporating domain knowledge as a specialized graph for medical report generation from images.
The model was extensively compared against several state-of-the-art models, showing superior performance.
Decision: Accept-Main