VR-DiagNet: Medical Volumetric and Radiomic Diagnosis Networks with Interpretable Clinician-like Optimizing Visual Inspection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 01:38:55

Abstract: Interpretable and robust medical diagnoses are essential traits for practicing clinicians. Most computer-augmented diagnostic systems suffer from three major problems: non-interpretability, limited modality analysis, and narrow focus. Existing frameworks can either deal with multimodality to some extent but suffer from non-interpretability or partially interpretable but provide a limited modality and multifaceted capabilities. Our work aims to integrate all these aspects in one complete framework to fully utilize the full spectrum of information offered by multiple modalities and facets. We propose our solution via our novel architecture VR-DiagNet, consisting of a planner and a classifier, optimized iteratively and cohesively. VR-DiagNet simulates the perceptual process of clinicians via the use of volumetric imaging information integrated with radiomic features modality; at the same time, it recreates human thought processes via a customized Monte Carlo Tree Search (MCTS) which constructs a volume-tailored experience tree to identify slices of interest (SoIs) in our multi-slice perception space. We conducted extensive experiments across two diagnostic tasks comprising six public medical volumetric benchmark datasets. Our findings showcase superior performance, as evidenced by heightened accuracy and area under the curve (AUC) metrics, reduced computational overhead, and expedited convergence while conclusively illustrating the immense value of integrating volumetric and radiomic modalities for our current problem setup.

Main paper: 
 related work: 
 To offer a thorough overview, we explore specific technologies closely intertwined with this paper, including volumetric image diagnosis, medical radiomics analysis, and MTCS-based planning. Volumetric Image Diagnosis. Deep learning has emerged as the dominant paradigm in medical image analysis (MIA) tasks in recent years. Convolution-based applications for medical volumetric images have adopted either 3D [25, 35] or 2.5D [31, 35] variants. These approaches, necessitating dense encoding, entail heightened computational costs and diminished data efficiency. Moreover, they are susceptible to over-fitting in scenarios with limited sample sizes. Contrarily, graph-based methodologies like GCN-CAP [14] and GSDG [4] leverage sparse encoding, reorganizing volume slices, effectively curbing computational complexity. This reduction is achieved by pre-extracting slice embeddings utilizing a frozen pretrained Convolutional Neural Network (CNN) backbone during pre-processing. Although this approach significantly decreases input data density, it potentially suffers from domain gap issues introduced by out-of-domain pre-training. Jang et al. [11] propose a hybrid approach that merges Transformers with 2D and 3D CNNs, wherein the 3D CNN facilitates 3D representation learning while the 2D CNN utilizes pre-training weights from ImageNet [5] for 2D representation learning. However, the introduction of 3D models escalates the risk of over-fitting. Despite their different focuses, all the works above fail to utilize radiomic features, an under-utilized feature modality, and address how to model the visual inspection process of clinicians. Furthermore, techniques extending beyond raw imaging, such as quantitative analysis and feature extraction within the radiomics domain, need more emphasis in such endeavors despite their general efficacy in elucidating clinical insights.Medical Radiomics Analysis. Radiomics features, with scales and calculation processes distinct from those of the imaging modality, complement the slice image modality and show promise in identifying slice-or volume-based biomarkers of therapy response. Reproducibility of radiomic features remains a challenge, addressed by initiatives like the Image Biomarker Standardization Initiative (IBSI) [41] . Tamal et al. [26] combine radiomics and machine learning for fast and accurate COVID-19 diagnosis from Chest X-ray images. Despite its good performance, classical machine learning remains dominant due to limited medical image samples and sparse radiomic data. Tanaka et al. [27] introduce a deep learning-based radiomics approach for early head and neck tumor regression prediction, outperforming traditional radiomics and clinical factors. However, its feature fusion stage precedes the machine learning decision layer, posing challenges in establishing deep-level complex𝒳 !,# 𝐦 !,# $ 𝒳 !,% 𝐦 !,# % 𝒳 !,& 𝐦 !,# & Experience Tree 𝐦 !,# ' strategy ℐ ! Vision Encoder (NHS) ⨁ 𝐞 !,# $ 𝐳𝐩𝐞 !,# $ 𝐦 !,# $Vision Encoder (NHS) correlations between the two modalities. Further exploration in multimodal tasks is warranted to elucidate radiomics' underlying value. Ge et al. [8] demonstrate the effectiveness of extracting complementary radiomics features from diverse imaging modalities for identifying and differentiating kidney diseases. Vanguri et al. [29] integrate and analyze radiomic, pathological, and genomic features using machine learning algorithms in treatment response prediction. Deep Learning Radiomics (DLR) revolutionizes predictive modeling by extracting deep features. Zheng et al. [38] design a parallel model structure with ResNet-50 [9] base networks ingesting images of two modalities as dual-modal inputs. DLR extracts deep features from images while separating them from traditional radiomic features. Ning et al. [20] extract radiomics and deep features from MRI modalities to quantify global and local information using a kernel fusion-based SVM classifier for glioma grading. Our work aims to model the correlation between classical radiomic and deep features, enabling intermodality interaction and exploring inter-group interaction within radiomic features to adapt to deep feature distribution.EQUATIONMTCS-based Planning. The emulation of the visual inspection process of clinicians is crucial for improving the interpretability of the model and inspiring the planner to align volumes with varying numbers of slices during the slice identification phase. Our methodology dissects volumetric images slice by slice, a tailored strategy for handling volumetric data. Furthermore, we integrate concepts reminiscent of tree-based planning algorithms, such as MCTS, as observed in AlphaGo [23] , and EG-MCTS [10] , which addresses retrosynthetic planning for drug compounds. Based on the tasks at hand, we fill the gap in the works above. During the planning process, we suggest an additional data modality to help construct a visual inspection strategy similar to a clinician's. We explore a hierarchical priori design, consisting of volume-specific refined radiomic features constructed over static raw radiomic features and a CLS embedding within the Multi-Head Attention (MHA) mechanism [30] . The hierarchical design is anticipated to enhance diagnosis robustness owing to its multimodal and multigranular nature. 
method 3.1 problem statement: 
 For accurate diagnostic purposes and by capturing the joint features between volumetric and radiomic features, our proposed multimodal method, VR-DiagNet, introduces a novel approach to medical image processing by integrating momentum planning (see §3. contains a planner f 𝑃𝐿 and a 𝐶-class classifier f 𝐶𝐿 , optimized alternately over 𝑁 𝑅𝑜 rounds. An overview of our method is illustrated in Figure 2 .First, we introduce the planner, the first constituent in our method. To mimic human-like visual inspection akin to clinicians, we propose a computationally feasible planning methodology. In each round 𝑟 ∈ [1, . . . , 𝑁 𝑅𝑜 ], the planner constructs a volume-specific experience tree T (𝑟 ) X 𝑖 comprising 𝐿 layers (excluding the root node). This tree is built using voxels and radiomic features extracted from X 𝑖 and x 𝑖,𝑅𝑎𝑑 , enabling the selection of the most informative SoIs in a class-agnostic manner. This class-agnostic approach employs information metrics to determine the most task-relevant slices, eliminating the need for class labels and enabling generalization to any classification task. We formalize this process as a MDP characterized by (S, A 𝑠 , P 𝑎 , R):(1) S represents the state space. EQUATIONFor a strategy with only one slice, NII is measured using a normalized negative variant of Shannon Entropy [22] , 𝑁 𝑆𝐸 (𝑠) = log 𝐶 + 𝐶 𝑐=1 p 𝑠,𝑐 log p 𝑠,𝑐 + 𝜖 /log 𝐶, serving as a certainty metric. p 𝑠 ∈ [0, 1] 𝐶 is the conditional probability distribution of the last slice in strategy 𝑠. For other strategies, NII is calculated using normalized mutual information: 𝑁 𝑀𝐼 (𝑠) = (H 𝑃 -H )/ √ H 𝑃 H , where H and H 𝑃 are Shannon Entropy values for the strategy 𝑠 and its ancestor, respectively. This calculation also relies on conditional probability distributions. We employ MCTS to solve the MDP above iteratively. The planner starts from the refined radiomics node and consists of four stages in each MCTS iteration: (1) w A 𝑠 ← NII(A 𝑠 ) ⊲ Eq. 1 5:X 𝑖,𝑡 (len(𝑠 )+1) ← WeightedSample(A 𝑠 , weights = w A 𝑠 )6:𝑠 ← 𝑠 ∪ X 𝑖,𝑡 (len(𝑠 )+1) 7: end while 8: return 𝑅(𝑠) strategy 𝑠 using the Upper Confidence Bound (UCB) strategy [2] , until reaching a leaf node or the maximum depth. By leveraging slice informativeness informed by NII, the search direction of the planner aligns with the optimization direction of the classifier. Nonetheless, the extensive state space often necessitates the integration of MCTS with additional techniques to enhance efficiency and convergence [24] . The specific techniques we have employed are enumerated below.EQUATIONEfficiency optimization. We employ the following strategies to enhance efficiency: 1) We adopt a layer-wise search strategy instead of exhaustively identifying all 𝐿 SoIs in consecutive MCTS iterations. Precisely, after every 𝑁 𝑀𝑐 iterations, we pinpoint one child slice with the highest UCB score for the current initial slice. This selected child slice is the subsequent initial slice for the next 𝑁 𝑀𝑐 iterations. Consequently, the iterative process involves 𝐿×𝑁 𝑀𝑐 MCTS iterations for each volume in each round. 2) We project backbone embeddings of the SoIs into a low-dimensional feature space to reduce computational overhead. 3) We perform pre-computation of shared and parallelizable parts over SoIs within each volume on GPUs to alleviate the workload on CPUs.Momentum planner. To ensure the stability of inter-round search directions, we propose a momentum-based initialization of 𝑁 (𝑠) and 𝑉 (𝑠):𝑁 (•) (𝑟 ) 𝑠𝑡𝑎𝑟𝑡 = 𝑚 × 𝑁 (•) (𝑟 -1) 𝑒𝑛𝑑 , 𝑉 (•) (𝑟 ) 𝑠𝑡𝑎𝑟𝑡 = 𝑚 × 𝑉 (•) (𝑟 -1)𝑒𝑛𝑑 , starting from the 3-rd round, where 𝑚 ∈ [0, 1] is the momentum factor. Cold start. At the beginning of the training, the planner does not adapt to medical images. Thus, we default to selecting the central 𝐿 slices from each volume and shuffling them to create ordered SoIs for warm-up training.Thus far, the proposed planner yields a fine-grained knowledge sequence, denoted as I 𝑖 = [X 𝑖,[𝑡 (1),...,𝑡 (𝐿) ] ], emulating a clinician-like visual inspection strategy, derived from X 𝑖 . This sequence facilitates volume-level label assignment to identified SoIs. Furthermore, it is worth noting that by maintaining a consistent number of tree layers across volumes, the planner can achieve alignment regarding the number of SoIs across volumes of varying lengths.We partition the classifier into three constituents: Neighbour-aware Hierarchical Slice encoding (NHS) module, Sequence-aware Strategy Encoding (SSE) module, and Decision Making (DM) layer. The combination of SSE and DM is akin to the value network in Al-phaGo [23] , which can assess the informativeness of newly incoming slices. We denote the complete classifier as f 𝐶𝐿 = {f NHS , f SSE , f DM }, and its learning trajectory is directed by the supervised signal from the volume-level annotation. Below, we first elucidate the refinement of radiomic features and introduce the three modules.Radiomic feature refinement. To align radiomic features with the same scale as the deep features, within each volume, we reorganize each raw radiomic features into corresponding groups, x 𝑖,𝑅𝑎𝑑 ↦ → [g 𝑘 ] 𝐺 𝑘=1 , based on the radiomics taxonomy (see group names in §4.3.1). Here, g 𝑘 represents a subset of raw radiomic features, and 𝐺 denotes the number of groups. Each feature vector is then projected by a group-specific linear layer into the same feature space for subsequent self-attention interaction. Finally, the interacted features are mapped back and concatenated into the radiomics dimension:EQUATIONwhere ∥ denotes the concatenation operation, Linear 𝑝 𝑘 and Linear 𝑟 𝑘 represent projection into uniform and back into raw radiomic feature sub-space, respectively. The resulting features are connected to the raw radiomic features by a shortcut implemented by a linear layer, as shown in panel (c), Figure 2 . Layer Normalization follows their summation to obtain a refined radiomic feature vector m 𝑖,𝑡 (0) :EQUATION⊕ indicates element-wise addition. Lastly, a linear layer is employed to map m 𝑖,𝑡 (0) to the deep-feature dimension.Neighbour-aware hierarchical slice encoding. We observe that neighboring slices offer moderate 3D information that is beneficial for current 3D tasks. Hence, we introduce the notion X N (𝑛) 𝑖,𝑡 (𝑙 ) to represent the neighbor-aware version of the original X 𝑖,𝑡 (𝑙 ) , covering 𝑛 neighboring slices in total. Here, 𝑛 is an odd number; thus, additional ⌊𝑛/2⌋ slices before and after are concatenated with the target slice. For each augmented SoI, using ResNet-18 [9] as an example, a position-agnostic embedding e 𝑖,𝑡 (𝑙 ) = f NHS (X N (𝑛) 𝑖,𝑡 (𝑙 ) ) ∈ R 1024 is extracted from different encoder stages, specifically, the stem layer and four residual blocks, each followed by global average pooling. These embeddings are then concatenated to form a vector, as shown in panel (d), Figure 2 . Based on this, we compute an embedding sequence e 𝑖,• ∈ R 𝐿×1024 by parallelly applying f NHS on I 𝑖 . These embeddings are projected into a low-dimensional feature space e 𝑖,• = ReLU LayerNorm Linear e 𝑖,• ∈ R 𝐿×𝑟𝐶 for computational efficiency, 𝑟 is scale coefficient.Sequence-aware strategy encoding. We posit that the appearance of the next informative SoI is contingent upon all its predecessor SoIs within I 𝑖 and delineate the conditional distribution of X 𝑖,𝑡 (𝑙 ) as follows:EQUATIONWe utilize MHA equipped with the proposed m 𝑖,𝑡 (0) (Eq. 4), CLS embedding, dropout, and zero-centered position encoding zpe 𝑖,𝑡 (•) (details in §S.4, Supplementary Materials) to model this sequential relationship. For the (sub-) sequences of SoIs, we first calculate the updated conditional features m 𝑖,𝑡 (•) ∈ R 𝐿×𝑟𝐶 as follows:EQUATIONIn this context, MHA is performed slice-by-slice to guarantee that no descendant slice contributes to the computation of any ancestor slices. Each input embedding for the MHA corresponds to a node along the identified path within the experience tree.Decision making. We employ a shared linear layer to obtain the position-wise logits of all SoIs:EQUATIONThe volume-level prediction distribution ŷ𝑖 ∈ [0, 1] 1×𝐶 is Softmax (or Sigmoid for the binary classification setting) over the average of all slice logits:EQUATION)where 𝜏 represents temperature, however, for calculating the loss function, we opt for the slice-level distribution, i.e., without Avg above to get ŷ𝑖,𝑡 (•) ∈ [0, 1] 𝐿×𝐶 . Classification loss function. We formulate the likelihood of the identified SoIs as a joint probability:L 𝑖 (𝜃 ) = L 𝑖 (𝜃 ; I 𝑖 ) = 𝐿 𝑙=1 ŷ𝑖,𝑡 (𝑙 ) (X 𝑖,𝑡 (𝑙 ) |X 𝑖,[𝑡 (1),••• ,𝑡 (𝑙 -1) ] ; 𝜃 ) (9)This likelihood is optimized by maximizing the log-likelihood, which is equivalently represented by minimizing cross-entropy (CE):EQUATION)Penalty Term. The penalty term is crucial for aligning the search direction of the planner with the optimization direction of the classifier and incentivizing the discovery of the most informative SoIs. For the 𝑙-th layer (𝑙 ≥ 2), we define the penalty score between a strategy and its ancestor as:EQUATIONHere, 𝑠 (𝑙) represents a strategy corresponding to a 𝑙-layer tree path. Subsequently, the total penalty is calculated as follows:EQUATIONThe final loss function, balanced by 𝜆, is then given by:EQUATION)Training workflow. The planner and classifier optimization alternate, mutually reinforcing, as shown in Alg. 2. ⊲ Data set for the current round.for 𝑖 ← 1 to 𝑁 𝐷 do 7:← f 𝑃𝐿 (X 𝑖 , x 𝑖,𝑅𝑎𝑑 , 𝑁 𝑀𝑐 , 𝐿; 𝜃 (𝑟 -1) ) ⊲ Build Exp-tree and extract SoIs ( §3.2).8:𝐷 (𝑟 ) ← 𝐷 (𝑟 ) ∪ (I (𝑟 ) 𝑖 , x 𝑖,𝑅𝑎𝑑 , 𝑦 𝑖 ) 9:end for 10: 𝜃 (𝑟 ) ← AdamW f 𝐶𝐿 (; 𝜃 (𝑟 -1) ), 𝐷 (𝑟 ) , 𝑁 𝐸𝑝 ⊲ Train classifier ( §3.3). 11: end for 
experiments 4.1 dataset and pre-processing: 
 The MedMNIST v2 dataset [35] offers a comprehensive benchmark featuring volumetric representations illustrating lesions across six distinct 3D diagnostic datasets with 9998 cases span modalities such as MRI, CT, and electron microscopy. The benchmark encompasses binary and multi-class classification tasks, with tensor dimensions of 28 × 28 × 28 voxels per volume. Each 28 × 28 matrix along the first dimension is treated as a slice. We employ RandomResizedCrop() for data augmentation during the training phase. Value normalization is achieved using the z-score. These datasets are categorized into two problem domains following medical standards:(1) Task 1: anatomical structure and pathological condition analysis.This stratification encompasses the classification of organ types (Organ3D), neuronal synapses (Synapse3D), and lung nodules (Nodule3D). They heavily rely on texture-based features for accurate classification. Details in [35] . (2) Task 2: micro-structure classification. This stratification analyzes fracture micro-structures (Fracture3D), intracranial aneurysms (Vessel3D), and adrenal gland morphology (Adrenal3D). They emphasize morphology-based characteristics, enabling the construction of fine-grained masks over variable-length volumes. [35] 0.977 0.788 0.838 0.835 0.634 0.696 3D 𝑅18 [35] 0.996 0.907 0.863 0.844 0.820 0.745 ACS 𝑅18 [34] 0.994 0.900 0.873 0.847 0.705 0.722 FRM 𝑅18 [40] 0 We delineate the shared hyper-parameters employed across all datasets. Specifically, we fix 𝑁 𝑅𝑜 = 10, 𝑁 𝐸𝑝 = 20, 𝜆 = 0.1, and set the batch size to 64. The neighboring size (𝑛) remains constant at 5 within N (𝑛). We adopt 𝑐 = √ 2 as the balancing factor for UCB and employ AdamW [16] as the optimizer. For a comprehensive overview of hyperparameter configurations, please consult Table S6 , Supplementary Materials. The NHS module employs a ResNet-18 backbone [9] for comparison with prevalent convolutional models regarding model size, convergence speed, and computation cost due to the shared backbone. While our architecture could potentially utilize other visual backbones such as ViT [6] , it falls beyond the scope of our current focus. Model selection hinges upon the validation set's accuracy (ACC) score. Each reported score reflects the average over three runs to maintain consistency with [35] .Extraction of coarse-grained radiomic features. During the feature extraction phase, we utilize a cubic binary mask, the same as the shape of the entire tensor, to capture 107 radiomic features per volume. This extraction process is facilitated by the PyRadiomics platform [28] . These features cover a broad spectrum, encompassing 18 first-order statistics, 16 3D shape-based descriptors, and various matrices including the Gray Level Co-occurrence Matrix (GLCM) comprising 24 features, Gray Level Run Length Matrix (GLRLM) with 16 features, Gray Level Size Zone Matrix (GLSZM) with 16 features, Neighboring Gray Tone Difference Matrix (NGTDM) with 5 features, and Gray Level Dependence Matrix (GLDM) with 14 features. Subsequently, z-score normalization is applied to standardize the radiomic features across volumes. Given the disparate contributions of different radiomic features to various tasks and lesions and recognizing the limited descriptive capability of cubic masks in delineating anatomical structures of interest, we adhere to established medical research practices [26] . Specifically, we utilize one-way ANOVA testing for each dataset to identify discriminative radiomic features exhibiting statistically significant differences (𝑝 < 0.05) among different classes. This analytical process results in the following numbers of radiomic features for the respective datasets: Nodule3D: 62, Organ3D: 90, Fracture3D: 84, Adrenal3D: 68, Vessel3D: 83, and Synapse3D: 82. We quantitatively compared our proposed method's efficacy against SOTA techniques. The outcomes are summarized in Table 1 . Our approach showcases either SOTA performance or comparability across all datasets, spanning diverse methodological families by attaining the highest average ACC and securing the second position in terms of average AUC.Micro-structure Classification. Extraction of fine-grained radiomic features. Volumes from Fracture3D, Adrenal3D, and Ves-sel3D datasets possess a distinct pure black background. This characteristic enables the creation of a fine-grained mask instead of the coarse-grained cubic mask in §4.3.1. We intuitively argue that such a fine-grained mask could enhance the discriminability of extracted radiomic features and deep features derived from a multimodal model. Utilizing the PyRadiomics on the fine-grained binary mask and volumetric images, we obtained the following counts of radiomic features: Fracture3D: 85, Adrenal3D: 39, Vessel3D: 40. In contrast to task 1, this task accentuates the morphological characterization of the lesion.In addition to masking out background voxels for radiomic features, the exclusion of black slices results in volumes of variable-length.The datasets of such characteristics can be used to demonstrate our method's capability to process volumes of varying lengths. We benchmark our approach with and without variable-length volumes against the SOTA, as illustrated in Table 2 . The elimination of black slices (the last row) restricts the search space, resulting in higher efficiency in lesion retrieval, thereby enhancing diagnostic accuracy relative to searching within 28 slices (penultimate row).We proceeded to conduct experiments involving various designs of priori embeddings and present the results in Table 3 . These results unequivocally demonstrate an enhancement in model performance attributable to integrating radiomic features, thereby validating their utility in such tasks. The CLS embedding can be conceptualized as a [35] 0.712 0.508 0.827 0.721 0.874 0.877 ACS 𝑅18 [34] 0.714 0.497 0.839 0.754 0.930 0.928 FRM 𝑅18 [40] 0 S2 to S7, Supplementary Materials. Notably, it is evident that our adopted multimodal clinician-like diagnostic approach altered the distribution weights of features. Given our primary focus on computer algorithm design, we refrained from delving deeper into the associations between distinct radiomic features and various tissues or diseases, a pursuit commonly undertaken in clinical research. Instead, we treated the refined radiomic features post-feature selection as a collective feature vector. We hope these insights will benefit clinical research endeavors. We illustrate samples from two datasets: one with a black background, as illustrated in Figure 4 , and the other lacking such background, as demonstrated in Figure 3 . The planner gradually emphasizes the identification of the most informative SoIs as the learning iterates. Furthermore, these delineated SoIs stimulate clinicians' visual inspection process, thus bolstering the interpretability of the deep learning model. Figure S8 to S13, Supplementary Materials provides comprehensive visualizations across six datasets.We assume that the radiomic features possess an inherently "high-level" character, as they demarcate the 3D lesion of interest from a comprehensive 3D standpoint, typically integrated directly into traditional machine learning classifiers. Consequently, they may not be optimally conducive to deep learning models. We scrutinize this assumption by contrasting static raw radiomic features with trainable refined radiomic features, as delineated in Table 4 .The findings suggest that each feature type possesses unique merits across different datasets, with the static scheme yielding a marginally higher average ACC. Nevertheless, we have opted for the trainable design approach to delve deeper into the influence of multimodal deep models on radiomic features. From the perspective of varying granularity of masks used in extracting radiomic features, Table 5 examines the discriminability of deep features when incorporating radiomic features of different granularity as the second modality. It is observed that the utilization of fine-grained masks leads to enhanced macro-averaged ACC and AUC metrics, highlighting the advantageous impact of finer masks. Additionally, we present outcomes utilizing classical machine learning models, specifically an SVM classifier with an RBF kernel, a methodology commonly adopted in medical research as elucidated # Params (M) FLOPS (B) 3D 𝑅18 [35] 33.15 106.42 2.5D 𝑅18 [35] 11.17 35 .64 ACS 𝑅18 [34] 11.17 in [26] . The classification scores are presented in Table S7 , Supplementary Materials, provide insights into the discriminative capacity of deep learning-refined radiomic features compared to their static counterparts. Encouragingly, an overall performance enhancement is observed, suggesting a potential medical research advancement. We analyze the model size and computational overhead during training, juxtaposing our classifier with competing approaches, as detailed in Table 6 . Specifically, we choose to compare with R18+3D [35] , R18+2.5D [35] , and R18+ACS [34] , as they employ the same backbone architecture. Notably, ACS [34] distinguishes itself by achieving comparable accuracy with reduced computational complexity compared to the naïve 3D scheme. Our analysis reveals a significant reduction in floating-point operations per second (FLOPS) in VR-DiagNet, attributable to the downsampling of slices, thus alleviating computational burdens. In contrast, competitors operate on all 28 slices, resulting in higher FLOPS. Remarkably, our approach maintains a comparable model size to the 2.5D and ACS counterparts. Moreover, we conduct a convergence analysis by training R18+ACS [34] for 200 epochs to align with the total epochs specified in our methodology. The results, as depicted in Figure S1 , Supplementary Materials, showcase our method's superior convergence characteristics across all six datasets.More extensive ablation study results can be found in §S.3, Supplementary Materials. 
conclusions: 
 Modern medical automation needs to improve in several aspects: the black-box nature of the learning models, the single-faceted nature of their inference processes, and the limited ability to handle multimodal information. Our approach, VR-DiagNet, pioneers a model that integrates volumetric and radiomic features, from which information is extracted via an interpretable reasoning framework. This methodology consists of two crucial modules: a planner and a classifier. By incorporating clinicians' diagnostic reasoning into the framework of MDP, the planner utilizes MCTS to extract annotated SoIs, which serve as training data for the classifier. The introduction of class-agnostic information metrics ensures coherent optimization across both modules. We achieve SOTA performance in five of six diagnostic datasets for addressing volumetric tasks in MIA. With VR-DiagNet, we show that robust and accurate deep-learning models for medical use are entirely feasible.  
: 
 challenge for medical models because it renders their conclusions opaque and their inference processes cryptic and unintelligible [21] . This is unacceptable for conventional medical practice, which places high value on transparency in diagnosis and accountability of their results; this is, in fact, the hallmark of many mission-critical systems. In addition, most medically relevant learning models [32, 33] fail to adequately consider the multiple modalities of clinical data. With the increasing emphasis on personalized medicine [13] and its necessary prerequisites on highly customized features, it has become apparent that a single-modality approach towards healthcare has become woefully insufficient. Finally, most current learning models deployed can deal with a single [36] or, at most, a few aspects [37] of a medical problem. This is a significant obstacle to comprehensive and accurate diagnosis since, in most cases, medical conditions are complex and multifaceted, requiring the consideration and consolidation of disparate observations to reach the correct decision.Recent advances in three-dimensional (3D) volumetric imaging have enabled imaging results to present a multi-sliced view of any medical condition, furnishing clinicians with a depth-rich source of visual information. Despite the clear advantages of volumetricity, most convolutional [25, 31, 35] and attention-based [11] frameworks extract visual features at the dense voxel-level, constructing a "global view" of the volume while overlooking its multifaceted nature. In addition to volumetricity, multimodality has yet to be comprehensively harnessed for increased diagnostic robustness and accuracy. Although the idea of multimodality is expected in the medical scenario [1] , very few works have attempted to leverage the large and diverse informational content of imaging feature space, or radiomics, for multimodal integration [20, 38] . With the advent of high-precision imaging equipment, radiomics have emerged as a novel imaging-based data characterization and feature identification system that enables extracting image features, or phenotypes, that comprehensively quantify the specifics of radiographic images. These features have been shown to offer valuable insights into intraand inter-tumor heterogeneities [19] . Finally, interpretability [3] is an active and essential area of research in machine learning, especially in the medicine use scenario [3] . Moreover, it is apparent that interpretability and multimodal integration are often at odds with one another since multimodality (and especially integration thereof) often requires the design of a deep architecture, which is usually not amenable to interpretation [17, 18] . In this sense, the Monte Carlo Tree Search (MCTS) is a robust technique suitable for interpretable decision process planning and has been applied in the fields of Go [23] , retrosynthetic planning [10] , and games [15] .Having identified the critical bottlenecks in achieving interpretable, multimodal-multifaceted integration, we propose merging several essential paradigms to effectively unify these diverse yet crucial concepts. We illustrate our approach by considering two significant aspects of the problem: perceptual and inferential. At the perceptual level, we solve the problem of lack of multifacetedness and multimodality via the introduction of volumetrics and radiomics. Our approach combines descriptive features from both sources to optimize information extraction. Introducing our inference mechanism solves the problem of determining which features are descriptive. We have ensured that our inference logic is fully transparent and interpretable, thus ensuring the logical clarity of our approach.To this end, we propose VR-DiagNet, namely Volumetric and Radiaomic Diagnosis Networks, which jointly incorporates volumetric imaging data and extracted radiomic features to encompass the multimodal integration for the diagnosis capacity enhancement. Moreover, the MCTS algorithm for identifiable inference processes is leveraged as a planning module. Our model comprises two primary multimodal constituents: the class-agnostic planner and the classifier. The planner identifies SoIs with the highest informativeness, enabling the alignment of the number of slices within each volume. Subsequently, sequence-aware strategy encoding is built upon the extracted slice-level local features and global radiomic features in the classifier, transformed into final diagnosis decision-making. The classifier has the same model structure as the planner, consisting of the Neighbour-aware Hierarchical Slice encoding (NHS) module for reading slice representation, the Sequence-aware Strategy Encoding (SSE) module for encoding the decision sequence, and the Decision Making (DM) layer that classifies the slice-level features output from SSE. This comprehensive integration of multimodality, multifacetness, and interpretability shows that our model improves upon baselines. In summary, our contributions are as follows:• By identifying and surmounting key obstacles such as noninterpretability, narrow focus, and limited modality, we make decisive steps toward rectifying some of the severe issues hindering the effective application of deep learning in medical diagnosis; • We present a framework for achieving multifaceted, multimodal medical diagnosis by designing the first learning framework, to the best of our knowledge, that emphasizes the effective interaction between radiomics and volumetric information, mediated by a robust decision-making process; • We present a novel architecture consisting of an interpretable planner and a multimodal, multifacetted classifier, which closely mimics the inference strategies of human clinicians; • We conduct extensive experiments, achieving state-of-the-art (SOTA) results while quantitatively and visually validating the effectiveness of integrating volumetric and radiomic features in the current multimodal tasks. 
 
 Appendix:  


Decision: All reviewers agreed on the submission and responses as provided.

Reviews: review: paper_summary: paper_summary: The paper proposes a method called VR-DiagNet for medical volumetric and radiomic diagnosis. The method integrates volumetric imaging information with radiomic features and utilizes a planner and a classifier to simulate the perceptual process of clinicians. The planner uses a Monte Carlo Tree Search (MCTS) to identify slices of interest **Generated by ChatGPT, no copying allowed!** in the volume, while the classifier incorporates radiomic features and deep features to make the final diagnosis decision. The authors conducted extensive experiments on six public medical volumetric benchmark datasets and achieved superior performance compared to baselines.
strength_weakness: strength_weakness: 1) The paper addresses the important problem of interpretable and robust medical diagnoses, which are essential for practicing clinicians.
2) The proposed VR-DiagNet method integrates volumetric imaging information and radiomic features, which allows for a more comprehensive and accurate diagnosis.
3) The use of MCTS in the planner module provides an interpretable decision-making process, which enhances the transparency of the model.
limitations: limitations: 1) The evaluation of the proposed method is limited to 2 tasks. It would be beneficial to compare the performance of VR-DiagNet with other state-of-the-art methods on a wider range of datasets.
2) The paper could benefit from a more thorough discussion of the limitations and potential future directions of the proposed method.
paper_summary: paper_summary: The paper introduces an innovative diagnostic framework, VR-DiagNet, designed to enhance the accuracy and efficiency of clinical diagnostics using volumetric imaging data. The core components of the framework are a planner and a classifier, which are optimized in a cohesive and iterative manner. This framework simulates the perceptual processes of clinicians by integrating volumetric imaging information with radiomic features, allowing for a more comprehensive analysis.
strength_weakness: strength_weakness: VR-DiagNet enhances the interpretability of deep learning models by integrating volumetric imaging information with radiomic features, which clinicians can more easily relate to and understand.'

The framework overcomes the narrow focus of traditional models by utilizing a customized Monte Carlo Tree Search (MCTS) to construct a volume-tailored experience tree.

The proposed framework is specifically designed to achieve a multifaceted, multimodal approach to medical diagnosis. This is a pioneering step in the field, offering a cohesive system that integrates various types of medical data and simulates the perceptual and cognitive processes of clinicians.
limitations: limitations: The low resolution (28x28x28) of the MedMNIST images may not adequately represent the complexity and detail present in higher-resolution medical images typically used in clinical settings. As a result, comparing VR-DiagNet's performance with state-of-the-art (SOTA) methods using such a dataset could lead to biased or misleading conclusions about its efficacy and superiority.

As observed in Table 1 of the paper, VR-DiagNet did not achieve the best Area Under the Curve (AUC) for all three structures analyzed. Similarly, Table 2 shows that the proposed method did not produce the best results across the three structures evaluated. These results indicate that while VR-DiagNet shows promise, it does not consistently outperform existing methods on the chosen dataset, highlighting potential limitations in its current implementation or optimization.

To fully validate the effectiveness and applicability of VR-DiagNet, it is crucial to test the framework on datasets with higher image resolutions. For instance, using datasets like ChestX-ray14 [1], which features images with resolutions around 1024x1024, would provide a more comprehensive evaluation. Higher resolution images better mimic real-world scenarios, allowing for a more accurate assessment of the method's performance and its potential advantages over current SOTA techniques.

References:
[1] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, "ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit.*, 2017, pp. 3462-3471.
paper_summary: paper_summary: The manuscript addresses critical challenges in the application of deep learning to medical diagnosis, emphasizing the need for interpretability, multimodal integration, and comprehensive analysis. In response, they propose VR-DiagNet, a novel framework combining volumetric imaging and radiomic features to enhance diagnostic capabilities. This approach leverages Monte Carlo Tree Search (MCTS) for interpretable decision-making and integrates descriptive features from multiple sources for optimal information extraction. The model comprises a class-agnostic planner and a classifier, both designed to handle multifaceted and multimodal data, ensuring transparency and logical clarity in inference. The manuscript makes notable contributions by presenting a comprehensive and interpretable framework that significantly improves upon existing baselines in medical diagnosis tasks. Extensive experiments demonstrate the model's effectiveness, setting a new standard for integrating volumetric and radiomic features in multimodal medical diagnosis. This work represents a substantial advancement in the field, addressing key limitations and offering a robust solution for real-world clinical applications.
strength_weakness: strength_weakness: The approach proposed in the paper is solving a problem of significant importance. They have proposed a novel approach and demonstrated state-of-the-art results on multiple open datasets. They have described their approach, datasets, implementation details, evaluation metrics and results very well.
limitations: limitations: I don't see any major concerns with the paper. Although, I am a bit unsure how this paper is directly related to multimedia but the approach discussed in the paper can certainly be extended to multimedia use cases as well and hence can act as a source of inspiration which can drive further research in the field of multimedia.