# Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
## 1 Introduction  
Medical Visual Question Answering (Med-VQA) aims to generate accurate diagnostic answers from candidates by jointly analyzing medical images and natural language-described clinical questions [1, 2]. This task provides essential technical support for AI-assisted diagnosis, medical education, and clinical decision-making [3, 4]. In recent years,Med-VQA has demonstrated potential in applications such as pneumonia detection [5] and tumor classification [6], with current research focusing on precise multi-modal feature fusion and emulation of physicians' hierarchical diagnostic logic [7] to better align with clinical workflows [8]. Traditional Med-VQA approaches typically treat complex medical questions as single granularity tasks, neglecting the progressive reasoning pathway of \"Topic Existence \(\rightarrow\) Element Existence \(\rightarrow\) Attributes\" inherent to clinical diagnosis. To address this limitation, as illustrated in Figure 1, a traditional Hierarchical Medical Visual Question Answering (Hierarchical Med-VQA) task has been proposed [9]. These models simulate clinical diagnostic pathways by decomposing structured report questions into hierarchical levels, using autoregressive methods to sequentially predict multi-level medical questions corresponding to a single medical image. While demonstrating advantages over conventional image-question Med-VQA models in clinical relevance and automated hierarchical report generation, these models retain critical limitations: (1) The continued use of single-layer answer encoding architectures from traditional Med-VQA leads to hierarchical semantic fragmentation, stemming from feature space coupling and gradient optimization conflicts across hierarchical fine-grained question tasks. Specifically, shared visual representations must simultaneously satisfy divergent semantic requirements across hierarchy levels, /textcolorblueleading to attention competition and representational confusion during feature learning, loss gradients from subtasks exhibit directional conflicts during backpropagation, exacerbating optimization instability. This fragmentation amplifies at the data distribution level: Anatomical regions with long-tailed distributions suffer representational suppression due to gradient dominance from high-frequency tasks, resulting in imbalanced performance across hierarchical question levels. In conclusion, current hierarchical Med-VQA implementations achieve question hierarchy only at the dataset level without a corresponding architectural hierarchy. (2) The conventional Transformer-based concatenated self-attention fusion method [10] in such models shows critical limitations: Its implicit, coarse-grained cross-modal interaction mechanism fails to meet medical scenarios' demands for precise alignment and robustness. By simply concatenating image-text embeddings before self-attention layers, existing methods achieve global feature mixing but fundamentally rely on implicit cross-modal relationship learning, thereby obscuring crucial local semantic correlations in medical contexts. To address these challenges, this study proposes HiCA-VQA, a framework incorporating hierarchical modeling with cross-attention fusion. Within this architecture, questions at different fine-grained levels are processed through dedicated answer decoders. Specifically, we introduce a hierarchical prompting module and hierarchical answer decoders. Based on the fine-grained level of the current question, the hierarchical prompting module introduces distinct prompts to pre-align text prompts with corresponding image features before fusing them with question features for decoding. The primary objective is to guide the model in progressively shifting attention from global to local image regions, thereby completing stepwise reasoning from screening to detailed analysis. We introduce a cross-attention fusion module [11] where image features serve as queries and text features as key-value pairs.