# Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
## 3 Methodology
### Hierarchical Answer Decoders  
Hierarchical Answer Decoders employ cross-modal cross-head attention followed by a residual FFN block. The cross-modal attention layer aims to establish a dynamic correlation mapping between visual and text features. We use medical image embedding as the query and text embedding as the key value at the same time. This design idea is derived from a systematic analysis of the characteristics of heterogeneous modalities: medical reports have strong semantic coherence and high information density, which are suitable as semantic anchors in the attention mechanism, while medical images contain complex spatial distribution characteristics and fuzzy pathological representations, which are more suitable as query subjects to trigger semantic retrieval. By setting the text as a key/value pair, the model can effectively use the precise semantics of the diagnostic text to guide the semantic focus of image features--the attention mechanism reconstructs the visual representation in the form of weighted aggregation of text features (values) by calculating the similarity between the local area of the image (query) and the text semantic unit (key). In particular, since the attention output is essentially a probability-weighted combination of values, using text with higher information density as the value carrier can maximize the effectiveness of semantic fusion. For example, using the embedding vector of precise terms such as \"pneumonia\" as the value input can make the output features more accurately reflect the key pathological signs in medical images. Since the information density of the questions extracted from the structured medical reports is greater than that of the ordinary questions, and the medical image features are already the features that have been previously prompted according to the hierarchy, this design can better refer to the local area of the image and the contextual information of the question, while ensuring the efficiency of cross-modal interaction fully fits the objective characteristics of accurate text and complex images in the medical field, and provides the optimal multi-modal representation basis for subsequent disease classification and positioning tasks. The calculation process of cross-attention is as follows:  
First, the image-prompt embedding \(F_{p}\) is converted to query \(Q\), and the text embedding \(V_{q}\) is converted to \(K\) as Key and \(V\) as Value:  
\[Q=F_{p}W_{Q},K=V_{q}W_{K},V=V_{q}W_{V} \tag{5}\]  
\(W_{Q}\), \(W_{K}\) and \(W_{V}\in R^{d\times dk}\) are learnable weight matrices. The inner product of the query and key is calculated and scaled and normalized:  
\[F=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)} \tag{6}\]  
where \(F\) denotes the attention weight between image and text tokens. The attention score is applied to the value to obtain an output that combines the image and text embeddings:  
\[E=FV, \tag{7}\]  
where \(E\) is the combined embedding. A feedforward network is applied to the output of the cross-attention and a nonlinear transformation is performed. This further enhances the feature representation. The fused features are then used to perform multi-label classification on all answer candidates. However, we only consider outputs that are valid for the current question as the correct answer. For single-choice questions, we predict a single label by applying a softmax function to all valid answers. For multiple-choice questions, we predict multiple labels using a sigmoid function. For all question categories, due to the imbalanced class distribution within the dataset, the model fails to adequately learn relevant features for certain medical questions. To address this, we employ a weighted masked cross-entropy loss function where \(w\) represents a class-specific weight matrix Algorithm 1. Classes with fewer samples are assigned higher weights to mitigate data imbalance.