# Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
## 4 Experiments
### Training and Evaluation  
During training, we employ the teacher-forcing strategy [36], feeding the problem along with the previous layer's problem and answer as context input. Given that the level3 problem is a multi-choice question, we implement a weighted mask cross-entropy loss function, calculating the loss only for the labels relevant to the current problem. The AdamW optimizer is set with a learning rate of \(10^{-5}\), and the end-to-end training is conducted on an NVIDIA RTX4090 GPU (NVIDIA, Beijing, China) using the PyTorch-Lightning 1.8.3. framework. The number of epochs is dynamically determined based on the performance on the validation set. Additionally, we adopt data augmentation strategies [14; 37], including random dropping [38] and reordering of questions at the same level to prevent overfitting [39]. The model supports a multi-task output mechanism, using softmax for single-choice classification and sigmoid for multi-label classification in multiple-choice scenarios, while strictly constraining the valid answer space. The training process emphasizes hierarchical dependencies. Each sample in the training data is summarized as \(D=(p_{i},q_{i},I,y_{i})\), where \(p_{i}\) represents the prompt corresponding to each question, \(q_{i}\) represents the question, \(I\) is the medical image, and \(y_{i}\) is the true answer label corresponding to the question. One medical image receives a complete structured report, and a structured report includes three levels of medical questions. Each sample is split into a medical image and one of the medical questions within it. We freeze the image encoder and text encoder, training only the alignment module, hierarchical answer decoders, and MLP classifier.  
For fair and convenient comparison, we adopt the same evaluation method as Hi-VQA and context-VQA [29]. The evaluation system uses macro-average precision, recall and F1 metrics to cover all possible paths of hierarchical questions, and simultaneously calculates the accuracy of the complete report. The report-level accuracy is the proportion of all paths where all questions on that path are predicted correctly out of all paths. If the higher-level question on a path is predicted correctly but the lower-level question is incorrect, that path is considered a wrong prediction. The evaluation is auto-regressive, so the model utilizes the previously proposed questions and their predicted answers as historical context. The evaluation enforces hierarchical consistency constraints. In hierarchical visual question-answering tasks like HiCA-VQA, if the model predicts \"no\" for a higher-level question, the path reasoning is interrupted, and the lower-level sub-questions in the hierarchical structure are automatically determined as \"no\" or \"not selected\" (for the level3 question), thereby enforcing consistency in predictions. This aligns with the actual situation in clinical diagnosis, ensuring that the generated reports are consistent and coherent. If a medical expert determines from a global perspective that there are no abnormalities in the image, then more detailed and granular investigations are unnecessary. Finally, since the object, sign, or pathology of a patient may appear multiple times, when the model predicts \"yes\", the model will iteratively ask about further occurrences (e.g., \"Are there any other opaque areas in the lungs?\"). The model will limit the number of subsequent questions based on the maximum occurrence of each patient in the data to ensure data consistency. Due to the unclear order of appearance, instance matching is applied during the metric calculation for evaluation to achieve the highest F1 score for this discovery.