# Generative Models in Medical Visual Question Answering: A Survey
## 3 Dataset and Evaluation
### Evaluation Metrics
#### 3.2.2 Open-Ended Evaluation Metrics  
Open-ended MedVQA questions involve generating free-form text responses, requiring more sophisticated evaluation techniques beyond accuracy. Table 5 presents adopted evaluation metrics for open-ended questions.  
**Accuracy**: While traditional accuracy is difficult to apply directly to open-ended responses, some studies use exact-match accuracy, where a response is considered correct only if it matches the ground truth exactly. However, this metric is often too rigid for natural language generation tasks.  
**Recall**: Recall measures how many ground-truth tokens appear in the generated response. This metric is particularly relevant for MedVQA tasks where completeness of information is crucial, such as in diagnostic explanations.  
**F1 score**: The F1 score is the harmonic mean of precision and recall, providing a balanced measure of both false positives and false negatives.  
**BLEU (Bilingual Evaluation Understudy)**: BLEU is a widely used metric that measures the overlap of n-grams between generated responses and reference answers. It assigns higher scores to outputs that closely match human-written references.  
**WBSS (Word-Based Semantic Similarity)**: WBSS evaluates how semantically similar a generated response is to the reference answer by considering the relationships between words. Unlike BLEU, it captures meaning rather than exact word matches, making it more suitable for clinical applications where different phrasing can still convey the correct diagnosis or interpretation.  
**CBSS (Concept-Based Semantic Similarity)**: CBSS evaluates the similarity between two texts based on extracted biomedical concepts rather than exact word overlap with the help of MetaMap.  
**BERTScore (BERT-Sim)**: BERTScore computes similarity between generated and reference answers using contextualized embeddings from BERT. It assesses token-wise similarity in a way that accounts for synonymy and paraphrasing, making it a more robust metric for evaluating MedVQA models.  
**ROUGE-L**: ROUGE-L evaluates the longest common subsequence (LCS) between the generated and reference texts, offering a measure of lexical similarity that accounts for  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Evaluation Metric** & **Used by** \\ \hline Accuracy, Exact-match & [13; 14; 15; 17; 22; 25; 29; 31; 33; 34] \\ \hline BLEU & [6; 13; 14; 16; 23; 27; 31] \\ \hline WBSS & [6] \\ \hline CBSS & [30] \\ \hline F1 score & [13; 16] \\ \hline BERTScore & [13; 23; 25; 27] \\ \hline ROUGE-L & [27] \\ \hline Recall & [24; 35; 37] \\ \hline Clinical Evaluation & [25; 27; 29] \\ \hline GPT-4-Based Evaluation & [24; 27; 29] \\ \hline Deepseek-Based Evalutation & [35] \\ \hline VLM-Based Evaluation, not specified & [38] \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation metrics for open-ended questions in MedVQA.  
word order while being more flexible than n-gram-based metrics, making it suitable for tasks like summarization and report generation.  
**Clinician Evaluation**: Some studies ask clinician to score the model answers by giving quality scores on correctness in a 10-point [29] or 5-point range [25]. Some also score from different aspects such as relevance, completeness, coherence, and explainability [27].  
**LLM-Based or MLLM-Based Evaluation**: Recent studies leverage GPT-4, Deepseek, and some other LLMs and VLMs as an evaluator, scoring generated responses based on accuracy.  
By employing a combination of these metrics, researchers can obtain a more comprehensive assessment of MedVQA models, balancing precision, semantic similarity, and clinical utility.