# Generative Models in Medical Visual Question Answering: A Survey
## 2 The Evolution of Generative MedVQA
### Integrations of Vision-Language Pretraining with Various Answer Generators  
The advancement of generative models in the general domain has led to several MedVQA systems exploring different answer generator architectures, such as BART [20; 22] and T5 [21; 23]. Meanwhile, starting in 2021, vision-language pretraining (VLP) became a dominant strategy in building MedVQA systems. These generative models not only focused on answer generation but also incorporated VLP to improve image-text alignment, enabling more efficient handling of multimodal data. One notable model, M2I2 [34], integrates multimodal learning by unifying pretraining objectives across image and text features. The model, pretrained with tasks like Masked Image Modeling (MIM), Masked Language Modeling (MLM), Image-Text Matching (ITM), and Image-Text Contrastive Learning (ITC) on the ImageCLEF2022 dataset [40], consists of a 12-layer Vision Transformer (ViT) as the image encoder and a 6-layer transformer initialized with BERT for text encoding. It further integrates a 6-layer transformer with cross-attention for multimodal fusion and a 6-layer transformer-based decoder for answer generation. Another approach, MUMC [33], introduces Unimodal and Multimodal Contrastive Losses (MUMC) to complement pretraining tasks like MLM and ITM. The Unimodal Contrastive Loss (UCL) differentiates examples within the same modality (either image or text), while the Multimodal Contrastive Loss (MCL) aligns image and text embeddings in a shared space [33]. The architecture mirrors M2I2, and the model is pretrained on multiple datasets, including ROCO [41], MedICaT [42], and ImageCLEF2022 [40]. The Multimodal Prompt Retrieval (MPR) framework [23] leverages a T5 transformer [21] for auto-regressive answer generation. The framework integrates a retrieval module that selects relevant multimodal prompts from a pre-indexed database of medical image-text pairs. These prompts are then used as additional context during inference. The vision-language encoder, based on CLIP-ViT, extracts visual embeddings aligned with retrieved text embeddings for seamless multimodal fusion. A generative decoder, implemented using a pretrained GPT model, generates open-ended answers conditioned on the question, retrieved prompts, and encoded visual features. The pretraining follows a two-stage strategy: the first stage trains the model with self-supervised contrastive learning tasks, aligning vision and language features using a synthetic dataset of medical images with associated captions and metadata from ROCO; while the second stage fine-tunes the model on MedVQA datasets, such as Slake and PathVQA, using a retrieval-augmented generative learning objective. BiomedGPT [22] is a unified encoder-decoder model designed to handle vision, language, and multimodal tasks. Based on the BART architecture [20], it integrates a BERT-style encoder for bidirectional understanding and a GPT-style autoregressive decoder for text generation. Pretraining tasks include MIM, MLM, image captioning, VQA, and object detection (OD). The model is pretrained on a wide range of datasets, including CheXpert [43], CytolmageNet [44], Slake, PathVQA, Peir Gross [45], IU X-ray [46], PubMed Abstracts [47], MIMIC-III Clinical Notes [48; 49], and NCBI BioNLP Corpus, among others. The MISS (Multi-task Self-Supervised-learning-based) model [15] adopts a Joint Text-Multimodal (JTM) encoder, which unifies text and multimodal feature extraction to replace traditional dual-tower architectures. The model incorporates a ViT-based image encoder for visual feature extraction and a transformer-based text decoder for generating free-form answers. To address the challenge of limited multimodal medical datasets, MISS introduces Transfer-and-Caption (TransCap), a technique that leverages LLMs to generate structured captions for unimodal medical images, effectively expanding pretraining data.