# VL-Few: Vision Language Alignment for Multimodal Few-Shot Meta Learning
## 2 Related Work  
There are some research studies on multimodal learning to perform the VQA task. CLIPCap [53] proposes to use CLIP-ViT [10] to extract image features, interact with images and captions through a trainable mapper network, obtain image features containing text, use them as input prefixes, input them together with the question into GPT-2 [3], and finally generate an answer to the image viewing question. This method can enable language models to understand images by training only a small number of model parameters. However, it requires a large amount of data for training, thus making it difficult to apply to small sample scenarios. BLIP-2 [23] proposes a Q-former (querying transformer) structure to map image features into language feature space. Specifically, this method extracts image features through the frozen image encoder CLIP-ViT [10], and it inputs both image features and learned queries into the Q-former. The Q-former fuses image features into learned queries, outputs learned queries, and obtains new image features through a fully connected layer. The new image features are used as prefix text and input together with the question into the language model. Finally, the language model generates the answers based on the image and question. This method obtains image features through a learnable sequence, which is then fused with the questions to obtain image features related to the problem, thus ultimately resulting in a good representation of the image features. However, this method requires a large number of data for pretraining, which is a total of 129 M, thus making it difficult to apply it to small sample scenarios. NExT-GPT [55] proposes to extract features from the data of different modalities through corresponding encoders and then map the features to the language space through the projections. The core task is understood through a language model, and the language features are mapped to different modality spaces through instruction following. Finally, the target modality data are generated through the diffusion model of the different modalities. This framework proposes a unified approach to convert data from different modalities into data from other modalities. However, this method requires a large amount of aligned data during the training phase, which poses challenges for small sample scenarios. There are some research studies regarding multimodal few-shot learning to perform the VQA task. Frozen [56] proposes a multimodal few-shot method that freezes the language model, trains only the visual encoder, and directly inputs image features into the language model. Frozen not only reduces the number of training parameters by updating the visual encoder parameters but also enables image features to be understood by language models. In addition, Frozen provides demonstrations of the task that help  
Figure 1: VL-Few VQA model architecture. We construct the whole model structure network to align the vision feature into language space. the model understand the task form and the input and output for the task. However, this method requires a large amount of data to train the visual encoder, thus making it difficult to apply to small sample scenarios. MML [54] proposes the meta mapper to map the image features to language feature space. Specifically, MML extracts image features through a visual encoder and inputs them together with a visual prefix into the meta mapper. The meta mapper maps image features to visual prefixes through the network structure. Subsequently, the visual prefix is inputted into the language model along with the sentence to help the model understand the images and generate the answer. This method freezes the visual and language models and only trains the meta mapper network, which reduces the number of trainable parameters and has good scalability. However, this method still requires a certain amount of data for training. The above research studies need a lot of data for training, even large-scale data for pretraining.