# VL-Few: Vision Language Alignment for Multimodal Few-Shot Meta Learning
## 1 Introduction  
With the development of language models [1; 2; 3; 4; 5; 6], vision models [7], and multimodal models [8; 9; 10; 11; 12; 13; 14], there are many applications that bring people more efficiency and convenience in their lives based on this state-of-the-art technology, such as smartphone assistants [15], chatbots [16], smart healthcare [17], smart transportation [18], smart shopping guides [19], autonomous driving [20], and so on. The success of applying this technology requires large different modal data for training like text, image, voice, video, table, graph, machine recording, and so on to train the models. However, how to make the language models understand other different modal data is a different matter. In the visual question answering (VQA) [21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32] task, the main modes of the data are vision and language because vision represents the eyesight and language represents the thinking of the brain, which are the key functions people use to interact with the environment and complete tasks. Making language models understand data from other modalities such as images, videos, speech, point clouds, etc., is a promising ability that will help language models understand real-world knowledge, complete more complex and realistic tasks, assist humans in completing tasks, and improve productivity. The core of multimodal learning ability is modal alignment. For example, aligning visual features into language space enables language models to understand images or videos. The traditional method is to train a multimodal model from scratch, but this approach requires preparing a large amount of high-quality multimodal alignment data in advance. For example, CLIP [10] constructs a large number of image text pairs, but itcannot understand the modalities that have not been learned, thereby resulting in limited scalability. Inheriting the modal processing ability learned by the pretrained model to complete multimodal tasks is a promising research direction. This problem is more serious in the few-shot scenario because of scarce training data. So, to alleviate the low-resource problem and improve the generalization ability of the model, meta learning [33; 34; 35; 36; 37; 38] and few-shot learning [39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52] can be considered. Meta learning can construct a task pool to improve the generalization ability of the model. Few-shot learning can reduce the dependence on the data size of the model. Therefore, to solve the multimodal few-shot problem, we propose VL-Few, a multimodal few-shot meta learning framework as shown in Figure 1. VL-Few uses a small amount of data, combines meta learning to build a task pool, and constructs a small number of training inputs with small samples. Through multimodal feature mappers, the language model can understand visual features and reduce the number of parameters for model training, thus reducing the dependence of the model on data volume and ultimately completing the VQA task. There are some differences between other research studies and ours. For model architecture, other methods require retraining of the vision model (Frozen) or only consider mapping vision image features and language question features into the same (CLIPCap [53]) or fewer (MML [54]) tokens through mappers to help the model understand vision image features. However, our method realizes that the prominent problem with understanding multimodal data is that the features are not in the same feature space, and the failure to align multimodal features effectively results in the model being unable to understand other modal data. Therefore, we consider both vision image features and the interaction features between vision images and language questions. We ultimately fuse them with other auxiliary features through a fully connected layer to form the final input feature. In addition, our experiment used less data to achieve better performance.