# A Survey on MLLMs in Education: Application and Future Directions
## 2 Preliminaries of MLLMs
### Examples of Prominent MLLMs
#### 2.4.1 Open-Source MLLMs  
**NVLM 1.0**: Developed by NVIDIA, NVLM 1.0 is a frontier-class family of multimodal large language models designed to excel in vision-language tasks while maintaining or even improving text-only performance [28]. NVLM introduces three architectural variants: the decoder-only NVLM-D, the cross-attention-based NVLM-X, and the hybrid NVLM-H, each catering to different multimodal processing needs. With model sizes up to 72 billion parameters, NVLM combines state-of-the-art performance with high flexibility for handling diverse multimodal inputs. NVLM's training process incorporates a meticulously curated dataset blend, including high-quality multimodal and text-only supervised fine-tuning (SFT) datasets. This approach preserves text-only capabilities while enhancing vision-language performance. NVLM-D processes multimodal input directly within the LLM, achieving unified reasoning capabilities. NVLM-X employs gated cross-attention for efficient high-resolution image processing, while NVLM-H combines elements of both architectures, providing superior reasoning and computational efficiency. One of NVLM 1.0's key strengths is its performance on diverse benchmarks, where it rivals or surpasses proprietary models like GPT-4o and Claude 3.5. For instance, NVLM-D demonstrates exceptional OCR capabilities, achieving the highest scores on benchmarks such as OCRBench and VQAv2. NVLM-H leads in multimodal reasoning tasks like MathVista and multidisciplinary reasoning, proving its versatility for complex educational applications. Despite these advancements, NVLM models also present challenges. The large-scale models require substantial computational resources for both training and deployment, \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Key Capabilities** & **Educational Applications** & **Open Source** \\ \hline \multirow{2}{*}{**PaLM-E**} & Integrates vision and language for robotics and embodied AI. & Enhances interactive, physical learning environments, especially in STEM and robotics education. & No \\ \hline \multirow{2}{*}{**LLaVA**} & Combines vision and language for general-purpose understanding. & Visual question answering, image captioning, supporting visually enriched content in learning platforms. & Yes \\ \hline \multirow{2}{*}{**Kosmos-G**} & Processes text and images for multimodal comprehension. & Facilitates interactive content, supports collaborative and visual learning tools. & No \\ \hline \multirow{2}{*}{**GPT-4o**} & Extends LLM capabilities to visual data, enabling conversational responses to images. & Supports interactive learning with text and visual input, such as image-based Q& No \\ \hline \multirow{2}{*}{**MM1**} & Achieves state-of-the-art performance in multimodal tasks by combining high-resolution visual processing and language models. & Supports tasks like in-context learning, multi-image reasoning, and few-shot learning, useful for assessments and exploratory learning. & No \\ \hline \multirow{2}{*}{**LIama 3-V**} & Combines vision, language, coding, reasoning, and tool usage with multilingual support. & Enables adaptive and multilingual learning, coding education, and collaborative projects. & Yes \\ \hline \multirow{2}{*}{**NVLM 1.0**} & Frontier-class multimodal model with exceptional vision-language reasoning and text-only improvements. & Supports OCR tasks, multimodal math reasoning, and document analysis in educational environments. & Yes \\ \hline \multirow{2}{*}{**BLIP**} & Pre-trained for text-image retrieval and multimodal content generation. & Facilitates creative content development and storytelling in visual education. & Yes \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Overview of selected prominent MLLMs and their educational applications. which may limit accessibility for smaller institutions. Additionally, while NVIDIA plans to open-source the training code and model weights, the complexity of implementation might pose challenges for non-specialist users without significant expertise or infrastructure.