### Meeting notes

- 2024-09-12
    - dissection VLM (simple but comprehensive study?)
    - small phenomenon → dig deeper?
        - internal thoughts before CoT
        - eye sight test
- 2024-09-10
    - Refer to Zeyuan Zhu’s tutorial, and run some small-scale controlling experiments
    - Need to gain more insights on each topic first, and then decide if we can commit to or continue with the topic
- 2024-09-07
    - Some LMMs focus on both language and vision abilities (e.g., GPT-4V, GPT-4o), while others prioritize multi-modal capabilities. The latter typically report their models' performance on multi-modal benchmarks without pure language benchmark scores (e.g., Qwen2-VL, LLaVA-OV).
    - Curating a comprehensive table with all desired information is not that easy. It's more practical to build a small-size table aligned with our specific focus. However, we can first identify which variables and metrics are relevant to our research (see 240907 slide).
    - Make it clear that the main objective of our LMM project is to understand the performance of LMMs. Refer to "Topics being investigated" for specific details.
    - Document all discussions and ideas in Notion to maintain a record of our progress.

### Materials:

- Slides:

[jal_meeting_slides - Google Drive](https://drive.google.com/drive/folders/1OXG-eR8bnZBiwxaWLo_5gcHSofX0R97H?usp=sharing)

- Useful links:
    - https://rank.opencompass.org.cn/home
    - https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
    - https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
    - https://llava-vl.github.io/blog/
    - https://qwen2.org/vl/
    - https://github.com/OpenGVLab/InternVL
    - https://openxlab.org.cn/apps/detail/kennyutc/open_mllm_leaderboard
    - https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models

### Project: Understand the performance of LMMs (ongoing)

- Selected related papers:
    - Law of Vision Representation in MLLMs (Stanford, 2024-08-29)
        - select the optimal vision encoder based on **AC** (**A**lignment, **C**orrespondence) policy
        - poor notation, good motivation
    - A Survey on Evaluation of Multimodal Large Language Models (NTU, 2024-08-28)
    - Building and better understanding vision-language models: insights and future directions (HuggingFace, 2024-08-22)
        - **VERY insightful, offer lots of directions which deserve to be explored further!**
    - LLaVA-OneVision: Easy Visual Task Transfer (LMMs Team, 2024-08-06)
        - see their blogs for details
        - how to interpret the data mixing policy in this paper?
    - MiniCPM-V: A GPT-4V Level MLLM on Your Phone (OpenBMB, 2024-08-03)
    - A Survey of Multimodal Large Language Model from A Data-centric Perspective (HKUST, 2024-07-18)
    - Observational Scaling Laws and the Predictability of Language Model Performance (Stanford, 2024-07-02)
        - good paper, interesting story and complete experiments
    - Long Context Transfer from Language to Vision (LMMs Team, 2024-06-30)
    - Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (Shanghai AI Lab, 2024-06-20)
        - tricky title, boring idea
    - Frozen Transformers in Language Models Are Effective Visual Encoder Layers (UIUC, 2024-05-06)
        - Interesting idea, might enhance our understanding of how LLMs interact with visual information
    - Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (NYU, 2024-04-25)
    - MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training (Apple, 2024-04-18)
        - very solid experiment, can gain some insights from the ablations
    - Compression Represents Intelligence Linearly (HKUST, 2024-04-15)
        - simple idea, good motivation
    - When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (Google, 2024-02-26)
        - explores how various factors affect the finetuning stage of LLMs, solid experiments with scaling laws
    - Aligning Modalities in Vision Large Language Models via Preference Fine-tuning (UNC, 2024-02-17)
    - VILA: On Pre-training for Visual Language Models (NVIDIA, 2023-12-12)
    - An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models (Microsoft, 2023-09-18)
    - Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (Alibaba, 2023-09-12)
        - rejection sampling, pre-training loss v.s. performance after fine-tuning
    - Scaling Laws for Generative Mixed-Modal Language Models (FAIR, 2023-07-03)
    - GPT4Image: Can Large Pre-trained Models Help Vision Models on Perception Tasks? (PKU, 2023-06-07)
    - Reproducible Scaling Laws for Contrastive Language-Image Learning (LAION, 2023-06)
        - dataset scaling law
    - Scaling Laws of Synthetic Images for Model Training ... for Now (Huawei)
        - synthetic dataset scaling law
- Topics (one-sentence version):
    1. Investigate how language instruction data helps with LMMs.
    2. How to train an LMM without compromising its language capabilities? Or, how to train an LMM to improve both its language capabilities and its multimodal capabilities?
    3. Predict the performance of LMMs on multi-modal benchmarks based the performance of the corresponding LLMs on language benchmarks.
    4. Multi-modal needle in the haystack, focus on not only long context but also complex context/semantic.
        1. focus on perception, excluding reasoning and knowledge
    5. Try to figure out how LLMs affect the perception ability of LMMs. 
        1. can we design controlled experiments based on synthetic dataset?
        2. any further thoughts based on the observations of “eye sight test”
    6. 
- Topics being investigated:
    - 💡 Evaluate how a multi-modal task can be affected by non-LLM modules in LMMs?
        - Fix the LLM part, and calculate the performance gap of LMMs on multi-modal benchmarks between the best and the worst
        - Difficult to obtain meaningful results due to significant variations in LMM training quality
        - Vision Encoders
        
        ![image.png](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/d8d5b59b-d86c-45ef-8885-9c9f172b2dd5/image.png)
        
        []()
        
    - 💡 Performance of LLM on a language task *v.s.* Performance of LMM on the same language task
        - Need to evaluate the LMMs by ourselves
        - Less output token from LMM compared to LLM → inference compute? → mitigating via annealing/alignment methods?
    - 💡 Predict the performance of LMMs based on LLM features?
        - feature 1: benchmark scores
            - data contamination
            - overfitting
        - feature 2: abstract abilities
        - …
    - 💡 Performance of LLM on a language task *v.s.* Performance of LMM on a multi-modal task
        - **Goal: Check if we can predict the performance of LMM only based on its LLM’s performance**
        - Exp. 1: MMLU v.s. multiple multi-modal tasks
            - Varying LLM model series, but select the LMM with the best performance for each LLM
            - Source: OpenCompass
            - Model (LLM): InternLM2-7B-Chat, InternLM2.5-7B-Chat, Llama3-8B-Instruct, Llama3-70B-Instruct, Qwen1.5-7B-Chat, Qwen2-7B-Instruct, Qwen2-72B-Instruct
            - Results (R-square):
            
            ![image.png](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/226feb83-bd88-43d6-a224-263ae56a258c/image.png)
            
            - Comment: If we don’t control variables, just predict the multi-modal performance based on the performance of language, it’s impossible to get accurate predictions.
        - Exp. 2: MMLU v.s. multiple multi-modal tasks
            - Varying LLM model series, but fixed training recipe and vision encoder
            - Source: https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/
            - **R-square: 0.87 (MMMU), 0.87 (MathVista), 0.63 (AI2D), 0.91 (LLaVA-W)**
            - The behavior of Mistral-instruct-v-0.2-7B looks strange…
        
        ![image.png](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/3f98fa46-4be9-4191-bed9-b52b8b05a32d/image.png)
        
        - Exp. 3: Multiple language tasks vs. multiple multi-modal tasks (Qwen1.5 series only)
            - Fixed language model series, training recipe, and vision encoder
            - Model:
                - LLM: Qwen1.5, 0.5B ~ 72B
                - Vision encoder & training recipe (fixed): https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/
            - Results (R-square):
            
            ![image.png](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/f3b61d03-8f52-4aa4-8f41-99f9b55cabfc/image.png)
            
        - Comment: From observing the above two experiments, it seems that we can not infer the performance of VLMs on a certain multi-modal tasks only based on the LLM’s performance on language tasks. Model family matters!
        - Conclusion: We can somehow predict the performance of LMM based on its LLM’s performance on language tasks, but the (LLM) model series, training recipes, and vision encoders should be fixed to get more accurate predictions.
    - 💡 Investigate how language instruction data helps with LMMs:
        - InternVL, MM1, and LLaVA-OV claim to "maintain or further enhance language capabilities," but how exactly? To what degree?
    - 💡 How to train an LMM without compromising its language capabilities? Or, how to train an LMM to improve both its language capabilities and its multimodal capabilities?
        - Previously, GPT-4 and GPT-4V ⇒ now, only GPT-4o?
        - What’s the language performance of LLaVA, InternVL2, and etc?
    - prepend the data with specific tags
        - automatically data selection?
    
    ![image.png](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/78016cd8-87af-46a7-b856-27c154255562/image.png)
    
    - …
- Exploratory qualitative experiments (not very reliable):
    - In “eye sight test”, LMMs with small-size LLMs perform more poorly than LMMs with large-size LLMs.
        - This experiment doesn't require knowledge or reasoning abilities from the LLM side, so **small-size LLMs are likely difficult to align with visual encoders.**
        - somehow like semantic mismatch?
        
        ![20240911151154.jpg](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/b80dd015-cc82-41d3-984a-aad687103156/20240911151154.jpg)
        
    - LLMs seem to perform better than LMMs on language benchmarks
        - more reasonable intermediate thoughts
        - but personally I think, the difference reflected in the benchmark will not be too big.
        - LMMs tend to generate less words than LLMs? not super sure
        - source:
            - https://www.modelscope.cn/studios/opencompass/CompassArena/summary?fullScreen=1
            - https://internvl.opengvlab.com/
    - current LMMs generally fail **color blindness test** and **spot the difference**
        - no further thoughts…
        - can we extend it to an evaluation benchmark?
        
        ![20240911150438.jpg](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/8f00e586-1ea6-4565-91fb-72c87434b53f/20240911150438.jpg)
        
        ![image.png](https://prod-files-secure.s3.us-west-2.amazonaws.com/99ea616b-a979-41c7-81dd-723df646c9c8/2af40ef7-32e3-4c13-b5d1-9bf8b8f1b8ed/image.png)
        
    - ~~consistency check~~
        - useless

### Project: Evaluate the potential of instruction tuning (on hold)

- Why on hold?
    - require large computational resources
    - work from Google has explored the scaling laws in the fine-tuning stage
- But:
    - will still keep an eye on related papers
    - will consider restarting it once we come up with an easy way to obtain scores of LLMs after IT
- Some thoughts:
    - can **learning curve extrapolation** be adapted to LLM prediction?
    - how to understand




上面是我的笔记记录，告诉我你觉得最值得做的方向？