Improving Multimodal Large Language Models Using Continual Learning
Authors: Shikhar Srivastava, Md Yousuf Harun, Robik Singh Shrestha, Christopher Kanan


Reviewer paper summaries: The paper starts from the valid observation, that even though multimodal LLMs (MLLMs) are LLM backbones trained further to “understand” image tokens as well, they are not as capable as LLMs on linguistic understanding. The authors try to alleviate the “forgetting” of linguistic capabilities in MLLMs with continual learning strategies. A careful choice of strategies can reduce the performance degradation of language understanding in MLMs trained from LLMs. But language understanding is still less than the original LLMs and still takes a loss in vision & language understanding too compared to a standard MLM.

**Comments after Author Discussion**: I have read and acknowledge the author's response, however I kept my score.

The authors present their work of adapting continual learning methods for dealing with the issue of linguistic forgetting when fine-tuning LLMs for vision-language tasks.
Summary of strengths: 1. The main strength of the paper is in its idea, namely to indentify the shortcomings of MLLMs: they are initialised with an LLM capable of NLU and NLG, but by further training on vision and language data, the MLLM gains vision and language capabilities, to the detriment of language-only capabilities. To alleviate the problem, the paper makes the nice contribution of thinking about this in the context of continual learning. 

1. The paper uses an extensive batch of MLLMs. 

1. It tries out quite an extensive number of continual learning strategies, namely 5, which prove to be somewhat effective.

Novel work of adapting continual learning methods for dealing with the issue of linguistic forgetting when fine-tuning LLMs for vision-language tasks
Summary of weaknesses: 1. **Limited novelty**: While a great contribution of the paper is framing the problem described in S1 as a continual learning problem, the paper makes no novelties in the spectrum of continual learning, but just applies known and established continual learning strategies to the problem of vision and language. This limits the contributions and novelties of the paper quite a bit.  

1. Interesting, but **maybe not the most useful framework**: It is a bit underwhelming that a fancy name of “continual learning” applies to strategies like careful fine-tuning with LoRA and giving the model some data from earlier tasks in later training stages, to not forget them – since the forgetting is largely a data problem: imo the forgetting happens because the VL data is  much less in numbers and much less linguistically diverse in comparison to text-only data. This is also found by this paper which in L389 shows that instruction tuning (which has more linguistic variance than bland images and captions) almost single-handedly circumvents the problem. This makes the whole (admittedly interesting) framework of continual learning, a bit obsolete. 

1. **Little experimental insight**: The paper does extensive engineering in experimenting to see what sequence of training VL tasks is the best, but unfortunately an empirical and engineering contribution is not an outstanding piece of research. Particularly, there is no gained insight, not even a hypothesis, into **why** the exact sequence of VL tasks training is the best, and not another one.

- Experiment with only one NLG dataset can be biased towards the dataset.
- Table 4 has the main results of CL for VL learning of LLMs, but the results are not consistent and do not support the claims well.
- Lack of experiments about VL forgetting when fine-tuning LLMs with multiple VL datasets
- The models used in the experiments of Tables/Figures keep changing without clear reason.
Comments, suggestions and typos: L185: what does “linear” in “linear alignment network” refer to? Does this mean that the network for the alignment is linear (but how could this be, because it has two layers, and we know that 2 linear layers are as good as a single one) or is there anything else meant by the word “linear”? 

Why do you say in the introduction in the contributions (L062) you experimented with 7 MLLMs, when you experimented with 9?  

L385: broken reference

- Figure 1: Could not find description of how this experiment was performed. No discussion about this figure.
- Figure 2: The linguistic forgetting of most of LLMs is not significant except one of Pythia models. Quite different from the linguistic forgetting of Figure 1.
- Line 375: Table ??
- Line 448: Disagree. There are many exceptions.
- Line 450: Disagree. Not quite consistent results.
Overall assessment scores: 2.5

2