CogVLM: Visual Expert for Pretrained Language Models
Authors: Weihan Wang
Qingsong Lv
Wenmeng Yu
Wenyi Hong
Ji Qi
Yan Wang
Junhui Ji
Zhuoyi Yang
Lei Zhao
Song XiXuan
Jiazheng Xu
Keqin Chen
Bin Xu
Juanzi Li
Yuxiao Dong
Ming Ding
Jie Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Learning
Vision and Language
Representation Learning
Large Language Model
Primary area: machine_vision
TL;DR: We introduce CogVLM, a powerful open-source visual language foundation model.

Abstract: We introduce CogVLM, a powerful open-source visual language foundation model. Different from the popular \emph{shallow alignment} method which maps image features into the input space of language model, CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. As a result, CogVLM enables a deep fusion of vision language features without sacrificing any performance on NLP tasks. CogVLM-17B achieves state-of-the-art performance on 17 classic cross-modal benchmarks, including 1) image captioning datasets: NoCaps, Flicker30k, 2) VQA datasets: OKVQA, TextVQA, OCRVQA, ScienceQA, 3) LVLM benchmarks: MM-Vet, MMBench, SEED-Bench, LLaVABench, POPE, MMMU, MathVista, 4) visual grounding datasets: RefCOCO, RefCOCO+, RefCOCOg, Visual7W. Codes and checkpoints are available at Github.

CogVLM Visual Expert for Pretrained Language Models
Weihan Wang"l.2, Qingsong Lv Wenmeng Yu', Wenyi Hong 12, Ji Qi Yan Wang', Junhui Ji', Zhuoyi Yang! 1.2 Lei Zhao', Xixuan Song 12 Jiazheng Xu Keqin Chen', Bin Xu², Juanzi Li,, Yuxiao Dong 2 Ming "ing"",Jic Tang
'Zhipu Al Tsinghua University
ming wwng@ziipuai.ccom
{yuxiaod, jjstang edu.
Abstract
We introduce CogVLM, a powerful open-source visual language foundation model. Different from the popular shallow alignment method which maps image features into the input space of language model, VLM bridges the gap between the frozen pretrained language model and image encoder by trainable visual expert module in the attention and FFN layers. As result, Cog VLM enables deep fusion vision language features without sacrificing any performance on NLP tasks. CogVLM-17B achieves state-of-the-art performance 15 classic cross- modal benchmarks, including 1) image captioning datasets: NoCaps, Flicker30k, 2) VQA datasets: OKVQA, ScienceQA, 3) LVLM benchmarks: MM-Vet, MMBench, SRED-BBnch, L.LaV,Benc,, POPE, MMMU, MathVista 4) visual grounding datasets: RefCOCo, RefCOCO+, RefCOCOg, Visual7W. Codes and checkpoints are available at Github.
Introduction
Vision language models are versatile and powerful. Many vision and cross cooss-modllity tasks can be formulated as next token prediction, e.g., image captioning (Agraaal al. [2019), visual question answering (Antol et al. 2015), visual grounding (Yu et al. 2016)] and even segmentation (Chen et al. 2022a]. Useful abilities like in-context learning (Tsimpoukelli et al. 202 Sun et al. 2023b Alayrac ct alal]]]00 [2022] also emerge along with the improvement of downstream tasks when scaling up VLMs. However, to train large language model is already non-trivial, and it is more challenging train a VLM from scratch with the same NLP performance as well-trained pure language models like
LLaMA2 Touvron al. 2023 Therefore, it is natural to investigate how to train VLM from an off-the-shelf pretrained language model.
The popular shallow alignment methods represented by InstructBLIP [Li et al. 2023b) and MiniGPT-
4 [Zhu et al.) 2023) connect a frozen pretrained vision encoder and language model via trainable Q-Former or linear layer, mapping the image features into the input embedding space of the language model. This method converges rapidly, but its performance noticeably inferior to that of LLA with trainable language parameters, despite their model sizes and training datasets being almost identical.
The primary challenge in the performance of shallow alignment methods within VLMs can be attributed to the lack of deep fusion between visual and linguistic data, Shallow alignment methods struggle because they rely on 'frozen' language model weights, which are intrinsically trained to
WW & QI. contributed equally.
"ooresponndig authors Work done when WW, JH, ZY, XS, JX and KC interned at Zhipu AI.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Description
Chart Unberstaning
Send Me
What difaranz snd inko: Amrreci sky sigs
incad
pecortage of 1i60. The anatter hur reprosers Castuda peseange pacseaage
Whand Mask
Tour désinageiz
deaal
CogVLMM: tice
"Tighe on X ubou lipinng duck, Zack siad Hent Me LOCATION: tuly graceeds otfeed Colosceen nut
CagYEMt: the emge showasos nalvey philcrm lrncdd
exaraiay foex eagas the platfors ther
Mattion 2. A paen mas is appeeack sids raor
Referring Exprestion Comprehension
spacc"
will ligh?
the bukkhord sig plation that mekk: Ploso Use the Fonthodge Cross The us Youe Regair Aaceracc tackgread, ducck
QC: boyee teas dues zaily inaccing?
Plase
Lat becl on the ccecer the tece, il searo Suck sally mavalngg potseding
Brassning
Meme Understanding
Grounding Visual Question Answering
Qt: Whs: cobe
tanb are baking Bovan Lec's dank vay dap.
Fooo
uadyoe why disqures sosraees soll'
CagitAA read fower HE SOLSS Cleck gei gei
U Cas enalia stannt
CupYLM:
serecal donati countaad bhe caddy of do touge The peppeticc
tous
Coghil lik
with
balsen laktu up
dosta 1154,000
in luss silo offeres The wisn sos ficionl mocting of vanoo eyyitcal arcalmms, oids a ztique coapaornul lpps mmacc. sxproiei the lrage. Pallkdyy
vichi mcacially the with glow fiom the hillow and the calt bautiges mraks ware remtnng aimopticns the Agtt
bocaing surgec akeeben denentet
slll
dide afficia li otatneet ofte made by plane aucaiiaa comppadds builts drogs.
macting. piskees hemoreiiy repecient cifince oik offce cocane sugaetin that drag sange undeeraidd offeca.
Grumnding with Caption
Can pervid decr plan lmage and you'
CogyLMM pige 11
lnc tokiing Ereend task sug.
02: What ju whoon
Knuwlagge
K2noE 9671) (ast 1942 525-763 200 385.577 217 200 20u www.s//............. a22,099).
and grewery
Chock yuan (100001 201 (yees IJ's ainis soagel gey. wnsver gryy
2018 Cap?
Pengaas tams
pertigase and in His Worki Cae the ione Work
Oup Oridaoo for woud gad.
Figure Samples generated by CogVI.M.
process text tokens. This presents significant mismatch issue, as visual features lack a direct equivalent in the textual input space. Consequently, when these visual features undergo multi-layer transformations, they tend to deviate from the expected input distribution of the deeper language model layers. This misalignment is particularly evident in tasks like image captioning, where the specificity of a task such writing style and caption length can only be superficially encoded into visual features through shallow methods.
A common strategy, as seen in PaLI (Chen et al. 2022b] and Qwen-VL Bai ct al. 2023]. involves direct training of LLM during the pre-training supervised fine-tuning (SFT) phase. However, this approach can compromise the models' generalizability, particularly for tasks focused on textual outputs. Conventionally. LLMs are pretrained on extensive text-only datasets [Raffel al. [2020). leading to a significant divergence in data distribution when compared to image-text pair datasets like LAION [Schuhmann al.) 2022] and COYO Byeon al. 2022]. This shift often results in catastrophic forgetting, a phenomenon where the model's profciencyy in its original domain deteriorates. This issue is evident in Figure 2, which shows marked decline MMLU (Hendryckss) et al. 2020 score the model becomes attuned to the LAION dataset, thus validating

---

hypothesis. This trend is not isolated: similar effects have been observed in models like PaLM- [Driess al. [023] and Flamingo [Alayrac al. 2022]. For instance, adapting an 8B parameter language model for VI.M pretraining can lead to an 87.3% reduction in natural language generation (NLG) performance [Driess et al. 2023
MMLU Szor
46-
Trainig Loss
288
24
MMLU Score
23
Training Loss
22
1000
2000
3000
4000
5000
Training Steps
Figure 2: MMLU score and training loss over multimodal pretraining phase. When directly training the language part of the VLM using the LAION dataset, the model's 's score on the pure text dataset MMLU rapidly decreases, dropping 24.9 at 2500 steps.
The discussion above raises an important question: is it possible to retain the NLP capabilities of the large language model while adding top-notch visual understanding abilities it?
VLM gives a "yes" answer. CogVLM instead adds a trainable visual expert to the language model. In each layer, the image features in the sequence use the text features. Visual expert doubles the number of parameters while keeping the FL.OPs the same. Since all the parameters in the original language model are fixed, the behaviors are the same as in the original language model if the input sequence contains no image. This inspiration arises from the comparison between P-Tuning (Liu et al. 2023T] and LoRA (Hu al. [021] efficient finetuning, where p-tuning learns task prefix embedding in the input while LoRA adapts the model weights in each layer via low-rank matrix. As result, LoRA performs better and more stable. A similar phenomenon might also exist in VLM, because in the shallow alignment methods, the image features act like the prefix embedding in P- P-Tuning.
QKV matrix and MLP layer with
Our contributions in this work are as follows:
We introduce the VLM model, which deeply integrates visual and linguistic features while retaining the full capabilities of a pretrained large language model. CogVLM-177, trained from Vicuna-7B. 7B, achieves state-of -hettrrrrt across 17 classic cross-modal benchmarks.
Through extensive ablation studies, we validated the effectiveness of our proposed visual expert module and the importance of deep fusion. We further delved into multiple critical factors in multimodal pertaining, including the scale of visual encoder, variants of attention mask, the most impactful parameters in VLMs, and necessity of incorporating self- supervised image loss, etc.
We have made the weights of Cog and the dataset used in the SFT phase available to the public. We anticipate that the open sourcing of VL. will significantly contribute to the research and industrial application of visual understanding.
3

---

Position ids for RoPE
FFN
FFN
Test features
Concat
Image features
LoyerNorm
Image feature sequence] ]
MLP Adapter
TTxt feature sequence
Muzi-
VT encoder
Word embedding
Q
K
QKV matrix
QKV matrix
A fox is sitting on the books.
Image features
LayerNorm
Patchifled images
Paired Text
(a) The input of visual language model
(b) The visual expert built on the language model
CogVLM. (a) The illustration about the input, where an image is processed by a pretrained VIT and mapped into the same space the text features. (b) The Transformer block in the language model. The image features have different QKV matrix and FFN. Only the purple parts trainable.
Figure The architecture
Method
2.1 Architecture
Cog VLM model comprises four fundamental components: vision transformer (ViT) encoder, MLP adapter, a pretrained large language model (GPT), and a visual expert module. Figure (ssurs shows an overview of the CLM architecture. The components' design and implementation details provided below:
VIT encoder. We utilize pretrained EVA2-CLIP-E [Sun al. 2023a in CogVLM-17B, Note that the final layer of ViT encoder is removed because it specializes in aggregating the [CLS) features for contrastive learning.
ML.P adapter. map the output of ViT into the same space the text features from word embed- ding, we use an MIP adapter, two-layer MLP (SwiGLU (Shazeer] (020)). For implementation convenience, all image features share the same position id in the language model.
Pretrained large language model. CogVLM's model design is compatible with any off-the-shelf GPT-style pretrained large language model. Specifically, CogVLM-17B adopts Vicuna [Chiang et al. 2023] for further training. A causal mask is applied to all the attention operations, including the attention between image features.
Visual expert module. We add visual expert module to each layer to enable deep visual-language feature alignment. Specifically, the visual expert module in each layer consists of QKV matrix and an MLP in each layer. The shapes of the QKV matrix and MLP are identical to those in the pretrained language model and initialized from them. The motivation is that each attention head in the language model captures a certain aspect of semantic information, while a trainable visual expert can transform the image features align with the different heads, therefore enabling deep fusion.
Formally, suppose that the input hidden states of an attention layer are X E BX D where the batch size, L, and LT are the lengths of image and text sequences, is the number of attention heads, and D is the hidden size. In the attention with visual expert, X is first split as image hidden states X, and hidden states Xr, and the attention computed as:
Tril(QK VD
Attention W7, WT) softmax
V
(I)

---

Q = concat ....W.))
(2)
K Noncat(XX,W)) Xr.
(3)
V concat (X .x,
(4)
where Wr, WT are the QKV matrices of the visual expert and original language model, and Tril(.) means lower-triangular mask. The visual expert in FFN layers performs similarly,
(XN concat(FFN, (Xr), (FNT(XT))
(5)
where FFN, and FFN, are the FFN of the visual expert and original language model.
Position embedding. In the RoPE within LLM, allow all visual tokens to share single position id, as they already encapsulate positional information when inputted into the ViT. This approach mitigates the impact of remote attenuation between tokens in the LLM. Given that image can occupy hundreds to thousands of tokens, and a typical input sequence is structured as 'cimage embed> query', using conventional positional encoding would result in excessively lengthy encoding sequences, Moreover, it would lead the query to focus more on the image sequences closer to it, namely the lower part of an image.
2.2 Pretraining
Data. The image-text pairs for pretraining are all publicly available, including LAION-2B and COYO-700M. After removing the broken URLs, NSFW images, images with noisy captions, images with political bias and images with an aspect ratio > or 1/6, about 1.5B images are left for pretraining.
We also crafted a visual grounding dataset of 40M images. Each noun in the image caption is associated with bounding boxes to indicate the positions in the image. The construction process basically follows [Peng et al.), which extracts nouns via spaCy (Honnibal and Johnson 2015) and predicts the bounding boxes using GLIPv2 (Zhang et al. 2022]. The image-text pairs are sampled from LAION- 115M, a subset of LAION-400M filtered by [Li et al. 2023b]. We filter and retain a subset of 40 million images to ensure that over 75% of images contain at least two bounding boxes.
Training. The first stage of pretraining is for image captioning loss, i.e. next token prediction in the text part. We train the CogVLM-11 LLM-17B model on the 1.5B image-text pairs introduced above for 120,000 iterations with batch size of 8,192. The second stage of pretraining is mixture of image captioning and Referring Expression Comprehension (REC). REC is a task to predict the bounding box in the image given the text description of an object, which is trained the form of VQA, i.e., Question: Where is the object? and Answer: [[zo, yo, T1, 31]]. Both z and y coordinates range from 000 999, meaning the normalized position in the image. We only consider the loss of the next token prediction in the "Answer" part. We pretrain the second stage for 60,000 iterations with a batch size 1,024 on the text-image pairs and visual grounding datasets introduced above. During the final 30,000 iterations, we change the input resolution from 224 x 224 to 490 490. The total number of trainable parameters 6.5BB
2.3 Alignment
In the instruction alignment phase, we trained two generalist models: CogVLM-Chat and CogVLM- Grounding. CogVLM-Char accepts natural language inputs and outputs, while TogVLM-Grounng accepts inputs and outputs with bounding boxes.
CogVLM-Chat. In our study, we integrated data from variety of open-source visual question- answering datasets, including VQAv2 [Antol et al. 2015). OKVQA [Marino et al. 2019].
TextVQA (Singh et al.) 2019]. OCRVQA [Mishra et al. 2019) ScienceQA [Lu et al.) 2022 as well as datasets formatted as multi-turn dialogues such as Innstrutt LLu et al. 2023c
LRV-Instruction [Liu et al., 2023a LLaVAR Zhang et al. 2023b]. We then conducted unified instruction-sun sureiviiie fine-tuning (SFT) across these diverse datasets. The integrity and quality of SFT data crucial; notably, the ALaVA-Instruu dataset, initially generated through a language-only GPT-4 pipeline, contained certain inaccuracies. We meticulously corrected these errors through manual inspection and annotation to ensure data quality.
VQA datasets typically feature concise, often one-word answers, contrasting with the dialogue datascts that provide detailed responses with extensive reasoning. To accommodate this variability,
5

---

Table 1: Performance Image Captioning benchmarks. All tasks CIDEr as the evaluation metric. OOD refers to out-of-domain test set. Karp. refers to the Karpathy test split.
Method
Train Data
NoCaps val
NoCaps test
Fliekr
COCO
TextCaps
OOD
overall
OOD
overall
Karp.
Karp.
rest
Human
95.7
87.1
91.6
85,3
125.1
VinVL. Zhang et al.
2021
8.9M
83.8
94,3
78.0
92.5
130.8
Wang ala
115.2
112.2
109.5
CoCa
Yu
2022
4.8B
122.4
120.6
143.6
LEMON
el 20/22
20
120.2
117.3
110.1
114.3
139.1
Flamingo
Alayya 0
2022
2.3B
67.2
138.1
Prismer
L
20255
12.7M
113.5
112.9
110.8
136.5
BLIP-2
Li ct 20225
129M
124.8
121.6
144.5
Instruc13I.PP
Dii ml
2023
129M
123.1
82.4
UniversalCap
Corma el
35M
123.4
114.3
GIT [Wang el
20224
0.8B
127.1
125.5
122.0
123.4
49.6
144.8
138.2
GIT2
Wang et al 202e5
12.9B
130.6
126.9
122.3
124.8
50.7
145.0
145.0
Qwen-
Bai a A026
1.48
121.4
PaLI-17B
Clen
2022b
1668
127.0
149.1
135.4
Pal.1-X-558
Chan
2023b
126.3
124.3
149.2
147.0
CogVI.M (ours)
1.5B
132.6
128.3
128.0
126.4
94.9
148.7
144.9
we employed prompts formatted as Question: Short answer: for concise responses and Question: Answer: for extended discourse in the SFT phase.
During training, the model underwent 6000 iterations with learning rate of le-5 and batch size of 1024. To enhance and ensure the stability of the training, we activated the visual encoder's parameters and adjusted its learning rate be one-tenth of that used for the remaining training parameters.
Cog MogVLL-Grounding order to endow our model with consistent, interactive visual grounding capabilities, we collect high-quality dataset covering 4 types of grounding data: (1) Grounded Captioning (GC) image captioning datasets where each noun phrase within the caption is followed by the corresponding referential bounding boxes; (2) Referring Expression Generation (REG)- image-oriented datasets that each bounding box in the image is annotated with a descriptive textual expression that accurately characterizes and refers to the content within the specific region; (3)
Referring Expression Comprehension (REC) text-oriented datasets that each textual description is annotated with multiple referential links associating the phrases with corresponding boxes; (4)
Grounded Visual Question Answering (GroundedVQAa QA)) VQA-style datasets where the questions may contain region references in given image. The sources of grounding data all publicly available, including Flickr30K Entities [Plummer et al. 2015), RefCOCO (Kazemzadeh et al. 2014
Mao al. 2016 Yu et al. [016). Visual7W [Zhu et al. 20166. VisualGenome [Krishna al. 2017 and Grounded CoT-VQA Chen et al. 2023a] [bor] in this section is in the format of [ Mo. 1 v1l.
It is noteworthy that the curated datasets exhibit a versatility of visual grounding capabilities, and many datasets can be adapted and repurposed across different tasks. For instance, grounded captioning datasets can be reformulated to suit REG and REC tasks. Taking the example of "A man [bor]] and a woman [bors] are walking together.", ", this can be reframed into question answering pairs like ("Describe this region [boor))." ", "A woman.") ") and ("Where the man?". "[bozi] ]"). Similarly, REC datasets can be translated into REG tasks by switching the input and output, and vice versa. However, certain conversions might lead ambiguities. For example, when presented with the isolated query "Where is another man?" from the caption "A man [box1] is running, while another man [bozz] is looking.", " the distinction between [boz]] and [boz2] becomes unclear, potentially leading to errors.
3 Experiments
To rigorously validate the superior performance and robust generalization of our base model, we conduct quantitative evaluations on an array of multi-modal benchmarks. These benchmarks can be categorized into three broad areas covering comprehensive range of measuremen [
Image Captioning. The main purpose of these tasks to generate textual captions sum- marizing the major content of given image. We utilize prominent datasets including
'etailed summary of all benchmarks and corresponding metrics available at Appendix A.2

---

Table 2: Generalist performance VQA and LVLM benchmarks. donates the dataset has been trained during SFT stage. We compared with the latest state-of tae--of-thhttrr generalist models, including
MiniGPT-4 Zhu et al. 2023), IDEFICS-Instrree InstructBLIP [Dai et al.
(Laurençon et al.) 2023]. OpenFlamingo (Awadalla
et al.
2023 DreamLLM [Dong et al. 2023).
2023J. Fuyu (Bavishi et al.
2023 Owen-VL. [Bai al., 2023 L AV-11. 1.5 (Liu et al. 2023b InternI.M-XCom M--XCMppossr
Zhang
et al. 2023a mPLUG-Ow12
Ye et al. 2023].
SPHINX Lin et al. 2023b]. Emu2
(Sun et al. 2023b
VQA
LVLM-Benchmarkk
Method
LLM
VQAv2 OKVQA ScienceQA MM SEED MMBench LLaVA POPE MMMU Math Vista
MiniGPT-4
Vicuna-7B
39.6
22.1
47.4
23.0
45.1
23.1
IDEFICS-Instruct
LLaMA-65B
37.4
36.9
61.8
39.7
53.2
54.5
56.9
26.2
OpenFlamingo
MPT-7B
53.0
38.3
44.8
24.8
42.7
5.7
34.2
26.3
18.6
DLeamL.LM
Vicuna-7B
56.6
44.3
35,9
49.9
InstruciBLIP
Vicuna-7B
60.5
26.2
58.8
33,9
59.8
53.8
25.3
Fuyu
Fuyu-8B
74.2
60.6*
27.4
Qwen-VL-Chat
Qwen-7B
78.2
56.6*
68.8
65.4
61.8
67.7
32.9
33.8
LLaVA-1.5
Vicuna-7B
78.5*
66.8
30.5
58.6
64.3
60.7
85.9
23.6
Internl MM-Coomposer
Internl. M-7B
35.2
66.9
74.4
29.8
mPLUG-Ow12
LLaMA2-7B
79.4*
57.7*
36.2
64.1
64.5
25.0
86.2
32.1
25.3
Unified-1O2
UIO-2XXL
79.4*
55.5*
86,2*
65.6
71.5
87.7
LLaVA-1.5
Vicuna-138
80.0*
71.6
35.4
67.7
64.6
85,9
33.6
26.1
SPHINX-2K
LLaMA2
80.7*
62.6*
32,9
Emu2-Chat
a33MA333
CogVLM- VLtChaat
Vicuna-7B
82,3*
64,8*
91.21
51.1
72.5
77.6
27.8
87.9
41.1
34.5
CogVLM LogVLM-Chal Chat
LLaMA3-8B
83.4*
64.1*
92.5*
60.4
75.9
80.5
86.4
88.2
44.3
38.1
NoCaps [Agrawal et al. 2 COCO (Lin et al. 2014). Flickr30K (Plummer et al. and TextCaps [Sidorov et al., 2020 for evaluation.
2015
Visual Question Answering. The VQA tasks require models to answer questions that may focus on distinct visual contents based on the given image. Our assessment covers diverse datasets, including VQAv2 Antol et al. 2015). OKVQA [Marino et al. 2019 and
ScienceQA (Lu et al. 2022].
LVLM Benchmarks. LVLM benchmarks are primarily employed to assess the advanced capabilities of large multimodal models, such as object recognition and localization, OCR, visual description, and visual knowledge reasoning. We conduct multidimensional evalua- tions of the models on datasets including MM-Vet (Yu al. 2023]. MMBench [Liu et al. 2023g SEED-Bench [Li et al. 2023a VA-BBhB Liu et al. 2023c POPE Li et al.
2023c
MMMU Yue al. 2023 and Math Vista [Lu al.) 2023
Visual Grounding. Visual grounding involves a set of tasks that establish referential links
between textual mentions in sentence and specific regions in an image. We evaluate our model on the typical datasets, including Visual7w [Zhu et al.) 2016), RefCOCO [Liu et al.,
2017), RefCOCO+, and RefCOCOg to ensure completeness.
3.1 Image Captioning
We evaluate the image captioning capability of our pretrained base model on the aforementioned four benchmarks. In a zero-shot evaluation on the Nocaps and Flickr datasets, we assess the precision of our model in describing tail visual concepts. Additionally, we present results from finetuning on the COCO and TextCaps datasets.
The detailed performance is shown in Table I Overall, our model achieves the SOTA or compatible performance across the board. Specifically, on the NoCaps benchmark, our base model outperforms the previous best method, GIT2, across four splits with a maximum of 5.7 points in the out-domain set while only consuming 10% of the pretraining data (1.5B vs 12.9B). On the Flickr benchmark. our model achieves a SOTA score of 94.9 surpassing the concurrently released Qwen- VL model by 9.1 points. These results demonstrate the remarkable capability and robustness our pretrained model on the image captioning task. We also evaluate our model on the COCO [Lin et al.) 2014) and TextCaps, where the latter specifically designed to integrate the textual information of the given image into captions. Though training without the dedicated OCR data, encouragingly, our base model
7

---

reveals a significant text-reading ability and obtains a competitive performance with PaLI-X-55B, and outperforms the previous best model of the same scale, PaLI-17B, by 9.1 points score.
Table 3: Results on Referring Expression Comprehension and Grounded Visual Question Answering.
Type
Model
RefCOCO
ReCCOCO+
RefCOCOg
Visual7W
val
Teal-A
test-B
val
Test-A
test-B
val
test
test
OFA-L* WWag et
202.2
79,96
83.67
76.39
68.29
76.00
61.75
67.57
67.58
VisionL MM-H
Wamper et au:
2023b
86.70
Shikm-7B
Chein et mi M8746
87.01
90.61
80.24
81.60
87.36
72.12
82.27
82.19
Generalist
Shikra-13B
20254
87.83
91.11
81.81
82.89
87.79
74.41
82.64
83.16
85.33
Qwea-
Bau al. 2023
89.36
92.26
85.34
83.12
88.25
77.21
85.58
85.48
Ferret
Yourel
84.36
75.17
CogVL.M-Crounddng
92.76
94.75
88,99
88.68
92.91
83,39
89,75
90.79
91.05
[ Ju G-DINO-.
2023e
90.56
93.19
88.24
82.75
88.95
75.92
86. 13
87.02
Specialist
UNINEXT-H
92.64
94.33
85.24
63
79.79
88.73
89.37
ONE, PEACE
Wang al. 20231a
92.58
94.18
89.26
88.77
92.21
83,23
89.22
89.27
3.2 Visual Question Answering
As illustrated Table 2
our CogVLM model demonstrates outstanding performance and a sig- nificant lead over models of similar parameter scale across variety of tasks, including daily-life image question-answeet Vex and knowledge-demanding datasets like OKVQA and ScienceQA. This success showeases the model's robust generalization capabilities and potential across diverse domains.
dataset VQAv2, text-intensive image question-answering datasets such as
3.3 LVLM Benchmarks
Our findings, detailed in Table demonstrate that VLM achieved state-of-the-art results in all 7 .VL.M-benchmarks, markedly surpassing all other models. It also outperformed multimodal models that utilized larger language models, such as LLava with Vicuna- 13B and Emu-2 with LLAMA- 33B, leading by 15.7 and 2.6 points on MM-vet, 9.9 and 14.0 points on MMBench, respectively. Compared to EDEFICS-Instrue and 20.9 points on Seed-Bench MMBench, and LLaVA-Bench respectively. Furthermore, CLM achieved a score of 41.1 on the MMMU dataset, and also scored 87.9 on the hallucination assessment dataset POPE, along with 35.2 on the multimodal mathematical reasoning benchmark Math Vista. These impressive results not only showcase its robust reasoning abilities and multi-task generalization capabilities but also clearly demonstrate that CogVI.N is significantly outpacing other models in these domains. Notably, shallow fusion models such as InstructBLII and MiniGPT-4 underperformed across most benchmarks, despite InstructBIII's extensive training on instructional data, underscoring the necessity of deep fusion for enhanced performance.
trained on LLaMA-65B, CogVLM' scores exceeded by 19.3, 23.1,
After using a stronger and larger LLaMA-3 language model the backbone, our model achieved significant improvements on all benchmarks, fully demonstrating the robustness of our proposed method. The experimental results using other language models as backbones can be found in Appendix C.
Table 4: Ablation studies for various components and training settings. VE refers visual expert.
Abluted Aspects
Original Setting
Ablated Setting
Trainable params
COCO CIDExt
OKVQA top1
TestVQ toplf
VQAr2
CIDErt
top1t
VE-fult every layer
LLM+MLP Adapter
MLP Adapter
140M
131.2
140.3
111.5
55.1
40.7
73.8
Tuned paramcters
MLP Adapter
VE-fuli every 4th layer
VE-FFN layer
4.48
1.78
140.0
138.7
118.7
117.4
58.2
58.9
44.1
45.1
77.6
78.6
Init method
From LLM
Random init
6,68
138.0
117.9
55.9
44.0
79.1
Visual ottention mask
Causal musk
Full mask
6,68
141.0
117.2
57.4
45.1
79.6
Image SSI, loss
V (chip feature)
6,68
142.9
1198
58.7
45.9
79.7
Visual encoder
EVA2-E
EVA2-L
141.4
59.2
79.0
X
79.4
CogVLA (ours)
6.6B
142.8
120.1
59.3
45.3
$0.0
8

---

3.4 Visual Grounding
Table 3 shows the result on the standard visual grounding benchmarks. We find that our generalist model achieves state-of-the-art performance across the board, with significant advantage over the previous or concurrent models. As shown in the bottom part of [ our model even surpasses models that are specifically trained for individual tasks, achieving SOTA performance on 5 of 9 splits. For instance, in the ReiCOCO val subset, our model attains a score of 92.76, surpassing UNINEXT- H's 92.64; in the RefCOCO+ test-A subset, it scores 92.91, exceeding ONE-PEACE's 92.21; and in the RefCOCOg test subset, it achieves 90.79, outperforming UNINEXT-H's 89.27. These results suggest remarkable visual grounding capability of our model incorporating our training paradigm.
3.5 Ablation Study
To understand the impact of various components and settings our model's performance, we conduct an extensive ablation study for 6,000 iterations and a batch size of 8,192. Table 4 summarizes the results about the following aspects:
Model structure and tuned parameters. To investigate the effectiveness of CogVLM's model, we conduct ablation studies on several structure variants and tuning strategies, including: 1) tuning only the MLP Adapter layer: 2) tuning all LLM parameters and the Adapter without adding visual expert; 3) only adding visual expert at every 4th LLM layer; and 4) only add visual expert to FFNs at all layers.
From the results, we can see that shallow vision-language alignment, i.e. only tuning the adapter layer (similar the method used in BLIP-2), results in a significantly inferior performance. Also, the performance of training the visual expert is higher than that of training the LLM, especially on the datasets that require external knowledge, even though the training parameters are roughly the same. We also compare with other variants of adding visual expert, including a. inserting an expert module every 4 layers and b. removing the attention part from the expert. Both of them result in a certain degree of performance decline, but within an acceptable range, which provides some guidance for balancing computational overhead and model performance.
Initialization Method. As for visual expert's initialization method, we compare initialization with weights from LLM to random initialization. Our results across various datasets demonstrate that initialization with LLM's weights consistently achieves superior performance. This indicates that the transformer architecture pre-trained language data possesses a certain capability process visual tokens. Moreover, it can serve as a more effective starting point for multimodal pre-training initialization.
Visual Attention Mask. We empirically find that using a causal mask on visual tokens yields a better result in comparison with full mask. This is slightly counterintuitive, using bidirectional attention mask allows access to more information than a causal mask. We hypothesize the possible explanation for this phenomenon is that the causal mask better fits the inherent structure of LLMs.
Image SSL Loss. We also investigated the self- self-supervised learning loss on image features, where each visual feature predicts the CLIP feature of the next position for visual self-supervision. Align with the observation from PaLI-X [Chen el al., 2023b), we find brings no improvement on downstream tasks, although indeed observed improvements in small models in early experiments.
Visual Encoder.
we substituted the 300M-pam -prammmeee EVA2-L model for the 4.4B-parameter EVA2-E to investigate the impact of visual encoder parameters on various tasks. The results indicated that there was only slight decrease performance across most benchmarks. However, a notable exception was observed in the text-oriented dataset VeA, where we recorded a decline of 2.5.
EMA. We utilize EMA (Exponential Moving Average) during pretraining. The ablation results show that EMA often brings improvements across various tasks compared to not using it.
4 Conclusion
In this paper, we introduce CL.M an open visual language foundation model. Cog VLM shifts the paradigm for VLM training from shallow alignment to deep fusion, achieving ofate-of-the-art performance on 15 classic multi-modal benchmarks.

---

The VLM training is still in its infancy, and there are many directions to explore, for example, better SFT alignment, RLHF and amiihhallucination. Since the previous famous VLMs are mostly closed-source, we believe Cog VLM will be a solid foundation for future multi-modal research.
Acknowledgments
This work is supported by the Natural Science Foundation of China NSFC 62276148 and 62425601, a research fund from Zhipu, New Comerstone Science Foundation through the XPLORER PRIZE and Daimler Greater China Ltd. and Tsinghua University Joint Institute for Sustainable Mobility, National Engineering Laboratory for Cyherlearning and Intelligent Technology, and Beijing Key Lab of Networked Multimedia.
A Appendix
A.1 Details of Training Settings
We report the details of parameter settings during pre-training and multitask training in Table 5 and
Table 6
Table 5: Hyperparameters for pre- pre-training model.
Hyperparameters
Stage 1
Stage 2
Total steps
120, 000
600000
Warmup steps
12, 000
1,200
Batch size
8, 192
1,42
Learning rate
1e-4
1e-5
Learning rate decay
Cosine
Weight decay
0.05
Dropout ratio
0.1
Adam
1e-8
Adam /
(0.9, 0.95)
Textual encoder
Vicuna-1.55 115-7B
Visual encoder
EVA2-CLIP-E
Patch size
14
Input resolution
224
224 > 4902
Table 6: Hyperparameters for multitask finetuning CLg
Hyperparameters
Multitask
Learning rate
1e-5
Total steps
6,000
Batch size
1,024
Adam W
1e-8
AdamW W 8
(0.9, 0.95)
Weight decay
0.1
Dropout ratio
0.1
Input resolution
4902
A.2 Details of Associated Datasets
In this section, we introduce the details of datasets and their use our evaluation process for all associated benchmarks.
10

---

Table 7: Summary of the evaluation benchmarks.
Task
Dataset
Deseription
Split
Metries
NoCaps
Captioning of natural images.
val
CIDtt (t)
Image Caption
Flickr
Captioning natural images.
karpatty-
CIDEr (1)
COCO
Captioning natural images.
karpatly-test
CIDE (t)
TexeCaps
Captioning natural images containing text.
test
CIDEr (1)
VQA+2
VQA on natural images.
test-dev
VQA Score(t)
OK-VQA
VQA natural images requiring outside knowledge.
VQA Soote
General VQA
ScienceQA
Multi-choice VQA on a diverse set of science topics
test
Accuracy (1)
TDIUC
VQA natural images with detailed question types.
VQA Soote
MM. Vet
Open-ended ended VQA on diverse set of topics
test
GPT4 Score(1)
SEED-Bens Bench
Mulii-choice VQA a diverse set of lopics
Accuracy
MMBeench
Multi choice VQA on a diverse set of topics
test
Accuracy (7)
LVLM Benchmarks
LLaVA -ennchh
Open-ended VQA lesting instruction following abilities
In-the
GPT4 Scure(
POPE
Multi choice VQA for testing ballacinations
overall
Accuracy (7)
VQA diverse set lopicss
lest
Accuracy
Vash
VQA for Measuring Mathematical Abilities
tost-mini
Accuracy (7)
RefCOCO
Refer grounding on natural images.
overall
Accuracy (?)
Grounding
RefCOCO+
Refer grounding natural images.
overall
Accuracy (7)
RefCOCOg
Refer grounding on natural images.
overall
Accuracy (?)
Visual7W
VQA with referential regions selection.
val
Accuracy
A.2.1 Image Captioning
COCO [Lin 2014 The Captions in COCO dataset are collected using Amazon's
Mechanical Turk (AMT) workers who are given instructions to control the quality. The dataset contains 330K images, where the train, validation and test sets contain 413,915 captions for 82,783 images, 202,520 captions for 40,504 images, and 379, captions for 40,775 images respectively.
NoCaps (Agraall et al.) 2019]. NoCaps is large-scale benchmark for novel object captioning, containing nearly 400 novel object classes compared to COCO. The validation and test set comprised of 4,500 and 10,600 images, respectively, sourced from the Open Images [Kassin et al.) 2017] and annotated with 11 human-generater captions per image, and each set is subdivided into three domains: "in", "near", and "out", with objects in the "out-domain" never appearing in the COCO dataset.
Flickr30K [Plummer et al. 2015). Flickr30K is high-quality dataset consists of 31,783 images of everyday life activities, envets and scenes (all harvested from the online website Flickr) and 158,915 captions (obtained via crodsourcing). Each image in this dataset is described independently by five annotators who are not familiar with the specific entities and circumstances depicted them.
TextCaps [Sidorov al. 2020) Textcaps a dataset with 145k captions for 28k images. The design purpose of the TextCaps dataset is to effectively integrate textual information with visual context into captions, requiring the model have both excellent OCR capabilities and strong captioning abilities.
A.2.2 General VQA
VQAv2 Antol al. 2015] VQAv2 encompasses over 200,000 images, paired with more than 1.1 million questions that have collectively garnered over 11 million answers. Questions span various types, including yes/no, counting, and open-ended queries.
OKVQA Marino et al. 2019] The OK-VQA (Outside Knowledge Visual Question An- swering) dataset is specifically designed to probe visual question answering capabilities that necessitate external knowledge or common sense beyond image content. It has 14,055 open-ended questions and 5 ground truth answers per question.
ScienceQA [Lu et al. 2022] The ScienceQA dataset comprises 21,208 multimodal multiple- choice questions spanning three diverse subjects: natural science, language science, and social science. Each question is annotated with explanations linked to relevant lectures.
TDIUC (Shrestha et al. 2019) The TDIUC dataset features 1.6M questions across 170K images from MS COCO and Visual Genome. Categorized into 12 distinct question types, it ranges from basic tasks like identifying objects or colors to more advanced reasoning like counting positional discernment.
11

---

A.3 LVLM Benchmarks
MM-Vet (Yu al. 2023
MM-Vet defines six core VL capabilities and examines 16 integrations of interest derived from the combinations of these capabilities. It employs an evaluator based on LLMs for open-ended outputs, capable of assessing across different question types and answer styles, thus deriving a unified scoring metric.
SEED-Bench [i al. 2023a SEED)-Bench a dataset comprising 19K multiple-choice questions with precise human annotations, covering 12 evaluation dimensions, including understanding of image and video modalities. It obtains accurate answer options through manual annotations, enabling objective and efficient assessment of model performance.
MMBench [Liu et al. 2023g] MMBench comprises approximately 3000 multiple-choice questions, covering 20 different capability dimensions, aimed at evaluating various abilities of visual- Lasual-anguage models. MMBench adopts a hierarchical capability dimension structure, including two high-level capability dimensions: perception and reasoning, as well as fine- grained capability dimensions such object localization and attribute inference.
(Liu et al.) 2023c] VA-nnch (In-the- is benchmark dataset com- prising 60 questions, designed to evaluate the multimodal instruction following capabilities of LMMs. It includes indoor and outdoor scenes, memes, paintings, sketches, etc., and is equipped with highly detailed, manually curated descriptions and appropriate question selections.
LLa
POPE al. 2023c The POPE dataset is binary classification query dataset specifically designed to evaluate object hallucination issues in LMMs. The random, popular, and adversarial subsets within the POPE dataset are constructed through different sampling strategies, totaling 8,910 entries.
MMMU (Yue et al.) 2023] The MMMU dataset a large-scale, multidisciplinary multi- modal understanding and reasoning benchmark set, containing 11,5K questions. It covers 6 major disciplines, topics, and 183 subfields, with question types including multiple-choice and open-eneee open-ended questions. The dataset includes 30 types of images, such as charts, tables, chemical structures, photographs, paintings, musical scores, etc., testing the multimodal perception capabilities of models and their performance in expert-level tasks.
Math Vista [Lu et al. 2023 Math Vista is a new benchmark dataset that combines mathe- matical and visual understanding, comprising 31 existing multimodal datasets and 3 newly created datasets, totaling 6141 examples. These datasets encompass a diverse range of mathematical reasoning abilities, including seven types: algebra, arithmetic, geometry. logic, numerical common sense, science, and statistics. The goal is to comprchensively evaluate the capabilities of existing foundational models in mathematical reasoning and visual understanding.
A.3.1 Grounding
RefCocO/RefCOOO+ [Liu et al. 2017 RefCOCO and RefCOCO+ evolved from the ReferltGame. Both subsets focus images with two more similar objects. RefCOCO, with 142,209 expressions across 19,994 images, places no linguistic constraints. Conversely, RefCOCO+ emphasizes appearance-centrie descriptions, omitting locational terms, and comprises 141,564 expressions over 19,992 images.
RefCOCOg Mao et al. 2016) The RefCOCOg subset was amassed through Amazon Me- chanical Turk, where workers penned natural referring expressions for objects in MSCOCO images; boasts 85,474 referring expressions spanning 26,711 images, each containing 2 to 4 objects of the same category.
Visual7W Zhu et al. 2016). The Visual7W dataset is predominantly designed for VQA tasks, with a dedicated subset crafted for grounded VQA. this subset, models are presented with image accompanied by "which"" -type question, such as "Which is the small computer in the comnrr"". Participants are then given four bounding boxes within the image, from which they must select the correct one as the answer. The grounded Visual7W part consists of 25,733 images and 188,068 questions.
Flickr30K-Ente [Plummer al., 2015). The Flickr30K Entities dataset, precursor in the realm of grounded captioning, encompasses a collection of 3,7783 images accompanied
12

---

by 158k captioning annotations, Every caption in this dataset has been meticulously anno- tated such that each noun phrase linked with manually delineated referential bounding box. In total, there are 276k such annotated bounding boxes provided within this dataset.
VisualGenome Krishna et al., 2017]. The VisualGenome dataset stands as a comeestone in understanding the multifaceted relationships present within images. With a collection of over 100k images, each image is annotated in detail, capturing an average of 21 objects, 18 attributes, and inter-object relationships. unique aspect of this dataset is the alignment of objects, attributes, relationships, and region descriptions with standardized terminologies from WordNet. Specifically tailored for the REG and REC tasks, each annotated region in image comes with a corresponding descriptive text, making rich resource for image understanding and semantic modeling. We use the subset with around 86k images and million region-caption pairs for visual grounding.
Previous SOTA
Sentiment Understanding
CogVLM-1778
Utility Affordances
Activity Recognition
Positional Reasoning
Object Recognition
Sport Recagnition.
Other Attributes
Color Attribute
Object Presence
Counting
Average
Figure 4: Performance on TDIUC benchmark with ane-ggrined questions classes.
B Additional Fine-grained Experiments
To comprehensively investigate the proposed model on specific topics and question types, we further conduct extensive experiments on a representative benchmark, TDIUC [Kafle and Kanan 2017]. We use the publicly available split of val set evaluation data, and the VQA accuracy calculated from their official scripts as the evaluation metric.
The experimental results on TDIUC compare our model against the specialist SOTA method MUREL [Cadene et al., 2019] are shown in Figure 4 From the experimental result, we can see that our model consistently outperforms the previous model on 12 specific question types, resulting in a 94.0 accuracy score compared to the previous SOTA of 88.2 on the overall dataset. These results demonstrate that our model exhibits comprehensive boobvm--oolviggeg skills on general VQA tasks.
C Alternative Language Models Results
Table 8: Comparison of different language models as backbones.
LLM
MM-Vet
OKYQA
Math Vista
MMBench
Vicuna-7B-1.5
52.0
64.8
34.5
77.6
Beeuma-13B-11
56.8
66.7
37.2
78.1
LLaMA3-8B
60.4
64.1
38.1
80.5
GLM3-32B
64.5
68.2
45.1
82.3
13

---

As shown in the Table 8 our visual expert module is integrated into the LLM, and it can significantly benefit from the scaling of the LLM. These results demonstrate that our approach can effectively leverage the benefits of LLM scaling to improve performance on multimodal tasks.
D Computational Efficiency
In this section, we compare the computational efficiency of our model with other state-of-ther ah-ttt--rt models, considering both pretraining and finetuning data from datasets such as VQAv2 and TextVQA Owing to an optimized architecture and the utilization of high-quality pretraining data, our model demonstrates marked reduction in resource consumption during training relative models with comparable parameter magnitudes.
Table Comparison of different models based on their computational efficiency. PFLOPS*day: metrics.
We use
Model
Pretraining Data
Pretraining compute
VQAv2 finetuning
TextVQA finetuning
PaLI-3B
1.6B
56
1.1
0.2
PaLI-17B
1.6B
4.5
0,9
Flamingo-80R
2.3B
1381
N/A
N/A
GIT2-5.1B
12.9B
5513*
N/A
N/A
CogVLM
1.5B
230.1
1.2
0.13
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of this paper accurately reflect the contributions and scope of the research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [NA]
Justification:
The paper has no limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper fully discloses all the information needed to reproduce the main experimental results, including training details, training parameters, and model weights, which have been made open-source.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The paper provides open access to the dataset used in the instruction fine-tuning phase and the training code, along with sufficient instructions to faithfully reproduce the main experimental results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
http://ppppbpi.p...u.cooo///o/aa htt////ooeSSSiiseeeeeeeeeee SoroodeSumisssioPPP... wwriiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all the training and test details, including data splits, hyperparameters, their selection process, the type of optimizer, etc., either the main text in the appendix, providing a comprehensive understanding of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars or other statistical significance information because the experimental results distinct enough to be interpreted without them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides sufficient information on the computer resources required to reproduce each experiment, including the type of compute workers, memory, and time of execution, in both the main text and the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Justification: The research conducted in the paper fully conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets such as code, data, and models used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thooe cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented, and the documenta- tion is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
References
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen. Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee. and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3948-8957 2019.
MannBaptistee Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308. 010.0,013,,,, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar Xrv: 12966, 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşılaa. Introducing multimodal models, 2023. URL https://wwww adept ai/ blog/fuyu-8b
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https: :tttps///gite httpa////..lllaaobaaaa//ooooeeea dataset 2022.
Remi Cadene, Hedi Ben- Youne,, Matthieu Cord, and Nicolas Thome. Murel: Multimodal relational reasoning for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1989-1998 1998, 2019.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023a.
Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David Fleet. A generalist framework for panoptic segmentation of images and videos. arXiv preprint 1xXiv:2210006366 2022a.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0677 2022b.
Xi Chen, Josip Djolonga. Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.78566 2023b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wa, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa. Imsys. org (accessed 14 April 2023), 2023.
21

---

Marcella Cornia, Lorenzo Baraldi, Giuseppe Fiameni, and Rita Cucchiara. Universal captioner: Long-tail vision-and -nd--andddaaee and-language model training through content-style separation. arXiv preprint urXiv:2111.12722) 1(2):, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards ceneral-purpose vision-language models with instruction tuning, 2023.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309. 1499, 2023.
Danny Driess, Fei Xia, Mehdi SM Sajaadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzean Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03377. 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.000 033000 2020.
Matthew Honnibal and Mark Johnson. An improved mon-montooiic transition system for dependency parsing. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1373- 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0963 2021.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. pre-training for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17980- 17989,7989, 2022.
Scaling up
Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE international conference on computer vision, pages 1965-1973, 2017.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, al. Openimages: public dataset for large-scale multi-label and multi-class image classification. Datasel available from https://github
com/openimages. 2(3):18, 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12559 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.1035. 10355, 2023c.
22

---

Fangjian Lin, Jianlong Yuan, Sitong Wu, Fan Wang, and Zhibin Wang. Uninext: Exploring a unified architecture for vision recognition. arXiv preprint arXiv:2304.13700. 2023a.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014; 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575 2023b.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arX1v:2310.03744, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint XXiv:2304.08488 2023c.
Jingyu Liu, Liang Wang, and Ming-Hsuan Yang. Referring expression generation and comprehension via attributes. In Proceedings of the IEEE International Conference on Computer Vision, pages 4856-4864, 2017.
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar, Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv:2303.02550 2023d.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0.55 050999, 2023e.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023f.
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06256 2023g.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei neiWei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 11-20, 2016.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824,
23

---

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 2022.
Noam Shazeer. Glu variants improve transformer. arXiv preprinl arXiv:2002.052000 2020.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. Answer them all! toward universal visual question answering models. In Pruceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 10472-10481, 2019.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 742-758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8317-8328 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15388 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint ar arXiv:2307.052: 2023b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Maria Tsimpoukelli. Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-211 2220000-1,2200 2021.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 2201:22005.14000 2022a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-ue sowennce-t--seuencee learning framework. In International Conference Machine Learning, pages 23318-233444 PMLR, 2022b.
Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities. arXiv preprint arXiv:2305. 2023a.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.1 11135..117..... 2023b.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.100 10904, 2021.
24

---

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li. Junfeng Tian, al. mplug-docowe Modularized multimodal large language model for document understanding. arXiv preprint urXiv:2307.02433 2023.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXhv:2310007744 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arX1v:2205.01917, 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In Computer Eision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 11 14, pages 39-85. Springer, 2016
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308 02490,, 2023.
Xiang Yuc, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint 1..Xii233111655000 2023.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, (enq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. anternlm-xcomposee A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112 15112, 2023a.
Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang. Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision- vision-language models. Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 5579-5588, 2021.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004, 2016.
25


Reviews:
[{"review_id": "ZryC3nIaOz", "reviewer": "Reviewer_nFmo", "summary": "This paper proposed a large vision language model, CogVLM, which shows capabilities in various benchmarks. In contrast to popular solutions that fuse the vision and language token in LLM with shared trainable parameters, CogVLM bridges the gap between the frozen pre-trained language models and image encoders by trainable visual expert modules in the attention and FFN layers. In this way, CogVLM excels in maintaining the language abilities of the original LLM and achieving good performance on various vision-language benchmarks, including captioning, VQA, and visual grounding.", "questions": "See weakness", "limitations": "The limitations are not discussed in this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The idea of Visual Expert is simple but effective. It is impressive that newly introduced Visual Expert Modules with large parameters can be well-optimized and perform well.  \n3. The proposed CogVLM is extensively evaluated using various benchmarks. It can be widely adopted for captioning, VQA, and Visual Grounding.", "weaknesses": "1. The visual expert will introduce a large number of parameters. As a result, the CogVLM with a 7B LLM backend will finally have 17B parameters. Although this won't lead to high computational costs in inference, the significantly increased parameters will require much more GPU memory. It would be helpful if the author could discuss whether there are alternative solutions to save memories.\n2. CogVLM adopts Visual Expert Modules to retain LLM abilities via freezing it. However, how about involving pure language data in the SFT stage, as in [1]? Whether the LLM ability can be retained this way needs to be discussed.\n3. CogVLM allows all visual tokens to share a single position ID in the RoPE. An ablation study should be reported.\n\n[1] InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thorough review and constructive feedback on our paper. Your insights are valuable for improving our work. We address each of your points below:\n\n## 1. Increased number of parameters and memory usage\n\nWe acknowledge that the Visual Expert introduces additional parameters, increasing static GPU memory usage during training and inference. This represents a trade-off between performance and computational cost. Notably, CogVLM outperforms models like LLaVA-Next, which use much larger language models, demonstrating the efficiency of our approach.\nSimilar parameter-heavy approaches have been employed in NLP models, such as DeepSeek-v2[1] (236B total parameters, 21B activated). \n\nWe explored the LoRA expert method as suggested by Reviewer 7nCy, using a LoRA rank of 128. However, we found its expressiveness limited:\n- It required 3.7 times more steps to reach the same loss level as our current method.\n- The per-step computation time was nearly identical to our current approach.\n\nThese results indicate that while LoRA expert can reduce parameter count, it may come at the cost of training efficiency.\nIn our experiments with language models up to 32B parameters, the additional memory overhead remained acceptable. For future extensions to larger language models, we will further investigate parameter reduction techniques.\n\n## 2. Retaining LLM abilities\n\nWe appreciate this excellent question. Our approach improves the model structure, while InternLM-XComposer2 focuses on data-side enhancements. These methods are not mutually exclusive. In fact, our latest experiments show that the best multimodal fusion is achieved by using Visual Expert modules, unfreezing LLM parameters, and simultaneously training on multimodal and pure text data. We will include a discussion on this in the revised paper.\n\n## 3. Shared position ID for visual tokens in RoPE\n\nThank you for raising this important point. Our positional encoding scheme addresses the \"remote attenuation effect\" in RoPE, where attention weights decrease with increasing token distance. This prevents the query from overfocusing on nearby image tokens.\n\nComparative experiments demonstrate the benefits:\n- At 224x224 resolution, our method and the original achieve the same pre-training loss.\n- At 490x490 resolution, our method achieves 5% lower loss.\n- On the DocVQA task (1120x1120 resolution), our method improves accuracy from 47.7% to 49.1%.\n\nThe concurrent work Mousi[2] observed a similar phenomenon, further validating our approach.\n\n\n[1] DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model\n\n[2] MouSi: Poly-Visual-Expert Vision-Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o4AxCzOWxu", "reviewer": "Reviewer_LKA5", "summary": "This paper aims to add multimodal capability while maintaining the language capability of LLM. The authors thus propose a feature fusion strategy, without sacrificing any performance on NLP tasks. The experimental results on multiple datasets are impressive, which demonstrate the validity of the proposed method to some extent.", "questions": "None", "limitations": "None", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The motivation of this paper sounds reasonable. Most of current VLMs do not take into account the degradation of LLM when training multimodal data, which is a good research point.\n\n2. The method looks interesting. A simple feature fusion strategy seems to tackle the above problem.\n\n3. The writing of this paper is good, and the structure is easy to follow.", "weaknesses": "The experiment still has room for improvement: \n(1) The results on the representative VLM benchmark MME are not given, which is widely used by current methods, such as LLaVA, InternVL, and Qwen-VL. It is suggested to add the results of MME in Table 2.\n(2) In order to maintain the capability of LLM, we can also add text data while using multi-modal data, is there a corresponding comparison? Some common text data might also serve this purpose.\n(3) Alternatively, we could freeze the parameters of the LLM but train the visual encoder and the connector, which might also do the trick.\n(4) It would be good to see CogVLM's ablility on the representative multimodal video benchmark Video-MME [1]. The authors could input multiple video frames, i.e., generalize to multiple images. GPT-4V, InternVL-Chat, Qwen-VL-Max, and Qwen-VL-Chat all have good generalization on Video-MME. It is suggested to add the results of CogVLM in this paper.\n\n[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis.\n\nGiven the limited time available for rebuttal, the authors could incorporate the above experimental results and discussions into the final accepted version or the next version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thorough review and insightful feedback. Your comments are invaluable in helping us improve our work. We are pleased to address each of your points below:\n\n## 1. MME Benchmark Results\n\nThank you for suggesting the inclusion of MME benchmark results. We have evaluated CogVLM using LLaMA-3 as the base model on the MME benchmark. The results are as follows:\n\n| Model | MME | OCR | artwork | celebrity | color | count | existence | landmark | position | posters | scene | perception | \n|---|-----|-----|---------|-----------|-------|-------|-----------|----------|----------|---------|-------|------------|\n|CogVLM-LLaMA3-32B| 2043.54 | 132.50 | 150.75 | 172.06 | 168.33 | 153.33 | 195.00 | 174.50 | 103.33 | 147.62 | 150.75 | 1548.18 | \n|CogVLM-GLM3-32B| 2211.76 | 155.00 | 149.75 | 155.29 | 165.00 | 175.00 | 200.00 | 182.75 | 131.67 | 185.37 | 155.50 | 1655.33 | \n\nModel | code_reasoning | commonsense_reasoning | numerical_calculation | text_translation | reasoning |\n|--       |--------|-----|------|-----|----|\n|CogVLM-LLaMA3-32B |70.00 | 132.86 | 107.50 | 185.00 | 495.36 |\n|CogVLM-GLM3-32B| 120.00 | 151.43 | 92.50 | 192.50 | 556.43 |\n\nWe will include these results in the revised version of our paper to provide a more comprehensive comparison with other state-of-the-art models.\n\n## 2. Maintaining LLM Capability with Text Data\n\nThis is an excellent point. Our method primarily focuses on improving the model structure, while approaches like VILA and InternLM-XComposer2 make improvements on the data side. These methods are not mutually exclusive.\n\nIn fact, our latest experiments show that the best multimodal fusion is achieved by:\n1. Using Visual Expert modules\n2. Unfreezing LLM parameters\n3. Simultaneously training on multimodal and pure text data\n\nWe will include a discussion on this synergistic approach in the next version of our paper, demonstrating how structural and data-based improvements can be combined for optimal performance.\n\n## 3. Freezing LLM Parameters\n\nYou're correct that this approach has been explored, notably in the BLIP-2 series of models. However, as seen in Table 2 of our paper, this method often underperforms on many benchmarks. Our experiments also indicate that freezing language model parameters can lead to issues with instruction following and multimodal fusion.\n\nWe believe that fine-tuning the language model parameters is crucial for optimal performance in multimodal tasks, which is why our approach allows for updating these parameters.\n\n## 4. Video-MME Benchmark\n\nWe agree that extending image model capabilities to video is an interesting and valuable area of exploration. We are currently conducting research in this direction and plan to evaluate our model on the Video-MME benchmark in the near future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "46aDOmffm4", "reviewer": "Reviewer_gmvD", "summary": "The paper introduces CogVLM, an new open-source visual language foundation model. Contrary to the popular method of adapting LLMs by fine-tuning their original weights, CogVLM introduces new weights specifically for processing the visual tokens. Concretely, CogVLM copies all weights of the LLM to form visual expert modules at every attention and FFN layer. During training only the visual expert weights are updated such that the LLM retains its language modeling capabilities when no image input is given. CogVLM is evaluated extensively on image captioning, VQA, LVLM and Referring Expression Comprehension benchmarks.", "questions": "I would be great if the authors could clarify my concerns from the weaknesses section.\n\nApart from that, here is one more minor suggestion.\n- Figure 1 shows interesting qualitative example. I suggest to reference the figure in the main text and extend the caption to explain the displayed capabilities.", "limitations": "Limitations and broader impact are not discussed although they certainly exist. The authors should add a discussion on both (see weaknesses).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The design choice to add additional parameters for the vision tokens is reasonable and implemented well such that it defaults to the original model without image input.\n2. The experimental evaluation is extensive. CogVLM achieves convincing results across the board.\n3. Ablations shed light on important questions such as the importance of the visual expert parameters and the causal attention mask.\n4. As an open-source model, CogVLM can have a high impact on the research community.", "weaknesses": "1. The prompting differences between the tasks is not clear. For instance, how can a user make sure that CogVLM returns bounding box information or prevent CogVLM to output bounding boxes?\n2. While the results of CogVLM are impressive, it is not a parameter efficient method. Trying to scale CogVLM to LLMs of better language modeling capacity will double the parameters which is not sustainable for big LLMs.\n3. As VLMs become more powerful over time, there will be higher expectations of their capabilities. One aspect that has not been mentioned at all is whether CogVLM can handle multi-image inputs or multi-turn conversations with interleaved image-text context. Based on the examples given and the training data, the paper currently suggests this is a model limitation.\n4. Limitations are not discussed at all. The checklist states that the paper has no limitations without explanation, which is a superficial statement and hardly ever true. The previous points are likely limitations.\n5. Similarly, the authors do not provide a discussion on the broader impact of the paper. Since CogVLM is an open model it has the potential to have a high impact, both positive and negative. Since CogVLM puts a particular effort into retaining the original LLMs performance, it also means that it carries over all biases and limitation from the base LLM. It is important that papers of generative models reflect on the societal impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thorough review and insightful feedback. Your comments have been invaluable in helping us improve our work. We address each of your points below:\n\n## 1. Prompting differences between tasks\n\nThank you for highlighting this important aspect. In our approach, we use two separate models for chat and grounding tasks. This design choice is based on the following considerations:\n\n- Using a single model wouldn't necessarily decrease performance, but some ambiguous prompts can make it difficult for the model to determine whether coordinate outputs are required. For example, \"Where is the dog in the image?\"\n- To avoid confusion, we believe it's necessary to use very clear prompts for grounding-type questions during both training and inference. For instance, \"Specify the exact coordinates of the dog in the image\" or adding task-specific prefixes like <grounding>.\n\nWe will clarify this in the revised paper to ensure users understand how to effectively prompt CogVLM for different tasks.\n\n## 2. Parameter efficiency\n\nWe acknowledge that the Visual Expert introduces additional parameters, increasing static GPU memory usage. This represents a trade-off between performance and computational cost. Notably, CogVLM outperforms models like LLaVA-Next, which use much larger language models, demonstrating the efficiency of our approach.\nSimilar parameter-heavy approaches have been employed in NLP models, such as DeepSeek-v2 (236B total parameters, 21B activated). \n\nWe explored the LoRA expert method as suggested by Reviewer 7nCy, using a LoRA rank of 128. However, we found its expressiveness limited:\n- It required 3.7 times more steps to reach the same loss level as our current method.\n- The per-step computation time was nearly identical to our current approach.\n\nThese results indicate that while LoRA can reduce parameter count, it may come at the cost of training efficiency. In our experiments with language models up to 32B parameters, the additional memory overhead remained acceptable. For future extensions to larger language models, we will continue to explore parameter reduction techniques.\n\n## 3. Multi-image inputs\n\nWe appreciate you bringing this to our attention. Our latest version of the model has been enhanced using MMC4 and OBELICS datasets to improve multi-image capabilities. We will update the results in the subsequent version of the paper to reflect these advancements.\n\n## 4. Limitations discussion\n\nWe sincerely apologize for the oversight in not discussing limitations. We will add a dedicated section on limitations in the revised paper, addressing the points raised by reviewers, including:\n- The impact of Visual Expert modules on parameter count\n- Support for multi-image inputs\n- Potential biases inherited from the base LLM\n\n## 5. Broader impact\n\nWe agree that discussing the broader impact is crucial. We will add this point in the limitations section, addressing the negative potential impacts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y8Z1tHHePY", "reviewer": "Reviewer_7nCy", "summary": "This paper introduce CogVLM, a powerful open-source visual language foundation model. CogVLM bridges the gap between the frozen pretrained language model and image encoder by a trainable visual expert module in the attention and FFN layers. The contributions are summarized:\n1. introduce the CogVLM model, which deeply integrates visual and linguistic features while retaining the full capabilities of a pretrained large language model. CogVLM-17B, trained from Vicuna-7B, achieves state-of-the-art across 17 classic cross-modal benchmarks.\n2. validated the effectiveness of our proposed visual expert module and the importance of deep fusion. \n3. the weights of CogVLM and the dataset used in the SFT phase available to the public.", "questions": "1. Why train a separate grounding model? Would it degrade the chat model if the chat model also had grounding capabilities?\n2. Why not try tuning the visual encoder?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's results have achieved state-of-the-art (SOTA) performance, demonstrating excellent capabilities across multiple benchmarks.\n2. The paper is well-organized and the expression is clear.\n3. The starting point of the paper: addressing the issue of performance degradation in LLM due to parameter updates during multimodal training, is very meaningful and valuable.", "weaknesses": "1. A new set of QKV and FFN would nearly double the number of model parameters. Why not consider using LoRA (Low-Rank Adaptation), where text tokens use QKV/FFN and image tokens use QKV+LoRA/FFN+LoRA? Additional experiments are needed to support this.\n2. There is a lack of ablation studies to compare the impact of different RoPE (Relative Positional Encoding) strategies on visual tokens on the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thorough review and insightful feedback. Your comments are invaluable in helping us improve our work. We address each of your points below:\n\n## 1. Parameter efficiency and LoRA consideration\n\nThank you for this good question. The Visual Expert indeed introduces additional parameters, increasing static GPU memory usage during training and inference. This represents a performance-cost trade-off, enabling CogVLM to outperform models like LLaVA-Next, which use much larger language models.\n\nSimilar approaches have been employed in NLP models, such as DeepSeek-v2 [1], which has 236B total parameters but only 21B activated parameters.\n\nWe explored the LoRA expert method with a LoRA rank of 128. However, we found its expressiveness limited:\n- It required 3.7 times more steps to reach the same loss level as our current method.\n- The per-step computation time was nearly identical to our current approach.\n\nThese results indicate that while LoRA can reduce parameter count, it may come at the cost of training efficiency.\nIn our experiments with language models up to 32B parameters, the additional memory overhead remained acceptable. For future extensions to larger language models, we will continue to explore parameter reduction techniques, including more efficient implementations of LoRA.\n\n## 2. Ablation studies on RoPE strategies for visual tokens\n\nWe appreciate you highlighting this important point. Our positional encoding scheme addresses the \"remote attenuation effect\" in RoPE, where attention weights decrease with increasing token distance. This prevents the query from overfocusing on nearby image tokens.\n\nComparative experiments demonstrate the benefits:\n- At 224x224 resolution, our method and the original achieve the same pre-training loss.\n- At 490x490 resolution, our method achieves 5% lower loss.\n- On the DocVQA task (1120x1120 resolution), our method improves accuracy from 47.7% to 49.1%.\n\nThe concurrent work Mousi [2] observed a similar phenomenon, further validating our approach.\n\nWe will include these results in the revised paper to provide a clearer comparison of different RoPE strategies.\n\n## 3. Separate grounding model\n\nTraining a unified model had minimal impact on overall performance. Our experiments show that simultaneously training on both grounding and chat data led to:\n- A 1.5 point decrease in the Nocaps dataset score (from 128.3 to 126.8)\n- A 0.8 point increase in the VQAv2 task score (from 82.3 to 83.1)\n\nThe main reason for using two models is to address ambiguous prompts where it's unclear whether coordinate outputs are required. For example, \"Where is the dog in the image?\" With two models, we can provide clearer instructions for grounding tasks without affecting the chat model's performance.\n\n## 4. Visual encoder tuning\n\nWe apologize for any confusion. We did indeed train the ViT parameters, as mentioned in line 152 of the original paper: \"we activated the visual encoder's parameters and adjusted its learning rate to be one-tenth of that used for the remaining training parameters.\"\n\n[1] DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model\n\n[2] MouSi: Poly-Visual-Expert Vision-Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces CogVLM, a large-scale visual language model designed to integrate visual and linguistic features while preserving the language capabilities of pre-trained language models. The approach involves adding Visual Expert modules to handle visual tokens, enabling CogVLM to achieve state-of-the-art performance across multiple benchmarks.

The ratings among reviewers are mostly positive, with many acknowledging the model's strong performance and the quality of the paper. However, concerns about parameter efficiency and model limitations remain which raises difficulties to reproduce in the academic community. Despite these concerns, the majority of reviewers support acceptance, with a few maintaining their original ratings due to unresolved issues. As a result, the ACs recommend acceptance and suggest the authors to integrate the rebuttals into the final version.
Decision: Accept (poster)