Octopus: A Multi-modal LLM with Parallel Recognition and Sequential Understanding
Authors: Chuyang Zhao
YuXin Song
Junru Chen
KANG RONG
Haocheng Feng
Gang Zhang
Shufan Ji
Jingdong Wang
Errui Ding
Yifan Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal large language model
large language model
object detection
Primary area: machine_vision

Abstract: A mainstream of Multi-modal Large Language Models (MLLMs) have two essential functions, i.e., visual recognition (e.g., grounding) and understanding (e.g., visual question answering). Presently, all these MLLMs integrate visual recognition and understanding in a same sequential manner in the LLM head, i.e., generating the response token-by-token for both recognition and understanding. We think unifying them in the same sequential manner is not optimal for two reasons: 1) parallel recognition is more efficient than sequential recognition and is actually prevailing in deep visual recognition, and 2) the recognition results can be integrated to help high-level cognition (while the current manner does not). Such motivated, this paper proposes a novel “parallel recognition → sequential understanding” framework for MLLMs. The bottom LLM layers are utilized for parallel recognition and the recognition results are relayed into the top LLM layers for sequential understanding. Specifically, parallel recognition in the bottom LLM layers is implemented via object queries, a popular mechanism in DEtection TRansformer, which we find to harmonize well with the LLM layers. Empirical studies show our MLLM named Octopus improves accuracy on popular MLLM tasks and is up to 5× faster on visual grounding tasks.

Octopus: A Multi-modal LLM with Parallel Recognition and Sequential Understanding
Chuyang Zhao
Yuxin Song
Junru Chen'
Kang Rong' Errui Ding'
Haocheng Feng' Yifan Sun 18
Gang Zhang!
Shufan Ji
Jingdong Wang'
Baidu VIS
2 Beihang University
Equal Contribution
Corresponding Author
{zhaochuyang
www.:uuuuuaangooouuuuunuounioan songyuxin02
suniifan01)@@oaaa. com
Abstract
A mainstream of Multi-modal Large Language Models (MLLMs) have two es- sential functions, i.e., visual recognition (e.g., grounding) and understanding (e.g., visual question answering). Presently, all these MI.LMs integrate visual recognition and understanding in same sequential manner in the LLM head, i.e., generating the response ooken-by-ooken for both recognition and understanding. We think uni- fying them in the same sequential manner is not optimal for two reasons: 1) parallel recognition more efficient than sequential recognition and is actually prevailing in deep visual recognition, and 2) the recognition results can be integrated to help high-level cognition (while the current manner does not). Such motivated, this pa- per proposes novel "parallel recognition - sequential understanding" framework for MLLMs. The bottom LLM layers are utilized for parallel recognition and the recognition results are relayed into the top LLM layers for sequential understanding, Specifically, parallel recognition in the bottom LLM layers implemented via object queries, popular mechanism in DEtection TRansformer, which we find to harmonize well with the LLM layers. Empirical studies show our MLLM named Octopus improves accuracy on popular MILM tasks and is up to 5x faster visual grounding tasks.
Introduction
Visual recognition and understanding two essential abilities for Multi-modal Large Language Models (MLLMs), While earlier MILMs 023 focused on the high-level visual understanding ability (e.g., visual question answering), recent literature finds that visual recognition ability (i.e., identifying and locating the objects) are no less important. The importance lies in two aspects: 1) Many newly-merged MLLM usages are directly related to visual recognition. e.g., visual grounding (5) [6] and referential dialog (6). 2) More generally, visual recognition is potential to benefit all understanding tasks as recognition results are important compositions for understanding. In this paper, we are interested in better harmonizing these two essentials for MLLM.
Presently, MLLMs unify visual recognition and understanding in sequential paradigm. In this paper, the terms "sequential" and "parallel" refer specifically to the inference of LLM head, rather than the visual encoder. Typically, an MILM consists of a visual encoder and an LLM head. During inference. the LLM head sequentially generates the response token-by-token. regardless of whether the task more aligned with visual recognition (e.g., grounding) understanding (e.g., visual question answering), as in Fig. I (left. Sequentially referring the recognition results, particularly textualized coordinates, is relatively slow. This sequential manner a legacy of the LLM structure and, more fundamentally, stems from the inherently sequential nature of language.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Sequential Recognition & Understanding
Sequenial Understaningg (top LLM layas)
This funny becoude (o 0,47, 0.45, 0.69) is lying on the sofa [0.00, 0.21, 0.55,100 1.00] with remote control [0.3 32, 0.68, 0.40, 0.82] nearby, giving the impression that watcing TV.
This pic picture is funny becouse is lying on the sofo with a pontrol nearity. giving impression that is watching TV.
This picture is funny because cat (0.15, sofa (0.00, 0.21, 0.55, 100)
lying on the
This picture is funny becouse coT (0:15, 0:47, 0:45, 0.69)
What is funny about this picture? Please give positions the objects you mentioned.
Parallel Recognation (botoon LLM layers)
Prior MILMs
User
Our Octopus
Figure 1: Comparison between prior MLLMs (left) and our Octopus (right). Left: Prior MLLMs typically adopt the purely sequential inference: the LLM head infers the response token-by-token. regardless whether the response token is more aligned with recognition (e.g., detection) or under- standing. Sequentially inferring the position slow. Right: contrast, Octopus establishes "parallel recognition - sequential understanding" framework. The bottom LLM layers first recognize potential objects via visual grounding or referring segmentation (in Appendix B). The recognition results (coupled with visual tokens) are relayed into top LLM layers. The top L.M layers thus do NOT infer the object position but instead, they select boxes (or masks) that have already been detected. The entire Octopus LLM head (recognition understanding) is trained end-to-end.
We conjecture the purely sequential paradigm might not be an optimistic framework for MILM, especially regarding visual recognition and its cooperation with understanding. There are two reasons:
First, for both human and deep Icarning, visual recognition relies heavily on parallel processing for high efficiency. Before the MILMs era, most deep recognition models are built on the parallel paradigm. For instance, the segmentation model infers the semantic for all pixels simultaneously. and the detection model detects all the objects using parallel anchors 7
g or queries (
10 I1 12
Humans also use parallel processing for simple recognition sequential paradigm lacks efficiency for visual recognition.
[13 14
In contrast, current purely
Second, regarding the cooperation between visual recognition and understanding. there is hierarchy of "parallel recognition - sequential understanding". as revealed by psychology and neurobiology discoveries [4 [5]. Relatively easier and low-level recognition results integrated via more complex mental operations to help high-level cognition [I3]. This hicrarchy allows the understanding to take advantage of the recognition results, while the purely sequential paradigm does not offer such a benefit.
This paper proposes the Octopus (the octopus animal has a central brain and multiple parallel "auxiliary brains") framework to improve the efficiency of recognition, and to harness the benefit of the aforementioned cognition hierarchy. Octopus separates visual recognition and understanding into parallel and sequential processes, respectively, and then re-integrates them "parallel recognition _ sequential understanding" hierarchy. The comparison between purely sequential MLLMs and Octopus is illustrated in Fig.
Given visual tokens from the visual backbone, Octopus's LLM head uses the bottom layers to detect the potential objects in parallel. The detection results, coupled with the visual tokens, are fed into the sub-sequential LLM layers for further understanding. Though the understanding remains sequentially bakennbb-token, it deviates from previous MILMs by eliminating the need infer the position of objects. Instead, it selects previously-detecte boxes and associates them with the objects, markedly improving efficiency. For example, on Flickr30k dataset (average 4 objects per image), Octopus reduces the time of recognizing all objects to about 21% (3.80s to 0.82s per image, 5x increase in speed). Moreover, we empirically find that Octopus improves the accuracy on range of understanding tasks compared to its purely sequential counterpart. It indicates that the initial

---

parallel recognition can effectively support the understanding, revealing clear advantage of the brained-ings -ispeedinspired cognitive hierarchy.
Another significant feature of Octopus is: it can automatically adjust its recognition modes, oscillating between class-agnostic and class-specific, based user instructions. This flexibility provides versatile usages for various tasks. For instance, in a grounding task where users specify the particular interest ee, g., cat, the recognition part of Octopus becomes class-specific and and predicts multiple candidates for the object of interest. The understanding part then selects the best candidate for the final response. In contrast, in another scenario, where the users do not specify any particular interest and request detailed enumeration (including the position), the recognition part automatically shifts into a class-ape aaasssag detector. Correspondingly, the understanding part then assigns semantics the detection results. This flexibility originates from the knowledge in the bottom LLM layers designated for recognition.
Our main contributions are as summarized as follows:
We investigate the cooperation between recognition and understanding in current MLLMs. As result, we identify an efficiency issue with the purely sequential paradigm, as well as a significant discrepancy from human cognitive processes.
We propose the Octopus framework for MLLM. Octopus separates recognition and understanding into parallel and sequential processes, respectively, and re-integrates them into a "parallel recognition sequential understanding" hierarchy.
Extensive experiments show Octopus improves inference efficiency and enhances the accuracy, compared to the purely sequential counterpart.
2
Related Works
Multi-modal Large Language Model. The recent success of large language models (LLMs) has spurred research into integrating LLMs with computer vision for visual understanding. Flamingo [66 adds trainable cross-attention layers to each LLM decoder layer learn visual information. BLIP introduces the Q-Former to align visual and language spaces. Mini-GPT4 2 and mPLUG- also use the Q-Former for visual understanding. LLaVA [8]] connects the pretrained CLIP 19 visual encoder to the LLM with a simple vasion-language connector, achieving strong performance. These efforts demonstrate the potential of Multi-modal Large Language Models
2
3
OWL 17
(MLLMs) for complex multi-modal tasks.
Using MLLMs for Visual Recognition. Inspired by that LLM have unified various NLP tasks into a generation problem in one architecture, recent MLLM works manifest to solve traditional visual recognition tasks in unified MLLM architecture. Object detection, key visual recognition task, poses a challenge in expressing positional information in language within MLLM frameworks. Some convert the bounding box into natural language format and directly generate them in text response. However, representing bounding boxes in text form may not be optimal since the bounding box coordinates
literature 6 20 21 2
numerical data and are typically predicted by regression. Some works 22 (33 use output embeddings of L.LM as the understanding pivot to call an object detector for detection. However, the LLM can not benefit from the object detection results from the detector to enhance its own understanding. A more challenging scenario is the segmentation task, in which the ground-truth can be in random shape and seems indescribable via language. To tackle this problem, VisionLLM 24 represents the segmentation mask in text format by representing the mask by the coordinates the mask polygon. Sphinx [2] and LLaVA-Plus (5) utilize an offline model SAM (26) for segmentation. They first generate the bounding box of the object to segment using MILM,, then they prompt SAM to generate the segmentation mask. LiSA (27 integrates SAM into MILM for joint training. They introduce a special "<SEG>" token to predict the segmentation mask. However, single "<SBG>" token cannot differentiate between multiple instances, limiting to outputting only one segmentation mask.
3

---

+2 gloss-d811
Large Languaee Model (top layen)
defection resutts
Visuil Encader
FEN
Cross Attentoo
Large Language layens)
Sclf-Amention
Describe
Object Queries
Imago Tokers
Text Prompis
Object Querties
Resporses
Figure 2: The overall training process of Octopus. We omit the visual backbone to highlight the LLM head. Octopus inputs multiple object queries into the LLM head, in addition the image tokens and text prompt tokens,. After passing through several bottom LLM layers, the object queries are fed into DETR decoder for parallel recognition. The recognition is the canonical close-set detection, but can be class-agnostic detection, visual grounding, referring segmentation, etc., depending on the user prompt (as shown in Fig Afterwards, these object queries, coupled with the image token and text tokens, are then sent into the upper LLM layers for sequential understanding. When the users ask for spotting the mentioned objects (e.g., visual grounding). Octopus finds the object query that detects each object (e.g., the 2nd object query detects the cat) and points out this object query via corresponding index token "<d2>". The structure of Octopus and its training details are elaborated in Section 3.2 and Section 3,3
3 Approach
3.1
Preliminaries
Sequential inference in prior MILMs. Prior MILMS 12 solve traditional visual recognition tasks (e.g., object detection, instance segmentation) in sequential generation paradigm. Specifically, they represent the outputs of visual recognition in natural language format. For example, the bound- ing boxes are denoted by the coordinates of their corners: "( "[xminn Vmin Iman Mmaz]] ]", and segmenta- tion masks are represented by coordinates of points in polygon mask: "[x1 31, 22, y2, ..,... J Hereby, each numerical value is expressed as multiple text tokens. Sequentially generating all these position tokens can be quite cone-consuming, e.g., 25 tokens for a bounding box "10.152,0.475.0.44 1,0.61, Moreover, generating each token its core, is based classification. In contrast, coordinates inherently numerical and are typically predicted through regression. This discrepancy suggests that sequential generation may not be the optimal approach for visual recognition.
revisit to DEtection TRansformer (DETR). Octopus uses light- DETR decoder to cooperate with the bottom LLM layers for parallel visual recognition, illustrated in Fig. 2 We first give brief revisit to DETR below.
DETR ( is an end-to-end object detection approach based on transformer. A DETR model consists of visual backbone, an encoder, and transformer decoder. The backbone and the encoder transforms an input image into image feature F. Afterwards, the DETR decoder employs a set of parallel object queries {q1, 92..... ] to absorb image features through stacked cross-attention layers, which is formulated as:
Q = -Decoder Q),
in which Q the output state of object queries. Finally, the DETR decoder append class and box predictors upon the object queries to predict their category and bounding box, respectively:
B box Q):
S = cls (Q),

---

Describe image with positions. Two girls cd50> <d26> sit on the gress <d19> beside lake. .d18).
User:
Octopus:
User: Where white pink umbrella? Octopus: <d9>
User: What fruits are the plate?
Octopus: It contains apples and rombuton.
(8)
(b)
(c)
Figure 3: Visualization the Results. We visualize the detection results and the LLM output from Octopus on three tasks: (a) Spotting Caption, (b) Referring Expression Comprehension (REC), and (c) Visual Dialogue. In the Spotting Caption task. DETR identifies all foreground objects (highlighted in blue), while in REC, DETR locates only the object-of-intere
(highlighted in green) as dictated by the prompts. DETR seamlessly transitions between these two modes based on user input. In the case of Visual Dialogue, since the users do NOT ask the MILM output object positions, the L.MM output does not contain box information, correspondingly. However, Octopus can still localize the objects from the intermediate recognition results (we visualize the detection results with a classification score greater than 0.5).
which the bounding boxes and S the classification scores. In Octopus we employ DETR as a class-agnostic detector, S E R denotes whether an object is foreground or background. Consequently, DETR is independent of the number of classes in the training dataset.
We choose DETR decoder to implement visual recognition for the following reasons: 1) The object queries allow for parallel recognition. 2) The versatility of DETR offers the potential to handle more visual recognition tasks. In addition to object detection, DETR can be adapted for many other recognition tasks, e.g., segmentation (28] 30,, pose estimation [31 32 33, and object tracking [34 33| (3)] 3) The object queries can absorb user prompts through attention layers. This allows Octopus to recognize random objects described in natural language, as will be detailed the following section.
3.2
The structure of Octopus
The overall architecture of Octopus is shown in Fig. 2 The image features F extracted the visual The text prompts are tokenized and encoded into text tokens T = {t1, f }. Based on this standard MLLM structure, Octopus ad- ditionally employs & object queries Q = {q1, 4....., 4k }. Each qi E IR." has the same dimension D as the image and text tokens. The object queries are placed after image tokens and prompt text tokens. Consequently, V.T and Q are concatenated and jointly fed into the LLM head, which facilitates parallel recognition and then sequential understanding below.
encoder is projected into image tokens V (V1, 12, u2,,.... ].
Parallel recognition. In the bottom LLM layers, the object queries Q interact with the image tokens V and text tokens T through attention. This interaction aligns the object queries with the hidden states of user prompts and image tokens. In benefit of the language knowledge that embedded in the LLM layers, the object queries thus absorb information from the user prompts and get understanding of the user interest. It makes the recognition adaptive to user prompts and crucial for visual grounding tasks, e.g., referring expression comprehension (REC), Afterwards, the object queries are fed into lightweight (4-layers) DETR decoder to detect the objects. Both the Q-V QT interaction and the object detection DETR decoder are in parallel, yielding the complete recognition for Octopus.
Discussion: It's worth noting that the key difference between the integrated DETR decoder in Octopus and traditional DETR trained on closed-set set is that in Octopus the DETR decoder makes predictions based on user prompts. The predictions made by Octopus DETR can be agnosticc detection, visual grounding, referring segmentation, efc., depending on the user prompt. Fig. 3 shows DETR predicts the objects-of-interee depending on user prompts. For example, e.g., tries to locate all objects in the image, given the user prompt "Please detect all objects in the image". Given another
5

---

user prompt "Please spot the bluck-and-whing cat in the image", DETR focuses on identifying the specified cat rather than other objects.
Sequential understanding. The recognition results, i.e, the object queries output from the DETR decoder, coupled with the hidden states of V and T, are relayed into the following LLM layers. These recognition results will be absorbed into the final output tokens which form the model response, e.g., image captions or visual question answers,
sequential manner.
In visual grounding tasks, the model response is expected to contain bounding boxes for the mentioned objects. In the previous MLLMs, the bounding box is represented in natural language form, which takes multiple tokens and is time-consuming to generate (Section 3.J). In contrast, Octopus does not generate the text of the bounding box in sequential manner, but instead, it selects the detection the 2nd object query from the DETR decoder detects the cat. In the LLM outpur, Octopus generates a special token "<d2>" which indexes the 2nd object query after the "cat" token. We name the token that indexes an object query as index token. Consequently, Octopus becomes aware of the position of the cat by selecting the bounding box corresponding to the predicted index token. How Octopus learns to predict the index tokens is elaborated in the following Section
boxes predicted by the DETR. For example, in Fig. 2
3.3 Training Octopus
Training Octopus involves supervision of two components: the DETR output and the L.M output. These components trained jointly, meaning that both the DETR decoder and the LLM are optimized together.
Supervision on the DETR output. Given the detection predictions Y = (($1,(().... by)...., (sk. by)
and the ground-truth objects Y b1), b1)...., (1, by DETR uses the Hungarian algorithm to find the optimal assignment of (-), where each ground- object is assigned to its best-matched prediction. Here, (s;, b,) indicates the predicted classification scores and bounding boxes from query gs All objects in the targets are treated as foreground objects, and thus, binary classifier is used to predict whether an object query is foreground or background (non-object). The classification loss lets(.) is computed using binary cross-entropy, and the box regression loss box (-) is computed using L1 box distance and GloU loss:
LDTRR Σ (Cets(Do())) Bn) Cbox (bo(t)) bn)));
Supervision on the LLM output. The LLM output is supervised through the next-token prediction manner. However, supervising the object position differs significantly. We recall that for spotting objects, the LLM head does not generate the bounding boxes through text, but predicts index token that points to the corresponding detection result. Correspondingly, during training, Octopus is trained predict the index token following each mentioned object. The ground -tuth index token is not fixed, but dynamically determined on-the-fly in each training iteration.
Identifying the index token requires matching the ground-trui -tutt object to its nearest object query. To this end, we get the predicted bounding box b (the subscript is omitted) of all object queries, and then find the nearest query for the ground-truth object b by:
Cloc = | bl (1 GloU(b, ()).
(4)
We replace the bounding boxes in the response with the index tokens the matched object queries. This process introduces slight overload to the training (increasing ~ 10% training time). In inference, the index tokens are directly generated in the response. Since exclusively use index tokens to represent bounding boxes without the need to generate bounding boxes token-by-toker our method significantly faster than previous MLLMs.
The LLM computes the training loss as the language modeling loss using next-token prediction, the same as prior MLLMs.
K
LIM
Σ
log P(ylve)),
(5)
where yi represents the target token at position i. The overall training objective the combination of DETR training loss and language modeling loss.

---

Table 1: Results on Referring Expression Comprehension benchmarks. We note that Octopus with resolution 224/336 is up to 5x/4x faster than a purely sequential counterpart Shikra (resolution 224)
and achieves higher accuracy. More details of the inference speed comparison are reported in Sec. 4.5
RefCOCO
RefCOCO+
RefCOCOg val
Model Type
Method
Res.
val
test-A
test-B
val
test-A test-B
test
OFA-I
37
480
79.96
83.67
76.39
68.29
76.00
61.75
67.57
67.58
Vi
224
86.70
Generalists
Shikra 6
224
87.01
90.61
80.24
81.60
87.36
72.12
82.27
82.19
MiniGPT-v2
38
448
88.06
91.29
84.30
79.58
85.52
73.32
84.19
84.31
Octopus
224
88.77
91.93
82.28
83.05
88.87
75.12
83.11
84.78
Octopus
336
89.02
92.63
83.42
83.55
89.40
76.02
84,25
86.19
G-DINO-L
39
512
90.56
93.19
88.24
82.75
88.95
75.92
86.13
87.02
Specialists
UNINEXT-H
40
640
92.64
94.33
91.46
85.24
89.63
79.79
88.73
89.37
4 Experiments
4.1 Settings
Training details. We train Octopus via three stages, i.e. stage-1 for pretraining vision- lasion-language align- ment, stage-2 for pretraining the DETR recognition module, and stage-3 for end-to-end instruction fine-tuning. The details are as below:
1) Stage-1 pretrains the vision-language connector for vision- alignment using LLaVA pretraining data [18]. The visual encoder and LLM are frozen and only the lision-language connector is trained. 2) Stage-2 pretrains the DETR module on small-scale grounding detection datasets
(RefCOCO [41) RefCOCO+ [42], RefCOCOg [42 and Flickr30k (43) to quickly obtain the recognition ability. We freeze LLM and visual encoder and only train the DETR module in this stage. Stage-2 is mainly for training efficiency, iee. fast adapting the DETR decoder to the LLM layers. 3) In stage-3, we finetune the whole LLM head and DETR decoder on ALLVA--Insteee [18], REC data (ReíCOCO, RefCOCO+, RefCOCOg, Visual Genome 44 and Flickr30k end-to-end. Please refer to the Appendix for details of the training datasets.
We adopt AdamW as the optimizer and cosine annealing scheduler. The learning rate initialized to le-4 for stage-1 and stage-2, and 2e-5 for stage-3. The entire training takes about 2 hours for Stage-1 (I epoch), 4 hours for Stage-2 (2 epochs) and 120 hours for Stage-3 on 8 NVIDIA A100 GPUS.
Architecture details. Octopus adopts the ViT pre-trained from CLIP as the visual encoder. All the vision-language connectors one-layer MLP with random initialized. The LLM head is initialized with
[45] and the DETR decoder consists of 4 standard DETR decoder layers. We employ 64 object queries and place the DETR decoder after the 16-th LLM layer.
4.2 Evaluation on REC datasets
We evaluate Octopus's 's recognition ability on 3 popular referring expression comprehension datasets, i.e., RefCOCO, RefCOCO+ and RefCOCOg. Table Octopus achieves the highest accuracy on seven out of eight dataset splits, among the compared generalist MILMs. For example, Octopus out- performs MiniGPT-v2 [46] by +0.96% on val split and +1.34% on test-A split of RefCOCO dataset. On RefCOCO+, the superiority is even larger, e.g., surpassing MiniGPT-v2 by +3.97% on val split, +3.92% on Aest-A split and +2.70% on test-B split.
We particularly note the comparison against Shikra, purely sequential counterpart that adopts the same backbone as our Octopus and shares the same training data. Octopus (resolution 224) is up to 5x faster than Shikra under the same image resolution and consistently achieves higher accuracy, e.g. +1.511 on RefCOCO+ test A. When scaling up the image resolution to 336, Octopus still 4x faster than Shikra (resolution 224) and further enlarges the accuracy superiority to + 2.04% on RefCOCO+ test A. These observations show that parallel recognition improves both the efficiency and accuracy for recognition.
7

---

Table 2: Results Visual Question Answering benchmarks. Note that specialists fine-tuned on each individual evaluation dataset. We gray out those specialists methods, as well as the fine-tuned results of generalists.
#LLM
Model Type
Method
Res.
VQAv2
OKVQA
GQA
VizWiz
SciQA
Params
BLIP2
11B
224
65.0
45.9
41.0
19.6
61.0
InstructBLIP
11B
224
49.2
34.5
60.5
Unified-IO XL
2.7B
256
77.9
54.0
PaLM-E-12B
12B
224
76.2
55.5
Generalists
Shikra
6
7B
224
77.4
47.2
Octopus
7B
224
78.5
56.0
62.29
45.6
65.7
1L5
49
7B
336
78.5
62.0
50.0
66.8
Qwen-ViL VL-Chat [50
7B
448
78.2
56.6
57.5
38.9
68.2
Octopus
7B
336
79.2
57.2
63.3
50.1
67.7
GIT [1
0.7B
384
78.6
68.0
Specialists
GIT2 61
5.1B
384
81.7
71.0
PaLI-17B 52
17B
580
84.3
64.5
71.6
Table 3: Results popular VL benchmarks. MMB is MMBench [53], LLaVA is LLaVA-Bench
(In-the-Wil (iid) [18] and MM-V is MM- Vet Benchmark [54]. POPE (55] is reported on the average F1 score of three splits (Adersarial, Popular and Random).
#LLM
Method
Resolution
MMB
LLaVA
SEED
MM-V
POPE
Params
BLIP-2
13B
224
38.1
46.4
22.4
InstructBLIP
7B
224
36.0
60.9
53.4
26.2
InstructBLIP
13B
224
58.2
25.6
78.9
IDEFICS 16
7B
480
48.2
IDEFICS [6
65B
480
54.5
1LaVA-1.5
49
7B
336
64.3
65.4
58.6
31.1
84.2
Qwen Vie-Val [50)
7B
448
60.6
58.2
Shikra
7B
224
58.8
83.9
Octopus
7B
224
66.2
63.9
58.6
32.1
84.8
4.3 Evaluation on VQA datasets
General visual question answering (VQA) is a widely employed task for MLLMs. We compare Octopus on VQA datasets against multiple competing MI.LMs in Table 2 It observed that Octopus achieves competitive results under both 224 and 336 resolution settings. For instance, at 224 image resolution setting, Octopus outperforms Shikra 6 by 1.Moooorr, it surpasses InstructBLIP 4 by +13.09%, +11.1 11.1%% %,, and +5.2% on GQA, VizWiz, and SciQA, respectively.
We note that on these datasets, MILMs not asked to provide position of the mentioned objects, but Octopus still gains benefit from parallel visual recognition. This suggests that the recognition effectively supports the subsequent understanding, validating the benefit of "parallel recognition sequential understanding" hierarchy.
4.4 Evaluation recent MILM benchmarks
We report the performance on the recent popular MILM benchmarks in Table 3 It is observed that Octopus performs favably against purely sequential counterparts on the five benchmarks. For example, Octopus is higher than Shikra by +7.4% on MMBench and +0.9% on POPE, respectively.

---

Table 4: Comparison of inference speed. We compared the inference speed of our method with the baseline method Shikra on two benchmarks. The reported inference times (in seconds) are averaged for one record across the datasets. RefCOCO contains exactly bounding box in outputs, and Flickr30k contains on average 4 boxes in outputs. Our method is much faster (~ 5x) than the baseline in both resolutions.
#LLM
RefCOCO
Flickr30k
Method
Resolution
Params
infer time
FPS
infer time
FPS
Baseline
7B
224
0.89
1.12
3.80
0.26
Octopus (ours)
7B
224
0.17
5.88
0.82
1.21
Baseline
7B
336
1.16
0.86
6.08
0.16
Octopus (ours)
7B
336
0.27
3.70
1.49
0.67
78.7
79.4
79.5
79.1
78.2
77.0
76.5
(a)
12
16
18
20
(b)
100
120
Figure 4: Influence of the recognition layers and number of object queries. We evaluate our method RefCOCOg val benchmark. We did Visual Genome data (56) in training for efficiency, (a) Influence of the number of LLM layers employed for recognition. (b) Influence of the number of object queries.
Octopus also outperforms InstructBLIP by +3.0% on LLaVA VA-Bench, +5.2% on SEED and +6.5% on MM-Vet, Vet, respectively.
4.5
Ablation Study
The number of LLM layers employed for recognition. We study the optimal layer for positioning DETR within LLM. Fig. (a) illustrates method's performance the RefCOCOg val benchmark when placing DETR after different LLM layers. We find that the optimal placement for DETR is after the 16th LLM layer, which is the middle layer of the LLM (consisting of 32 layers in total). We infer that positioning DETR on a lower layer might limit its comprehension of the instruction prompt, while placing it higher layer harms the ability of LLM to infer the correct index tokens for the mentioned objects.
Number of object queries. We compare the impact of different object query number performance in Fig. 4 (b). We evaluate Octopus on the val split of RefCOCOg benchmark and observe that 64 and 80 object query achieved the best performance in our setup. We infer that using insufficient object queries will hamper the recognition capabilities of the bottom LLM layers. On the other hand, although excess of object queries enhance the LLM's perception ability for foreground objects, also increases the complexity for the LLM to select the correct object queries in the final output.
Inference speed. We compare the inference speed of our method against the traditional MILMM on the grounding detection data. As shown in Table our method is much faster (5x) on all benchmarks. This is attributed to that our method does not require generate bounding boxes in discrete tokens. For each bounding box, we only need to generate an index token to the corresponding detection results predicted by the DETR module. It saves 24 tokens in generating one box, which is more significant in scenarios where a larger number of objects exist.
Qualitative analysis. We visualize the recognition results predicted by DETR for non-grounding data, such as VQA and instruction dialogue, which no bounding boxes are provided and generated. As shown in Fig. 3 (c), DETR still locates the mentioned objects in the user prompt even if it is not in grounding data format. The detection results are helpful for VQA and some tasks, where the MLLM needs to locate the mentioned objects in the image and answer specific questions.

---

Limitation and Conclusion
Limitations. Due to limited GPU resources, we have been able to explore how Octopus would perform when scaling up on larger LLM and more open-ended instruction-tuninn data. Moreover, DETR applicable to many recognition tasks beyond detection. We leave it as future works.
Conclusion. We propose Octopus, a novel MLLM framework that disrupts the purely sequential inference paradigm for LLM head. Octopus perform parallel recognition through the lower LLM layers and a lightweight DETR decoder, and then passes the recognition results to the upper LLM layers for further understanding. Consequently, it reformulates the LLM head into "parallel recognition - sequential understanding" hierarchy. Empirical results show Octopus improves accuracy over range of MLLM tasks and significantly enhances inference efficiency when the task include recognition objectives.
References
[1] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36, 2024.
[2] Deyao Zhu, Jun Chen, Xiaoqian Shen. Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models, 2023.
[3] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: pre-training with frozen image encoders and large language models, 2023.
Bootstrapping language-image
[4] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[5] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023.
[6] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic, 2023.
[7] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[8] Joseph Redmon, Santosh Divvala, Ross Girshick. and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 779-788, 2016.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213-229. Springer, 2020.
[10] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201. 220., 2022.
[11] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arX.v.2010.04159, 2020.
[12] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605. 2022.
[13] Elena Agliari, Adriano Barra, Andrea Galluzzi, Francesco Guerra, Daniele Tantari, and Flavia Tavani. Hierarchical neural networks perform both serial and parallel processing. Neural Networks, 66:22-35 2015.
10

---

[14] Yoshihito Shigihara and Semir Zeki. Parallel processing in the brain's 's visual form system: an fimri study. Frontiers in human neuroscience, 8:506, 2014.
[15] Kang Li, Mikiko Kadohisa, Makoto Kusunoki, John Duncan, Claus Bundesen, and Susanne Ditlevsen. Distinguishing between parallel and serial processing in visual attention from neurobiological data. Royal Society Open Science, 7(1):191553, 2020.
[16] Jean-Baptiste Alayrac et al. Flamingo: a visual language model for few-shot learning. 2022.
[17] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
[18]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[20] Wenhai Wang, Zhe Chen. Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024.
[21] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575. 2023.
[22] Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. International Journal of Computer Vision, pages 1-19, 2024.
[23] Hao Zhang. Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Chunyuan Li, Jainwei Yang, et al. Alava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pages 19-35. Springer, 2025.
[24] Wenhai Wang, Zhe Chen. Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, 2023.
[25] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang. Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05433 2023.
[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.
[27] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.006: 2023.
[28) Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 12.01:2112.01557) 2021.
[29] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- 12,
der Schwing. Mask2former for video instance segmentation. 12, 2021.
[30] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR, 2022.
[31] Lucas Stoffi, Maxime Vidal, and Alexander Mathis. End-to-end trainable multi-instance pose estimation with transformers. arXi::2103.12115 2011.

---

[32] Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang. and Zhibin Wang. Tipose: Direct human pose estimation with transformers. arXiv:2103.55320 2021.
[33] Guillem Brasó, Nikita Kister, and Laura Leal- Taixé. The center of attention: Center-keypoint grouping via attention for multi-person pose estimation. In ICCV, 2021.
[34] Bin Yan, Houwen Peng. Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In ICCV, 2021.
[35] Moju Zhao, Kei Okada, and Masayuki Inaba. arXiv:2105.03817, 2021.
Trtr: Visual tracking with transformer.
[36] Fangao Zeng, Bin Dong, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. arXiv:2105. 03247, 2021.
[37] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sequeeee learning framework. In International Conference Machine Learning, pages 23318-23340. PMLR, 2022.
[38] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. 2023.
[39] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
[40] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval, 2023.
[41] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[42] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[43] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting eegion-to-phran correspondences for richer image-to-sentence models. In Proceedings of the IEFE international conference on computer vision, pages 2641-26499 2015.
[44] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[45] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[46] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model unified interface for vasion-language multi-task learning. arXiv preprint arXiv:2310.09478. 2023.
[47] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-moda tasks, 2022.
12

---

[48] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023.
[49] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[50] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding. localization, text reading, and beyond, 2023.
[51] Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: generative image-to-text transformer for vision and language, 2022.
[52] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski. Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-inan -mage model, 2023.
[53] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint (rXii:2307002888 2023.
[54] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 024900, 2023.
[55] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- languae models, 2023.
[56] Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
13

---

A Comparison between the intermediate and final recognition results
In Octopus, there are two components that can give recognition results, i.e. the intermediate outputs from the DETR and the final LLM outputs. The intermediate recognition results are parallel and re- dundant, i... predicting multiple boxes for each object of interest. Based on these parallel recognition results, the final LLM outputs choose single one for each object via the indexing tokens.
We compare these two different results for recognition objectives in Table []] Since the intermediate recognition results (DETR results) have multiple predictions, we evaluate the rank-k accuracy, which indicates the recall of the ground-truth. We draw two observations as below:
First, the final LLM outputs yield higher accuracy than the rank-1 DETR results, egg., +1.18% and 2.12% under the 224 and 336 resolution settings, respectively. shows that the LLM top layers further improves recognition accuracy after understanding all the recognition results.
Second, the redundant DETR results achieve high recall of the object-of-interre e.g., 94.56% rank-5 accuracy under the 224 resolution. Since these intermediate detection results are relayed into the LLM top layers, we infer the high recall is important reason for the LLM to improve recognition accuracy by selecting one result for each object-of-iniee if-ineres.t
In addition to the benefit of improving recognition accuracy, the understanding part of Octopus has another important role: it organizes multiple recognition results into a complete description or response, e.g., in the spotting caption task (Fig. 5 in the main part).
Table 5: Comparison of Detection Performance.
Octopus
DETR results
Method
Resolution
Accuracy
R@1
R@2
R@3
R@4
R@5
Octopus
224
83.11
81.93
90.54
92.85
93.89
94.56
Octopus
336
84.25
82.13
92.79
94.58
95.36
95.95
B Application on referring segmentation
In addition to detection, Octopus is potential to acquire broader range of recognition abilities, taking advantage of the versatility of the DETR mechanism. Hereby, we endow Octopus with the referring segmentation ability by modifying its object queries into "mask" queries, common
Where is the bus in center?
Where the chocolate cake?
Where is the front-mot bus?
Harry Pucmery Aol Prich
Where the left yogurt?
Where the right sandwich?
Where is the right bowl?
Figure 5: Visualization of the referring segmentation results. Octopus directly makes pixel-wise predictions rather than predicting the vertexes of the segmentation mask.
14

---

application of DETRs. We add an additional mask head on top of the DETR decoder to predict the segmentation mask for each mask query. Under this setting, we train Octopus on the training datasets of referring segmentation datasets (RefCOCO, RefCOCO+, RefCOCOg). Visualization in Fig. 5
shows that Octopus gains referring segmentation ability. The ability of directly predicting pixel-wise segmentation results rather than vertexes of segmentation masks is valuable for MILMS..
Table 6: Referring segmentation results on RefCOCo.
Method
Resolution
val
testA
testB
Octopus
224
63.6
66.6
61.3
We quantitatively evaluate the referring segmentation performance on the RefCOCO benchmark using cloU. Table.[[shovv that our model achieves reasonable results. Due to time limit, we only implement Octopus using the low-resolution CLIP-ViT features (224 224) for this experiment. The input size is very small for segmentation task and is important reason that limits performance (e.g., 63.6 cloU on RefCOCO validation set). We note a recent state-of-thee 74.6. LISA uses external strong segmentation model, SAM [26). that is pretrained on SA-IB datasets and uses large input size. We conjecture that enlarging the input size and adding training data will bring further improvement to Octopus, well.
method LÍSA achieves
To sum up, by incorporating the bottom layers of LLM head with the DETR query mechanism, Octopus is potential to acquire various recognition abilities. We will explore more forms of recognition abilities for Octopus.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide our contributions both in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We express our limitations in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, claim our experimental settings in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification: Our code is currently being organized. We will release the code and data after the review process complete.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://ooooSuuiiss.oooeeee
//nips //nips.c//ppl wwcppblic/guticcccccccceeeooooee
httSSSabmiiiiooooiiioiooooii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, provide all training and test details in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited and cited the used assets in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "Izn1gD3hV2", "reviewer": "Reviewer_hBAZ", "summary": "Octopus first identify unifying visual recoginition and understanding is not optimal from two aspects, 1) parallel recognition could achieve better effiency, 2) recognition results can help understanding. Based on this insight, this work built a recognition then understanding pipeline for MLLM.  Extensive experiments are conducted to show the effectiveness of this work. MLLM achieves better accuracy on common MLLM tasks and is much faster on visual grounding tasks.", "questions": "see weakness part. \nif authors could address my concerns, i would be happy to raise my rating.", "limitations": "yes,", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. this work give thorough investigation about the connection between visual recognition and understanding in MLLMs.\n\n2.  i like the flow of writing in this work,  the proposed framework is originated from the observed insights, which is quite straightforward yet effective. \n\n3. Extensive experiments show Octopus improves inference efficiency and enhances the accuracy, demonstrating the effectiveness of this work.", "weaknesses": "1. More experimental investigations shall be counducted to support the motivation of this paper, especially more quantitative analysis. \n2. DETR framework is too old for visual recognition, it is more proper to introduce some efficient architecture for this part, e.g deformable-detr [1]\n3. the writing of this work can still be improved, and typos still exists. \n\n[1] Zhu, Xizhou, et al. \"Deformable detr: Deformable transformers for end-to-end object detection.\" arXiv preprint arXiv:2010.04159 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: More experimental investigations shall be conducted to support the motivation of this paper, especially more quantitative analysis.  \n**A1**: Thanks. During rebuttal, we conducted one more experiment according to your suggestion. This experiment along with the already-existing ones in Table 1 to 4 in the manuscript can well support our motivation, i.e., the \"parallel recognition $\\rightarrow$ sequential understanding'' hierarchy brings mutual benefits between recognition and understanding, and is efficient. The details are as below:\n\n**Existing experiments**. Table 1 and Table 4 in the manuscript show that our method improves open-world recognition (REC) accuracy (+2.01 on RefCOCO) and efficiency (5$\\times$ faster). Table 2 and Table 3 show the recognition results then benefit the understanding (+0.7 on VQAv2). \n\n**The new experiment in rebuttal**. The improvement on understanding task seems relatively small in Table 2 and Table 3, mainly because many understanding evaluation objective is not closely related to recognition. Hereby, we investigate some closely-related recognition tasks. We selected a subset (173621/214354) of the VQAv2 dataset consisting of \"What?\" (87868), \"How many?\" (23372), and \"Is?\" (62381) type questions, where grounding is needed to recognize the referred objects. We compared the results of our methods against the baseline methods on this subset.\n\n| Method  | LLM Params | Res | VQAv2 |\n| ------- | ---------- | --- | ----- |\n| LLaVA   | 7B         | 336 | 79.3  |\n| Shikra  | 7B         | 224 | 78.8  |\n| Octopus | 7B         | 224 | 81.0  |\n\nAs shown in the above table, the advantage of Octopus on this subset is more apparent, which further validate our motivation that parallel recognition can further improves the sequential understanding of MLLM.\n\n---\n\n**Q2**: DETR framework is too old for visual recognition, it is more proper to introduce some efficient architecture for this part, e.g deformable-detr.  \n**A2**: Thanks. During rebuttal, we  substituted the DETR backbone with Deformable-DETR pretrained on COCO-2017 dataset, and trained Octopus with Deformable-DETR on LLaVA-Instruct, REC and Flickr30k datasets. Results are evaluated on RefCOCOg val and test sets, as shown in the table below. We observe further improvement, i.e., +0.5 and +0.4 on validation and test set, respectively. Moreover, during training , we observed that Deformable-DETR significantly accelerates training in earlier stage. We will release  the code for both DETR and the deformable DETR implementations. \n\n| backbone        | val  | test |\n| --------------- | ---- | ---- |\n| DETR            | 79.5 | 80.2 |\n| Deformable-DETR | 80.3 | 81.1 |\n\n---\n\n**Q3**: The writing of this work can still be improved, and typos still exists.  \n**A3**: Thank you for your advice, we will continue to improve the writing and fix typos in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iF3ZEcKDLd", "reviewer": "Reviewer_aS3o", "summary": "This paper introduce a new “parallel recognition → sequential understanding” framework for MLLMs. Specifically, the bottom LLM layers are utilized for parallel recognition and the recognition results are relayed into the top LLM layers for sequential understanding. The parallel recognition in the bottom LLM layers is implemented via object queries, a popular mechanism in DETR. The authors conduct experiments on popular understanding MLLM tasks.", "questions": "What is the performance of the proposed method on Flickr30k and RefCOCO?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is interesting that transfers  visual recognition and understanding in a same sequential manner to “parallel recognition → sequential understanding” manner.\n2.The presentation is good and the figures are well-designed for understanding the proposed methods.", "weaknesses": "1. Missing many experiments. This paper propose a framework for both detection and understanding. However, the authors only report the performance of results on VQA MLLMs. They only report inference time and  fps on RefCoCo and Flickr30k, not performance.\n2. The compared methods are not newly proposed and current SOTAs, should add more recent methods for comparison, such as LLaVA 1.5, LLaVA-next and etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Missing many experiments. This paper propose a framework for both detection and understanding. However, the authors only report the performance of results on VQA MLLMs. They only report inference time and fps on RefCoCo and Flickr30k, not performance.  \n**A1**: Table 1 and Section 4.2 in the manuscript already provide performance (acc.) on RefCOCO, as well as other two datasets, i.e., RefCOCO+ and RefCOCOg. These are the three most popular datasets for the referring expression comprehension (REC) task. \nWe did not employ Filcker30k because its annotation does not satisfy the standard REC requirement, i.e., binding the bounding box with the corresponding object. No prior methods employed Flicker30k for REC evaluation yet either.  \nGiven this limitation, during rebuttal, we compute the Rank-1 accuracy as the alternative evaluation protocol, by computing the accuracy between the ground truth boxes and predicted boxes pairs of IoU > 0.5. We observe that our method improves the baseline, e.g.,  +2.4 (59.3 to 61.7) R1 score on Flicker30k test set.\n\n| Method  | val @ R1 | test @ R1 |\n| ------- | -------- | --------- |\n| Shikra  | 58.9     | 59.3      |\n| Octopus | 61.2     | 61.7      |\n\n---\n\n**Q2**: The compared methods are not newly proposed and current SOTAs, should add more recent methods for comparison, such as LLaVA 1.5, LLaVA-next and etc.  \n**A2**: The requested LLaVA-1.5 result is already listed in Table 2 in the manuscript (\"LLaVA\"). We will revise the notations from \"LLaVA\" to \"LLaVA-1.5\" to avoid confusion. Table 2 in the manuscript clearly shows that our method surpasses LLaVA-1.5, e.g., +0.7 on VQAv2 and +1.3 on GQA.  \n\nWe did not compare LLaVA-Next because it is not an official academic paper, but rather a technical blog. It does not release its training data and code, either.  Since our method is not restricted to any specific MLLM baseline, it can be plugged into LLaVA-Next and is potential to achieve some improvement, as well. \n\n---\n\n**Q3**: What is the performance of the proposed method on Flickr30k and RefCOCO?  \n**A3**: As explained in the responses to Q1, Table 1 in the manuscript already evaluated RefCOCO (+2.01 compared to Shikra baseline).\nFlicker30k is not quite usable for the standard REC task, so we use an alternative evaluation protocol and observe consistent improvement (58.9 $\\rightarrow$ 61.2 rank-1 accuracy). Please refer to the response to Q1 for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agvGJo7AO6", "reviewer": "Reviewer_Up36", "summary": "This paper introduces Octopus, a multimodal LLM that employs parallel recognition and sequential understanding. Inspired by the human brain, the authors aim to address the efficiency issues inherent in existing methods by proposing this novel approach. The paper makes three key contributions: (1) Identification of an efficiency issue with the sequential paradigm, (2) Proposal of the Octopus framework to enable parallel recognition and sequential understanding, and (3) Demonstration of superior performance and inference efficiency compared to the sequential approach.", "questions": "- How are the newly introduced object queries initialized? Does this affect the final performance?\n- Would it be better to use a pre-trained DETR decoder from off-the-shelf publicly available checkpoints?", "limitations": "Generally speaking, the proposed method is interesting and indeed more efficient than generating bounding box strings sequentially. As such, I am inclined to assign a borderline score (slightly leaning towards acceptance) at this stage. However, this paper also suffers from several issues that could lead to a rejection in the final decision.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to understand. The proposed Octopus framework is novel and intriguing.\n- The experimental results partially support the authors' claims.\n- The proposed framework not only performs object localization but also generalizes well to pixel-wise recognition tasks such as segmentation.", "weaknesses": "- The biggest problem is that the experimental results and comparisons in this paper are not comprehensive and sometimes unfair to some baseline methods. Octopus is built upon LLaVA and leverages additional data beyond LLaVA's dataset. Therefore, the results in Table 2 cannot conclusively demonstrate that Octopus benefits from parallel visual recognition (as stated in Line 251). It would be better to compare Octopus and LLaVA using the exact same training data. Moreover, in Table 3, some methods from Table 2, such as LLaVA and Qwen-VL, are not included. If the result comparisons are cherry-picked, the conclusions are unreliable, and the performance advantages are not significant.\n- Previous MLLMs with grounding capacity can be grouped into two categories: (1) generating box coordinates as natural language (like Shikra and Qwen-VL), and (2) introducing localization tokens (such as PaLI and Kosmos-2). This paper addresses the efficiency issue of the former methods (coordinates as language) but ignores the latter. What are the performance and efficiency advantages when comparing Octopus with the latter (introducing new localization tokens)?\n- In Line 200, it is stated that after bipartite matching, the response is modified by replacing the bounding boxes with the index tokens. Does this mean that for one sample (image, instruction, and response), it requires dual forward passes (first image + instruction + queries, then image + instruction + queries + modified response)? As the authors mentioned in Line 201, Octopus introduces non-negligible training overhead.\n- Considering the unfair comparisons in Tables 2 and 3, and given the slight performance advantages in Table 1, the advantage of \"Parallel Recognition\" does not seem significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The comparison in this paper are not comprehensive and sometimes unfair to some baseline methods. It would be better to compare Octopus and LLaVA using the exact same training data. Moreover, in Table 3, some methods from Table 2, such as LLaVA and Qwen-VL, are not included.  \n**A1**: Thank you for your advice. We have trained LLaVA with the same source of training data as Octopus (by adding the grounding data). The results in below Table show Octopus surpasses LLaVA with same training data.\n\n| Method  | \\#LLM Params | Res. | VQAv2 | GQA  | VizWiz | SciQA |\n| ------- | ------------ | ---- | ----- | ---- | ------ | ----- |\n| LLaVA   | 7B           | 336  | 78.8  | 62.5 | 48.8   | 65.9  |\n| Octopus | 7B           | 336  | 79.2  | 63.3 | 50.1   | 67.7  |\n\nWe will add the results of Qwen-VL and LLaVA into Table 3 in the manuscript. As shown in the below table, our method surpasses LLaVA and QwenVL-Chat on MMB, SEED, MM-V, and POPE benchmarks, but is lower than LLaVA on the LLaVA$^W$ benchmark.\n\n| Method      | \\#LLM Params | Resolution | MMB      | LLaVA$^W$ | SEED     | MM-V     | POPE     |\n| ----------- | ------------ | ---------- | -------- | --------- | -------- | -------- | -------- |\n| LLaVA       | 7B           | 336        | 64.3     | **65.4**  | 58.6     | 31.1     | 84.2     |\n| QwenVL-Chat | 7B           | 448        | 60.6     | -         | 58.2     | -        | -        |\n| Octopus     | 7B           | 224        | **66.2** | 63.9      | **58.6** | **32.1** | **84.8** |\n\n---\n\n**Q2**: Some MLLMs, such as PaLI and Kosmos-2 introdices localization tokens to fullfill grounding ability. What are the performance and efficiency advantages when comparing Octopus with these methods (introducing new localization tokens)?  \n**A2**: Thank you for your advice. We compare using localization tokens and our Octopus in the table below. It is observed that our Octopus achieves higher accuracy (e.g., + 5.9 test acc.) and is slightly faster. The results are reasonable: the localization tokens have coarse spatial resolution (7 $\\times$ 7 pixels per token) and is thus inferior for REC. We used the Kosmos-2 code for implementing the localization code and adopted the same backbone (i.e., LLaVA-1.5) plus the same REC training data for a fair comparison. We did not use PaLI because PaLI has not released its code yet. \n\n| Method   | Resolution | infer time | fps  | val  | test |\n| -------- | ---------- | ---------- | ---- | ---- | ---- |\n| Kosmos-2 | 224        | 0.23       | 5.34 | 73.2 | 74.3 |\n| Octopus  | 224        | 0.17       | 5.88 | 79.5 | 80.2 |\n\n---\n\n**Q3**: In Line 200, it is stated that after bipartite matching, the response is modified by replacing the bounding boxes with the index tokens. Does this mean that for one sample (image, instruction, and response), it requires dual forward passes (first image + instruction + queries, then image + instruction + queries + modified response)? As the authors mentioned in Line 201, Octopus introduces non-negligible training overhead.  \n**A3**: Yes, our model requires two forward passes to generate the correct response. However, the training overhead is relatively low (only \\~10\\%) because the first pass requires no gradients computation and is efficient. \n\n---\n\n**Q4**: Considering the unfair comparisons in Tables 2 and 3, and given the slight performance advantages in Table 1, the advantage of \"Parallel Recognition\" does not seem significant.  \n**A4**: \nThe advantage of \"parallel recognition\" on REC is non-trivial, regarding both efficiency (about $4\\times$ faster in Table 4) and accuracy (+1.76 over the baseline on Refcoco val). Please kindly note that we have already made the comparison fair by using the same training data during rebuttal. \n\nAs for understanding task (Table 2 and 3), the superiority does seem relatively small. This is because many understanding tasks are not closely related to recognition. During rebuttal, we add another experiment by selecting closely-related sub-tasks and observe larger improvement (+1.7 VQA v2). Please refer to R1-Q1 for details. \n\n\n---\n\n**Q5**: How are the newly introduced object queries initialized? Does this affect the final performance?  \n**A5**: We guess your question is about the training initialization for the object queries. At the start of training, we use the standard Gaussian initialization (the default PyTorch initialization) for these object queries (as well as for all the other vocabulary embeddings). After training convergence, all the object queries use the learned embeddings for each inference. \n\n---\n\n**Q6**: Would it be better to use a pre-trained DETR decoder from off-the-shelf publicly available checkpoints?  \n**A6**: Thanks. Utilizing a pre-trained DETR decoder can benefit training, as validated by our experiments in Stage-2 (L219). A pre-trained decoder helps in quickly adapting the DETR decoder to the LLM layers. If we do not use DETR pretraining, it takes a longer period (1.5$\\times$) to achieve similar results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lzmwrYOg5k", "reviewer": "Reviewer_u3Ag", "summary": "The paper presents Octopus, a combined MLLM for joint localization and visual chat tasks. The authors insert the DETR decoder into the LLM to perform visual grounding to replace LLM’s outputs for more precise localization. The bottom LLM layers are utilized for parallel recognition and the recognition results are relayed into the top LLM layers for sequential understanding. Extensive results on various MLLM tasks show the effectiveness of Octopus.", "questions": "See the weakness.", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, The overall writing is easy to understand and follow.\n\n2, The proposed model, Octopus, can perform both localization tasks and visual chat tasks in one shot. Moreover, it runs faster than previous works.\n\n3, The performance looks good compared with several previous MLLM, such as MiniGPT-V2 and Shika.", "weaknesses": "1, The motivation of combining DETR with MLLM is not new. Several previous works have explored the nearly same design. In particular, LLaVA-Grounding have the same design by introducing the Detr decoder into the output of LLM. However, as shown in Fig.4 (a), taking the intermediate LLM layer as the input of DETR, the effects are not that significant than previous design in LLaVA-Grounding.\n\n2, The technical novelty is limited. The meta-architecture is the same as previous works.\n[2] is on arXiv even one year ago. However, the authors adopt the nearly same idea but without a citation. This is disrespectful for the community.\n\n[1], LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models, arxiv-2023.\n\n[2], Contextual Object Detection with Multimodal Large Language Models, arxiv-2023.\n\n\n3, Several ablation studies are missing.   For example, the effect of pre-training data and the effect of transformer layers in DETR. \n\n4, No failure case analysis and no improvement analysis, compared with late fusion decoder.\n\n5, What about using LLM to predict the box in the end?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The motivation of combining DETR with MLLM is not new. Several previous works have explored the nearly same design. In particular, LLaVA-Grounding have the same design by introducing the Detr decoder into the output of LLM.  \n**A1**: Our motivation is not combining DETR with MLLM.\nOur motivation (and key contribution) is integrating the parallel recognition and sequential understanding into one LLM, and organizing them into a “parallel recognition → sequential understanding” hierarchy. This is fundamentally different and even opposite to previous \"MLLM + DETR'' methods because they all follow the \"understanding → recognition'' regime, i.e., first concept understanding and then recognition. \n\nOur novel hierarchical MLLM framework has several advantages, compared to previous \"DETR + MLLM'' methods, including:\n- The parallel recognition results can further augment the sequential understanding ability of current MLLMs. In contrast, in previous \"MLLM + DETR'' methods, the recognition does not benefit understanding. \n- Our DETR head can be used separately for recognition tasks, resulting in high efficiency. In contrast, previous \"MLLM + DETR'' methods is more computational expensive because they require full inference over the entire LLM. \n\n---\n\n**Q2**: The technical novelty is limited. The meta-architecture is the same as previous works. [2] is on arXiv even one year ago. However, the authors adopt the nearly same idea but without a citation. This is disrespectful for the community.  \n**A2**: We respectfully disagree. Though our method shares some similarity (\"MLLM + DETR''), the architecture and underlying mechanism are fundamentally different:\n- Architecture: as explained in the Q1-Answer, our Octopus is a novel \"parallel recognition → sequential understanding'' MLLM framework, while [2] adopts the popular \"understanding pivot → recognition head'' regime. \n- Mechanism: [2], as well as some recent methods [1,3], employs an MLLM as the understanding pivot to call a recognition head, so that the recognition head can receive and understand free-style human instructions. The LLM benefits the recognition head. In contrast, our Octopus uses bottom LLM layers to assist the recognition and in turn uses the recognition results to prompt the following understanding. The LLM and the recognition mutually benefit each other. \nWe will add the above comparison to the manuscript.\n\n---\n\n**Q3**: Several ablation studies are missing. For example, the effect of pre-training data and the effect of transformer layers in DETR.  \n**A3**: Thanks. The pretraining data consists of REC and Flickr30k and the ablation is shown in the table below. During rebuttal, we further increased the pretraining data by adding Visual Genome (3rd row) and observed another round of improvement.  \n\n| pre-training data | accuracy |\n| ----------------- | -------- |\n| REC               | 79.2     |\n| +Flickr30k        | 79.5     |\n| +Visual Genome    | 79.9     |\n\nThe impact of the number of transformer layers in DETR is presented in the table below. We observe that increasing the number of transformer layers results in a slight improvement in accuracy from 4 to 6 layers, reaching an optimal performance at 6 layers. Considering training efficiency, we use 4 layers in our experiments.\n\n| \\#DETR layers | accuracy |\n| ------------- | -------- |\n| 4             | 79.5     |\n| 5             | 79.7     |\n| 6             | 79.8     |\n| 7             | 79.7     |\n\n---\n\n**Q4**: No failure case analysis and no improvement analysis, compared with late fusion decoder.  \n**A4**: We guess the “late fusion decoder'' you mentioned is the ”understanding $\\rightarrow$ recognition'' methods [1,2,3]. Compared with late fusion decoder, our method used relatively fewer LLM layers (16 out of 32 layers) for informing the DETR head with the object-of-interest. It is possible that in some extreme cases where the object-of-interest is very obscure, our model do not capture the user intention and missed the object.  \nThe advantage of our approach, on the other side, is that the recognition is much more efficient for recognition (REC, in particular). More importantly, our recognition results will be injected into the following LLM layers to assist the understanding. In contrast, the late fusion mode does not benefit understanding. \n\n---\n\n**Q5**: What about using LLM to predict the box in the end?  \n**A5**: Directly using LLM to predict the position is the \"sequential recognition'' manner introduced in Line 29 to 32 in the manuscript and is inferior to ours. One important motivation of this paper is that the sequential recognition manner has low efficiency. Table 4 in the manuscript shows that our method significantly improves the efficiency by about 4 $\\times$ faster. Table 1 in the manuscript shows that our method has higher accuracy (e.g., 2.01 on RefCOCO) in comparison, as well.  \nReviewer Up36 kindly reminds there is another manner, i.e., using localization tokens, as in Kosmos-2. This manner has relatively lower REC accuracy (-5.9 acc. on RefCOCO test set) and is inferior to our method. Please refer to Reviewer Up36 Q2 for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable comments. We are grateful to Reviewer Up36 and Reviewer aS3o for recognizing the novelty of our method, and to Reviewer hBAZ and Reviewer u3Ag for acknowledging its effectiveness and straightforwardness. In the rebuttal, we provide detailed responses to each reviewer’s feedback. We look forward to your further comments. Thank you.

Meta-review: Unlike previous MLLMs which operate in the sequence-to-sequence fashion, this work proposes a new "parallel recognition -> sequential understanding" framework to work better for visual recognition and understanding tasks. The parallel recognition is implemented via object queries, motivated from DETR, and the outputs are forwarded to the latter sequential understanding. 

All reviewers like the idea of "parallel recognition -> sequential understanding" framework and the paper is easy to read. And the main concerns are 1) more experiments need to be conducted to better support the claims and for more fair comparison; 2) the overall improvement over LLava is marginal; 3) limited novelty given some existing works, e.g. "LLava-grounding". After the rebuttal, most of the concerns are resolved, and all reviewers lean to accept. Please follow the reviewers' suggestion to prepare the final version.
Decision: Accept (poster)