Interfacing Foundation Models' Embeddings
Authors: Xueyan Zou
Linjie Li
Jianfeng Wang
Jianwei Yang
Mingyu Ding
Junyi Wei
Zhengyuan Yang
Feng Li
Hao Zhang
Shilong Liu
Arul Aravinthan
Yong Jae Lee
Lijuan Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Foundation Model
Vision and Language
Interleave
Segmentation
Detection
Large Language Model
Large Multimodal Model
Primary area: machine_vision

Abstract: Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in Fig.1, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights.  (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.

Interfacing Foundation Models' Embeddings
Xueyan Zou Linjie Li"F, Jianfeng Wang", Jianwei Yang', Mingyu Ding!, Junyi Wei
Zhengyuan Yang". Feng Li", Hao Zhang", Shilong Liu Arul Aravinthan Yong Jae Lee Lijuan Wang
UW-Madison Microsoft
UC Berkeley HKUST & Tsinghua University
Equal Advisory Contribution
Main Technical Contribution
"Equal Contribution
https:/ /ttl//iittte www.hitt.ccom com/ Soom/UX--r TXX-Decoder/E
FIND
https: /ttps://an github........ com UX Dww.Deeooodrreeeee /vlco
Interleave Segmentation
antmaal known rocerionally
long neck and lngs, and distrastee
oost bronn patovee secarated
lighter
tigher tranchea trens primuntyy
reed teaea and shoots.
Segmentation
Long Segmentation
Interastive Sogmentation
Interleave Grounding
Input
ekiing)
[on the slope] (under]
skiing)
slope] (under)
surfing on)
mon a red jacket and woman
jacket
ridinge surtbard shawcasing balance
also in jacket are shanding ding on
Output
snow-dowered skops They are prepar
anoxboard. They preparing
and control surfooard.
lidee down this anowl mountaiin underithhe bright blue sle!.
under
Interleave Retrieval
Input
playling
aitting bench),
Change
Top2
Top3
Top2
Top1
Output
Figure 1: The proposed FIND interface is generalizable to tasks that span granularity (pixel to image) and modality (vision to language). The retrieval space for this figure is the COCO validation set.
Abstract
Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power foundation models, we present FIND, generalized interface for aligning foundation models' embeddings with unified image and dataset-leve understanding spanning modality and granularity. As shown in Fig. 1 lightweight transformer interface without tuning any founda- tion model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive new tasks, and new models. In light of the interleaved embedding space, we in- which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.
troduce
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Muttimodal:
Intertoove:
X-Decader Vision Encodar
Interleaved
Vision Encoder
Segmentation
Retrieval
Grounding
Visuel
Visual
Visual
Text Visual
(a)
Text
Texrt
Text
Text
Visual + Text
FIND_Inteert Interface
Muttimodil::
Visuel
Visual
Text
Text
LLaMA
Gemma Language Encoder
Mistral
(b)
Languaage Encoder
Language Encoder
Interleave:
Visuai
(1) Interface on Foundation Models
Multimodal Interieave
Figure 2: (1) The concept of interfacing foundation models embedding, the black arrow means active attached modules and the gray arrow means the option that it can switch to. On the right, we show the difference of Multimodal and Interleave (2.a) in the context of embeddings matching; (2.b) in the context of embeddings interaction for reasoning and generation.
Introduction
With the exhilarating progress in foundation models across the vision and language domains, such as GPT4(V) (30), DALLE-3 (31), SAM (I9), and LLaMA (38), etc., we have reached a stage where deep learning models achieve remarkable performances on both vision and language domains (5) Specifically, models like GPT-4(V) (30) have showcased human-level perception and reasoning
skills (46).
Despite their impressive capabilities in information memorization, processing, and reasoning, these models tend to be specialized for specific output types. However, their output types are limited to language for GPT, images for DALLE, masks for SAM, etc. In this work, we aim leverage the privileged properties of foundation models' embeddings to expand their output space (e.g., extend to pixel-level outputs), unlocking their potential for interleaved understanding and reasoning.
To accomplish this, we introduce an INterface for Foundation models' embeDdings (FIND), which utilizes the pre-trained foundational model embeddings to jointly handle downstream tasks of varying granularities (from pixel to image) in an interleaved manner. As illustrated in Fig I, the FIND inter- face processes embeddings from vision and language foundation models, and outputs segmentation, grounding, and retrieval results.
As all vision- language tasks trained uniformly in FIND, interleaved shared embedding space created where vision and language references can be interchanged and augmented. For example, in 2 during mapping interleaved representation loosens the kingle-mddligg constraint the source and target domain. And during reasoning. interleaved sequences enhance information exchange between vision and language compared to multimodal sequences.
To effectively align and evaluate the interleaved embedding space, we construct a new dataset named FIND-Bench. This dataset uses COCO images and includes new annotations for integrated grounding and segmentation. These annotations are generated by GPT-4 which, despite not processing visual input, can directly link specific image segments and annotation IDs with generated descriptions (e.g.. <id>(the golden retriever) ...). This unique capability enables the creation of training and evaluation datasets for retrieval and grounding in interleaved context.
In summary, we claim the following contributions:
We introduce the FIND interface that is is generalizable, flexible, and extendable to various downstream tasks and foundation models.
Through the effective training scheme of FIND, an interleaved shared embedding space is created interfacing foundation models.
We propose a new Benchmark, FIND HND-Benchh, which includes new training and evaluation ground truths for interleave segmentation and retrieval.
Our model achieves SoTA performance on interleave retrieval and grounding and shows better or comparable performance on generic, interactive, grounded segmentation and image-text retrieval.
Related Work
Foundation Models. Recent years have seen speedy evolution of foundation models in diverse areas such computer vision (47), natural language processing (39 10 30), and their interactions (1

---

For example, GPT-3 4 heralds breakthroughs in natural language understanding and generation tasks, As a vision foundation model, Florence (47 42) can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, etc. eta.innmi (I) bridges powerful pre-trained vision-only and language-only models by token fusion with cross-attention. BLIP-2 (23) proposes an efficient pretraining strategy that bootstraps vision-languanl pre-training with lightweight Q-Former in two stages. Different from previous multi-modal approaches, such as Flamingo ①. LLa VA (26) and Q-Former (BLIP-2) (23) that feed the vision foundation model output into a language decoder and use the LLM as an interpreter, our goal is to interface foundation model embeddings that LLMs and vision models can be unified in the embedding space.
23 44).
Interleaved Image-Text Understanding. Previous works have explored interleaved visual un- derstanding in the context of visual question answering, visual dialogue, image captioning, and interleaved image retrieval (20) 13: T. In addition, recent works (48) explore contextual detection that associates phrases with visual content in sentence. We notice that these earlier works, though reveal interleaved capabilities for image understanding, lack an evaluation benchmark, as well as a complete training dataset. (51 21 ( propose a new benchmark on interleaved generation and under- standing of image and document level, while there is no benchmark available for the interleaved tasks between interactive image parts and phrases. To this end, we introduce the interleaved segmentation and interleaved retrieval tasks with our carefully designed benchmark FIND "IND-Bench, which we believe to be essential for the field.
33 have dominated wide range of key image understanding tasks, such as image retrieval, detection, and segmentation. Some multimodal methods 7 24 50) have shown good performance for retrieval tasks. On the other hand, open-vocabulary segmentation methods have recently drawn much attention, including generic segmentation (6 53 (D) interactive segmentation [4 that separates objects by actively integrating user inputs, and grounded segmentation (53) 52 that grounds object segments from language descriptions. We notice that there is currently no available work that achieves image-level retrieval, pixel-level segmentation, and interleaved vision-language In this work, we propose FIND as a unified interface that can support all the above tasks, while maintaining good performance, and further enabling two new tasks of interleaved segmentation and interleaved retrieval. We unify these tasks by interfacing foundation models' embeddings.
Image Understanding. Vision Transformers (6 40
understanding in a single model.
3 Method
Foundation models such as CLIP (32, SAM (9), LLaMA (38), etc. can process vision or language inputs for reasoning, understanding, and generation. The embeddings generated by these models contain rich and structured information (35 3), making them extremely well-suited for understanding tasks. Aligned with the Platonic Representation Hypothesis (17), we believe foundation models can easily communicate with each other. Therefore, we designed the FIND interface to project vision and language embeddings from foundation models into unified space. The created space enhances both multimodal and interleaved understanding.
Since no prior benchmark exists for interleave understanding, we believe it is meaningful formally define the interleave retrieval and segmentation problems and create a dataset for benchmarking them.
3.1 FIND Benchmark
Our new benchmark supports two tasks: interleave retrieval and interleave grounding. It evaluates both dataset-level and image-level interleave alignment, focusing on reasoning and matching capabilities. Additionally, we created training and evaluation datasets to further enhance interleave understanding.
3.1.1 Task Definition
Interleave Retrieval! [ An interleave entry (F) consists of sequence of images (I), texts (T), and connections (C), and can be represented as F = (Na, N2,..., ,N, N; E (T, C)), where (-) is an ordered sequence. The bottom part of the Table. ?? clearly illustrates an example of an interleave entry. We denote the source domain (D.) of interleave retrieval as D. = E Eg,... En }. as shown
in Fig. 3 (Left), and the target domain (D,) as D; = {41, 12,... (2,....,..,, Jn). J. as shown in Fig. 3 (Right).
The task of interleave retrieval is find the closest entry I. E D, for each E E D., excluding itself. Formally, we define this as VE E D., I, arg max 1a. ().
Unless we stated as interleave text retrieval, we refer interleave visual retrieval Fig. shown.
3

---

Context for Image
GT: Ground Truth image caption labeled by human.
2. PD: Pscudo image Description generated by VLM model.
3. Box: All Ground truth bounding box labeled by human.
SI: Segment Information for cach box arca including index, bbox, category, descriptions, etc.
SP: Segment Proposal for the generated description.
Prompt for GPT4 Engine
''Generate image captions with grounded entities and attributes with the following information:
ground truth image captions: <(}>,
pseudo image description: <{)>. ground truth bounding boxes ([r0, Vo, /t]: (xo. yo) is the top-left corner; (wx, h) is box size); segment_ingg j_nff: <[ { and segment_ pproppsstrpool:::::: <{}>.
An example output format would be: "[index]<A <A womanz sitting next to findex]<a handsome man>,
with their hands holding together under [index]<the blue sky>.". where [index] and <xx>> are associated with the ground truth bounding boxes.
Generated caption constraints: (1-6) Please refer appendix." format(GT, PD, Box, SI, SP)
Retrieve Visual Sample with SEEM
Given the scarch dataset (Q) with the segments in all images denoted as (SD), we compute all embed- dings S representing each segment using SEEM (53) with S SEEM(SD) E Baxd Given the similarity matrix W
x S where Wy represents the similarity between segment and segment j. the index of the closest segment for segment i is Match(i) = arg m W, where Match(E) returns the index that has the highest similarity to segment i.
Integrated Response of GPT4 and SEEM
[5721674|<A baseball player in black and white uniform> crouches (1345187]<e sandy
ground> near 1t) playing fields,
holding a [1778208]<blacck leather bascball
glove>
taking break.
Table I: Pseudo code for Data Engine. We show the pipeline to create the FIND-Bench from data preparation, text prompting using GPT4, visual prompting with SEEM to integrated result.
Interleave Grounding Fl An image contains sequence of objects segments (O) represented as
I [O,, Oz...................... On). We provide an example of objects the bakery image Fig. 3 upper part. These objects form the target domain D, = 7 [O,, Oz....,
for interleave grounding. Unlike interleave retrieval, where interleave entries constitute the source domain, interleave grounding focuses each component of the interleave entry, with the entities (N) in the interleave entry forming source domain. Specifically, D. = [N1, .N...... N₁, N₂ E {1,T}} C. E. We show an example of interleave entry decomposition in the lower part of Fig. 3 The task of interleave grounding is to find the closest entry O. E D, for each E D,,, excluding itself. Formally, we define this as VN E D., O, arg oo O).
3.1.2 Data Engine
We reuse the images and ground truth annotations from the COCO dataset to create FIND-Bench. In the first part of Table. 1 demonstrate the input data used in-context learning for GPT-4. In addition to the COCO ground truth, we generate pseudo-image descriptions using VLM models, such LLaVA (26). to enrich the information. In the second part of Table. [ we present the prompt template for our data engine. This template generates the text part for the interleaved captions in part 4 of Table. [ providing language descriptions associated with annotation IDs. The segments corresponding to these IDs are highlighted in the same color in the example image shown in Table. [
As stated in Sec. 3.1.1 the source and target components are exclusive. We leverage the strong visual understanding capabilities of SEEM (53) to find replacements for the visual components in the entry. The retrieved and replaced visual components are shown in part 4 of Table. T with the exact segment highlighted in the same color as the corresponding reference text. For example, <the playing field> is
Unless we stated as interleave text grounding, refer to interleave visual grounding Fig. 3 shown.

---

Image::
Text:
Connection: C
071.0% [CI
O:
O.
02
D3
UQ
(1) [c]
4
[C]
[C]
wooden surface holds donuts with diverse icings-pan pink
playing with red frisbee)
phocolats, minbow donus,
Datrich, Girafiii
(1) Interlcave Retrieval
(2) Interleave Grounding
(3) Interactive Segmentation
(4) Generic Segmentation
Figure Task Unification for retrieval, grounding. and segmentation. The corresponding components
are labeled with the same color or connected with a line arrow.
associated with the COCO annotation ID [3171 126] and similar playing field (marked in blue) in another image. In this way, the data engine can generate comprehensive interleaved descriptions for each image the COCO dataset. This is sufficient build D, and D, for the interleave retrieval and grounding tasks introduced in Sec. 3.1.1
3.2 FIND Approach
With benchmarks introduced in Sec. 3.1 to evaluate the model's interleaved visual understanding ca- pability, now present our approach for interfacing foundation models' embeddings multimodal and interleave understanding. We begin with the preliminaries on task unification and terminology.
3.2.1 Preliminary
Task Unification. In this work, we focus retrieval, grounding, and segmentation in both multimodal and interleaved manners. In Fig. we demonstrate four example tasks: interleave retrieval, interleave grounding, interactive segmentation, and generic segmentation. From abstract perspective, can regard all visual understanding tasks as the problem of matching candidates from the source domain to the target domain. Formally, define the source domain D, and the target domain as D. Example elements in D, or Dr includes interleaved entry E, an image 7, an object or segment O, texts T. For each visual understanding task U(D,, D,), the goal is find the closest Y DD for each X E DD, Formally we write:
vX E Y arg max sim (X,Y)
Y YEP
where X, and Y are base element of D., and D, respectively, and sim(X, Y) denotes the similarity between and : For example, in generic segmentation (Fig. 3),, D, is the set all objects (seg- ments) in the image: D fO1.. {O1..... ....... On,,) } and D, is the set of category names: D, (Ti, ....... Tn D.
For each object in D., we will find the corresponding category
Terminology. Here we will introduce important model terminology, including prompts (P) and queries (Q). Our model supports three kinds of inputs: vision (I), language (T), and interleaved Vasion-language (E). The vision and language foundation models predict the embeddings for those inputs. As shown in Fig. I, by sampling the embeddings, we obtain vision prompts (Pr), language prompts (PT), and interleave prompts (Pg). Additionally, trainable queries initialized with random parameters will accumulate information from the prompts. For example, in generic segmentation, object queries (Qo) gather information from visual prompts. Interestingly, queries just act like "buckets" accumulating "water" (prompts) in the FIND interface, as shown Fig. 41.
3.2.2 Model Pipeline
Our model designed to interface with a pair of arbitrary vision and language foundation models. Prompts and Queries Preparation. Given image (I), text (T), and interleave (F) inputs, the vision encoder (F.) and language encoder (Fi) will encode these inputs to sequences of embeddings M:
M, = (T),), My = FT( ME = FF.F (E)
where, M R and n. d is the embedding number and dimension respectively. Similar to SEEM (53), we use an embedding sampler to sample customized prompts for downstream tasks. Example sampling strategies include downsampling, ROI pooling for the region, and rearrangement of embeddings for interleave prompt. The sampling procedure does not alter the embedding distribu- tion. After sampling, we obtain (Pr. Pr, .)....} Emb_ Laib_SSmple(N)) (M, Mr. MF). Additionally, the embedding sampler is responsible for sampling queries ([QF, QT, (.....)) from the pool of
learnable queries. We allow duplication in the sampling procedure of learnable queries. These queries
5

---

[Boforo Interface]
playing with)
Empty Query
[i friddww)).
Leamable Queries
Human Input
OO00
(.T.))
Vision Encoder
LLM Encoder
Content Attention
[nterface]
Mocel
EmbeddingSomppee
Add Norm
Sampler
Vision, Language, interlavee Prompta
- O
Quarieaa
O00%
Maiio]
Conditional.o ConditionalAAttntioo Attention
O
FING Interfasa@Prompth, Quersss, Alton Heak)
[Languudg]
Add Norm
D0-0 Object
- Languugg
Queries
FFN
Interleave]
Interleavee
Imagle
Caption Interlewve
O
|Ahter interficc))
Semante Propection
Plool Projection
Prompra
Retiievall
Grounding
Segmentation
(1) Pholiminary
(2) Model Pipeline
FIND Interface
Figure 4: (a) Preliminaries on the terminology of prompts and queries. (b) FIND approach pipeline.
The shape of different polygons represents different embedding types, and the color (vision, language) of the polygons represents input modality. (c) Detailed architecture of the FIND Interface.
and prompts are the inputs of FIND interface. Technically, the embedding sampler is usually an interpolation or grid sample layer in PyTorch.
FIND Interface. The FIND interface primarily consists of two operations: content attention A, and conditional attention Ad, shown in Fig. 4. Content attention allows queries to accumulate information from the corresponding prompts, while conditional attention enables prompts and queries to reason internally (e.g. self-attention object queries to avoid duplication). With initial prompts p = {P E, po, PP...... J. and initial learnable queries Q² = (Q) Q7, Qi...... content attention and conditional attention are formally defined as:
(2) where St C (Pi, Q') is subset of queries and prompts, - represents the attention mask. For example, - Q] means that Q is able to attend P during the attention. In this way, prompts act as the information source, and queries act as the bucket. In Fig, 42, we unfold the prompts and queries for some tasks supported by FIND interface.
Q+11 = AP(P Q"; [P' - Q')), Q't1, p Aa(P', Q¹: + (',(P''P'))
+
Projection The outputs of the FIND interface are a sequence of queries: Q² (Q), O/ Q "/.....) We then project the queries using linear layers, MLP, and MLP,,. for semantic and pixel projection, respectively. The semantic and pixel queries are computed as RR and Q" = MLP,(Q")) E R= where n, is the total instance number, and d is the embedding dimension. The semantic outputs are used for retrieval, category mapping, etc., while the pixel outputs are used for mask prediction.
Q' MLP, (Q")
Task Head With the projected queries, as illustrated Sec. 1.2.1 each understanding task can be repre- sented as a similarity mapping procedure. Formally, segmentation result (Mask) can be computed given initial image embedding M, E IR" xd where n2p is the pixel number. The similarity scores (Score) can be computed directly from Q*. The outputs for each task a subset of (Mask, Score}.
Mask = Q" M,T E Rm,x", Score Q" x Q =
Loss FIND is trained with a linear combination of losses for panoptic segmentation, grounded segmentation, interactive segmentation, image-text retrieval, interleave retrieval with visual entities from the same image, and interleave grounding. We demonstrate the loss details in the Appendix.
4 Experiments
Datasets. We use COCO (25) as our main training and evaluation dataset, which spans diverse annotation types. We make use of the annotations from COCO-panoptic, Ref-COCO 45 28 29). coco-Karpathy (18), and the new datasets generated with the data engine in FIND-Bench. We generate two sets of new annotations, including COCO-Entity and COCO-Paragrany the detailed
statistics are shown in the table below:
Trainin
Evalustion
Eniiyy Assaciation
Average
Images Capiion Emilies Imagge
Captions
Eniises
Mask Phinse Visua
EmiiyyTimge
35219
4090
47500
TST1S
COcO-Pangraph
Settings. We benchmark our method on three different model sizes: Tiny (FocalNet), Base (Davit-d3), and Large (Davit-d3). The vision backbone is fixed and reuses the X-Decoder pre-trained weights

---

RLEDCCOL COCO REAN EXCO Paozed
COCO Par
SAM 10 21
566
(M6
SAM
SAM-1190
550
23333 08
pact-at 00 1386
5i1 16.1
56.3
050
696
535
56
534
23
15.5
X-ccaee (1) D4
COcOeIT (2:2
52A 61.
124
48.2
SARVE22
5657663
S.weele La
MAY
123
566
7387
215
334
-13
2ND-D
1700
510
011
5641
11.1
02.
485
850
150
65.
967
174
774
Table 2: Benchmark on Generalizable multi-modal understanding tasks with one model architecture joint training for all. *Unlike Mask2Former and SEEM, FIND is not trained with a deformable vision encoder. We report un-ensemblleenseemble results for X-Decoder, and the finetuned/pre-trainee results for blip2. Note that we compute the ITC score for blip2 instead of ITM.
unless specified as SAM. The language backbone is a fixed LLaMA- unless specified as UniCL. During training, we train the FIND-Interface jointly on all the tasks unless specified.
Metrics. We evaluate all the tasks with their standard evaluation metries. For the newly proposed interleave retrieval, we use IR @5 and IR @ 10 interleave-to-image Retrieval accuracy at rank 5/10). For interleave grounding, we evaluate based cloU (pixel-wise IoU), and mloU (image-wis, wise IoU) between the predicted interleave masks and the ground truth masks.
Baselines. We use ImageBind (3), FROMAGe (20), BLIP2 (23) as baselines for the interleave retrieval task; Grounding-SAA -AM (27). SEEM (53) for interleave grounding. We claim to make every effort to design the baseline evaluation protocol to achieve the best possible performance.
4.1
Main Results
In the main experiments, we focus on evaluating FIND on Generalizable, Interleavable, and Extend- able capabilities as claimed in the abstract.
(1) Generalizable Segmentation, Grounding, and Retrieval. Table comppres FIND with strong baselines on generic segmentation tasks including panoptic segmentation, instance segmentation, and semantic segmentation. In addition, we demonstrate the segmentation capability in both referring segmentation (RefCOCO-g: one sentence is associated with one instance) and grounded segmen- tation (COCO-Entity and COCO-Paragraph. one sentence is associated with multiple instances) settings. Moreover, we also benchmark FIND's performance in image-text retrieval on three different ground truth types on COCO, where the average sentence length for the splits (Karpathy, Entity, and Paragraph) gradually increases. Below are the takeaways:
The instance segmentation resull stands out: similar models like Mask2Former, X-Decoder, and SEEM, achieving a performance 2.2 points higher than Fask2Former (L), which additionally uses deformable convolution. Notably, the segmentation training data is identical for both Mask2Former and FIND. The performance gain likely results from our unified segmentation and grounding pipeline, which mutually benefits from the semantic ground truth of each domain.
Our approach with large vision encoder outperforms
Mutual benefits of grounded and referring segmentation: In FIND, unify grounded and referring segmentation using queries and prompts. As shown in Table 2 our model achieves state-of of the- art performance on COCO-Entity and COCO-Paragrayy and outperforms strong baselines on the Ref-COCOg dataset.
Interactive segmentation performance is preserved in the unified settings. Unlike SEEM which is only trained on image-only tasks, FIND is trained also on image-text tasks, such as image-text retrieval. With the smart design of queries, prompts, and attention mechanisms, training interactive segmentation and image-text retrieval does not interfere. Thus, it enables our approach to achieve competitive performances (i.e. FIND 88 08.55/8.557777 vs. SEEM 88. $8.5//9.6676655)
Less optimal image-text retrieval results: The sub-optimal performance in image-text retrieval is due to batch size during fine-tuning. Pilot experiments with X X-Decoder showed that different resolutions (e.g., 1024 for images and 224 for language) do not generalize well across tasks. Thus, FIND is trained with the same resolution for all tasks. In Table models are either 384x384 with batch size 384 or x024x 1024 with batch size 192 for all tasks. Other tables show results with a 640x640 training resolution and 192 batch size.
7

---

Coco-Esily
COCD-Purge
COCO Beily
Coco-Burpahh
Cla
Visal Cuetes
Destisiie
zeli
mit
Ar50
del
AP5
1885
RAA
mist
PV
RAA
nkll
TAP
Guanding SxM
(81)g 1g
58.9
557
652
561
56.6
62.
D50
51.
30.8 7323
25.8/477
22.
27.115
2 52
33A
413
61A
24.7
2638
16.2
43.4
578
08.6
53.4
23.3
23.1
21.1
87.6
15-8
66.3
66.3
923
522
163
15.
16.0
17.5
SIEM 113 53
08.3
60.0
17.8
673
654
7710
96.9
46.
83.5
260
26-4
18.1
73.2
6200
(END
81.8
65.
70.7
51.1
M4
13.4
55.
080
516
26.7
36.1
HIND
76.3
66.7
81.7
76.3
00.6
787
534
66.7
427
754
50.7
5L6
53.
41.
36.1
57.1
27.8
33.2
26.8
Table 3: Benchmark on interleaved understanding with the jointly trained model on all tasks with one
set of weights. We evaluate interleave grounding, retrieval, and generic segmentation.
Generic Segmentation
Grounding
Interactive
Retrieval
Class
Description mAP
g-Ref
VOC
COCO-Karpatly
Vision
Language
PQ
mAP
mloU
PQ
mloU
cloU
I-loU
IR@1
TR@ I
X-Decoder (T)
52
UniCL 43
48.5
39.0
61.4
12.4
20.7
18.9
61.3
82.6
40.4
54.0
X-Decoder (T) (52)
LLaMa 38
48.5
38.9
61.2
19.5
30.2
35.5
61.6
82.5
40.2
52.2
SAM (B) 19
UniCL (4
42.5
37.6
53.6
4.5
17.7
17.9
64.9
81.6
29.1
39.5
SAM (B) 19
LLaMa (38
42.5
36.9
53.0
6.1
15.6
16.6
58.9
81.5
27.0
35.5
Table 4: Ablation study on different foundation model architectures.
(2) Interleavable on vision and language modalities. In Table. 3 we evaluate FIND on the interleaved dataset- and image-level understanding tasks in FIND-Bench. In the columns of COCO- Entity and EDCO-Paragrapp we replace the text entity with visual reference on 0.5 probability, unlike Table. 2 the columns are purely evaluated on language- data.
Interleaved Segmentation: We build an interleaved segmentation baseline using the SEEM model. Instead of formulating the grounding task in an interleaved format that SEEM doesn't support. we simply separately infer visual, and text entities using the interactive or grounding function of SEEM. As shown in Table 3 FIND outperforms SEEM on interleave segmentation with around +8 points on both COCO-Entic and COCO-Paragrapy under cloU metrics.
Interleaved Retrieval: We also explore cross-image interleave retrieval on FIND, Since the interleaved reference objects are from the same validation set, IR@ 1 is not meaningful, so we report @ and @ @ 10 in this setting. For ImageBind and BLIP-2, we use ensemble scores of texts, sentences, and images. Following FROMAGe's ' settings for interleaved image-text retrieval, our performance significantly higher than the baselines, demonstrating the effectiveness of our interleaved shared embedding space.
Generic Segmentation: Beyond classic evaluations using class names fixed indices, we replace categories with class descriptions (long descriptions) or visual prompts (average features for object queries for each class). Leveraging LLMs, FIND excels in description- bbased segmentation, benefiting from smoother representations and better handling of long contexts. We also demonstrate FIND's effectiveness in the visual context setting.
(3) Extendable to arbitrary foundation models and tasks. In the main experiments, we use X- Decoder as the vision encoder, and LLaMA as the language encoder, which shows convincing performance on all the tasks. X-Decoder has been trained to pair up vision and language embeddings, however, SAM is only trained segmentation data without any semantic meaning. Thus, SAM as an ablation vision foundation model, to study how important is vision encoder trained with semantic data. For the language encoder, we adopt UniCL which has the same size Bert study the difference between a standard language encoder, and an LLM encoder. As shown in Table 4 UniCL and LLaMA usually have very similar performance with X-Decoder as vision encoder, except that LLaMA is extremely effective on long description reasoning. Although the performance of SAM much worse than its counterpart X-Decoder on semantic understanding after training the interface, our approach also shows that without any modification to SAM, it applies to semantic understanding tasks on generie, grounded segmentation, and image-text retrieval.
4.2 Ablation Study
We ablate our approach from two perspectives: (1) What is the effectiveness of each task in the unified pipeline? (2) The effectiveness of using intermediate layers of the L.M representation.
Independent task effectiveness: We assess task effectiveness by gradually removing tasks in Table 5 Removing image-text retrieval significantly reduces interleave retrieval performance. Further remov-
8

---

COCO
g.Kel
Emity cloU
Karpulby
Enitiy
PQ
mAF
mloU
cloU
Point
IR:11
TR
1R.61
TR
73.0
54.0
Retrieval
48.5
39.0
61.1
60.6
73.2
82.8
44.3
44.8
Task
Grounding
48.6
39.1
61.3
40.9
82.8
45.3
46.2
Interactive
48.6
38.8
61.0
36.5
31.4
33.4
Interleave
48.9
39.3
61.0
[-11
7350
Language
[-6]
47.8
38.8
60.4
60.3
72.9
81.3
38.
49.9
48.1
47.5
Lovel
[-12]
48.5
39.0
61.4
61.3
73.0
82.6
40.4
54.0
50.8
51.9
[-18]
72.6
[-24]
48.5
38.8
61.5
61.6
72.9
82.6
40.2
52.2
50.5
51.3
[-30]
48.1
39.2
61.1
60.1
73.3
82.4
37.9
49.3
49.4
50.0
Table 5: Ablate each training task and language encoder feature level.
ing the grounding task decreases entity-based grounding performance. Since interleave grounding is
related to interactive segmentation, removing it also reduces interleave segmentation performance. Finally, training only panoptic segmentation yields similar performance to other settings, indicating the unified interface's ' consistency with basic task training.
Varying the feature embeddings layer for LLM: LLMs process language tokens, with embeddings near input and output layers being less semantic. We hypothesize that intermediate layers align better with vision embeddings. Table shows performance across tasks using emebddings from layers -1 (output) to -30 (input). Layer- -12 emebddings perform best, while top and bottom layers perform worse for image-text retrieval on COCO-Karparti splits. Thus, we use layer -12 emehddings for LLLaA throughout the paper.
4.3 Demonstration Results
Interleave Album Search. The queries in our FIND approach support linear complexity interleave album search. Given image, interleave, text input, model retrieve and segment all the photos in the album. Below, we show an example using the COCO validation set as the search space.
ls standing on the rook under the trees.
Interleave Video Localization. We formulate the video frame localization problem maage-text retrieval task. This allows us to reason about and identify corresponding objects based on given instructions, as illustrated below. We believe FIND is useful for robot navigation.
you green
3D Feature Field. Foundation model embeddings are utilized to create a 3D feature field for robot manipulation, localization, and reasoning. We believe that the interleave embedding space, with its pixel-level understanding capabilities, has significant potential in the 3D feature field. Below, we compare scene trained with FIND embeddings versus CLIP embeddings.
FIND 3D Feature Field
CLIP 3D Faature Feeld
Conclusions and Future Work. This work introduces the FIND Interface, generalized interface for aligning foundation models' embeddings, along with the FIND Benchmark for training and evaluation. In Sec. 4.3 demonstrate potential applications such interleave album search, video localization, and 3D feature fields. These examples clearly illustrate the potential of our model for personalized foundation models and robotics.
Linitations. Our model is only trained and evaluated on the COCO dataset. With the limitation of data quantity, mention that the method may not be well adapted to the in-the-wild settings.
Broader Impact. Our proposed approach inherits ethical or social issues (e.g. bias amplification, privacy risks, energy consumption) of foundational models.

---

Acknowledgement. This work was supported in part by NSF CAREER IIS2150012, NASA 8ONSSC2IK0295 the Institute of Information and communications Technology Planning and Evalu- ation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of Al Autonomy and Knowledge Enhancement for AT Agent Collaboration). This research project has benefitted from the Microsoft Accelerate Foundation Models Research (AFMR) grant program.
References
[1] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A.,
Millican, K., Reynolds, M., et al.: Flamingo: visual language model for few-shot learning.
Advances in Neural Information Processing Systems 35, 23716-23736 (2022)
[2] An, J., Yang, Z. Li, L.,, Wang, J., Lin, K., Liu, Z., Wang, L., Luo, J.: Openleaf: Open-domain interleaved image-text generation and evaluation. arXiv preprint arXiv:2310.07749 (2023)
[3] BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., Reddy,
S.: Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.0596 (2024)
[4] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877-1901 (2020)
[5] Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E.,, Lee, P., Lee, Y.T., Li, Y., Lundberg, S., et al.: Sparks artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 121.::333....1222... (2023)
[6] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint Xix: 1atiii::1060000 1706055877 (2017)
[7] Chen, Y., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: UNITER: universal image-text representation learning. In: ECCV. vol. 12375, pp. 104-120 (2020)
[8] Cheng, B., Misra, I., Schwing. A.G., Kirillov, A., Girdhar, R.: Masked-attention mask trans- former for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290-12902 1290-12990000 (2022)
[9] Cherti. M. Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2818-2829 (2023)
[10] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (I) (2019)
[11] Ding, M., Lian, X., Yang, L., Wang, P., Jin, X., Lu, Z., Luo, P.: Hr-nas: Searching efficient high- resolution neural architectures with lightweight transformers. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 2982 2982-2992 (2021)
[12] Ding, M., Xiao, B., Codella, N., Luo, P., Wang, J., Yuan, L.: Davit: Dual attention vision transformers. In: European Conference on Computer Vision. pp. 74-92. Springer (2022)
[13] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, Imagebind: One embedding space to bind them all. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 15180-15190 (2023)
[14] Grady, L.: Random walks for image segmentation. 1EEE transactions on pattern analysis and machine intelligence 28(11), 1768-1 1783 (2006)
[15] Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jégou, H. Douze, M.: Levit: vision transformer convnet's clothing for faster inference. In: ICCV, pp. 12259-12269 (2021)
[16] Heo, B., Yun, S., Han, D., Chun, S., Choe, J., Oh, S.I.: Rethinking spatial dimensions of vision transformers. In: ICCV. pp. 11936- 11966--1995511111 (2021)
10

---

[17] Huh, M.. Cheung, B., Wang, T., Isola, P.: The platonic representation hypothesis. arXiv preprint arXiv:2405.07987 (2024)
[18]
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3128- 3137 (2015)
[19] Kirillov, A., Mintun, E.,, Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02443 (2023)
[20] Koh, J.Y., Salakhutdinov, R., Fried, D.: Grounding language models to images for multimodal inputs and outputs (2023)
[21] Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A.M., Kiela, D., et al.: Obelics: An open web-scale filtered dataset of interleaved image-text documents. In: Thirty-sevent Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023)
[22] Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L.,, Gao, J.: Multimodal foundation models: From specialists general-purpose assistants. arXiv preprint arXiv:2309. 1, 2 (2023)
[23] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.1259) (2023)
[24] Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L.., Wang, L.., Hu, H., Dong. L., Wei, F., Choi, Y., Gao, J.: Oscar: Object-semantics aligned pre-training for vision- rision-language tasks. In: ECCV. pp. 121-137 (2020)
[25] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer VisVon-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer (2014)
[26] Liu, H., Li, C., Wu, Q., Lee, YJJ.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023)
[27] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0 05499 (2023)
[28] Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 11-20 (2016)
[29] Nagaraja, V.K., Morariu, V.I., Davis, L.S.: Modeling context between objects for referring expression understanding. In: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 1-14, 2016, Proceedings, Part IV 14. pp. 792-807. Springer (2016)
[30] OpenAl: Gpt-4 technical report. Tech. rep., OpenAl (2023)
[31] OpenAl: Improving image generation with better captions. Tech. rep., OpenAl (2023)
[32] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. 8748-8763. PMLR (2021)
[33] Riquelme, C., Puigcerver, J., Mustafa, B.. Neumann, M., Jenatton, R.. Susano Pinto, A., Keysers, D., Houlsby, N.: Scaling vision with sparse mixture of experts. NeurIPS 34 (2021)
[34] Ryoo, M.S., Piergiovanni, A., Arnab, A., Dehghani, M., Angelova, A.: Tokenlearner: What can 8 learned tokens do for images and videos? arXiv: Computer Vision and Pattern Recognition (2021)
11

---

[35] Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., Khandeparkaa, H.: A theoretical analysis of contrastive unsupervised representation learning. In: International Conference on Machine Learning. pp. 5628 3628-5637. PMLR (2019)
[36] Srinivas, A., Lin, T.Y., Parmar, N., Shlens, J., Abbeel, P. Vaswani, A.: Bottleneck transformers for visual recognition. In: CVPR. pp. 16519 165299 (2021)
[37] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: ICML. pp. 10347-10357. PMLR (2021)
[38] Touvron, H.,, Lavril, T., Izacard, G., Martinet, X., Lachaux. M.A., Lacroix, T., Rozière, B.. Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 13971 (2023)
[39] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I:: Attention is all you need. In: NeurIPS (2017)
[40] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: ICCV (2021)
[41] Wu, K., Peng, H., Chen, M., Fu, J., Chao, H.: Rethinking and improving relative position encoding for vision transformer. In: ICCV. pp. 10033-10041 (2021)
[42] Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., Yuan, L.: Florence-2: Advancing unified representation for a variety of vision tasks. arXiv preprint arXiv:2311 1,66422 (2023)
[43] Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, L., Gao, J.: Unified contrastive learning in image-text-label space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19163-19173 (2022)
[44] Yao, L.,, Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang. X., Li, Z., Jiang, X., Xu, C.: FILIP: fine-grained interactive language-image pre-training. In: ICLR (2022)
[45] Yu, I.., Poirson, P., Yang, S., Berg, A.C., Berg, T.I.: Modeling context in referring expressions. In: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer (2016)
[46] Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: Mm-vet: Evaluating large multimodal models for integrated capabilities (2023)
[47] Yuan, L., Chen, D., Chen, Y.L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al.: Florence: new foundation model for computer vision. arXiv preprint arX (2021)
[48] Zang, Y., Li, W., Han, J., Zhou, K., Loy, C.C.: Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305.1822 (2023)
[49] Zhai, X., Kolesnikov, A., Houlsby, N., Beyer, Scaling vision transformers. arXiv: Computer Vision and Pattern Recognition (2021)
[50] Zhang, P., Li, X., Hu, X., Yang. J., Zhang, L., Wang, L., Choi, Y., Gao, J.: Vinvl: Making visual representations matter in vasion-language models. arXiv preprint arXiv:2101 100009990 (2021)
[51] Zhu, W., Hessel, J., Awadalla, A., Gadre, S.Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang. W.Y., Choi, Y.: Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06933 (2023)
[52] Zou, X., Dou, Z.Y., Yang, J., Gan, Z., Li, L., Li, C., Dai, X., Behl, H., Wang, J., Yuan, L., et al.: Generalized decoding for pixel, image, and language. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15116-15127 (2023)
[53] Zou, X., Yang, J., Zhang, H., Li, F,, Li, L.., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. anXiv preprint arXiv:2304.06711 18 (2023)
12

---

A Method Details
A.1 Task Specific Interface
In Section 3.2.2 we provided a comprehensive overview of the general pipeline of FIND. Here, we focus on the task-specific interface design choices. The pipeline comprises three main components: (I) Embeddings, which include prompts and queries as introduced in Section 3.2.2 Prompts are multimodal embeddings containing relevant information, while queries are learnable embeddings that aggregate information from the prompts. For instance, for image prompts (a.k.a visual features of an image) we denote them p. image. (2) Operators, which incorporate both content and condition attention, and are responsible for information accumulation and exchange. The arrows +, denote the attention direction. (3) Projection, which maps the queries into semantic or pixel space. Table. 6 below shows details of all task-specific design choices for the FIND interface, including embeddings, operators, and projection.
Task
Prompts
Tokenization
Queries
Content Atlenionn
Reasoning
Coudition Attention
Defokenization Projection
Generic Segmentation
clana objectt, cl
9.clasa p.clasu
9.00joct p.
p.
Pixel, Serantic
Grouaded Segmentation
ixage, isage,
grounding,
grommding p-trage
E.grounding p.text
Pixel, Besantic
Image Text
Leage, caption
inage, captica
caption caption
g-inage image
p.* p.*, q.* to q.*
Bemantic
Interactive Segmentation
image, epatial
cegnent, epatial
q. 9egaaat piaage
sppptttl 4- p.spatial
P.* p.*. q.* q.*
q. q.segrent e- p.spatsinl
Fixel, Semantie
Interlove Grecaddin
trage, interleave Laave
antity, interlaave
9 Leaver 6 pp. inte:llave
9. eatity + p. p.image
p.v e p.*, q. ++ q."
quaatlly 4- p.interlen Pixel, Serancic
Interleove Retrieval image, interleane
inage.
interllaaee
9._ iatorleave + intezleare
g-3mage + p.ieage
p." p.t. 9. q.
Semantic
Table Task specific FIND Interface. We define each task under the prototype of the FIND interface that enables a shared embedding space, and a unified and flexible architecture for future tasks. Where p. stands for prompts, queries, and arrows stand for attention direction. The colors red, blue, and olive are the embeddings of vision, language, and interleave modality.
A.2 Loss Functions
The training tasks include panoptic segmentation, interactive segmentation, grounded segmentation, image-text retrieval, interleave retrieval with visual entities from the same image, and interleave grounding. Losses for each task are standardized loss functions including LBCE for binary cross- entropy loss, Len for cross-entropy loss, LDICE for dice loss, Le for contrastive loss. Below the loss function for FIND.
L CppFCCR_p panno pano + BpLBCR_ + "pCDICF_Pmm pano + 0gg CtFCFR + Big BgCCCF grd + 7igCrICE gnd + a, + 3,CBCE_ive isg + 7:CDiCE_sse + @LvLc_imgex + iftr_intr + CinfCEE_int + Big LDICE_ + Yig Licr
where, pano denotes panoptic segmentation, grd denotes grounding, iseg denotes interactive segmentation. imgtextr denotes image-text retrieval, intr denotes interleave retrieval, intg denotes interleave grounding. For more implementation details on the loss function, please refer to the code.
A.3 Case Study: Interleave Grounding
As shown in Table. 6 the input embeddings of interleave groundings for FIND interface contain prompts and queries. Image prompts are the image features with a shape of [h x w, 512], while interleaved prompts are langal-language tokens of sentences like "A baseball player in black and
white uniform crouches on near the token length). Entity queries are learnable embeddings for object proposals of the image, shaped 1000, 512). Interleave queries are learnable embeddings for gathering information from the interleave prompts, shaped [n, 512), where is the total number of meaningful entities. For example, the interleave sentence shown above has entity numbers of 4. Specifically the entity contains ['A
holding a taking break." with shape of [r. 512] (1
13

---

baseball player in black and white uniform, referencing to Fig. S
which the total number of [2' and [4]
After getting full sense of the imput embeddings of interleave grounding, including P image. p. inteerleaee, q qeeniity q iinterleave. We then introduce the operation on top of those embeddings. As introduced in Sec. 3.2.2] the operations contain content attention A, and conditional attention Ag. Formally we could write the attention mechanism for the specific input embeddings of interleave grounding with the following equations:
q. eneiity, q. g.interleave
entity, .q. interleave]; [p. image, interleave] M.);
(5)
q.*, = A.(q., p p] [q.*, p. M.+]:Ma)
(6)
where A(query; key=value; M) is the attention operator with query, key, value and mask. Given the order p. image, p. interleave, q entity, q interleave, the content and condition attention masks written below:
F
F
F
F
F
F
F
F
F
F
M.
M
(7)
F
T
F
F
T
F
T
The index of matrix coordinates follows the input order. After the input prompts and queries fully communicated, we will compute the projected pixel and semantic embeddings for output in the following manner:
q entity', q. interleave" MLP. MLP.(q.entity. q iitterleaee) entityy = MLP, entity)
(8)
(9)
where .P are semantic and pixel projection respectively. This way, queries are projected into semantic and pixel space to compute the final output. The dimension of q. entity" and q. entity" are both [100, 512]. addition, q interleavee has dimension [n, 512) where n the entity number. With those projected queries and image features M, in the pixel projection space with shape [h, w, 512]. could get the final output mask associated with each entity with the following operation:
Index = arg max sim entity', q. interleave") dimm0
(10)
O = q ententttyy (anity"[Indeex
(11)
Mask O x Mj
(12)
(13)
In this way, associate the grounding entity with the desired mask segment of the image, as shown the top right figure in Table. I
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims generalizable, prototypable, extendable, and interleavable are clearly demonstrated in the method and experiment section. The contribution mentioned is also proved in method and experiment section.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have a paragraph on limitation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: Our paper is application-bast based paper, and does not have theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our code is public available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide open access to the code with training details in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccppplliiggudeeeeeeeeeeeee Cooee///ddeeebmimiia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify them in the training and inference code public available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our model is evaluated on a large number of datasets with enough data points. The number is empirically very stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
We have provided the details in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the border impacts in the last paragraph.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our model is on the side of understanding instead of generation, so that safeguards not applied.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have properly cited and acknowledged the prior works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new benchmark is documented in the paper the code base.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not need crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not contain human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "uNvalkMnzW", "reviewer": "Reviewer_iZKM", "summary": "The authors propose a benchmark for evaluation of what they call 'interleave understanding', or tasks which depend on the embeddings which are aligned across both modalities and task granularity, and they call this benchmark FIND-Bench. Find-Bench includes variants of segmentation and retrieval tasks, derived mostly from COCO datasets. The authors then propose a method for lightweight  fusion of llm and vision features with a multi-task objective, and find the model to be effective at various tasks.", "questions": "See weaknesses. Also, the FIND pipeline uses unimodal LLM and Vision Encoder, and unifies them with some training. There are many multimodal encoders these days like LLaVA[3], why not use those?\n\nTo focus the rebuttal, from my end, additional experiments supporting the weakness of missing baselines are nice-to-have but improved text clarity, related works, discussion of method is more important to me. \n\n[3] Liu, Haotian, et al. \"Visual instruction tuning.\" Advances in neural information processing systems 36 (2024).", "limitations": "Limitations are discussed in minimal way, but I see no major societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The idea of making a universal benchmark for grounding, retrieval, and segmentation at various granularities is  novel. The solution of aligning existing foundation models with light-weight multi-task tuning seems to both be an effective solution and in line with the underlying motivation for foundation models. \n\nQuality: The model design is reasonable, and the experiments are relatively extensive.  The ablations are interesting, especially the feature embeding layer for the llm. \n\nClarity: The motivation and findings, in addition to most of the technical details, of the paper are clearly presented. See weaknesses for a few cases where more exposition is needed. \n\nSignificance: The ability for a model to become a universal model for vision tasks is important, and the benchmark does test a form of universality. It would be helpful to further strenghen this point with motivation from real-world use-cases. See weaknesses.", "weaknesses": "Missing related works: The problem of Composed Image Retrieval e.g. ([1]) is quite related to interleaved retrieval. The authors should discuss the differences between the proposed work and the existing Composed Image Retrieval works. \n\nMissing Method Description: Specifics of the sampler, especially for text, are missing in the main paper.  In fact, it is unclear what the 'Embedding Sampler' in Figure 4(b) does. \n\nMissing Baselines: Although the number of baselines is relatively large, it does seem like LLaVA-type models (eg. [2]) can handle interleaved image-text inputs. Is it possible to retrieve with them? \n\nImproved Clarity: In some cases, clarity needs to be improved. In addition to some missing method description, there is also a missing task description of Interactive Segmentation in section 3.1.1. \n\nMotivation: Although the benchmark proposed is interesting, there is a limited connection to what capabilities it has the ability to unlock. Benchmarks, in order to be very impactful, should somehow be connected to an ability models should have \n\n[1].Saito, Kuniaki, et al. \"Pic2word: Mapping pictures to words for zero-shot composed image retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Jiang, Dongfu, et al. \"Mantis: Interleaved multi-image instruction tuning.\" arXiv preprint arXiv:2405.01483 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comprehensive reviews. We are motivated to address your concerns in detail below. We hope this clarifies your questions and enables you to have a better impression of our paper.\n\n[Q1] Missing related work with “Pic2word”.\n\nThanks for pointing out this related work, this composed image retrieval is very related, we have evaluated their method on demo examples shown in the **rebuttal PDF**, section Compared with Pic2Word Baseline. \n\n- It clearly shows that FIND has better multimodal reasoning capability, pic2word may tend to favour one of the entities in the query sentence and neglect the reasoning between multiple entities.\n- Meanwhile, FROMAGe is a concurrent work with Pic2work that is much closer to our setting, we have compared interleave retrieval with FROMAGe in Table. 3 and showing privilege results.\n- Additionally, we want to mention that our approach has more capabilities than Pic2work where FIND can do genetic, interactive segmentation, and grounded interleaved segmentation that is not achievable by Pic2Work.\n\n[Q2] Missing details on embedding sampler.\n\nWe appreciate your attention to this issue. We address it in the main rebuttal section under **[Common Question 2]**.\n\n[Q3] Missing baselines in comparison with LLaVA or LLaVA types of models (e.g. Mantis). \n\n- We have discussed the relationship with LLaVA in the related work section in L63 - L66. In addition, the evaluation benchmark of LLaVA and FIND is very different because their output type is different. Please refer to **[Common Question 1]** in the main rebuttal for a detailed explanation.\n- LLaVA does not support interleave understanding, LLaVA is multimodal understanding (order of image, text doesn’t matter), in Figure. 2 (2) we clearly compare multimodal and interleave understanding.\n- In addition, **LLaVA only supports one image input, even in LLaVA 1.6**, which does not support any of our tasks as well. As of last month, the LLaVA-Next-Interleave [1] technical report began supporting multiple images as input, which is later than our implementation. However, they still do not support retrieval and segmentation.\n- **Mantis is definitely concurrent work**; their first arXiv version was released on May 2, 2024, making it not a desirable comparison with a NeurIPS submission. Additionally, they focus on language output like QA or captioning, which is not comparable with FIND, as stated in **[Common Question 1]** for LLaVA-type models.\n\n[Q4] Missing Description on Interactive Segmentation.\n\nThank you for pointing this out. We have not described what interactive segmentation is in FIND. We will add this to the camera-ready version, if applicable. By default, interactive segmentation involves locating the relevant segment in the image with human reference, such as a point, bounding box, or scribbles. SAM [2] and SEEM [3] are two good references for understanding interactive segmentation.\n\n[Q5] Unclear benchmarking capabilities and lack of strong baselines.\n\n- Our FIND-Bench for interleave retrieval and grounding **benchmarks the capabilities of joint vision and language understanding**. For example, in the scenario “A [Image: Dog] is sitting on the [Text: Bench]”, we want the phrase “is sitting on” to be reasoned across both the image and the text contents. Our evaluation metrics will penalize instances where the content cannot be accurately interpreted (e.g. wrong retrieval/segmentation).\n- Actually, at the time of our submission, Grounding-SAM, BLIPv2, and ImageBIND were all very strong baselines, **utilizing industry-level computational resources**. However, we appreciate you mentioning this. We create a new benchmark on the latest SAMv2 dataset, and benchmark it with the most capable multimodal foundation model GPT-4o.\n    \n→ SAMv2 Benchmark statistics.\n\n| Dataset | # Images (videos) | # Instances | Avg Text Length |\n| --- | --- | --- | --- |\n| sav_val | 136 | 532 | 221 |\n| sav_test | 116 | 451 | 209 |\n\n→ SAMv2 Experiment results.\n\n|  | sav_val |  |  | sav_test |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Model | mIoU | cIoU | AP50 | mIoU | cIoU | AP50 | Latency |\n| FIND_Focal-T_LLaMA_640 | 58.9 | 63.7 | 65.6 | 57.8 | 59.4 | 66.5 | 0.0809 s/iter |\n| FIND_Focal-T_UniCL_640 | 58.3 | 65.7 | 65.9 | 60.1 | 60.3 | 68.7 | 0.0528 s/iter |\n| FIND_Davit-d5_LLaMA_640 | 62.5 | 69.8 | 71.6 | 60.2 | 64.6 | 71.3 | 0.1978 s/iter |\n| FIND_Davit-d5_UniCL_640 | 61.7 | 66.0 | 72.3 | 62.0 | 65.6 | 72.2 | 0.1699 s/iter |\n| SoM + GPT4o | 75.3 | 83.6 | 75.3 | 76.7 | 81.2 | 76.7 | ~10 s/iter |\n\n- Lastly, we want to emphasize to the reviewer that the benchmark is only one aspect of our contributions. Our main contribution lies in proposing the FIND interface, which leverages the raw foundation model embeddings for various downstream tasks.\n\n[Q6] Why not use LLaVA vision encoder?\n\n- LLaVA does not have its own vision encoder; it uses the pretrained CLIP encoder, which can be found in: https://github.com/haotian-liu/LLaVA/blob/c121f0432da27facab705978f83c4ada465e46fd/llava/model/multimodal_encoder/clip_encoder.py#L7. Moreover, they do not fine-tune the weights, as indicated by the use of torch.no_grad() in the forward path of: https://github.com/haotian-liu/LLaVA/blob/c121f0432da27facab705978f83c4ada465e46fd/llava/model/multimodal_encoder/clip_encoder.py#L133.\n\n[Q7] Rebuttal focus: new baseline numbers (Important), improved text clarity, related works, discussion of method (More important).\n\nThanks for your clarification, we have summarize the rebuttal in the following contents:\n\n- New Results: Q1, Q5.\n- Discussions: Q2, Q3, Q4, Q6.\n\n[1] Li, Feng, et al. \"LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models.\" *arXiv preprint arXiv:2407.07895* (2024).\n\n[2] Kirillov, Alexander, et al. \"Segment anything.\" ICCV 2023\n\n[3] Zou, Xueyan, et al. \"Segment everything everywhere all at once.\" NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pAzwkhsiGK", "reviewer": "Reviewer_kGcR", "summary": "The paper explores a unified multimodal embedding space across three image-text interleaved tasks, covering different granularity levels from image-level to pixel-level tasks.", "questions": "Please see the weakness.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The work investigates various multimodal tasks under image-text interleaved inputs, including grounding, retrieval, and segmentation, providing rich semantic understanding from image-level to pixel-level. The exploration of a unified embedding space is meaningful.\n2. The paper proposes a new benchmark, FIND-Bench, which includes new training and evaluation ground truths for interleaved segmentation and retrieval.\n3. The writing and visualizations are clear and well-presented.", "weaknesses": "1. It would be valuable to explore the effectiveness of the proposed method on larger datasets.\n2. The authors also mention recent multimodal models such as Llava and BLIP-v2. It would be interesting to discuss how the current approach compares with these recent methods, as they also aim for modality unification. More insights into the differences between these approaches and whether the current method achieves similar results with less data or solves problems that the aforementioned models do not address, such as pixel-level visual tasks, would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your clear and insightful comments. We have made our best effort to address them carefully below and in [Common Question 1], and [Common Question 3]:\n\n[Q1] Experiment on a Larger dataset.\n\nThank you so much for your interest in scaling up the training recipe. Unfortunately, we do not have sufficient computational resources at the moment. However, we have proposed a new dataset in FIND-Bench, incorporating the latest SAMv2 dataset for interleaved grounding in video frames, to enable further benchmarking. We hope this new dataset will unlock more capabilities of our FIND interface. We are open to future collaborations on scaling up.\n\n→ SAMv2 Benchmark statistics.\n\n| Dataset | # Images (videos) | # Instances | Avg Text Length |\n| --- | --- | --- | --- |\n| sav_val | 136 | 532 | 221 |\n| sav_test | 116 | 451 | 209 |\n\n→ SAMv2 Experiment results.\n\n|  | sav_val |  |  | sav_test |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Model | mIoU | cIoU | AP50 | mIoU | cIoU | AP50 | Latency |\n| FIND_Focal-T_LLaMA_640 | 58.9 | 63.7 | 65.6 | 57.8 | 59.4 | 66.5 | 0.0809 s/iter |\n| FIND_Focal-T_UniCL_640 | 58.3 | 65.7 | 65.9 | 60.1 | 60.3 | 68.7 | 0.0528 s/iter |\n| FIND_Davit-d5_LLaMA_640 | 62.5 | 69.8 | 71.6 | 60.2 | 64.6 | 71.3 | 0.1978 s/iter |\n| FIND_Davit-d5_UniCL_640 | 61.7 | 66.0 | 72.3 | 62.0 | 65.6 | 72.2 | 0.1699 s/iter |\n| SoM + GPT4o | 75.3 | 83.6 | 75.3 | 76.7 | 81.2 | 76.7 | ~10 s/iter |\n\nWe also compare our model with the current most capable model GPT4o with SoM labeled mark, the marks are computed by SEEM with most capable vision backbone.\n\n[Q2] Difference and benefits in addition to LLaVA and BLIPv2.\n\nThis is a very good question that deserves the attention of all reviewers. We have compared our work with LLaVA and BLIPv2 in [Common Question 1] and discussed the importance of mapping-based methods in the era of steerable models in [Common Question 3].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NY5Yp7eRFd", "reviewer": "Reviewer_kpHd", "summary": "FIND is a generalized interface for aligning foundation models' embeddings using a lightweight transformer without tuning pretrained model weights. It supports various tasks like retrieval and segmentation, is adaptable to new tasks and models, and creates a shared embedding space through multi-task training. FIND-Bench, an extension of the COCO dataset, showcases its effectiveness, achieving state-of-the-art performance in interleave segmentation and retrieval.", "questions": "Please see Weakness part.", "limitations": "No Limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The starting point and motivation of the paper is good.\n2. The experimental results are very good.", "weaknesses": "1. This paper is somewhat difficult to understand. For example, what is the **embedding**? Is it the embedding generated by the tokenizer or the feature generated by the foundation model? And what does **interleaved** mean? My understanding is that different modalities are interleaved. It is not limited to image+text or text+text; it can be image+text+image+text. I am not sure if my understanding is correct. However, it took me many careful readings to figure this out. It would be best to clearly define and emphasize key terms and settings at the very beginning. \n2. The method in section 3.2.2 is somewhat simplistic and lacks innovation. It doesn't seem to have much new design. It needs to be clarified why this interleaved approach is challenging (for example, if all the inputs are just treated as a sequence, will the performance be significantly worse? If so, why?).  More analysis and intuition about that will be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We greatly appreciate your comprehensive comments in the weakness section. We sincerely hope that, after reading our rebuttal, you will have a new perspective.** We believe most of the confusion stems from terminology common within the small multimodal understanding community. It would be beneficial if you could also consider the high-level comments from Reviewer-kGcR and Reviewer-iZKM, as these comments highlight our contributions from a broader perspective. *We believe a paper’s impact should be evaluated from both high-level and detailed viewpoints.*\n\n[Q1] The definition of embedding is confusing.\n\n**Embeddings** is a commonly used term in the context of foundation models. It typically refers to “the transformation of raw data into continuous vector representations that capture the semantic relationships and essential features of the data, enabling efficient and meaningful processing by the model [1]”.\n\n- A well-known early work, “word2vec,” introduced the concept of a continuous vector space for natural language. This continuous vector space is commonly referred to as “the embedding space.”\n- In addition, a recent well-known work, “The Platonic Representation Hypothesis [3],” presented as a position paper at ICML 2024, uses the term “embeddings” extensively, mentioning it 12 times throughout the paper.\n\n[Q2] Confusing in the definition of interleave.\n\nFirstly, thank you for taking the time to understand the term “interleave.” We will provide a more comprehensive explanation here.\n\n- Your understanding “It is not limited to image+text or text+text; it can be image+text+image+text” is correct. This is clearly illustrated in Figure 2 (2) of our main paper.\n- The terminology “interleave” was not introduced by us; it first gained attention within the community through the Flamingo paper [4] by Google, which had a significant impact in 2023. In Fig.7 of Flamingo, it clearly provides an example of an interleaved token “<BOS>Cute pics of my pets!<EOC><image>My puppy sitting in the grass.<EOC><image> My cat looking very dignified.<EOC>”. Hope this can help you to have a better understanding on interleave.\n\n[Q3] The method is somewhat simplistic and lacks innovation, not much new design.\n\nI would suggest that simplicity with minimal overhead is best for model design. \n\n- As stated on the first page of LLaMA3 [5] paper, it indicates:\n    \n    *“We believe there are three key levers in the development of high-quality foundation models: data, scale, and **managing complexity**.”*\n    \n    Managing simplicity is very important for the potential of success for scaling up.\n    \n- Our main contribution in the methods section is the design of a sophisticated interface that leverages the attention mask for various downstream tasks without any overhead. We have provided a comprehensive unification demonstration of all the proposed tasks under the FIND interface in [Common Question 4]. We will give a preview (notations are defined in the PDF) below for fast reference:\n\n[Q4] Why the interleave approach is challenging.\n\n- Interleave retrieval was introduced in the FROMAGe paper [6] without any benchmark, prompting us to propose an interleave retrieval benchmark and use FROMAGe as a baseline for comparison.\n- For interleave retrieval, we compared FIND with strong baselines like ImageBIND and BLIPv2, which lack interleave understanding, and FROMAGe, which does. Table.3 demonstrates that FIND achieves the best performance in interleave retrieval.\n- We are the first to propose interleave grounding, a challenging task because it requires joint reasoning of vision and language information. As shown in Table 3, separating grounding vision and language tokens (SEEM baseline) results in significantly worse performance on interleave grounding compared to FIND, with differences as high as ~10 points across various scales and datasets.\n\n[Q5] Why not just treat vision and language tokens as a sequence?\n\n- Most interleave methods treat vision and language information sequentially. However, pixel-level information may require multiple tokens (e.g. 512 tokens) for accurate representation. To address this, we separate queries and tokens, allowing us to compress token information into queries for interleave retrieval and segmentation.\n\n[1] GPT4-o with prompt “what does embedding means in the context of foundation model in short”.\n\n[2] Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. ICLR 2023.\n\n[3] Huh, M., Cheung, B., Wang, T., & Isola, P. (2024). The platonic representation hypothesis. ICML 2024.\n\n[4] Alayrac, Jean-Baptiste, et al. \"Flamingo: a visual language model for few-shot learning.\" NeurIPS 2023.\n\n[5] Dubey, Abhimanyu, et al. \"The Llama 3 Herd of Models.\" *arXiv preprint arXiv:2407.21783* (2024)\n\n[6] Koh, Jing Yu, Ruslan Salakhutdinov, and Daniel Fried. \"Grounding language models to images for multimodal inputs and outputs.\" ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "METYZYrlv4", "reviewer": "Reviewer_AFHH", "summary": "The paper proposes to construct an interface to connect the embeddings and predictions from different foundation models. With the designed interface, the overall system has a promise to interleave any modality in a flexible manner. To showcase this flexibility, this paper constructs a benchmark, FIND, by leveraging existing COCO labels and text-only GPT-4 model. As a generalist model, the paper shows that FIND generally performs better than existing generalist model and sometimes even specialized model.", "questions": "Benchmark\n- From Table 1, it is not clear to me how does the text-only data engine resolve the cases where there are multiple instances within the same classes. For example, in `Prompt for GPT4 Engine`, say the image contains ten mugs on a table. How does data engine resolve the mapping between the bounding boxes of each mug to the captions? \n\nMethod\n- How does the embedding sampler work for Llama? Specially, which embeddings is used? Is it the last hidden states of the last tokens? Do the authors have any insights in extracting the embeddings from an auto-regressive model? It is non-trivial to use the embeddings in an auto-regressive mdoels, this paper [1] specifically ablate the design choice.\n- Does the approach involves any similarity calculation? In section 3.2.1 task unification, the authors use a similarity measurement to connect the source and the target domains. While in the Fig. 4, I feel the FIND interface is purely relied on a transformer network followed by several projection heads. Can the authors clarify the method?\n\n\nExperiments\n- In Table 2\n  - Why SEEM, X-decoder, BLIP-2 is marked as not “jointly trained” (the third column). Please clarify it.\n  - What does * mean? It’s shown in the first column and the seventh column (mIoU for RefCOCO-g).\n- In section 4.1, the authors claim that the less optimal image-text retrieval results are due to the batch size during fine-tuning. In L245-248, I am very confused with the justification. Here are my questions:\n  - What resolution FIND trained with?\n  - “Pilot experiments with X-decoder showed … well across tasks.” Is this described in X-decode paper or is it reproduced by the authors? Also, I checked the section 4.1 in X-decoder paper, I believe the authors want to say “1024 for segmentation and 224 for image-text data”.\n  - I am not sure if the authors are claiming that the main issue is the batch size or the resolutions. If it’s the batch size, why not use gradient accumulation? \n\n\n[1] Vision-Language Models Provide Promptable Representations for Reinforcement Learning", "limitations": "The limitations include\n1. Presentation is not clear enough and has a lot to clarify\n2. The paper works around foundation models but it's limited to COCO dataset. \n3. The benchmark data engine requires more qualitative examples and justification of the design choices.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The paper idea is novel and promising if extended well.\n- The paper conducts an extensive experiments on COCO and compares with valid baselines.", "weaknesses": "Method and experiment\n- Lack of baseline of instruction-tuned vision-language models, such as LLaVA.\n- Lack of out-of-domain evaluation. The advantages of foundation models is that they have a higher possibility to generalize to unseen domain due to their scale. However, it is unclear when the COCO-trained interface is able to generalize to other domains.\n\nWriting to improve\n- $\\textbf{sim}$ is first introduced in L109 and is later explained in L149\n- Figure 3 is not self-contained. It’s hard to follow what E, O mean. There are too many notations and too little text to understand the tasks and how the task unification works here.\n- In Figure 4 (center), the caption says “the shape of different polygons represents different embedding types” I cannot find where in the text specifying different “types of embeddings”.\n- Missing citations in L206,207 after each model\n- Where is “interleaved segmentation” in Table 3 mentioned in L256", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate your thorough and comprehensive reading of our paper.** We understand your frustration with the confusing implementation details despite your diligent effort. We apologize for any confusion caused. **We kindly ask that you consider the paper from a higher-level perspective**, as highlighted by the strengths noted by Reviewer-kGcR and Reviewer-iZKM. Meanwhile, we will address these constructive detailed comments in the camera-ready version if any.\n\n[Q1] Lack of Instruction baseline e.g. LLaVA.\n\nIn **[Common Question 1]**, we provide a comprehensive discussion on why instruction-based methods are not comparable to our approach, considering both the output type and the specific problem we are addressing. Additionally, in **[Common Question 3]**, we thoroughly explain the importance of multimodal understanding and mapping in the steerable era.\n\n[Q2] Lack of out-of-domain evaluation.\n\nThank you for pointing this out. We have created a new benchmark on SAMv2, using an improved protocol outlined in Table.1 for interleave grounding (we have illustrated this improved protocol in the rebuttal PDF). This benchmark is compared with SoM+GPT-4o, which serves as a high-standard evaluation protocol, the SoM mark is labeled with our best power SEEM model.\n\n→ SAMv2 Benchmark statistics.\n\n| Dataset | # Images (videos) | # Instances | Avg Text Length |\n| --- | --- | --- | --- |\n| sav_val | 136 | 532 | 221 |\n| sav_test | 116 | 451 | 209 |\n\n→ SAMv2 Experiment results.\n\n|  | sav_val |  |  | sav_test |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Model | mIoU | cIoU | AP50 | mIoU | cIoU | AP50 | Latency |\n| FIND_Focal-T_LLaMA_640 | 58.9 | 63.7 | 65.6 | 57.8 | 59.4 | 66.5 | 0.0809 s/iter |\n| FIND_Focal-T_UniCL_640 | 58.3 | 65.7 | 65.9 | 60.1 | 60.3 | 68.7 | 0.0528 s/iter |\n| FIND_Davit-d5_LLaMA_640 | 62.5 | 69.8 | 71.6 | 60.2 | 64.6 | 71.3 | 0.1978 s/iter |\n| FIND_Davit-d5_UniCL_640 | 61.7 | 66.0 | 72.3 | 62.0 | 65.6 | 72.2 | 0.1699 s/iter |\n| SoM + GPT4o | 75.3 | 83.6 | 75.3 | 76.7 | 81.2 | 76.7 | ~10 s/iter |\n\n[Q3] Sim is first introduced in L109 and is later explained in L149.\n\nWe apologize for the confusion. We will add a reference the first time we use the term “sim.” Initially, we did not include it because “sim” seemed straightforward to understand.\n\n[Q4] Hard to follow what E, O mean in Fig.3.\n\nWe have clearly defined E in L103, and O L110 of the main paper.\n\n[Q5] Too many notations and too little text to understand the tasks and how the task unification works here.\n\nThe mathematical symbols are carefully curated to abstract and unify the tasks, as highlighted by Reviewer-kGcR in strength 3: “The writing and visualizations are clear and well-presented.” We provide a comprehensive textual explanation of task unification in the rebuttal PDF, section xxx.\n\n[Q6] Cannot find where specifying different “types of embeddings” in Fig.4.\n\nThe types of prompts and queries are clearly labeled in Fig. 4. You can find the text “Vision, Language, Interleave Prompts” directly above the pentagon (5-edge-polygon), and “Queries” above the gray square.\n\n[Q7] Missing citations in L206-207 after each model.\n\nThanks so much for pointing this out, we will add the citation in the camera-ready version if any.\n\n[Q8] What is interleaved segmentation in Table.3.\n\nThanks for pointing out, Interleave Segmentation actually means Interleave Grounding defined in L110. We will edit this in Camera Ready if any.\n\n[Q9] How does the text-only data engine resolve the cases where there are multiple instances within the same classes?\n\n- In Table. 1 **SI** contains object descripts generated by LLaVA, together with bounding box and annotation ID, the GPT4 data engine is able to identify the object appearance. We show an example to resolve the confusion on “ten-mug problem” in the section [Qualitative examples on FIND-COCO-Bench] of the **rebuttal PDF**.\n- In addition, for the newly created SAMv2 benchmark, we are using SoM [1] + GPT4-V to further reduce object hallucination. We show 7 random examples for FIND-SAMv2-Bench in the **rebuttal PDF**.\n\n[Q10] How does the embedding sampler work for Llama.\n\nFor language embedding, we use identity sampling for all language embeddings. Please refer to [Common Question 2] for more details.\n\n[Q11] Any insights into extracting the embeddings from an auto-regressive model?\n\nYes, this is one of our main contributions, clearly stated in L49-50 and L66. Additionally, Table 5 shows that the -12 layers feature in LLaMA aligns best with visual representation in terms of semantic meaning. We have demonstrated this insight in L290-295.\n\n[Q12] Related work “Vision-Language Models Provide Promptable Representations for Reinforcement Learning”.\n\n- The related work uses the last layer embeddings from LLMs as a knowledge prior for policy generation. They ablate the design choices of “No Prompt; No Generation; Change Aux.; Text Oracle Detector” in Table 8, which are very different in content and objective from our approach.\n- The referenced paper is a concurrent work with the NeurIPS template, updated in May 2024 on ArXiv.\n\n[Q13] Does the approach involve any similarity calculation?\n\nTask unification is achieved by treating all tasks as similarity mapping problems. This similarity mapping is applied after “the transformer network and several projection heads”; it is used to compute the loss and identify the output. We have provided a use case study in the rebuttal PDF, Sec.xxx, demonstrating how similarity mapping is applied to interleave grounding.\n\n[Q14] What is jointly trained?\n\nJoint training means whether the model is jointly trained on all the tasks with the number indicated in Table.2. We apologize for the typo that X-Decoder and SEEM should be jointly trained.\n\n[Q15] What does * mean in Table.2 for mIoU?\n\nWe apologize for the confusion. * means the model can evaluate this number, but it was not reported in the original paper, so we did not include it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
→ We thank all the reviewers for their constructive comments with the **following strengths listed**:

**[Novelty (iZKM, kGcR, kpHd, AFHH)]**: The idea of making a universal benchmark for grounding, retrieval, and segmentation at various granularities is novel. The exploration of a unified embedding space is meaningful. The starting point and motivation of the paper is good. The paper idea is novel and promising if extended well.

**[Experiment (iZKM, kpHd, AFHH)]**: The experiments are relatively extensive, and the ablations are interesting. The experimental results are very good. The paper conducts an extensive experiments on COCO and compares with valid baselines.

**[Writing (iZKM, kGcR)]**: Paper is clearly presented for technical details and findings. The writing and visualizations are clear and well-presented.

**[Benchmark (iZKM, kGcR)]**: The paper proposes a new benchmark, FIND-Bench, which includes new training and evaluation ground truths for interleaved segmentation and retrieval. The benchmark does test a form of universality of vision problems.

In general, the reviewers regard the paper with good novelty and strong experimental results. 

→ We also addressed the common question raised by the reviewers:

**[Common Question 1]**: Clarification of the main idea and the relationship with instruction-based methods (e.g., **LLaVA**) and Q-Former styled methods (e.g., **BLIPv2**).

- Logically, our method is an experimental proof of “The Platonic Representation Hypothesis [1],” which suggests:
    
    *“Neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.”*
    
    Our experiments in Table 4 clearly indicate that although SAM and LLaMA are trained on very different datasets, the embeddings of the two models can be projected into the same embedding space for similar semantic meanings. Additionally, it shows that models trained on very different objectives (segmentation and text generation) can converge to a shared statistical model.
    
- LLaVA uses a fixed vision model as a tokenizer for LLM to enable generative visual QA. Meanwhile, BLIPv2 aligns the vision output with the language input as well. Both models have tuned the LLM weights, which does not provide clear insight into how vision foundation models and LLMs are connected. Unlike these, FIND aligns the vision backbone features (embeddings) with the intermediate layer features (embeddings) of LLMs, which is distinctly different from LLaVA and BLIPv2 (L62 - L64, Table 5).
- The output modality of LLaVA and BLIPv2 is very different from ours. They use LLMs as generative language decoding tools for QA and captioning, whereas we focus on mapping problems (e.g., retrieval, segmentation, grounding, etc.). **We are not supposed to compare with LLaVA, as it does not support retrieval and segmentation, and we do not support question answering.** For BLIPv2, it supports retrieval, so we compare the retrieval results in Table 3.
- The training budget of LLaVA and BLIPv2 is much heavier than FIND. (1) We only tune the vision projector (6 layers of transformers) and the FIND interface (9 layers of transformers) in a single stage. In contrast, both BLIPv2 and LLaVA involve multistage training and LLM tuning, requiring much higher computation for tuning.

**[Common Question 2]**: **Embedding sampler** is not well defined.

- The embedding sampler is a variant of the visual sampler presented in “Segment Everything Everywhere All at Once [2].” In L173, we mention that “Technically, the embedding sampler is usually an interpolation or grid sample layer in PyTorch.”
- Except for interactive segmentation or interleave segmentation with visual reference (they are visual samplers in [2]), the embedding sampler is an identity grid sampling layer (i.e., it keeps all the embeddings). We maintain the potential for any future sampling for long language context, so by using the term embedding sampler.
- As stated in our main paper:
    
    *“Technically, the embedding sampler is usually an interpolation or grid sample layer in PyTorch.”*
    

**[Common Question 3]**: Why understanding and **mapping is important in the steerable era**.

- Understanding the mapping is crucial for comprehending the embedding properties of foundation models.
- Aligning the mapping serves as an intermediate step toward steerable interaction, where instruction-tuned models decode embeddings into human-understandable language.
- As shown in Table 5, vision embeddings have the best alignment with the intermediate embeddings of LLaMA. This finding supports the design choice in the multimodal Llama3, as discussed in “The Llama 3 Herd of Models [3],” which utilizes cross-attention layers to interact with vision embeddings. We believe this design choice is not ideal for both LLaVA and BLIPv2, as noted in L64-66.

[1] Huh, M., Cheung, B., Wang, T., & Isola, P. (2024). The platonic representation hypothesis. ICML 2024.

[2] Zou, Xueyan, et al. "Segment everything everywhere all at once." NeurIPS 2023.

[3] Dubey, Abhimanyu, et al. "The Llama 3 Herd of Models." *arXiv preprint arXiv:2407.21783* (2024)

Meta-review: This paper was reviewed by several experts where a plurality argued for accepting the paper.  The proposed dataset and approach seemed generally accepted by reviewers.  Some minor concerns still remain in terms of the positioning of the paper and the significance of the results.  For example, the authors argue that they only support a subset of tasks, but that seems to be contrary to the general purpose of a "foundation model" by only focusing on some tasks.  While this can be argued as common definitions, it is imprecise and should be discouraged.  The approach itself can also be seen as essentially combining multiple models, but this also has other alternatives, such as model soups.  While this wouldn't add more task flexibility, it does result in a more effective model for the tasks it does support.  These weaknesses and others noted by reviewers call into question the significance of the work, but not enough to argue for overturning the plurality recommendation by reviewers.
Decision: Accept (poster)