Interfacing Foundation Models' Embeddings

Authors: xueyan  zou
linjie  li
jianfeng  wang
jianwei  yang
mingyu  ding
junyi  wei
zhengyuan  yang
feng  li
hao  zhang
shilong  liu
arul  aravinthan
yong jae lee
lijuan  wang
madison ♯ microsoft
berkeley † hkust
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 16:30:25

Abstract: Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in Fig.1, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights.  (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.

Main paper: 
 introduction: 
 With the exhilarating progress in foundation models across the vision and language domains, such as GPT4(V) (30) , DALLE-3 (31) , SAM (19) , and LLaMA (38) , etc., we have reached a stage where deep learning models achieve remarkable performances on both vision and language domains (5; 22) . Specifically, models like GPT-4(V) (30) have showcased human-level perception and reasoning skills (46) .Despite their impressive capabilities in information memorization, processing, and reasoning, these models tend to be specialized for specific output types. However, their output types are limited to language for GPT, images for DALLE, masks for SAM, etc. In this work, we aim to leverage the privileged properties of foundation models' embeddings to expand their output space (e.g., extend to pixel-level outputs), unlocking their potential for interleaved understanding and reasoning.To accomplish this, we introduce an INterface for Foundation models' embeDdings (FIND), which utilizes the pre-trained foundational model embeddings to jointly handle downstream tasks of varying granularities (from pixel to image) in an interleaved manner. As illustrated in Fig. 2 .1, the FIND interface processes embeddings from vision and language foundation models, and outputs segmentation, grounding, and retrieval results. As all vision-language tasks are trained uniformly in FIND, an interleaved shared embedding space is created where vision and language references can be interchanged and augmented. For example, in Fig. 2 .2, during mapping an interleaved representation loosens the single-modality constraint on the source and target domain. And during reasoning, interleaved sequences enhance information exchange between vision and language compared to multimodal sequences.To effectively align and evaluate the interleaved embedding space, we construct a new dataset named FIND-Bench. This dataset uses COCO images and includes new annotations for integrated grounding and segmentation. These annotations are generated by GPT-4, which, despite not processing visual input, can directly link specific image segments and annotation IDs with generated descriptions (e.g., <id>(the golden retriever) ...). This unique capability enables the creation of training and evaluation datasets for retrieval and grounding in an interleaved context.In summary, we claim the following contributions:• We introduce the FIND interface that is is generalizable, flexible, and extendable to various downstream tasks and foundation models.• Through the effective training scheme of FIND, an interleaved shared embedding space is created interfacing foundation models.• We propose a new Benchmark, FIND-Bench, which includes new training and evaluation ground truths for interleave segmentation and retrieval.• Our model achieves SoTA performance on interleave retrieval and grounding and shows better or comparable performance on generic, interactive, grounded segmentation and image-text retrieval.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The claims generalizable, prototypable, extendable, and interleavable are clearly demonstrated in the method and experiment section. The contribution mentioned is also proved in method and experiment section.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Foundation Models. Recent years have seen a speedy evolution of foundation models in diverse areas such as computer vision (47) , natural language processing (39; 10; 4; 30), and their interactions (1; 23; 44) . For example, GPT-3 (4) heralds breakthroughs in natural language understanding and generation tasks, As a vision foundation model, Florence (47; 42) can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, etc.Flamingo (1) bridges powerful pre-trained vision-only and language-only models by token fusion with cross-attention. BLIP-2 (23) proposes an efficient pretraining strategy that bootstraps vision-language pre-training with a lightweight Q-Former in two stages. Different from previous multi-modal approaches, such as Flamingo (1), LLaVA (26) and Q-Former (BLIP-2) (23) that feed the vision foundation model output into a language decoder and use the LLM as an interpreter, our goal is to interface foundation model embeddings so that LLMs and vision models can be unified in the embedding space.Interleaved Image-Text Understanding. Previous works have explored interleaved visual understanding in the context of visual question answering, visual dialogue, image captioning, and interleaved image retrieval (20; 13; 1). In addition, recent works (48) explore contextual detection that associates phrases with visual content in a sentence. We notice that these earlier works, though reveal interleaved capabilities for image understanding, lack an evaluation benchmark, as well as a complete training dataset. (51; 21; 2) propose a new benchmark on interleaved generation and understanding of image and document level, while there is no benchmark available for the interleaved tasks between interactive image parts and phrases. To this end, we introduce the interleaved segmentation and interleaved retrieval tasks with our carefully designed benchmark FIND-Bench, which we believe to be essential for the field.Image Understanding. ) that grounds object segments from language descriptions. We notice that there is currently no available work that achieves image-level retrieval, pixel-level segmentation, and interleaved vision-language understanding in a single model. In this work, we propose FIND as a unified interface that can support all the above tasks, while maintaining good performance, and further enabling two new tasks of interleaved segmentation and interleaved retrieval. We unify these tasks by interfacing foundation models' embeddings.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have a paragraph on limitation.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Foundation models such as CLIP (32) , SAM (19) , LLaMA (38) , etc. can process vision or language inputs for reasoning, understanding, and generation. The embeddings generated by these models contain rich and structured information (35; 3), making them extremely well-suited for understanding tasks. Aligned with the Platonic Representation Hypothesis (17) , we believe foundation models can easily communicate with each other. Therefore, we designed the FIND interface to project vision and language embeddings from foundation models into a unified space. The created space enhances both multimodal and interleaved understanding. Since no prior benchmark exists for interleave understanding, we believe it is meaningful to formally define the interleave retrieval and segmentation problems and create a dataset for benchmarking them.Our new benchmark supports two tasks: interleave retrieval and interleave grounding. It evaluates both dataset-level and image-level interleave alignment, focusing on reasoning and matching capabilities. Additionally, we created training and evaluation datasets to further enhance interleave understanding.Interleave Retrieval Retrieve Visual Sample with SEEM Given the search dataset (Q) with the segments in all images denoted as (SD), we compute all embeddings S representing each segment using SEEM (53) with S = SEEM(SD) ∈ R n×d . Given the similarity matrix W = S × S T , where Wij represents the similarity between segment i and segment j, the index of the closest segment for segment i is Match(i) = arg max j̸ =i Wi where Match(i) returns the index j that has the highest similarity to segment i. Table 1 : Pseudo code for Data Engine. We show the pipeline to create the FIND-Bench from data preparation, text prompting using GPT4, visual prompting with SEEM to integrated result. We reuse the images and ground truth annotations from the COCO dataset to create FIND-Bench. In the first part of Table . 1, we demonstrate the input data used to in-context learning for GPT-4. In addition to the COCO ground truth, we generate pseudo-image descriptions using VLM models, such as LLaVA (26) , to enrich the information. In the second part of Table . 1, we present the prompt template for our data engine. This template generates the text part for the interleaved captions in part 4 of Table . 1, providing language descriptions associated with annotation IDs. The segments corresponding to these IDs are highlighted in the same color in the example image shown in Table . 1.As stated in Sec. 3.1.1, the source and target components are exclusive. We leverage the strong visual understanding capabilities of SEEM (53) to find replacements for the visual components in the entry. The retrieved and replaced visual components are shown in part 4 of Table . 1, with the exact segment highlighted in the same color as the corresponding reference text. For example, <the playing field> is With benchmarks introduced in Sec. 3.1 to evaluate the model's interleaved visual understanding capability, we now present our approach for interfacing foundation models' embeddings on multimodal and interleave understanding. We begin with the preliminaries on task unification and terminology.Task Unification. In this work, we focus on retrieval, grounding, and segmentation in both multimodal and interleaved manners. In Fig. 3 , we demonstrate four example tasks: interleave retrieval, interleave grounding, interactive segmentation, and generic segmentation. From an abstract perspective, we can regard all visual understanding tasks as the problem of matching candidates from the source domain to the target domain. Formally, we define the source domain as D s and the target domain as D t . Example elements in D s or D t includes interleaved entry E, an image I, an object or segment O, texts T . For each visual understanding task U(D s , D t ), the goal is to find the closest Y ∈ D t for each X ∈ D s . Formally we write:∀X ∈ D s , Y * = arg max Y ∈Dt sim(X, Y )where X, and Y are base element of D s , and D t respectively, and sim(X, Y ) denotes the similarity between X and Y . For example, in generic segmentation (Fig. 3 .4), D s is the set of all objects (segments) in the image: D s = {O 1 , . . . , O ns }, and D t is the set of category names:D t = {T 1 , . . . , T n }.For each object O in D s , we will find the corresponding category T ∈ D t .Terminology. Here we will introduce important model terminology, including prompts (P ) and queries (Q). Our model supports three kinds of inputs: vision (I), language (T), and interleaved vision-language (E). The vision and language foundation models predict the embeddings for those inputs. As shown in Fig. 4 .1, by sampling the embeddings, we obtain vision prompts (P I ), language prompts (P T ), and interleave prompts (P E ). Additionally, trainable queries initialized with random parameters will accumulate information from the prompts. For example, in generic segmentation, object queries (Q O ) gather information from visual prompts. Interestingly, queries just act like "buckets" accumulating "water" (prompts) in the FIND interface, as shown in Fig. 4 .1.Our model is designed to interface with a pair of arbitrary vision and language foundation models.Prompts and Queries Preparation. Given image (I), text (T), and interleave (E) inputs, the vision encoder (F v ) and language encoder (F l ) will encode these inputs to sequences of embeddings M :EQUATIONwhere, M ∈ R n×d , and n, d is the embedding number and dimension respectively. Similar to SEEM (53), we use an embedding sampler to sample customized prompts for downstream tasks.Example sampling strategies include downsampling, ROI pooling for the region, and rearrangement of embeddings for interleave prompt. The sampling procedure does not alter the embedding distribution. After sampling, we obtain {P E , P T , P I , . . .} = Emb_Sample(M I , M T , M E ). Additionally, the embedding sampler is responsible for sampling queries ({Q E , Q T , Q I , . . .}) from the pool of learnable queries. We allow duplication in the sampling procedure of learnable queries. These queries Q 0 = {Q 0 E , Q 0 T , Q 0 I , .. .}, content attention and conditional attention are formally defined as:EQUATIONwhere S l ⊆ {P l , Q l } is a subset of queries and prompts, → represents the attention mask. For example, [P → Q] means that Q is able to attend P during the attention. In this way, prompts act as the information source, and queries act as the bucket. In Fig. 4 .2, we unfold the prompts and queries for some tasks supported by FIND interface.Projection The outputs of the FIND interface are a sequence of queries:Q L = {Q L O , Q L T , Q L I , Q L E , . . .}.We then project the queries using linear layers, MLP s and MLP p , for semantic and pixel projection, respectively. The semantic and pixel queries are computed asQ s = MLP s (Q L ) ∈ R nt×d and Q p = MLP p (Q L ) ∈ R nt×d ,where n t is the total instance number, and d is the embedding dimension. The semantic outputs are used for retrieval, category mapping, etc., while the pixel outputs are used for mask prediction.Task Head With the projected queries, as illustrated Sec. 3.2.1 each understanding task can be represented as a similarity mapping procedure. Formally, segmentation result (Mask) can be computed given initial image embedding M I ∈ R np×d , where n p is the pixel number. The similarity scores (Score) can be computed directly from Q s . The outputs for each task is a subset of {Mask, Score}.EQUATION) Loss FIND is trained with a linear combination of losses for panoptic segmentation, grounded segmentation, interactive segmentation, image-text retrieval, interleave retrieval with visual entities from the same image, and interleave grounding. We demonstrate the loss details in the Appendix.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 Datasets. We use COCO (25) as our main training and evaluation dataset, which spans diverse annotation types. We make use of the annotations from COCO-panoptic, Ref-COCO (45; 28; 29), COCO-Karpathy (18) , and the new datasets generated with the data engine in FIND-Bench. We generate two sets of new annotations, including COCO-Entity and COCO-Paragraph, the detailed statistics are shown in the Settings. We benchmark our method on three different model sizes: Tiny (FocalNet), Base (Davit-d3), and Large (Davit-d3). The vision backbone is fixed and reuses the X-Decoder pre-trained weights unless specified as SAM. The language backbone is a fixed LLaMA-7B, unless specified as UniCL.During training, we train the FIND-Interface jointly on all the tasks unless specified.Metrics. We evaluate all the tasks with their standard evaluation metrics. For the newly proposed interleave retrieval, we use IR@5 and IR@10 (Interleave-to-image Retrieval accuracy at rank 5/10).For interleave grounding, we evaluate based on cIoU (pixel-wise IoU), and mIoU (image-wise IoU) between the predicted interleave masks and the ground truth masks.Baselines. We use ImageBind (13) , FROMAGe (20) , BLIP2 (23) as baselines for the interleave retrieval task; Grounding-SAM (27) , SEEM (53) for interleave grounding. We claim to make every effort to design the baseline evaluation protocol to achieve the best possible performance.In the main experiments, we focus on evaluating FIND on Generalizable, Interleavable, and Extendable capabilities as claimed in the abstract.(1) Generalizable to Segmentation, Grounding, and Retrieval. Table 2 compares FIND with strong baselines on generic segmentation tasks including panoptic segmentation, instance segmentation, and semantic segmentation. In addition, we demonstrate the segmentation capability in both referring segmentation (RefCOCO-g: one sentence is associated with one instance) and grounded segmentation (COCO-Entity and COCO-Paragraph: one sentence is associated with multiple instances) settings. Moreover, we also benchmark FIND's performance in image-text retrieval on three different ground truth types on COCO, where the average sentence length for the splits (Karpathy, Entity, and Paragraph) gradually increases. Below are the takeaways:The instance segmentation result stands out: Our approach with a large vision encoder outperforms similar models like Mask2Former, X-Decoder, and SEEM, achieving a performance 2.2 points higher than Mask2Former (L), which additionally uses deformable convolution. Notably, the segmentation training data is identical for both Mask2Former and FIND. The performance gain likely results from our unified segmentation and grounding pipeline, which mutually benefits from the semantic ground truth of each domain.Mutual benefits of grounded and referring segmentation: In FIND, we unify grounded and referring segmentation using queries and prompts. As shown in Table 2 , our model achieves state-of-theart performance on COCO-Entity and COCO-Paragraph and outperforms strong baselines on the Ref-COCOg dataset.Interactive segmentation performance is preserved in the unified settings. Unlike SEEM which is only trained on image-only tasks, FIND is trained also on image-text tasks, such as image-text retrieval. With the smart design of queries, prompts, and attention mechanisms, training interactive segmentation and image-text retrieval does not interfere. Thus, it enables our approach to achieve competitive performances (i.e. FIND 88.5/89.5/77.4 vs. SEEM 88.5/89.6/76.5).Less optimal image-text retrieval results: The sub-optimal performance in image-text retrieval is due to batch size during fine-tuning. Pilot experiments with X-Decoder showed that different resolutions (e.g., 1024 for images and 224 for language) do not generalize well across tasks. Thus, FIND is trained with the same resolution for all tasks. In (2) Interleavable on vision and language modalities. In Table . 3, we evaluate FIND on the interleaved dataset-and image-level understanding tasks in FIND-Bench. In the columns of COCO-Entity and COCO-Paragraph, we replace the text entity with visual reference on 0.5 probability, unlike Table . 2 the columns are purely evaluated on language-based data.Interleaved Segmentation: We build an interleaved segmentation baseline using the SEEM model. Instead of formulating the grounding task in an interleaved format that SEEM doesn't support, we simply separately infer visual, and text entities using the interactive or grounding function of SEEM.As shown in Table 3 , FIND outperforms SEEM on interleave segmentation with around +8 points on both COCO-Entity and COCO-Paragraph under cIoU metrics.Interleaved Retrieval: We also explore cross-image interleave retrieval on FIND. Since the interleaved reference objects are from the same validation set, IR@1 is not meaningful, so we report IR@5 and IR@10 in this setting. For ImageBind and BLIP-2, we use ensemble scores of texts, sentences, and images. Following FROMAGe's settings for interleaved image-text retrieval, our performance is significantly higher than the baselines, demonstrating the effectiveness of our interleaved shared embedding space.Generic Segmentation: Beyond classic evaluations using class names or fixed indices, we replace categories with class descriptions (long descriptions) or visual prompts (average features for object queries for each class). Leveraging LLMs, FIND excels in description-based segmentation, benefiting from smoother representations and better handling of long contexts. We also demonstrate FIND's effectiveness in the visual context setting.(3) Extendable to arbitrary foundation models and tasks. In the main experiments, we use X-Decoder as the vision encoder, and LLaMA as the language encoder, which shows convincing performance on all the tasks. X-Decoder has been trained to pair up vision and language embeddings, however, SAM is only trained on segmentation data without any semantic meaning. Thus, we use SAM as an ablation vision foundation model, to study how important is vision encoder trained with semantic data. For the language encoder, we adopt UniCL which has the same size as Bert to study the difference between a standard language encoder, and an LLM encoder. As shown in Table 4 , UniCL and LLaMA usually have very similar performance with X-Decoder as vision encoder, except that LLaMA is extremely effective on long description reasoning. Although the performance of SAM is much worse than its counterpart X-Decoder on semantic understanding after training the interface, our approach also shows that without any modification to SAM, it applies to semantic understanding tasks on generic, grounded segmentation, and image-text retrieval.We ablate our approach from two perspectives: (1) What is the effectiveness of each task in the unified pipeline? (2) The effectiveness of using intermediate layers of the LLM representation.We assess task effectiveness by gradually removing tasks in Varying the feature embeddings layer for LLM: LLMs process language tokens, with embeddings near input and output layers being less semantic. We hypothesize that intermediate layers align better with vision embeddings. Table 5 shows performance across tasks using emebddings from layers -1 (output) to -30 (input). Layer -12 emebddings perform best, while top and bottom layers perform worse for image-text retrieval on COCO-Karparthy splits. Thus, we use layer -12 emebddings for LLaMA throughout the paper.Interleave Album Search. The queries in our FIND approach support linear complexity interleave album search. Given an image, interleave, or text input, our model can retrieve and segment all the photos in the album. Below, we show an example using the COCO validation set as the search space.Interleave Video Localization. We can formulate the video frame localization problem as an image-text retrieval task. This allows us to reason about and identify corresponding objects based on given instructions, as illustrated below. We believe FIND is useful for robot navigation.3D Feature Field. Foundation model embeddings are utilized to create a 3D feature field for robot manipulation, localization, and reasoning. We believe that the interleave embedding space, with its pixel-level understanding capabilities, has significant potential in the 3D feature field. Below, we compare a scene trained with FIND embeddings versus CLIP embeddings. Limitations. Our model is only trained and evaluated on the COCO dataset. With the limitation of data quantity, we mention that the method may not be well adapted to the in-the-wild settings.Broader Impact. Our proposed approach inherits ethical or social issues (e.g. bias amplification, privacy risks, energy consumption) of foundational models.In Section 3.2.2, we provided a comprehensive overview of the general pipeline of FIND. Here, we focus on the task-specific interface design choices. The pipeline comprises three main components: (1) Embeddings, which include prompts and queries as introduced in Section 3.2.2. Prompts are multimodal embeddings containing relevant information, while queries are learnable embeddings that aggregate information from the prompts. For instance, for image prompts (a.k.a visual features of an image) we denote them as p.image. (2) Operators, which incorporate both content and condition attention, and are responsible for information accumulation and exchange. The arrows ←, ↔ denote the attention direction. (3) Projection, which maps the queries into semantic or pixel space. Table . 6 below shows details of all task-specific design choices for the FIND interface, including embeddings, operators, and projection. Interleave Grounding image, interleave entity, interleave q.entity ← p.image q.interleave ← p.interleave p.* ↔ p.*, q.* ↔ q.* q.entity ← p.interleave Pixel, SemanticInterleave Retrieval image, interleave image, _interleave q.image ← p.image q._interleave ← p.interleave p.* ↔ p.*, q.* ↔ q.* Semantic Table 6 : Task specific FIND Interface. We define each task under the prototype of the FIND interface that enables a shared embedding space, and a unified and flexible architecture for future tasks. Where p, q stands for prompts, queries, and arrows stand for attention direction. The colors red, blue, and olive are the embeddings of vision, language, and interleave modality.The training tasks include panoptic segmentation, interactive segmentation, grounded segmentation, image-text retrieval, interleave retrieval with visual entities from the same image, and interleave grounding. Losses for each task are standardized loss functions including L BCE for binary crossentropy loss, L CE for cross-entropy loss, L DICE for dice loss, L C for contrastive loss. Below is the loss function for FIND.L = α p L CE_pano + β p L BCE_pano + γ p L DICE_pano + α g L CE_grd + β g L BCE_grd + γ g L DICE_grd + α i L CE_iseg + β i L BCE_iseg + γ i L DICE_iseg + θL VLC_imgtexr + ϕL IC_intr + α ig L CE_intg + β ig L DICE_intg + γ ig L ICE_intg (4)where, pano denotes panoptic segmentation, grd denotes grounding, iseg denotes interactive segmentation, imgtextr denotes image-text retrieval, intr denotes interleave retrieval, intg denotes interleave grounding. For more implementation details on the loss function, please refer to the code.As shown in Table . 6, the input embeddings of interleave groundings for FIND interface contain prompts and queries. Image prompts are the image features with a shape of [h × w, 512], while interleaved prompts are visual-language tokens of sentences like "A baseball player in a black and white uniform crouches on near holding a taking a break." with a shape of [l, 512] (l is the token length). Entity queries are learnable embeddings for object proposals of the image, shaped [100, 512]. Interleave queries are learnable embeddings for gathering information from the interleave prompts, shaped [n, 512], where n is the total number of meaningful entities. For example, the interleave sentence shown above has entity numbers of 4. Specifically the entity contains ['A baseball player in a black and white uniform, , , ], which is the total number of [T ] and [I] referencing to Fig. 3 .After getting a full sense of the input embeddings of interleave grounding, including p.image, p.interleave, q.entity, q.interleave. We then introduce the operation on top of those embeddings. As introduced in Sec. 3.2.2, the operations contain content attention A t and conditional attention A d . Formally we could write the attention mechanism for the specific input embeddings of interleave grounding with the following equations: q.entity, q.interleave = A t ([q.entity,q.interleave]; [p.image,p.interleave]; M t ),EQUATIONwhere A(query; key=value; M) is the attention operator with query, key, value and mask. Given the order p.image, p.interleave, q.entity, q.interleave, the content and condition attention masks are written below:EQUATIONThe index of matrix coordinates follows the input order. After the input prompts and queries are fully communicated, we will compute the projected pixel and semantic embeddings for output in the following manner: q.entity s , q.interleave s = MLP s (q.entity, q.interleave) (8) q.entity p = MLP p (q.entity)where s , p are semantic and pixel projection respectively. This way, queries are projected into semantic and pixel space to compute the final output. The dimension of q.entity s and q.entity p are both [100, 512]. In addition, q.interleave s has dimension [n, 512] where n is the entity number. With those projected queries and image features M I in the pixel projection space with shape [h, w, 512].We could get the final output mask associated with each entity with the following operation: Index = arg max dim=0 sim(q.entity s , q.interleave s ) (10)EQUATIONEQUATIONIn this way, we associate the grounding entity with the desired mask segment of the image, as shown in the top right figure in Table . 1. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our model is on the side of understanding instead of generation, so that safeguards is not applied.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited and acknowledged the prior works.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Unless we stated as interleave text retrieval, we refer to interleave visual retrieval as Fig.3.1 shown.Unless we stated as interleave text grounding, we refer to interleave visual grounding as Fig.3.2 shown. 
: 
 Figure 1 : The proposed FIND interface is generalizable to tasks that span granularity (pixel to image) and modality (vision to language). The retrieval space for this figure is the COCO validation set.Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in Fig. 1 , a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.  
 
 Appendix:  


Decision: This paper was reviewed by several experts where a plurality argued for accepting the paper.  The proposed dataset and approach seemed generally accepted by reviewers.  Some minor concerns still remain in terms of the positioning of the paper and the significance of the results.  For example, the authors argue that they only support a subset of tasks, but that seems to be contrary to the general purpose of a "foundation model" by only focusing on some tasks.  While this can be argued as common definitions, it is imprecise and should be discouraged.  The approach itself can also be seen as essentially combining multiple models, but this also has other alternatives, such as model soups.  While this wouldn't add more task flexibility, it does result in a more effective model for the tasks it does support.  These weaknesses and others noted by reviewers call into question the significance of the work, but not enough to argue for overturning the plurality recommendation by reviewers.

Reviews: review: paper_summary: paper_summary: The authors propose a benchmark for evaluation of what they call 'interleave understanding', or tasks which depend on the embeddings which are aligned across both modalities and task granularity, and they call this benchmark FIND-Bench. Find-Bench includes variants of segmentation and retrieval tasks, derived mostly from COCO datasets. The authors then propose a method for lightweight  fusion of llm and vision features with a multi-task objective, and find the model to be effective at various tasks.
strength_weakness: strength_weakness: Missing related works: The problem of Composed Image Retrieval e.g. ([1]) is quite related to interleaved retrieval. The authors should discuss the differences between the proposed work and the existing Composed Image Retrieval works. 

Missing Method Description: Specifics of the sampler, especially for text, are missing in the main paper.  In fact, it is unclear what the 'Embedding Sampler' in Figure 4(b) does. 

Missing Baselines: Although the number of baselines is relatively large, it does seem like LLaVA-type models (eg. [2]) can handle interleaved image-text inputs. Is it possible to retrieve with them? 

Improved Clarity: In some cases, clarity needs to be improved. In addition to some missing method description, there is also a missing task description of Interactive Segmentation in section 3.1.1. 

Motivation: Although the benchmark proposed is interesting, there is a limited connection to what capabilities it has the ability to unlock. Benchmarks, in order to be very impactful, should somehow be connected to an ability models should have 

[1].Saito, Kuniaki, et al. "Pic2word: Mapping pictures to words for zero-shot composed image retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Jiang, Dongfu, et al. "Mantis: Interleaved multi-image instruction tuning." arXiv preprint arXiv:2405.01483 (2024).
questions: questions: See weaknesses. Also, the FIND pipeline uses unimodal LLM and Vision Encoder, and unifies them with some training. There are many multimodal encoders these days like LLaVA[3], why not use those?

To focus the rebuttal, from my end, additional experiments supporting the weakness of missing baselines are nice-to-have but improved text clarity, related works, discussion of method is more important to me. 

[3] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).
limitations: limitations: Limitations are discussed in minimal way, but I see no major societal impacts.
paper_summary: paper_summary: The paper explores a unified multimodal embedding space across three image-text interleaved tasks, covering different granularity levels from image-level to pixel-level tasks.
strength_weakness: strength_weakness: 1. It would be valuable to explore the effectiveness of the proposed method on larger datasets.
2. The authors also mention recent multimodal models such as Llava and BLIP-v2. It would be interesting to discuss how the current approach compares with these recent methods, as they also aim for modality unification. More insights into the differences between these approaches and whether the current method achieves similar results with less data or solves problems that the aforementioned models do not address, such as pixel-level visual tasks, would be helpful.
questions: questions: Please see the weakness.
limitations: limitations: Yes.
paper_summary: paper_summary: FIND is a generalized interface for aligning foundation models' embeddings using a lightweight transformer without tuning pretrained model weights. It supports various tasks like retrieval and segmentation, is adaptable to new tasks and models, and creates a shared embedding space through multi-task training. FIND-Bench, an extension of the COCO dataset, showcases its effectiveness, achieving state-of-the-art performance in interleave segmentation and retrieval.
strength_weakness: strength_weakness: 1. This paper is somewhat difficult to understand. For example, what is the **embedding**? Is it the embedding generated by the tokenizer or the feature generated by the foundation model? And what does **interleaved** mean? My understanding is that different modalities are interleaved. It is not limited to image+text or text+text; it can be image+text+image+text. I am not sure if my understanding is correct. However, it took me many careful readings to figure this out. It would be best to clearly define and emphasize key terms and settings at the very beginning. 
2. The method in section 3.2.2 is somewhat simplistic and lacks innovation. It doesn't seem to have much new design. It needs to be clarified why this interleaved approach is challenging (for example, if all the inputs are just treated as a sequence, will the performance be significantly worse? If so, why?).  More analysis and intuition about that will be better.
questions: questions: Please see Weakness part.
limitations: limitations: No Limitations.
paper_summary: paper_summary: The paper proposes to construct an interface to connect the embeddings and predictions from different foundation models. With the designed interface, the overall system has a promise to interleave any modality in a flexible manner. To showcase this flexibility, this paper constructs a benchmark, FIND, by leveraging existing COCO labels and text-only GPT-4 model. As a generalist model, the paper shows that FIND generally performs better than existing generalist model and sometimes even specialized model.
strength_weakness: strength_weakness: Method and experiment
- Lack of baseline of instruction-tuned vision-language models, such as LLaVA.
- Lack of out-of-domain evaluation. The advantages of foundation models is that they have a higher possibility to generalize to unseen domain due to their scale. However, it is unclear when the COCO-trained interface is able to generalize to other domains.

Writing to improve
- $\textbf{sim}$ is first introduced in L109 and is later explained in L149
- Figure 3 is not self-contained. It’s hard to follow what E, O mean. There are too many notations and too little text to understand the tasks and how the task unification works here.
- In Figure 4 (center), the caption says “the shape of different polygons represents different embedding types” I cannot find where in the text specifying different “types of embeddings”.
- Missing citations in L206,207 after each model
- Where is “interleaved segmentation” in Table 3 mentioned in L256
questions: questions: Benchmark
- From Table 1, it is not clear to me how does the text-only data engine resolve the cases where there are multiple instances within the same classes. For example, in `Prompt for GPT4 Engine`, say the image contains ten mugs on a table. How does data engine resolve the mapping between the bounding boxes of each mug to the captions? 

Method
- How does the embedding sampler work for Llama? Specially, which embeddings is used? Is it the last hidden states of the last tokens? Do the authors have any insights in extracting the embeddings from an auto-regressive model? It is non-trivial to use the embeddings in an auto-regressive mdoels, this paper [1] specifically ablate the design choice.
- Does the approach involves any similarity calculation? In section 3.2.1 task unification, the authors use a similarity measurement to connect the source and the target domains. While in the Fig. 4, I feel the FIND interface is purely relied on a transformer network followed by several projection heads. Can the authors clarify the method?


Experiments
- In Table 2
  - Why SEEM, X-decoder, BLIP-2 is marked as not “jointly trained” (the third column). Please clarify it.
  - What does * mean? It’s shown in the first column and the seventh column (mIoU for RefCOCO-g).
- In section 4.1, the authors claim that the less optimal image-text retrieval results are due to the batch size during fine-tuning. In L245-248, I am very confused with the justification. Here are my questions:
  - What resolution FIND trained with?
  - “Pilot experiments with X-decoder showed … well across tasks.” Is this described in X-decode paper or is it reproduced by the authors? Also, I checked the section 4.1 in X-decoder paper, I believe the authors want to say “1024 for segmentation and 224 for image-text data”.
  - I am not sure if the authors are claiming that the main issue is the batch size or the resolutions. If it’s the batch size, why not use gradient accumulation? 


[1] Vision-Language Models Provide Promptable Representations for Reinforcement Learning
limitations: limitations: The limitations include
1. Presentation is not clear enough and has a lot to clarify
2. The paper works around foundation models but it's limited to COCO dataset. 
3. The benchmark data engine requires more qualitative examples and justification of the design choices.

Comments: title: Thanks for your feedback : )
Thank you
comment: Thanks for sharing the information Figure 1 is easy to be misunderstood, we will improve the writing according to the reviewer's comments for sure. Lastly, thanks for your encouragement on "the community can gain more insights rather than just publishing a paper".
Thank you for the prompt explanation. I just feel that there's no need to even consider the situation with multiple language decoders here, and Figure 1 can easily cause misunderstandings, leading readers to mistakenly think that your model selects one from multiple language encoders (if that's indeed how it works). Of course, this is just a minor issue; the main issue is still the writing of the paper, which makes it difficult for readers to grasp the motivation behind the work.

In any case, I hope the author can improve the writing in the revision, so that the community can gain more insights rather than just publishing a paper.  Wish the author all the best. :)
Dear Reviewer AFHH,

There are only four hours left before the discussion deadline, it would be appreciated if the reviewer could take a look at the answers and give the final decision! We again appreciate your reviewing efforts : )

Best,
Authors
Thanks for the reviewer's response, it is quite encouraging. And we again thank you for your suggestion on the gradient for paper improvement.

[Question] Multiple language encoders in Figure 1.
If reading the figure zoomed in or in detail, there is only **one solid black arrow**, which is the language encoder that we are using. We just show the potential of our model that can integrated with **one of** the language encoders or vision encoders.

Again, thanks for the understanding efforts in the rebuttal session, we will improve the camera-ready version if any.
Thanks for the author's reply, which has given me a better understanding of the paper's motivation. 

To be honest, the writing of the paper needs significant improvement (although this shouldn't be the primary criterion for judging whether the paper should be accepted). I hope the author will improve the writing in future versions of the paper. 

Additionally, regarding the method in the paper, for example, in Figure 1, there are multiple language encoders—why are multiple language encoders necessary? Isn't one sufficient? Is it a good choice to make the model redundant in order to achieve a slight performance improvement? Having multiple task decoders is fine because the tasks are varied.

After reading the author's rebuttal, I have raised my score. However, I still hope the author will improve the writing in the final version, especially focusing on the motivation and related aspects.
Thanks so much for the reviewer's response : ) We are actually on the same page for the simplicity of the design choice, especially when the reviewer gives the following comments:

*“I agree that the simpler the design of network methods, the better. The academic work that involves additional design for improving performance is just a paper”*

The reviewer actually gives comments in a neutral and constructive way. But we the authors want to emphasize, that our work is not lacking in contributions; **it represents a highly efficient and effective unification of foundation model embeddings across both granularity and modality, enabling seamless communication.** Achieving this unification is a significant, non-trivial effort. The authors focused on integrating all components with minimal overhead, which may make the individual contributions appear limited. Although we prepared similar explanations for R4-AFHH, who also found the paper challenging to understand, ***we want to re-emphasize the content here to endeavor the reviewer’s support***.

> **Impact and Novelty**
> 

On the road towards the most powerful “GPT4-V”, there are three major contents for the final models:

1. Tokenization across modality.
2. Effective communication and reasoning between modalities.
3. Detokenized to human language.

Our model focuses on the effective communication and reasoning between modalities (**Point 2**), where these components should work together seamlessly. In the multimodal section (7.2) of LLaMA 3.1 [1]. their approach is similar to ours in terms of design choices, but we extend this concept to a finer granularity (pixel-image, phrase-paragraph). **Our approach is novel in this direction, addressing an area that is essential yet underexplored**.

For approaches focusing on **instructional tuning** (e.g., LLaVA), **they primarily work on Point 3**, detokenizing into human language for steerable interaction. *We emphasize that our exploration is a parallel effort, addressing different aspects of the problem.*

> **High-Level Motivation**
> 
1. We aim to align embeddings across different granularities (e.g., segments and images for visual understanding; paragraphs and phrases for language) within foundation models like X-Decoder and LLaMA.
2. We seek to align embeddings between foundation models across modalities (e.g., Vision and Language), consistent with the Platonic representation hypothesis.
3. We aim for the aligned embeddings to communicate seamlessly and efficiently.
4. We want to identify the optimal design choices for embedding extraction from different foundation models.
5. We believe all understanding tasks fall under the broader scope of information mapping.

> **Interface Design aligned with Motivation**
> 
1. We design the interface to incorporate queries and prompts that unify granularity within a foundation model. Specifically, queries can attend to an arbitrary number of prompts through content attention, enabling unified granularity embeddings.
2. Additionally, since queries can attend to prompts across different modalities through content attention, we align these modalities within the same representation space via queries.
3. The condition attention allows queries that span granularity and modality to communicate efficiently.
4. We utilize object queries for vision embeddings and exhaustively test which layer embeddings are optimal for language models like LLaMA.
5. Projection and similarity mappings enable the unification of all the recognition tasks under a consistent framework.

> **Experiments proves the Design choice and Motivation**
> 

We evaluated generic segmentation, interactive segmentation (Table 2, Main Paper), and text-paragraph grounding (Fig. 1, Second Row, Main Paper) to demonstrate unification across granularity within the foundation model, in both vision and language.

1. We assessed grounded segmentation and image-text retrieval (Table 2, Main Paper) to validate cross-modality alignment (vision and language). Additionally, we confirmed that this alignment is invariant across foundation models trained independently (Table 4, Main Paper).
2. To evaluate the effectiveness of cross-granularity and cross-modality communication, we developed the FIND-Bench for interleave grounding and retrieval. FIND-Bench is compared with other models focused on information fusion in Table 3.
3. The design choices for vision have been explored in previous works (e.g., Mask2Former, X-Decoder, SAM). We conducted an ablation study on language design choices in Table 5 (Main Paper, Lower Part) to determine how to extract the correct information from language models.
4. In Table 5 (Upper Part), by gradually removing each task in FIND, we demonstrated the effectiveness of similarity mapping on recognition tasks.

[1] The Llama 3 Herd of Models
Thanks very much for the author's response. Regarding the question about Q3, I think the author misunderstood my point. I agree that the simpler the design of network methods, the better. The academic work that involves additional design for improving performance is just a paper; the industry would never adopt complicated methods, and in this regard, I believe I am on the same view as the author. However, simplicity does not mean a lack of contribution, or perhaps the way the author presents their ideas makes it difficult for me to grasp the key points of contribution. 

Figure 4 is particularly difficult to understand.

If the other reviewers find it easy to understand, AC can disregard my opinion. :) 
I apologize, but I still prefer to maintain my original score.
We again appreciate your review **actually helped us rethink our model in a more logical way** than ever, and we hope ***the following contents could facilitate your understanding towards the final decision*** because we do believe our work is *novel and inspiring for future works.* We also aligned your question with the structure provided below.

> **High-Level Motivation**
> 
1. We aim to align embeddings across different granularities (e.g., segments and images for visual understanding; paragraphs and phrases for language) within foundation models like X-Decoder and LLaMA.
2. We seek to align embeddings between foundation models across modalities (e.g., Vision and Language), consistent with the Platonic representation hypothesis.
3. We aim for the aligned embeddings to communicate seamlessly and efficiently.
4. We want to identify the optimal design choices for embedding extraction from different foundation models.
5. We believe all understanding tasks fall under the broader scope of information mapping. → [Q3, Q13]

> **Interface Design aligned with Motivation**
> 
1. We design the interface to incorporate queries and prompts that unify granularity within a foundation model. Specifically, queries can attend to an arbitrary number of prompts through content attention, enabling unified granularity embeddings. → [Q6, Q10]
2. Additionally, since queries can attend to prompts across different modalities through content attention, we align these modalities within the same representation space via queries. → [Q6, Q10]
3. The condition attention allows queries that span granularity and modality to communicate efficiently. → [Q4]
4. We utilize object queries for vision embeddings and exhaustively test which layer embeddings are optimal for language models like LLaMA. → [Q11]
5. Projection and similarity mappings enable the unification of all the recognition tasks under a consistent framework. → [Q5, Q13]

> **Experiments proves the Design choice and Motivation**
> 

We evaluated generic segmentation, interactive segmentation (Table 2, Main Paper), and text-paragraph grounding (Fig. 1, Second Row, Main Paper) to demonstrate unification across granularity within the foundation model, in both vision and language.

1. We assessed grounded segmentation and image-text retrieval (Table 2, Main Paper) to validate cross-modality alignment (vision and language). Additionally, we confirmed that this alignment is invariant across foundation models trained independently (Table 4, Main Paper).
2. To evaluate the effectiveness of cross-granularity and cross-modality communication, we developed the FIND-Bench for interleave grounding and retrieval. FIND-Bench is compared with other models focused on information fusion in Table 3. → [Q2, Q9]
3. The design choices for vision have been explored in previous works (e.g., Mask2Former, X-Decoder, SAM). We conducted an ablation study on language design choices in Table 5 (Main Paper, Lower Part) to determine how to extract the correct information from language models.
4. In Table 5 (Upper Part), by gradually removing each task in FIND, we demonstrated the effectiveness of similarity mapping on recognition tasks.

> **A Missing Piece for Alignment**
> 

We believe that embeddings from different models within the same modality can also effectively communicate. Several concurrent pilot studies, such as [1, 2, 3], are exploring this direction.

> **Future Research**
> 

On the road towards the most powerful “GPT4-V”, there are three major contents for the final models:

1. Tokenization across modality.
2. Effective communication and reasoning between modalities.
3. Detokenized to human language.

Our model focuses on the effective communication and reasoning between modalities (**Point 2**), where these components should work together seamlessly. In the multimodal section (7.2) of LLaMA 3.1 [4]. their approach is similar to ours in terms of design choices, but we extend this concept to a finer granularity (pixel-image, phrase-paragraph). **Our approach is novel in this direction, addressing an area that is essential yet underexplored**.

For approaches focusing on **instructional tuning** (e.g., LLaVA), **they primarily work on Point 3**, detokenizing into human language for steerable interaction. ***We emphasize that our exploration is a parallel effort, addressing different aspects of the problem.*** → [Q1]



[1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

[2] MouSi: Poly-Visual-Expert Vision-Language Models

[3] Theia: Distilling Diverse Vision Foundation Models for Robot Learning

[4] The Llama 3 Herd of Models
Thanks for your positive comments : )
Dear Reviewer kpHd,

We sincerely appreciate your thorough review of our paper and the valuable feedback you have provided. We would be grateful if you could review our rebuttal and share any additional feedback or questions you might have. If the rebuttal clarifies any points of confusion or if the comments from other reviewers (such as iZKM and AFHH) offer new insights into the paper, we kindly ask you to consider re-evaluating your rating. Thank you once again for your time and efforts!

Best,
Authors.
After reading the rebuttal, I will maintain my positive score.
After the FIND-SAMv2-Bench is prepared, we evaluate the interleave grounding with the FIND model. Because the video/frame number is too few in SAV val and test sets we do not propose interleave retrieval evaluation here. 

For interleave grounding, we use the following protocol to generate the query: For each entity we want to do the grounding, 0.5 probability is applied to determine whether we want to use visual or text reference for grounding. An example grounding example is shown below:
```
[1]<A woman> sitting next to [2]<a handsome man>, with their hands holding together under [3]<the blue sky>.
```
Thus, <> would either be text or visual reference of the instance. The visual reference is the masked scribble part of the original instance in the image, one better option could be the tracked instance in another frame.

We evaluate the FIND-SAMv2-Bench on our Tiny and Large (d5) model with both UniCL (CLIP style language encoder) and LLaMA language encoder. As shown in the table [SAMv2 Experiment results], in addition to the effective interleave grounding in the out-of-domain dataset, we have also observed that our interface is more effective when the vision backbone is stronger (e.g. davit-d5-Florence) with evidence on the comparison results between LLaMA or UniCL language encoder.

To further prove the effectiveness of our FIND approach, we compared it with the strongest multimodal baseline of GPT4-o, as it is not able to do interleave grounding from scratch, we use SoM [1] as the adapter to bridge pixel-level visual grounding in GPT4-o. It is clearly shown in the table [SAMv2 Experiment results], that FIND similar results on AP50 metrics with GPT4-o while 58 times faster than GPT4-o.

The exact prompt for evaluating GPT4-o are shown below:
```
You are given a marked image with masks and number in each region.
Given the full sentence {}, and the corresponding viusal reference, you are instructed to select the region that best matches the query entities {}.
The output format should be: ##output##: [Entity_id]: (Region number), [Entity_id]: (Region number), ...
```

[1] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023).
Thanks so much for your reply, **feel free to ask any further questions if needed** : )

For the additional table in [Q2], we leverage the newly released SAMv2 [1] dataset that originally worked on video object/part segmentation. We provide a link [2] below for your reference on the dataset explorer. The dataset contains one validation set, and one test set, we generate new benchmark labels for those videos. SAMv2 (a.k.a SAV) dataset has the following benefits in solving your confusion:

(1) Video datasets are definitely in another domain compared with the COCO dataset, e.g. they contain motion blur and the scene is usually different from COCO-style scenes. This will solve your confusion on [Q2] better.

(2) Benchmarking on the video dataset will unleash more potential for our model, we have observed qualitatively that FIND can do interleave video tracking on SAMv2 dataset. If there is a future version, we will add those results (the rebuttal cannot add figures or links).

-> We create the FIND-SAV-Bench using the following protocol:

(1) We generate instance-level annotations using SEEM-d5 [3] and filter out the annotations with (a) Low confidence. (b) Very small region.

(2) We use Set-of-Mark prompting [4] to annotate the visual images with the generated annotations. This will give better results than purely using language prompts as shown in Table.1 (Main paper).

(3) We prompt GPT4-V with the following exact prompt, this is an improved version of the method shown in Table.1 in solving the lack of fine-grained annotation in SAMv2:

```
1. Can you describe each instance in the [image1] with detailed appearance, in the order of how confident you are on the recognition? \n

2. Can you selecte 3-5 instances in the [image1] that also likely appear in the [image2]? \n
After these steps, generate image captions with grounded entities and attributes with following instructions: \n
1. [image1] is the sampled image to generated grounded caption. \n
2. [image2] is the reference image help to select which entities in [image1] should be included in the generated caption. \n
3. Numbered instances in [image1] are the proposed candidate entities. \n
5. The number in [image1] and [image2] are different, you should use in the number in [image1]. \n
an example output format would be: ##output##:"[entity_id]<A woman> sitting next to [entity_id]<a handsome man>, with their hands holding together under [entity_id]<the blue sky>.", where [entity_id] and <xxx> are associated with the ground truth bounding boxes. \n
generated caption constraints:
1. [entity_id] should be the same as the number in [image1], e,g: [1]. \n
2. Try to find the entities in [image1] that also appear in [image2]. \n
3. The selected entity should be the instance you are very confident on recoginition, selecte around 3-4 entities would be fine. \n
4. The entity description in <> should be accurate and detailed to identify the instance, e.g. <the dog with black and white dot>, <the second bottle from the front>. \n
5. Focus more on instance classes instead of stuff classes. \n
Please generate the grounded caption for [image1] accordingly.\n
```

(4) We do human pruning (approve/decline) to remove any ridiculous examples.

The final examples of FIND-SAMv2-Bench are shown in the **rebuttal PDF** with the section name "Qualitative examples on FIND-SAMv2-Bench". And the statistics are shown in the [SAMv2 Benchmark statistics], with the metrics below:

(1) # Images (videos): The total number of videos/images in the splits, because we only annotated the first image in a video for benchmarking as the frame looks similar inside one video. This labeling strategy will maximize evaluation capability with minimum effort.

(2) # Instances: Total number of instances annotated in the split.

(3) Avg Text Length: The average sentence length of the grounding annotation in character.


[1] Ravi, Nikhila, et al. "SAM 2: Segment Anything in Images and Videos." arXiv preprint arXiv:2408.00714 (2024).

[2] https://ai.meta.com/datasets/segment-anything-video/

[3] Zou, Xueyan, et al. "Segment everything everywhere all at once." Advances in Neural Information Processing Systems 36 (2024).

[4] Yang, Jianwei, et al. "Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v." arXiv preprint arXiv:2310.11441 (2023).
Thanks for the detailed clarification.

I read through the rebuttal. Overall, I start to appreciate the idea of the proposed framework *in high-level*, but I need some time to put the submission and the clarification in the rebuttal together to refine my understanding. 

To facilitate the discussion, I want to start by adding a follow-up question:
1. I appreciate the effort for the new dataset [Q2]. Can the author give some interpretations of the provided table?
Thanks so much for your prompt reply! We are happy that the rebuttal materials solve your confusion, for the embedding sampler, it seems that this creates confusion for nearly all the reviewers, so that we will correct this in the camera-ready version if any. Thanks again for increasing the rating!
I thank the authors for their responses. They have adequately addressed most clarity concerns (e.g. comparison to Pic2Word) and discussion (e.g. details on embedding sampler). Therefore, I raise my score my one point. I strongly encourage all clarifications to make it into any final version. Moreover, because the embedding sampler is identity for the most part, my personal take is that introduces unneeded complexity into the paper writing. I encourage the authors to consider this perspective in any further version.
**[Common Question 4]**: Details on **task unification across modality and granularity**.

In Section 3.2.2 (Main Paper), we provided a comprehensive overview of the general pipeline of FIND. Here, we focus on the task-specific interface design choices. The pipeline comprises three main components: (1) Embeddings, which include prompts (p), and queries (q) as introduced in Section 3.2.2 (Main Paper). Prompts are multimodal embeddings containing relevant information, while queries are learnable embeddings that aggregate information from the prompts. For instance, for **image prompts** (a.k.a visual features of an image) we denote them as **p.image**, *other notations in the table below also follow this naming protocol*. ****(2) Operators, which incorporate both content and condition attention, and are responsible for information accumulation and exchange. The arrows ←, ↔ denote the attention direction. (3) Projection, which maps the queries into semantic or pixel space. The table below shows details of all task-specific design choices for the FIND interface, including embeddings, operators, and projection.

| Task | Prompts | Embeddings | Content Attention | Condition Attention | Projection |
| --- | --- | --- | --- | --- | --- |
| Generic Segmentation | image, class | object, class | q.object ← p.image | p.* ↔ p.*, q.* ↔ q.* | Pixel, Semantic |
| Grounded Segmentation | image, text | grounding, text | q.grounding ← p.image, q.text ← p.text | p.* ↔ p.*, q.* ↔ q.* | Pixel, Semantic |
| Image-Text Retrieval | image, caption | image, caption | q.image ← p.image, q.caption ← p.caption | q.grounding ← p.text | Semantic |
| Interactive Segmentation | image, spatial | segment, spatial | q.segment ← p.image, q.spatial ← p.spatial | p.* ↔ p.*, q.* ↔ q.* | Pixel, Semantic |
| Interleave Grounding | image, interleave | entity, interleave | q.entity ← p.image, q.interleave ← p.interleave | q.entity ← p.interleave | Pixel, Semantic |
| Interleave Retrieval | image, interleave | image, _interleave | q.image ← p.image, q._interleave ← p.interleave | p.* ↔ p.*, q.* ↔ q.* | Semantic |

*Table 1. Task-specific FIND Interface. We define each task under the prototype of the FIND interface that enables a shared embedding space, and a unified and flexible architecture for future tasks. Where p, q stands for prompts, queries, and arrows stand for attention direction.* 

A comprehensive case study on Interleave Grounding is in the **rebuttal PDF**.
[Q16] What resolution FIND trained with?

We have clearly stated in L245-248:

*“In Table 2, models are either 384x384 with batch size 384 or 1024x1024 with batch size 192 for all tasks. Other tables show results with a 640x640 training resolution and a 192-batch size.”*

[Q17] “Pilot experiments with X-decoder showed … well across tasks.” Is this described in X-decode paper or is it reproduced by the authors?

The experiments are produced by the authors.

[Q18] Presentation is not clear.

We hope you have a better understanding after the rebuttal and will update the camera-ready version if needed. As Reviewer-kGcR noted, “The writing and visualizations are clear and well-presented.”

[Q19] Limitation to COCO dataset.

Thank you for the suggestion. We have addressed this by introducing the new FIND-SAMv2-Bench. Please refer to [Q2] for more details.

[Q20] Need more qualitative examples for the data engine and justifications.

We have provided additional justifications for the design in our response to [Q9], and more qualitative examples are included in the **rebuttal PDF**.