OLIVE: Object Level In-Context Visual Embeddings (2024)

Abstract: Recent generalist vision-language models (VLMs) have demonstrated impressive reasoning capabilities across diverse multimodal tasks. However, these models still struggle with finegrained object level understanding and grounding. In terms of modeling, existing VLMs implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and inevitably introduces noisy spurious background features. Additionally, these models struggle when generalizing to unseen visual concepts and may not be reliable for domain-specific tasks without further fine-tuning. To address these limitations, we propose a novel method to prompt large language models with in-context visual object vectors, thereby enabling controllable object level reasoning. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Furthermore, we propose region-level retrieval using our object representations, facilitating rapid adaptation to new objects without additional training. Our experiments reveal that our method achieves competitive referring object classification and captioning performance, while also offering zero-shot generalization and robustness to visually challenging contexts.1

heading: 1 Introduction
text: Despite the popularity, many existing VLMs such as LLaVA (Liu et al., 2023), MiniGPT4 (Zhu et al., 2023a), and mPLUG-OWL (Ye et al., 2023) handle the entire image for visual understanding, leading to two major shortcomings. First, these VLMs use a visual transformer to split an image into a grid of image patches and embed them into a lengthy array of image patch embeddings that have object level features scattered around different positions of the array. This leads to the different granularity
1Our code and models are available at https://github. com/tossowski/OLIVE
between the image patch tokens and text tokens, further creating difficulty in aligning and grounding visual objects to text concepts. Second, feeding all image patch embeddings to the large language model (LLM) decoder is problematic due to the resulting long context and inefficiency of including in-context examples from multiple images.
To improve fine-grained visual alignment, recent region-based VLMs are pre-trained to integrate object level information into the LLM decoder. GPT4ROI (Zhang et al., 2023b) pre-trains LLMs to understand ROIAlign features (He et al., 2017) extracted from bounding boxes. Other similar methods such as Shikra (Chen et al., 2023) or Kosmos-2 (Peng et al., 2024) ground and refer to objects using text in multimodal referential dialogues. FERRET (You et al., 2023) and ViP-LLaVA (Cai et al., 2023) further support free-form shapes as referring input by summarizing visual features sampled within the region of interest. Although these methods provide improvement to object level reasoning, they still fail at recognizing unseen/rare objects and are sensitive to spurious background features, as shown in §5. Even powerful closed-source multimodal models such as GPT4V are unreliable to deploy in high-stakes domain-specific situations such as the medical domain (Senkaiahliyan et al., 2023).
A straightforward way to handle generalization to unseen visual content is to integrate a retrieval component. Methods such as REVEAL (Hu et al., 2023) and MuRAG (Chen et al., 2022) provide retrieved multimodal facts as supplementary context to help VLMs generalize to new concepts without further training. However, these models do not consider object level retrieval and in-context prediction. Models such as Flamingo (Alayrac et al., 2022) and Qwen-VL (Bai et al., 2023) allow for in-context examples from multiple images, yet do not support object level retrieval and reasoning.
To address the above issues, we propose to encode object level in-context visual embeddings
5170
(OLIVE) to enhance LLMs with region-level reasoning capabilities. Critically, we omit lengthy image patch features and encode visual object embeddings by a lightweight encoder of 20 million parameters, allowing for faster training and direct connection to existing LLMs. This preserves the full functionality of the original LLMs, while also introducing novel multimodal reasoning abilities. Furthermore, our object level retrieval module allows for more precise queries and retrieved information to help the model adapt to domain-specific tasks with limited training data. Our contributions are summarized below and in Table 1:
• We propose a lightweight object encoder that can be connected to existing LLMs to enable controllable object level multimodal reasoning with free-form input annotations.
• Our model omits image patch features and summarizes object features into a single vector, significantly reducing context length for more efficient training and inference, and allowing for in-context examples from multiple images.
• We conduct extensive experiments with regionretrieval of object level features and showcase rapid adaptation to unseen visual concepts.

heading: 2 Preliminaries
text: Generative VLM Architecture Recent generative VLMs (e.g., LLaVA, BLIP-2) adopt a similar architecture that connects a pre-trained visual encoder ϕv and a pre-trained language model decoder ϕt through a lightweight fusion neural network, denoted as ϕc. Specifically, the fusion module first uses a projection function to map a visual feature v ∈ V to the text embedding space X of the language model decoder, and then fuse the visual and text embeddings as input to language model decoder. Formally, given an image v and a text prompt x, the decoder takes in the combined feature x to autoregressively predict the output y.
xt =TxtEmbed(x;ϕt) ∈ X (1) v =ImgEncoder(v;ϕv) ∈ V (2) xv =Project(v;ϕc) ∈ X (3) x =Fuse(xv,xt;ϕc) (4)
pvlm(y|v, x) = |y|∏
j=0
pϕt(yj |x, y<j) (5)
Different from prior fusion modules (e.g., linear projection in LLaVA, gated cross-attention in
Flamingo, and Q-former in BLIP-2) that project the whole image features, we propose an object level encoder (§3.1) that captures fine-grained region features and speeds up training and inference.
Visual Instruction Tuning We adopt a similar visual instruction-tuning approach as Liu et al. (2023) by fine-tuning parts of the VLM parameters (e.g., ϕc and/or ϕt) on instruction-following data. The training objective is based on maximum likelihood estimation for next-token predictions given the input image and the text prompt. Different from prior work using pure text prompts, our object encoder and retrieval module (§3.1, §3.2) enables the usage of code-switched prompt sequence mixing text tokens and image object tokens, and the rapid adaptation to unseen domains via in-context prediction.

heading: 3 Method
text: This section as well as Figure 1 highlights the main components of our method. We first design an object encoder (§3.1) to learn visual object embeddings in a shared vision-text space, then apply a similarity search over object embeddings to retrieve relevant visual objects (§3.2), and finally construct a code-switch multimodal prompt to integrate the retrieved object information for generation (§3.3).

heading: 3.1 Object Encoder
text: Following popular region-grounded models such as FERRET (You et al., 2023), we allow for free-form annotation of objects using the object segmentation mask omask as input. Specifically, we first encode an image v with a vision transformer (Dosovitskiy et al., 2020) to obtain patch-level features v:
v = ImgEncoder(v;ϕv) ∈ R(n 2+1)×d, (6)
where n is the grid size and d is the dimension of hidden states. To further obtain an object level feature vobj from the image, we first extract a subset of the image features vmasked corresponding to the binary object segmentation mask omask:
vmasked = v[Flatten(omask)] ∈ Rl×d (7)
where omask is a n × n binary matrix, indicating the corresponding image patches occupied by an object in the image, and l denotes the number of the occupied patches. These segmentation masks can be created by automatic segmentation tools such as SAM (Kirillov et al., 2023) or provided by human selection on the image. The segmentation
mask is first flattened and used to select object patches vmasked from v. Finally, we obtain the object embedding by compressing vmasked into a single vector vobj.
vobj = ObjectEncoder(vmasked;ϕc) ∈ Rd, (8)
where the object encoder uses a lightweight 2-layer transformer that acts similar to a visual resampler (Zou et al., 2023a; Li et al., 2023), followed by a learnable linear layer to further project the visual representation to the text space (Liu et al., 2023).

heading: 3.2 Visual Object Retrieval
text: In many cases, the object of interest does not resemble anything seen during training. With our visual object embeddings, we can easily perform object level retrieval to match an open class of visual objects and integrate the retrieved information into the language decoder for predicting unseen or rare objects from specific domains (e.g., biomedicine). To this end, we assume access to a retrieval set R = {(oi, di, vi)}mi=1, where each triple consists
of an object’s segmentation mask oi, the object’s text description di and the image vi containing this object. To retrieve relevant objects from R, we use a similar object encoding as §3.1 except that we use the mean pooling of vmasked as the object encoder in Eq. (9), since this simple strategy does not require any learnable parameters for projection to the text embedding space and visual object embeddings can be pre-computed before any fine-tuning. However, we use a learnable object encoder in Eq. (8) to connect object embeddings to the LM decoder during instruction-tuning for text generation (§3.3).
vobj = MeanPool(vmasked) ∈ Rd, (9)
During retrieval, we compute a query vector vquery for a given object, and compute the cosine similarity scores between vquery and all the visual object embeddings from R to obtain the top k closest triples, denoted as K = {(oi, di, vi)}ki=1.

heading: 3.3 In-context Prompt Construction
text: As the visual object embeddings are projected into the text embedding space of the LM decoder, this
allows us to construct a code-switched prompt that mixes visual objects with text tokens for the LM decoder (e.g., Llama 2 (Touvron et al., 2023)). In addition, as our object encoder compresses a visual object into a single vector vobj, this significantly shortens the length of the visual tokens that the LM decoder needs to fuse with text tokens. Therefore, we can easily integrate multiple retrieved object embeddings into the prompt to augment the LM decoder for in-context text generation. Specifically, we define a special vocabulary token [obj] which can be inserted flexibly in the user prompt x. For example, the user can ask “[obj] Describe this part of the image" to perform region-level description. The embedding of this token is directly replaced with its corresponding visual object embedding. Formally, given a text prompt x that contains indexed [obj] tokens referring to an object vobj of interest in an image v and its relevant objects in K, we define a prompting function that replaces the text embedding of [obj] with its corresponding visual object embedding, and integrates the top k most similar objects K as in-context examples. For example, a prompt with retrieved in-context examples can be “The top [k] related objects are: [obj_1] is a [label],...[obj_k] is a [label]. [obj_query] What is this?”. We provide more details about in-context prompt templates and construction in Appendix A.
x = Prompt(x,vobj,K) (10)
Finally, we feed the multimodal prompt x into the LM decoder for text generation following Eq. (5). Note that compared to prior VLMs (e.g., LLaVA) that directly fuse the patch-level features v of the whole image (Eq. 6) with object information scattering around different positions in v, our object encoding is computationally more efficient and speeds up the training that involves multiple in-context objects in the multimodal prompt.

heading: 4 Experimental Settings
text: In this section, we first describe two main objectlevel tasks for evaluation (§4.1) together with the datasets used (§4.2). Finally, we describe three variants of our model (§4.3), the training details §4.4), and the other baselines in comparison (§4.5).

heading: 4.1 Object-level Tasks
text: Referring Object Classification Given an object referred by its image location (e.g. segmentation mask/bounding box), the model is instructed to
generate a text that predicts the object’s class label in a predefined label set, C ∈ {c1, c2, ...cn}. We provide the ground truth segmentation mask to eliminate localization errors and focus on evaluating the models’ understanding of image objects.
Referring Expression Generation Given an input image object referred by a segmentation mask, the model is instructed to generate a natural language expression which semantically matches multiple ground-truth references R ∈ {r1, r2, ...rm}. We use METEOR (Banerjee and Lavie, 2005) and CIDEr (Vedantam et al., 2015) score for evaluating generated description quality.

heading: 4.2 Datasets
text: This section describes the different datasets used in our experiments, with more details in Appendix C.
Common Objects in Context (COCO) (Lin et al., 2014) is a popular visual reasoning dataset with over 800,000 object-level annotations for 80 categories of objects. We use it to train our model to understand region input since it contains highquality segmentation annotations. We use the standardized train and validation 2017 splits for the detection task, and discard a few (<1%) small segmentation annotations that fail to be converted into a binary mask. Following (Zhong et al., 2022), we evaluate in the setting where ground-truth segmentations are provided as input to eliminate localization errors. We use the standard metric of mean average precision (mAP) for object detection using the COCO API,2 as well as overall accuracy.
refCOCOg (Kazemzadeh et al., 2014) is a variant of the COCO dataset with about 50,000 annotations for objects and their description. We use the data to train our model to describe image regions and use their standardized train/validation split.
ChestX-Ray8 (CXR8) (Wang et al., 2017) is a medical dataset consisting of 108,948 frontal-view X-ray images. The image annotations for the 8 possible pathologies are text-mined from the radiology reports using NLP tools. A small subset of 984 images contains bounding box annotation of the pathology. We use this subset for our zero-shot domain adaptation experiments, splitting the data into 16% retrieval set and 84% test data. The retrieval set consists of 20 examples of each pathology, and we use overall accuracy as the evaluation metric.
2https://github.com/cocodataset/cocoapi

heading: 4.3 OLIVE Variants
text: OLIVE-R (Retrieval-only) This retrieval-only method predicts the answer to the user question by taking a majority vote of the top k retrieved examples. For simplicity, we fix k = 5 for this setting unless otherwise specified and analyze the effect of k in Figure 6. Although simple, this baseline proves to be effective and provides salient additional context as described in §4.3. However, this discriminative model does not allow for free-form text generation for tasks such as region captioning.
OLIVE-G (Generative-only) This model is trained to generate free-form text based solely on the user question and corresponding object features. We omit the retrieved information to observe the capability of the standalone object representations. We find that even without retrieval, the model can learn to perform more challenging object-level tasks such as region description. The final decoder input can be expressed as a variant of Eq. (10):
x = Prompt(x,vobj). (11)
OLIVE-RG (Full) Our full model generates text outputs based on in-context object examples from retrieval. The multimodal in-context prompt is constructed using Eq. (10). This prompt includes the retrieved object features, their labels, and their similarity scores. The exact construction can be found in Appendix A. The top k retrieved multimodal documents in K are obtained using the same retrieval described in §3.2 and ordered in increasing relevance score. Both OLIVE-G and OLIVE-RG use greedy decoding for text generation.

heading: 4.4 Training Details
text: Our model uses a frozen ViT-L/14 vision transformer from a CLIP model to obtain patch-level features. For our LLM backbone, we use either Llama 2-7B or GPT-2 (124M) (Radford et al., 2019). The LLM is finetuned with LoRA (Hu et al., 2021) as we find this improves model performance. We use the train splits of two different region-level datasets (i.e., COCO, refCOCOg) as our training data for their respective tasks, and evaluate models on their corresponding validation splits because their test data does not have object-level annotation. More details are in Table 7 and we leave the other hyperparameter search to future exploration. We additionally find that we can train a multi-task model by combining the datasets for all object-level tasks (Details in Appendix E).

heading: 4.5 Other Baselines in Comparison
text: CLIP (Radford et al., 2021) Contrastive Language Image Pretraining learns a joint visionlanguage space between images and their matching captions. We use this method for zero shot object classification by predicting the target with the highest cosine similarity to the cropped region.
BioMedCLIP (Zhang et al., 2023a) The authors train a CLIP model aligned to biomedical imagetext pairs, achieving state of the art on a variety of medical tasks. We use this model as a baseline for object classification in the medical domain.
RegionCLIP (Zhong et al., 2022) This model learns region-text level alignment through softlabels obtained from CLIP. We use it for referring object detection based on ROIAlign features.
Kosmos 2 (Peng et al., 2024) This generative VLM trains a LLM decoder to perform a variety of visual grounding tasks from their newly introduced grounded image-text (GRIT) dataset. We compare with their results on referring expression generation on the refCOCOg dataset.
Flamingo (Alayrac et al., 2022) This generative model learns to connect frozen visual features and LLMs by training on interleaved image-text data. We evaluate Flamingo’s few-shot performance on referring expression generation on cropped image regions. We use an open-source implementation trained on the multimodal C4 (Zhu et al., 2023b) and LAION-2b (Schuhmann et al., 2022) datasets.

heading: 5 Results and Analysis

heading: 5.1 Referring Object Classification
text: Unseen Object Classification One of the benefits of our retrieval augmented system is its rapid
generalization to unseen visual concepts. We estimate this capability by training on the COCO dataset and evaluating object classification on an unseen medical dataset which has drastically different types of images and limited training data. Table 2 illustrates the performance of our method on the CXR8 dataset in either a classification or generative setting. Even with as little as 20 examples per class in the medical retrieval set, OLIVE-R achieves competitive performance compared to domain-adapted models (i.e., BioMedCLIP), which we hypothesize is because of our region-level retrieval and in-distribution retrieval set. We also note that our generative approach OLIVE-RG can utilize the retrieved in-context examples and achieve similar performance to BioMedCLIP, despite only being trained on COCO images. Without retrieval, the generative model fails catastrophically with 0% accuracy, and zero-shot CLIP achieves about the same performance as random guessing.
Rare Object Classification We also investigate our model’s performance on rare, but seen objects. Figure 3 shows our method’s performance on the top 5 rarest classes in the COCO dataset. For OLIVE-G and OLIVE-RG, we use a 224 pixel resolution visual encoder to match the CLIP visual encoder. OLIVE-G tends to have lower performances on the rare classes. However, when combining retrieval with parameterized methods in OLIVE-RG and OLIVE-RG-336px, the performance on rare classes improves significantly, with OLIVE-RG-336px performing better than CLIP on all rare classes. OLIVE-RG also achieves better performance on three out of five classes despite being trained on less data. Our model’s overall performance can be found in Appendix D.

heading: 5.2 Referring Expression Generation
text: Captioning Unseen Objects In addition to referring object classification, we investigate our model’s ability to caption out-of-distribution objects. Figure 2 illustrates an example of asking our model to describe animals not seen during training. Without retrieval, OLIVE-G fails to describe the shark and turtle. However, after manually adding just 5 labeled objects of turtles and sharks to the existing retrieval set, OLIVE-RG accurately describes the object and provides supporting examples for its prediction. The label description for each object in the retrieval set is only the name of the animal, but the model generates additional characteristics in its description. Appendix B shows more examples of zero-shot adaptation to unseen visual concepts in the object classification setting.
Challenging Visual Context To test the quality of the representations generated from our object encoder, we qualitatively evaluate our model prediction in adversarial visual contexts. Figure 7 shows a white dog and a black cat in a “yin-yang” shape. We observe that free-form annotation allows for more precise user queries and object descriptions, and illustrates other properties such as scene content awareness and patch-level details as shown in Appendix B. While many VLMs can accurately understand normal scenes, Figure 4 illustrates an example in which an object-level representation may be necessary, with recent works struggling to caption the snowboarder on the beach. The detailed performance of our model on the refCOCOg captioning task can be found in Appendix F.

heading: 5.3 In-context Example Size
text: Since our method omits image patch features and compresses object information into a single vector, it can process many objects from different images at once. In Figure 5, we highlight the difference in context length for various methods when prompted with multimedia examples. We assume an average prompt length of 30 accompanying each in-context image example for all models. Even approaches designed for interleaved image-text data such as Flamingo insert multiple latent vectors for each image, incurring a higher cost than our approach.

heading: 5.4 Sensitivity on Retrieval: Coverage and k
text: In Figure 6 we analyze the effect of changing the size of the object retrieval set as well as the number of retrieved examples, k. To thoroughly test various settings, we evaluate the retrieval-only based approach (OLIVE-R) on the validation split of the COCO dataset using different sized subsets of the training data for retrieval. We ensure the retrieval set contains an equal amount of each object class when possible. Our results indicate that the optimal value of k depends on the size of the retrieval set. With a small retrieval dataset (red), performance is lower and the optimal k tends to be smaller. Larger retrieval sets (blue, green) benefit from retrieving more examples and have greater performance.

heading: 5.5 Object Vector Visualization
text: Having a single vector representation for each object allows for visualization using dimensionality reduction. In Figure 8, we perform principal com-
ponent analysis (PCA) on the hidden states of object vectors at different layers in the LLM decoder. We plot 200 examples from each of 10 object categories and note several patterns. First, objects from the same class tend to appear together, even though they appear in different visual contexts. This suggests that the object encoder has semantic understanding of the visual concepts. Second, the object vectors naturally form hierarchical clusters where objects from the same super class such as vehicle, animal, or fruit have overlapping clusters. Lastly, the clustering appears similar across all layers, with only minor variations.

heading: 6 Related Work
text: Grounding in Language and Vision A popular approach for aligning vision and language embeddings is contrastive learning methods such as CLIP and ALIGN (Li et al., 2021). However, these methods align the entire image representation, leading to poor reasoning on image details for downstream vision language tasks. RegionCLIP (Zhong et al., 2022) and GLIP (Li et al., 2022) address this issue by proposing fine-grained alignment with region-text pairs during pretraining. GLIPv2 (Zhang et al., 2022) further improves the pretraining and alignment by introducing localization, detection, and other tasks. Another recent popular approach involves training models on automatically curated region-level data from image-caption pairs (Peng et al., 2024). Many other works focus on region-level alignment during pretraining for greater vision-language understanding (You et al., 2023; Chen et al., 2023; Zeng et al., 2022b,a). More generally, a recent study (Bugliarello et al., 2023) shows that VLMs with fine-grained object-level
pretraining such as X-VLM (Zeng et al., 2022a) have better reasoning ability. Other works align vision and language using regularization or loss to create relation aware cross attention between modalities (Pandey et al., 2023; Ren et al., 2021).
Visual Resampling Visual resampling is a popular technique to compress long sequences of image features into a few rich vector representations. This is achieved by constructing a fixed amount of learnable vectors that attend to the visual features through cross-attention layers. Models such as BLIP (Li et al., 2023) first explore this idea to connect frozen vision features to LLMs efficiently by summarizing the content of the image. Other methods including X-Decoder (Zou et al., 2023a) or SEEM (Zou et al., 2023b) use resampling to encode various types of prompts or intents which improve the LLM decoding ability. Additionally, works such as Flamingo (Alayrac et al., 2022) and QwenVL (Bai et al., 2023) show that multiple images can be inserted in-context to the prompt by compressing image features with resamplers, enabling few shot capabilities. Our work visually resamples object representations for object-conditioned text generation, and only uses a single vector for the representation. This allows for more fine-grained reasoning and longer in-context prompting.
Retrieval Augmented VLMs In the text domain, learning to retrieve relevant documents to enhance the LLM query (Guu et al., 2020) has been explored extensively (Wang et al., 2023). Recent VLM works follow a similar approach to retrieve multimodal documents to improve performance on knowledge-intensive tasks and improve generalization to rare situations. Gao et al. (2022) summarizes visual content into natural language to use as
a query for dense passage retrieval. MuRAG (Chen et al., 2022) proposes a multimodal image-text memory bank to help models answer challenging knowledge-based visual questions such as “What shape is the pediment on top of the white house?" REVEAL (Hu et al., 2023) and RA-VQA (Lin and Byrne, 2022) learns a trainable multimodal retriever similar to REALM (Guu et al., 2020) during pretraining to fetch relevant documents to answer questions, achieving state of the art performance on datasets such as VQAv2 (Antol et al., 2015) and OKVQA (Schwenk et al., 2022). To the best of our knowledge, we are the first to integrate region-level retrieval with LLMs, in which the multimodal documents are indexed by object-level visual features.

heading: 7 Conclusion
text: We present a simple approach to insert object level visual embeddings into large language model decoders, enabling object level reasoning with flexible prompt structure. Our object encoder compresses fine-grained region level information into a single vector, enabling in-context prompting with objects from multiple images and more efficient training and inference. In addition, we introduce the idea of region retrieval, which allows for precise queries free of image background noise and rapid generalization to rare and unseen objects with no parameter updates. We hope our method may help researchers design vision language models which can adapt to their needs by simply updating the retrieval set or object encoder, while also being responsive to varying user intents using LLM prompting techniques.

heading: 8 Limitations
text: While our approach provides a flexible way for users to supply object-level prompts, it does not output bounding boxes or other region-level grounding. This may be addressed in future research by further finetuning on region-level instruction tuning data as done in FERRET (You et al., 2023), GLAMM (Rasheed et al., 2023), and other region-level VLM pretraining. At the moment, we also do not explore generic image tasks such as VQA or image captioning. However, a potential solution is to use our object encoder to connect to existing VLMs (e.g. LLaVA) which excel at these tasks. Lastly, our results in the retrieval setting depend on the quality of the retrieved examples. Curating a high-quality retrieval set at the object-level can be challenging. However, existing tools such as GLIPv2 (Zhang et al., 2022) allows for semi-automatic generation of region-level data as used in KOSMOS-2 (Peng et al., 2024) in developing the GRIT dataset.

heading: 9 Ethical Considerations
text: Biases From Pretrained LLMs Since our model uses existing pretrained LLMs such as Llama 2 or GPT2, it may inherent some of the social biases or toxicity acquired during their pretraining stages. While Llama 2 undergoes extensive alignment to ideal human values through reinforcement learning from human feedback (RLHF) (Griffith et al., 2013), some of these toxic behaviors may still be present in the morally aligned model. We make sure to only use images of common objects in the COCO dataset, which do not contain any of these biases or violent scenes to the best of our knowledge. Nevertheless, further testing to ensure the
impartiality of the model may be necessary before deploying in widespread technologies.
Domain Adaptation Some of our experiments involve evaluating our model in a data-scarce domain in a zero shot manner with in-context prompting. While this is a promising direction for efficient domain adaptation, users should take caution in directly using model prediction, as this is a challenging task due to distribution shift. We encourage human-in-the-loop interaction to sanity check the outputs. Different from other ICL prompting methods, we provide retrieved examples and similarity scores which can help determine the trustworthiness of the model prediction, which may be valuable for high-risk domains such as medicine.

heading: Acknowledgement
text: Ossowski and Hu are supported by the Wisconsin Alumni Research Foundation and the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number R01EB033782. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

heading: A Appendix
text: Table 3 contains all the prompts we use to instruct the LLM decoder.

heading: B Qualitative Examples
text: Here we include several selected examples showcasing the strengths and weaknesses of our approach.
Visual Concept Generalization In Figure 9 we demonstrate more examples of rapid generalization to new visual concepts. Many existing methods confidently predict concepts from their pretraining, while ours can predict new concepts on the fly.
Scene Content Awareness Even though our object representation involves masking out image patch features from other parts of the image, we have observed that the object vector still contains information about its surroundings. Figure 10 illustrates this phenomenon, where OLIVE can include the cow in its description, despite not including any image patches corresponding to the cow in the user selection.
Patch level Detail Our method also can identify and describe small objects at the patch level. Figure 11 shows an example of object classification on smaller objects.
Describing Partially Visible Objects We notice that our model can make mistakes when describing occluded or partially visible objects as seen in Figure 12. We hypothesize that the training data of refCOCOg does not include these kinds of image regions, which also limits its availability in retrieval data. This may be addressed with larger-scale pretraining on data such as GRIT which likely includes more occluded objects.
Errors in Detailed Description While our model can identify the object most of the time, it sometimes gets minor details incorrect. For example, the colors of a shirt or other piece of clothing are seen in Figure 12. This may be due to the extreme compression we learn into a single vector. Future work may consider visually resampling the object features into more than just 1 latent vector for detailed captioning, but still use the single vector representation for retrieval.

heading: C Dataset Information
text: Table 4 provides more details on the dataset splits used in our training and evaluation. Our COCO train and validation splits are slightly smaller than normal because of our approach of using segmentation masks. We decide to omit some excessively small segmentations which account for less than 1% of the data. For tasks that require training (COCO and refCOCOg), we use the train split of the COCO object detection dataset as our retrieval data. We make sure to omit the closest match when training object detection on COCO with retrieval to avoid label leakage. We also confirm that no images are repeated in the validation split from the training split for both datasets.

heading: D Referring Object Classification
text: This task requires the LLM to predict the object class label given a ground truth input annotation (e.g. bounding box, segmentation, etc). We follow a similar evaluation protocol used in (Zhong et al., 2022) and (Zareian et al., 2021), in which the ground truth annotation is supplied to avoid localization error. Table 5 shows the overall referring object classification accuracy and mAP 3 for our methods. We observe several findings. First, although retrieved examples help with domain adaption and rare objects, it does not improve the overall in-domain performance. Second, both the LLama 2 and GPT2 baseline have similar performances on the task, suggesting that even smaller models can learn vision-language grounding. Lastly, even our retrieval-only baseline, which requires no training, has better accuracy than some parameterized methods such as CLIP.

heading: E Multi-Task Model
text: We also explore the possibility of training a multitask model using a similar curriculum learning strategy to LLaVA (Liu et al., 2023). We first train the model on the referring object classification task to perform the object-word level alignment. The model is then trained on the referring expression generation task, and finally on an object instruction following dataset (Cai et al., 2023) with many different tasks. For each stage of training, we formulate the task in an instruction-following manner through the prompts in Table 3. This allows the
3To simplify the calculation, we assigned a confidence score of 1 to each prediction. Reported mAP may be lower than the true value when using more accurate probabilities.
model to be responsive to many different user intents (Figure 13)

heading: F Referring Expression Generation
text: We study our model’s overall performance on referring expression generation by quantitatively evaluating our model on the RefCOCOg validation
set shown in Table 6. Several findings can be observed. First, including retrieved multimodal documents results in slightly better performance. Second, the size of the LLM can be modified without much performance change, with GPT2 performing slightly better than Llama 2. Third, having global image context contained in the object representation is important, as methods that crop the image region (e.g. Flamingo) perform worse.

heading: G Training Hyperparameters
text: We provide the detailed training hyperparameters in Table 7.

heading: Future Work
text: Future work may consider visually resampling the object features into more than just 1 latent vector for detailed captioning, but still use the single vector representation for retrieval