FlexCap: Describe Anything in Images in Controllable Detail
Authors: Debidatta Dwibedi
Vidhi Jain
Jonathan Tompson
Andrew Zisserman
Yusuf Aytar
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language models
dense captioning
open-ended object detection
Primary area: machine_vision
TL;DR: New image captioning model that can generate short or detailed descriptions of specific image areas.

Abstract: We introduce FlexCap, a vision-language model that generates region-specific descriptions of varying lengths. FlexCap is trained to produce length-conditioned captions for input boxes, enabling control over information density, with descriptions ranging from concise object labels to detailed captions.  To achieve this, we create large-scale training datasets of image region descriptions with varying lengths from captioned web images. We demonstrate FlexCap’s effectiveness in several applications: first, it achieves strong performance in dense captioning tasks on the Visual Genome dataset. Second, we show how FlexCap’s localized descriptions can serve as input to a large language model to create a visual question answering (VQA) system, achieving state-of-the-art zero-shot performance on multiple VQA benchmarks. Our experiments illustrate FlexCap’s utility for tasks including image labeling, object attribute recognition, and visual dialog. Project webpage: https://flex-cap.github.io.

FlexCap: Describe Anything in Images in Controllable Detail
Debidatta Dwibedi
Vidhi Jain" Carnegie Mellon University
Google Deepmind debidatta@google.
vidhi j@addrew..de.rr. cmu. edu
Jonathan Tompson
Andrew Zisserman Google Deepmind
Yusuf Aytar
Google Deepmind
Google Deepmind
tompson@google.co com
zissermanOgoogle. com
yusufaytardgooggee.com com
Abstract
We introduce FlexCap, vision-language model that generates region-specific descriptions of varying lengths. FlexCap is trained to produce captions for input boxes, enabling control over information density, with descrip- tions ranging from concise object labels to detailed captions. To achieve this, create large-scale training datasets of image region descriptions with varying lengths from captioned web images. We demonstrate FlexCap's s effectiveness in several applications: first, it achieves strong performance in dense captioning tasks on the Visual Genome dataset. Second, we show how FlexCap's localized descriptions
input a large language model create visual question answering (VQA) system, achieving state-of-thh-aar zero-shot performance on multiple VQA benchmarks. Our experiments illustrate FlexCap's utility for tasks including image labeling, object attribute recognition, and visual dialog. Project webpage: https://flex- cappfplxxxcppppeee httpf/lxxxcap.ithhub... github.
Introduction
How does describe the world around not just in broad strokes but with the ability to zoom in and out, capturing both the grand scene and the minute details? Imagine pointing at a bustling market scene and asking, "What's happening here?" and receiving vivid description, not just of the market as a whole, but also a detailed account of the interactions between vendors and customers, the vibrant specific item held by passerby. This ability to controllably
colors of the goods on display. focus and describe visual content is what we call flexible captioning.
Traditional image captioning models, while adept at capturing the gist of an image, often struggle to pinpoint specific objects or attributes. On the other hand, object detection systems excel at localizing elements but may lack the vocabulary to describe them comprehensively. Dense captioning [26) attempts to bridge this gap by generating captions for multiple regions, but its expressiveness is limited by existing datasets. In this work, we introduce model called FlexCap that bridges the gap between holistic image understanding and localized inquiry. This enables the generation of captions that are both spatially precise and semantically rich (see Fig. [ (left) by specifying region of interest and the desired level detail in terms of number of words in the predicted caption. This allows us to integrate the strengths of image captioning, object detection, and dense captioning into model.
To be able to train such a model, we require dataset of images where many boxes are labeled with short and long descriptions. We propose a method to generate triplets of (i) image, (ii) proposed region within the image, and (iii) a caption of particular length, by using an apen-vocabulary object detector to label regions from captions of an image-text pair dataset. We demonstrate this at two
Work done as a student researcher at Google Deepmind.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Object Detection
cat
Bounding propraal
FlexCap
inin thi in irage?
LLM contect prompt You cne iesigert eniuu
Controllably rich captions
bioycle bicydde
pl nurnway
Weth the airplano polnted lihy"
Controlled Captions per box
einn
dark
The iiplone is poiened tc the right side the image il located on the right side of image und
eng or plane
whth cat difting on bioyyle
the of the paa al wind)
placs seggecee
vhiile bloycle darkk
blackk
the lift slse
wing
Dense Captioning
white block lookiing
Whht (iiht
wigin on the 2000 es
ric
The fighht &s operoted
Figure/ 1: FlexCap generates controllably rich localized descriptions for any region in image as shown on the left. It has the flexibility to produce captions in a controllable manner which allows the full spectrum of valid descriptions be explored from short object category names to fully-detailed captions. On the right, we demonstrate that rich localized captions generated by FlexCap, when coupled with large language models (LLMs), enable zero-shot visual question answering.
scales: 200 million triplets using YFCCI00M 47 captioned images; and 32 billion triplets using the WebLI [ captioned dataset. Training FlexCap on these datasets enables the model to generate spatially and semantically rich representations (bounding boxes and their descriptions) that focuses on objects, their attributes, and their contextually changing descriptions.
We evaluate the effectiveness of the descriptions generated with FlexCap by representing images in detail by describing its constituent objects and regions. A popular and effective strategy is to directly provide visual features as input tokens to L.MSs 22 14 29] 30 (33 59]. Instead we directly provide
textual descriptions of images as input to the LLM evaluate FlexCap similar to [21 42 57]. We show that this human interpretable representation, when combined with the power of LLMs, enables visual question answering and dialog. An example is shown in Figure [right). We also demonstrate that this combination can result in performance that is competitive with state-of-the-art VLM models on zero-shot image and video question answering benchmarks.
Our key technical contributions are: (i) controllable localized visual descriptions, using word count as proxy for complexity to modulate the output of a generative language model (ii) a large-scale dataset generated for image-text-box triplets to enable training of our model; (iii) the human-interpretable representations produced by FlexCap with the help of LLMs is comparable exceeds performance of state-of-the-ard VQA methods; and (iv) demonstrating that our dense captioning methods outperform baselines under comparable scenarios.
2 FlexCap
Length conditioning. For the same region in an image, there may be multiple valid captions. In the input image shown in Figure 2 all the following descriptions are correct: cat, grey and white cat, grey and white cat lying on shoes. Clearly, the task of describing a bounding box in the image does not have only one right answer. We equip the model with the capability of producing outputs of desired length by utilizing the idea of length conditioning. We condition the input by simply using an additional token indicating the desired length of the output caption. Training with length-conditionig is useful for three key reasons. First, the number of words used to describe is often proportional to the information content. We train the model to predict the next word in the sequence while accounting for the desired length, thereby the model learns to modulate the amount of information in the generated text. Second, length conditioning allows users to control the output of the model, further enabling the use of single model for many diverse tasks. Third, the length prefix provides a better conditioned initial state for the captioner. Figure ssows than one ground truth caption 
 a dog <e> or 
 dog playing with a frisbee <e>, If we use the first caption as ground truth and the words 
 a dog as the seen text, the nort-word prediction loss encourages the model to increase its score for the <e> token and decrease the score for the word playing due to the softmax loss. Whether this occurrence will on the dataset statistics. To quantify the prevalence of this problem, we compute a statistic: for each image box, we consider all pairs of captions and measure the fraction of pairs sharing prefix words. For instance, consider a box that has three captions: 
 dog <e>, 
 dog playing <e>, and
shows how the same box might have more
problem depends

---

grey and white cat lying on shoes <eos>
Text Decoder
LENGTH-7 grey and white cat lying on shoes
Image Encoder
Linear Projection
Prefix Token
Caption
cat
LENGTH-1 cat
[xminn ymin, xmax, ymax]
grey and cat
Append
LENGTH-4 grey and cat
grey and white cat lying
prefix"
LENGTH-7 grey and white cat lying on shoes
Input Image
Bounding Box
Conditioned Captions
Architecture and Training Setup. FlexCap outputs a object contained in the bounding box by taking (left) an image, (middle) coordinates of bounding box and (right) the length prefix and caption, as inputs. The training loss the standard next-word prediction loss that is used to train image captioning models.
Figure 2:
caption of the

 dog playing with frisbee <e> which share a prefix 
 dog. While another caption for the same box, 
 brown dog <e>, does not share prefix with captions beginning with 
 dog <e>, After averaging this metric across all images in our Localized Captions Dataset from WebLI (introduced later in Section 3). found that 30.8% of all caption pairs share a prefix. Using a length conditioning token instead of 
, the probability of prefix matching decreases from 30.8% 11.1%. The length conditioning helps the model in distinguishing between captions with the same prefix while also providing the model with a novel capability during inference.
Architecture. Our objective is to train model that takes input image and region of interest and outputs a description of a desired length of the region spanned by the box. We present FlexCap's architecture in Figure 2 The model takes an image, the coordinates of bounding box and the conditioning tokens as input, and outputs a textual description visual contents within the specified bounding box. Our model mainly consists of an image encoder (i.e. SOViT-400M/1 14 (I)) and transformer-basee based text-decoder. We pass the image through the vision model to produce outputs of dimensions bounding box coordinates (of dimensions 1 x 4) through a linear layer to produce the coordinate d). Both vision features and normalized bounding box features are
(where n is the number of patches and d is the embedding size). We pass the
features (of dimension concatenated to form input of dimension (n + 1) x d to text decoder. The text decoder consists of stack of Transformer layers. We use a decoder-only architecture in which all the vision and bounding box tokens remain unmasked but the text tokens are masked in a causal manner to enable next-word prediction training. We add the vision tokens and bounding box coordinate tokens so that the text decoder can access all of the visual context in the image and the exact bounding box location. In this work, we train text decoder composed of 12 self-attention transformer layers with a dimensionality of 768 and 12 attention heads.
In total, FlexCap has 590M parameters with 428M comprised of the image encoder (SOVIT) and the remaining parameters in the text decoder. A linear layer transforms the 1522-dimeniional output from the vision encoder into a f68-dimensional input for the text decoder. We initialize the vision encoder with SigLIP [60] weights, which is a contrastively trained image encoder using web-scale vision-text pairs from WebLI. We do not freeze the vision encoder during training.
Loss. We train FlexCap to predict the next token of the text. The text tokens are prefixed with the desired length of the caption and appended with end of sentence token <e> indicate the end of the caption. The target text tokens are obtained by shifting the padded text by 1. This is common training methodology for training generative language models like GPT [ Vim [51]. The loss is a classification loss over all the words present in the vocabulary. The loss ignored over the padded tokens that are used keep the size of the outputs same for all the captions in the batch. Formally, we represent a data sample as triplet T' = (X, B, W) consisting of image X, bounding box B and captions W. where W {LENGTH-K, W1, W2, ...wy). To enable batch training, we pad the tokenized captions to fixed size M. For a given data triplet, our objective is to maximize the following log- likeliooooodd. ((X, B, W)
Σ i= pow. wer X, B). Assume that we have a dataset
3

---

D= -= T......... Ty}. The overall loss function is:
N LD)= Σ B,, j=1
NV.
M
Σ
log (wg BB)
Implementation. We implement this model using the JAX framework 77. We train the entire model for about 400K steps using the AdamW optimizer with cosine learning rate schedule. The maximum learning rate is 1.6 x 10 with 10K warm-up steps. We use a weight decay of 0.05. We train with a batch size of 4096 and image resolution of 224 x 224. We use maximum text sequence length of 32 tokens. For each image in the batch, we sample a maximum of 8 bounding boxes.
Inference. At inference time, provide image, the target bounding box, and the desired length as input. We then decode in an auto-regressive manner till the end caption token <e> is encountered or the maximum number of decoding steps is reached. While we use standard sampling techniques used in setteeneeation like beam search, temperature sampling, or nucleus sampling [19] to generate multiple captions. We use greedy decoding in all experiments unless otherwise stated.
3 Localized Captions Dataset
In order to train the FlexCap model, build large scale dataset of image region descriptions of varying lengths. In the following section we deseribe how we produce such a dataset from existing image-text paired datasets. We leverage the web-based image-caption pairs datasets (like WebLI [ and YFCC100M [47] to create a localized captions dataset. The dataset generation pipeline is shown in Figure First, we create text queries using n-grams from the caption of the image: e.g. "dog" "brown dog". "brown dog playing with a disc". We specifically create n-grams where = (1, .8 and then filter out incomplete captions like "with red", "dog playing with". More details about the filtering step are mentioned in the appendix. Then we use the filtered n-grams as text queries for pre-trained region proposal models (i.e. OWL-ViT (36) extract boxes and select text-box pairs based on the similarity score (> 0.1). Multiple n-grams may match for box, and this results in several ways of describing box in the image as shown in Col. in Figure
WebLL. This data collection technique on the WebLI dataset results in 32 billion triplets from 2 billion images without requiring new manual annotation. Our captions show rich vocabulary that is close to common language used to describe objects in the context of an image. If we MS-COCO's vocabulary then all humans in the dataset would get labeled as person. However by building our vocabulary in a bottom-up manner we end with captions that contain more informative words such baby, nurse, policeman, firefighter, or baseball player to describe the person class. Please refer to the appendix for details of dataset statistics and examples.
YFCC100M. We also create localized captions dataset using YFCC100M images. Specifically use the same 14M images as the CLIP paper. The dataset creation method results in ~ 11M images with at least one valid box. On average each image has 20 boxes, making the size of this dataset ~ 0.2B maage-box-caption triplets. The number of YFCC100M triplets is 160 times smaller than the localized captions dataset created from WebL.I
As both OWI-ViT and the CLIP subset of YFCC100M are publicly available, the resulting localized captions dataset can be generated with open- models and public datasets. Since the WebLI dataset is not publicly available yet, YFCC100M triplet-dataset can serve as a reproducible benchmark. Concurrently large-scale grounded image-text dataset generation pipelines have also been proposed in [38 and [50].
Experiments
4.1 Correctness and Compliance of Generated Captions
Correctness. In this experiment, we solely evaluate the recognition capabilities of our model in zero-shot manner. We use the region classification task [53 on the MS-COCO dataset to assess how well our model recognizes objects at different scales and under occlusion. In this task, the image and the ground truth bounding box are provided as input to the model such that produces short description of what is contained in the bounding box. Our region classification pipeline (Figure 4

---

Web Scale Image-Ter Text Pairs
2 Extract n-grams
Associate n-grams with box proposals
32B Triplets Image- Box- Tox-Caption
brownn dog, playing, red...
OWL-VIT
A brown dog playing with a red disc grassy field.
brown dog, dog playing, red disc. grassy field...
Embeddings each n-grom)
Text Transformer Encoder
alisc
red disc
brown playing, red disc. grassy field...
n- n-gram & box simierity scores
field
grassy field
a brown dog playing with red cisc on a grassy field
Vision Transformer Encoder
Boxes
Thrsshckft
dog
brown dog
Figure 3: Dataset Generation. We use OWL-ViT to generate a dataset of triplets of image, bounding box and captions from web-scale dataset of noisy image-text pairs. Increasing levels of richness in captions is captured through different length descriptions for each box. Image Bounding Box
Ground Truth
Only used for calculating evaluation metrics
airplane cat
dog person
Predicted Captions
Text Encoder
Orange BMW
Mean
Text Encoder
Over Predicted
Dot product to get class similarities
BMW
Caption
2017 BMW
Embeddings
FlexCap
Pick highest simiarity
Predicted
Figure 4: Evaluating open-vocabulary outputs from FlexCap using the CLIP
text encoder.
passes the input image (of size 448 x 448) through FlexCap which generates 20 captions each for 4 caption lengths (1, 2, 3, 4) via nucleus sampling. These captions are then mapped to object class names using CL.IP's (39 text encoder. By comparing the mean of the text embeddings of predicted captions with those of ground truth class names, we obtain the classification scores. We report the results in Table Ta We find FlexCap outperforms contrastively trained approaches used for region classification. Furthermore, we observe a significant boost 13% mAP) in performance by producing multiple captions for each box. One reason for the boost in performance may be directly comparing text embeddings to produce classification scores as compared to baselines which use dot product of image and text embeddings.
Compliance. In Figure 5 we show qualitative examples of the FlexCap model producing different length captions for the same box. Note how the model progressively adds more information about the object by incorporating context in the longer sentences (in the jungle), attributes (pink flamingo kite), and alternative nouns (chevy, feline. We also measure how well our model complies to the desired caption length. To do so, we 1000 images from the MS-COCO dataset and use random object in the image produce descriptions with different lengths. We report the average length of the predicted caption, and the fraction times the predicted caption has a length equal to the target length in Table Tb We find FlexCap's outputs are mostly compliant with the target length.
4.2 Visual Question Answering
Visual question answering (VQA) often requires visually grounded rich semantic understanding of the content multiple levels of granularity depending on the question. These properties make VQA a great test-bed for our method which can generate dense spatially grounded information on visual content with desired semantic complexity.
Flex C.. In Figure I (right), we show how we use FlexCap with an LLM to solve visual questions. First, we convert an image to sequence of localized descriptions that describe the image terms the objects and regions present the image. To so, we need region proposals. We use OWL-ViTV2 [35] to localize important objects and regions in an image. We keep the top 128
5

---

Input: Image Boundiig Output: length ceptions
Imput: lmnege Bounding
Output: Lengthh ceptioon
Bephont
Kite
olephant
Flarnnngo kite
Elephann jungle
Pink flamiino kitw
Elephent thejuuge jungle
Colortul fammingo beach kite
Elephonn wallding
Bird blue
Ar elephent walking along a path
Flamingo kite the marina bey
with elephant
Pink flamnnge lite flying the wind
The clesp ant was the middle newehr
pink flamirn ningg kita flyling beach
Chevy
Typing
Red truck
Computeer
Working from home feilme
Car the Highway
sleeping computer
The back of truck
stretchinn
The back red truck
Con tying on top computer
The back of red pickup truck
sleeping the keyboard
the side the
Figure 5: Examples of length controlled captions generated by FlexCap. Note that attributes ("pink flamingo kite") and context ("in the jungle") are generated as the length increases.
Method
mAP (t)
Target Length
Mcan Len Pred.
Accu- racy
Target Length
Mean Len Pred.
Accu-
racy
CLIP
53
29.2
CLIM
62.2
1.02
0.99
5,06
0.94
RegionCLIP 61
62.8
2
2.05
0.96
6
6.01
0.97
FlexCap (Top-1 caption)
72,0
3
3.06
0.95
7
7.02
0,95
FlexCap (Top-20 captions)
85.0
4
4.04
0.96
8
8.02
0.95
(a) MS-COCO Region Classification with
(h) Compliance metries. FlexCap produces length-compliant captions for different lengths.
ground truth bounding box.
Table 1: FlexCap's outputs are accurate and length compliant.
bounding boxes by their objeciness scores. then use FlexCap to describe each in the image in the context of the entire image. In order to produce holistic descriptions, we use multiple prefixes for each region. These prefixes are combination of length conditioning token and some initial text. context to the LLM (see Figure [
We add the boxes and their descriptions to text preamble that defines the setup where we are using an LLM to answer questions about an image. In all the experiments,
PALM2-S model 4 as the LLM of choice. We refer to this end-to-end system that takes an image and a question to output the answer as FlexCap-LLM.
To adapt the base FlexCap to have improved detection skills, output longer sentences, and identify OCR, we co-train FlexCap for 25k more steps on detection (COCO, VOC, Openlmages, LVIS), captioning (COCO Captions, Visual Genome) and OCR datasets (WebLI). For image captioning datasets, we use the bounding box that covers the whole image. We find this co-training step useful for downstream tasks using the LLM.
Hence we evaluate the effectiveness of FlexCap-LLM on several image VQA benchmarks such as OKVQA (34, VQAv2 [16), GQA 23 and VizWiz [8], and vidco question answering benchmarks such as MSRVTT 551 and MSVD 54 Diverse characterictics of these datasets helps gaining better
insight on FlexCap's capabilities. We report the commonly used accuracy metric for each dataset.
VQAv2, GQA, OKVQA and VizWiz image VQA benchmarks in a zero-shot setting, meaning that our approach not trained with the task the corresponding dataset. The results these benchmarks are presented in Table 2
Image Question Answering. First evaluate FlexCap-LLM
Standard VQA. The VQAv2 dataset is a standard for evaluating the performance of visual question- answering systems. In Table 2a present the results of our evaluation of FlexCap- CLp on this dataset. We find that FlexCap-LLMM outperforms other zero-shot baselines, such as BLIP-2. This performance is achieved by providing object and region level information to LLMs without requiring multi-modal fine-tuning.
Compositional VQA. The GQA dataset is for evaluating the performance on complex compositional questions. As FlexCap produces information for multiple visual elements in scene with their corresponding locations, it is quite well-suited for questions on compositional understanding of the image. On this benchmark, as shown Table 25 FlexCap-L.LM outperforms all the recent baselines except InstructBLIP [.].

---

Method
Acc.(%)
Method
Acc. (%) t
PalmE-562B
80.0
LGCN
70
55.8
VLMO 0
82.8
LXMERT
45
60.0
BEIT-3
84.2
NSM 22
63.0
PaLI-17B
9
84.3
CFR
37
72.1
Vew 25
47.7
BLIPv2
29
44.7
Flamingo
56.3
ViperGPI
48.1
BIIIP2 29
65.2
FlexCap-ILLM
48.8
FleXCap-LIMM
65.6
InstructBLIP
49.5
(a) VQAv2 results lest-dev set
(b) GQA results test-dev balanced set
Method
Acc.(%)
Method
Acc.(%)
PalmE-12B
14
55.5
Flamingo 32 Shot 14
49.8
PalmE-562B
66.1
Flamingo FT
65.7
PaLI-3B 9
52.4
PaLI-3B
67.5
PaLI-17B
64.5
PaLI-17B
74.4
BLIPv2 22
45.9
Flamingo 3
31.6
Flamingo
50.6
Emu
34.2
ViperGPT 44
51.9
InstructBLIP
34.5
FlexCap-LLMM
52.1
FlexCap-LLM
41.8
(c) OKVQA results on val set
(d) VizWiz results on test-dev set
Table 2: Zero-shot image question answering results. FlexCap-LLM is compared against recent
baselines. Grayed methods are trained on question answering datasets.
Method
mAPt
(%
FCLN 26
5.39
Method
MSRVTT
MSVD
Method
mAP
JIVC [56
9.31
Emu
8.3
18.8
FCLN (66
27.11
COCG 31
9.82
Flamingo 3
17.4
35.6
CAG-Net 58
36.29
CAG-Net 58
10.51
FlexCap-LLM
25.0
39.5
FlexCap
46.9
TDC+ROCSU
11.49
GRIT [52]
15.52
Table 3: Zero-shot video ques- tion answering results reported on MSRVTT-QA and MSVD-QA the test set. FlexCap-LLM is better than other zero-shot baselines for video VQA benchmarks.
(a) Captioning GT Boxes
FlexCap GRiT Boxes
16.2
(b) Dense Captioning
Table 4: Captioning boxes in Visual Genome dataset. Flex- Cap exceeds performance of other methods. All methods have been fine-tuned on Visual Genome captions.
VQA with External Knowledge. OKVQA dataset is particularly designed for evaluating the ability to answer questions about images that require external knowledge which is not readily available on the image. Hence it requires multiple levels of understanding of the content, and reasoning with that information, which is well-suited for applying FlexCap. In Table 2e|we we show our performance strong baselines such Flamingo and ViperGPT which highlights the
OKVQA is superior effectiveness of the mix of generic and specific descriptions generated by FlexCap. Unlike other baselines which use the question, FlexCap generates captions without having access the question.
VQA with atypical images. We also evaluate on VizWiz, which contains visual questions asked by people who are visually impaired. Unlike web content, in these images the objects and the scene not always well-centered, hence this dataset contains many out-of-distribution samples compared to typical web-crawled datasets. We report the results of this experiment in Table 2d Nevertheless, approach significantly outperforms Flamingo 2 and InstructBLIP
the zero-shot setting.
Video Question Answering. We also evaluate FlexCap-LLM on zero-shot video question answering
datasets MSRVTT-QA and MSVD-QA 54 The results on these benchmarks are presented in
Table 3 For processing the video, sample 8 frames uniformly from the video. We pass each of these frames through FlexCap produce captions of objects and regions. We then combine all the object captions from the different frames into prompt for the LLM. We observe FlexCap-LLM
7

---

significantly exceeds the performance of the Flamingo model in the zero-shot setting. These results highlight the zero-shot effectiveness of our method, which can solve tasks in the video domain even though both FlexCap and the LLM were not trained for those tasks.
4.3 Dense Captioning
Dataset and Evaluation Metries. The dense captioning task is defined producing both the regions and the corresponding descriptions for each region. For this experiment, we use the Visual Genome (28] dataset. In this dataset, each image is annotated with multiple bounding boxes and each box has corresponding caption. We use the train-test splits and evaluation metric as proposed in (26). The paper proposes to use a mean of Average Precisions (mAP) over pairwise thresholds of both IOU thresholds (0.3, 0.4, 0.5, 0.6, 0.7) and Meteor score thresholds (0.0, 0.05, 0.1, 0.15, 0.2, 0.25). We use the same preprocessing of text and boxes as mentioned in [52].
Fine-tuning FlexCap. We fine-tune the pretrained FlexCap model on the Visual Genome train split for 60k steps with lower learning rate of 1e 6 at resolution of 448 x 448.
Captioning GT boxes. Following the evaluation procedure from (6], we evaluate captioning of the ground-truth boxes in Visual Genome. Since this setting removes the localization task, we have
cleaner evaluation of only the region captioning problem. The results of this experiment are provided in Table in which we show that FlexCap achieves better performance compared to other approaches evaluated in this setting.
Captioning GRIT boxes. In this experiment, we want to compare against other approaches that perform both localization and captioning. We are measuring how well FlexCap performs when provided with box proposals. Table Abjoows FlexCap obtains better performance compared to other approaches evaluated in this setting. Since our work we do not propose any localization module, we use GRIT's 52 region proposals as the input boxes for our model. This also allows us to directly compare our captioning capabilities against GRIT. We find that our approach outperforms GRIT at this task even though we test at a lower resolution of 448 x 448.
4.4 Open-Ended Object Detection
Open-ended object detection, that is identifying all the objects in the image with rich descriptions, can be achieved in two ways: (a) localize-then-describ or (b) describe-then--lcaliiee With Flex- Cap, we employ locadizzethenn--escriibe approach where boxes are generated with a box proposal mechanism (i.e. OWL-ViT) and then described by FlexCap. We compare these results with approach where we first ask a otate-of-the-art (SOTA) VLM (i.e. LLAVA (33) for a comprehensive description of the image with all objects and then localize these descriptions using a SOTA open-vocabulary object detection method (i.e. OWL-VVT (66). The Recall of all the objects in the image is major criteria in open-ended object detection. As shown in Figure (ja,, we demonstrate that localize- then thee-dee approach powered with FlexCap performs significantly better compared to describe- -lcaliiee approach using existing works. In Figure(o((be)) (b-e), we observe that our localize-then-desccibbe approach retrieves more parts of the image, particularly the small approach. This is mainly because
siscribe-then-locaae
and medium size objects. compared to the describe-then-localee most VLMs are trained to describe the salient objects in the image rather than exhaustively list all the objects present in the scene.
4.5 Discussion
Prefixing. Training on a large dataset also enables the extraction of different kinds of information from an image using prefixes. We can use this property to our advantage to extract object properties such as color and material. We show examples of this in Figure 7 Say we want to extract the color and material of objects, we design two prefixes: 1) "LENGTH-4 The color is ---- and 2) "LENGTH-5 This is made of --._". The pre-fixed length tokens guide the model to produce outputs that are just one more word. Note how the model is aware of the input prefix and completes based on the object in the bounding box without confusing it with other surrounding objects. We show more examples of using the prefix to define in Fig. 8 FlexCap can extract the following attributes with the corresponding prefix in the brackets: the actions of humans ("The person is is"), the uses of objects (" ("This is used for"), OCR ( ("The sign says"), "), names of books ( ("The book is called"), authors of books
"
8

---

A00 larg: 34edk
Dearth Bostandee
USA
USe Decrie Locket
DV5
Threshold
"OWLMT
OWLVVI
Store
(a) Recall wss IOU
(b) Number of Objects Found
(c) Number of Small
(d) Number of Med.
(e) Number of Large Objects Found
Objects Found
Objects Found
Figure 6: Open-Ended Object Detection on the Visual Genome dataset. We find that the localize- then-describe approach, which involves describing every detected box with FlexCap, achieves higher recall (see (a)) and produces more bounding boxes with good matching scores (see (b)) compared to describe-then-lond thenn-localzee approach with SOTA VLMs and open- vonabulayy detection. The difference between the two approaches is most stark for small and medium sized objects (see (c-e)). Conditioning Prefix: LENGTH-4 The color
Conditioning Prefix: LENGTH-5 This is made
purpie
blue
NINA
blue
leather
Figure 7: Extracting properties by conditioning FlexCap with prefixes. Examples of FlexCap extracting properties of objects of different categories by using relevant prefixes. Note how we able to retrieve a one-word answer from the model by controlling the length of the caption.
The person 15
is used for
The sign says
The book called
Whiten by
The photo was taken
AOSTEN SSCINDA CHENE BEYOND
SO YOUNDA CROR BEYOND
voodoo doughnut
INFINITY
INFINITY
carrying a surfboard
siting
beyond Infinitt
eugeenia cheng
from the Br
Figure 8: Generating conditional captions using prefixes. Conditional captions allow us to extract desired information from the input bounding box.
("Written by"), and the locations where the photos were taken ("The photo was taken in Note the same model can extract different information from the same bounding box: the name of the book title and the author name based on the prefixing (col 4-5).
Limitations. We use OWL-ViT and alt-text to generate the training dataset for the model rather than relying on human annotations. While this approach allows us to scale, there exist biases in the model and alt-text which will reflect in the outputs of FlexCap. Another limitation that the proposed FlexCapLLM system is not end-to-end trainable but is composition of a VLM and LLM. This can be alleviated by training a VLM with the localized captioning dataset.
Broader Impact. Localized and controllable image captioning enabled by our model may benefit applications like accessibility tools for visual impairments and intuitive human-computer interaction. However, biases in training data raise concerns around potential misrepresentation or exclusion of certain demographics. Dual-use risks also exist if this model is employed for unethical surveillance.
Related Work
Visual question answering (VQA), task designed to assess if a computer can answer questions about image, often requires grounding visual concepts and reasoning. Although initially introduced for supervised evaluation of the task [5], most recently, VQA also become one of the most powerful benchmarks for evaluating task and dataset independent visual dialog. Several existing models such

---

as ViperGPT [44, Flamingo (2) BLIP 29 30), PaLI [T0] show convincing zero- shoo-shot performance on the VQA benchmarks that rivals the supervised approaches. Unlike most previous zero-shot approaches, which tightly couple vision and language components in a single model, FlexCap generates a high-level human interpretable representation of an image and demonstrates that, through straight-forwar application of L.MMs, we can achieve comparable performance with state-of-the-art results across VQA benchmarks. Unlike others, ViperGPT [44], also decouples vision and language components and reinterprets visual questions with LLM generated programs, and executes them using existing visual perception tools. Whereas, in our case we use only one powerful vision tool, i.e. FlexCap, to generate all the necessary information and leave the reasoning to LLM. In that sense, FlexCap is quite complementary to ViperGPT as it can be one of the powerful tools that can improve the controllable visual understanding of the image for ViperGPT.
Open vocabulary object detection models like OWL-ViT 36 and ViL.D 17 enable the user to query any given text on the image and obtain matched bounding boxes for those queries. In these models the text is often encoded by a text encoder like CLIIP
and T5 44) The text embeddings are compared with the category-agnostic box proposals coming from the visual backbone. In this work, we use OWL-ViT's text and vision encoders to associate bounding boxes with text-queries to produce our training data. By training a localized captioning model, we remove the manual step of providing per-dataset or per-image text queries to use OWL- VIT. RegionCLIP [6]] obtained good performance on open- venn-ooabylaryy object detection by utilizing region-level vision- laniuue contrastive learning on large scale data. We differ from this work as we generate the description for each bounding box instead of associating text queries (defined manually) with bounding boxcs.
3
Dense captioning involves localizing salient regions of the image and describing them with natural language sentences, introduced in 22) In practice, the existing work often produces longer and more informative descriptions of objects or their compositions using visual attributes of objects [27 58) or contextual and global image cues [3 (56). However, the richness of descriptions in this line of work are often limited to existing image captioning datasets [28 32
By utilizing a large scale dataset of billions of noisy image-text pairs collected from the web (similar to (9] 224). we aim to generate more diverse sentences with a focus on describing the visual content in controllable detail using a richer visual descriptive space learned from the web.
Length-controlled image captioning has been explored ZeroCap 46 and LIC 13. ZeroCap [46 implements length control as post-pree post-processing step by changing the probability of sampling the -nd-of-sentence token. Hence the model is not naturally trained with word length conditioning in mind and cannot guarantee fine-grained length control at the level of number of words. On the other hand, LIC [3] generates length-controllable captions by conditioning the model with learned tokens that represent different length intervals. However there are considerable differences compared to FlexCap. First, our approach allows for controllability at the level of image regions, while LIC only provides full image captions. This is significant difference, as it allows to generate concise or detailed captions for all the objects in the image. Second, our approach has a more precise level of cuption-length control. LIC uses coarse subjective level of control with four five levels of length (e.g. short, medium, long, and longer), while our approach allows for an exact number of words to be specified. 48 also propose an approach to produce variable length descriptions for objects localized interactively. They use pre-trained image captioner to produce descriptions of objects and ChatGPT in post-hoc to output montiicoondi captions. While in our FlexCap model, the length tokens modulate the output produced by the captioner.
Conclusion
In this work, we introduce FlexCap. a flexible captioning model that can describe localized regions in an image with controllably rich captions. To train FlexCap, we generate a large-scale image-box. box- caption dataset that is rich in diversity of visual descriptions and their length. We achieve this by utilizing existing web-scale noisy image-text pairs and open- voccabulyyy object detection models. We show how localized rich descriptions provided by FlexCap can help us connect images and videos to LLMs and achieve strong performance on visual question answering and dense captioning tasks. We also show that our FlexCap model benefits from contrastive pretraining, localized captions dataset size scaling, model size scaling, and is compliant to length conditioning. We also demonstrate the effectiveness of FlexCap-enabled teeeiize-then--eeseeee approach over the describe-then--lcalie approaches.
10

---

Acknowledgements
The authors would like to thank Matthias Minderer. Lisa Anne Hendricks, Andy Zeng, Matthias Bauer, Anastasija Ili,, Alexey Gritsenko, Vincent Vanhoucke, Jonathan Hoech, Alex Irpan, and Shefali Umrania for their feedback and helpful discussions.
References
[1] I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer. Getting vit in shape: Scaling laws for compute- optimal model design. In Thirty Thirty-seventhh seventh Conference Neural Information Processing Systems, 2023.
[2] J.-B. Alayrac, J. Donahue, P. Luc. A. Miech, I. Barr, Y. Hasson, K. Lene. A. Mensch, K. Millican, Reynolds. R. Ring, E. Rutherford, S. Cabi, T. Han, Gong. Samangooei, M. Monteiro, Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisser- man, and K. Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, 0ess/204,14199, 2022.
[3] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong. S. Samangooei, M. Monteiro, Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisser- man, and K. Simonyan. Flamingo: a visual language model for Few-Shot learning. ArXiv, 141922 2022.
[4] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv e-prints, pages arXiv-2305, 2023.
[5] S. Antol, A. Agrawal, Lu, M. Mitchell, D. Batra, C. L.. Zitnick, and Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[6] H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei. Vlmo: Unified lasgune pre-training with mixture-of-modant seoodality-xxperte Advances in Neural Information Processing Systems, 35:32897 32912, 2022.
[7] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas. S. Wanderman- Milne, and Q Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neclakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901 2020.
[9] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Gryener, B. Mustafa, L.. Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint wwXiv:2209.0679.0 2022.
[10] X. Chen, X. Wang. S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Gryener, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. V. Thapliyal, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. R. Ruiz, A. P. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut. PaLI: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Representations, 2023.
[11] Dai. Li, Li. A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung. and S. Hoi. Instructblip: towards general-purpose vision-language models with instruction tuning. In Proceedings of the 37th International Conference Neural Information Pr Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
[12] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 226-335,70 2017.
[13] C. Deng, N. Ding, M. Tan, and Q. Wu. Length-controllabbe image captioning. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XIII 16, pages 712-729. Springer, 2020.
[14] D. Driess, F. Xia, M.
S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong. T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0333 2023.
11

---

[15] R. A. Google, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin. A. Passos. S. Shakeri. E. Taropa, P. Bailey. Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Mellsee G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Xu, Zhang, G. H. Abrego, Ahn, Austin, P. Barham, J. Botha, Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette- Choo, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng. V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez. G. Gur-Ari, S. Hand, H. Hashemi, I.. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Itycheeiah,, M. Jagielski, W. Jia, K. Kenealy, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, Li, H. Lim. H. Lin, Z. Liu, F. Liu, M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nado, J. Nham, E. Ni. A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov. R. Pope. S. Qiao, E. Reif. B. Richter. P. Riley, A. C. Ros, A. Roy. B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D. R. So, D. Sohn, S. Tokumine, D. Valter, Vasudevan, K. Vodrahalli,
Wang. P. Wang. Z. Wang. T. Wang. J. Wieting, Y. Wu, K. Xu, Y. Xu, L.. Xue, P. Yin, J. Yu, Q. Zhang, S. Zheng, C. Zheng, W. Zhou, D. Zhou, S. Petrov, and Y. Wu. Palm 2 technical report, 2023.
[16] Y. Goyal, T. Khot, D. Summers- Stay, D. Batra, and D. Parikh, Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398- 414, 2016.
[17] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open- vocabllayy object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921, 2021.
[18] D. Gurari, Q. Li. A. Stangl, A. Guo. C. Lin, K. Grauman, J. Luo, and P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3608-3617, 2018.
[19] A. Holtzman, J. Buys, L. Du M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 190.00991499 2019.
[20] R. Hu, A. Rohrbach, T. Darrell, and K. Saenko. Language- condiiiocoo graph networks for relational reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10294- 10303, 2019.
[21] Y. Hu, H. Hua, Z. Yang. W. Shi, A. Smith, and J. Luo. Promptcap: Prompt-guides image captioning for vqu with gpt-3. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2963-2975, 2023.
[22] D. Hudson and C. D. Manning. Learning by abstraction: The neural state machine, Advances in Neural Information Processing Systems, 32, 2019.
[23] D. A. Hudson and C. D. Manning. Gqa: a new dataset for compositional question answering over real-world images. ArXiv, abs/11922.09566, 2019.
[24] C. Jia, Y. Yang. Y. Xia. Y.T.. Chen, Z. Parekh, H. Pham, Q Le, Y.-.. Sung. Z. Li, and T. Duerig, Scaling up visual and vision-languen representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMIRR 2021.
[25] W. Jin, Y. Cheng, Y. Shen, W. Chen, and X. Ren. A good prompt is worth millions of parameters: Low-resoure resource prompt-basee based learning for vision-language models. arXiv preprint arXiv:21 10.08484, 2021.
[26] J. Johnson, A. Karpathy, and L. Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4565-4574, 2016.
[27] D.J. Kim, J. Choi, T.-H. Oh, and S. Kweon. Dense relational captioning: Triple-stream networks for relationship baation based captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6271-6280, 2019.
[28] R. Krishna, Zhu, Groth, Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73 73, 2017.
[29] J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping Language-Image pre-training with frozen image encoders and large language models. Jan. 2023.
[30] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language- -mage pre- nee-rrainng for unitied vision- language understanding and generation. In International Conference on Machine Learning, pages 12888- 12900. PMLR, 2022.
12

---

[31] X. Li, S. Jiang, and J. Han. Learning object context for dense captioning. In Pruceedings of the AAAI conference artificial intelligence, volume 33, pages 8650-8657, 2019.
[32] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 133th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[33] H. Liu, C. Li, Q. Wu, and J. Lee. Visual instruction luning. In NeurIPS, 2023.
[34] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 31 1190-3199, 2019.
[35] M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection, arXiv preprint 2.X0:23306.09683, 2023.
[36] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenbomn, A. Dosovitskiy, A. Mahendran, A. Arna,, M. Dehghani, Z. Shen, X. Wang. X. Zhai, T. Kipf, and N. Houlsby. Simple open-vocabulary object detection. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Isvael, October 23-27. 2022, Proceedings, Part X. page 728-755- Berlin, Heidelberg, 2022. Springer-Vette Verlag.
[37] B. X. Nguyen, Do, H. Tran, E. Tjiputra, Q. Tran, and A. Nguyen. Coarse-to- to-fint reasoning for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4558-4566 2022.
[38] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, Q. Ye, and F. Wei. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations, 2024.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMIRR 2021.
[40] C. Raffel, N. Shazeer, A. Roberts. K. Lee. S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning 2020.
Research, 21(1):5485-555
[41] Z. Shao, J. Han, D. Marnerides, and K. Debattista. Region-object relation-awar dense captioning via transformer. IEEE Transuctions on Neural Networks and Leurning Systems, 2022.
[42] Z. Shao, Z. Yu, M. Wang, and J. Yu. Prompting large language models with answer heuristics for knowledge- snowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14974-14983, 2023.
[43] Q. Sun, Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang. Emu: Gener- ative pretraining multimodality. The Twelfth International Conference on Learning Representations. 2024.
[44] D. Sur'is, S. Menon, and Vondrick. ViperGPT: Visual inference via python execution for reasoning. ArXiv, abs/2303.0812,, 2023.
[45] H. Tan and M. Bansal, Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: (.Xx::9908.0700 2019.
[46] Y. Tewel, Y. Shalev, I. Schwartz, and I.. Wolf. Zerocap: Zera-shot image-to-text generation for visual- semantic arithmetic. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 17918-17928, 2022.
[47] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, Ni, D. Poland, D. Borth, and L.-J. Li. Yfec100m: the new data in multimedia research. Communn ACM, 59(2):64 73, jan 2016.
[48] T. Wang, Zhang, Fei, Y. Ge, H. Zheng, Y. Tang, Z. Li, M. Gao, S. Zhao, Y. Shan, et al. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint ar arXiv:2305.02677 2023.
[49] W. Wang. H. Bao, L. Dong. J. Bjorck, Z. Peng. Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint arXiv:2208. 10442, 2022.
13

---

[50] W. Wang, M. Shi, Q. Li. W. Wang. Z. Huang. L. Xing, Z. Chen. H. Li. X. Zhu, Z. Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint wwXiv:2308.01907 2023.
[51] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904 2021.
[52] J. Wu, J. Wang. Z. Yang. Z. Gan, Z. Liu, J. Yuan, and L. Wang. Grit: A generative region-to-tex transformer object understanding. arXiv preprint arXiv:2212.00 002800, 2022.
[53] S. Wu, W. Zhang, L. Xu, S. Jin, W. Liu, and C. C. Loy. Clim: Contrastive language-image mosaic for region representation. In Proceedings of AAAI Conference on Artificial Intelligence, volume 38, pages 6117-6125, 2024.
[54] D. Zhao, Xiao, Wu, H. Zhang. He, and Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
[55] Xu, T. Mei. T. Yao, Y. Rui. Mst-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288 5296, 2016.
[56] L. Yang, K. Tang, J. Yang, and L.-J. Li. Dense captioning with joint inference and visual context. In Proce Proceedings of the IEEE conference computer vision and pallern recognition, pages 2193-2203 2017.
[57] Z. Yang, Z. Gan, J. Wang. X. Hu, Lu, Z. Liu, and L. Wang. An empirical study of gpt-3 for few-shot knowledge-baat beaseeeeeeeaedd vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4081-3089, 2022.
[58] G. Yin, L. Sheng, B. Liu, N. Yu, X. Wang, and J. Shao. Context and attribute grounded dense captioning, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2019.
[59] H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, Cao. S.-F. Chang, and Y. Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 2023.
[60] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre pe--raining. arXiv preprint arXiv:2303. 15343, 2023.
[61] Y. Zhong, J. Yang. P. Zhang, C. Li. N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li. et al. Regionclip: Region-based language-iman pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-1680) 2022.
14

---

Appendix
A Localized Captions Dataset Details
WebLL. Our dataset creation process on the WebLI dataset results in ~-32 billion triplets. In Figure 9a we show the distribution of caption lengths in the generated dataset using WebLL. We observe that the distribution is not uniform. This is due to the fact that there are more n-grams length
sample than length The average number of unique boxes in an image is 4.19, and the average number of captions per box is 4.04.
YFCCI00M. dataset creation process the CLIP subset of the YFCC100M dataset results in ~0.2 billion Lunge-Box-Cappe triplets. In Figure we show the distribution of caption lengths in the generated dataset using YFCC100M. The average number of unique boxes in image is 6.3, and the average number of captions per box is 3.3.
Distribution Caption Lengths
Distribution of Caption Lengths
Fraction of Captions
Fraction Captions
0.15
0.15
0.10
0.10
0.05
0.00
0.00
Caption Length
Caption length
(a) WebLI Dataset
(b) YFCC100M Dataset
Figure 9: Distribution of caption lengths in the Localized Captions Dataset. We show the distribution of caption length for the region-captions obtained using paired image-texts from two datasets: WebLI [9] and YFCC100M 47
N-gram Filtering. Before matching n-grams with boxes, filter out n- n-grams that do not form informative or grammatically correct captions for boxes. This is done with three steps: 1) Removing any captions composed only of uninformative words (image, background. wallpaper, hd wallpaper etc.); 2) Removing n-grams that begin with words with which sentences usually do not start (of, on, in etc.); 3) Removing n-grams that finish with words with which sentences usually do not end (a, the, to, on etc.). This step is essential to reduce noise present in the large-scale image-text pair dataset obtained from the web. is also important for the captioning model to produce grammatically correct informative sentences.
Dataset Samples. We show some samples from the dataset in Figure 10 The alt-text from which the box captions are generated is provided as the title of the image. Note the alt-text gets clipped due to display-length limits which is why the detected boxes might have captions not visible in the displayed alt-text directly. We next discuss how captions of varying lengths are matched with different objects in image.
B Ablations and Analysis
Large-scale Contrastive Pre- vee-Trainigg In this section, we study the impact of using a contrastively pre-trained vision encoder or training from scratch using only box-caption objective. For this experiment, we use VIT-B/16 as our backbone and train on the WebLI Region Captioning dataset. We report the results on ground truth (GT) box captioning and dense captioning on GRiT proposals on the Visual Genome dataset in Table[5a In Row 1, we observe that a congth-oondiiooed region captioning objective can be used for training both the backbone and text decoder from scratch and achieve competitive performance. Second, we observe using a large-scale contrastively pre- trained (CPT) model results in better performance. Note that these CPT weights have already been open-sourced [60].
15

---

Mecihpplll thicer murgo lamato sahe
lestliow brand somen Quatte
CHROUS
Sa
NEW ONT
Bosti wattem boiee calliee
All Teat cotran darmask tableclorh
12'eee hests oth
Carras- rabledun
Riba aet SPhentt partann
fox g
Too hand gloves bike
Ceedo ham plor
Toxaa soay hal bween sumpkin in smow tale
At TEat mollss 6 d0ug aphabet glant Ribor
8bo GAANT RADEE Fozzls
Figure 10: Samples from the Localized Captions dataset with images from the WebLI dataset 9 We only visualize maximum of boxes for each image avoid clutter.
16

---

VG mAP 1
VG mAP f
VG mAP f
CPT
GT
GRiT
Dataset
Size
GT
GRiT
Backbone
Params
GT
GRiT
43.6
15.6
YFCC100M WebLI
0.2B
38.5
14.2
VIT-B/16 SO-ViT/14
86M 428M
45.1
15.8
45.1
15.8
32B
45.1
15.8
46.9
16.2
(a) Contrastive Pre- training
(b) Data Scaling
(c) Model Scaling
Table 5: Ablations. We vary different aspects of model pre-training, dataset size, and model size and measure its effect on region captioning task on the Visual Genome dataset. In (a) and (b) the visual backbone is ViT-B/16 and in (b) and (c) all models start from contrastively pre-trained weights.
Figure 11:
Objects missed by LLAVA but found by FlexCap. We show some typical examples of objects (5 per image to avoid clutter) that FlexCap can find with OWL-ViT but LLAVA does not.
Data Scaling, In this experiment we measure how scaling the localized captions dataset affects performance. We first take VIT-B/16 encoder contrastively pre-trained with large image-text dataset (WebLI). We now train it on two different region captioning datasets of different scales. We show the results of this experiment in Table 5b We find that even after contrastive pre-training on large scale, a large region-caption dataset can result in significant performance boost (~5% mAP for GT box captioning). This shows the importance of designing a scalable dataset creation method such as the one introduced in Section 4.3 We also note that the dataset and pre-trained weights required to reproduce the model with YFCC100M are available publicly.
Model Scaling, Next, we measure the impact of changing the model size. We train two models: ViT-B/16 (85M) v/s SO-ViT/14 (428M). Just like the data scaling experiments, both of these have been pre-trained on the WebLI dataset in contrastive manner. We report the results of this experiment We find that the larger model results in better performance.
in Table 5c
C Open-Ended Object Detection
In this experiment, we compare the effectiveness of describe-then--ocaliee method (specif- ically LLAVA 1.5 (33 7B + OWL-ViTV2 (35)) with a localize-heen-deerriee method (specially FlexCap OWL-ViTv2) for the task of open-ended object detection. For the describe-then-localiz method, we generate a list of objects using the following promptteecrribe object names and regions in this image. We extract the nouns and use them as text queries for ViTV2 For the localize-then-deseee 128 objects ranked by objectness score by OWL-ViTv2 and describe them with different length prompts. In Fig. 5 in the main paper, we compare recall of the ground- regions annotated in the Visual Genome dataset for both these approaches. We find that while describe-then hhenlloallize with LLAVA can be effective approach for finding large objects, localize-then-descrie with FlexCap is significantly better at medium and small sized objects and marginally better for large-sized objects. We show some typical examples of missed detections in Fig.
approach, we take the top
17

---

Ougpus captlrr
Ouguus captors
condRoona capton
A man soanding
black
A lring rocm with lepttp
track
The photo bathrosem taken in
photo living taken In
taken
Notice the ra
Notice oif Untte from that
Notice inptoopop.oo
brann conbuctor
open
Figure 12: Diverse captioning with Prefixes. FlexCap can be used to perform conditional captioning of images. We show three prefixes: "a", "the photo taken in "notice" resulting in diverse captions for the same image. Note the input red bounding box is around the full image.
Maximum Length of Captions
Dataset
2
8
VQAv2 (val split)
57.4
59.9
66.4
67.8
GQA (testdev-balancee split)
43.6
45.3
47.7
48.8
Table Effect of caption lengths on FlexCap-LLM
D Different prefixes with same image
In Fig. 12 highlight how different prefixes can be used to generate diverse captions. Note how in column I the model correctly identifies the country even, possibly using the logo on the train. We find the prefix "Notice" leads to captions highlighting noteworthy aspects in an image which the un un-prefixed caption does no.
E
Visual Dialog
FlexCap-LLMM be used for the task of visual dialog 112 We first caption all the objects in the image using FlexCap and OWL-viTv2. We retain the top 128 boxes according to the objectness score from OWL-ViTv2 and describe each box using FlexCap. See Fig. and Fig. 4 in the main paper for more details. Once the image has been represented as the list of objects in text, we can interact with LLM by providing the conversation turns as additional context for each query to the LLM. We show some examples of conversations with the FlexCap-LLM system in Figure 13] Note how the model able to read text the image in the leftmost figure, recognize material the middle figure, and localize objects of interest in the rightmost figure. As we compute the object captions only once at the beginning VLM for each additional turn in the conversation. We show more examples of this on the webpage in the FlexCap-LLLM section of index. html.
the conversation, there is no additional overhead of querying large
Effect of Length on FlexCapLLM
In this experiment, study the effect of generating captions of different lengths downstream visual question answering tasks. We evaluate this by using captions of increasing lengths on the val split of the VQAv2 dataset. We report the results of this experiment in Table We find that increasing the length of generated captions results in better VQAv2 and GQA performance.
G FlexCapLLM Details
In Fig. 14 we show the whole pipeline how FlexCap interfaces with LLMs. We convert an image into list of localized texts. These texts passed onto the LLM with preamble indicating the task and the VQA task's question. We use the following prompts for the LLM the guestion-answering experiments.
VQAv2, OK-VQA and GQA.
Preamble:
18

---

o
What happening in the photo?
What this picture?
picture?
runway.
The image contains blue bows, knitted blanket, spoon, taoth, eyes, button, cereal,
image contains following objects: apples. banares, grapes, kowi fruit. lemons, mangoes. oranges, potatoes, tometoes
Which pointed? Why?
The airplane pointed the right image.. is located the right side the image, and the tail of the plane located on the left side of the image.
What fruit the image?
What is the color of the monster?
is red monster with white
Brown kiwi fruit
horns, teeth and eyes.
What is the price grapes?
What fight is it?
Where is the bowl placed on?
The price of grapes not listed.
The flight operated Singapore Airlines.
Blue blanket
Figure 13: FlexCap-LLM for Visual Dialog
Prompt LLM
Preamble intelligent
fence behind the tennis court
region detections
FlexCap
Box-caption pairs [:]20 196, 168). Tennis (acket)) (I (1188, 288, 828, 348 green sennda ball ground)....
LLM
Tahiits
Bounding Proposals (OWL-VITv2)
oreenn Jannis
ground
Answer Minbledon
White tennis shoes
Question
Caption Prefixes
the this sport's english competition?
Figure 14: FlexCap for VQA with bounding box proposals and an LLM. FlexCap generates captions for different regions in given image. To answer any open-ended questions, we prompt an LLM [5] with FlexCap's detections (box-caption pairs).
preanble "You helpful assistant anavering questions inages people. 1ist of object detections in the inage and ansver questions. The inage content may not be sufficient to rely
look the knovledge resources commonsense.
you may nany objects were desected. They are listed the folloving format: [object descriptions] [ex, bl, vhere ex is coordinate of the center, ey is the y coordinate of the center, ie the width and is the height of the bounding box of that object in the image."
questions,
image,
Image Size:
imago_ _tt___sie_________e______ _topppp "The height of the is (image_hoight) and width of the (imago_width)+
Image description:
Luage_deecriptton I"Full inages descriptione this inage Aluaage_captizan
Object representation:
19

---

(trane_idx) following objecta detected" for captions in bbjjett_captione__nn (capttions) = "The 11st of objects is followa:
for captions, (ex, ey, v, in sipfotject_captionn (bject_boxee)) objects_descripion 4- (capptions)) ((ex), (cy), (v), (b)],"
Question prompt:
question..proppt +*Q: (question) one
Full prompt:
foll_pronpt (preabble inage_aize.prone image_deacription dbjects_descriptinn + question.prompt)
VizWiz
Preamble:
helpful list of object detections in the image and answer questions. The image content not be sufficient to answer questions, nany objecta were detected. They are listed the folloving format: [object descriptions] [ex, [score], where ex is coordinate the center, cy ie the y coordinate of the center, is the width, height and likely inaccurate, and this often nakes the question unanswerable. You can anewer questions as upansverable'.
presuble
ansvering questione about inages people.
you may
image, raly external knooladge resources commonsense.
hl
object detaction. Lov
the detection
Image Size:
inago_ aarg___sz___prrree "Te height of the image is (inage_hoight)) and width of the image is (imago_width)'.
Image description:
inage_deacription r'Fall Imagea descriptions for this inage are: finage. captions]"
Object representation:
pajacts_descriptte "The 1ist of objecta follows:
for captions, (ex, ey, v, 1)) in 2ip(object_capte object_boxee)) boxes):
abjacta_descripttttt ('[captions) [(ex), (ey), {o), (bH]"
Question prompt:
question.proapt #1Q: (question) Ansser in one word. A:
Full prompt:
foll_proupt (preanble image_aize.prompt meaga_deacription eejjecta.escriptiinn question_ pronpt)
MSRVTT and MSVD
Preamble:
preamble "You are a helpful assistant ansvering questions about videos to people. You can look at the list object detections in frane and queations."
Object representation:
"In & video, many objects vere detected in each frane."
for frame_ide frame_idxes:
Question prompt:
question_prompt +:Q: (question) Ansver in one vord. A:*
Full prompt:
full_proupt (preanble objeeta_deeerrppiinn (ueapioonppropt)
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that we can train a vision language model conditioned on bounding box and desired caption length. We show in the experiments that train such model. Furthermore, we claimed that using this model we can describe an image densely using just words. We show that connecting these descriptions an LLM leads to better dense captioning performance and zero-shot visual question answering performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the Discussion section we have included limitations of the current work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:No theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes the dataset is produced using public model. The weights of our vision backbone are also open-sourced, The decoder is trained from scratch. We include results in the ablations in the appendix section for models trained on open-source YFCC1 data. We will release training code and model inference code. The part that cannot be open-sourced is the WebLI dataset. We will attempt to release the model trained on WebLI dataset though.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The base open-source dataset with which results can be replicated is YFCC100M. The object detector used is OWL-ViTv2. The vision backbone is SigLIP
pretrained SO400M.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
cublic/guides httpei/ggiiide/eoeettiieeesees SrodeSuumissiooPoll11
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips http:/uullcggguudeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification:
We mention these details in the implementation section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bars are not included as conducting a single run pre-training is very resource intensive due the large size of the dataset.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Dataset size, model size, batch size, hardware and number of training steps have been mentioned in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read the code of ethics and this work does not violate its guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]

---

Justification:
Refer to Broader Impact section of the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Models will be released after safeguards are put in place.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All work that directly used in development of data and models used in this paper have been appropriately cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
25

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palasets.papersstt paperswi thcode. com con/dataset package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
New assets are not released right now.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work did not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work did not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "6nK59Qxan8", "reviewer": "Reviewer_eaKh", "summary": "This paper introduces a versatile captioner capable of generating region-specific descriptions with controllable information density. This functionality enables dense captioning tasks and enhances visual question answering (VQA) by integrating with a LLM. The paper also presents a large-scale dataset containing image-text-box triplets, which is valuable for the community to explore region-controllable captioning capabilities. Leveraging this dataset, FlexCap can produce localized visual descriptions with adjustable lengths. By providing various localized textual descriptions of images as input to a LLM, FlexCap-LLM demonstrates strong performance in VQA tasks.", "questions": "Q1. The abstract is divided into two paragraphs (Line #6 and Line #7).\n\nQ2. Section 4.1 on Correctness lacks a reference of Table 1(a).\n\nQ3. Line 249 lacks a reference of Figure 8.\n\nQ4. Regarding weakness 1, can the authors provide some statistics to support their statement?", "limitations": "It is commendable that the authors discuss the biases present in FlexCap. However, FlexCap is quite basic, and FlexCap-LLM is not an end-to-end model. It would be intriguing to see how these components could be combined into a fully integrated vision-language model. As mentioned in Weakness 3, FlexCap lacks sufficient flexibility; incorporating more visual and textual controls is an important area for further development. But it actually takes a step forward.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed dataset can promote community research on visual-controllable captioning task, which is useful for the development of user-friendly vision-languague models.\n\n2. FlexCap is easy to follow, and its controllable captioning capability, with positional information and varying information density, is beneficial for downdream tasks like VQA.\n\n3. The experiments are comprehensive, demonstrating the capabilities of region control and length control (Sec 4.1, Sec. 4.3). The VQA results generated by FlexCap-LLM show good human-interpretable representation of the generated located descriptions.", "weaknesses": "1. The architecture of FlexCap and its training setup lack novelty, as it is a typical transformer-based captioning model. However, this does not lead me to reject this paper, as the contributions on task and dataset is useful.\n\n2. The authors should carefully consider their statements in the paper. While this paper achieves region and length control, there are many other controllable signals such as mask/point control in visuals and emotion/style control in text (as seen in Caption Anything [45]). Referring to FlexCap as a versatile flexible-captioning vision-language model might be an overstatement.\n\n3. In lines 66-68, the authors state that \"the next-word prediction loss encourages the model to increase its score for the <e> token and decrease the score for the word playing.\" This statement is intuitive. I believe the probability of \"<e>\" and \"playing\" depends on the frequency of occurrence of \"a dog <e>\" and \"dog playing …\" during training, i.e., depending on the training set. It would be beneficial to see the statistical probabilities of \"<e>\" and \"playing\" following \"dog\" to support the authors' statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and meticulous attention to detail in reviewing our paper. \n\nWe address the concerns raised by them below:\n\n>Referring to FlexCap as a versatile flexible-captioning vision-language model might be an overstatement.\n\nWe used this description to highlight the fact FlexCap can be conditioned using bounding box, desired caption length, and caption prefix to produce diverse captions of objects and regions in images. We show many examples of the flexibility in Fig.5, Fig. 8 and Fig. 13 in paper and in the supplementary webpage. That said, we agree that, when implemented effectively, there could be many more spatial and textual controls for captioning and with FlexCap we take a significant step towards properly exploring this space.  We will also soften the claims on versatility in the abstract by focusing more on the specific controls that we introduce (i.e. bounding boxes, length, and caption prefixes) while providing the larger context on potential spatial and textual controls.\n\nAlso, while theoretically the CaptionAnything system can produce outputs of different lengths, we tested it for length conditioning and it does not consistently return outputs of the desired length. \n\n\n> Regarding weakness about “statistical probabilities of \"<e>\" and \"playing\" following \"dog\" to support the authors' statement.”, can the authors provide some statistics to support their statement?\n\nWe agree that whether this occurrence will be a problem depends on the dataset statistics. To quantify the prevalence of this problem, we compute a statistic: for each image box, we consider all pairs of captions and measure the fraction sharing prefix words. For instance, one box has three captions: \"dog <e>\" \"dog playing <e>\", \"dog playing with a frisbee <e>\" which share a prefix \"dog\". While another caption for the same box: \"brown dog <e>\" does not share a prefix with captions beginning with \"dog\". After averaging this metric across all images in the dataset, we found that 30.8% of caption pairs share a prefix. The length conditioning token assists in distinguishing between captions with the same prefix while also providing the model with a novel capability during inference. When length conditioning is applied, the probability of prefix similarity decreases from 30.8% to 11.1%. We appreciate the encouragement to explore this issue quantitatively and plan to include this analysis in our paper.\n\nWe answer the other questions asked by them below:\n\n> Q1. The abstract is divided into two paragraphs (Line #6 and Line #7).\n\nWe will merge the two paragraphs into one.\n\n>Q2. Section 4.1 on Correctness lacks a reference of Table 1(a). \n>Q3. Line 249 lacks a reference of Figure 8.\n\nThanks for catching this. We will add a reference to Table 1(a) in section 4.1 and to Figure. 8 in Line 249.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uEPFkDHZmK", "reviewer": "Reviewer_bbcR", "summary": "This paper proposes a vision-language model termed as FlexCap, which given a specific region in the image represented as a bounding box, outputs a description of that region in a length-controllable fashion where the exact length of the generated description can be controlled via a prefix token. First the authors harnesses a dataset of region-text pairs using existing large-scale image-caption datasets such as YFCC100M and WebLI. Specifically, they extract n-grams from the caption, apply a filtering process, and each of those n-grams are then fed to OWL-ViT to get a bounding box. This allows for multiple variable-length captions for a region. They then train FlexCap on that dataset. Generating a large set of descriptions for many possible regions in the image allows for a complete detailed understanding of the image, and those generated descriptions can be fed to a LLM to perform VQA by considering and reasoning over those descriptions. Experiments are conducted on dense captioning (without localisation), zero-shot VQA (image and video), and other tasks such as image labeling, object attribute recognition, and visual dialog.", "questions": "Regarding [W1], there are some works [R2, R3, R4] published in ICLR 2024. But I also understand that they were officially published and presented shortly before the NeurIPS deadline, and therefore, I will not consider them as grounds for rejection. However, the authors should take note of them and cite them, as they are proposing very similar works. However, how does the proposed method compare to [R1]? Moreover, although the length-controllable concept is incremental, I believe it is important in such systems. And the authors address this problem well. \n\nI would also like to hear a clarification on [W2, W3, W6 and W7].", "limitations": "Limitations are discussed. No special issues on negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of using controllable caption generation for specific image regions is interesting an can be beneficial in the case of dense captioning, since some regions are more detailed than others and need more words to be described, while other regions can be described sufficiently by 1-2 words. Thus, having this controlability can be helpful and enhance efficiency, i.e., generating 3-words per region is much more faster and efficient than generating 20 words per region, especially if those descriptions are to be fed to a LLM, where the context length decreases with less words per description. \n- The authors show how to make use of existing VLM to serve as annotaters. This is very heloful, especially in scenarios where we don't have a specific dataset tailored to our needs. \n- Evaluation is performed on different tasks, and impressive zero-shot performance.", "weaknesses": "- [W1] The novelty is limited; it only involves adding a length prefix to condition the generation of length-controllable output. However, the idea of generating region proposals from image-text pairs, and then learning to describing those regions has been a hot research topic lately. Some related works:\n1. Osprey [R1] generates descriptions for fine-grained objects rather than coarse-grained bounding boxes with also the capabilities of intruction-based training on those fine-grained regions. \n2. Kosmos-2 [R2], where given an image-caption pair the text is processed in a similar way that the authors do to extract sentence chunks (of variable length), and each text chunk is then associated to a bounding box using the GLIP grounding model (instead, the authors use OWL-ViT as the grounding model) and an autoregressive transformer is learned to predict the description of each region. Actually, Kosmos-2 is also capable of grounding, and does not require an external model to perform grounding at test time.\n3. In the work of [R3] they use MiniGPT to extract a caption and then the noun from that caption, as well an OCR detector. They then feed these to a LLM to imagine possible other nouns, as well as BLIP model where they feed the cropped region and generate a caption for that cropped region, and extract the noun phrases\n4. [R4] takes a step futher and not just generates a caption of a region of interest, but also allow perform instruction-based training such that a user can chat with a given region of interest. \n5. There is also a much more close work to the authors: [R5], however, I will not consider this paper in my decision-making process, as it is published on arXiv and not yet published in a peer-reviewed venue. \n\nTherefore, I think this work is an incremental improvement of plugging in the length-control idea (which has also been well studied, and the authors mention that in the related work) in region-based captioning\n\n- [W2] How should one choose the length of the region descriptions? Isnt there a heuristic to estimate what regions need more words than others? What is the point of having a length-controlled method of generating region descriptions, if there still needs to be a manual selection of the length? \n\n- [W3] Controllable image captioning itself is not a new concept. The work of [R6] proposes to do this just in another way where the caption (and not the length) can be controlled. Similarly, their method can control the detail and length of the caption.  How does the method of the authors compare to this work?\n\n- [W4] InstructBLIP [R7] is not compared to, which seems to perform better than the proposed method (see GQA for example). Also, the authors dont compare on the standard VQAv2 test-dev set benchmark, which makes it hard to compare if their work outperforms previous works. Most works report on VQAv2. \n\n- [W5] The dataset that the authors generate, as well as the performance of FlexCap, is highly limited by the performance of OWL-ViT (since it serves as an annotation tool to extract regions). Any biases and errors in OWL-ViT will also be transferred to the dataset and FlexCap. Consequently, the model and dataset are heavily influenced by the quality of OWL-ViT.\n\n- [W6] L178, regarding the zero-shot setting, the authors finetune the model on COCO (according to L168). But the VQA dataset (or its variants) is actually built on COCO. While it is not finetuned on the VQA annotations of question-answers, the model is still trained and sees the image, its caption, and its bounding boxes, which all three I think cover well for VQA. Therefore, this method cannot be seen as zero-shot.\n\n- [W7] For VQA, the amount of captions generated seem to be huge (128 regions $\\times$ number of prefixes used), and therefore is this a fair comparision with other works? Couldn’t a similar or better performance from the compared models, be achieved by using more generated captions (e.g., by different samplings)? There is no ablation study on this, and on performance achieved by varying the number of generated descriptions, which I find important. \n\nMinor/Unclear:\n\n1. Table 1a is not cross-references \n2. Line 137, for each region, do the authors generate 20 captions for each of the (1,2,3,4) lengths? So that equates to 20x4 = 40 captions per bounding box? Are you then averaging over all the 40 captions in CLIP text encoder?\n3. Line 160, what language queries are used here? Is the process similar to how the dataset is built, using N-gram processing?\n4. Line 168, Isn't the model natively built to describe regions/boxes rather than localize them? How is the model adapted to localization? \n\nReferences:\n\n[R1] Osprey: Pixel Understanding with Visual Instruction Tuning, CVPR 2024\\\n[R2] Grounding Multimodal Large Language Models to the World, ICLR 2024\\\n[R3] Towards Panoptic Visual Recognition and Understanding of the Open World, ICLR 2024\\\n[R4] The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World, ICLR 2024\\\n[R5] GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest\\\n[R6] Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions, CVPR 2019\\\n[R7] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning, NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for their insightful feedback and thorough review.\n\n>However, how does the proposed method compare to [R1]? \n\nIn R1, the training dataset is generated by querying GPT4V with regions of interest found by SAM. While they do train OSPREY with this dataset, the main ability of describing regions was already there in GPT4V. We do not know publicly what the dataset and architecture used for GPT4V are. In our work, we are proposing a ground up way of creating a large-scale region description dataset from alt-text data with a publicly available object detector (OWL-ViT).  Furthermore, R1 shows a language model (VICUNA) can be trained further with their dataset to learn a visual instruction tuned model. We show that for describing objects we do not need a full-fledged language model but training a reasonably sized text decoder from scratch is sufficient. The other difference is length conditioning but that was not the focus of R1. \n\n>Regarding [W1], there are some works [R2, R3, R4] published in ICLR 2024. \n\nThank you for sharing these references. We will add them to related work as they all are relevant.\n\n> [W2] How should one choose the length of the region descriptions?... What is the point of having a length-controlled method of generating region descriptions, if there still needs to be a manual selection of the length?\n\nThe objective is to have one model that can produce long and short descriptions as needed. For example, if someone is interested in recovering only object names they can request for length1 captions from the model. However if one is interested in a longer description of an object then they can ask for length8 captions. In a way, our model merges the label spaces of object detection datasets and dense captioning datasets by using length conditioning. \n\nAlthough there is a common-sense based length prior that most people use for describing a region, people can also describe the content with a few or a lot more words when requested. Our model has this flexibility to be queried at a desired level of detail. But by fine-tuning the model with common-sense human captions (e.g. Visual Genome), our model adjusts itself to the average human biases in information density. We show examples of this in *flexcap-spatial.html* in supplementary material. Although some heuristics can be introduced, we believe fine-tuning with a dataset that captures the desired information density is a more configurable approach to the output information density problem.\n\n> [W3] Controllable image captioning itself is not a new concept. ... How does the method of the authors compare to this work?\n\nThere are several differences with R6. First, R6 only handles image captioning. While we handle object, regions, and full images by using the flexibility of conditioning with a bounding box. Second, their conditioning approach is a specific way of producing different captions by changing the order of detected objects as conditioning. They detect N objects in an image. The captioner is shown different orders of N objects to produce different captions. While this produces grammatically correct captions in different ways, it does not necessitate new information being generated in the captions. The objective in our work is to use length conditioning as a proxy for information content. Both are valid but different objectives of conditional image captioning. Finally, our approach allows for prefix conditioned captioning which can be used to extract attributes of interest: color, material, action, function and text. This capability emerges due to pre-training the model on a large-scale dataset which is missing from R6.\n\n>[W6] L178, regarding the zero-shot setting, the authors finetune the model on COCO (according to L168)... Therefore, this method cannot be seen as zero-shot.\n\nIn our study, we used the term \"zero-shot\" to indicate that the training process did not involve any VQA annotations of question-answers. The evaluation was conducted on non-train splits of COCO, ensuring that the model has not encountered the evaluation image was not seen in any phase of training. Hence, we maintain that the setting can still be considered zero-shot. To provide greater clarity, we will explicitly mention this in the final version of the paper.\n\n>[W7] For VQA, the amount of captions generated seem to be huge (128 regions × number of prefixes used), and therefore is this a fair comparision with other works?...\n\nThe VQA baselines systems are trained to produce answers using different approaches (mostly end-to-end). They do not have a similar system to ours where we generate a large chunk of text and then deduce a single answer from that. We will add an ablation on how the number of generated descriptions affects performance.\n\n>[W5] The dataset that the authors generate, as well as the performance of FlexCap, is highly limited by the performance of OWL-ViT ... \n\nYes we mostly agree with this assessment and have mentioned this in the limitations section. However, just to clarify, OWL-ViT does not define what to caption in images. This information comes from alt-text associated with the images in the form of n-grams. OWL-ViT mainly localizes provided n-grams on the images using the box proposals which can arguably cover almost all the interesting regions in the image. Moreover this automated approach allowed us to scale both in size and diversity which often is the main influencer of the performance, as demonstrated with many results in our experiments section. \n\n>[W4] InstructBLIP [R7] is not compared to ... Most works report on VQAv2.\n\nTo clarify any concerns, we will test our method on VQAv2 and provide the results on this dataset in the final version of the paper. We will also add InstructBLIP to the list of baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVWcT6Q67B", "reviewer": "Reviewer_T9RF", "summary": "This paper introduce a versatile flexible-captioning vision-language model called FlexCap, capable of generating region-specific descriptions of varying lengths. The FlexCap use caption length to control the information density of the generated sentences. The paper also introduces a large-scale dataset of image-text-box triplets for this flexible region captioning task. The proposed model outperforms baselines and achieve leading performance under this flexible captioning scenario.", "questions": "Please see my weakness comments.", "limitations": "Please see my weakness comments.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper proposes a novel mechanism to control the complexity for region captioning.\n\n2.\tIntroduce a large-scale image-text-box dataset.\n\n3.\tAchieve SOTA VQA performance on several benchmarks.", "weaknesses": "1.\tAbout the dataset building:1) It seems that there is no human involvement in the data construction process. I worry about the correctness and diversity of the generated sentences. 2) One of the key contributions of the data is the different lengths of descriptions for a region proposal. There should be at least a statistic about the length distributions. 3) Based on Table 1(b) and the cases presented in Figure 5, is the max length 8 and is it enough for a complex region/object?\n\n\n2.\tAbout the baselines: I think it is important to list the backbone types and parameter scaling for each baseline in the result tables. For example, the OWL-ViT has ViT-H/14, ViT-L/14, ViT-B/32, etc. The CLIPM could be applied to different backbones. The parameter scaling (especially the LLMs) will significantly affect performance.\n\n\n3.\tAbout the evaluation: 1) Using the CLIP similarity to map captions and objects is plausible but just one-sided. Have you considered adopting some rule-based verbalization methods to map captions and class names? 2) Back to the motivation, I wonder how to evaluate the information density for each generated caption? Do the authors have any insights?\n\n\n4.\tOverall, the proposed framework seems conventional. Can the author discuss what makes the proposed model outperform current similar architectures? How and to what extent does the prefix token influence the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reviewing our work. \n\nWe address the concerns raised below:\n\n> About the dataset building:1) It seems that there is no human involvement in the data construction process. I worry about the correctness and diversity of the generated sentences. \n\nWe were also curious if a dataset generated in a fully automated manner would be useful. For correctness, we were interested to know if the produced captions are useful for downstream tasks and found them useful for a variety of VQA tasks and region classification. Diversity-wise we choose the n-grams from alt-text of the captions instead of pre-selecting the words. The vocabulary is diverse because of the phrases found in the alt-text of images. Please see examples of the dataset in Figure 11. \n\n> One of the key contributions of the data is the different lengths of descriptions for a region proposal. There should be at least a statistic about the length distributions.\n\nThese statistics are given in Figure 10 in the Appendix. We agree that this is important.\n\n>Based on Table 1(b) and the cases presented in Figure 5, is the max length 8 and is it enough for a complex region/object?\n\nWe have empirically found the length to be sufficient for tasks like region classification and VQA tasks across different datasets where our model outperforms baselines with a max length of 8. We also found that the model  pre-trained with max length of 8 words can be fine-tuned to produce longer descriptions for regions like those present in the Visual Genome dataset. In the Supplementary material (flexcap-spatial.html), we show that the model produces a mix of short and long captions in the style of Visual Genome. There are 50 captions in the 40 images with length longer than 8. Finally, it is possible to use lengths longer than 8 words for pre-training. \n\n>About the baselines: I think it is important to list the backbone types and parameter scaling for each baseline in the result tables. For example, the OWL-ViT has ViT-H/14, ViT-L/14, ViT-B/32, etc. The CLIPM could be applied to different backbones. The parameter scaling (especially the LLMs) will significantly affect performance.\n\nYes, we agree. We will add this information to the final version of the paper.\n\n>About the evaluation: 1) Using the CLIP similarity to map captions and objects is plausible but just one-sided. Have you considered adopting some rule-based verbalization methods to map captions and class names? \n\nIt is possible to build a more complicated method than using CLIP similarity. But this is an evaluation technique that allows us to show the captions generated by our model are correct. So we use text-image embedding matching used commonly for evaluating zero-shot recognition in papers (CLIP). Even with this simplistic approach, we find that we significantly outperform the baselines. We will add an ablation using descriptive classnames in the final version of the paper.\n\n> 2) Back to the motivation, I wonder how to evaluate the information density for each generated caption? Do the authors have any insights?\n\nWe are not directly addressing information density in captions. Instead, we are using caption length as a proxy for information content.  We observe that captions incrementally add more information as we increase length and add this ability to the model for users to control. If a length 1 caption just mentioned object class name, length 4 would contain information about attributes, and length 8 would contain information about context and attributes. This can be seen in the examples shown in Figure 5 and the Supplementary material (flexcap-length.html). It is possible to parse the produced captions to show length 1 captions are mostly nouns, length 2-4 are adjectives and nouns, but for longer captions image context we need a more complex method of measuring information density than just parsing.\n\n> Overall, the proposed framework seems conventional. Can the author discuss what makes the proposed model outperform current similar architectures? \n\nOur contributions are: endowing vision-language models with a new capability of producing length controlled localized captions and \nproducing a large-scale dataset of image-box-caption triplets that can be used to train the model. We are *not* proposing architectural changes to achieve length-controlled localized captions.\n\n> How and to what extent does the prefix token influence the performance? \n\nThe prefix token enables a new capability for the captioning models. The same model can be used for producing short and long captions in a controllable manner. We further show our dataset creation technique allows us to generate length-controlled captions for both small and large regions in an image. Also, in the *Length Conditioning* subsection in Section 2, we describe how the length token helps when there are many captions for the same box.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper unanimously receives positive rates thanks to the dataset novelty of controllable captions and convincing experiments with the dataset. Although overall reviews are positive, the clarification in the rebuttal should be reflected in the final draft, including dataset analyses and additional details of experimental results.
Decision: Accept (poster)