Query-based Image Captioning from Multi-context 360° Images
Authors: Koki Maeda
Shuhei Kurita
Taiki Miyanishi
Naoaki Okazaki
Venue: EMNLP 2023 Findings (2023)
Keywords: Image Captioning
360-degree image
Vision and Language

Abstract: A 360-degree image captures the entire scene without the limitations of a camera's field of view, which makes it difficult to describe all the contexts in a single caption. We propose a novel task called Query-based Image Captioning (QuIC) for 360-degree images, where a query (words or short phrases) specifies the context to describe. This task is more challenging than the conventional image captioning task, which describes salient objects in images, as it requires fine-grained scene understanding to select the contents consistent with user's intent based on the query. We construct a dataset for the new task that comprises 3,940 360-degree images and 18,459 pairs of queries and captions annotated manually. Experiments demonstrate that fine-tuning image captioning models further on our dataset can generate more diverse and controllable captions from multiple contexts of 360-degree images.

Query-based Image Captioning from Multi-context 360 Images
Koki Maeda 12 Shuhei Kurita² Taiki Miyanishi Naoaki Okazaki'
"Tokyo Institute of Technology RRIKEN AIP *ATR
koki. www.aenlp.deccmmmm c. titech. ac. jp, shuhei kurita@riken, miyanishi@atr... jp, okazaki@c. titech. jc
Abstract
360-degree image captures the entire scene without the limitations of camera's field of view, which makes it difficult to describe all the contexts in single caption. We propose a novel task called Query-based Image Cap- tioning (QuIC) for 360-degree images, where a query (words or short phrases) specifies the context to describe. This task is more chal- lenging than the conventional image caption- ing task, which describes salient objects in im- ages, as it requires fine-grad fineggrained scene under- standing to select the contents consistent with the user's intent based the query. We con- struct dataset for the new task that comprises 3,940 360-degree images and 18,459 pairs of queries and captions annotated manually. Ex- periments demonstrate that fine-tuning image captioning models further generate more diverse and controllable captions from multiple contexts of 360-degree images.
Scene Query: Traffie
Caption: currently no vehicles travelling the
are several vehicles parked along the sides of the street.
Scene Query: Plants
Caption: There two Dower beds in the medians between the street that have blue, pink and white flowers planted along their edges.
Scene Query: Weather
Caption: sky mostly cloudy and gray ground wet has just reined.
dataset can
Figure Overview Query-baser Image Captioning from 360 images, task is generate relevant scene descriptions of images corresponding to linguistic information.
1 Introduction
Image captioning is a task describing the con- text of an image in natural language (Vinyals et al., 2015; Xu et al., 2015; Lu et al., 2017). Existing im- age captioning datasets, such as Microsoft COCO Captions (Lin et al., 2014) and Flickr30K (Young et al., 2014; Plummer et al., 2015), are based on the conventional images with limited camera field-of- view. For such images, the contexts of the images are selective and even simplified because people tend to choose what to photograph and crop out, Such images often include a clear context, such as "a bird flying over a body of water" or "a cat sitting on mat." Image captions in the existing dataset often follow such salient context, ignoring minor but negligiiee contexts of the image. This becomes particularly problematic in the case of 360-degree images captured by omnidirectional cameras, as they capture the entire visible context the scenes indiscriminately and preserve rich minor details.
In this paper, we explore a novel Query-based Image Captioning task for 360 images (QuIC- 360%). 360-degree images from omnidirectional cameras are indispensable in several fields, such as autonomous cars (Liao et al., 2022) and house- hold robots (Srivastava al., 2022). These images have no constraints on the camera field-of-view: therefore they capture rich contexts of the distant scenes. Indeed, multiple possible captions exist corresponding to the different aspects of the 360- degree images. Thus is expected to generate textual descriptions that align with the user's in- terests and the relevant contexts when we develop systems that describe remote contexts from 360- degree images. Unfortunately, such rich contexts in 360-degree images are fully expressed in the ex- isting image captioning tasks. Conventional image captioning models concentrate on several salient contexts, ignoring other contexts that can be in-

---

Standard Image Captioning
Controllable Image Captioning
Query-based Image Captioning (Ours)
Caption
Caption
Caption
Image Captioning Model
Image Captioning Model
Image Captioning Model
Context
Additional
Scene
Information
Query
(e.g, POS tags, Scene graphs)
Figure 2: Comparison between image captioning tasks. Our task is form of controllable image captioning for multi-context 360 images. In addition to the image, our task requires query, a word or short phrase to represent the intention of humans, as a signal.
terest to the users. To generate user-controllable image captions from multi-context images, we in- troduced textual scene query to the conventional image captioning task. Figure presents exam- ple of the 360-degree image. textual scene queries, and captions. The image encompasses various con- texts, such as weather and surrounding traffic. is impractical to mention all contexts in single coherent human-readable caption. Therefore we introduce pre-defined scene queries that specify the contexts of the scene to describe. By querying the image, image captioning models can generate concrete captions from multi-context images.
Related Work
2.1 Image Captioning
Research on image captioning has made significant progress since the introduction of the Attention mechanism (Xu et al., 2015) for visual contexts. particular, much attention has been drawn to con- trolling the described contents in captions in recent years. Controllable image captioning allows users to select and prioritize what should be described in the image (Cornia al., 2019). Prior studies have explored methods control the caption gen- eration with information of different modalities in addition to the images, such as POS tags (Desh- pande et al., 2018), sentiments (You et al., 2018), semantic roles (Chen et al., 2021), and abstract scene graphs (Chen et al., 2020a). Dense Caption- ing (Johnson et al., 2016) attaches detailed captions on each object in the conventional images. How- ever, their approach often requires detailed and costly annotations and may overlook the coherent description of the entire scene. We explored visual disieetiiiianiiitiiion for multi-contexts images with QuiC by controlling caption generation with a short textual scene query that is easily interpretable by most users.
To achieve the QulC-360 task, we constructed a human--nnottan dataset' for fine-tuning cap- tioning models (Section 3). We collect 3,940 im- ages taken by 360-degree cameras from Flickr and 18,459 manually annotated captions aligned with queries for our dataset. The captioning models leverage linguistic and visual information by train- ing on our dataset involving different query-caption pairs for the same image.
We conduct experiments to assess our proposal dataset (Section 4). The quantitative results demon- strate that further fine-tuning existing image cap- tioning model with the proposed dataset improves the performance, controllability, and diversity in the QuIC task. Fine-tuned models also have the ro- bustness novel scene queries and out-of-domain images. In addition, the comparison of automatic evaluation metrics and human evaluation reveals that CLIP-based metrics are unsuitable for evaluat- ing the QuIC task. We further investigate how mod- els trained on the proposed dataset can generate captions aligned with given queries by qualitative analysis.
Several studies employ topic modeling typified by LDA and NMF, to automatically extract topics from the existing image-captioning datasets and uti- lize them in their proposed models (Yu et al., 2019; Mao et al., 2018; Al-Qatf et al., 2022). However, these studies rely on the existing image captioning datasets such as MS COCO Captions (Lin et al., 2014), which typically includes limited contexts in each image. We explore the scene captioning prob- lem with images that have multiple contexts, and hence models require extrapolated scene queries to choose image contexts to described. We prepare
Chttps://githhe wtteil...cc//////////// www.///viaaass/euuucc.......

---

the annotated captions following these pre-defined scene queries.
can be important direction for real-world appli- cations.
2.2 Language Conditioned Visual Task
2.3 360 -image Dataset
QuIC, models take an image and short textual query as input. The input differences from exist- ing image captioning methods are summarized in Figure 2. It is also considered that QuIC is similar to the existing visual question answering (VQA). However, as our task is based on image captioning, QuIC models expect more robust queries to scene images and generate more extended captions than those of VQA tasks.
For scene captioning, we search for general domain 360-degree image sets for our annotation. There are several existing 360-degree image sets. However, notice that most of these datasets cover limited domains, such as indoor scenes (Cruz et al., 2021; Chou al., 2020b; Chang et al., 2017: Zioulis al., 2018; Karakottas et al., 2019; Zioulis et al., 2019), room layout sets and outdoor scenes (Sekkat et al.. 2020; Liao et al., 2022), such as driving recordings.
Visual Question Answering. Visual question an- swering (VQA) is a task of answering question given an image (Antol et al., 2015). It is similar to this research in that it focuses on local portions of an image and provides appropriate captions, and research on 360-degree images has also been con- ducted (Chou et al., 2020a; Yun et al., 2021). The answers the VQA datasets are typically short, and VQA models often serve as a selection model from pre-defined answer vocabulary set. Com- pared to the limited answer diversities in VQA, QuIC generates diverse scene captions following human-interppetablee short queries.
Datasets for saliency detection (Zhang et al., 2018;
Hu et al., 2017) and gaze prediction (Lo et al., 2017; Fremerey et al., 2018; Xu et al., 2018) comprise 360-degree videos posted video streaming ser- vice. Mazzola al. (2021) collected 16 outdoor 360-degree videos for moving pedestrian recog- nition. We consider these datasets unsuitable for the general domain image captioning dataset be- cause of the small number of videos. Some others are constructed by the integration or part of the existing image datasets (Wang et al., 201 8; Cirik et al., 2020). We also refer to SUN360" (Xiao et al., 2012), collection of 360-degree images from 360cities and the past de-facto standard 360- degree image dataset. However, SUN360 not publicly available online on June, 2023.
Referring Expression Comprehension. Refer- ring Lxpression Comprehension (REC) is another major branch of langunge--onnditioned visual tasks. In REC, models specify the bounding box of the corresponding object following the textual refer- the given im- age (Kazemzadeh et al., 2014; Plummer et al., 2015; Yu al., 2016; Mao et al., 2016). This is sim- ilar to the visual context disambiguation of QuIC, although the output of REC is an object bound- ing box. not textual captions. Cirik et al. (2020) proposed the referring expression comprehension task on 360-degree images. Nowadays, several joint models of vision-and- languuge tasks, includ- ing VQA and REC, have been proposed (Chen et al., 2020b; Li et al., 2020). Among these mod- els, we utilize the OFA model (Wang et al., 2022) for the joint visual and language input and textual output modeling baseline. Referring expression comprehension videos is also emerging task and has much attention because it essential when we speciy some objects in video cameras (Li et al., 2017: Chen et al., 2019). Recently, Kurita et al. (2023) has proposed first-person vision- iision-based refer- ring expression comprehension of RefEgo, which
3 QuIC-360 Dataset
ring expression as "query"
We introduce the QuIC-360" dataset, which con- tains 3,940 panoramic images annotated with query-focused 18,459 captions. This is the first dataset that uses linguistic information to control the scene description for 360-degree images. We first describe the collection of 360-degree images from the Web and present how to annotate these im- ages. We also present statisties of our QuIC-360 datasct.
3.1 Image Collection
We first collected 360-degree images of the diverse scenes for QuIC-360 and its annotation. Existing 360-degree image sets are often based on limited domains of scenes such indoor (Antol al., 2015; Cruz al., 2021; Chou al., 2020b) or out- door (Liao et al., 2022). Therefore we newly assem- bled 360-degree images of various domains from Flickr, popular community website for photo shar-

---

ing, We downloaded images from group² sharing photos of 360-degree scenery views. On the Flickr website, multiple resolutions are available for some images. We chose the highest-resolution images for downloading. The total number of downloaded images is 12,930. However, these images often take different 360-degree image formats and in- clude duplicated images for the same scenes. To improve the image dataset quality, we further fil- tered the downloaded images with the following steps: (i) To ensure consistency in image format, we utilized 360-degree images only the equirect- angular cylinder method. (ii) To filter similar im- ages captured at the same location, we discarded images posted within an hour by the same user. (iii) To ensure image diversity, we sampled at most 100 images from the same the quality of the collected images, we manually re- move inappropriate or duplicate images, After the filtering steps, we obtained 3,800 images total. Furthermore, we sampled additional 140 im- ages by randomly selecting 20 images about each scene category (c.g., Restaurant and Shop) from Refer360 (Cirik et al., 2020), which are part of the SUN360" (Xiao al., 2012). We treated these images as a separate test set of the different image sources in the experiments. We annotated these images with captions and queries and called Refer-test.
Split
Images
Captions
Vocab.
Avg. Length
Train
3,000
9,438
8,090
20.2
Valid
400
1,251
3,098
20.2
Test
400
6,000
6,043
18.4
Refer-test
140
1,770
2,968
18.8
Total
3,940
18,459
10,862
19.4
Table I: Dataset split and statistics in QulC-360",
Query
Query
Location
1,169
Activity
1.59
People
1.042
Sea
158
Weather
952
Street
146
Building
850
Lake
145
Trees
667
Houses
109
Arthitecture
523
Garden
84
Landscape
492
Happenings
84
Interior
398
Corridor
66
(iv) To further improve
What they doing
342
Paintings
53
Cars
296
Stores
Small objects
272
Trains
41
Furniture
243
Fashion
37
Mountains
223
Animals
35
Plants
197
Traffic
33
Art
195
Foods
24
Rivers
Planes
22
Monuments
161
Accidents
Total
9,438
Table 2: scene queries and number of each query selected in the train split of QuIC-360* dataset.
0.20
MS-COCO
3.2 Caption Annotation
Flickr30k
0.15
QuiC-360 30urs)
In contrast to conventional image caption datasets, we annotated captions based textual scene queries specifying single context from the multi- context 360-degree images. In our task, queries play an important role describing part of the context from the 360-degree scene. When carefully selecting queries, we confirmed that the existing categories used for the image classification task were inappropriate as queries for the collected 360- degree image data. Therefore we pre-defined 34 scene queries suitable for describing 360-degree scenes by manually reviewing the collected images. See Table for the pre-defined 34 scene queries.
Density
0.10
0.00
10
20
30
40
50
Number of words in caption
Figure 3: Distribution of caption length on MS-COCO, Flickr30k, and QuIC-360" datasets.
queries from the pre-defined 34 queries for each image and then wrote down captions related to each query To prevent generating brief captions, we asked MTurk workers to write captions no shorter than eight words. Then, we manually removed un- informative captions (e.g., "There is something on something in this image.") and obtained 18,459 captions.
We employed annotators using a crowd-sourcing service, Amazon Mechanical Turk (MTurk), to ob- tain captions following the pre-defined queries. For our task. each query for some image is expected to be relevant to that image and corresponding con- texts. To this end, annotators chose three different
Phttps: //www. http://ww.llckeee flickr. wwwlicrr.co/gruuuuuei conn/groups//6dderrees
AAppendix A presents details of the annotation website.

---

Model
Fine- tune-
Query
Vocab,
Len.
BLEU@4 METEOR
ROUGE-L
CIDEr
SPICE
X
417
9.9
5.2
8.8
23.4
8.5
6.5
OaAs
395
13.2
10.2
11.3
28.0
14.3
7.2
/
674
13.0
15.7
15.6
34.8
26.9
12.4
X
x
521
10.9
5.9
9.4
24.7
10.1
6.9
OFAga
X
405
13.1
10.0
11.2
28.0
14.2
7.1
V
590
12.0
16.2
15.3
34.7
25.0
11,8
X
X
455
10.7
8.1
10.2
26.1
12.5
8.1
BLIP-2
X
865
14.2
12.8
14.2
32.6
20.4
10.2
(FlanTōxL))
V
796
18.4
13.1
15.8
33.8
23.0
12.3
Table 3:
Comparison of query-based image captioning (QuIC) task performance using test split of the QuIC- 360 dataset. All methods optimize the cross-entropy loss during finetuning.
3.3 Dataset Analysis
split.
Table presents the statistics of the QuIC-360* dataset. In total, our dataset contains 3,940 images and 18,459 captions for their related 34 queries. The vocabulary size of our dataset is 10,862, and the average caption length 19.4, longer than exist- ing image caption datasets such MS-COCO and Flickr30k. We divided the QuIC dataset into train, valid, test, and Refer-test splits. The images in the test/Refirr-tet splits have at least four captions for each query to properly evaluate the caption genera- tion performance.
4.1 Baseline Model for QuIC
To ascertain the effect of additional linguistic in- formation in the form of queries alongside input images, we compared our method to the vanilla image captioning models OFA (Wang al., 2022) and BLIP-2 (Li et al., 2023) as baselines. Both baseline models exhibit high performance in conventional image captioning tasks. We gave these baseline models both an image and modi- fied prompt with scene queries as input to perform the query-based image captioning. However, also noticed that these off-the-shelf models use fixed textual prompts in the image captioning train- ing, independent of the various possible queries. The fixed prompt may cause under-fitting for the QuIC task, which requires the ability to generate captions for diverse queries. We therefore fine- tuned these off-the-shelf models with our modi- fied prompts including scene query. To examine the contribution of the query and fine-tuning, we compared the performance in three distinct scenar- ios, (i) w/o QuIC fine-tuning, (ii) fine-tuning with QuIC-3609 in the conventional image captioning way, and (iii) fine-tuning with QuIC-360 by the modified prompts that include scene queries. We used the cross-entropy loss for the sentence predic- tion, following the conventional image captioning training of OFA and BLIP-2.
Table 2 lists the frequency at which the anno- tators selected each query. The top six queries account for 55% of the total. We assume that these queries are easy to extract the context to describe the scene from the 360-degree images. We con- sider these six queries major and the rest minor. In the experiment, we observe the captions on minor queries generated by model trained with only cap- tions on major queries to validate the robustness to unknown queries (see Sec 4.3).
Figure: depicts the distribution of the cap- tion length for MS-COCE (Lin et al., 2014), Flickr30k (Young et al., 2014), and our QuIC-360 dataset. The distribution of existing datasets peaks at around ten, while about half (48.5% (%))ll) of the cap- tions in QuIC-360 have over 20 words. This in- dicates that the captions of our dataset are longer than existing sets and have detailed information about images compared existing datasets.
4.2 Metrics
We measured the similarity to human Coimtn-written cap- tions using BLEU (Papineni et al., 2002), ME- TEOR (Denkowski and Lavie, 2014), ROUGE (Lin, 2004). CIDEr (Vedantam et al., 2014), and SPICE (Anderson al., 2016). In addition, we used CLIPScore and RefCLIPScore (Hessel et al., 2021) metrics, which have been found to have the
4 Experiment
We validate that the model learning with the QuiC- 360 dataset surely enhances the controllability of scene captioning with baseline models. We use 3,000/400/400 images and 9,438/1,22 1438/1,251/66000 corre- sponding captionn pairs the train/valid/tes

---

0.25
Method
Fine-tuning
Query
Relevance
Difference CIDtr
0.0
Human
3.87
0.15
x
X
3,40
OFAmue
x
3,40
3.70 v
0.05
0.00
BIJI--2
x
x
3,52
X
3.64
(FlanT55xL)
V
3.69
anim
what they are Soing
pannes
peogle
cars
wealthe chet
houp ce
seruung mom monteeeee
traffic
sttres
local
happanir aarden
inteero mounte mounten
corratt
foodd
trains
Table 4: Human evaluation of model outputs for image- query pairs sampled from the test split of the QuIC-360".
small
Model
Fine-tuning
Query
CLIP-S
Re(CLIP-S
Figure 4; The difference of performance (CIDEr) in QuiC task per query between OFAbase fine-tuned with/without queries.
x
0.710
0.729
OFAhusn
x
0.661
0.707
/
0.646
0.725
0.733
0,730
OFAtarge
x 0.654
0,701
highest correlation with human assessments in con- ventional image captioning tasks. These metrics evaluate the similarity between the holistic context of an input image and given caption using the CLIP model (Radford et al., 2021). Since the pro- posed QuIC task must focus on the local contexts of the image, also confirmed whether these CLIP- based metrics are suitable for evaluating the QuIC task by using human agreement. Furthermore, compared the average length and the vocabulary size of generated captions to examine whether a query enhances the diversity of captions (Wang and Chan, 2019).
0.648
0.729
x 0.734
0.747
BLIP-2
X
0.661
0.733
(Flan'155x))
0.671
0.739
Table Comparison of the query-based image cap- tioning (QuIC) task performance by CLIPScore and RefCLIPScore.
tioning models to select the contexts of the images and improves performance on the QuiC task. We further validated whether the proposed models can generate diverse captions with qualitative analysis, detailed in Sec. 4.4.
4.3 Quantitative Analysis
Performance Improvement per Query. We ex- amined the performance difference per query to determine which queries benefit from fine-tuning in the QuIC task. Figure 4 depicts the difference in CIDEr scores for each query on the test between the model fine-tuned on the query-bascd image cap- tioning and the conventional image captioning task. The result indicates a clear trend of improvements in most queries. We observe that queries with sig- nificant performance improvements provide cap- tions not generated by standard image captioning. Although some queries showed performance drops, assume this is due the limited training data corresponding these queries. This can be solved by collecting more diverse query-caption pairs on 360-degree images.
We first evaluated the effect of introducing queries for 360-degree images. Then conducted hu- man evaluation to test whether the models generate captions relevant the queries. Finally we ana- lyzed our model with out-of-domain settings and the effect of tuning specifically for image caption- ing (i.e., CIDEr optimization).
Effect of Fine-tuning with Query. Table presents the results of query-based image caption- ing tasks on the test split of the QuIC-360 dataset. Models trained on our QuIC-360 dataset outper- formed models that were pre-trained or fine-tuned in the conventional image captioning training on all metries and successfully generated captions closer human annotations. In addition, our fine-tuning method with queries significantly increased vocab- ulary size and the average length of captions com- pared to the pre- trained model. These results in- dicate that the introduction of queries helps cap-
Human Evaluation. We conducted human eval- uation using MTurk assess whether the outputs of models were relevant to given queries. First, we randomly selected 150 image-query pairs from the

---

Training Data
#
Len.
B@4
M
R-L
C
Split
|Opt.
# #V
Len.
[@4
M
S
0
399
9.8
5.7
9.0
24.4
10.2
6.6
x
674
13.0
15.7
15.6
34.8
26.9
12.4
test
major
537
13.0
14.8
14.6
33.7
25.6
11.1
/
749
17.7
15.0
17.5
35.3
30.5
13.3
major- minor
|558
13.8
15.2
15.7
35.3
28.3
11.7
Refer-
x
418
14,0
14.1
15.2
32,9
22.9
12.5
test
/
457
17.9
12.6
16.5
33,1
26.2
13.5
Table Comparison of the query-based image caption- ing (QuIC) task performance on novel minor queries with different training set. The column colored in gray indicates the training contains captions the minor queries in the test set.
Table 8: Comparison of the query-based image caption- ing (QuIC) task performance of OFA with/without CIDEr optimization.
treated the six largest queries as the major queries and the rest as the minor queries. this experi- ment, we trained the OFAbase model on the QuIC task with captions on major queries and evaluated it with those minor queries.
Ft.
Q
#V
Len.
@@4
M
R-L
C
S
X
X
206
10.4
4.3
8,5
22.7
7.8
6.8
X
236
14.0
7.5
11.2
27.0
12.9
8.1
418
14.0
14.1
15.2
32.9
22.9
12.5
Table 7: Comparison of the query-based image cap- tioning (QuIC) task performance on the out-of-domain images using Refer-test split.
Table 6 reports the result of the query-based im- age captioning performance the minor queries in the QuIC-360" dataset with different training strategies. Although the model trained only with major queries performed slightly worse than the in-domain setting, it significantly outperformed the other model in all evaluation metrics without fine-tuning. The increased vocabulary and aver- age length indicate that our model generates more diverse and controllable captions. These results imply that training the QuIC task even with limited number of queries enables the models select the contexts from novel queries.
test split and generated captions corresponding to the image-query pairs by the image captioning mod- els. Then, the workers rated the generated captions by a five-point Likert scale on how relevant these captions were to the given image-query pair. Ta- ble 4 presents the human evaluation of each model output with or without fine-tuning tuning or query. The best results for both OFA and BLIP-2 models are obtained when fine-tuned with query, indicating that the proposed query-bases image captioning surely select the context aligned with query from the image.
Adaptability to Out-of-domain Images. To eval- uate our models' adaptation capacity out-of- domain images, we utilized the Refer-test split, listed in Table 1. While test images of the QuIC- 360 are collected from Flickr, images of the Refer- test (from Refer360%) are collected from 360cities. Table 7 compares the query-based caption genera- tion performance the Refer-test split. Despite the images being collected from different image sources, fine-tuning with the QuIC-360 on the QuIC task improved the performance in the Refer- test split compared to other strategies, with no fine- tuning and with fine-tuning on the standard im- age captioning task. In addition, the introduction of queries increased the vocabulary size and av- erage length of generated captions. As with the in-domain results in Table 3, query-based im- age captioning framework allows models to gener- ate more diverse and accurate captions for out-of- domain images.
Evaluation with CLiP-based scores. Table 5 displays the query-based image captioning perfor- mance with the CLIPScore and Re/CLIPScore. We found that both the models without fine-tuning or a query exhibited the highest values on the CL.PP. based scores, while human and automatic evalua- tions exhibited the lowest performance as presented in Tables 3 and 4. This suggests that the CLIP- based scores are correlated well with humans in the standard image captioning task but not in the proposed task. Thus these metrics seem to be un- suitable for evaluating the QuIC task. We did not evaluate the performance with CLiP-based metries in the subsequent experiments.
Adaptability to Novel Queries. Preparing queries that cover possible 360-degree images is challeng- ing requirement. We assessed the adaptability of our model novel queries, which are not included in the training data. As introduced in See. 3.3, we sorted the queries by the number of captions and
Empirically, optimizing the image captioning model using the test metric (i.e., CIDEr optimization) signif-
Effect of CIDEr Optimization.

---

Building
People
Landscape
Weather
Small objects
Weather
wyo guery
cloudy gray
cloudy
lange fallen middiie of the forest
lange fallen muddle of forest
Three people sinting on middle of
people sitting rocks middle the forest
and the sky goay
Wi quary Ooua
brilding
woman standing her hand.
landscape forest
jrreen
chining: through
siiting rocks forest
sunny bright
of
Cars
Landscape
Architecture
Rivers
Lake
Trains
veto query
The skyy is blue with a lot of clouds in the sky
The skyy is bloe with a Jot of clouds in the sky
The sky blue with white The sky blue with whine clouds and few clouds in clouds and few clouds in the sky
small pand with u lat of july pads on the surfhce of the water
ammall pood with lot of hiy pads on surface of the water
the sky
My
quen (Our)
black truck is parked in the grass next to green track
green grassy fildd with trees in the background
The beisdge is of wood has metal Erame
small river running through middlle
small lake filled with lily pads in the middle of gransy Gilld
through the woods on the
A red traveling
bridge
Figure 5: Qualitative examples. We colored examples that are relevant to the queries.
icantly improves the image captioning task's per- formance (Rennie et al., 2017). Table presents the performance difference between models based on whether CIDEr optimization was conducted. Both models are trained on the QuIC task with our dataset. For our new task, the model tuned with CIDEr obtained higher values of automatic evalua- tion metrics for both test/Refer-test sets. It is also observed that models generated longer captions due to the CIDEr optimization. Thus the decrease in BLEU can be attributed to this. These findings confirm the usefulness of tuning models with test metrics for better performance the QuIC task.
main content of the images. contrast, the model tuned with image-caption-queyy triplets produced diverse captions based on the given query.
Although the models trained on the QuIC task can produce correct captions from 360-degree im- ages in most cases, it sometimes generates incorrect captions. As depicted in the top-right and bottom- right examples, our model generated captions for non-exist objects related to a given scene query. For example, "camera" and "red train" not actu- ally exist in the top-right and bottom-right images, but our model wrongly gave descriptions including these objects following given queries. In future work, it is essential to address such hallucinations to ensure generating captions that correspond ap- propriately the image-query pairs.
4.4 Qualitative Analysis
We compared the captions from models fine-tuned with queries to those from models fine-tuned with- out queries to clarify how the generated captions are diverse and consistent with the given query. Fig- ure 5 illustrates examples of the based image caption generation". The models trained without queries generated the same sentences regardless of the query provided, indicating that the vision-and- language model tuned with image-caption pairs only relied on image contents and described the
Conclusion
We have proposed a query-based image caption- ing for 360 images and constructed the QuIC- 360 dataset for our new task, containing human- annotated captions focused queries. In QuIC, models leverage textual queries to describe images with multiple visual contexts from 360-degree im- ages. Experimental results show that further fine- tuning existing image captioning models with our dataset can improve the controllability and diver-
'Wo have attached other examples in Figure 9 in Ap- pendix C.

---

sity of captions on 360-degree images. In addition, models trained on our dataset are robust to both un- known queries and out-of-domain images to some extent. Further investigation also reveals that CLIP- based metrics are unsuitable for the evaluation of our task, highlighting the target difference between QuIC and the conventional image captioning task.
retrieval-basee topics re- re-weighting. IEEE Transac- tions on Multimedia, pages 1-16.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic Proposi- tional Image Caption Evaluation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 182-398. Springer.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, VQA: Visual Question An- swering. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2425-2433.
Limitations
and Devi Parikh. 2015.
Images. Our proposed dataset consists of images collected by web crawling. Although we gathered diverse 360-degree images set including indoor and outdoor scenes, it still has a limitation in the coverage of the real world-scenes.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Mat- terport3d: Learning from rgb-d data in indoor envi- ronments. 2017 International Conference on 3D Vision (3DV), pages 667-676.
Annotations. We employed MTurk workers to obtain the captions for our QuIC-360" dataset. We provided clear instructions and examples for the workers and asked them to remove inappropriate or harmful captions. We also manually supervised assure this. However, do not ensure that all biases are completely removed.
Long Chen, Zhihong Jiang, Jun Xiao, and Wei Liu. 2021. Human controllable image captioning with verb- specific semantic roles. In 2021 IEEF/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 16841-16855
Others. As mentioned in the qualitative analysis, the annotators tend to select queries from the pho- tographed contents when they assign captions to images. Thus the models trained on our dataset might give wrong descriptions in the QuIC task when the image does not contain the object related to the given queries.
Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. 2020a. Say as you wish: Fine-grained control of image cap- tion generation with abstract scene graphs. In 2020 IEFF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 9959-99959999 9968.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. Uniter: Universal image-text representation learning. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), page 104-120 Berlin, Heidelberg. Springer-Verlay
Ethics Statement
We constructed this dataset based on the publicly available images posted on Flickr. We distribute the URL list to the Flickr images instead of the col- lected images themselves. Some of the images con- tain people, but we carefully removed captions that can be used to identify them. When used properly, our image and annotation dataset is beneficial as many other image datasets such as MS-COCO (Lin et al., 2014) and Flickr30K (Young et al., 2014; Plummer et al., 2015).
Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Yee Kenneth Wong. 2019. Weakly-Supervised Grounding Natural Sentence in Video. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Shih-Han Chou, Wei-Lun Chao, Wei-Sheng Lai, Min Sun, and Ming- Hsuan Yang. 2020a. Visual ques- tion answering on 360 images. In 2020 IEEE Win- ter Conference Applications of Computer Vision (WACV), pages 1596-1605.
Acknowledgements
Shih-Han Chou, Cheng Sun, Wen- Yen Chang, Wan- Ting Hsu, Min Sun, and Jianlong Fu. 2020b. 360- indoor: Towards learning real world objects in 360 indoor equirectangular images. In 2020 IEFF Win- ter Conference on Applications of Computer Vision (WACV), pages 834-842.
This work was supported by JSPS KAKENHI Grant Number JP22K17983, and by JST PRESTO
Grant Number JPMJPR20C2.
Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson,
References
Majeed Al-Qatf, Xingfu Wang, Ammar Hawbani, Amr Abdusallam, and Saeed Hammod Alsamhi. 2022. Image captioning with novel topics guidance and

---

Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Sealing instruction-finttmee language models.
Computer Vision and Pattern Recognition (CVPR), pages 1396-1405.
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2016. Densecap: Fully convolutional localization networks for dense captioning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4575-4574.
Volkan Cirik, Taylor Berg- Kinkparrka,,r,,, and Louis- Philippe Morency. 2020. Refer3609: A referring expression recognition dataset in 360 images. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 7189- 7202, Online. Association for Computational Lin- guistics.
Antonios Karakottas, Nikolaos Zioulis, Stamatis Sama- ras, Dimitrios Atalogiou. Vasilcios Gkitsas, Dimitrios Zarpalas, and Petros Daras. 2019. 360 surface re- gression with a hyper-sphere loss. In 2019 Interna- tional Conference on 3D Vision (3DV), pages 258- 268.
Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2019. Show, control and tell: A framework for gen- erating controllable and grounded captions. In 2019 IEEE/CVF Conference on Computer Vision and Put- tern Recognition (CVPR), pages -299-
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014, ReferltGame: Referring to objects in photographs of natural scenes. In Proceed- ings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787- 798, Doha, Qatar. Association for Computational Linguistics.
Steve Cruz, Will Hutchcroft, Yuguang Li, Naji Khosra- van, Ivaylo Boyadzhiev, and Sing Bing Kang. 2021. Zillow indoor dataset: Annotated floor plans with 360 panoramas and 3d room layouts. In 2021 IEEE/CVF Conference on Computer Vision and Pat- lern Recognition (CVPR), pages 2133-2143.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao. Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment anything. arXiv:2304.02643.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Shuhei Kurita, Naoki Katsura, and Eri Onami. 2023, Refego: Referring expression comprehension dataset from first-person perception of ego4d. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15214-15224.
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexan- der G. Schwing. and David A. Forsyth. 2018. Di- verse and controllable image captioning with part-of- speech guidance. CoRR, abs/1805.12589.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-ing -imaag pre- training with frozen image encoders and large lan- guage models.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. 2022. Eva: Exploring the limits of masked visual representation learning at scale.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang. Houdong Hu, Li Dong. Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre-training for vision-langusel tasks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 121-137, Cham. Springer International Pub- lishing.
Stephan Fremerey, Ashutosh Singla, Kay Meseberg, and Alexander Raake. 2018. Avtrack360: An open dataset and software recording people's 's head rota- tions watching 360 videos on an hmd. In Proceed- ings of the 9th ACM Multimedia Systems Conference, MMSys 18, page 403-408, New York, NY, USA. Association for Computing Machinery.
Zhenyang Li, Ran Tao, Efstratios Gavves, Cees G. M. Snoek, and Arnold W. M. Smeulders. 2017. Tracking by Natural Language Specification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jack Hessel, Ari Holtzaan, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Yiyi Liao, Jun Xie, and Andreas Geiger. 2022. Kitti- 360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions Pattern Analysis and Machine Intelligence.
Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, and Min Sun. 2017. Deep 360 pilot: Learning a deep agent for piloting through 360 sports videos. In 2017 IEEE Conference on
Chin-Yeu Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

---

Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco:
Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179-1195.
Common objects in context. In Computer Viston
ECCV 2014, pages 740-755, Cham. Springer Inter- national Publishing.
Ahmed Rida Sekkat, Yohan Dupuis, Pascal Vasseur. and Paul Honeine. 2020. The omniscape dataset. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1603-1608.
Wen-Chih Lo, Ching-Ling Fan, Jean Lee, Chun- Ying Huang, Kuan-Ta Ta Chen, and Cheng-Hsin Hsu, 2017. 360 video viewing dataset in moad-mouteed virtual reality. In Proceedings of the 8th ACM Multime- dia Systems Conference, MMSys' 17. page 21 211-216, New York, NY, USA. Association for Computing Machinery.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach. Roberto Martin-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, Silvio Savarese. Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. 2022. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Proceedings of the Sth Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 477-490. PMLR.
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via visual sentinel for image captioning. In 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3242-3250.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous ob- ject descriptions. In 2016 IEEE Conference on Com- puter Vision and Puttern Recognition (CVPR), pages 11-20.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2014. Cider: Consensus-based image de- scription evaluation. CoRR, abs/ 141 1,5726.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural im- age caption generator. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156-3164.
Yuzhao Mao, Chang Zhou, Xiaojie Wang, and Ruifan Li. 2018. Show and tell more: Topiceoriented multi- sentence image captioning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4258-4264. International Joint Conferences on Artificial Intelli- gence Organization.
Fu-En Wang, Hou-Ning Hu, Hsien-Tzu Cheng, Juan- Ting Lin, Shang-Ta Yang, Meng-Li Shih, Hung-Kuo Chu, and Min Sun. 2018 Self-supervieed learning of depth and camera motion from 360 videos. In Asian Conference Computer Vision, pages 53-68. Springer.
Giuseppe Mazzola, Liliana Lo Presti, Edoardo Ardiz- zone, and Marco La Cascia. 2021. A dataset of anno- tated omnidirectional videos for distancing applica- tions. Journal of Imaging, 7(8).
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: Unifying ar- chitectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Pro- ceedings of the 39th International Conference on Machine Learning (ICML), pages 23318-23340
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Qingzhong Wang and Antoni B. Chan. 2019. Describ- ing like humans: On diversity in image captioning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4190-4198.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockemmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2641- 2649.
Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shan- shan Zhao. Ying Shan, and Feng Zheng. 2023. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint arXiv:230..02677.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, International Con- ference on Machine Learning, pages 8748-8763.
Jianxiong Xiao, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. 2012. Recognizing scene view- point using panoramic place representation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 2695- 2702.

---

Kelvin Xu. Jimmy Ba, Ryan Kiros, Kyunghyun Cho. Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pages 2048-2057, Lille, France. PMLR.
Yanyu Xu, Yanbing Dong, Junru Wu, Zhengzhong Sun, Zhiru Shi, Jingyi Yu, and Shenghua Gao. 2018. Gaze prediction in dynamic 360 immersive videos. In 2018 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5333-5342.
Quanzeng You, Hailin Jin, and Jiebo Luo. 2018. Image captioning at will: versatile scheme effectively injecting sentiments into image descriptions. CoRR, abs/1 101.10111...
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Licheng Yu, Patrick Poirson, Shan Yang. Alexander Berg, and Tamara L. Berg. 2016. Modeling context in referring expressions. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 69-85. Springer International Publishing.
Niange Yu, Xiaolin Hu, Binheng Song, Jian Yang, and Jianwei Zhang. 2019. Topic-oriented image caption- ing based on order-embedding. IEEE Transactions on Image Processing. 28(6):7733-2744
Heeseung Yun, Youngjae Yu, Wonsuk Yang. Kangil Lee, and Gunhee Kim. 2021. Pano-avqa: Grounded audio- visual question answering on 360 vidcos. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 201 1-2021.
Ziheng Zhang, Yanyu Xu, Jingyi Yu, and Shenghua Gao. 2018. Saliency detection in 360 videos. In Proceedings of the European conference computer vision (ECCV), pages 504-520, Cham. Springer In- ternational Publishing.
Nikolaos Zioulis, Antonis Karakottas, Dimitrios
Zarpalas, Federico Alvarez, and Petros Daras. 2019, Spherical view synthesis for self-supervised 360 depth estimation. In 2019 International Conference on 3D Vision (3DV), pages 690-699.
Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, and Petros Daras. 2018. Omnidepth: Dense depth estimation for indoors spherical panora- mas. In Proceedings of the European Conference on Computer Vision (ECCV), pages 453-471, Cham. Springer International Publishing.

---

Appendix
A MTurk Annotation Details
We employed workers using Amazon Mechanical Turk to collect annotations for the 360-degree images. To ensure the quality, we hired those who resided in the U.S. and whose recent assignment approval rate greater than 95%. Each annotation time was measured by a pilot task, and paid rewards for the average time worker at the rate of $12 per hour. We have attached snapshot of the annotation website in Figure
By draqzing the photo, you can view various directions of 360 degree pholo.
Theme
Themed Description
Required
Aequredl
freenees
scene precitely everyone imagite image
Themet
Descripion theme (ecuirec)
Themme
Dastripiin thme ((equrac))
Descrpiee plen theme (ecurrec)
(Optional) When you consider this image is inappropriate for themed description
Fill in the descriptions even if you check here juoljusl wriling pisceholdersj Otherwise, your HIT may be rejected descrittion (e.g. nsturel
image eppropriale
Figure 6: An screenshot of annotation in MTurk. The workers select three queries related to the presented images and give descriptions of each query.
Model Architecture
OFA. Figure 7 shows the illustration of our OFA architecture. OFA employs Transformer- encoder- decoder architecture, which unifies various vision and language tasks such image captioning, visual grounding, and visual question answering in single architecture. In the image captioning task, OFA uses fixed prompts with images as textual input for the conventional image captioning task. Taking advantage of the prompt's 's characteristics, we modified OFA be suitable for the QuIC task. We modified the original prompt to include the given scene query and fed it to the model to generate context-dependent captions.

---

Caption
Caption sky appears be cold and foggy.
The sky appears to be cold and Toggy:
LLM
Fully Connected
Transformer Encader
Transformer Demader
Image
Tuken Embedding
Image Encoder
Q-Former
Writee description on weather for thas Image.
Encoder
Prompt
What does image
describe weather?
Weather
Weather
Queries
360 Image
(Trainable Parameter)
Query
Image
Query
Figure 7: OFA architecture for the QuIC task.
Figure 8: BLIP-2 architecture for the QuIC task.
Cars
Landscape
Architecture
Rivers
Landscape
People
rocky waterfall and few rocks in the middle of
group people sirting They
parking
including palm eypreses
building
Activity
Trees
Cars
Weather
Interior
What doing
group people doing bicycles in park.
of trees background lmnagn.
There are cars parked the street.
cloudy, slight chill
The interier trinn statio has tot windows and glass onitnng
group people standing
wailing to
Figure 9: Additional qualitative examples generated by the models trained on our QulC-360 dataset.
BLIP-2. Figure 8 delineates the architecture of BLIP-2. BLIP-2 integrates Querying Transformer (Q- (Q-former) to bridge the gap between frozen LLM and pre-trained image encoder. Q-Former contains transformer submodules for visual feature extraction and for processing textual input, which share the same self-attention layers. In the experiments, we used ViT-g/14 from EVA-CLIP (Fang al., 2022) for the image encoder, and Flan T5x1. (Chung al., 2022) for encoder-decoder based LLMss. We added the original prompt containing the scene query to the model.
Additional Examples
Figure 9 illustrates additional examples produced by our method using the QuIC-360' We can observe that most outputs describe detailed information in response to the queries. The captioning model correctly recognized the materials of the ceiling and the fact that people bottom right example. On the other hand, they fail mention in detail when they do not acquire enough information from images. In the bottom left example, the description of trees gives less information about images. The model tends to generate such simple captions when the size of the object related to the query is small, as depicted in the example.
waiting in the station in the

---

mrddlelocat tion
stanning
sun
Lnstod outa de
ailt peop
wear
Mongh
one
da
day
Wa
wwindww OW
mat
park
La
lalking
oOU
eet
main arear
a one
oigg
water
peeple
ight
PORR Sunny
ualGing
AMssma
M
road
O
& orpup
biing
SOVA Shaming
S
Side
brewn dbkaLazs
whire
well Sieee
inside
mountain
a
around
S Sitting
son
around
weather
I
blue
bat sma mimall Ted
Ligh perauee
"chursh yol
many city
dewo
Location
People
Weather
Building
wearing anggarr ar
tandi ng ture
sesen
person oon
pund
Side ecedd
Seen
F1oor
graffit Wa
larg gE
twork
Whi Lite THO Deop e theiih
baaeedbrr bedbrie onany
street
wa
tón
walking
Sttakings gi tting
Meingsiten tur
gre
baldinn zone
bui Iding
sma
Cellirg
Ror
sma I SIS vellor
Lane
S Side Say
Du ding dhyey
empty
mader
people
ReOD Le
Ii traf
Oadlight
Light
side wā ter Boyinng
wa Fr
sign
Rade ADII
parkeddusst stonsttstttssssooo
area da the
What they are doing
Small objects
Art
Street
Figure 10: Visualization of word distributions of our QuIC-360 dataset by query.
Word Distribution
To clarify the context that the queries represent, we visualized word distributions for the four queries each major and minor query in Figure 10. We can observe that each query contains the word itself and various words describing properties actions. Interestingly, the caption for the query "Location" contains words related to other queries, such as "Building", "People", and "Trees". These queries are required to select contexts from multiple choices and describe the most salient objects, which are more complicated.
E Leveraging Non-language Data
We also tested method to control the caption generation by once specifying the region focus with query and then using the obtained area as an aid. Specifically, we employed OF'A for the reference expression comprehension task, and combined with the Caption Anything Model (Wang al., 2023y3, which internally uses the Segment Anything Model (Kirillov et al., 2023). First, OFA model fine-tuned with RefCOCO Dataset predicts the bounding box from a given scene query and image. Then, pre-trained CA model generates captions based on the bounding box and images. We performed the same experiment on this pipeline, as presented in Table 3. However, CIDEr and CLIPScore were 4.5 and 0.670, respectively, the worst result despite using the state-of-the-art models.
'We used sam sit h for the segmenter, and Salestorcehhlip2--p--..7 for the captioner as is.


Reviews:
[{"review_id": "Sud2T6DZr2", "reviewer": "Reviewer_jdnx", "summary": "", "questions": "Why do you feel that the model proposed and the approach taken controlled image captioning is motivated and well suited to the particular challenges of the 360° image says you are working with?\nI do not claim to be an expert or have experience working with image captioning for 360° image data sets, but a question I would have is would it not be more relevant to generate captions that actually acknowledge and take advantage of the special properties of the 360° image. Specifically would it not be useful to generate captions that are aware of the special arrangement relative to the centre of the 360° image assuming it to be the front? Similar to this the extreme left and right of the image would assumed to be the back, and associated with this would be challenges around generating images relative to stitch together content. It strikes me that this would be a task that will be more natural for the 360° dataset, rather than just focusing on standard issues around lexical controllability.\nCan you give more information on the Refer360 dataset since it's an important part of the test approach? \nYou mentioned that refer-test is a distinct statuses that was based on the Refer360 dataset, but then later you say that refer-test is built from a split of the main dataset. Can you provide clarification on exactly what this data subset is built from? \n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Focus on the particular challenges with 360° images\n\nOur task is scene captioning with 360-degree images and therefore we concentrate on the comprehension of scenes through images with textual scenery topics, not solving the 360-degree visionary issues such as distorted coordinates.  360-degree images used in this dataset contain richer contexts than the conventional images in existing image captioning datasets, and our study proposes a new image captioning task for scenes addressing these multiple contexts of 360° images. Furthermore, the experiment presented in Table 3 demonstrates that the query-based captioning method is more effective for tackling this challenge compared to the methods for the conventional image captioning task.\nWe agree that it is also an interesting direction to leverage other properties of 360° images for future research. Our proposed 360° query-based image captioning dataset will offer a first step in this direction.\n\n> Clarification on Refer360 dataset and our refer-test split\n\nThe refer-test split is a subset of QuIC-360° and contains images with a different source from the main dataset. We collected 3,800 images by crawling Flickr for the main dataset and  selected 140 images from the Refer-360 dataset for the refer-test split. For both dataset, we newly annotated captions via MTurk. While we mentioned in lines 466-468 that the images in the Refer-test split differ from the test split of the main dataset, this explanation might have been insufficient. We will add a clarification in our camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed new task domain along with the associated data set is a valuable contribution for the controlled image caption and community. Also a systematic investigation of a particular architecture for controlled caption generation based on context specified as simple words and phrases also is part of a trend that has been seen in the community lately, but is none the less valuable and of benefit to the community. \nThe analysis itself on the data and the model is high-quality and worthy of publication.\nThe related work section is high quality and covers areas directly and indirectly related to the current publication.\n", "reasons_to_reject": "My biggest problem with this paper is that it isn't clear whether the paper is really trying to propose a new task and associated data sets or a particular model for controlled image caption generation. I find that the model used based on individual context words for controlling the airport could be just as applicable to any other form of data and is in no way tied to the particular challenges associated with 360° images. Therefore the arguments made about the links between 360° datasets and this particular form of controllable image captioning do not hold up well. I would prefer to see a task that is more focused on the particular challenges with describing the 360° days assess rather than one that's broader in terms of specifying basic contacts that would be as applicable to any other image type.\nIn the Experiment section, the autbors state that {We validate that the model learning with the QuIC- 360◦ dataset surely enhances the controllability of scene captioning with baseline models.\". This is surely self evident, i.e. that the use of a new day success with a particular property will provide better overall results when a testate assess with that same property is then used for the evaluation. ", "questions_to_authors": "Why do you feel that the model proposed and the approach taken controlled image captioning is motivated and well suited to the particular challenges of the 360° image says you are working with?\nI do not claim to be an expert or have experience working with image captioning for 360° image data sets, but a question I would have is would it not be more relevant to generate captions that actually acknowledge and take advantage of the special properties of the 360° image. Specifically would it not be useful to generate captions that are aware of the special arrangement relative to the centre of the 360° image assuming it to be the front? Similar to this the extreme left and right of the image would assumed to be the back, and associated with this would be challenges around generating images relative to stitch together content. It strikes me that this would be a task that will be more natural for the 360° dataset, rather than just focusing on standard issues around lexical controllability.\nCan you give more information on the Refer360 dataset since it's an important part of the test approach? \nYou mentioned that refer-test is a distinct statuses that was based on the Refer360 dataset, but then later you say that refer-test is built from a split of the main dataset. Can you provide clarification on exactly what this data subset is built from? \n\n\n", "paper_topic_and_main_contributions": "In this paper the authors present a dataset/task that is focused on the generation of captions for images coming from 360° cameras. The technical novelty of the work is focused on the use of context words to drive to controlled caption generation process. The task is called Query-based Image Cap- tioning (QuIC) for 360-degree images. Usefully the work contributes a new novel dataset that comprises 3,940 360-degree images and 18,459 pairs of queries and captions annotated manually. The queryset included 34 items that were put together after the dataset was reviewed. Almost all queries were categories of items, however one query was the noun phrase what are they doing.\n\nThe authors applied their datasets to OFA and BLIP-2 both in terms of vanilla uses of these models, but also with variants that were fine tuned to the QuIC dataset in the normal way, and also variants that used the OFA and BLIP-2 architectures with the textual prompts. \n\nThe results do demonstrate that finetuning an architecture that is primed with the context word does outperform OFA and BLIP that have not received such fine tuning. Given the very different nature of the dataset, this is not surprising -- I would have been surprised if that was not the case. The evaluation is however high quality and will be useful for others to build upon.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tr4R8bPDdY", "reviewer": "Reviewer_4vda", "summary": "", "questions": "Can this dataset(QuIC), like SA[0], use \"points\" or bounding boxes as cues to generate corresponding captions?\n\n[0] Kirillov A, Mintun E, Ravi N, et al. Segment anything[J]. arXiv preprint arXiv:2304.02643, 2023.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Limited number of models\n\nWe prioritized the ablation studies over the validations of numerous models. We emphasize that  OFA and BLIP-2 were two state-of-the-art image captioning methods before submission. Furthermore, the effectiveness of query-based image captioning was verified using these two methods. The automatic evaluation metrics and the human evaluation in Table 3 and Table 4 proved that the introduction of queries helps captioning models to select the contexts from the images. \n\n> Lack mismatched cases\n\nAlthough the absence of the topic from images is interesting, discriminating the irrelevant topics  are well-known as notoriously difficult similar to the “no-answer” or “unanswerable” instances of the previous visual question answering dataset (e.g. Ray et al. [2]) and reading comprehension dataset of SQuAD v2. We hence consider it is an advanced ability for the current models to deal with mismatched cases and hence make them out-of-scope of the current dataset, concentrating on the annotations where topiced contexts surely exist in 360-degree images. We also believe we can develop the augmented pseudo mismatched cases with random topics and images with our annotations in the future work.\n\n> The impact of pre-training on this dataset for other tasks\n\nWe designed QuIC-360° explicitly for fine-tuning models on the QuIC task. Using this dataset as a pre-training for other tasks is clearly outside the scope of this paper, so we do not think it is a reason to reject. We appreciate the reviewer's suggestion, but it would be helpful if you could tell us which task we should use it for.\n\n> Can this dataset(QuIC), like SA[0], use \"points\" or bounding boxes …\n\nIn the problem setup of the query-based image captioning task, we take an image and a scene query as input and generate a caption corresponding to the query. So we cannot use any points or bounding boxes directly as a cue. Instead, it is possible to predict the bounding box from the scene query and image, and indirectly generate a caption based on the detected bounding box.\n\nMotivated by your insight, we implemented an additional query-based image captioning method using the state-of-the-art referring expression comprehension model of OFA and a Caption Anything [1] that internally uses SA [0]. First, the OFA (fine-tune with RefCOCO) predicts a bounding box from the scene query and image. Subsequently, the Caption Anything generates captions based on this predicted bounding box.\n\nWe conducted additional experimental using this pipeline method and summarize the evaluation results as follows:\n\n[Setup]\n- Visual Grounding Model: OFA Large (fine-tuned with RefCOCO)\n- Caption Anything:\n  - Segmenter: sam_vit_h_4b8939.pth, \n  - Captioner: Salesforce/blip2-opt-2.7b\n\n[Metrics]\n- BLEU-4: 0.0246\n- METEOR: 0.0599\n- ROUGE: 0.2020\n- CIDEr: 0.0451\n- SPICE: 0.0347\n- CLIPScore: 0.6710\n\nWe evaluated the SoTA model with  SA[0], but its performance was inferior to our methods. We will add this experiment in the appendices in the camera-ready version. \n\nWe appreciate the constructive feedback and hope our clarifications address the concerns. \n\n[1] Caption Anything: Interactive Image Description with Diverse Multimodal Controls (Wang et al., 2023)\n\n[2] Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions (Ray et al., 2016)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The QuIC dataset has been proposed, which to some extent, enhances the Image Captioning scene in the field of panoramic vision.\n\n2.The paper conducted comprehensive testing and evaluation of the dataset from various perspectives, and presented the results of human evaluations.", "reasons_to_reject": "1.The experimentation involved a limited number of models. For a new dataset, it would be beneficial to conduct evaluations on a more diverse set of architectures.\n\n2.The dataset lacks certain mismatched cases, such as when the object corresponding to the Query is absent in the image, resulting in a caption like \"Nothing.\" Including such cases would add valuable real-world scenarios to the dataset.\n\n3.The impact of pre-training on this dataset for other Image Captioning tasks remains unknown. Further investigation is required to assess its potential benefits for related tasks.", "questions_to_authors": "Can this dataset(QuIC), like SA[0], use \"points\" or bounding boxes as cues to generate corresponding captions?\n\n[0] Kirillov A, Mintun E, Ravi N, et al. Segment anything[J]. arXiv preprint arXiv:2304.02643, 2023.", "paper_topic_and_main_contributions": "1. A new Image Captioning task and dataset, named Query-based Image Captioning for 360◦ images (QuIC-360◦), has been introduced.\n2. They further investigate how models trained on the proposed dataset can generate captions aligned with given queries by qualitative analysis.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZvMqpRmF8j", "reviewer": "Reviewer_WuwK", "summary": "", "questions": "A. Many of the scene queries do not seem mutually exclusive: e.g. \"cars\" and \"traffic\", \"activity\" and \"what they are doing\", \"interior\" and \"furniture\", \"art\" and \"paintings\", etc. Would captions generated based on these very similar queries be sufficiently different? If so, in what way?  \n\nB. Why, do you think, are the CLIP-based metrics poorly suited for the data/task presented in this paper? Is it also the case for other types of controllable image captioning described in previous research, where only certain aspects of the image are described in the caption?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Difference from other types of controllable image captioning, guided by textual thematic \"prompts”.\n\nWe have been aware of using “prompts” for guided image captioning in the course of our research. While both methods involve similar inputs and outputs, PromptCap requires much more detailed “questions” instead of our “queries” to control caption generation. This approach seems to require prior comprehension of the images, which deviates from our goal of automatic scene understanding.  We will cite this in our manuscript in the revision and clarify these differences.\nIt is also notable that although the paper “PromptCAP”[2] was published in late 2022 to arXiv and was accepted in the ICCV2023, we do not necessarily assume it is worth mentioning as a part of “reasons to reject” in the EMNLP2023 review because we believe they were done in the same timeline.\n\n> Use of detailed regular images\n\nAs a part of the scene captioning task where scene descriptions come from solely one 360-degree camera, we chose 360-degree images not only for their rich contexts but also because they capture entire scenes without selection bias. The contexts of normal images are almost always selective by photographers and even simplified because people tend to choose what to photograph and crop out. We stated this point in lines 24-41, and we will try to express more clearly the reason for choosing the 360-degree images in the camera-ready version.\n\n> Many of the scene queries do not seem mutually exclusive\n\nIn this dataset, we selected topics that are frequently illustrated in scenery images. We do not impose that the topics are mutually exclusive and hence accept that some topics may overlap in the resulting generation. However, even in the topics that are apparently similar, we expect differences of the focused contexts, e.g., in the “cars” topic models describe the details of the cars while in “traffic” the models describe the road traffic states, including pedestrians and traffic lights. We allow these words overlapping in the query set to enhance the model’s robustness.\nWe hope that the query set will become open-vocabulary and the captioning model will be able to generate captions that accurately capture the nuances of those related words, and our study serves as a dataset to facilitate this.\n\n> Reasons why the CLIP-based metrics poorly suited for our data/task\n\nA prior work [1] showed that CLIP-based metrics are sensitive to whether captions depict salient objects in the image. On the other hand, our proposed task requires captions to focus on various objects or scenes corresponding to the given queries.  In particular, totally different captions can be correct for the same image, depending on the queries. For this reason, CLIPScore, which treats holistic images and texts in the joint space, is poorly suited for our data/task.\n\n[1] An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics, (Ahmadi and Agrawal, 2023)\n\n[2] PromptCap: Prompt-Guided Task-Aware Image Captioning (Hu et al., 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a new take on controllable image captioning, and the dataset and reported results can become a benchmark for researchers interested in this topic. The performance of the fine-tuned models on the novel dataset demonstrate their potential for creating diverse captions focusing on different aspects of the images, given suitable training.", "reasons_to_reject": "The task may not be sufficiently different from other types of controllable image captioning, guided by textual thematic \"prompts\", e.g. (Hu et al., 2022). The only obvious difference is the focus on 360-degree images but it doesn't seem from the paper that 360-degree images are (or should be) treated any differently from regular ones. The authors' claim is that 360-degree images contain more topics and details, but many regular images can also be quite detailed, and the examples provided in the paper (e.g. the forest, lake, bridge scenes in Fig. 5) are not much more detailed than non-panoramic images.\n\nHu, Y., Hua, H., Yang, Z., Shi, W., Smith, N. A., & Luo, J. (2022). Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699.\n", "questions_to_authors": "A. Many of the scene queries do not seem mutually exclusive: e.g. \"cars\" and \"traffic\", \"activity\" and \"what they are doing\", \"interior\" and \"furniture\", \"art\" and \"paintings\", etc. Would captions generated based on these very similar queries be sufficiently different? If so, in what way?  \n\nB. Why, do you think, are the CLIP-based metrics poorly suited for the data/task presented in this paper? Is it also the case for other types of controllable image captioning described in previous research, where only certain aspects of the image are described in the caption?", "paper_topic_and_main_contributions": "The paper introduces a new task, Query-based Image Captioning for 360-degree images (QuIC-360). In this task, given a 360-degree panoramic image and a specific topic (a \"query\"), the model produces a caption that describes the aspects of the image related to the query. The paper also presents a novel dataset collected for this task and experimental results achieved by fine-tuning existing image captioning models on this dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "For clarity, it would be helpful to give an explicit definition of \"(image) context\" in Section 1 -- how this (generally vague) term is going to be used in this paper in particular.\n\n31-32: \"what to be photographed and cropped out\" -> \"what to photograph and crop out\"\n\n191: \"comprises\" -> \"comprise\"\n\n260-261: \"the existing categories used for the image classification task\" -- which categories and task are referred to here? If it is a concrete previous work, it should be cited.\n\n401-403: \"We observe that queries with significant performance improvements provide captions not generated by standard image captioning\" -- would be nice to illustrate this statement with examples, to make the claim more clear and convincing.\n\n416-417: \"how well these captions were relevant to...\" -> \"how relevant these captions were to...\"\n\n488: \"whether\" -> \"based on whether\"\n"}}]

Author rebuttal:

Meta-review: This paper presents a new task for image captioning which requires a model to generate captions based on (1) a 360-degree picture, (2) a controlling query. This task is novel and interesting to the community. However, reviewers have raised some concerns as well: (1) It is not clear the benefit of using a 360-degree picture instead of a standard picture, (2) The experiments section can be improved.

Specifically, reviewer 4vda has provided several good suggestions to improve the quality of the paper, including more baselines and more challenging examples.
Decision: Accept-Findings