Location-Aware Visual Question Generation with Lightweight Models
Authors: Nicholas Collin Suwono
Justin Chen
Tun Min Hung
Ting-Hao Kenneth Huang
I-Bin Liao
Yung-Hui Li
Lun-Wei Ku
Shao-Hua Sun
Venue: EMNLP 2023 Main (2023)
Keywords: Location-aware Visual Question Generation
Visual Question Generation
Question Generation
Lightweight Models
TL;DR: We propose a novel Location-aware Visual Question Generation (LocaVQG) task, create a dataset, and present a method that can learn a lightweight model to address the task.

Abstract: This work introduces a novel task, location-aware visual question generation (LocaVQG), which aims to generate engaging questions from data relevant to a particular geographical location. Specifically, we represent such location-aware information with surrounding images and a GPS coordinate. To tackle this task, we present a dataset generation pipeline that leverages GPT-4 to produce diverse and sophisticated questions. Then, we aim to learn a lightweight model that can address the LocaVQG task and fit on an edge device, such as a mobile phone. To this end, we propose a method which can reliably generate engaging questions from location-aware information. Our proposed method outperforms baselines regarding human evaluation (e.g., engagement, grounding, coherence) and automatic evaluation metrics (e.g., BERTScore, ROUGE-2). Moreover, we conduct extensive ablation studies to justify our proposed techniques for both generating the dataset and solving the task.

Location-Aware Visual Question Generation with Lightweight Models
Nicholas Collin Suwono!?2 1.2
Chih Yao Chen'
Tun Min Hung'
Ting-Hao 'Kenneth' Huang I-Bin Liao*
Yung-Hui Li*
Lun-Wei Ku'
Shao-Hua Sun?
Institute of Information Science, Academia Sinica' National Taiwan University?
Pennsylvania State University Hon Hai Research Institute
r1 11894022@ntu. edu. tw cyaochenêcs. unc. edu
hungtungmingegmaii. com
txh710@psu. edu yunghui. ww@foxconn..nommmmm com
ibin. Mlin.liaoe www.1lffffooooo....mmmmm com inveiissssnnca edu shaohuas@ntu. edu. tw
Abstract
Location-aware Information
Streetview Images
This work introduces novel task, location- aware visual question generation (LocaVQG), which aims generate engaging questions from data relevant to particular geographi- cal location. Specifically, we represent such location-aware information with surrounding images and a GPS coordinate. To tackle this task, we present dataset generation pipeline that leverages GPT-4 to produce diverse and sophisticated questions. Then, we aim to learn a lightweight model that can address the Lo- ca vQQ task and fit as a mobile phone. To this end, we propose a method which reliably generate engag- ing questions from location-awar information. Our proposed method outperforms basclines regarding human evaluation (e.g-, engagement, grounding, coherence) and automatic evalu- ation metrics (e.g.. BERTScore, ROUGE-2 2). Moreover, we conduct extensive ablation stud- ies to justify our proposed techniques for gen- erating the dataset and solving the task.
West
GPS Coordinate
40%75 79.05"N -73995 70. 14 "W
Generated Engaging Question
Does anyone know any famous movies or TV shows that have featured the Queensboro Bridge their scenes?
Figure 1: Location-aware Visual Question Genera- tion (LocaVQG) involves generating engaging ques- tions from a specified location, represented by GPS coordinate of vehicle and set of street view images captured by on-car cameras.
edge device, such
driver in a conversation? While initiating a conver- sation with general questions may not interest the driver, delving into driver-specific inquiries raises privacy concerns since it requires personal infor- mation. Our key insight is to engage the driver in conversation by posing questions based on the location-aware information, composed of both the geographical coordinate of the car and surrounding visual perception represented by pictures captured by on-car cameras. Such rich location-awer aware infor- mation allows for producing diverse and relevant questions, enabling a system to initiate engaging conversation.
1 Introduction
Driving is an integral part of our daily routines, playing a significant role in our lives. Whether commuting to work, running errands, or embarking on exciting adventures, we heavily rely on automo- biles to get us from one place to another. Despite its undeniable convenience, driving requires con- stant focus the road, the need to remain alert, and the mental strain of navigating through traffic. Hence, staying behind the wheel after long work- ing hours or during long-distance trip can give rise to hazardous circumstances. To combat this, passengers often engage in conversation to keep the driver awake and attentive.
In this work, we introduce a novel task, Location- aware Visual Question Generation (LocaVQG), which aims to produce engaging questions from a GPS coordinate of a vehicle and set of street-view images captured by on-car cameras, as illustrated in Figure 1. We make the first attempt to tackle this task by developing a data generation pipeline that can create dataset containing high-quality samples for the LocaVQG task. To this end, we leverage the recent advances in large language mod- els (LLMs) (Liu et al., 2023; Touvron et al., 2023).
Can we develop system running lightweight device that automatically engages the
Project page can he found at https://giite httis:///gttuubooom LtademiiSinttcalLLLL//.ooo.

---

Specifically, we collect data from Google Street View and design a prompt according to the address obtained by reverse geocoding the GPS coordinate and the captions street-view images provided by an off-the-shelf image captioning model. While LLMs can generate questions relevant to the pro- vided location-awasl aaationnwwwaeeeeee information, the produced questions may not always be engaging. Therefore, we further propose to train an engaging question classifier that can filter out non-engaging questions. Our proposed dataset generation pipeline is illus- trated in Figure
most current commercialized autonomous vehicles are categorized as SAE (the Society of Automo- tive Engineers) (International, 2018) Level 2 (e.g., Tesla, Hyumdai, Kia) or Level 3 (e.g., Mercedes). When driving an SAE Level 2 vehicle, the driver must always hold the steering wheel. With an SAE Level 3 vehicle, the driver must still be ready to take control of the vehicle at all times when prompted by the vehicle. That being said, driving in modern days still requires the driver's attention and therefore can be assisted with the task and the system proposed in this work.
We present method, FDT5, that can learn a lightweight model and reliably address the Lo- caVQG task. We compare proposed method to various small and mid-size language models Iearn- ing from the generated dataset. The experimental results demonstrate that our proposed FDT5 out- performs the baselines regarding human evaluation (e.g., engagement, coherence, grounding) and auto- matic evaluation metrics, e.g., BERTSSore (Zhang et al., 2019), ROUGE-2 (Lin, 2004). Our FDTS with only 15M parameters achieves competitive performance even compared to large language model (i.e., GPT-4). This highlights the effective- ness of the proposed dataset generation pipeline as well as the proposed training techniques.
In-car intelligent assistant system. Developing in-car intelligent assistant systems, such as a voice assistant (Lin et al., 2018; Braun et al., 2019), is emerging research area. Large et al. (2017) discovered that engaging drivers in conversation could effectively reduce driver fatigue. In contrast, this work focuses on raising drivers' attention by formulating task and devising system produce location-aware engaging questions.
Visual question generation (VQG). VQG con- cerns generating questions from visual in- puts (Mostafazadeh et al., 2016). Compared to this work, recent works (Lu et al., 2021; Yeh et al., 2022) that explore VQG do not leverage geograph- ical information (e.g.. GPS). On the other hand, Zhang and Choi (2021) presented a dataset with questions respecting geographical and temporal contexts; yet, it does not utilize visual inputs. contrast, the task and the dataset proposed in this work leverage images captured by on-car cameras.
The main contributions of this work are three- fold as follows:
Task. We propose Location-aware Visual Question Generation (LocaVQG), novel task that aims to produce engaging questions from GPS coordinate of a vehicle and a set of street-view images captured by on-car cam- eras. This will lead to the development of more intelligent in-car assistant systems.
Large language models (LLMs). Recent ad- vances in LLMs (Liu al., 2023; Touvron et al., 2023) have led promising results in various do- mains (Wei et al., 2022: Zhang al., 2023). How- ever, these gigantic LLMs with billions of parame- ters (Liu et al., 2023; Touvron et al., 2023) cannot be deployed on lightweight devices and therefore are not ideal for in-car intelligent assistant systems, This work aims to develop lightweight models that can run on edge devices like mobile phones.
To address Loca we introduce dataset generation pipeline that can produce diverse and engaging questions from speci- fied location by leveraging pre-trained LLMss
Dataset.
Method. We present a method FDT5 that outperforms all the lightweight baselines re- garding human evaluation (e.g., engagement, coherence, grounding) and automatic evalua- tion metrics (e.g., BERTScore, ROUGB-2).
Lightweight language models. Existing mobile- friendly language models (Sun al., 2020: Mehta and Rastegari. 2021) struggle at language genera- tion tasks. This work aims to devise lightweight models that can address VQG tasks and achieve competitive results even compared to LLMs.
2 Related Works
Self-driving cars. Despite the recent advances in developing self-driving cars (Parekh et al., 2022),

---

Streetview Images
Captions
Prompt
city
some uption ybout suncundings You are now shiong Pitsburgh, PA 1522., USA.
people walking
#100
Image
city strcet cars drfiing
Captioning Model
On your Nort, city sreet with tall bui buildings and people wailing down
cll stioet cars diing.
Cruyour
your tu buildings.
West
People walk'ng ldings
On your West, people walking in plaas front tall buildings.
GPS
Address
16"
Reverse Geocoding
Third Awenue, Pitsburgh, 16222,
Hease genemst engag from mose doscriptons
-00000 33""0033.000"" 30"000W
Prompt
Generated Questions
Classifier Output
complex consisting Eetwaiin 1981 and 19647 Can anyon guiss the and srect who deskg add ramarable comalex?
Die
which completed
Engaging
Engaging Question
Consicering ing the trafl cord ors on the west side, can anyone guess what me day might be?
Hot engaging
GPT-4
Classifier
Innaperiater used by locals Prisburgh How
tranepentiionn cotioes your cit?
Eogaging
Figure 2: Data Generation Pipeline. This pipeline produces questions from a given Loca VQG task tuple consisting of four street view images Vv, Va, Vs, Vw and GPS coordinate X. We caption each image using an image captioning model and infer the address by reverse geocoding the GPS coordinate. Then, we construct prompt that describes in detail the location-aware information and leverage GPT-4 to generate questions. We further employ an engaging question classifier to filter out non-engaging questions. Finally, the remained questions included in the dataset with the given LocaVQG task tuple.
3 Loca VQG: Location-aware Visual Question Generation
To ensure the location diversity, select 3,759 ordinates with their directional images, excluding the upper/sky view, which is usually not observ- able by the driver. We denote the geographical coordinate of each location as X and its surround- ing images as Va with = [N, E, S, W). standing for each direction. We define our LocaVQG task tuple T as: T [VN, VE, Vs, Vw. X]. Given a LocaVQG task tuple T, our goal is to produce engaging question Q with model f: f(T) Q.
We introduce novel task, Location-aware Visual Question Generation (QocaVQG). This section for- mally defines this task and describes how we collect data to construct the LoC task tuples.
3.1
Location-aware Information
Location-aware Information includes data or con- tent specifically relevant to, or influenced by a par- ticular geographical location. With such informa- tion, applications can offer location-specific rec- ommendations, directions, local weather updates, nearby points of interest, targeted advertisements, etc. Since our goal to produce engaging ques- tions with an in-car device based on location-awe aware information, we limit it to the information that is easily accessible even without the internet. Specif- ically, we consider the surrounding visual percep- tion and the geographical coordinate.
Generating Loca
LocaVQG VQG Dataset
Our goal to train lightweight models to address the LocaVQG task. Therefore, we aim to "label" the task tuples described Section 3. Annotating the task tuples with engaging questions requires creativity and location-specific domain knowledge. which can be challenging for human annotators. In this work, we propose automatically generating questions from LocaVQG task tuples by leveraging the recent development of LLMs. An overview of the proposed dataset generation pipeline is depicted in Figure 2.
3.2 LQc Task Tuple
To collect the surrounding visual perception and the geographical coordinate of diverse locations, we propose leverage Google Street View Dataset (Zamir and Shah, 2014). The dataset contains 10,343 coordinates, and each coordinate comes with 5 corresponding directional images (North, East, South, West, and Upper/Sky view).
4.1 Prompting GPT-4
This section describes how utilize GPT-4 (Ope- nAI, 2023) to produce questions from LQG task tuples by processing task tuples and designing LocaVQG prompts.

---

Engaging Questions
The city of Pittsburgh is known its numerous bridges. How many bridges do you think are in the city, and why do you think there are so many?
What types of events or festivals might take place in this park throughout the year? As we look towards the south, can you guess the purpose of this brick building with cars parked in front? Perhaps an office building, a restaurant, or something else?
Non-En -un-Enggging Questions
Speaking of the hospital, does anyone know the range of medical services provided Prince George's Hospital?
What some ways that city plunners might improve traffic flow at busy intersections? Noticing the mixture of architoctural styles, can you guess which era had the most significant influence the city's development?
Table I: GPT-Generated Questions. We provide examples of GPT-4 generated questions that are classified engaging and non-engaging by the engaging question classifier. Answering these non-engaging questions often requires specific domain knowledge and therefore may interest only limited audience. blue-colored text indicate visual cues, red-colored text indicate directional cues, teal-colored text indicate location-specifie information.
Street view images + captions. While GPT-4
4.2 Filtering GPT-Generated Questions
is a multimodal model, its feature of taking im- age inputs is not yet publicly accessible as of May 2023. Hence, to inform GPT-4 with the street view images, we caption street view images using an off-the-shelf image captioning model (Wang et al., 2022).
While GPT-4 can generate numerous diverse ques- tions from our designed prompts, we empirically find that some generated questions are not particu- larly engaging (e.g., requiring domain knowledge), as shown Table 1. To combat this, we propose to learn a BERT-based (Devlin et al., 2019) engaging question classifier to filter out non-engaging ques- tions. We construct the training data for this classi- fier with non-engaging questions from SQuaD (Ra- jpurkar et al., 2016) and engaging questions from MVQG (Yeh et al., 2022). The key insight is that SQuaD questions are made for question-answering tasks, thus, solely revolves around passage, while MVQG questions are collected with engagement in mind.
GPS coordinate >> address. To leverage the GPS coordinate, we reverse geocode it using Google's Reverse Geocoding API (Google), translating the coordinate into a street address. We found that with the decoded street address, GPT-4 can often infer nearby famous landmarks, or information, and generate relevant questions.
Constructing prompts. We aim to prompt GPT-4 with the processed location-aware input and pro- duce engaging questions. We first design system prompt that infuses GPT-4 with tour guide role, enforcing it engage users. Then, we design chat prompt that encapsulates processed location- aware information and requires GPT-4 to generate engaging questions. The two prompts are presented as follows.
With this trained engaging question classifier, for each LocaVQG task tuple T, we filter out non- engaging questions generated by GPT-4, and the remained questions are included in the dataset the "labels" for this task tuple.
4.3 Dataset Statistics
Applying the procedures described in Section 4.1 and Section 4.2 results in a dataset with 3759 task tuples and 35K questions. The basic statistics of the dataset are described in Table 2.
System prompt: You are a tour guide and you are driving in a car with your tourists. You want to engage with them with any kind of information you have around you.
of LocaVQG Task Tuples
3759
# of Task Tuples from Pinsburgh
919
# of Task Tuples from Orlando
611
Chat prompt: Here are some descriptions of your surroundings You are currently driving on [Street Address]. On your North, [mage Caption]. On your East, [mmaee Caption]. On your South, [Image Caption). On your West, Image Caption]. Based on those descriptions. please ask 10 engaging questions.
# of Task Tuples from New York
2217
of Questions After Filtering
35551
Average Sentence Length
16.6
Average Question Length
30.8
Table 2: Dataset Statistics. We present the statistics of our location-aware visual question generation dataset

---

Average Sentence Length (Avg Length) computes the average length of sen- tences across the whole dataset, representing how rich and detailed a dataset is.
Sent.
20000
# of Questions
3000
# of Questions
15000
2000
10000
Syntactic Complexity calculates the degree of variation, sophistication, and elaboration of the questions in a dataset (Ferraro et al.. 2015). We report the mean of Yngve Score normalized by the sentence length.
1000
5000
6
10
25 # of Tokens
15
20
30
35
40
2 23 Sentences
Figure 3: Question Length. We present the question length statistics in and tokens and sentences.
Percentage of Abstract Terms (% Abstract Term) computes the ratio of visual and non- visual concepts covered by a dataset, based on the abstract terms defined by Vanderwende et al. (2015).
4.3.1 Question Length
We present the histograms of question lengths in terms of of tokens and # of sentences in Figure 3.
4.3.2 Frequent Trigrams and Words
Average Term Depth is calculated based on WordNet, where noun words with a lower depth indicate higher-level concepts (Lu et al.. 2021).
We present the top 15 frequent trigrams in the dataset in Table The questions often start by trying to intrigue the respondent (e.g., Did you know, What do you). Also, open-ended questions r.g.,have you noticed, have you ever) appear quite frequently. Most frequent words, presented in Ta- ble 4, require or lead the attention of the respondent (e.g., considering, looking, notice).
Compared the MVQG dataset, the results show that our proposed LocaVQG dataset contains significantly more diverse and sophisticated ques- tions. In fact, the questions included in the MVQG dataset are collected from human annotators. This highlights the effectiveness of generating questions by leveraging the recent advances in LLMs (e.g.. GPT-4), as proposed in this work. Further eval- uations on the generated questions be found in Section D and Scction E.
Did you know
What do you As we drive
Can you
Can guess
How do you Can anyone What are your continue
Have any of
Have you noticed
Have Are you familiar
Can you identify
What
Table Top 15 Frequent Trigram of Questions. The frequent trigrams appearing in the dataset show that the questions aim intrigue or engage respondent.
Criteria
MVQG
LocaVQG (Ours)
Vocabulary Size
608
3046
Average Sentence Length
12.341
17.168
Yngve Score f
2.271
3.761
%6. Abstract Terms f Average Term Depth J
0.127
0.167
Speaking
Considering
Based
Looking
Notice New
7,906
7.259
Since
Perhaps
Observing
Residential
Look
Pittsburgh
Given
Orlando
Table 5: Question Quality Comparison with MVQG.
Turning
Judging
Noticing
Resides
Table Top 20 Frequent Words.
This table presents 20 frequent words appearing in the dataset aside from commonly used words, such as "have", "can".
5 Learning and Evaluating Lightweight Models
We aim to train and evaluate lightweight models learning from the proposed LocaVQG dataset.
4.3.3 Question Quality
We compare our proposed dataset to engaging question dataset, the MVQG dataset (Yeh et al., 2022), regarding the following criteria and report the results in Table 5.
5.1 Baselines
We compare method the following baselines. Text-To-Text Transfer Transformer (TS). We experiment with a family of T5 pre-trained lan- guage models (Raffel et al., 2020; Tay et al., 2021), which includes T5- (770M), T5- Base (220M),
Vocabulary Size (Vocab Size) measures the number of distinct words in a dataset.

---

Model
#Parameters
Engagement
Naturalness
Coherence
Common Sense
Grounding
Overall
MVQG-VL-TS
254M
3.64
3.65
3.84
3,76
MVQG-VI-
254M
3.96
3.82
3.82
3.99
3.66
3.85
T5-Large
770M
3.92
3.81
3.78
4.03
3.83
3,87
T5-Basc
220M
3.92
3.81
3.73
3.97
3.78
3,84
T5-Tiny
15.6M
3.96
3.79
3.67
4.01
3.81
3.85
HDT5 (Ours)
15.6M
4.03
3.83
3.96
4.05
4.03
3.98
GPT-4
IT
4.12
3.99
4.01
4.01
4.04
Human Anmotator
4.06
3.87
3.90
4.06
3.88
3.95
Table 6: Human Evaluations. Each question is rated by three AMT workers. Among all the light-weight weight models. the fewest parameters, Note that while the exact
our proposed FDT5 achieves the best overall performance number of parameters GPT-4 is not revealed, many believe it is at least times larger than GPT-3 (Brown al., 2020) (175B).
and T5-Tiny (15.6M). We fine-tune the pre-trained T5 models on our LocaVQQ dataset. Specifically, for each LocaVQG task, the models learn to map the prompt presented in Section 4.1 to one of the ground truth questions generated by GPT-4.
questions. Specifically, given LocaVQQ task, our method keeps generating questions until accepted (i.e., classified as "engaging") by the classifier.
Our proposed method FDT5 combines the two techniques described above.
MVQG-VL-TS. Cho et al. (2021) introduced Vision-and-Languaa. T5 (VLT5) for vision-and- language tasks. Yeh et al. (2022) adapted it for generating questions from set of images. We adopt this method, dubbed MVQG-VL-TS, and fine-tune the pre-trained model on our LocaVQG dataset. The input of MVQG- VL-T5 consists of 4 street view images and the street address. More details can be found in Section F.
5.3 Human Evaluation
We provide human evaluation of the questions gen- erated by all the methods.
5.3.1 Evaluation Metrics
We randomly sampled 100 LocaVQG task tuples and the questions produced by all the models. Each question evaluated by three Amazon Mechanical Turk (AMT) workers according to the following metrics. We adopt a 5-point Likert scale for all the evaluation metrics.
5.2 Our Approach
We propose Filtered Distilled Ti-Tiny (FDT5). Distillation. While T5-Tiny has the fewest of pa- rameters and can on mobile phones, capac- ity might be limited to a complex task like Lo- caVQG. Therefore, we propose to learn a T5-Tiny model by distilling a learned T5 T5-Large model. In- spired by Chen et al. (2020), during training, utilize both the questions generated by GPT-4 from the dataset and the questions generated by the T5- Large model, resulting in the objective:
You find the question engaging and you would want to answer the question.
Engagement:
Naturalness: It is natural to ask this question given the information you have.
The question is coherent with the information you have.
Coherence:
Common Sense: It makes sense to ask these questions given the information you have.
(()) a Chara + +(1 a) (0),
Grounding: The question asked about things related to the information you have.
where a balances the relative importance of learn- ing from each loss and 0 parameterizes the model. The hard-label loss Lhard (ground truth target) opti- mizes cross-entropy, while the soft-label loss Leoft (teacher model) optimizes KL Divergence. Filtering. To further improve the engagingness the questions produced by our method, we pro- pose to utilize the engaging questions classifier described in Section 4.2 to filter out non-engaging
We also provide the evaluation of the questions generated by GPT-4, which can be considered upper bound as GPT-4 has unparalleled number of parameters compared the lightweight models. Furthermore, to compare the performance of these LMs against humans, we crowdsource and collect 100 questions on AMT based on the same set

---

Model
Engagement
Naturalness
Coherence
Common Sense
Grounding
Overall
Filtered Datasct (Ours)
3,92
3,97
3,78
3,84
Unfiltered Dataset
3.89
3.76
3,78
3.85
3,78
3.81
Table Engaging Question Classifier for Dataset Generation. Employing the engaging question classifier in the dataset generation process to filter out unengaging questions improves the quality of generated questions.
Model
Engagement
Naturalness
Coherence
Common Sense
Grounding
Overall
Filtered Inference (Ours)
4.03
4.05
3.98
Unfiltered Inference
3.96
3.78
4.04
3.79
3.88
Table Engaging Question Classifier for Inference. Our proposed FDT5 employs the classifier during inference to filter unengaging questions. Excluding the filtering phase results in significantly worse performance.
LocaVQQ task tuples. More details on AMT can be found in Section G.
We further evaluate the questions generated by all the models with some automatic evaluation met- rics. To compare two questions based on exact wording, we are using BLEU-4 (Papineni et al., 2002), ROUGE-2 (Lin, 2004). To compare the questions based semantic similarity, are using ML-Based evaluation: BERTScore (Zhang et al., 2019), and BLEURT (Sellam al., 2020). The results are presented in Table 7. Our proposed method FDT5 achieves the best performance on ROUGE-2, BERTScore, and BLEURT. T5-Large outperforms others in terms of BLEU-4. This veri- fies the effectiveness of the filtering and distillation technique employed in FDT5.
5.3.2 Results
The human evaluations are presented in Table 6. FDT5 outperforms all the lightweight models. Our proposed method FDT5 achieves the best over- all score with the fewest parameters. This justifies the effectiveness of our adopted distillation scheme. Furthermore, average score of 3.98 indicates that our model can reliably generate satisfactory questions from location-aware information.
MYQG-VI-T5. MVQG-VL-T5, without learning from our dataset, achieves the worst performance, demonstrating the importance of constructing and learning from dataset dedicated to the LocaVQG task. Alternatively, the MVQG-VL-TS model fine- tuned dataset (MVQG-VL-TS fine-taned) struggles grounding, aligning with the findings discussed in (Ych al., 2022).
5.5 Ablation Study
We conduct extensive ablation studies to investi- gate the effectiveness of employing the filtering classifier (Section 5.5.1), the impact of incorporat- ing GPS coordinates into the question generation process (Section 5.5.2), and the effect of varying dataset sizes (Section 5.5.3).
GPT-4 asks better questions than humans. The questions produced by GPT-4 are preferred by the workers compared to those provided by human an- notators on all the metrics, except for common sense. This justifies our proposed dataset genera- tion pipeline, which collects questions from GPT-4 instead of humans.
5.5.1 Employing Engaging Question Classifier
We propose to learn engaging question classi- fier for (1) filtering out non-engaging questions generated by GPT-4 during the dataset generation process (Section 4.2), and (2) filtering out non- engaging questions produced by our model during inference (Section 5.2). This section examines the effect of employing this classifier.
5.4
Automatic Evaluation Metrics
Model
BLEU-4
ROUGE-2
BERTSeore
BLEURT
Dataset Generation. To verify the effectiveness of filtering out questions from GPT-4 with the clas- sifier, we train T5-Base model to learn from unfiltered dataset that contains all the questions produced by GPT-4 (Unfiltered Dataset). We com- pare the performance of this model to the T5-Base model trained on our proposed filtered dataset (Fil- tered Dataset) and report the human evaluation re-
VLTS
0.2712
0.0342
0.5093
.0.7208
T5-Large
0.2756
0.0380
0.5165
-0.7336
TS-Base
0,2746
0.0388
0.5163
-0.7305
TS-Tiny 0.2635
0.0371
0,5164
-0.7419
FDTS (Ours)
0.0393
0.5190
-0.7073
Table Automatic Evaluation. Our proposed FDTS achieves the best performance on 3 out of metrics (i.e., ROUGE-2, BERTScore, and BLEURT).

---

Streetview Images
Generated Questions
Address
Colonial Dr. FL 32501, USA
6 Queensboro Bridge York, USA
T5-Large
you that the ng and only var ous avens? any of you stlonded event at cimiarr location betore!
popu spct octh locais sour
aseeraoro Rrdige hoi breen numnan conacuu mossss and Inlecis
shows. anyone name or show features the lconic bridge?
T5-Tiny
Eipnacinng
you ony iegugum intereeiigg par-edt side the smeet?
youe favorioo
Fledgg
curmently
thom
marker?
FDTS
Can identity
parked
of the nood? think
other scenic destinatinn atrrations lookang fonward seeting duiing to New York Ciy?
tout
(Ours)
onea that you'se vislie?
Human
|What you think ascu: Crlands comp pared to ohher placos?
How long feet do you the ed iooo bridge in New York 8s7
Figure 4: Qualitative Results. We present sampled generated questions from T5-Large, T5-Tiny, our proposed method FDTS, and human annotators, together with corresponding streetview images and addresses. With only 15M parameters, FDTS can reliably generate engaging location-awer -wwrre questions.
sults in Table 8. model learning from the filtered dataset achieves better performance. justifffyn the efficacy of em- ploying the classifier.
The results demonstrate that the
5.5.3 Varying Dataset Sizes
We investigate the impact of varying dataset sizes with T5-Tiny and our proposed FDT5, and report the results Table 11. FDT5 achieves better perfor- mance with fewer data points and performs compa- rably T5-Tiny when dataset size increases. This indicates that our method is more data efficient.
Inference. We propose filter out non-engaging questions generated during inference, adopted in our method FDT5. We conduct human evaluations on filtered generation questions (Filtered Inference) and unfiltered questions (Unflttrrd Inference), re- ported in Table 9. The results show that filtering non-engaging generated questions with the classi- fier can significantly improve the question quality on all the metrics. This justifies the effectiveness of employing the classifier during inference.
Model
#Samples
BLED1
ROUGE-2
BERTSeore
BLEURT
0.7K
0.2566
0.5160
-0.7666
1.7K
0.2629
0.0341
0,5139
40,530
TS-Tiny
2.7k
0.2604
0.164
.07589
3.7K
0,2635
0.0371
0,5156
0,7419
4.7K
0.2639
0.0361
0.5145
4.7398
0.7k
0.2565
0.5201
-0.1449
1.7k
0,2000
0,042
0.5214
.0.7245
HDTS (Ours)
2.7k
0.2675
0.0422
0.5211
.0.1126
3.7k
0.2661
0.0393
0.5190
40.7033
4.7k
0.2706
0.0336
0.3180
-0.7256
5.5.2 Incorporating GPS Coordinates
Table 11: Effect of Dataset Size. From the results, FDT5 is more data efficient as it could achieve better performance with smaller sample size
While Yeh et al. (2022) explored generating ques- tions from set of images, our work further in- corporates addresses (reverse geocoded from GPS coordinates) into the question generation process. This section investigates the effect of employing such information. We compare the questions gen- crated by GPT-4 with or without the address in the prompt and report the results in Table 10. The results show that incorporating the address leads to richer and more diverse questions, verifying the unique value of the proposed LocaVQC task.
5.6 Qualitative Results
As human evaluations can be subjective, we present qualitative results in Figure 4 for the readers to bet- ter understand the generated questions. The results show that FDT5 with only 15M parameters can re- liably generate engaging location-awe aware questions.
Conclusion
In this work, we propose a novel task, location- aware visual question generation (LocaVQG), which aims to generate engaging questions from data relevant to particular geographical location. Specifically, we represent location-aware informa- tion using four directional street view images and GPS coordinate. To address this task, we intro- duce a dataset generation pipeline that leverages the recent advances of large language models (i.e.,
Criteria
w/o address
w/ address (ours)
Vocabulary Size f
450
525
Average Question Length+ f
25.0 02.02
30.18
Yngve Score t
3,531
3.693
Table 10: Effect of Leveraging Street Address The questions generated with street addresses are richer and more diverse.

---

GPT-4) to generate diverse and sophisticated ques- tions. To ensure the engagingness of the questions produced by GPT-4 we employ an engaging ques- tion classifier to filter out non-engaging questions. Our proposed dataset contains richer and various questions compared to existing datascts.
erating question without considering continuing conversation. Second, we evaluate the generated questions with an AMT interface where the AMT workers read and evaluate the questions. However, in a driving scenario, interacting with a virtual as- sistant by reading a question is impractical. Hence, evaluating the generated questions by connecting to text-to-speech system and requiring the annota- tors rate the questions by listening to them would align better with the application.
To learn from the proposed Loca dataset with lightweight models, we present Filtered Dis- tilled T5-Tiny (FDT5) method. We extensively evaluate the proposed method and the baselines with human evaluation and automatic evaluation metrics. Our proposed FDTS, with the fewest pa- rameters, demonstrates superior performance on most metrics. We conduct extensive ablation stud- ies to verify the effect of employing the filtering classifier, the effectiveness of incorporating GPS coordinates into the question generation process, and the impact of varying dataset sizes. We hope this work will encourage researchers to explore the Loc task and its applications.
Distractingness of generated questions. This work makes the very first attempt to develop an in-car visual question generation system that can ask engaging questions to initiate conversations with drivers. However, such engaging questions can potentially distract drivers and lead to danger- ous situations in the worst case. To address such a concern, we encourage future works along this line to consider the "distractingness" of generated ques- tions. In particular, developing evaluation metrics to determine a question would distract a driver from the road and devising methods to produce en- gaging yet non-distracting questions are promising and interesting research directions.
Limitations
We discuss the limitations and how we can poten- tially address them in this section.
Biases in AMT workers. We notice that the AMT workers involved in human evaluation might be biased due to their demographic. This can poten- tially be addressed by ensuring the diversity of their background.
Ethics Statement
Since our proposed dataset generation pipeline in- volves collecting questions from GPT-4, the data inherits any biases of GPT-4. Moreover, our pro- posed method learns from the dataset, and therefore will also be biased. Therefore, inheriting biases can lead to generating inappropriate, sexist, racist ques- tions or descriptions. Fortunately, addressing ethi- cal issues has been an active research area (Liang et al., 2021; Baldini et al., 2021; Yan et al., 2023; Kasneci al., 2023). We wish to incorporate the advances in the field to alleviate ethical concerns.
Location-aware information. Aiming to develop an in-car intelligent assistant, this work proposes representing location-aware information as GPS coordinate and set of images captured by on-car cameras. Incorporating more detailed information, such as local news and weather, can potentially lead to more diverse and engaging questions, and is left for future research.
Address-aware LLMs. Our proposed dataset gen- cration pipeline heavily relies on GPT-4. This partially limits the generated questions to loca- tinss/ddresssss that are known by GPT-4 and there- fore this pipeline might not produce coherent ques- tions given locations that are less known by GPT-4, We can potentially address this by employing a more sophisticated external information retrieval system to extract information from a location
Acknowledgement
This work supported by the National Science and Technology Council of Taiwan under grants 111- 2634-F-002-022- and by the Academia Sinica and Hong Hai Research Institute collaborative project OST-1 AST-1110523-1Q Shao-Hua Sun was partially sup- ported by the National Taiwan University and its Department of Electrical Engincering, Graduate In- stitute of Communication Engineering, and College of Electrical Engineering and Computer Science, and the Yushan Fellow Program by the Ministry of Education, Taiwan. We thank the online crowd workers for participating in the study.
Human evaluation setup. While our motivation is develop an in-car intelligent system that can engage a driver in a conversation to keep the driver awake, this work falls short from the following perspectives. First, our work solely focuses on gen-

---

References
David Large, Gary Burnett, Vicki Antrobus, and Lee Skrypchuk. 2017. Stimulating conversation: Engag- ing drivers in natural language interactions with autonomous digital driving assistant counteract passive task-related fatigue. In International Confer-
Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ra- mamurthy, Mikhail Yurochkin, and Moninder Singh. 2021. Your faimess may vary: Pretrained language model faimess in toxic text classification. arXiv preprint
ence Driver Distraction and Inattention.
Paul Pu Liang, Chiyu Wu, Louis-Philipg Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In ICML.
Michael Braun, Anja Mainz, Ronee Chadowitz, Bastian Pileging, and Florian Alt. 2019. At your service: Designing voice assistant personalities to improve automotive user interfaces. In CHI Conference on Human Factors in Computing Systems.
Chin- Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out. ACI..
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Shih- Chich Lin, Chang Hong Hsu, Walter Talamonti. Yunqi Zhang, Steve Oney, Jason Mars, and Lingjia Tang. 2018. Adasa: conversational in-vehicle digi- tal assistant for advanced driver assistance features. In ACM Symposium User Interface Software and Technology. Association for Computing Machinery.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang. Jiaming Tian, Hao He. Antong Li. Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang. Dingang Shen, Tianming Liu, and Bao Ge. 2023. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.0 2.04.01844..2
Ven-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020. Distilling the knowledge of BERT for text generation. In ACL.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and- rision-and-language tasks via text genera- tion. In ICML.
Zexin Lu, Keyang Ding. Yuji Zhang. Jing Li, Baolin Peng, and Lemao Liu. 2021. Engage the public: Poll question generation for social media posts. In ACL- IJCNLP.
Tim Detimers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient
Sachin Mehta and Mohammad Rastegari. 2021. Mo- bilevit: light-weight, general-purpose,, and mobile- friendly vision transformer. arXiv preprint 11Xiv:2110.02178.
finetuning of quantized LLMs. arXiv preprint
arXiv:2305.1453 14314.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Nasrin Mostafazadeh, Ishan Misra, Jacob Deylin, Mar- garet Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In ACI.
Francis Ferraro, Nasrin Mostafazadeh, Hag Huang. Lucy Vanderwende, Jacob Devlin, Michel Galley, and Margaret Mitchell. 2015. A survey of cur- rent datasets for vision and language research. In EMNLP.
OpenAl. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.0877.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In ACL.
Google. Reverse geocoding.
HuggingFace. 16-cos-vl.
Darsh Parekh, Nishi Poddar, Aakash Rajpurkar, Man- isha Chahal, Neeraj Kumar, Gyanendra Prasad Joshi, and Woong Cho. 2022. A review on autonomous vehicles: Progress, methods and challenges. Elec- tronics.
Sae International. 2018. Taxonomy and definitions for terms related driving automation systems for on- road motor vehicles. SAE international.
Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi. and Aniruddha Kembhavi. 2020. Grounded situation recognition. In ECCV.
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva. Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chatgpt for good? op- portunities and challenges of large language models for education. Learning and Individual Differences.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits

---

of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research.
Min-Hsuan Yeh, Vincent Chen, Ting-Hao Huang, and Lun Wei Ku. 2022. Multi- AQQti Generating engag- ing questions for multiple images. In EMNLP.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Amir Roshan Zamir and Mubarak Shah. 2014, Image ero-localization based on multiplenearest neighbor feature matching usinggeneralized graphs. PAMI.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems.
Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xi- ang Ren, Minsuk Chang, Shao-Hur Sun, and Joseph Lim. 2023. Bootstrap your own skills: Learning solve new tasks with large language model guidance. In Conference Robot Learning.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-OKVQA: A benchmark for visual question an- swering using world knowledge. arXiv preprint
Michael J.Q. Zhang and Eunsol Choi. 2021. Situat- edQA: Incorporating extra-linguistic contexts into QA. EMNLP.
arXiv:2206.01718.
Tianyi Zhang, Varsha Kishore, Felix Wu. Kilian Q Wein- berger, and Yoav Artzi. 2019. BERTScore: Evalu- ating text generation with BERT, arXiv preprint arXiv:1904.09675.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: learning robust metries for text generation. arXiv preprint ar 2ivi:2004.0000 04696.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: compact task-agnostic bert for resource- limited de- vices. arXiv preprint vrXiv:200402988
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Met- zler. 2021. Scale efficiently: Insights from pre- training fine-tuning transformers. arXiv preprint arXiv:2109, 10686.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13977.
Lucy Vanderwende, Arul Menezes, and Chris Quirk. 2015. An AMR parser for English, French, German, Spanish and Japanese and new AMR-annotated corpus. In NAACL.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying chitectures, tasks, modalities through simple sequence-to-sequuecce learning framework. In ICML.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi. Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. TMLR.
Lixiang Yan, Lele Sha. Linxuan Zhao, Yuheng Li. Roberto
Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. 2023. Practical and ethical challenges of large language models in education: A systematic literature review. arXiv preprint arXiv:2303.1337 13379.

---

A
Additional Diversity Analyses GPT-4 Generated Questions
on average, indicating that the engaging question classifier requires FDT5 to perform 1.73 additional inference trials for each LocaVQG task on average. Note that this post-filtering phase can be shut down for latency-critical scenarios, and FDT5 without post post-filtering still outperforms T5- 75-T in human evaluation, according to Table 6 and Table 9.
We perform further diversity analyses on the ques- tions provided in our LocaVQG dataset.
Question type analysis: While Table 3 shows the top 15 most frequent trigrams of gener- ated questions, we have performed an addi- tional trigram analysis during the rebuttal pe- riod to examine the diversity of the gener- ated questions. In particular, we followed Yeh et al. (2022) and identified 2437 ques- tion types among our 35K generated questions. This highlights the diversity of the generated questions.
C Filtering Out Non-engaging Questions Generated by GPT-4 Using GPT-4
This work proposes to train an engaging question classifier to filter out non-engaging questions gener- ated by GPT-4; alternatively, evaluate and filter out non-engaging questions that generates. To investigate this possibility, we feed the questions generated by GPT-4 back into GPT- for scoring (i.e., determining each generated question is engaging or not).
use GPT-4 to
Pairwise cosine similarity: Inspired by Schwenk al. (2022), which computes the average pairwise cosine similarity between each pair of generated questions encoded by a sentence transformer Mauuli-qa-MiniLMM-- cos-vl provided by HuggingFaco in dataset, we have performed this evaluation on our gen- erated dataset. We obtained an average cosine similarity of 0.1698, indicating that the gen- erated questions are not highly correlated and therefore ensuring the diversity of our pro- posed dataset.
Specifically, we provide GPT-4 with 10 ques- tions generated by itself and asked determine if each question engaging not. The prompt and the response of GPT-4 are shown in Table 12. As a result, GPT-4 classifies all the questions it generated as engaging. This suggests that GPT-4 may not be useful to score its own generated ques- tions, and motivates learning external model to score the questions, such as proposed engaging question classifier.
B Latency Analysis
Note that this finding aligns with the statement, "we observe that GPT-4 assigns significantly higher scores to its own outputs compared to human rat- ings," stated in Dettmers et al. (2023).
Our goal is to develop lightweight models that can run on mobile devices. To examine the applicability of our proposed model FDT5 and the baselines, we measure and report the latency of MVQG-VV VL- T5, T5-Large, T5-Base, T5-Tiny, and our proposed FDT5 in Table 15. Each inference and yost-filtering time is computed by averaging over 300 trials to reduce the variance.
D Effect of Incorporating Address Information
We are using the address the main cue for GPT- 4 to retrieve some information regarding those places. While it is possible to produce general, non- location-specific questions based on hand-crafted templates, we found that GPT-4 can certainly pro- duce location-specifie questions that require knowl- edge of specific locations, which can potentially be more engaging. Some examples are presented as follows.
Latency (sec)
Loading Model
Inference
Post-Filtering
MVQG-VL-TS
7.09
6.38
N/A
TS-Large
12.79
10.04
N/A
TS-Base
10.34
5.9
N/A
T5-Tiny
2.02
N/A
2.27
3.92
Table 15: Latency Testing of the trained models.
The results show that FDT5 and T5-Tiny, with the same model architecture and the same number of parameters, enjoy significantly reduced time for loading models and running inference. The post-filterigg phase of FDT5 takes 3.92 seconds
Did you know that Fort Duquesne Boulevard is named after the historic Fort Duquesne, which was a key location during the French and Indian War? Have any of you studied that period in history?

---

Prompting GPT-4 for Self-Evaluation
You will be given 10 questions. Please classify this question inlo engaging and non-enguging questions.
Did you know that Pittsuurgh is known the City of Bridgesdue to its 446 bridges?
Can you spot any nearby bridges from this location?
2. The tall buildings on the south side are a part of Pittshurgh's beautiful skyline. Can anyone name some of the prominent buildings that make up the skyline?
As we pass by these parked cars on the north side, can you guess the most popular car brund in Pittsburgh based on what you see?
4. Pittsburgh has rich history of steel production. earning it the nickname Steel Caty.Can you see any indications of this industrial past the architecture of the buildings around us?
The traffic light at the empty intersection on our east might make you wonder ahout the city's 's traffic patterns. What do you think contributes to the current flow of traffic in this area?
Pittsburgh known for its vibrant arts scene. Can anyone spot a nearby gallery or theater we drive along these city streets?
7. As we pass by the buildings on the west, let's 's talk about the city's 's diverse architectural styles. Can you identify diffcrent styles just by looking at their exteriors?
Pittshurgh also famous for rich sports culture, with successful teams in foothall, basehall, and hockey. Can you spot any sports fans or themee establishments in our surroundings?
9. The city has growing lech scene, with companies like Google and Uber opening offices here. Can you any tech companies or startups we drive through these streets?
10. Pinthurgh is known for its friendly locals, often called yinzersdue to their unique accent. Have you encountered any friendly
yinzers during your visit so far, and what has been your favorite interaction with them?
GPT-4 Response
1. Lingaging
2. Engaging
Engagiigg
Fingaging
5. Fingaging
Engaging
Engaging
8. Ennaaging
9. Finaaging
10. Engaging
Table 12: GPT-4 Self Evaluation Results.
The city of Pittsburgh is known for its numer- ous bridges. How many bridges do you think are in the city, and why do you think there are so many?
Address: 1250 Penn Ave, Pittsburgh, PA
15222, USA
Generated question: "As we drive along 1250 Penn Ave, there any upcoming events, festivals, or celebrations in the area that you't 'd like to learn more about?"
The 59th Street Bridge, also known the Ed Koch Queensboro Bridge, connects Manhat- tan to Queens. Can you identify any famous movies or TV shows that have featured this iconic bridge?
Observation: GPT-4 knows this location has hosted several events in the past and therefore asks about upcoming events.
Address: 333 Boulevard of the Allies, Pitts- burgh, PA 15222, USA
We cannot obtain these questions by simply re- placing the city name other questions, nor should these questions be asked at a different location.
Generated question: Did you know that the Boulevard of the Allies is named honor the Allies of World War I? What do you think about the significance of this historical connection?
Moreover, while generating question with an address can increase the vocabulary size and av- erage question length by simply inserting the ad- dress into the question, we still observe some ques- tions that are generated based on the knowledge extracted by GPT-4 according to the address. We provide some example questions as follows.
Observation: Based on knowing the his- tory of the Boulevard of the Allies, GPT- asks about World War I.

---

E Importance of Incorporating Visual Input and Learning to Generate Questions
F Implementation Details
F.1
GPT-4 Parameters and Expenses
Setup. When generating our proposed dataset us- ing GPT-4, OpenAI API, with 0.7 temperature and 0.1 pres- ence penalty.
Since it is possible to generate questions solely based the fetched address, we aim to further analyze the effect of employing visual inputs to produce questions. Also, discussed in Section D, we aim to quantitatively compare generating ques- tions by our learned model and producing questions using general hand-crafted templates. To this end, we labeled 100 questions generated by FDTS based on the following two criteria:
the model "gpt4" listed in the
Expenses. average, each request of our task uses up around 500 tokens, costing us around $0.001 USD. In total, generating the dataset and experimenting with it cost around $150- 200 USD.
F.2
T5 and FDT5
The generated questions contain visual infor- mation (w/ vis) or not (w/o vis)
Input. Similar to GPT-4, the T5 models and FDT5 take image captions input. The text input the T5 models and FDT5 is modified from the chat prompt provided to GPT-4. Specifically, we prepend "generate questions:" prefix to each input. resulting in the model input as: generate questions: You are currently driving on [Street Address). On your North, [Image Caption]. On your East, [Im- age Caption]. On your South, [Image Caption). On your West, [Image Caption].
The generated questions are based some templated (templated) or a learned language model (leared
Specifically, we went through each question and
Determined contains visual information (e.g., describing surroundings). If so, this question is labeled as w/ vis; otherwise, it is labeled w/o vis.
Implementation. We adopt the basic pre-trained T5 models available on the Hugging Face platform. Training, During the training, we use questions for each LocaVQG task tuple. We train each model for 20 epochs with a learning rate of 10-4
Decided it can be generated based on some templates (e.g., the city's name can be re- placed with another city and the question still makes sense). If so, this question is labeled as templated; otherwise, it is labeled learned.
F.3 VL-T5
Input. The input of VL-T5 contains the following prompt prefix, street address, visual embeddings, and visual semantic groundings.
Then, we analyze the engagement score and di- versity each group of questions. Regarding en- gagement scores, the questions containing visual information (w/ vis) achieve an average score of 4.007, slightly outperforming the questions without visual information (w/o vis) with an average score of 3.915, indicating that the visual-related ques- tions may be more engaging. On the other hand, as the reviewer anticipated, the templated questions have a higher average score of 4.002 compared to learned questions (3.901).
Prompt prefix. The prompt prefix is generate question: which guides the model to generate questions with the instruction.
Street address. The street address is the spe- cific street address the pictures that is ver- balized, e.g., You are currently driving in Penn Avenue, Pinssurgh.
Then, we analyzed the diversity of each group of questions and found that the learned questions are more diverse (with pairwise Cosine similar- ity score of 0.3614), outperforming the templated questions with pairwise Cosine similarity score of 0.3995.
Visual embeddings and visual semantic groundings. We extract the visual embed- dings from the whole image and the im- age regions with Faster-RCNN (Ren et al., 2015). Also, adopted the grounded situ- ation recognition (GSR) (Pratt al., 2020) model extract information on the sequence of images to understand the semantics. Prefix of the directions the images is also added, e.g., North: /Visual embeddings]
In conclusion, we believe that generating engag- ing and diverse questions still requires incorpo- rating visual inputs with a well-learned language model.

---

Training. During the training, we used the pre- trained baselines presented in (Yeh al., 2022), specifically the adapter. We train the model for 30 epochs with learning rate of 10 10-6. We also employ gradient accumulation steps of and warm- up steps of 10.
worker is provided with the street address and 4 streetview images of this location. Then, the worker required to rate the generated question ac- cording to 5 evaluation metrics: Engagement, Nat- uralness, Coherence, Common Sense, and Ground- ing. The descriptions of the metrics are as follows.
F.4 Engaging Question Classifier
Engagement This is an engaging question for this set of photos. You would want to answer respond to this question
The engaging question classifier is trained the questions from the two datasets: SQuaD (20239 questions) and MVQG (31098 questions). The engaging question classifier is a BERT-based clas- sifier with 110M parameters. We train the classifier to classify the questions sampled from SQuaD as non-engaging and those sampled from MVQG as engaging for 10 epochs. We use the ADAM opti- mizer with learning rate of 10-5
Naturalness Given the pictures and location that you are in, it is natural to ask this ques- tion.
Coherence This question asks something about the description and information that could be found in the image relevant to the location.
Some example questions are follows:
Why is that man playing billiards by himself?
Common Sense This question provide enough Common Sense. The question asks something that makes sense according our common sense.
(Engaging)
How did you celebrate your last birthday party? (Engaging)
Grounding This question mentions the essen- tial objects or information of the images, and it is mentioned the right direction or talking about the location.
What document was signed in 1999? (Non- engaging)
What did John Milton do for world literature?
(Non-engaging)
This AMT interface is shown Figure 5
We report the performance of the learned engag- ing question classifier on the training (train), vali- dation (val), and testing (test) sets in Table 14. The accuracy evaluates if the classifier can correctly distinguish the questions in the MVQG dataset from those in the SQuaD dataset. The results show that the trained engaging question classifier can accurately distinguish the questions from the two datasets.
G.2 Collecting Human Generated Questions In order to compare GPT-4 to humans regard- ing the ability to produce engaging questions from location-aware information, we crowdsource human human-generrn generated questions from human annotators on AMT. We use three-stage questions to collect questions follows.
Question 1. Pick 1 (or more) pictures that you want to focus on, and write down the object (or event) that stands out the most for you.
train
valid
test
Accuracy 99.9%
98.9%
99.0%
Question 2. Please describe the object (or event).
Table 14: Performance of the Learned Engaging Question Classifier.
Question 3. Please write question based on it.
Amazon Mechanical Turk Details
This AMT interface is shown in Figure 6. For each LocaVQG task tuple, we require three AMT workers to write question, and then take the best question from the three questions. We present some questions produced by the workers in Table 14.
G.1 Human Evaluation Details
Section 5.3 conducts human evaluation on AMT to compare questions generated by all models, GPT-4, and humans. For each generated question, each

---

There is big strong building around Do you want try come and visit it?
Can you give me some tips to how to drive like you?
what do you think this building is used for?
where are your coming from and which place are going to visit?
Look at the building around us, what do you think about the building? find it looking really sturdy
On the east side, you could see a cargo with few people around it. What do you think is inside the cargo?
Have you ever wondered what beyond that colossal vertical gateway adorned with glass windows elegance?
I wonder if could it be gallery perhaps stock exchange?
is Pittsuurgh ya know, the Steel Cityihe possibilitics are endless!
What you think about Garland Avenue compared to other streets Orlando?
Table 14: Human-Generated Questions.

---

you puukitk rug diny twee phace cllveng
den ogen
Imagin you driing in your strentiee of respecively
Given the location that you are in. it netural ask question.
Notice the truck parked front building to our north. Can anyone guess many trucks operating city Pittsburgh?
Haln
bucacler
images anyone guess
right
parked building
how many trucks are operating in city like Pittsburgh?
3. This question is coherent the description and information that could be found in the image or relewant to the location
Notice the truck parked in front of the building to our north. Can anyone guess how many trucks are operating in city like Pittsburgh?
Palr te quily tp Silanna anciocome
This question provide enough Common Sense, The queston ask something that make sence according our common serse Notice the truck parked in front of the building to our north. Can anyone guess many trucks are operating city Pittsburgh?
qurty doneemeeeee peeeenn wnterneeng reg.ggi.e
engaging question
Notice the truck parked front building north. Can anyone guess how many trucks are operating in city like Pittsburgh?
Figure 5: Human Evaluation AMT Interface.

---

will oere sepures sen ahe munuee thi Serevvss
plaer ane dee and
Vetural Questin canticn sooe
AbO ebtic ietieed phett colecs.
Qencue pepoliioo
ligeging Q Cuedtos suseer upenc
Locatin thommaiie more segeciig cerdice
scw you wore agpelidd
tat pore the plase
ploon
Please answeer the following questions,
Qo. (ju non] pictures you foous wrtke down ofject moeed) than counde mout you "Oiper Pearpe makin
0r. Please describe the object (or everd in sentences
Figure 6: Location-aware Question Collection AMT Interface.


Reviews:
[{"review_id": "3cn2ojun3q", "reviewer": "Reviewer_BRvQ", "summary": "", "questions": "Is it necessary to have an externally trained engaging scorer? Wouldn't it be feasible to simply input the generated questions back into GPT-4 for scoring?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n### Responses to Questions\n\n> The thoughts of prompting LLMs to collect data has limited novelty.\n\nAs pointed out by the reviewer, prompting LLMs to collect data has been recently explored in [1, 2]. While [1] focuses on using LLM to generate instruction data for instruction tuning, [2] aims to leverage LLM as a teacher in the distillation process. To the best of our knowledge, **our work is the first to prompt LLMs to generate data for visual question generation (VQG) tasks**. Specifically, this work proposes a dataset generation pipeline that leverages GPT-4 for location-aware VQG. The contributions of the proposed dataset generation pipeline include converting images to text with an image captioning model, reverse geocoding to obtain an address, designing effective system and chat prompts, and utilizing an engaging classifier to filter generated questions. We believe these contributions are novel and valuable.\n\n> The effectiveness of the engaging question classifier in the study seems to be uncertain. ...  Also, There is a lack of evaluation regarding the engagement of the generated question.\n\nWe propose to utilize the proposed engaging question classifier to filter out unengaging questions generated by GPT-4 when generating the dataset and filter out unengaging questions produced by our proposed question generation method FDT5. **We extensively evaluate and discuss the effectiveness of this proposed engaging question classifier in Section 5.5.1 with the results presented in Table 8 and Table 9.** We provide an overview of the results as follows.\n- **Dataset generation pipeline**: The results presented in Table 8 show that the model learning from the filtered dataset achieves better performance in terms of both the Engagement and Overall scores in human evaluation, compared to the model learning from the unfiltered dataset. This justifies the efficacy of employing the proposed engaging question classifier in the dataset generation pipeline.\n- **Filtering FDT5 generated questions**: The results presented in Table 9 compare the questions produced by our proposed method FDT5 with or without filtering out the generated questions deemed unengaging by the proposed engaging classifier. With the proposed filtering procedure, the generated questions achieve higher Engagement and Overall scores in human evaluation. This verifies the effectiveness of employing the proposed engaging question classifier during inference.\n\nPlease refer to section 5.1.1 for detailed discussions on the effectiveness of the engaging question classifier.\n\n> There is a lack of evaluation regarding the performance of the trained engaging question classifier.\n\nWe thank the reviewer for the suggestion. We report **the performance of the learned engaging question classifier on the training (train), validation (val), and testing (test) sets** as follows. The accuracies evaluate if the classifier can correctly distinguish the questions in the MVQG dataset from the questions in the SQuaD dataset. We will revise the paper and incorporate this information into the paper.\n\n| | train | val | test | \n| --- | --- | --- | --- |\n| Accuracy | 99.9% | 98.9% | 99.0% |\n\n> Is it necessary to have an externally trained engaging scorer? Wouldn't it be feasible to simply input the generated questions back into GPT-4 for scoring?\n\nWe thank the reviewer for the suggestion. We did not use GPT-4 to determine whether the questions generated by itself are engaging or not because recent studies have shown that **LLMs tend to be biased and prefer their own generated answers**. This is evident from [3], which states that “we observe that GPT-4 assigns significantly higher scores to its own outputs compared to human ratings.\"\n\nTo empirically verify the above statement, we have conducted an additional experiment that inputs the generated questions back into GPT-4 for scoring (i.e., decide if each generated question is engaging or not). We provided GPT-4 with ten questions generated by itself and asked it to determine if each question is engaging or not. The prompt and the GPT-4 output are shown as follows. As a result, **GPT-4 classifies all the questions it generated as engaging**. This suggests that GPT-4 may not be useful to score its own generated questions, and motivates learning an external model to score the questions, such as our proposed engaging question classifier. \n\nWe thank the reviewer for this suggestion. We will incorporate this additional experiment and discussion into our revision to strengthen our paper.\n\nPrompt to GPT-4\n```\nYou will be given 10 questions. Please classify this question into engaging and non-engaging questions.\n\n1. Did you know that Pittsburgh is known as the \\\"City of Bridges\\\" due to its 446 bridges? Can you spot any nearby bridges from this location?\n2. The tall buildings on the south side are a part of Pittsburgh's beautiful skyline. Can anyone name some of the prominent buildings that make up the skyline?\n3. As we pass by these parked cars on the north side, can you guess the most popular car brand in Pittsburgh based on what you see?\n4. Pittsburgh has a rich history of steel production, earning it the nickname \\\"Steel City.\\\" Can you see any indications of this industrial past in the architecture of the buildings around us?\n5. The traffic light at the empty intersection on our east might make you wonder about the city's traffic patterns. What do you think contributes to the current flow of traffic in this area?\n6. Pittsburgh is known for its vibrant arts scene. Can anyone spot a nearby art gallery or theater as we drive along these city streets?\n7. As we pass by the buildings on the west, let's talk about the city's diverse architectural styles. Can you identify different styles just by looking at their exteriors?\n8. Pittsburgh is also famous for its rich sports culture, with successful teams in football, baseball, and hockey. Can you spot any sports fans or sports-themed establishments in our surroundings?\n9. The city has a growing tech scene, with companies like Google and Uber opening offices here. Can you see any tech companies or startups as we drive through these streets?\n10. Pittsburgh is known for its friendly locals, often called \\\"yinzers\\\" due to their unique accent. Have you encountered any friendly yinzers during your visit so far, and what has been your favorite interaction with them?\n```\n\nGPT-4’s Response\n```\n1. Engaging\n2. Engaging\n3. Engaging\n4. Engaging\n5. Engaging\n6. Engaging\n7. Engaging\n8. Engaging\n9. Engaging\n10. Engaging\n```\n\n### References\n\n- [1] Wang et al. “Self-Instruct: Aligning Language Models with Self-Generated Instructions” ACL 2023\n- [2] Hsieh et al. “Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes” ACL 2023\n- [3] Dettmers et al. “QLoRA: Efficient Finetuning of Quantized LLMs” arXiv 2023\n\n### Conclusion\n\nWe are incredibly grateful to the reviewer for the detailed and constructive review. We believe our responses address the concerns raised by the reviewer. Please kindly let us know if there are any further concerns or missing experimental results that potentially prevent you from accepting this submission. We would be more than happy to address them if time allows. Thank you very much for all your detailed feedback and the time you put into helping us to improve our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The acceptance of this article could potentially facilitate the application of LLMs within specific vertical domains. This work demonstrates a comprehensive pipeline for the application of LLMs in a specific vertical field, which involves annotating data by prompting LLMs, filtering data for desired feature (engagement here), and training more time-efficient models.", "reasons_to_reject": "1. The thoughts of prompting LLMs to collect data has limited novelty.\n2. The effectiveness of the engaging question classifier in the study seems to be uncertain. There is a lack of evaluation regarding the performance of the trained engaging question classifier. Also, There is a lack of evaluation regarding the engagement of the generated question.", "questions_to_authors": "Is it necessary to have an externally trained engaging scorer? Wouldn't it be feasible to simply input the generated questions back into GPT-4 for scoring?", "paper_topic_and_main_contributions": "1. This work introduces a novel task to generate engaging questions from data relevant to a particular geographical location. This task can be applied to in-car intelligent assistant system to raise drivers' attention and reduce their fatigue by produce location-aware engaging questions.\n2. This work introduces a location-aware VQG dataset by prompting GPT-4. To improve the engagement of the dataset, this work proposes to learn a BERT-based engaging question classifier to do filtering.\n3. This work proposes to use reverse geocoding to handle GPS coordinate and image caption model to deal with four streetview images to incorporating location information into the model.\n4. This work proposes to train a lightweight filtered distilled T5-tiny from the generated dataset to fit this task on an edge device, outperforming all the lightweight models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bHWuagISCZ", "reviewer": "Reviewer_Gg8Z", "summary": "", "questions": "- How are the images selected from Google Streetview?\n- How do you select which question to use as the ground truth for fine-tuning? Is this randomly selected?\n- Is there a way for you to measure the diversity of the questions you're generating? (see above point for more details)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n### Responses to Questions\n\n> I'm not sure how valuable this task will be for the NLP community. I agree that it's an important and challenging task, but I'm not sure that it's a benchmark that a lot of people will be working on.\n\nWe appreciate the reviewer for recognizing the importance and the challengingness of the task introduced in our work. We aim to leverage the rapid advancement in NLP to revolutionize how we interact with technology and build systems that enhance human ability. To this end, our work makes the initial attempt to develop an in-car visual question generation system that can ask engaging location-aware questions to keep drivers awake. **We believe our work will inspire further innovation in various visual question generation tasks and a wide range of systems that promote road safety.**\n\n> I'm also not entirely convinced that having the visual part of the pipeline actually helps. I wouldn't be surprised if just using the locations can provide enough information for the model to generate something informative. For instance, a lot of the \"engaging questions\" in Table 1 don't refer to the images at all and can be generated solely by using the questions.\n\nWe thank the reviewer for raising this concern. We acknowledge that the questions presented in Table 1 are not particularly relevant to the visual part (i.e., streetview images) of LocaVQG task tuples. This is because we only aimed to compare engaging and unengaging questions in Table 1. **We provide several GPT-4 generated questions relevant to visual inputs as follows.** \n- \"As we look towards the south, can you guess the purpose of this brick building with cars parked in front? Perhaps an office building, a restaurant, or something else?\"\n- \"What do you think the history of this western city street might be? Pittsburgh has a rich history, and every street has a unique story to tell.\"\n- \"What types of events or festivals might take place in this park throughout the year?\"\n\n**The above questions refer to a building, a park, or a street that could be seen from the images, which are based on the visual part of the LocaVQG task tuples.** We will revise Table 1 to include these questions to avoid confusion. We will also include a discussion on the importance of employing visual cues for visual and language tasks [1, 2] in the revision.\n\n> I'm not very convinced with the engaging question classifier. A lot of the \"engaging\" questions in Table 1 seem very generic (e.g. about food, architecture, etc.) that you can just swap out the name of the city and it's going to be a perfectly valid Barnum question for any location you want. \n\nWe agree with the reviewer that one can potentially generate some questions with some hand-crafted templates. However, designing such templates requires expertise and may be time-consuming. Instead, our work proposes to minimize human effort by prompting GPT-4. While, as pointed out by the reviewer, some questions in Table 1 are generic, **GPT-4 can certainly produce location-specific questions**. We provide some example questions sampled from the proposed dataset below. \n- “Did you know that Fort Duquesne Boulevard is named after the historic Fort Duquesne, which was a key location during the French and Indian War? Have any of you studied that period in history?”\n- “The city of Pittsburgh is known for its numerous bridges. How many bridges do you think are in the city, and why do you think there are so many?”\n- “The 59th Street Bridge, also known as the Ed Koch Queensboro Bridge, connects Manhattan to Queens. Can you identify any famous movies or TV shows that have featured this iconic bridge?”\n- “The plaza to the south is often used for various events and activities throughout the year. Have any of you attended or heard about any events that take place at PPG Place?”\n\n**We cannot obtain these questions by simply replacing the city name in other questions, nor should these questions be asked at a different location.** This highlights GPT-4’s ability to generate location-specific and diverse questions. We will revise the paper to clarify this and incorporate these questions into Table 1 to avoid confusion.\n\n> Is there a way for you to measure the diversity of the questions you're generating?\n\nWe agree with the reviewer that examining the diversity of the generated questions is indeed crucial. **We have analyzed the diversity of the generated questions from the following perspectives.**\n- **Comparison to an existing VQG dataset (MVQG)**: Therefore, we have compared our proposed dataset to an existing visual question generation dataset (MVQG) [2], and the results are presented in Table 5 (Question Quality Comparison with MVQG). The comparisons are based on vocabulary size, average sentence length, syntactic complexity (Yngve score normalized by the sentence length), percentage of abstract terms, and average term depth. **The results show that the proposed dataset contains significantly more diverse and sophisticated questions compared to MVQG.**\n- **Question type analysis**: While Table 3 shows the top 15 most frequent trigrams of generated questions, we have performed an additional trigram analysis during the rebuttal period to examine the diversity of the generated questions. In particular, **we followed [2] and identified 2437 question types among our 35K generated questions**. This highlights the diversity of the generated questions.\n- **Pairwise cosine similarity**: Inspired by [3], which computes the average pairwise cosine similarity between each pair of generated questions encoded by a sentence transformer (multi-qa-MiniLM-L6-cos-v1 [4]) in a dataset, we have performed this evaluation on our generated dataset. We obtained an average cosine similarity of 0.1698, indicating that **the generated questions are not highly correlated and therefore ensuring the diversity of our proposed dataset.**\n\nWe will incorporate the additional analysis into our paper. While we have strived for comprehensive diversity analysis during the rebuttal period, if the reviewer has other specific diversity metrics in mind, we will be happy to use them to evaluate our proposed dataset and report them in the revision.\n\n> I'm also not entirely convinced with the results in Table 6. T5-Large, T5-Base, and T5-Tiny all perform around the same\n\nWe agree with the reviewer. The performance of the three T5 models with different sizes (T5-Large: 3.87, T5-Base: 3.84, T5-Tiny: 3.85) is very similar. Therefore, this motivates us to develop a method based on T5-Tiny. We would like to emphasize that our proposed method **FDT5 outperforms all the T5 models** with an overall score of 3.98.\n\n> … the gap between the GPT-4 (4.04) and T5-Tiny (3.85) is very small\n\nTo answer the reviewer’s question, we have performed a t-test statistical analysis on the performance of T5-Tiny and GPT-4. As a result, we obtained a p-value of 0.1459. This indicates **there is statistical significance in the difference in the performance of T5-Tiny (3.85) and GPT-4 (4.04)**. This motivates us to develop a method that performs more comparably to GPT-4 and has significantly fewer parameters.\n\n> I would have wanted to see a discussion on speed/latency. This usually isn't an issue for most papers, but this paper claims one of its main points to be that it can work on edge devices such as mobile phones, which opens up the whole can of worms on latency/inference time. For instance, they mention that they filter out poor questions until the model produces a question that's classified as engaging. I'm wondering how many times the model generated a rejected question before it generates one that is accepted.\n\nWe thank the reviewer for raising this concern about speed and latency. We also believe that incorporating this information into our paper will strengthen the paper.  **We have measured and reported the latency of MVQG-VL-T5, T5-Large, T5-Base, T5-Tiny, and our proposed FDT5 below.** Each inference and post-filtering time is computed by averaging over 300 trials to reduce the variance.\n\n| Latency | Loading Model (sec) | Inference (sec) | Post-filtering (sec) |\n| :---: | :---: | :---: | :---: |\n| MVQG-VL-T5 | 7.09 | 6.38 | N/A |\n| T5-Large | 12.79 | 10.04 | N/A |\n| T5-Base |  10.34 | 5.9 | N/A |\n| T5-Tiny |  3.89 | 2.02 | N/A |\n| FDT5 |  4.25 | 2.27 | 3.92 |\n\nThe results show that FDT5 and T5-Tiny, with the same model architecture and the same number of parameters, enjoy a significantly reduced time for loading models and running inference. The post-filtering phase of FDT5 takes 3.92 seconds on average, indicating that the engaging question classifier requires FDT5 to perform 1.73 additional inference trials for each LocaVQG task on average. Note that this post-filtering phase can be shut down for latency-critical scenarios, and FDT5 without post-filtering still outperforms T5-Tiny in human evaluation, according to Table 6 and Table 9.\n\nWe will incorporate the results and the discussion above in the revision.\n\n> How are the images selected from Google Streetview?\n\nThe Google Streetview dataset contains locations that are very close to each other. Therefore, we subsampled the dataset by picking one coordinate from every three adjacent coordinates. Each coordinate contains five directional images: North, East, South, West, and Up (upward view). We discarded the upward view and used the remaining four directional images to form the images in a LocaVQG tuple.\n\n> How do you select which question to use as the ground truth for fine-tuning? Is this randomly selected?\n\nYes. At each training iteration, we first randomly sample a batch of LocaVQG tuples from the dataset. Then, we randomly sample a question from each LocaVQG tuple as the ground truth question for training.\n\n> Line 406 -- 3.98 instead of 3.88? Or is the table wrong?\n\nWe thank the reviewer for pointing out this error. It should be 3.98. We will revise the paper and fix it.\n\n### References\n\n- [1] Huang et al., “Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization” INLG 2023\n- [2] Yeh et al., “Multi-VQG: Generating Engaging Questions for Multiple Images” EMNLP 2023\n- [3] Schwenk et al. “A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge” arXiv 2022\n- [4] [multi-qa-MiniLM-L6-cos-v1 on Hugging Face](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1)\n\n### Conclusion\n\nWe are incredibly grateful to the reviewer for the detailed and constructive review. We believe our responses address the concerns raised by the reviewer. Please kindly let us know if there are any further concerns or missing experimental results that potentially prevent you from accepting this submission. We would be more than happy to address them if time allows. Thank you very much for all your detailed feedback and the time you put into helping us to improve our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They propose a new task (location-based VQG), and I think these new tasks are nice for the community especially now that people are excited about the capabilities of these new language models. In terms of the applications track, I also think this is a well-done application-based paper because it's both well-motivated in the real world (useful for drivers), as well as interesting from a research standpoint.\n\nThey do a distillation onto a smaller model, which is getting quite popular these days with large models. Usually I would be wary of simply distilling large models as a research idea, but in this case, I think it's justified because the authors are using a very tiny model. \n\nThorough analysis of the dataset. Evaluations make sense. Nice to see that they have human evaluations since automatic evaluations wouldn't really mean much for this task. The paper is generally clear and well-written.", "reasons_to_reject": "I'm not sure how valuable this task will be for the NLP community. I agree that it's an important and challenging task, but I'm not sure that it's a benchmark that a lot of people will be working on. I'm also not entirely convinced that having the visual part of the pipeline actually helps. I wouldn't be surprised if just using the locations can provide enough information for the model to generate something informative. For instance, a lot of the \"engaging questions\" in Table 1 don't refer to the images at all and can be generated solely by using the questions.\n\nI'm not very convinced with the engaging question classifier. A lot of the \"engaging\" questions in Table 1 seem very generic (e.g. about food, architecture, etc.) that you can just swap out the name of the city and it's going to be a perfectly valid Barnum question for any location you want. I think perhaps a test for diversity could be helpful here. \n\nI'm also not entirely convinced with the results in Table 6. T5-Large, T5-Base, and T5-Tiny all perform around the same, and the gap between the GPT-4 (4.04) and T5-Tiny (3.85) is very small, which makes me somewhat question whether the human evaluations are actually suitable for this task or if humans are just poor at scoring these questions.\n\nI would have wanted to see a discussion on speed/latency. This usually isn't an issue for most papers, but this paper claims one of its main points to be that it can work on edge devices such as mobile phones, which opens up the whole can of worms on latency/inference time. For instance, they mention that they filter out poor questions until the model produces a question that's classified as engaging. I'm wondering how many times the model generated a rejected question before it generates one that is accepted.", "questions_to_authors": "- How are the images selected from Google Streetview?\n- How do you select which question to use as the ground truth for fine-tuning? Is this randomly selected?\n- Is there a way for you to measure the diversity of the questions you're generating? (see above point for more details)", "paper_topic_and_main_contributions": "The paper proposes a new task called location-aware visual question generation (LocaVQG), which is basically visual question generation conditioned on a given location. They motivate this by citing how engaging drivers in conversation/QA can reduce fatigue and help the drivers stay attentive.\n\n**Dataset** -- The input consists of a location (e.g. \"PPG Place, Third Avenue, Pittsburgh, PA\") and 4 corresponding images (north, south, east, west cameras). The images were taken from the Google Streetview Dataset and the location is reverse-geocoded. Their dataset contains 3,759 such examples.\n\n**Methodology** -- To generate engaging questions, they do the following: 1) caption each image to convert to caption. 2) generate a prompt given the location + 4 captions (north, south, east, west images) (e.g. \"You are driving in [...]. On your north, [...]. On your south, [...]. Please generate 10 engaging questions.\") 3) This prompt is then fed into GPT-4 to generate questions. 4) The questions are fed into an engaging question classifier to classify which questions are engaging.\n\n**Experiments and Evaluation** -- They train a T5-Tiny model (15.6M) by distilling from a T5-large model and from GPT-4. Evaluation is done through MTurk human evaluation (asking humans to evaluate Engagement, Coherence, etc. on a 1-5 scale), as well as through automatic evaluation metrics (BLEU, ROUGE, BERTScore, etc.) They show that their lightweight T5-Tiny model performs better than other lightweight models. Lastly, there is a thorough ablation study on the various parts of the pipeline/dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 406 -- 3.98 instead of 3.88? Or is the table wrong?"}, "extra_text": {}}, {"review_id": "tsQYWvLOsd", "reviewer": "Reviewer_BPM2", "summary": "", "questions": "A. Regarding the questions that urge the driver to pay attention to particular aspects of the surrounding area (e.g. the architectural style of the buildings). Couldn't they be considered dangerously distracting from the road conditions and traffic? Maybe the questions should be judged not only as \"engaging\" and \"not engaging\" but also as \"distracting\" and \"not distracting\". For example, a question about the local cuisine doesn't require the driver to take their eyes off the road and therefore is not as distracting as the \"architectural style\" question.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n### Responses to Questions\n\n> The paper is missing details of the engaging question classifier (dataset size and composition, model and training details).\n\nWe thank the reviewer for pointing this out. We provide the missing details as follows and will incorporate them into the revision. \n\nThe engaging question classifier is trained on the questions from the two datasets: SQuaD [1] (20239 questions) and MVQG [2] (31098 questions). The engaging question classifier is a BERT-based classifier with 110 M parameters. We train the classifier to classify the questions sampled from SQuaD as non-engaging and the questions sampled from MVQG as engaging for 10 epochs. We use the ADAM optimizer [3] with a learning rate of 1e-5. Some example questions are as below.\n- “Why is that man playing billiards by himself?” (engaging)\n- “How did you celebrate your last birthday party?” (engaging)\n- “What document was signed in 1999?” (non-engaging)\n- “What did John Milton do for world literature?” (non-engaging)\n\n> The metrics used in the automatic evaluation (Section 5.4) rely on comparing the generated text to some \"gold standard\", ground truth text. In this case, it is not clear what is used as the ground truth. Questions generated by GPT-4 for the original dataset? But if some model generates a different question from what GPT-4 has generated previously, it does not mean at all that the new question is any worse. This section should be extended with a more detailed description of how the automatic evaluation was conducted and how the authors interpret its results given the concerns above.\n\nGiven a LocaVQG task tuple sampled from our dataset, the automatic evaluation compares a generated question with a sampled GPT-4 generated question from this tuple. More details on the automatic evaluation are described as follows. \n- **BLEU** and **ROUGE** compare two questions based on the exact wording; therefore the scores may not exactly reflect the quality of generated questions.\n- **BLEURT** and **BERTScore** use an ML model to compare two questions, which may better reflect the similarity in semantics.\n\nWe agree with the reviewer that the automatic evaluation may not precisely reflect the quality of the generated questions. Therefore, our work uses human evaluation as our main evaluation metric. Yet, **the automatic evaluation can still indicate how well the trained models fit the dataset distribution**. As a result, we believe it is still informative to provide the automatic evaluation results. We will revise the paper to incorporate this information.\n\n> At least some of the results reported in Table 10 seem to be trivial. Adding concrete address information to the prompt leads to named entities (location names) being generated in the questions, which trivially increases the vocabulary size and average question length (the latter by simply adding more input that questions are based on). These findings also do not necessarily relate to the quality of the resulting questions (how engaging/relevant they are, etc.).\n\n**We provide the experimental results presented in Table 10 because we aimed to explicitly differentiate our work from MVQG [2].** Specifically, MVQG generates questions from a set of images, while our work generates questions from a set of images and a GPS coordinate. Therefore, we believe it is still informative to provide such a study for readers familiar with MVQG and its extensions, even if the results may seem trivial.\n\nMoreover, while generating a question with an address can increase the vocabulary size and average question length by simply inserting the address into the question, **we still observe some questions that are generated based on the knowledge extracted by GPT-4 according to the address**. We provide some example questions as follows. \n- **Address**: 1250 Penn Ave, Pittsburgh, PA 15222, USA\n    - **Generated question**: “As we drive along 1250 Penn Ave, are there any upcoming events, festivals, or celebrations in the area that you'd like to learn more about?”\n    - **Observation**: GPT-4 knows this location has hosted several events in the past and therefore asks about upcoming events.\n- **Address**: 333 Boulevard of the Allies, Pittsburgh, PA 15222, USA \n    - **Generated question**: Did you know that the Boulevard of the Allies is named to honor the Allies of World War I? What do you think about the significance of this historical connection?\n    - **Observation**: Based on knowing the history of the Boulevard of the Allies, GPT-4 asks about World War I.\n\n>  Regarding the questions that urge the driver to pay attention to particular aspects of the surrounding area (e.g. the architectural style of the buildings). Couldn't they be considered dangerously distracting from the road conditions and traffic? Maybe the questions should be judged not only as \"engaging\" and \"not engaging\" but also as \"distracting\" and \"not distracting\". For example, a question about the local cuisine doesn't require the driver to take their eyes off the road and therefore is not as distracting as the \"architectural style\" question.\n\nWe thank the reviewer for pointing this out. Our work makes the very first attempt to develop an in-car visual question generation system aiming to alleviate driver’s fatigue by asking engaging questions to initiate conversations. During our preliminary study, we found that generating location-aware, engaging questions is particularly challenging, and therefore this work aims to tackle this problem. \n\nWe agree with the reviewer that future works along this line should consider the “distractingness” of generated questions. In particular, developing evaluation metrics to determine if a question would distract a driver and devising methods that can produce engaging yet non-distracting questions are promising and interesting research directions. We will revise the paper and incorporate this into our limitations and future work.\n\n> Table 2 caption, lines 344, 382, 390: missing a full stop at the end.  \n\n> 434: \"the two techniques employed in FDT5\" -- unclear, better to explicitly state what these techniques are\n\nWe thank the reviewer for the suggestions. We will thoroughly revise the paper to fix them.\n\n> References for GPT-4, BERTScore, ROUGE-2 should be given at their first mentions in the paper (currently they appear much later than this). References for BERT, GPT-3 should also be given when they are mentioned in the paper.\n\nWe thank the reviewer for this suggestion. We will revise the paper to properly provide the references.\n\n### References\n\n- [1] Rajpurkar et al. “SQuAD: 100,000+ questions for machine comprehension of text” EMNLP 2016\n- [2] Yeh et al. “Multi-VQG: Generating engaging questions for multiple images” EMNLP 2022\n- [3] Kingma et al. “Adam: A Method for Stochastic Optimization” ICLR 2015\n\n### Conclusion\n\nWe are incredibly grateful to the reviewer for the detailed and constructive review. We believe our responses address the concerns raised by the reviewer. Please kindly let us know if there are any further concerns or missing experimental results that potentially prevent you from accepting this submission. We would be more than happy to address them if time allows. Thank you very much for all your detailed feedback and the time you put into helping us to improve our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed task is novel and interesting, with potentially important real-world applications. The presented dataset and method, which shows a strong performance, can serve as a benchmark and baseline for future research.", "reasons_to_reject": "1. The paper is missing details of the engaging question classifier (dataset size and composition, model and training details).\n2. The metrics used in the automatic evaluation (Section 5.4) rely on comparing the generated text to some \"gold standard\", ground truth text. In this case, it is not clear what is used as the ground truth. Questions generated by GPT-4 for the original dataset? But if some model generates a different question from what GPT-4 has generated previously, it does not mean at all that the new question is any worse. This section should be extended with a more detailed description of how the automatic evaluation was conducted and how the authors interpret its results given the concerns above. \n3. At least some of the results reported in Table 10 seem to be trivial. Adding concrete address information to the prompt leads to named entities (location names) being generated in the questions, which trivially increases the vocabulary size and average question length (the latter by simply adding more input that questions are based on). These findings also do not necessarily relate to the quality of the resulting questions (how engaging/relevant they are, etc.). \n\n\nUPD AFTER REBUTTAL: I trust the authors will include the explanations regarding these 3 points (just like they did in the rebuttal) in the revised version, so I'm raising the scores relying on that.", "questions_to_authors": "A. Regarding the questions that urge the driver to pay attention to particular aspects of the surrounding area (e.g. the architectural style of the buildings). Couldn't they be considered dangerously distracting from the road conditions and traffic? Maybe the questions should be judged not only as \"engaging\" and \"not engaging\" but also as \"distracting\" and \"not distracting\". For example, a question about the local cuisine doesn't require the driver to take their eyes off the road and therefore is not as distracting as the \"architectural style\" question.", "paper_topic_and_main_contributions": "This paper presents a new task, location-aware visual question generation (LocaVQG), along with the related dataset and a model (FDT5). The task aims to generate engaging and relevant questions based on the geographic coordinates of the user's location and the images of the user's surroundings. The dataset for the task is collected by employing GPT-4 to generate such questions given a prompt that includes location information and automatically produced captions of the images. The resulting questions are further filtered to keep only the engaging ones. The authors also present a lightweight model to produce such questions given the same input data. The model FDT5 is a pre-trained T5-Large model, fine-tuned based on the dataset of questions generated by GPT-4, followed again by \"engagement\" filtering.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 2 caption, lines 344, 382, 390: missing a full stop at the end.\n\n434: \"the two techniques employed in FDT5\" -- unclear, better to explicitly state what these techniques are\n\nReferences for GPT-4, BERTScore, ROUGE-2 should be given at their first mentions in the paper (currently they appear much later than this). References for BERT, GPT-3 should also be given when they are mentioned in the paper."}}]

Author rebuttal:

Meta-review: The paper proposes the task of location based visual question generation. They use GPT-4 to produce diverse engaging questions. And then share studies on both the data generation as well as solving the problem.

The reviewers find the paper exciting and sound after the rebuttal. As one reviewer indicates, there are flaws but it's nonetheless interesting. The authors should include several points raised by the reviewers so as to improve the paper: clarifications on the prompting and filtering, metrics and eval changes suggested by reviewers, and question classifier details to name a few.
Decision: Accept-Main