GazeVQA: A Video Question Answering Dataset for Multiview Eye-Gaze Task-Oriented Collaborations
Authors: Muhammet Furkan ILASLAN
Chenan Song
Joya Chen
Difei Gao
Weixian Lei
Qianli Xu
Joo Hwee Lim
Mike Zheng Shou
Venue: EMNLP 2023 Main (2023)
Keywords: video question answering
human-robot collaboration

Abstract: The usage of exocentric and egocentric videos in Video Question Answering (VQA) is a new endeavor in human-robot interaction and collaboration studies. Particularly for egocentric videos, one may leverage eye-gaze information to understand human intentions during the task. In this paper, we build a novel task-oriented VQA dataset, called GazeVQA, for collaborative tasks where gaze information is captured during the task process. GazeVQA is designed with a novel QA format that covers thirteen different reasoning types to capture multiple aspects of task information and user intent. For each participant, GazeVQA consists of more than 1,100 textual questions and more than 500 labeled images that were annotated with the assistance of the Segment Anything Model. In total, 2,967 video clips, 12,491 labeled images, and 25,040 questions from 22 participants were included in the dataset. Additionally, inspired by the assisting models and common ground theory for industrial task collaboration, we propose a new AI model called AssistGaze that is designed to answer the questions with three different answer types, namely textual, image, and video. AssistGaze can effectively ground the perceptual input into semantic information while reducing ambiguities. We conduct comprehensive experiments to demonstrate the challenges of GazeVQA and the effectiveness of AssistGaze.

Gaze A Video Question Answering Dataset for Multiview Eye-Gaze Task-Oriented Collaborations
Muhammet Furkan Ilaslan 1.2 Weixian Lei' Qianli Xu Joo Hwee Lim?
Chenan Songl* Joya Chen'
Difei Gaot
Mike Zheng Shou'
SSow Lab, National University of Singapore
PInstitute for Infocomm Research, Agency for Science, Technology, and Research (A (A*STAR), Singapore (m. Furkanilaslar chenan song) (ong)@uu.ee nus nus. .uuu mike. zheng. shou@gma com (stuimf, quu, joohwee] @ aww.aaasaa..... www.aaa.rar.u........ edu.sg
Abstract
Question: Which component will be assembled next?
The usage of exocentric and egocentric videos in Video Question Answering (VQA) is a new endeavor human-robot interaction and col- laboration studies. Particularly for egocentric videos, one may leverage eye-gaze information understand human intentions during the task. In this paper, we build novel task-oriented VQA dataset, called VQA, for collabora- tive tasks where gaze information is captured during the task process. Vaze is designed with novel QA format that covers thirteen different reasoning types capture multiple aspects of task information and user intent. For each participant, GazeVQA consists of more than 1,100 textual questions and more than 500 labeled images that were annotated with the assistance of the Segment Anything Model. In total, 2,967 video clips, 12,491 labeled images, and 25,040 questions from 22 participants were included in the dataset. Additionally, inspired by the assisting models and common ground theory for industrial task collaboration, we pro- pose new AI model called AssistGaze that is designed to answer the questions with three different answer types, namely textual, image, and video. AssistGaze can effectively ground the perceptual input into semantic information while reducing ambiguities. We conduct com- prehensive experiments to demonstrate the chal- lenges of VQA and the effectiveness of AssistGaze².
Textual Answer (Multiple Choice)
Image Answer
A1: Hub cover bolts
A2: Casing bolts
A3: Input hub cover Output hub
Video Answer (Step2-6 video dipp was used) 00: t2: 00:30
TPV-enface
TPV-sideward
FPV-raw
Paze
Figure 1: Gaze dataset contains gaze info with ego/exocentric videos to answer the questions that have semantic knowledge about human-robot studies.
have achieved remarkable progress (Deng et al., 2009; He al., 2016; Russell al., 2008; Feichten- hofer al., 2019; Tapaswi et al., 2016; Krizhevsky et al., 2017; Yu et al., 2019). Many projects (Wu
et al., 2017; Gao et al., 2021; Sood et al., 2021; Antol et al., 2015; Kafle and Kanan, 2017: Lei et al., 2022; Gao et al., 2022) in recent years have shown promising developments in the usage of multimodal information. Morcover, there are eral VQA studies, such as TutorialVQA (Colas et al., 2020), AGQA (Grunde-MeLaughlin et al., 2021), TVQA (Lei et al., 2018), KnowlT VQA (Garcia et al., 2020), EQo (Fan, 2019), Ego- TaskQA (Jia et al., 2022). GQA (Hudson and Man-
1 Introduction
Computer vision systems for understanding data, such as video clips, photos, and generated synthetic images, have been studied for few decades and
"Equal contribution.
Corresponding author.

---

ning, 2019), DramaQA (Choi et al., 2021). VQA- MHUG (Sood al., 2021), PoinIQA (Mani et al., 2020) that target at different aspects such as instruc- tional assistance, subtitle inclusion, and composi- tional reasoning. Despite the notable progress in VQA studies, most of the works use videos from one or multiple third-person views (TPV). Con- strained by the fixed viewing angle, TPV video may not be able capture subtle information on human operation in industrial tasks such as assembly and disassembly. In particular, the TPV may not be suf- ficient to capture the interaction that forms the basis of human- human-huuan/robbe shared tasks. view (FVP) videos containing gaze information are beneficial to understanding the tasks performed in beeembly-disaseembl) studies and overcoming spatial ambiguities.
gaze information to facilitate semantic understand- ing in collaboration lasks. As shown in Figure 1, GazeVQA contains gaze-augmented QA pairs that are related to human- interaction, which can assist a person in learning task knowledge with an AI assistant that answers questions.
To tackle the challenges of visual understanding presented in GazeVQA, we propose AssistGaze -an assistive model for task-collaborative HRI-QA, It combination of different models answer the questions with three different answer types, includ- ing textual answers, video retrieval answers, and image answers. The main purpose of AssistGaze is to predict correct answers for task-specific ques- tions in GazeVQA by using gaze information. We inquire into different configurations of the proposed model, such availability of gaze, choice of en- coders, answer types, etc., and compare our model with a few benchmarks, such as random guess and VIOLET (Fu et al., 2021). First, random guess models and masked versions of all video features are examined. Secondly, the effectiveness of TPV videos is examined. Finally, FPV videos with gaze information are added to the experiments. It is observed that the model obtains the highest perfor- mance when all features included.
A novel task-oriented collaborative VQA is pro- posed in this research based on the collaboration theory, which is defined as coordinated, syn- chronous activity resulting from a continuous effort to construct and maintain a shared understanding of a problem (Dillenbourg et al., 1996; Roschelle and Teasley, 1995). This theory encompasses four key aspects: situation, interaction, process (grounding), and effect (Dillenbourg et al., 1996). Situation is characterized by degree of division of labor, which sets the stage for interactions. Interaction is characterized by cognitive process such as ground- ing and mutual modeling, leading to effect that contribute to shared understanding (Dillenbourg et al., 1996). The ultimate goal of collaboration, according to this theory. is to build the common grounds necessary for effective cooperation. In the context of interaction between two subjects, there is requirement to develop shared language for task completion. This concept is called grounding mechanism (Dillenbourg et al., 1996), which oper- ates on a functional principle: while perfect mutual understanding is not attainable, nor necessary, is required to sustain sufficient mutual understanding to continue the task (Clark and Brennan, 1991). In addition, every collaboration has a unique user ex- perience. However, the TPV FPV videos may not adequately capture the semantic meaning that rep- resents users' intentions and unique experiences.
There are three contributions of this paper. (1) We propose a novel collaborative VQA task to sup- port collaboration in industrial applications such as the assembly and disassembly processes. (2) We design GazeVQA, a new VQA dataset on collab- orative scenarios, with carefully curated QA pairs and multimodal inputs including videos captured from multiviews and eye gaze information in the FPV video. (3) We create new assistant model, AssistGaze, to generate answers in three formats, while using gaze information to resolve spatial am- biguities. The model assists a human user in collab- oration and interaction by answering task-specific questions.
Related Works
2.1 Video QA
There is increasing momentum of VQA studies re- cently (Antol et al., 2015; Lu et al., 2019; Goyal et al., 2017; Gao et al., 2020; Tapaswi et al., 2016; Maharaj et al., 2017: Mun al., 2017: Jang et al., 2017: Gao al., 2023; Jang et al., 2017). With the acceleration of the studies carried out in this field, numan-centered action recognition studies were conducted with video clips taken from TV
The VQz task aims establish common ground for collaborative tasks involving human- robot interaction. Furthermore, different from other existing VQA studies, VQA leverages com- mon grounding theory (Li et al., 2006) and uses eye-

---

Total Hours
# of Videos
Avg. L. (min.)
# of Verbs
# of
# of Objects
Dataset
Actions
Ego4D (Grauman et al., 2022)
120
74
87
MECCANO (Ragusa et al., 2021)
6.9
20
20.7
12
21
61
IKEA ASM (Ben-Shabat et al., 2021)
35
371
5.6
12
10
33
Assembly 101 (Sener et al., 2022)
513
4,321
7.1
24
90
1,380
EPIC-KITCHENS (Damen et al., 2021)
100
700
8.5
97
300
4,053
GazeVQA(gaze)
31.5
895
2.5
28
16
40
VazeVQA
125
2,967
2.5
28
16
40
Table 1: Comparison of the Gaze VQA and other action datasets in the context of total hours of video clips, number of videos, average length of video clips (in minutes), number of verbs, actions, and objects.
series (Lei et al., 2020). A detailed comparison of VQA and current VQA models is in the appendix, Tables and 9. Egocentric videos have become popular the computer vision commu- nity due to the prevalence small cameras and the ease of collecting FPV videos. These cam- eras are useful to collect a varicty of videos in dif- ferent domains, such as manufacturing (assembly- disassembly) (Tan et al., 2020), education, behav- ior (Bambach et al., 2017; Wong al., 2022), and sports (Bertasius et al., 2017: Shi and Bertasius, 2017). These videos also enable analysis such as object detection (Fathi et al., 2011; I; Fan et al., 2018; Furnari et al., 2017), hands detection (Bambach et al., 2015; Chen et al.. 2023), gaze detection and prediction (Huang et al., 2018).
appendix for more comprchensive comparisons be- tween GazeVQA and existing VQA datasets.
2.2 FPV and gaze information
Visual grounding is recent research topic in the VQA studies. (Zhu et al., 2016; Hudson and Manning, 2019) proposes method to localize ob- jects answer the questions with visual grounding. However, human-robot interaction requires more cost-effective solutions. Thus, the usage of hand gestures, i.e. pointing at objects, shows a useful ap- proach (Mani et al., 2020), which nevertheless, suf- fers from tedious process for recording hand ac- tions and visual processing. Alternatively, human gaze can be used to decode the human intention with respect to localizing objects and recognizing actions.
To train powerful VQA models, datasets with reasonable scale and accurate annotation are necessary. Large-scale datasets such as EPIC- KITCHENS (Damen et al., 2021) or Ego4D (Grau- man et al., 2022) are not suitable for annotation due to their enormous sizes. Other studies for industrial assembly-disassembbl are Assembly 101 (Sener et al.. 2022), IKEA ASM (Ben-Shabat et al., 2021). and MECCANO (Ragusa al., 2021). Table 1 presents comparison of various works alongside VQA. GazeVQA is the first dataset that deals with beeembly-ddsassembbe in the industry context that contains eye gaze information. The availability of eye gaze information is a unique feature of our dataset, which can augment semantic understand- ing of the tasks by harnessing human intentions. Compared MECCANO and IKEA ASM datasets, VQA superior in terms of total recording hours, number of video clips, and number of differ- ent action words. Consequently, supports model development towards worldword applications. In- terested readers may refer to Tables 8 and 9 in the
Gaze information has been used for action recog- nition studies for a while, e.g., show how gaze- indexed frames beneficial (Li et al., 2015) and how noisy information can be recognized with min- imum error (Li al., 2018). However, using gaze information for VQA studies has not become pop- ular yet.
While VQA datasets have involved video with subtitles (Tapaswi et al., 2016; Lei et al., 2018. 2020), GazeVQA does not depend upon the addi- tional explanation. AGQA (Grunde-MeLaughhinn et al., 2021) works on spatial-temporal reason- ing and NexT-QA (Xiao et al., 2021) deals with temporal-cauu causal reasoming. Assembly 101 (Sener et al., 2022) focuses assembly and disassembly upplications, but does not involve collaboration and interaction. Gaze works on common ground- ing for collaborative interactions by using gaze information. It is formulated based on collabora- tive upplications and aims to make predictions with text, image, and video answers.

---

3 Dataset
information from that kind of action in the videos. VQA studies usually require a large number of QA lists to train a model. Questions are asked to assist the subjects with the necessary information to perform the task. To do so, some questions are prepared by using specific verbs such as "look", and "gaze at". They are used in the dataset uti- lize eye-gaze information. short, questions are prepared to mitigate challenges in temporal action segmentation, action recognition, and object recog- nition, which can be answered by using gaze infor- mation, as well TPV enhanced approach to improving our VQA dataset.
3.1 Video and QA Collection
Vaz is a dataset based on simulated scenarios for industrial applications of nobon collabo- ration. Figure shows the spatial layout of the data collection. In our use case of industrial assembly and disassembly, the 3D printed version of the Type C Gearbox-AGNEE Shaft Mounted Speed Reducer (SMSR) product was used.
Task-oriented, instruction-based procedures. Existing datasets feature multi-step activities fol- lowing strictly ordered recipe (Micch et al., 2019; Zhou al., 2018; Zhukov al., 2019), scripted (Ragusa al., 2021); or non-scripted (Sener et al., 2022). Vaz consists of instruction- based lask-triented collaborative activitics. In particu- lar, the assembly/disaseemblyy was performed by two partners in a collaborative relationship. One partner acts as an instructor, and the other is novice/subject to be trained. During the experi- ment, the subjects from the instructor. The subject can be warned and assisted by the instructor in cases of incom- plete/incorrect actions.
INSTRECTOO
INSTRUX TOR
SURIECT
SUBJECT
ask for clarification and help
Figure Spatial layout for GazeVQA data collection, and the images of QA tasks with the opening scene.
Synchronization of exo/egocentric angles. Fixed features of the scene or objects in the video that do not change over time are called static in- formation, such as the shape of an object or the layout of room. Such information is often im- portant for understanding the context of a scene the properties of an object. Information that expresses properties that can change over time called dynamic information, such as the movement objects or humans, the changing expressions on a human's 's face, or the actions performed by the subject of the video. Capturing dynamic informa- tion enables the analysis of events changes in the scene over time. The FVP video provides subjective view of the action. can offer an in- sight into a human's focus of attention. Some tasks are designed for the interaction of multiple objects, such as screwing bolts onto the gearbox. In this example, the gearbox should be put horizontally, and then the nut and bolt should be assembled in the correct matching position. In challenging steps where multiple actions performed simultanc- ously, FPV videos cannot provide sufficient data for the assistive model estimate human intention. Therefore, static and dynamic information might be missed by FPV. We propose that eye-gaze in- formation is the key to understanding the semantic
Different Perspectives. None of the 22 par- ticipants is knowledgeable about this industrial product. Most experiment sessions were recorded from three different camera angles (one FPV and two TPV angles), except two sessions that were recorded by two camera angles (one FPV and one TPV) due to the malfunctioning of a TPF camera. The use of these different videos has five different goals. First, TPV videos are positioned as shown in Figure 2, providing with spatial depth. As future study projection, this depth provided by the angle differences of the cameras also be used for 3D environment extraction. Second, it is used spatially locate small components that might invisible from an angle. Third, the frames taken from FPV videos might not contain the an- swer to the question asked. This is because the FPV camera has a limited field of view and the sub- jects constantly move the head from side to side, leading to partial coverage of the scene. Fourth, FPV captures the user's perspective which reflects his/her attention. Finally, with eye gaze in FPV videos, which provides additional information on the user's attention and intention. It helps to model to answer questions such as "Where did the subject

---

look at?" and "Which object will be assembled after the current object is assembled?"
not only textual answers but also video retrieval and/or image answers depending on the types of questions.
There are variations in the task completion pro- cess where the sequence may deviate from standard ones. For instance, "screw in the oil level indica- tors" should be completed in one action. In some cases, the subject realizes that as separate sequen- tial tasks. Therefore, the fixed type of QA list can- not be used. However, QA lists could be integrated into the new actions manually. For example, some of the subjects applied the actions "Put the input hub cover" and "Put the small hub cover" at the same time. Thus, the questions are updated for the combination of these steps. Additionally, there are cases where alternative sequences are performed by a subject, e.g., putting the small hub cover before the input hub cover. The detailed visualization is shown in Figures 10 and 11 (in the appendix part).
Similar to other VQA datasets, we propose a multiple choice (MC) answering format for our textual responses. Since video retrieval answers have been recorded from 3 different video angles (TPVenface, TPVsideward. and FPV with and without gaze information), 4 different video re- trieval responses are provided for each participant (except for two subjects). Additionally, manually created ground truth (GT) image answers with the help of segment anything model (SAM) (Kirillov et al., 2023) are prepared for comparing the gaze location in the frame to generate the image answer.
3.3 GazeVQA Statistics and Analysis
The Gaze dataset is composed of the assembly process, which includes a maximum of 22 steps, and the disassembly process, which includes max- imum of 19 steps (not symmetrical). The number of questions for the assembly task is 631; and the numbers of textual, image, and video answers are 563, 306, and 479, respectively. The number of questions for the disassembly task is 521, while the numbers of textual, image, and vidco answers are 474, 266, and 389, respectively.
3.2 VQA Task Formulation
For collaborative tasks to be effective, communi- cation (information sharing) is crucial (Tan et al.. 2020). The collaborative interaction of instructor and subject has two phases, which are the presenta- tion and acceptance phases according to the com- mon ground theory (CGT) (Clark and Brennan, 1991; Li al., 2006). CGT requires searches and updates of common ground to maintain effective communication. This continuous process during communication requires mutual agreement, con- stant attention to feedback, and the adjustment of communication strategies according to the context. The concept is used to analyze the efficiency of communication in different settings, including in- terpersonal communication and human-computer interaction. In our experiments, the instructions from the instructor (speaker) define the presenta- tion phase, while the listener's 's understanding of the given instructions and the execution of the instruc- tions defines the acceptance phase.
There is a total of 25, 25.040 questions, which can be broken down into 8,509 "What", 3,017 "Which", 2,798 "Where", 785 "When", 44 "Who", 2,392 "How", 7,407 "Did", 44 "Is" and 44 "Was". The number of unique textual questions is 1,091. The detailed analyses of the QA pairs are shown in the appendix. In total, there are more than 22,000 tex- tual answers, 12,400 labeled images that are used by one participant's video clips, and 2,967 video clips in the datasct. Additionally, 18,700 video answers are expected, according to the questions. Statistics on frequent words and questions are pro- vided in Figure 5 and 6 in the appendix.
We formulate a new QA task involving collabora- tive interaction processes that exist between human- robot/human. The aim of this interaction is to un- derstand the instructions given by the instructor and to implement them faithfully as a sequence of atomic actions. More than 1100 questions are prepared for each participant following guidelines of 13 different reasoning types that are commonly used in VQA studies. Interested readers may refer to Figure in the appendix for a detailed list of reasoning types. The subject can be assisted by
Model
To tackle the challenges presented GazeVQA, es- pecially related to the information, we propose a new AssistGaze model as shown in Figure 3.
multimodal
The first critical challenge relates to the encod- ing of multimodal inputs, especially video input. Video encoding serves the purpose of obtaining ob- ject information in video embedding (Wang et al., 2022b,a). Text encoding involves question encod-

---

textual answers
A1: Hub cover bolts
A3: imput hub cover
AZ: Casing bolts
CLIP
Pastar T (A)
A4: Output hub cover
Which component will assembled
CLIP
Peature
Muli Layer mer
question/input
Grounding Module
Featmn xCdc)
Prediction GdA
Festure Vx (6.)
Trange (6)
Featue Y (4+) (m))
videos/input
Peature (ffe)
R-CNN
Gi X1l.)
image answers
Figure 3: AssistGaze Model. Firstly, the features of questions and videos are taken as inputs; additionally, the
features of textual and labeled images extracted. Then the encoder structure is where the feature representations are obtained by input encoders. Finally, it performs answer prediction by contrastive loss check.
ing and text answer encoding. CLIP is used for the text embedding. This ensures that the video features and the text features are projected onto the same dimensional space. Each question has a shape of 1 x d, and the text answer will have shape of x d, where dg. and d, are the dimen- sions of features. The model needs to select the correct text tensors from the choices provided.
the Region of Interest (ROI). It can crop the image within the bounding box and convert into features with the shape of x dm-
After all features are extracted, question fea- tures q E R R!xd,, video features R5xd; video answer features age answer features x dm and each text answer feature Rlxd are concatenated. These fea- tuples. The
Rlxds with each im-
For the video encoder, CLIP (Radford al., 2021) used extract the features. The feature ex- tractions from the video clip consist of sequence of five distinct phases. We sparsely sampled frames from each action step and encoded each frame with CLIP. The feature of each frame has shape of x d, where d, is the encoding dimension. Each video answer consists of four videos of which there
tures are called one-question-ansseeeeeeeeeess--- concatenated feature for each question-answer tu- ple m X n x / is with shape of 8 x dy, where d is the feature dimension.
Each tuple will be passed through 3-layer trans- former encoder and we only use the class token of the output tensor from the transformer encoder. It has a shape of 1 x dy and then will be passed through fully connected layer with the shape of x 1. This will give us prediction of whether this pair is correct incorrect.
FPV videos and two TPV videos. One of the FPV videos is the version without gaze information and the other is the augmentede version, where the gaze point is represented by blue dot. While four phases represent the feature extraction of four different types of videos, the last phase represents the concatenation of these four views. The model needs to select the correct step from the 5 steps in the video.
A notable challenge in VQA short-cut learn- ing, which has been addressed by various strate- gies, such as acting on training data, different learn- ing strategies, and architectural priors (Dancette, 2023). A more balanced approach is suggested in (Goyal al., 2017), where the VQAv2 dataset is designed make it difficult answer ques- tion using only the image. (Johnson al., 2017). and (Hudson and Manning, 2019) have tried reduce conditional biases with the rejection sam- pling approach. We tackle this issue by designing
The Faster R-CNN (Ren al., 2015) is imple- mented for image encoding. Since the task requires the model to select the correct bounding box out the "n" choices. To get the features inside the bounding box, the Faster R- R-CNN suitable due to

---

Methods
Answering Type
mAP
Random Guess
Text
0.0711
VIOLET (Fu al., 2021)
Text
0.2549
(Ours) Frames of CLIP Features
Text
0.2717
(Ours) 16 Frames of CLIP Features
Text + Image + Video 0.4919
(Ours) FPV Video Add TPV Video
Text + Image Video 0.5040
(Ours) 8 Frames of CLIP Features (standard)
Text Image Video
0.6719
Table 2: The results for baseline methods which are prepared in different frames with CLIP (Devlin al., 2019) features and FPV video additions to the TPV videos. The benchmark has been done with the VIOLET (Fu et al.. 2021) model by considering only the textual answer type. Not only does our novel moultiple-answeriig approach, but single-type answering approach also performs better against the SOTA textual answer models.
GazeVQA and AssistGaze with multi-modality an- swers (text, image, or video). We also prepared a number of counterfactual QA pairs to reduce the possibility of shortcut learning, as inspired by (Goyal et al., 2017; Ramakrishnan et al., 2018).
video, which is suggested by ClipBERT (Lei et al., 2021). As shown in Table 2, the mAP result which is 0.6719 reflects that 8 frames perform well in our task and more frames, 16, even undermine the performance of the model 0.4919. In the standard model, the input modality includes two TPV videos and two FPV videos, which achieves mAP of 0.6719. To evaluate the contribution of different features, we test a few ablated feature combinations. For example, we use the FPV video with the TPV video with a trainable hyper- nyper-parameter a and get a finalized video features ax 2x df + (1-x) 2x de. the mAP decreases to 0.5040. As shown in Table 2, with only part of the features, the performance is lower than the standard model.
5 Experiments
Data Splits. GazeVQA consists of total of 3,008,400 question-answer tuples. These tuples are randomly split into the training and valida- tion set with a ratio of 8 2.
Evaluation Metrics. For each question we have X n X I qqestiionannswt answerr tuples. Among these tuples, there is only one tuple that can be labeled as correct. Specifically,
answer is considered cor- rect only if all three types of answers (text, image, video) are correct; otherwise, it is labeled as wrong, Therefore, only if all answers are answered cor- rectly, then the prediction will be labeled as correct with a ratio of positive samples to negative samples being m X n x I - 1. We use mean Average Precision (mAP) as the primary evaluation metrics (Oksuz et al., 2021).
Video Inputs
Metrics mAP
FPV
TPV
EyeGaze
X
0.5004
X
V
0.5061
X
0.4982
0.6719
Table 3: Ablation studies and results for the usage of different multi-view inputs (FPV, TPV, and Rye-Gaze).
TruePstu:
Precision
The above results show the challenging nature of the GazeVQA problem. requires varying modali- ties of answers while existing SOTA models require only single-modality answers. As shown in Tables and 5, the performance lower when only one or two kinds of answer types are involved (Note that the question and video features are preserved). The lowest mAP (=0.2274) is obtained when only the video feature used. However, the performance of our model is more suitable and better when multi- ple answer types are needed.
+ False
Implementation Details. We use PyTorch (Paszke et al., 2019) to perform the experiments. The AdamW (Loshchilov and Hutter, 2019) opti- mizer is used with learning rate of 6 10 The batch size is set to 1024.
5.1 Ablation Studies and Results
We first show the performance comparison of few models. As shown in Table 2, random guess has an mAP of 1. Our standard baseline uses frames of one video to be the representation of the
To evaluate the effect of different encoders, we use MViTv2 (Li et al., 2022) extract video fea-

---

Learning Rate:1.2e-3
Learning R
LR:8e-4, Unfrozen Layer of BERT
Epoch
mAP
mAP
mAP
0.0939
0.1047
0.2318
2
0.0977
1042
0,2542
0.1006
1039
0,2506
0.1056
0.2549
Table 4: The ablation studies that are conducted by using the VIOLET Model (Fu al., 2021). It shows the Fine-tuning of the VIOLET model's fully connected layer with different learning rates. The last column represents the fine-tuned linear layer of the VIOLET (Fu et al., 2021) by unfreezing the last layer the BERT (Devlin et al., 2018). LR: Learning Rate.
Answer Types
Metries mAP
encoder. Finally, when the encoder layer set 3, the model performance achieves the best result, with an mAP of 0.6719.
Video
Text
Image
X
0.3321
X
V
0.4482
Metrics
# of Encoder Layers
X
V
X
0.2717
2
V
V
X
0.2987
mAP 0.4557 0.6719 0.5289
X
X
0.2274
X
0.3523
Table 7: Ablation studies for the usage of different numbers of encoder layers to get accurate results.
V
0.6719
Table 5: The results of the usage of different answer types which are video, textual, and image.
Next, we the effectiveness of the recent SOTA model VIOLET (Fu et al., 2021) on our dataset. VIOLET supports only textual answer type and we compare it with our method on textual answer results. VIOLET uses a Video Swin Trans- former (Liu et al., 2022) to encode the images and BERT (Devlin et al., 2018) to encode the text. We fine-tuned the classification head of the VIOLET model while keeping the parameters of the Swin Transformer and BERT frozen. However, the mAP platcaued at 0.10 (as shown in Table 4), indicating the limited performance of VIOLET on our dataset when only fine-tuning the classification head.
tures and BERT (Devlin et al., 2018) to extract text features (Devlin al., 2019), compared to the CLIP-based model. Each step video is encoded with MViTv2 and we take the class token of the features as the representation that step. Since MVIT is not co-trained with BERT the same manner as in CLIP, the model faces challenges in determining the relationships among multimodal features. The performance analysis and compari- son between MViT, BERT, and CLIP are displayed in Table 6.
To better align the features with our task, we un- froze the last layer of the BERT encoder. After four epochs of training, the mAP of the VIOLET model converged a value 0.2549 which is slightly lower than our baseline result. Comprehensive ab- lation results are shown in Table
Encoders
Metrics
Video
Text
Object
mAP
V
BERT
Faster R-CNN
0.3993
CLIP
CLIP
Faster R-CNN
0.6719
Finally, the benchmarking results of textual an- swers with the VIOLET model are given in Ta- ble 2. AssistGaze employs the CLIP to encode both image and video features, while VIOLET the Swin Transformers for video encoding and the BERT for text encoding. CLIP efficiently learns visual concepts from natural language su- pervision. Consequently, using the CLIP as an encoder may result in better performance in visual question-anse answering tasks.
Table 6: Ablation studies on feature encoders to deter- mine the use case differences CLIP and MViTv2 for textual and video features, while Faster R-CNN is used for object features.
Furthermore, the encoder plays critical role in the model. As shown Table 7, the mAP result is 0.4557 with layers of the encoder. Addition- ally, the mAP value 0.5289 with 4 layers of the

---

6 Conclusions
National Research Foundation, Singapore under its NRFF Award NRF-NRFF13-2021-000 and Mike Zheng Shou's Start-Up Grant from NUS.
In this paper, with new dataset VQZ, we propose novel approach to task-oriented collab- orative QA towards real-world upplications like humannhhuman/robtt collaboration. The GazeVQA depicts the natural QA collaboration interaction between an instructor and a subject during guided assembly and disassembly tasks. Moreover, we propose new method with a multiple-choice an- swer format for all types of questions designed perpetuate the common grounding theory for collaboration tasks. Unlike the legacy VQA stud- ies, we address a challenging task in the industrial settings on collaborative task execution. The pro- posed AssistGaze is a new baseline for evaluating the VQA model's 's ability to address this challenging task.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question an- swering. In 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-2433, "Santiago, Chile". IEEE Computer Society.
Sven Bambach, David J. Crandall, Linda B. Smith, and Chen Yu. 2017. An egocentric perspective on tive vision and visual object learning in toddlers. In 2017 Joint IEEE International Conference on De- velopment and Learning and Epigenetic Robotics, ICDL ICDL-EpiRob 2017, Lisbon, Portugal, September 18- 21, 2017, pages 290-295, "Lishon, Portugal". IEEE.
In future studies, further developments are re- quired improve the collaboration of computer vi- sion with semantic understanding to inquire into the collaboration of AssistGaze and CazeVQA on real- world collaborative HRT applications. More efforts are needed to address high-level challenges such as temporal action segmentation, action recognition, object recognition, and spatial understanding tasks. We hope that the proposed novel gaze-enhanced VQA dataset "GazeVQA" and the new assistant model "AssistGaze" will enable the community to move forward with task orieetee VQA.
Sven Bamhach, Stefan Lee, David Crandall, and Chen Yu. 2015. Lending A hand: Detecting hands and rec- ognizing activities in complex egocentric interactions. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7- 13, 2015, pages 1949-19577 "Santiago, Chile". IEEE Computer Society.
Yizhak Ben- Ren-Shhbat, Xin Yu, Fatemeh Sadat Saleh, Dy- lan Camphell, Cristian Rodriguez Opazo, Hongdong Li, and Stephen Gould. 2021. The IKEA ASM dataset: Understanding people assembling furniture through actions, objects and pose. In IEEE Win- Conference Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021. pages 846-858, "Waikoloa, HI, USA". IEEE.
Limitations
The current work has the following limitations: (1) The GazeVQA dataset was designed with a new task-oriented perspective, which is a novel part of the dataset. However, the existing models need to be updated in their architectures to utilize the advantages of VQz (2) The number of tasks is limited, and this limitation could indicate a chal- lenge for the other collaborative applications. (3) Like many guze-related researches that mobile eye-tracker, gaze information in the current dataset is still noisy. Novel algorithms to de-noise the gaze data are needed to enhance the consistency and validity of the dataset.
Gedas Bertasius, Hyun Soo Park, Stella X. Yu, and Jianbo Shi. 2017. Am baller? basketball perfor- mance assessment from first-person videos. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, 2017.pages 2196 2204, "Venice, Italy". IEEE Computer Society.
Joya Chen, Difei Gao, Kevin Qinghong Lin, and Mike Zheng Shou. 2023. Affordance grounding from demonstration video to target image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6799-6808.
Seongho Choi, Kyoung-Woon Woon On, Yu-Jung Heo, Ah- jeong Seo, Youwon Jang, Min Su Lee, and Byoung- Tak Zhang. 2021. Dramaqa: Character-centered video story understanding with hierarchical QA. Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty-Thire Conference In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9. 2021, pages 1166 1174, "Vir- tual". AAAI Press.
A Acknowledgement
This research is supported by the Agency for Sci- ence, Technology and Research (A *SAAR) under its AME Programmatic Funding Scheme (Project A 118A260046). The NUS team is supported by the

---

Herbert H. Clark and Susan E. Brennan. 1991. Ground- ing in communication. In Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley, editors, Perspec- rives on socially shared cognition, pages 127-149. American Psychological Association.
Germany, September 8-14, 2018, Proceedings. Part III, volume 11131 of Lecture Notes in Computer Sci- ence, pages 124-137, "Munich, Germany". Springer.
Alireza Fathi, Xiaofeng Ren, and James M. Rehg. 2011. Learning to recognize objects in egocentric activi- ties. In The 24th IEEE Conference
Anthony Colas, Seokhwan Kim, Franck Dernoncourt, Siddhesh Gupte, Zhe Wang. and Doo Soon Kim. 2020. TutorialVQA: Question answering dataset for tutorial videos. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 5450-5455, Marseille, France. European Language Resources Association.
Computer Vi- ston and Puttern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pages 3281- 3288, "Colorado Springs, CO. US IEEE Computer Society.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming Hc, 2019. Slowfast networks for video recognition. In 2019 IEEE/CVF International Con- ference on Computer Vision ICCV 2019, Sevul, Ko- rea (South), October 27 - November 2, 2019, pages 6201 12100 "Seoul, Korea (South)". IEEE.
Dima Damen. Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kaza- kos, Davide Moltisanti, Jonathan Munro, Toby Per- rett, Will Price. and Michael Wray. 2021. The EPIC-KITCHENS dataset: Collection, challenges and baselines. IEEE Trans. Pattern Anal. Mach. In- tell., 43(1 1255-4441.
Tsu-Jui Fu, Linjie Li. Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang. and Zicheng Liu. 2021. VIOLET End-to-end video-language trans- formers with masked visual-token modeling. CoRR, abs/2111. 1.
Corentin Dancette. 2023. Shorteut Learning in Visual Question Answering. Theses, Sorbonne Université.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Mi- ami, Florida, USA. pages 248-255, "Miami, Florida. USA" IEEE Computer Society.
Antonino Furnari, Sebastiano Battiato, Kristen Grau- man, and Giovanni Maria Farinella. 2017. Next- active-object prediction from egocentric videos. J. Vis. Commun. Image Represent., 49:401-411.
Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2020. Multi-modal graph neural network for joint reasoning on vision and scene text. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12746-12756.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.0488)
Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. 2021. Env-qa: A video question answering bench- mark for comprehensive understanding of dynamic environments. In 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 1655-1665, "Montreal, QC, Canada". IEEF.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume (Long and Short Pa- pers), pages 4171-4188 "Minncapolis. MN. USA". Association for Computational Linguistics.
Difei Gao. Ruiping Wang, Shiguang Shan, and Xilin Chen. 2022. Cric: vqa dataset for compositional reasoning on vision and commonsense. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 45()):5561-5578
Pierre Dillenbourg, David Traum, and Daniel Schnei- der. 1996. Grounding in multi-modal task-oriented collaboration. In Proceedings of the European Con- ference on Al in Education, pages 401-407, "Lisbon, Portugal".
Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou. 2023. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 14773-14783.
Chenyou Fan. 2019. Egovqa an egocentric video question answering benchmark dataset. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Ko- rea (South), October 27-28, 2019, pages 4359-4366, "Seoul, Korea (South)". ". IEEE.
Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. 2020. Knowit VQA: answering knowledge-based questions about videos. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020. The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New
Chenyou Fan, Jangwon Lee, and Michael S. Ryoo. 2018 Forecasting hands and objects in future frames. In Computer Vision ECCV 2018 Workshops - Munich,

---

York, NY, USA, February 7-12, 2020, pages 10826- 10834, "New York, NY, USA". AAAI Press.
Munich, Germany, September 8-14, 2018, Proceed- ings, Part IV, volume 11208 of Lecture Notes in Com- puter Science, pages 789-804, "Munich, Germany". Springer.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Honolulu, HI, USA, July 21-26, 2017, pages 6325-6334, "Honolulu, HI, USA". IEEE Computer Society.
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reason- ing and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1700-6709, "Long Beach, CA, USA". Computer Vision Foundation/ / IEEE
Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger. Hao Jiang. Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Ra- dosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu. Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Frapalli, Christoph Fe- ichtenhofer. Adriano Fragomeni. Qichen Fu, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jáchym Kolár, Satwik Kottur, Anurag Kumar, Fed- erico Landini. Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro. Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi. Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Gio- vanni Maria Farinella, Christian Fuegen, Bernad Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Han- byul Joo, Kris Kitani, Haizhou Li, Richard A. New- combe, Aude Oliva, Hyun Soo Park, James M. Rehg. Yoichi Sato, Jianbo Shi. Mike Zheng Shou, Anto- nio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2022. Ego4d: Around the world in 000 hours of egocentric video. In IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18973-18990. IEEE
Yunscok Jang, Yale Song, Youngjac Yu, Youngjin Kim, and Gunhee Kim. 2017. TGIF-QA: toward spatio- temporal reasoning in visual question answering. In 2017 IEFE Conference on Computer Vision and Put- tern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1359-1367, "Honolulu, HI, USA". IEEE Computer Society.
Baoxiong Jia. Ting Lei. Long-Chhu Zhu, and Siyuan Huang. 2022. Egotaskqa: Understanding human tasks in egocentric videos. In NeurlPS, "New Or- leans, Louisiana. ABD", IEEE
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitmick, and Ross B. Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual rea- soning. In 2017 IÊEÉ Conference Computer Vi- sion and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017. pages 1988- 1988-1997. IEEE Computer Society.
Kushal Kafle and Christopher Kanan. 2017. An analysis of visual question answering algorithms. In IEFE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29. 2017, pages 1983- 1991, "Venice, Italy". IEEE Computer Society.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023, Segment anything. arXivv2304.00264:
Madeleine Grunde- aughtl,, Ranjay Krishna, and Maneesh Agrawala. 2021. AGQA: A benchmark for
compositional spatio-temporal reasoning. In IEEE Conference Computer Vision and Pattern Recog- nition, CVPR 2021, virtual, June 19-25, 2021, pages 11287- "Virtual". Computer Vision Founda- tion/ / IEEE
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hin- ton. 2017. Imagenet classification with deep convolu- tional neural networks. Commun. ACM, 60M)):84990
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg. Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and- aide--and-lnnguage learning via sparse sampling. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7331-7341, virtual. IEEE.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, "Las Vegas, NV, USA". IEEE Computer Society.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. TVQA: Localized. compositional video ques- tion answering. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 1369-1379, Brussels, Belgium. Association for Computational Linguistics.
Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. 2018. Predicting gaze in egocentric video by learn- ing tesk-dependent attention transition. Computer Vision ECCV 2018- 15th European Conference,

---

Jie Lei. Licheng Yu. Tamara Berg, and Mohit Bansal. 2020. TVQA+: Spatio-temporaa grounding for video question answering. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 8211-8224 Online. Association for Computational Linguistics.
Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C. Courville, and Christopher Joseph Pal. 2017. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolula, HI, USA, July 21-26, 2017, pages 7359-7368, "Honolulu, HI, USA". IEEE Computer Society.
Weixian Lei, Difei Gao, Yuxuan Wang, Dongxing Mao, Zihan Liang, Lingmin Ran, and Mike Zheng Shou. 2022. Assistsr: Task-oriented video segment retrieval for personal ai assistant. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022. pages 319-338.
Arjum Mani, Will Hinthorn, Nobline Yoo, and Olga Russakovsky. 2020. Point and ask: Incorporating pointing into visual question answering. CoRR, abs/2011.13681
Shuyin Li, Britta Wrede, and Gerhard Sagerer. 2006. A computational model multi-modal grounding for human robot interaction. In Proceedings of the SIG- DIAL 2006 Workshop. The 7th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 15-16 July 2006, Sydney, Australia, pages 153-160, Sydney. The Association for Computer Linguistics.
Antoine Miech, Dimitri Zhukov, Jean- -ran-Baptiste Alayrac. Makarand Tapaswi, Ivan Laptev, and Josef Sivie. 2019. Howtol00m: Learning a text-video embed- ding by watching hundred million narrated video clips. In 2019 IEEE/CVE International Confer- ence on Computer Vision, ICCV 2019, Seoul, Ko- rea (South). October 27 - November 2, 2019, pages 2630-2640, "Seoul, Korea (South)". IEFE
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. 2022. Mvitv2: Improved multiscale vision transformers for classification and detection. In IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 4794 4804, "New Orleans, LA, USA". . IEEE.
Jonghwan Mun, Paul Hongsuck Sco, Ilchae Jung, and Bohyung Han. 2017. Marioqa: Answering ques- tions by watching gameplay videos. In IEFF Interna- rional Conference on Computer Vision, ICCV 2017, Venice, Itll, October 22-29, 2017, pages 2886-2894, "Venice, Italy". IEEE Computer Society.
Yin Li, Miao I. Liu, and James M. Rehg. 2018. In the eye of beholder: Joint learning of gaze and actions in first person video. In Computer Vision - ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V. volume I 11209 of Lecture Notes in Computer Science, pages 639-655, "Munich, Germany". Springer.
Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. 2021. Imbalance problems in object detec- tion: A review. IEEE Trans. Puttern Anal. Mach. Intell., 43(10)33388344
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Z. Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035, "Vancouver, BC, Canada". IEEE
Yin Li, Zhefan Yc, and James M. Rehg. 2015. Delving into egocentric actions. In IEEE Conference on Com- puter Vision and Pateern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 287-295, "Boston, MA, USA". IEEE Computer Society.
Ze Liu, Jia Ning, Yue Cao. Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2022. Video swin trans- former. In IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2022, New Or- leans, LA, USA, June 18-24, 2022, pages 3192-3201. IEEE.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, "New Or- leans, LA, USA". OpenReview.net.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages "Virtual". PMLR.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and and language tasks. In Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 14, 2019. Vancouver, BC, Canada, pages 13-23, "Vancouver, BC, Canada". IEBE
Francesco Ragusa, Antonino Furnari, Salvatore Li- vatino, and Giovanni Maria Farinella. 2021. The MECCANO dataset: Understanding human-object

---

interactions from egocentric videos in an industrial- like domain. In IEEE Winter Conference on Applica- tions of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pages 1568-1577, "Waikoloa, HI, USA" IEEE
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen. Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In 2016 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1-340, "Las Vegas, NV, USA". IEEE Computer Society.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Ste- fan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. In Advances Neural Information Processing Sys- tems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8. 2018, Montréal, Canada, pages 1348-1555,
Alex Jinpeng Wang, Yixiao Ge. Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022a. Object-aware video- language pre-training for retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 3303-3312, "New Orleans, LA, USA" IEEE
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. In Ad- vances in Neural Information Procesxing Systems 28: Annual Conference on Neural Information Process- ing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91-99, "Montreal, Quehec, Canada", IEEE.
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022b. All in one: Exploring unified video-language pre-training. CoRR, abs/2203.07300.
Jeremy Roschelle and Stephanie D. Teasley. 1995. The construction of shared knowledge in collaborative problem solving. In Computer Supported Collab- orative Learning, pages 69-97, Berlin, Heidelberg. Springer Berlin Heidelberg.
Benita Wong, Joya Chen. You Wu, Stan Weixian Lei. Dongxing Mao, Difei Gao, and Mike Zheng Shou. 2022. Assistq: Affordance--entricc question-driven task completion for egocentric assistant. In Euro- pean Conference Computer Vision, pages 485- 501. Springer.
Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2008. Labelme: A database and web-based tool for image annotation. Int. J. Com- put. Vis., 77(133)1577-133
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony R. Dick, and Anton van den Hengel. 2017 7. Visual question answering: A survey of methods and datasets. Comput. Vis. Image Underst., 163:21-40.
Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang. and Angela Yao. 2022. Assembly 101: A large-scale multi-view video dataset for understanding procedural activities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 21074, "New Orleans, LA, USA" J.EE.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question- answering to explaining temporal actions. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2021, virtual, June 19-25, 2021, pages 9777-9786, "Virtual", Computer Vision Foundation IEEE.
Jianbo Shi and Gedas Bertasius. 2017, Using cross- model egosupervision to learn cooperative basketball intention. In 2017 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2017, Venice, Italy, October 22-29, 2017, pages 2355-2363, "Venice, Italy". IEEE Computer Society.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined at- tention over appearance and motion. In Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View CA, USA, October 23-27, 2017, pages 1655-1653, "Mountain View, CA". ACM.
Ekta Sood, Fabian Kögel, Florian Strohm, Prajit Dhar, and Andreas Bulling. 2021. VQA-MHUG: A gaze dataset study multimodal neural attention in visual question answering. In Proceedings of the 25th Con- ference Computational Natural Language Learn- ing, pages 27-43. Online. Association for Computa- tional Linguistics.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenen- baum. 2020. CLEVRER: collision events for video representation and reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, "Addis Ababa, Ethiopia". OpenReview.net.
Hui Li Tan, Mei Chee Leong, Qianli Xu, Liyuan Li, Fen Fang, Yi Cheng, Nicolas Gauthier, Ying Sun, and Joo Hwee Lim. 2020. Task-oriented moltl question answering for collaborative applications. In IEF. International Conference on Image Processing, ICIP 2020, Abu Dhabi, United Arab Emirates, Oc- lober 25-28, 2020, pages 1426-1430, "Abu Dhabi, United Arab Emirates". IEEF.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao. Yuet- ing Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. The Thirl AAAI Con- ference on Artificial Intelligence, AAAI 2019, The

---

Thirty-First Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pages 9127-9134, "Honolulu, Hawaii, USA". AAAT Press.
instructor. The users' videos were cropped 5 actions were performed consecutively. This is de- signed to reduce the processing cost of long videos to more optimum values. Moreover, videos with common steps increase the variety of uses in re- sponse to the questions asked.
Luowei Zhou, Chenliang Xu, and Jason J. Corso. 2018. Towards automatic learning of procedures from web instructional videos. In Proceedings the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18). the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Sym- posium on Educational Advances in Artificial Intel- ligence ( (EAAI-18). New Orleans, Louisiana, USA, February 2-7, 2018, pages 7590-7598, "New Orleans, Louisiana, USA". AAAI Press
QA Pairs. We have also published the raw ver- sions of our questions. This will facilitate better understanding of industrial QA pairs and the devel- opment of textual inputs.
Images. Each of the labeled images was pre- pared manually using the Segment Anything Model (Kirillov et al., 2023). When the eye-gaze data is on the object related to the answer to the question, the object or that answer is labeled as GT. It also contains two different incorrect labels.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question an- swering in images. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 4995-5004, "Las Vegas, NV, USA". IEEE Computer Society.
Figure shows the Gaze Architecture.
A.2 Words and Questions
One of the motivations for preparing QA pairs is to focus on the use of different verbs, words, and question types. Considering the small number of in- dustrial datasets, the textual QA pairs in Gaze VQA are useful supplement to existing studies.
Dimitri Zhukov, Jean- Baptistte Alayrac, Ramazan Gok- berk Cinbis, David F. Fouhey, Ivan Laptev, and Josef' Sivic. 2019. Cross-task weakly supervised learn- ing from instructional videos. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3537-3545, "Long Beach, CA, USA". Computer Vi- sion Foundation/ IEEE.
Figures 5 and show the diagram of most frequent words and questions.
A Appendix
A.3 VQA Datasets
One of the other motivations is the lack of datasets prepared for task-oriented collaborative scenarios. VQA leverages industrial collaborations by closing this gap and taking advantage of different targets, video sources, and purposes.
GazeVQA has been designed to offer a fundamen- tal and generalizable approach to human-human and human-robot interaction, especially for indus- trial collaborative task studies. AssistGaze pro- poses an assistive model that can support people in task-collaborative studics. The purpose of de- sign protocols is task completion by using eye-gaze information. It is predicted that studies such as human intention estimation and next action esti- mation with over 3,000,000 QA pairs and gaze location information will also benefit robotic ap- plications. Both contributions of this paper are essential for task-oriented human-robot interaction with the question-answering project.
Tables 8 and display the comparison of the Gaz dataset and existing VQA datasets.
Figure 9 displays the comparison of the GazeVQA dataset with existing VQA datasets in terms of reasoning types.
A.4 Assembly and Disassembly
Assembly and disassembly are crucial tasks for industrial applications. This leverages another mo- tivation to create a new dataset for real-world task- oriented collaborations.
A.1 VazeVQA Architecture
GazeVQA consists of videos obtained from 22 users, questions prepared separately for each action performed by each user, and labeled images from FPV's which are coordi-
Figures 7 and 8 show the ussembly and disas- sembly actions with synchronized multi-view video clips.
nated with eye-gaze information.
Figure 10 and 11 show the assembly and dis- assembly steps and protocols.
Video Clips. Each user performed an indus- trial task by following the instructions given by an

---

Step1
Question 1 to
22 Participants
Assembly
S005
Step21
Question 1 to n
QA Lists
S008
Disassembly
Step1
Question 1 to n
Step19
Question 1 to
$035
GazeVQA
Step1
Labeled Image1 to
Labeled
S005
Assembly
Images
S008
Step21
Labeled Imagel
Step1
Labeled Imagel
Disassembly
S035
Step19
Labeled Image1 to
S005
TPV-sideward
Video Clips
Step1 to Step5
S008
TPV-enface
Step2 Step 6
FPVgaze
S035
FPVraw
Step36 to Step40
Figure 4: VQA Architecture. It consists of three different parts. QA Lists, Labeled images, and Video Clips. There are participants, and each participant has their own sub-structures that are divided into assembly and disassembly parts for QA lists and Labeled images. additionally, they have 4 different video clips (TPV-sideward, TPV-enface, FPVgaze, and FPVraw) which explain why they were collected separately in the Dataset section of the paper. Each step has its QA lists, labeled images, and video clips.
Diagram of the Most Frequent Words
17500
15000
Frequency
12500
10000
7500
5000
2500
1000
50000
Infoo
huppor
goot
bale
bitt conne bol
eancal og meo
PM des Steass
Words
Figure 5: Diagram of the Most Frequent Words of the VQzeVQA. "Subject" is the word that is most frequently used in the dataset as a subject, and then "did" is used to check whether the actions/tasks are completed or not. the diagrams the datasets are checked more than 20 action verbs are used, and their frequency is moderately higher than the other words used. The reason that the usage of more verbs could be helpful to increase the possibility of the usage of the VQA QA list on the other industrial projects' applications. Consequently. it could increase the generalizability of the Gaze VQA.

---

Diagram of Question Words Frequency
8500
7500
6500
Frequency
5500
8500
5500
2500
1500
500
Di
Questions
Figure 6: Diagram of the Question Frequency of the Vaz. "What" and "Did" are the most frequent questions that are used to prove the application realization, object, and action recognition. On the other hand, "Which" and "Where" questions are helpful to object localization to increase the answers' accuracy of the spatial questions. The importance of the "How" question is to show the completion of the tasks.
Datasets
Video Source
# of V.
# of QA
Annota.
MSVD-QA (Xu et al., 2017)
Web Vidcos
2K
50K
Auto.
TVQA+ (Lei et al., 2020)
TV Programs
22K
153K
Manual
TutorialVQA (Colas et al., 2020)
Instructional Videos
408
6.2K
Manual
ActivityNet-QA (Yu et al., 2019)
Web Videos
5.8K
58K
Manual
CLEVRER (Yi et al., 2020)
Synthetic
10K
305K
Auto.
AGQA (GrundeMMeLaughlin et al., 2021)
Indoor Activity Videos
9.6K
192M
Auto.
NEXT-QA (Xiao al., 2021)
Web Videos
5.4K
52K
Manual
How VQA69M (Miech et al., 2019)
Instructional Web
10K
10K
Auto.
Gaze VQA (Ours)
Task Collaboration
2,967
3M
Manual
Table Comparison of Gaze and existing VQA datasets by evaluating the number videos, QA pairs, and
annotation types.
Datasets
Target
Text A.
Im. A.
V.
MSVD-QA (Xu et al., 2017)
Description
X
X
TVQA+ (Lei et al., 2020)
Subtitle inclusion
TutorialVQA (Colas al., 2020)
Instructional Assistance
X
X
ActivityNet-QA (Yu al., 2019)
Description
X
X
X
CLEVRER (Yi al., 2020)
Causal Reasoning
X
X
X
AGQA (Grunde-MeLaughhin et al., 2021)
Compositional Reasoning
NEXT-QA (Xiao al., 2021)
Teusoo-Temporal Relation Generalization
X
HowToV VQA99M (Miech et al., 2019)
X
X
Common grounding for collaboration by gaze
GazeVQA (Ours)
V
V
Table 9: Comparison of existing VQA datasets and GazeVQA which targets to obtain all answer formats with Multiple Choices (MC) answer types by using novel gaze information.

---

edfeee
IN
TPV syeges
Figure 7: VQA assembly action includes synchronized nulti-view video clips of assembling the gearbox.
eeface
PW #yegos
Regioness
Riggeeler
Enbral Frrlii
Figure 8: VQz disassembly action includes synchronized multi-view video clips of disassembling the gearbox.
GazeVQA
Explanation
Is Sevent A responsible for <event B+?
happen next, Sevent
EgoTaskKA
MarloQA TGIF-QA
Counterfactual
What would have happened <event A>?
EnvQA
Repetition Counting
How many times does the sperson>> <action>?
ActivityNet QA
TVQA
Activity Recogaition
What does the person do before/aferrwwilee <action??
MSRVTT-QA MSVD-QA
Concept
<concept occur?
Movie FIB
PeltionRela
What/Who When Where/How did <relation> <objecr??
Video-QA
Action-Object
Did they internct with <object> before after <actionp??
Action-Relationship
Did they crelation> something before after <action??
Action-OOjeee-RRlaaimmbha
they <relation <object> while Saction???
Superlatives
What were they <action> fins/last?
Action Sequencing
What did the person do after <actiou??
CLEVRER (synthetic)
Logical Combinations
they <redation> <object/>> <olject2>?
Duration Comparison
<action <actionI> for longer?
AGQA
Causal Reasoning
Why was the <actiou>> happened the end of the video?
NecTQA
Social Interaction
How is the discussion between A and
Social
Figure 9: Vaz focuses visual, semantic, spatial, location, temporal, and action-object relationships. well as social interaction information extracted from questions that are asked from different reasoning types. Figure 9 is prepared compare the current dataset by benefiting from AGQA MoanddeMMLauughie al., 2021).

---

3. Place the cosing top on the pullet. Exterior facing (AGINEE visblee).
2. Put the culput hub cover on the casing top hole.
c. Place casing the pallet, exterior side facing up
Put the output hub cover.
Put other cover. (NO ORDER REQUREMENT)
4. Screw in all the hub bolts. (monus)
5. Unload the casing
bolts (One
cover/boltt time].
time)
11. Insert three sub- assemblles collaooation.
12. Check gear meshes (rotate inpur shaft).
15. Put the gearbox in vertical position paller shoulder.
15. Screw the case bolts (Two hand operotion; casling bolt ie from cosing tap side, NO order requirement)
(collaborativel)
13, lestall the casing top onto the casing bane. the white necessary)
Ovrett
16a. Screw oil level
indicator.
18. Check gear meshes.
indicat-- (NO order requiremeet).
19. Put geas box horisontally.
20. Assembly completed.
Figure 10: Assembly Steps and Protocols: The assembly task has steps. However, we deleted the "pressing the white pins" step, which is 14, because realized by the subject in two different steps. For instance, there are two oil level indicators. so screwing two oil level indicators is shown as one step in the figure, but some subjects have realized this in two steps. Thus, we have a maximum of 22 steps in the assembly task.
common task among the subjects. Some tasks separated and
1. Put gear box in vertical position..
2. Remove two oll level indicators.
Put gear box in horizontal position. cover bolts
6a Remove output hub using hands. Remove small//nput hub cover. Remove input/umall
7. Put the casing upright position pallet shoulder.
10. Put the casing in horizontal position.
Unplug breather plug in colleboration.
11. Remove the top casing.
B. Unscrew all the casing bolts.
12. Pull out all the three shafts in collaboration. the bottom hub
15a. Remove output hub cover (frm caing base). 15b. Remove small
the small hub cover.
the casing Disassembly campleted.
Figure 11: Disassembly Steps Protocols: The disassembly task 17 steps. However, some tasks are separated and realized by the subject in two different steps. For instance, there two oil level indicators, so removing two oil level indicators shown as one step in the figure, but some subjects have realized this two steps. Thus, we have maximum of steps the disassembly task. The picture on the right-bottom is the annotation example show how to annotate the steps atomic actions.


Reviews:
[{"review_id": "okowoBlRIQ", "reviewer": "Reviewer_Ab7b", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal for Reviewer Ab7b\n\nThe GazeVQA dataset focuses on task-oriented collaborative applications such as assembly-disassembly to show that there is a strong link between VQA studies and instructional videos.\n\n---\n\n### **W1: Scenarios of the dataset**\n\n* __Assembly-disassembly tasks are the fundamental scenarios which are commonly realized in every kind of industrial-based application__. \n    * These tasks cover instructional procedures. Therefore, GazeVQA represents better approaches in different areas such as human-robot interaction, and task assistance by conducting these experiments with the collaborative version of these tasks.\n    * Our question set is prepared to be adapted to different industrial assembly-disassembly datasets by completing the name of object adjustments.\n    * These explain why assembly disassembly constitutes a valid process that is generic enough to support broad industrial scenarios/tasks.\n\n* __Existing task-oriented datasets usually focus on specific scenarios.__\n    * While EPIC-KITCHEN [1] and the Breakfast Actions Dataset [2], which are widely adopted in the literature, focus on __a limited number of niche kitchen scenarios__, MECCANO [3], and Assembly101 [4] also focus on toy-based assembly-disassembly scenarios.\n\n* __Diversity of products, scenarios, and questions__\n    * Instead of using a small-sized toy as used in Asembly101 [4] and MECCANO [5] studies, __GazeVQA uses the 3D-printed, and one-to-one scale of the gearbox__ in the experiments.\n    * The assembly and disassembly processes are __designed based on real practices in the industry__, which makes them of practical value and challenging for visual analytics approaches.\n    * The usage of “a real-world industrial product, more than 25K adaptable questions, more than 40 components and scenarios in the dataset” increases GazeVQA's __generalizability, applicability, and adaptability to different real-world applications__ compared to current studies.\n\n* __New opportunities__\n    * GazeVQA has unique characteristics that are not available in EPIC-KITCHEN [1], Breakfast Actions Dataset [2], MECCANO [3], and Assembly101 [4] datasets.\n    * Focusing on specific improvements in developing areas could be more beneficial for model design and training.\n    * GazeVQA propose __novel components and scenarios for task-oriented studies__.\n    * GazeVQA __offers a new combined subfield to the communities__ for new research contributions.\n\n---\n\n### **W2: Details of the dataset**\n\n* __Numbers and types of the components__\n    * Maximum of 22 steps in the assembly task and 19 steps in the disassembly task.\n    * Assembling is started with the instruction which is “___Place the casing top on the pallet. Exterior facing up___ and over with the instruction which is “ ___Put the gearbox horizontally___ ”.\n    * Disassembling is started with the instruction which is “___Put the gearbox in a vertical position.___ ” and over with the instruction “___Unload the casing base___ ”.\n    * There are 1 casing top, 1 casing base, 2 output hub cover, 1 input hub covers, 3 small hub covers, 22 hub cover bolts, 3 sub-assemblies (1 input shaft, 1 transfer shaft, and 1 output shaft), 6 casing bolts, 2 oil level indicators, and 1 breather plug. __In total, 42 components are used to assemble or disassemble this product.__  \n    * __Components and steps are illustrated with relevant instructions as shown in Figures 10 and 11.__ \n    * We will include detailed procedures with visualization in the supplementary material.\n\n---\n\n### **W3: Comparison with existing frameworks**\n\n* __Different model structure from existing methods__\n    * AssistGaze predicts 3 different types of answers by using multimodal inputs, which __has not been covered in detail in the literature__.\n* __Fair comparison__\n    * The most effective result is achieved when all features are used together. \n    * __Due to significant differences such as input-output types, modalities, and the diversity of the numbers and types of output predictions__, a fair and direct comparison between these studies could be infeasible.\n    * We compared our model with the recent SOTA model “VIOLET” [5] (100+ citations to date) whose answer type only has textual answers. VIOLET uses a Video Swin Transformer [6] to encode the images and BERT [7] to encode the text.\n    * Due to time constraints, we fine-tuned the VIOLET [5] model with its pertained weight with 1 epoch. __We intend to provide updated results during the designated discussion period__, once model convergence through the fine-tuning process. __Table below shows the results__.\n\n| Methods        |Answering Type | mAP                    |\n| -------------  | ------------- |-------------           |\n| Random Guess   |Text           | 0.07117 (from paper)   |\n| VIOLET [5]     |Text           | 0.09396 (to be updated)|\n| Ours           | Text          | 0.27173 (from paper)   |\n| Ours           | Text + Video + Image | 0.67199 (from paper) |\n\nWe only fine-tuned the classification head of the VIOLET [5] model, while keeping the parameters of Swin Transformer [6] and BERT [7] frozen due to the time and computation resources limit.  Hence the VIOLET model is underperforming compared to our results.\n\n---\n\n\n### **REFERENCES**\n\n1. Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., ... & Wray, M. (2022). Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, 1-23.\n2. Kuehne, H., Arslan, A., & Serre, T. (2014). The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 780-787).\n3. Ragusa, F., Furnari, A., Livatino, S., & Farinella, G. M. (2021). The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1569-1578).\n4. Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., & Yao, A. (2022). Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21096-21106).\n5. Fu, T. J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021). Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681.\n6. Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022). Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3202-3211).\n7. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a new Gaze VQA dataset, which can promote robot-human interaction and collaboration studies.\n2. It implements a Gaze VQA pipeline and provides some ablations to show the effectiveness of different components.", "reasons_to_reject": "1. The scenarios of the dataset are limited and only contain assembly and disassembly.\n2. The details of the dataset are insufficient, such as the number(type) of the assembled objects/steps\n3. The paper lacks comparisons (results on the current dataset) with existing frameworks, like VideoQA and ImageQA.", "questions_to_authors": "", "paper_topic_and_main_contributions": "**Topic**: This paper aims to provide a task-oriented collaborative QA application with eye gaze information, and builds an FPV video dataset(including TPV and multiple-choice QA pairs) and model. \n\n\n**Contributions:**\n1. a new VQA dataset(GazeVQA) with gaze information, which is composed of the assembly task with FPV and TPV videos.\n2. a pipeline (or a model) for the implementation of the Gaze VQA", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lPhP5N7FGP", "reviewer": "Reviewer_3C3r", "summary": "", "questions": "The dataset is compiled from 22 participants. This sample size might not be adequately representative to capture the diversity and nuances of human gaze and intent in collaborative tasks, leading to potential biases.\n\nThe emphasis on gaze data might make the VQA approach presented overly reliant on this particular modality. Real-world applications may encounter scenarios where gaze data isn't as indicative of intent, or where such data is unavailable or unreliable.\n\nHow does the utilization of gaze information in VQA differ from its use in action recognition studies, given that it has been employed in the latter for some time?\n\nIn Table2, why 8 Frames of CLIP Features achieves better performance than 16 Frames of CLIP Features?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal for Reviewer 3c3r\n\nI would like to clarify the concerns of the reviewer in four aspects that are diversity, of the dataset, applicability and generalizability of the dataset, comparison of our model with existing approaches, and utilization of gaze information.\n\nThe GazeVQA dataset creates a new sub-area and opportunities for task-oriented collaborative applications by combining two high-level research topics. Thus, the communities could do unique research to find solutions to the collaborative studies with instructional video by using new video analytics multi-modal approaches.\n\n---\n\n### **W1: Diversity of the dataset**\n\n* __Existing task-oriented datasets usually focus on specific scenarios.__\n    * Unlike kitchen-based niche tasks like EPIC-KITCHEN [1], and the Breakfast Actions Dataset [2], __GazeVQA is focused on task-oriented collaborative applications__.\n    * It is unclear how models which are trained with products such as toys that may be difficult to adapt to real-world applications, such as MECCANO [3] and Assembly101 [4], will perform in studies with real-world products. \n    * Although the sample number of 22 participants raises concerns, GazeVQA mitigates this concern with __125 hours of video data from 3 different video angles, more than 40 components, and more than 40 action steps that are not symmetrical__. \n\n---\n\n### **W2: Applicability and Generalizability of the dataset**\n\n* __Assembly/disassembly is a representative process in industry tasks and covers many important scenarios__.\n    * In our experiment, while the operation processes are similar across subjects, there are __many variations including the layout of the components, different styles of users completing a step, and collaboration interaction between instructor and user__. As such, we hope to strike a good balance of procedure consistency and variations, which is important in industrial operations.\n\n* __Successful contribution of gaze information__\n    * Tables 2,3,4,5 and 6 show that gaze information contributes to the model making correct predictions __when the gaze information is used in addition to other features__.\n    * Besides providing semantic information for industrial task assistance studies of gaze information, __it can be used as valuable input for multi-modal models in procedural planning with instructional videos, and human-centered user modeling studies__.\n\n* __Broad Review for Applicability and Generalizability of the gaze-enhanced dataset__\n    * __Gaze information plays a role as a potential contributor to studies in multi-disciplinary areas__  such as psychology, computer vision, NLP, and many more. For instance,\n        * Estimation of users' intentions and final decisions [5],\n        * Face-to-face communications in augmented reality environments [6],\n        * Users' manipulation of objects in virtual environments [7],\n        * Exploring the interaction of gaze, image, and users' description [8],\n        * Statistical word learning using eye movements [9],\n        * Analyzing the self-corrections for image descriptions with the help of eye-tracking corpus [10],\n        * Inspection of the images via spoken descriptions and gaze measurements [11].\n\n* In the context of task-oriented collaborative studies, GazeVQA can be generalized for numerous collaborative tasks such as __assembly-disassembly tasks__, __task-assisted video question-answering (VQA)__, __human decision-making__, __intention prediction research__, and __AR/VR studies__ by using the AssistGaze model strategy that utilizes more than twenty-five thousand generated questions and leverages gaze information for multimodal answer predictions.\n---\n\n### **W3: Comparison with existing approaches**\n\n* __AssistGaze structure is different from existing approaches__\n    * Unlike models that focus on one type of answer prediction, __AssistGaze is capable of multimodal input and three different answer predictions__.\n* __Baselines of GazeVQA__\n    * __Promising results are proposed with our baselines to follow the gaze-enhanced models for task-oriented collaborative industrial applications__. \n\n    * The recent SOTA model “VIOLET” [16] (100+ citations to date) which has only textual answer type, is used for benchmarking. VIOLET uses a Video Swin Transformer [17] to encode the images and BERT [18] to encode the text.\n    * Due to the limited time period for rebuttal, the VIOLET [16] model with its pertained weight with 1 epoch was fine-tuned. __We intend to provide updated results during the designated discussion period__, once model convergence through the fine-tuning process. __Results are shown in the Table below__.\n\n| Methods        |Answering Type | mAP                    |\n| -------------  | ------------- |-------------           |\n| Random Guess   |Text           | 0.07117 (from paper)   |\n| VIOLET [5]     |Text           | 0.09396 (to be updated)|\n| Ours           | Text          | 0.27173 (from paper)   |\n| Ours           | Text + Video + Image | 0.67199 (from paper) |\n\n* __We only fine-tuned the classification head of the VIOLET [16] model, while keeping the parameters of Swin Transformer [17] and BERT [18] frozen due to the time and computation resources limit__.  Hence the VIOLET model is underperforming compared to our results.\n\n* According to the experiments and ablation studies done for the study, we have found the reason for performance differences between 8 and 16 frames of CLIP. The reason 8 Frames of CLIP Features achieve better performance than 16 Frames of CLIP Features is that __the 16-frame version doubles the sequence length, making it relatively difficult to converge within 20 epochs__. In addition, 16 frames per video means more information is taken into consideration when the model is performing classification tasks. __There could be more noise in the features of 16 frames which results in the underperformance of the model__.\n\n___\n\n### **W4: Utilization of gaze information**\n\n* __Gaze information for action prediction__\n    * Gaze information has been investigated in the literature as gaze prediction [12].\n    * Action recognition studies can be performed by estimating gaze information using egocentric perspectives [13].\n    * It is also common to use head/hand motion and hand location/pose [14] or to use sensors in virtual environments [15] to estimate gaze location information.\n\n\n* __Gaze information for intention prediction__\n    * When processing a video of an action from an egocentric perspective, predicting the user's next steps based on various surrounding objects can hinder effective and precise answer predictions. However, __collecting the user's specific gaze location information enables user-object interactions and facilitates more accurate predictions of the user's intentions in the subsequent steps__.\n    * __The results of the experiments conducted in this study (shown in Fig. 1) illustrate that with the gaze information used in addition to other features, the model can yield more precise answer predictions__. \n\nGaze information is not overly reliant on a particular kind of modalities, but it also has crucial contributions in many different research areas and application scenarios. Consequently, differing from recent studies, the utilization of gaze information for textual, image, and video answers in VQA and task-assistance research further highlights the contribution we present to the communities.\n\n---\n### **REFERENCES**\n1. Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., ... & Wray, M. (2022). Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, 1-23.\n2. Kuehne, H., Arslan, A., & Serre, T. (2014). The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 780-787).\n3. Ragusa, F., Furnari, A., Livatino, S., & Farinella, G. M. (2021). The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1569-1578).\n4. Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., & Yao, A. (2022). Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21096-21106).\n5. Huang, C. M., Andrist, S., Sauppé, A., & Mutlu, B. (2015). Using gaze patterns to predict task intent in collaboration. Frontiers in psychology, 6, 1049.\n6. Rivu, R., Abdrabou, Y., Pfeuffer, K., Esteves, A., Meitner, S., & Alt, F. (2020, June). Stare: gaze-assisted face-to-face communication in augmented reality. In ACM Symposium on Eye Tracking Research and Applications (pp. 1-5).\n7. Pathmanathan, N., Becher, M., Rodrigues, N., Reina, G., Ertl, T., Weiskopf, D., & Sedlmair, M. (2020, June). Eye vs. head: Comparing gaze methods for interaction in augmented reality. In ACM Symposium on Eye Tracking Research and Applications (pp. 1-5).\n8. Yun, K., Peng, Y., Samaras, D., Zelinsky, G. J., & Berg, T. L. (2013). Studying relationships between human gaze, description, and computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 739-746).\n9. Yu, C., & Smith, L. B. (2011). What you learn is what you see: using eye movements to study infant cross‐situational word learning. Developmental science, 14(2), 165-180.\n10. Van Miltenburg, E., Kádár, A., Koolen, R., & Krahmer, E. (2018, August). DIDEC: The Dutch image description and eye-tracking corpus. In Proceedings of the 27th International Conference on Computational Linguistics (pp. 3658-3669).\n11. Vaidyanathan, P., Prud’Hommeaux, E., Pelz, J. B., & Alm, C. O. (2018, July). SNAG: Spoken narratives and gaze dataset. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 132-137).\n12. Li, Y., Liu, M., & Rehg, J. (2021). In the eye of the beholder: Gaze and actions in first person video. IEEE transactions on pattern analysis and machine intelligence.\n13. Min, K., & Corso, J. J. (2021). Integrating human gaze into attention for egocentric activity recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1069-1078).\n14. Li, Y., Fathi, A., & Rehg, J. M. (2013). Learning to predict gaze in egocentric video. In Proceedings of the IEEE international conference on computer vision (pp. 3216-3223).\n15. Emery, K. J., Zannoli, M., Xiao, L., Warren, J., & Talathi, S. S. (2021, March). Estimating Gaze From Head and Hand Pose and Scene Images for Open-Ended Exploration in VR Environments. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 554-555). IEEE.\n16. Fu, T. J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021). Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681.\n17. Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022). Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3202-3211).\n18. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The inclusion of both exocentric and egocentric videos with gaze information provides a unique perspective on human intentions during task processes. \n\nThe introduction of the AssistGaze model demonstrates a practical application of the GazeVQA dataset. \n\nThe extensive experiments performed showcase the efficacy of AssistGaze and the intricacies of GazeVQA.", "reasons_to_reject": "While the GazeVQA dataset's innovation is commendable, it addresses a very specific niche within the broader NLP community. The focus on gaze data and its application to VQA may not be of broad interest, which could limit its appeal and applicability to a wider audience.\n\nThe paper introduces the AssistGaze model but does not extensively compare it with existing models or approaches in the VQA domain. Such comparisons would have provided a clearer understanding of where AssistGaze stands in the current research landscape.\n\nGiven the novel nature of the dataset and the specific contexts in which it was collected, there might be concerns about the generalizability of the findings and the model to other domains or tasks beyond the ones explored in the paper.\n\nThe paper might lack an in-depth technical exploration of how the AssistGaze model effectively grounds perceptual input to semantic information.", "questions_to_authors": "The dataset is compiled from 22 participants. This sample size might not be adequately representative to capture the diversity and nuances of human gaze and intent in collaborative tasks, leading to potential biases.\n\nThe emphasis on gaze data might make the VQA approach presented overly reliant on this particular modality. Real-world applications may encounter scenarios where gaze data isn't as indicative of intent, or where such data is unavailable or unreliable.\n\nHow does the utilization of gaze information in VQA differ from its use in action recognition studies, given that it has been employed in the latter for some time?\n\nIn Table2, why 8 Frames of CLIP Features achieves better performance than 16 Frames of CLIP Features?", "paper_topic_and_main_contributions": "The paper's central focus is the incorporation of exocentric and egocentric videos in Video Question Answering (VQA) studies, a cutting-edge approach in robot-human interaction and collaboration studies. The paper's unique contribution lies in the creation of a novel task-oriented VQA dataset, referred to as GazeVQA, which incorporates gaze information during collaborative task processes to understand human intentions.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "This paper is well organized and written."}, "extra_text": {}}, {"review_id": "yR86a8cqfO", "reviewer": "Reviewer_doaa", "summary": "", "questions": "A. Can you further explain what is the \"Grounding Module in Figure 3?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal for Reviewer doaa\n\nI would like to clarify the concerns of the reviewer in three aspects that are generalizability of the dataset, the application of the gaze data in real-world scenarios, and grammar improvement.\n\n---\n\n### **W1: Generalizability of the dataset**\n\n* __Assembly and disassembly tasks are processes within industrial tasks, encompassing a wide array of significant scenarios__. \n    * __They are foundational scenarios frequently encountered in various industrial applications__.\n    * While the experiment process is similar across users, many variations which are **\"the layout of the components, users’ diverse styles to complete the steps, and interaction between instructor and user,\"** exist in GazeVQA. Therefore, we focus on striking a good balance of procedure consistency and variations.\n    * __These highlight the reason why assembly-disassembly serves as fundamentals for a wide range of scenarios__.\n\n* __Dataset comparisons__\n    * EPIC-KITCHEN [1], and Breakfast Actions Dataset [2] are niche studies focused on daily kitchen actions.\n    * MECCANO [3] and Assembly101 [4] are also studies that focus on the same environment and objects.\n    * Unlike MECCANO [3] and Assembly101 [4], __GazeVQA uses a one-to-one scale 3D version of real-world products__.\n    * __GazeVQA has a high potential to be used in real-world applications__ with more than 125 hours of video clips, more than 40 objects, and steps. \n    * The generalizability concern of the reviewer could be mitigated by the examples that were given to the reviewer Ab7b\n        * Video clips are collected in 3 different video angles (125 hours). More than 40 components and action steps have existed in GazeVQA. \n\nFor evaluation in terms of task-oriented collaborative studies, GazeVQA can be generalizable for numerous collaborative studies such as **human-robot interaction (HRI) studies, human co-worker task assistance, task-assisted video question answering (VQA), and human intention prediction**.\n\n---\n\n### **W2: Applicability of the dataset**\n\n* The recent SOTA model “VIOLET” [9] (100+ citations to date) which has only textual answer type, is used for benchmarking. \n    * VIOLET uses a Video Swin Transformer [10] to encode the images and BERT [11] to encode the text.\n    * Due to the limited time period for rebuttal, the VIOLET [9] model with its pertained weight with 1 epoch was fine-tuned. \n    * __We intend to provide updated results during the designated discussion period__, once model convergence through the fine-tuning process. __Results are shown in the Table below__.\n\n| Methods        |Answering Type | mAP                    |\n| -------------  | ------------- |-------------           |\n| Random Guess   |Text           | 0.07117 (from paper)   |\n| VIOLET [5]     |Text           | 0.09396 (to be updated)|\n| Ours           | Text          | 0.27173 (from paper)   |\n| Ours           | Text + Video + Image | 0.67199 (from paper) |\n\n* __We only fine-tuned the classification head of the VIOLET [9] model, while keeping the parameters of Swin Transformer [10] and BERT [11] frozen due to the time and computation resources limit__.  Hence the VIOLET model is underperforming compared to our results.\n\n\n* __Review for applicability of the gaze information__\n    * Head/hand motion and hand location/pose [5],\n    * Sensors in virtual environments [6]\n    * AR/VR studies [7, 8]\n\n* Gaze information can help to predict which object the user will interact with in the next step by recognizing relevant action (as shown in Fig. 1).\n\n* Gaze information can help multi-modal alignment with high precision for ensuring to processing of the video in real time.\n    * __As shown in Fig. 1, gaze information consists of valuable information to answer questions__ such as “Where does the user gaze at”, “looking at”, “Which component will be assembled next?”, or \"How to assemble <next object>?\".\n    * Gaze information __ensures to location of the objects and components with the help of detailed labeling as shown in our study.__ Thus, it obtains better performance for object recognition or action recognition tasks.\n\n* For the grounding module, it could be mentioned that the Image encoded by Fast RCNN has a different length compared to the clip features within the modules. First, we need to convert them into the same dimensions, like a Conv1D layer to make them have the same length. Then we get the all-feature extractions to put in a multilayer transformer, before the loss check.\n\n---\n\n### **W3: Grammar style and improvements**\n\n* __Ungrammatical Sentences__\n    * The paper has been read many times and has been checked by using multiple grammar-checking programs. **Mistakes were corrected**, such as:\n        * The incorrect sentence was “The first-person view provides a subjective view of the action and can insights into the human’s focus of attention”\n        * It was corrected by adding the word “offer” before the word “insights”.\n    \n* __Dataset Question Styles__\n    * We analyzed a meticulous examination of various question styles and addressed their implications in our dataset. __There are no grammatically incorrect questions but inverted questions prepared to challenge the model__.\n    * The experiments have been done multiple times with different versions of QA pairs. According to our results and efforts for fair benchmarking, we have not encountered a situation that potentially affects the benchmarking.\n\n* __Explicit Captions for Tables__\n    * Finally, I would like to mention that the captions of the tables have been changed as follows:\n        * Table 1: Comparison of the GazeVQA and other action datasets in the context of total hours of video clips, number of videos, average length of video clips (in minutes), number of verbs, actions, and objects.\n        * Table 2: The results for baseline methods which are prepared in different frames with CLIP features and FPV video additions to the TPV videos.\n        * Table 3: Ablation studies on feature encoders to determine the use case differences of CLIP and MViTv2 for Textual and Video features, while Faster R-CNN is used for object features.\n        * Table 4: Ablation studies for usage of different multi-view inputs\n        * Table 5: Ablation studies for the usage of different numbers of encoder layers to get accurate results.\n        * Table 6: The results of the usage of different answer types which are video, textual, and image, individually and collaboratively.  \n___\n\n### **REFERENCES**\n\n1. Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., ... & Wray, M. (2022). Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision, 1-23.\n2. Kuehne, H., Arslan, A., & Serre, T. (2014). The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 780-787).\n3. Ragusa, F., Furnari, A., Livatino, S., & Farinella, G. M. (2021). The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1569-1578).\n4. Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., & Yao, A. (2022). Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21096-21106).\n5. Li, Y., Fathi, A., & Rehg, J. M. (2013). Learning to predict gaze in egocentric video. In Proceedings of the IEEE international conference on computer vision (pp. 3216-3223).\n6. Emery, K. J., Zannoli, M., Xiao, L., Warren, J., & Talathi, S. S. (2021, March). Estimating Gaze From Head and Hand Pose and Scene Images for Open-Ended Exploration in VR Environments. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 554-555). IEEE.\n7. Rivu, R., Abdrabou, Y., Pfeuffer, K., Esteves, A., Meitner, S., & Alt, F. (2020, June). Stare: gaze-assisted face-to-face communication in augmented reality. In ACM Symposium on Eye Tracking Research and Applications (pp. 1-5).\n8. Pathmanathan, N., Becher, M., Rodrigues, N., Reina, G., Ertl, T., Weiskopf, D., & Sedlmair, M. (2020, June). Eye vs. head: Comparing gaze methods for interaction in augmented reality. In ACM Symposium on Eye Tracking Research and Applications (pp. 1-5).\n9. Fu, T. J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021). Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681.\n10. Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022). Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3202-3211).\n11. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The new dataset is large in size and video/image quality seems high. The introduction of this new resource can help promote research in surrounding fields.", "reasons_to_reject": "1. The generalizability of this work is questionable as it only contains videos of assembly and disassembly tasks. Furthermore, all the examples in the paper seem to be conducting assembly and disassembly on the same object? and in the same environment? If this is the case for the whole dataset then generalizability will be a very serious issue.\n2. Throughout the paper, there are many incoherent sentences such as detailed in the grammar/typo section below, which can be quite confusing for readers. While this is less of a problem if just in paper writing, the question in the example in Fig.1 \"What was put by the subject after the output hub cover was put?\" is also not grammatical. This problem could potentially affect the benchmark evaluation if frequently present in the dataset (No code nor data is provided with the submission so its not possible to verify). Thus, the paper and dataset will likely benefit from another round of revision/validation.\n3. From the broad explanation in the introduction (L72-82) alone, it is unclear how Gaze data actually benefits the VQA task in real world scenarios. It would help if the authors can back up this claim with concrete examples from the dataset.", "questions_to_authors": "A. Can you further explain what is the \"Grounding Module in Figure 3?", "paper_topic_and_main_contributions": "This work defines a new VQA task with video input from multiple viewing angles and constructs a video dataset from new recorded videos tailored to the task. The tasks focus on assembly and disassembly of a product, with fpv and tpv videos recorded. The authors also implement a naive baseline fusing together video data and question/choice features in a multilayer transformer.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The are many ungrammatical sentences such as L71-72, L282-284, L284-286, L307-308.\n2. Figure 3 and the tables in page 8 can be better organized.\n3. Tables are missing explicit captions."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has received positive feedbacks from all reviewers. All reviewers agree the dataset is a unique and important contribution to robot-human interaction and collaboration studies, while there have been concerns over dataset generalization and a lack of experimental comparison with other VQA methods, the authors provided satisfactory responses leading reviewers to increase their scores.
Decision: Accept-Main