THAVQA: a German task-oriented VQA dataset annotated with human visual attention
Authors: Moritz Kronberger, Viviana Ventura (Technische Hochschule Augsburg, 2024)
Topic: Language Models

Abstract
Video question answering (VQA) is a challenging task that requires models to generate answers by using both information from text and video. We present Task-oriented Human Attention Video Question Answering (THAVQA), a new VQA dataset consisting of third- and first- person videos of an instructor using a sewing machine. The sewing task is formalized step-by-step in a script: each step consists of a video annotated with German language open-ended question and answer (QA) pairs and with human visual attention. %This data collection is the first step toward a larger goal: to have an AI assistant help the user solve the task by answering questions about how to properly use the sewing machine. The paper also includes a first assessment of the performance of a pre-trained Multimodal Large Language Model (MLLM) in generating answers to the questions of our dataset across different experimental settings. Results show that our task-oriented dataset is challenging for pre-trained models. Specifically, the model struggles to answer questions requiring technical knowledge or spatio-temporal reasoning. %%Model performance improves when it has as input text that contains the specific technical knowledge.

Introduction
This paper presents a new VQA dataset based on demonstrating basic sewing machine operations.To our knowledge, THAVQA,which is also annotated with human visual attention, is the first task-oriented VQA dataset in German language. The dataset building is a first step in the larger project aimed at developing an AI-assistant for a sewing machine workshop held at the Technische Hochschule Augsburg. This AI-assistant would support students when using sewing machines for the first time. For example, this could mean answering questions about basic machine settings or explaining fundamental sewing skills. Our dataset poses unique challenges for VQA models and is almost unique in the state-of-the-art VQA datasets since it is user- and task-oriented: the questions collected are those that a real user would ask for help while using the sewing machine. The process of operating the sewing machine was decomposed in a script into steps and sub-steps that were recorded and on which questions and answers were annotated. Specialized knowledge of the process and understanding of spatial and temporal relationships is required for answering the questions collected. In addition, the limited visual variety of the video scenes and the specialized language and dictionary challenge the models for VQA. Annotating human attention in the video inputs of VQA models has recently been shown to improve their performance in user- and task-oriented datasets. In our dataset, the workshop instructor's eye gaze has been used as a proxy for human visual attention. The concept behind it is that visual human attention integrated as input into models for VQA can help the model distinguish between video frames, especially in datasets in which recorded scenes are very similar to each other as there are few participants and staged events. Our paper also provides a first assessment of the VQA performance of the pre-trained MLLM Gemini 1.5 Pro on THAVQA. Indeed, new releases of LLMs, such as Gemini 1.5 but also GPT-4, Llama 2, or Claude 3, now allow for visual inputs, making it possible to perform VQA tasks using pre-trained models directly. To sum up, this paper presents: A new dataset with third-person videos of an instructor operating a sewing machine and first-person videos annotated with visual human attention, QA pairs in German, and a script in German of the steps required to operate the machine. An evaluation of the performance of a pre-trained MLLM on generating open-ended answers from questions and videos of our dataset.

Conclusion
We provide a new task-oriented, German-language VQA dataset on demonstrations of sewing machine operation with open-ended human QA pairs and human visual attention: THAVQA. We then compared the VQA performance of Gemini 1.5 Pro on THAVQA varying the model inputs. We found that the task-oriented scenario of THAVQA was specific enough, such that the model could not rely on only its inherent knowledge to generate satisfactory responses. The questions contained in our dataset were over the capacity of the model to reason about the video data. Combining textual instructions with a first person video resulted in the best performing model across all reasoning types of questions. When looking towards the design of a VQA model for a future, practical sewing machine assistant, video inputs could therefore be used mainly to improve the model's perception abilities, while a retrieval system for textual information could provide the necessary specialized knowledge.