RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Crowdsourcing.
Crowdsourcing via the Amazon Mechanical Turk platform was utilized for conducting human evaluation. To ensure reliable results, we restricted participation to workers based in Australia, Canada, New Zealand, the United Kingdom, or the United States, with a minimum HIT approval rating of 98%.
Compensation for scene description checking and correction was set at a rate of $15 per hour. Our data collection is classified as an approved exempt protocol from the Institutional Review Board. Details of the interface can be found in Appendix A .
Acknowledgements
We thank our reviewers for their supportive feedback. This work is supported by an Amazon AWS AI/ML Research Award and AWS Cloud Credit for Research. This material is based upon work supported in part by the National Science Foundation under Grants #1821415 and #2048122 REU. The authors are solely responsible for the contents of the paper, and the opinions expressed in this
publication do not necessarily reflect the official policy or position of associated funding agencies or past or present employers of the authors. The contents of this paper are not intended to provide, and should not be relied upon for, investment advice.
A.1 Human Evaluation Interface
We demonstrate the interface of our human evaluation in scene description checking in Figure 9 . We employ manual procedures to guarantee the exclusion of personal information and the absence of offensive content during human evaluations.
Figure 1: The Video Infilling and Prediction Dataset consists of two ways to describe keyframes: an unstructured dense caption and a structured scene description with five components: Focus, Action, Mood, Objects, and Setting ( FAMOuS ). The unstructured dense captions are highly detailed dense captions that can promote visually descriptive reasoning tasks, while structured scene description provide a concise, visual description of the keyframe that can aid in more focused reasoning tasks.; The Video Infilling and Prediction Dataset consists of two ways to describe keyframes: an unstructured dense caption and a structured scene description with five components: Focus, Action, Mood, Objects, and Setting ( FAMOuS ). The unstructured dense captions are highly detailed dense captions that can promote visually descriptive reasoning tasks, while structured scene description provide a concise, visual description of the keyframe that can aid in more focused reasoning tasks.
Figure 2: Distribution of VIP ’s real-world video domains, weighted to emphasize videos containing significant visual change.; Distribution of VIP ’s real-world video domains, weighted to emphasize videos containing significant visual change.
Figure 3: Overview of the pipeline to generate the scene descriptions provided in the VIP Dataset. We first process a video and extract the important frames ( section 3.1 ), then generate scene descriptions by extracting visual information from each keyframe, along with grounding information from the video to offset model hallucinations. We then feed in the extracted information into GPT-4 to generate the dense captions and structured scene descriptions. ( section 3.2 ).; Overview of the pipeline to generate the scene descriptions provided in the VIP Dataset. We first process a video and extract the important frames ( section 3.1 ), then generate scene descriptions by extracting visual information from each keyframe, along with grounding information from the video to offset model hallucinations. We then feed in the extracted information into GPT-4 to generate the dense captions and structured scene descriptions. ( section 3.2 ).
Algorithm 1 frame_extract( v , c , f ); frame\_extract returns a list of f selected keyframes from a video v . First, we extract c candidates using Katna. These keyframes are embedded using CLIP in the image space; Detic extracts objects from the keyframes into a textual representation, which are also embedded with CLIP . Then, we iteratively prune the keyframe with the highest cosine similarity with adjacent frames until f keyframes remain.
Figure 5: Given a number of context frames, the frame prediction task requires models to predict the following n frames. In this example, we provide two FAMOuS scene descriptions and use Vicuna and GPT-4 to predict the next three frames. Results emphasized in red differ from the ground truth.; Given a number of context frames, the frame prediction task requires models to predict the following n frames. In this example, we provide two FAMOuS scene descriptions and use Vicuna and GPT-4 to predict the next three frames. Results emphasized in red differ from the ground truth.
Figure 6: Input prompt templates used to test for prompt variation ( Table 2 ). To reduce complexity, all subsequent results are reported using Prompt 1, which was designed to align with the widely recognized chain-of-thought prompting approach, adapted for video: “Let’s think frame by frame.”; Input prompt templates used to test for prompt variation ( Table 2 ). To reduce complexity, all subsequent results are reported using Prompt 1, which was designed to align with the widely recognized chain-of-thought prompting approach, adapted for video: “Let’s think frame by frame.”
Figure 7: Input prompt with a single in-context demonstration for the Infilling-1 task.; Input prompt with a single in-context demonstration for the Infilling-1 task.
Figure 8: Input prompt with a single in-context demonstration for the Prediction-1 task.; Input prompt with a single in-context demonstration for the Prediction-1 task.
Figure 9: Amazon Mechanical Turk Platform Interface, where crowdworkers are asked to qualitatively judge the accuracy of a FAMOuS scene description.; Amazon Mechanical Turk Platform Interface, where crowdworkers are asked to qualitatively judge the accuracy of a FAMOuS scene description.
Statistics of video datasets: whether they include frame-level ( Frame ) and structured descriptions ( Structured ) as well as the average video token length ( Vid. Len. ), average caption token length ( Cap. Len. ), and number of inference samples ( Test Samples ). VIP is the only video reasoning dataset that has an open video domain and frame-level descriptions and provides novel structured frame descriptions.
Average GPT-4 performance across three prompts ( Figure 6 ) on Infilling-1 task
reported as mean \pm one standard deviation. The low standard deviation indicates prompt stability.
Model performance on infilling and prediction tasks when outputting three frames, with the best results underlined .. We vary the number of context frames as indicated by the dash in task name (e.g., Infilling-2 uses two previous and two future context frames to predict three intermediate masked keyframes; Prediction-2 , uses two preceding keyframes to predict the subsequent three keyframes). We evaluate models using both FAMOuS structured scene descriptions and unstructured dense captions. Models show weak performance overall.
Model performance on the Prediction-3 task; the two leftmost numerical columns report the aggregate results, while the five rightmost columns report the individual FAMOuS component results. The best results on the component level is underlined for each metric. Language models perform worse predicting Action and Setting with textual representation inputs, highlighting where video representations could be most beneficial.
GPT-4 performance for videos partitioned by the physical (changes due to outside, real forces) and social (changes related to social cues) on the Infilling-1 task with structured, FAMOuS descriptions. Significant differences between partitions are underlined .
Average model performance across the Infilling-1 , Infilling-2 , Prediction-1 , Prediction-2 , and Prediction-3 tasks, partitioned by domain. Categories such as Games, Sports, People & Society, Health, and Science perform better on average, while others such as Jobs & Education, Law & Government, and Food & Drink perform less well. The Best results for each metric and textual representation are underlined .