SlowGuess/NeurIPS_2023_OCR huggingface dataset:
2. Roughly speaking, we expect about  $20 - 25\%$  of the questions that we have provided to be found as good. However, this is not necessary and the  $\%$  age can be smaller or larger depending on each three-minute clip.
Edge Cases:
1. If the asked question has multiple answers and at least one of them aligns with the correct answer while none of them align with any of the other wrong answers, then provided that the top 5 conditions are met, we can mark the set as good.
2. If two questions are very similar (even within different clips) and both are GOOD, only choose one as GOOD and reject the other one with a comment mentioning this. We do not expect this to happen more than 1 or 2 times in a 100.
3. There might be more such edge cases, please feel free to contact me in such cases and we can expland.
8 Model evaluation by different temporal certificate length
We labeled temporal certificates for 86 clips and classified them into three distinct categories, delineated by the length of the certificate: Category A (30 to 75 seconds), Category B (75 to 133 seconds), and Category C (133 to 180 seconds). These categories encompassed 37, 35, and 14 instances, respectively. Further, we evaluated the performance of 5 frame mPLUG-Owl, a state-of-the-art 7 billion parameter model, across these defined categories. The accuracy rates were found to be  $35\%$ ,  $20\%$ , and  $20\%$  for Categories A, B, and C, respectively. These results underscore the
inherent complexities associated with longer temporal lengths in the context of model performance, even among advanced models. We hypothesize the reason for the accuracy drop in mPLUG-Owl to be the train test discrepancy the model faces in long videos in inference. mPLUG-Owl is trained with images only and uses per-frame image representations for representing video. Hence, with a large number of frames, the performance starts degrading since the network is not trained to handle it in training. We leave it to future work in long-term video understanding to further experimentally verify this hypothesis and mitigate this challenge.
Certificate sampling. Building upon the aforementioned experiment, we further extended our investigation by benchmarking the 5-frame mPLUG-Owl model specifically on the same corpus of 86 clips. However, in this analysis, frames were exclusively sampled from within the annotated temporal certificate. The derived accuracy for this method was  $31.7\%$ , which is  $7\%$  higher than the baseline accuracy of  $24.7\%$  that was obtained when frames were sampled uniformly throughout the entirety of the clip. This outcome further reinforces the significance and validity of the temporal certificate concept.
9 Annotator agreement.
To rigorously evaluate the robustness and consistency in the interpretation of the "temporal certificate" concept, we conducted an experiment involving two independent annotators. These annotators were tasked with evaluating 86 clips from EgoSchema, in order to measure the inter-annotator agreement using the intersection over union metric between the marked certificate sets. We find the IoU agreement to be  $54.3\%$  over the entire set. This denotes that while the exact certificate sets do have some subjectivity, annotators still largely agree on the long-term nature of certificates. Note that in the EgoSchema collection, we only use temporal certificates as a qualifying criterion ( $>30$  seconds) and have a median length of 100 seconds.
10 Benchmarking details
10.1 Violet
Violet is a video language model comprised of a visual encoder, text encoder, and multimodal transformer pretrained on a variety of masked visual modeling tasks ranging from simple ones such as RBG pixel values up to more high levels ones such as spatially focussed image features. It performs competitively on a variety of video-language tasks such as Video-QA and Video-Text Retrieval. We evaluate one pre-trained model and 3 models finetuned on lsmdc-mc, msvrtt-qa, and msvrtt-retrieval. We evaluate using both 5 frames and 75 frames and choose the model with the best overall accuracy.
10.2 mPLUG-Owl
By default, mPLUG-Owl does not possess inherent capabilities for direct video question answering. As such, we undertook several experiments to adapt it to our required format. One approach involved inputting all answer choices in the form of a shuffled test. However, this resulted in a bias towards selecting the first option in most cases. For another approach, FrozenBiLM offered a methodology for frozen zero-shot models to operate in the context of multiple-choice video question answering, which inspired us to adapt this methodology for mPLUG-Owl. As mPLUG-Owl utilizes word-level tokenization, we could extract the confidence score for each generated token, particularly the 'Yes' token. We recorded the 'Yes' token confidence score for each answer option. In instances where the 'Yes' token was absent, we assigned the confidence score as zero, though empirically, in most cases, the model output was positive and contained the 'Yes' token. Ultimately, we selected the answer option with the highest 'Yes' confidence score as the model output given the question. In scenarios where multiple options scored the same highest confidence for the 'Yes' token, we randomly selected the answer from these top-scoring options. It should be noted that mPlug-Owl was originally trained to process a single image, and its capacity to handle additional frames is an emergent ability that has not been thoroughly tested to date."
10.3 InternVideo
The two most closely aligned formats supported by InternVideo are open-ended Video Question Answering and Zero-shot Multiple Choice tasks. In the case of open-ended Video Question Answering, the task is to predict the answer to a question posed within a video. However, due to the restricted vocabulary of open-ended answers in open-ended Video Questions Answering, we decided to formulate EgoSchema within the context of a Zero-shot Multiple Choice task. This task aims to identify the correct answer from a set of given options, without the inclusion of a question. InternVideo has provided finetuned weights for two datasets: MSRVTT and LSMDC. We selected the model finetuned on MSRVTT because it shares greater contextual similarity with EgoSchema.
10.4 Human
To conduct human benchmarking, we engaged a distinct team of ten employees within the same data annotation company to carry out human benchmarking on our dataset. The answers were randomized and presented in the form of a test. The following are the precise instructions provided to the annotators:
- Setting 1: Unlimited setting -- The goal is to get answers as accurately as possible without worrying about time.
- Setting 2: 1 minute timed setting -- In this case, the test taker (annotator) has only 1 minute to spend per question (including watching video/reading text/ everything). If they do not have the answer, just guess the best based on their intuition and move on.
- Setting 3: 3 minutes timed setting-- Same as above but with 3 min instead of just one.
- Setting 4: Video -> Text setting -- In this case, the taker is not allowed to read the text before looking at the video and at the video after reading the text. In other words, the test taker can spend as much time as they want to look at the video first and then must move on to answering the question. They cannot go back to the video once they start reading the text. This is an untimed setting -- they can take as much time as they want per question.
- Setting 5: 180 frames setting -- This is the same as the untimed setting except the annotator has access to only 1 frame per second (ie the video feels like a GIF with one frame per second instead of the usual 30 frames per second) -- each video is still 3 minutes long, but it feels more jittery. All instructions remain the same as in untimed setting.
Word Count Distribution Across Question, Correct Answer and Distractor Answers
Truncated Word Frequency Distribution for Question, Correct Answer and Distractor Answers
Figure 9: Interactive Version of these statistics visualizations can be found at the statistics page on our website.