SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models
Authors: Haotian Xia
Zhengbang Yang
Junbo Zou
Rhys Tracy
Yuqing Wang
Chi Lu
Christopher Lai
Yanjun He
Xun Shao
Zhuoqing Xie
Yuan-fang Wang
Weining Shen
Hanjie Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Models
Sports Understanding
Benchmark
Primary area: datasets and benchmarks

Abstract: Multimodal Large Language Models (MLLMs) are advancing the ability to reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models' ability to reason about sports solely through question-answering (QA), without requiring visual inputs; SPORTU-video, consisting of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluated four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. GPT-4o achieves the highest accuracy of 71\%, but still falls short of human-level performance, highlighting room for improvement in rule comprehension and reasoning. The evaluation for the SPORTU-video part includes 6 proprietary and 8 open-source MLLMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. GPT-4o performs the best with only 57.8\% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models' capabilities in sports understanding and reasoning. The dataset is available at [https://github.com/chili-lab/SPORTU](https://github.com/chili-lab/SPORTU).

Published as a conference paper at ICLR 2025
SPORTU: A COMPREHENSIVE SPORTS UNDER- STANDING BENCHMARK FOR MULTIMODAL LARGE LANGUAGE MODELS
Haotian Xia Zhengbang Yang Junbo Zou Rhys Tracy* Yuging Wang
Chi Lu Christopher Lai' Yanjun He² Xum Shao² Zhuoqing Xie Yuan-fang Wang' Weining Shen Hanjie Chen
'Rice University "University California, Irvine *George Mason University
"University of California, Santa Barbara Stanford University
fxiah6, weinings) } @uci.edu, hanjie@rice.edu
ABSTRACT
Multimodal Large Language Models (MLLMs) are advancing the ability reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strat- egy understanding. This component focuses on testing models' ability to reason about sports solcly through question-answering (QA), without requiring visual inputs: SPORTU-video, consisting of 1,701 slow-motion video clips across dif- ferent sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluated four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. GPT-4o achieves the highest accuracy of 711 %,, but still falls short of human-level performance. highlighting room for improvement in rule comprehension and rea- soning. The evaluation for the SPORTU-vide part includes proprictary and 8 open-source MILMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. GPT-4o performs the best with only 57.8% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models' capabilities in sports understanding and reasoning. The dataset is available at lattps://githu> com/ http/chiiiiii -allilllabaaee 1aa/--aaa//PPOTUU SPORTU
INTRODUCTION
The sports domain has witnessed surge in interdisciplinary research, combining Natural Language Processing (NLP) and computer vision (CV) to tackle a wide range of applications. For instance, NL.P-baedd approaches have been leveraged for automated sports news generation, producing de- tailed summaries and news articles from game data (Huang et al.) 2020a Wang et al. (2022)). Con- currently, hate speech detection has been employed mitigate the impact of toxic content on social media (Vujičić Stanković & Mladenović 2023). enabling athletes to maintain focus on their game. In the realm of CV, action recognition (Zhu et al.) [20226] [i et al.) 2021). player detection (Maglo) al.) 22222, Vandeghen et al.) (2022), and tactical analysis (He et al. [024bb] Xia et al.) (023) have been explored, enhancing visual content for analysis and fan engagement. The recent emergence of
Large Language Models (LLMs) (OpenAl 2024c; AT@ Meta 2024; Jiang et al. 2024 Anil et al., 2023) and Multimodal LLMs (MLLMs) (OpenAl 2024b) Gemini Team
2024a Anthropic 2024a Lin et al., 2023) has further accelerated this trend, enabling researchers to develop novel tasks such
as Al-assisted refereeing (Held et al. 2024a 2023). where models analyze game videos to identify
Work done Rice University

---

Published as a conference paper at ICLR 2025
Somario base Quartor::
Qverttr
weeeiing the that All paes Royer acdiumally rop den
Teams
second tiek Trum Flaer be?
Exfevatoon Aecadiigg FBll
Unda slesing
knsent fere dew
gestivee
Saquiee
Hard Level
(Surdirs W0i i fail videe
Easy Dueran quat till
tie alon the nettise the wolking in oeiteed
from bndce the thee anter fic
Dovee
Figure 1: SPORTU consists of two parts: SPORTU-text, which evaluates sports understanding through text-based multiple-choice questions, and SPORTU-video, which assesses multi-level rea- soning through video- QA tasks. SPORTU provides a comprehensive sports understanding evaluation of multi-leveled reasoning beyond perception. Right side is a sample from the scenario- based question in SPORTU-text, along with examples of both hard-level and easy-level questions from SPORTU-video.
fouls and violations, and interactive sports education Zhang et al. 2025 Zeng et al. 2023 where users engage with LLMs to learn rules, strategies, and game-related content.
However, the effectiveness of these applications depends crucially on the model's deep understand- ing of sports knowledge. While LLMs act as study guides, helping users learn rules and general strategies through text, MLLMS extend this knowledge to video-based tasks that require video and action perception, well as the ability to connect movements with context-based rules. For ample, when answering a question like "Why is it a rule violation in the video?", a model must distinguish the series of actions performed by the players and understand the corresponding rules that define the fault. This capability underscores the deeper reasoning required for real-world sports comprehension. The challenges, ranging from general video recognition deep sports knowledge reasoning, highlight the need for a dedicated sports-focused ceestion-answering (QA) dataset to improve the model's 's ability to comprehend and contextualize sports information effectively.
Existing sports QA datasets, either text-based or video-based, have limitations that hinder a com- prehensive evaluation of sports understanding. Text-based datasets (bench authors] [023) Liu al.]
2020 Xia et al. 2024aa Jardim et al. 2023) primarily assess comprehension of numerical data, rules, and context extraction, but lack detailed explanations to evaluate the underlying reasoning processes. In addition, video-based datasets, such as SportsQA (Li et al.) 2024a) and SoccerNet-XFoo ct al., 2024bb), focus on action understanding and multi-view QA, but are constrained by their nar- row scope, covering only single sport lacking multi-level reasoning. For example, in SportsQA, questions like "What does TEAM A do hefore/after TEAM B's action?" mainly require recognizing sequences of actions and their outcomes, rather than connecting these actions to the underlying rules and game dynamics. This highlights the need for a comprehensive multimodal sports benchmark to evaluate the capabilities of MLLMS across a diverse range of sports, with varying levels of difficulty, to assess their ability to apply deep reasoning and based understanding in real-world scenarios.
(Held
To address this gap, we introduce SPORTU, comprehensive Sports Understanding Benchmark for sports knowledge and slow-motion multilevel video reasoning. As multifaceted benchmark, SPORTU comprises both text and video components to facilitate thorough assessment of models' capabilities. As illustrated in Figure [ our dataset includes two parts: SPORTU-tex and SPORTU- video. The text component, SPORTU-text, features 900 multiple-choice questions with human- annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models' ability to reason about sports through snuestion-asswenen independent of visual inputs. The video component, 300 clips with varying camera angles and 12,048 QA pairs, categorized into three difficulty levels. The easy level is designed to be answerable without requiring sports domain knowledge, while the
POORTU-vddeo comprises 1,701 slow-motion video clips, including
2

---

Published as a conference paper at ICLR 2025
hard level demands in-depth rule comprehension and accurate video perception. This tiered struc- ture allows SPORTU-video to evaluate the sports understanding capabilities of MLLMs in a more nuanced and progressive manner. The use of slow-motion clips is crucial, as most fouls involve brief and subtle actions that may be overlooked in real-time footage. By providing models with a better opportunity to perceive and interpret these critical moments accurately, evaluate their performance.
more effectively
To gain deeper insights into SPORTU, we evaluated 4 LLMs on SPORTU-text and 14 MLLMs on SPORTU-video, covering both open-source and proprietary models. For SPORTU-texI, we selected GPT-40, Claude-3.5- 3.nnnnnet, LLaMA-3.1, 1, and Gemini 1.5 Pro, using few-shot learning paradigms. For SPORTU-video, we evaluated a broader range of models, including GPT-4o, Gemini 1.5 Flash, and Video-ChatGPPT, using multi-frame inputs.
Key results reveal that GPT-4o achieves the highest accuracy on SPORTU-text at 71%, highlight- ing its relatively strong performance in text-based sports reasoning. On SPORTU-video, Qwen2- VL 72B achieves the highest overall accuracy of 70.94%. but struggles on hard-level tasks with only 44.12% 12% accuracy. Claude-3.5-Sonne demonstrates the best performance on hard-level tasks (52.57%). Yet, results across all models reveal significant challenges in handling complex reasoning and rule comprehension, particularly in tasks requiring deep sports knowledge. We also systemati- cally applied different prompt strategies to evaluate reasoning performance in SPORTU-video. The results reveal that models generally achieve the highest overall performance when directly predicting the answer without providing rationale. However, performance declines when models are required to generate a rationale first and then predict the answer. For instance, Claude-3.5-Sonnee racy dropped from 52.57% with direct answer prediction 39.32% when reasoning the rationale first. This indicates current models struggle to maintain consistency and robustness in reasoning for complex tasks. These findings highlight the need for future advancements reasoning capabilities.
accu-
2 RELATED WORK
2.1 MULTIMODAL SPORTS ANALYSIS
Recent surveys, such as (iaa et al. 2024b), have highlighted the importance of integrating sports with NLP and CV techniques to advance research in the sports community. Traditional text-based applications have have encompassed a range of tasks, including sentiment analysis (Baca et al. 2023
Ljajić et al. 2015), game predictions (Beal et al.. |021] Xia et al., 2022 Oved al. 2020 Tracy
et al.] 2023), game statistics summarization (Thomson et al. 2020 Hu et al. 2024a), and game news
generation and narrative construction (Sarfati et al. [023] Huang et 20200b Wang et al. 2022a
Hu et al. 2024b). In the CV domain, studies have focused on sports action recogmition (Xu et al.,
2024b et al. [202cc] Yuan et al. 202T Zhu et al. 2022a),, sports action quality assessment (Zahan
et al. 2024| Xu et al. 2024a),, and tactic analysis (He et al. 2024b). Notably, Chen et al. (2022)
has demonstrated the efficacy of multimodal integration by leveraging natural language input to
enhance sports videos with visualizations. While prior work has made significant strides in sports understanding, the advent of MLLMS offers fresh perspective on this domain. In contrast to earlier multimodal approaches, MLLMs boast distinctive blend of scalability, expressiveness, and flexibility, thereby enabling the tackling of intricate sports understanding tasks with unparalleled depth and nuance.
2.2 MULTIMODAL SPORTS QA
Question answering (QA) has been widely adopted to evaluate the comprehension abilities of LLMs across various domains. Several QA datasets have been introduced to evaluate models' understand- ing of textual information, ranging from factual recall to multi-hop reasoning (Joshi et al. 2017] Yang et al.) 2018 Clark et al.) 2019). In the multimodal domain, QA Benchmarks have been ex- tended to evaluate MLLMs' image and video understanding by answering questions based on visual inputs (Fu et al. 2024 Yue et al. 2024a b He et al. 2024a Zhou et al., 2024). However, sports- related QA is underrepresented in these general datasets, and even when sports topics are included, the questions often lack sufficient difficulty due to the datasets' broader focus. Existing general QA datasets tend to focus on surface-level aspects of sports, such as historical facts or well-known events in text-based tasks, and perception-level tasks, like object action identification in video-
3

---

Published as a conference paper at ICLR 2025
based tasks. As result, they often fail to assess the deep, domain-specifet knowledge and reasoning required for a nuanced understanding of sports.
Existing sports ceatts-domii--sccicii QA benchmarks are limited in their ability to test models' sports understanding. Text-based datasets, such as BIG-bench (bench authors (023), LiveQA (Liu et al, 2020), and QASports (Jardim et al. 2023), focus factual recall, while SportA (Xia et al. 2024a) is a notable exception, introducing rule- questions that require scenario- based reasoning. How- ever, even SportQA lacks explanations, which are crucial for evaluating MLLMs' reasoning pro- cesses, particularly for complex tasks. The absence of explanations limits the ability to fully assess model's 's reasoning capabilities.
On the other hand, sports poots-domain-speciff VQA datasets that combine video and text modalities test comprehensive sports understanding are scaree. Sports-QA (Li et al., (024a) stands out by covering eight sports, including volleyball and basketball, with videos sourced from MultiSports (Li al. 2021) and FineGym (Shao al., 2020). However, although it provides detailed action recog- nition annotations, Sports-QA does not assess models for understanding sports rules, such as foul detection. This limitation arises because MultiSports does not include foul clips, and FineGym focuses solely on granular gymnastic actions, which do not encompass rule-based scenarios.
Foul detection requires models to recognize player actions and determine rule violations, making critical component of sports understanding. Another benchmark, SoccerNet-XFoon (Held et al. 2024b), evaluates soccer understanding, including rule violation detection and explanation. How- single sport limits its generalizability, may not extend to others with different rules and dynamics. To address the limitations of previous works, we introduce SPORTU, comprehensive benchmark that spans multiple sports and incorpo- rates both rule-based reasoning and foul detection across text and video modalities, offering a more diverse and in-depth evaluation of sports understanding.
ever, its focus
MLLM's performance in sport
SPORTS UNDERSTANDING BENCHMARK
Officily Ecay
Corest
liness cosnd &
Streeedy Relstid
Offic.lyy
upppring corili the tiik and bucenee
Haman Expleatioo: walieo.
eeslsss sneg conk for comsend look
bat ge theniig right
loed bookk and mor can regi-
dacias
Senoacio Related
OificuPl Hord
de suttonnde mady some dhos to (lew Why retroe dedked wond to
Thecadio Bwo foeell ou dn funcle bell of del lunktlt vur io be
delnche Teas must colles saalo
aff
Cores Kameel
cimereee...... Hemas
Opeater Mhee lod in
Areiging Lighnng beandng Tt duuiing
Figure 2: Examples of SPORTU-Iext (left) and SPORTU-video (right). Figure D also shows an example of an open-ended SPORTU-video question.
To address the limitations in the current sports-domain QA dataset and evaluations, introduce SPORTU. This benchmark consists of two datasets:
SPORTU-text the first pure-text-based sports QA dataset designed to evaluate models' under- standing of rules, factual knowledge, scenario-base based situations, and strategies with explanations. It contains 900 QA pairs, with each question having one or correct answers.
4

---

Published as a conference paper at ICLR 2025
SPORTU-video the first VQA dataset covering multiple sports, designed to evaluate MLLMs' sports understanding abilities. It features a multilevel question design using slow-motion videos, with some clips offering multiple camera angles. The tasks range from easy-level tasks, such as sports recognition, to medium-level tasks, such as Team Role Recognition, and hard-level tasks, such rule violation explanations. The dataset contains 1,701 slow-motion video clips across sports, with 12,048 QA pairs designed to test models' multi-level vidco-text reasoning capabilities.
Overall, SPORTU provides a comprehensive evaluation of MLLMS' ability to understand and apply sports knowledge. It fills the existing gap in current sports QA benchmarks by offering detailed assessment across both text-based reasoning and multimodal video tasks.
3.1 QUALITY CONTROL
To guarantee the accuracy and consistency of annotations and question generation in SPORTU, employed a rigorous quality verification process. Our team of annotators consisted of nine experts: two intercollegiate student-athletes with over 12 years of experience, and seven were players who had undergone at least 5 years of training in their respective sports. It helped maintain the high standard of explanations and annotations for both SPORTU-text and SPORTU-video.
During the training phase, each team member worked with twenty examples per batch for both text- based questions and VQA tasks. They were asked to: Annotate explanations for the SPORTU-text dataset. Collect videos for SPORTU-video and annotate the key information necessary for generat- ing questions. These key annotated variables were later used in a template-based system to generate questions. Once each annotator demonstrated full mastery of the annotation process, we officially launched the large-scale annotation phase. As part of our verification protocol, annotators were re- quired to double-review the videos they collected to ensure accuracy and quality. We prioritized the removal of controversial or hard-to cointtoointerrret clips, especially those where even human experts might disagree feel unsure about the decision, to minimize the risk of mislabeling.
3.2 SPORTU-TEXT: PURE TEXT QA
SPORTU-text is designed as the first pure-text- sports QA dataset that provides detailed expla- nations for each question option, aiming to evaluate models' understanding of rules, factual knowl- edge, scenario-based reasoning, and strategies. The dataset consists of 900 QA pairs, with each question having one or more correct answers and accompanied by human- annotated explanations to ensure high-quality assessment of reasoning processes. SPORTU-text can also serve as a bench- mark for model explainability, allowing researchers to compare models' generated reasoning with human-provided explanations.
Among all sports-specific QA benchmarks, only SportQA (Xia et al, 2024a) Level-3 questions assess models' deep understanding sports knowledge by covering rule-, strategy- and scenario-based questions. However, these questions are mixed together without clear labels for each type. To build SPORTU-IeXI, we randomly selected 900 questions from five differ- ent sports: American football, soccer, volleyball, basketball, and tennis, and our expert annotators manually categorized the selected questions. Rule-related questions focus on explicit sports rules, strategy-related questions involve tactical or strategic decisions, and scenario-related questions pro- vide specific context or player interaction (e.g.. "Player performs action X, and Player B reacts"). While some questions could fit multiple categories, annotators assigned each question to the cate- gory that most accurately reflected its primary focus. Each question was carefully annotated, with whether detailed explanations provided for each - correct incorrect. Annotators explained why an option was correct or not, offering clear insights into the reasoning required for each answer. This additional annotation step ensures more structured dataset and enables the evaluation of mod- els' reasoning capabilities across distinct aspects of sports knowledge. Examples can be found in the appendix O
Dataset Construction
3.3 SPORTU-VIDEO MULTIMODAL VIDEO QA
SPORTU-video is the first multimodal video QA dataset designed to evaluate MLLMs' sports un- derstanding across various tasks, with a specific focus on integrating visual and textual reasoning. It is unique in its use of slow-motion video clips across multiple sports and includes multilevel
5

---

Published as a conference paper at ICLR 2025
question design to test range of model abilities, from simple recognition to complex rule-based reasoning.
SPORTU-video consists of 1,701 slow-motion video clips across 7 different sports, including soc- cer, basketball, volleyball. ice hockey, tennis, baseball, and badminton. We also ensured that for some sports, the videos featured multiple camera angles to challenge the models' ability to capture consistent judgments across different perspectives. Our expert annotators manually cropped video clips from replay footage to include multiple perspectives of the same foul, such replays are stan- dard in sports broadcasts. This process ensures accuracy while minimizing additional manual work. Specifically. 300 video scenes were selected to include multi-angle views. Each clip is accompanied by one or more QA pairs. for a total of 12,048 QA pairs (with 10,973 Multiple Choice Questions and 1,075 open-ended questions based on the explanations that the annotators labeled), with three levels of complexity: Easy: 25.36%, Medium: 50.22%, Hard: 24.42%.
Dataset Construction The construction of SPORTU-video began by identifying the types of tasks that could be asked based on the sports domain. Some questions, such as sports recognition and identifying the number of players, are common across all sports. Other questions are sports-specitt requiring knowledge of rules strategies for each sport. Full question templates can be found in the appendix N. To classify the questions, we considered three different difficulty levels based on the required sports knowledge and reasoning. Easy-level questions: These tasks rely on commonsense, such as basic sports recognition. For example, questions like "What sport does the video show?" or "How many players shown?" do not require sports knowledge. Medium-level questions: These questions require sports knowledge beyond commonsense. For example, models are asked to identify which team on offense based on the video or recognize specific roles, such libero in volleyball, by identifying the libero's 's jersey color. Hard-level questions: These tasks involve deep rule-based reasoning. For example, identifying rule violations or technical errors requires models to understand the sport's 's rules in detail and apply them to the specific context of the video.
Once the question types were defined, the annotators collected the appropriate video clips and la- beled the corresponding ground truths that the videos showed, as multiple questions could often be answered from a single video. There no restriction on the data source for these slow-motion clips as long as the annotators follow the original copyright and license requirements, and each video was manually cropped to ensure high-quality footage suitable for detailed action analysis and rule com- prehension. For multiple-choice questions, annotators labeled the ground truth category, such the specific foul ("handball" or "offsid"), which was then used to generate multiple-choie questions with distractors derived from other categories. For open-ended questions, annotators provided de- tailed explanations for the rule violation or foul observed in the video. These explanations were used as the ground truth for generating open-ended questions, allowing models be tested reasoning and explanation generation.
Explanations One of the unique aspects of SPORTU-video is its emphasis on tasks involving foul detection and technical errors, where open-ended questions are accompanied by detailed human- annotated explanations. These explanations provide insights into why certain fouls or errors oc- curred, helping to evaluate models not only on their accuracy but also on their ability to explain the reasoning behind their answers. Due to limited annotation resources, these explanations are only provided for the most challenging tasks involving rule violations and technical errors, these require models to combine action recognition with textual rule understanding to determine the vio- lation. We believe that these tasks offer the most rigorous test of a model's ability to connect sports knowledge with video comprehension.
4 EXPERIMENT
We compare MILM's performance on the SPORTU benchmark, We also evaluate the ability of models to produce explanations. We start by describing the MILMS in our experiments and their experimental vettings($4.)) followed by prompting strategies (44.2). and evaluation metrics ( 4.3)
4.1 MODELS
SPORTU-text Evaluation: We evaluated several leading language models on the SPORTU-text, including open-source models like 1amma-3.1-4055 (Dubey al. 2024), and closed-source models

---

Published as a conference paper at ICLR 2025
such GPT-4o (2024-08-06 version) OOpenAl 2024a), Gemini-1.5 Pro(Gemini Team 2024a), and
M.ade-3,5-Sonne (20240620 version) (Anthropic 2024b). Access to these models was facilitated through their respective APIs.
SPORTU-video Evaluation: We investigate a range of MLLMs, including close-source models
and 6 open-source models. For close-source models, evaluated GPT-4o (2024-08-06 version)
and -4omini (OpenAl 2024a), Gemini 1.5 Pro (Gemini Team 2024a), Gemini 1.5 Flash (Gemini
Team
2024b),
(20240620 version) (Anthropic 2024b) and-3.0 Haiku Anthropic
(2024]
For open-source models, we evaluated ChatUni Vi (Jin et al. 2023), LLaVA-NeXT (Liu et al.
2024..
mPLUG-OW13 (Ye al., 2024). Tarsier (Wang al.) 2024), Indeo-ChatGPT (Maaz et al,
2024
VideoChat2 et al.] (024b), ST-LLM (Liu et al. (025), and Qwen2-VL-722 (Bai et al,
For the closed-source models, we adhered to the default settings provided by their official APIs. GPT and Claude family models processed ten image frames extracted from the video con- tent as input. The Gemini family models processed the entire video, as their API supports video input. Due to computing resource limitations, we used the 7B versions of all open-source mod- els in this evaluation except Qwen2-VL-72B, which can be accessed by API. For VideoChat, we set 'max frammess to 100, while for the other open-source models, used 16 frames as input. Across all closed and open-source models, we set the temperature parameter to to ensure consistent response generation. All inferences are on an NVIDIA RTX A6000.
2023
4.2 PROMPTING STRATEGIES
We apply three different prompting strategies to generate answers and/or explanations. We represent the input as X for the question and answer options, Y for the answer, and R for the explanation (rationale). Our three strategies are:
X + Y: No-CoT, which directly predicts the answer.
X + RY: Reasoning where answer inference is conditioned to the rationale. This strategy asks the model to engage in step-by-step reasoning first, and then answer the question. This approach is based on chain of thought (CoTT) prompting, which has been shown to improve LLMs' prediction accuracy across various reasoning tasks Wei et al. (2022). The zero-shot CoT method is adapted from Kojima al. (2022). We prompt the model 'think step by step, then provide the correct answer."
This strategy asks the model to answer the question first, followed by the rationale for why the model chose that option. This prompting method has proven effective on REV Chen et al. (2023)
X - RR:
For the SPORTU-text evaluation, LLMS have demonstrated their capability for in-context learning by utilizing exemplars through few-shot prompting (Brown) 2020). Therefore, we use four prompt- ing baselines for evaluation: zero-shot X + Y (OS), zero-shot X + RY (OCoT), five-shot X - Y (5S), and five-shot X + RY (5CoT). For the five-shot method, we provide five exemplars with only the answers. The five exemplars for the five-shot CoT method include both the answers and annotated rationales.
For the SPORTU-video evaluation, we use all three prompting methods. The zero-shot X + YR prompting method is applied to conduct human error analysis. More details of models' specific prompts are shown in the appendix L
4.3 EVALUATION METRICS
We use accuracy (prediction compared to ground truth) to evaluate the predictions of each model in multiple-choice tasks. We explore several methods to evaluate different aspects of model-generater explanations and open-ended questions: ROUGE-L (Lin) 2004). BERTScore (Zhang et al.] 2019). BLEURT (Sellam al. 2020), CTC-Preservation (Deng et al. (022), and G-Eval Score (Liu et al. 2023). ROUGE-L computes the surface-form similarity between model-generatee explanations and reference
explanations. BERTScore and BLEURT measure semantic similarity using pre-trained BERT (Devlin) 2018) and fine-tuned BERT models respectively. CTC metrics evaluate the information alignment of moodel-generate explanations. G-Eval is a framework that uses large language models with chain-of-thought reasoning and form-filling paradigm to assess
7

---

Published as a conference paper at ICLR 2025
the quality of NLG outputs. We apply this framework evaluate the accuracy, conciseness, and relevance of the model-genee model-generated explanations compared to the ground truth, reporting these as an overall score. Since LLMs might favor their answers, utilize all four models evaluated in the SPORTU-text part to implement the G-Eval process for SPORTU-tex results and three models (excluding I.Lam3.1 1- 1-405B) for the SPORTU-video part. We also calculate the average score of the models to obtain an overall score. The detailed prompt can be found in the Appendix [J] To ensure the reliability of the G-Eval scores, we conducted human evaluation on randomly selected set of 20 questions per sport across sports, totaling 140 questions. This subset was used to compare the scores across 14 models. By using the same set of questions
G-Eval scores with for all models, we ensured consistency in comparison, allowing to assess whether the G-Eval scores were line with human judgment. The human annotators rated the model- -enerrted content using the same criteria as G-Eval, providing a reference point for how well the automated evaluation aligns with human evaluation.
RESULTS
We compared the performance of different models across both SPORTU-text and SPORTU-video.
Overall results for SPORTU- text are shown Tad Ta- ble Z while the multiple-choice QA results for SPORTU-video presented in Table 3, and the open-ended QA results are displayed in Table 4 Among all models, GPT-4o performs the best in SPORTU-text, achieving the highest accuracy of 71% in the five-shot CoT prompt set- ting. along with a G-Eval score of 4.16 the zero-shot CoT prompt setting. The G-Eval score confirms that GPT-4o can somewhat grasp sports-related rules, strategies, and scenario-based basedd questions. However, there remains a notable gap compared expert performance, which exceeds 90%, as mentioned in Xia et al. (2024a).
Table 1: Performance of LLMs with Standard Prompt Settings
Setting
Model
Acc.(%)
Claude-3.5-Somne
64.33
gemini-1.5 Pro
63.33
X-Y((0))
GPT-4o
70.22
Llama3.1-405B
66.67
Claude-3.5-
68.44
gemini-1.5 Pro
64.11
X-Y((5))
GPT-4o
70.78
Llama3. 1-405B
66.33
Table 2: Performance of LLMS on SPORTU-text evaluation across CoT settings. Metrics include: Accuracy (Acc), ROUGE-L (R-L), BERTScore (B-S), BLEURT (BL), CTC Preservation (CTC), GPT-based G-Eval (G-E), Gemini-based Eval (GEM), Claude-based Eval (CL), Llama-based Eval (LL), and Average G-Eval score (AVG). GEM uses Gemini 1.5 pro, CL uses and LL uses Llama3. 1lmma.1-405BB for evaluation.
Setting
Model
Acc(%)
R-L
B-S
BL
CTC
G-E
GEM
CL
LL
AVG
Claude 3,5 Sonnet
3.5-oneet
64.67
0.26
0.65
0.57
0.43
3.78
3.25
3.28
4,07
3.60
X (-CY(
gemini-1.5 Pro GPT-44
62.67
0.28
0.62
0.53
0.43
3.79
3.57
3.39
3.98
3.68
68.78
0.27
0.66
0.57
0.43
4.16
3.42
3.43
4.37
3.85
Llama3.1 1-405B
64.44
0.25
0.64
0.55
0.43
3.89
3.19
2.74
3.90
3.39
Claude-3,5 3.5-nnnct
65.22
0.27
0.65
0.56
0.43
3.98
3.43
3.39
4.15
3.74
(5CoT)
gemini-1.5 Pro
61.22
0.30
0.62
0.53
0.43
3.73
3.51
3.49
3.38
3.53
GPT-4o
71.00
0.33
0.68
0.58
0.44
4.13
3.52
3.59
4.15
3.85
L.amma3.
65.22
0.67
0.57
3.33
4.02
3.61
For the SPORTU-video multiple-choice task. Qwen2-VL-72B achieved the highest overall accuracy at 70.94% on the X - Y setting, followed by Claude-3.5 1iiee-3,---onneee (70.18%). Models tend to perform well easy-level questions but show significant gap in hard-level questions, indicating lack of domain knowledge, particularly in rule comprehension. For example, GPT-4o leads on the hard- level tasks with only 57. 57.84%. Among the open-source models, LLAVA-NeXTT outperformed others.
By comparing three different prompting strategies, we observed that X - Y achieved the highest overall performance across most models, outperforming both X YR and X - RY. For most
8

---

Published as a conference paper at ICLR 2025
Table 3: Overall performance of MILMs SPORTU-video for multiple-choice questions. The best results are bolded. The results highlight that models perform best with the X 7 Y. followed by X + R. and X + RY.
Accuracy(%)
Model
Close source Model
Claude-3.0-aal
48.07
47.19
47.95
3.5
70.18
Gemini 1.5 Pro Gemini 1.5 Flash GPI-ommiii
63.13
63.04
64.93
$9.97
46.68
62.52
57.24
42,06
GPT-4o
68.00
65.56
68.79
Opea-source Model
ChatUniVi HaVA--NeT
42.35
32,58
41.89
68.89
62.16
63.72
mPLUG-OwI3
59,26
61.27
60.80
ST-LLM
41.59
40.09
46.39
Tarsier
60.99
Video ChurGPT
44.63
42.36
34.05
VideoChat12
61,55
62.79
61.53
Qwen2- VL-72B
70.94
X-YRR X RY. A detailed comparison models, the accuracy follows the order: x
of the prompting strategies across different difficulty levels is provided Appendix G
We also found that when models generated the rationale first (in X + RY), the final answer pre- diction was often influenced by incorrect or hallucinated reasoning processes. Examples illustrating these errors can be found in Appendix H This observation aligns with the findings of of et al. 2023). further emphasizing the challenges of reasoning based approaches in complex tasks.
For open-ended questions, Close source models, along with the Qwen2-VL-72B model, achieved higher G-Eval and human rating scores compared to the 7B open-source models. This indicates that close-source models exhibit stronger reasoning abilities than the 7B open-source models. GPT- 4o again led with a G-Eval score of 1.84, a result further validated by human annotators. Even with human evaluations, the score closely aligned with G-Eval's assessment, with the model not exceeding a score of 3 when evaluated using all three LLMs. A score of I indicates very poor performance, and 2 suggests poor performance based on the evaluation criteria. This highlights the model's struggle to connect observed actions with relevant domain knowledge, such as identifying technical errors specific rules. Overall, none of the MILMS achieved average score above 3, demonstrating gap in deep domain knowledge required for video sports understanding. We also noticed that among the evaluated metrics, G-Eval demonstrates the closest alignment with human ratings, with Pearson correlation coefficient of 0.41. However, as this and other metrics exhibit low correlations with human ratings, highlights the need for developing a domain- specific metric for evaluating sports content in the future. More can be found in Appendix
Additionally, we notice that the performance of models on multi-angle videos shows variability depending on the camera perspectives for the same scene, indicating that models struggle with con- sistent understanding across different camera angles. More details can be found in the appendix [
5.1
ERROR ANALYSIS
To gain deeper insights into the limitations of MLLMs, we applied the X-+Y prompting method, where models first generated an answer and then explained their reasoning. This provided valuable information about how models approached reasoning, especially in complex tasks like foul detec- tion. We analyzed errors across both open-ended and multiple-choice tasks, selecting incorrect examples per sport for each model, resulting in total of 3920 errors.
Figure shows the radar chart representing the distribution of different error types. The most fre- quent error Question Understanding Error, particularly in questions like "Why is it foul in the video?" where models incorrectly responded that there was no foul despite the question's 's clear presupposition. Another common issue was Hallucination Error, such when model mentioned a referce in its explanation, even though no referee was visible in the video. Detailed examples of these errors and further case studies provided in Appendix M

---

Published as a conference paper at ICLR 2025
Table 4: Model performance SPORTU-video open-ended tasks. Metrics include ROUGE-L (R-L), BERTScore (B-S), BLEURT (BL), CTC Preservation (CTC), GGPT-based G-Eval (G-E), Gemini-based Eval (GEM), Claude-based Eval (CL), Llama-based Eval (LL), and Average G-Eval score (AVG). GEM uses Gemini 1.5 pro, CL uses Claude-3.55 3.--Sonnet and LL uses Llama3. 1llmm3.1-4055 for evaluation and Human Rating (H-R*). denotes human ratings conducted to verify the reliability
of the G-Eval scores as mentioned in 4.3
Model
R-L
B-S
BL
CTC
G-E
GEM
CL
AVG
H-R*
Close-source Models
Claudc-3.00 3.0oo-3.--Hiikk
0.08
0.41
0.43
0.39
1.55
1.80
1.63
1.66
1.93
Blaudc-3.5-Somme
0.05
0.40
0.43
0.39
1.62
1.89
1.59
1.70
2.13
Gemini 1.5 Pro
0.08
0.38
0.36
0.38
1.11
1.16
1.20
1.16
1.19
Gemini 1.5 Flash
0.13
0.42
0.39
1.34
1.70
1.62
1.55
1.84
GPT-4omini
0.05
0.39
0.36
0.38
1.60
1.94
1.65
1.73
2.17
GPT-4o
0.07
0.41
0.43
0.39
1.84
1.17
1.75
1.59
2.51
Open-source Models
ChatUniVi
0.07
0.39
0.37
0.38
1.27
1.39
1.45
1.37
1.48
LLVV--NeX
0.17
0.47
0.38
0.40
1.47
1.63
1.75
1.61
1.62
mPLUG-OW13
0.15
0.44
0.37
0.39
1.38
1.60
1.75
1.58
1.46
Tarsier
0.12
0.45
0.36
0.40
1.36
0.70
1.78
1.28
1.63
Indeo-ChatGPT
0.08
0.39
0.35
0.38
1.08
1.11
1.36
1.19
1.22
VideoChat2
0.23
0.49
0.35
0.40
1.43
1.73
1.79
1.65
1.48
ST-LLM
0.13
0.38
0.20
0.41
1.30
1.50
1.52
1.44
1.22
0.10
0.42
0.39
0.41
1.62
1.94
1.72
1.76
2.08
Dusdoon Lederardd
CPTY
Gentnn
Cla
Helucradom
Figure 3: Error type distribution across different MILMs on SPORTU-video tasks. The analysis reveals that Question Understanding Error is the most common issue, followed by Hallucination Error. Each error type highlights specific model limitations comprehending the task.
CONCLUSION
In this paper, we introduced SPORTU, benchmark designed to evaluate the sports understanding capabilities of Multimodal Large Language Models (MLLMs). SPORTU comprises two compo- nents: SPORTU-text, which assesses models' comprehension of rules, strategies, and scenarios through multiple-choice questions, and SPORTU-video. which evaluates their ability apply this knowledge to real-world sports footage, including tasks like recognition, foul detection, and rule application. By integrating both text and video tasks, SPORTU provides a holistic assessment of reasoning abilities across different levels of complexity. Our results reveal that while models like GPT-4o show progress in text-based reasoning, they struggle with scenario-based reasoning and con- necting visual actions with domain-specific rules. Error analysis highlights issues such question misunderstanding and hallucination, emphasizing
need for improved reasoning capabilities fu- ture models. We also discuss the broader impacts of our findings in Appendix B We hope SPORTU will inspire advancements in MLLMs and contribute to robust weal-woldd sports understanding.
REFERENCES
Meta. Llama model card, 2024. URL hestee//ggiuuu..com///tttaaaaaae /
slob/ma n/MOD CARD md
10

---

Published as a conference paper at ICLR 2025
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry. Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa De- hghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxi- aoyu Feng. Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Itycheriah,, Matthew Jagielski, Wenhao Jia, Kathleen Ke- nealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hycontack Lim, Hanzhao Lin, Zhongtao Liu, Freder- ick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Mous- salem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Mar- tin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy. Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli. Xuezhi Wang. Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report,
Gaurav Mishra, Erica Moreira, Mark
2023. URLhttps: hettp::/arxivv.rg//aa
10403
Anthropic. Claude 3.5 sonnet model card addendum, 2024a URL hwtps::/ttwtwttt/ anthropic. cww wecnm/ffeeclllccc wtt.ffeccc1111111111111111111110 23011338888111111222233555555550
Card_ Claude_ Addenddum, pdf
Anthropic. Introducing the next generation of claude, 2024b. URL.ttpp:://ww..atbbopr.
com//ews/claudee- 3
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL http:::/tpwpccoott
anthropic com/ htaprpppic..co///bbabb.bbbbbb
.ddf
Luis Baca, Nátali Ardiles, Jose Cruz, Wilson Mamani, and John Capcha. Deep learning model based on a transformers network for sentiment analysis using nlp in sports worldwide. In Mayank Singh, Vipin Tyagi. P.K. Gupta, Jan Flusser, and Tuncer Ören (eds.). Advances Computing and Data Sciences, pp. 328-339. Springer Nature Switzerland, 2023, ISBN 978-3-031-37940-6
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Ryan Beal, Stuart E Middleton, Timothy Norman, and Sarvapali Ramchurn. Combining ma- chine learning and human experts to predict match outcomes in football: A baseline model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 15447-15451, 2021.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL.
net https::/cpenrev// .wt.ee.e.eeeeeeeeeeeeeeeeeeeeeee / forum (atpmmmmd==uyTLLBBB 1o
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, and Swabha Swayamdipta. REV; Information-theooreie evaluation of free-text rationales. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2007-2030, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18.18813/v 1/2023,c22l/l/ Icn. 112. URL https: htpps///cci anthho wtw.hhelooy..org2222333 llng. 112
11

---

Published as a conference paper at ICLR 2025
Zhutian Chen, Qisen Yang, Xiao Xie, Johanna Beyer, Haijun Xia, Yingcai Wu, and Hanspeter Pfis- ter. Sporthesia: Augmenting sports videos using natural language. IEEE transactions on visual- ization and computer graphics, 29(1): (111))991-99985 928,,,22222222 2022.
Christopher Clark, Kenton Lee, Ming-Wo Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprssing difficulty of natural yes/no questions. arXiv preprint UuXivv:105. 10044, 2019.
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. Compression, trans- duction, and creation: A unified framework for evaluating natural language generation. arXiv preprint arXiv:2 arXiv:2109.06377 2021.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rx.v:18100048000 04000, 2018.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal lims in video analysis. arXiv preprint arXiv:2405.21075. 2024.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text, 2024a.
Gemini Team, Gemini flash, 2024b. URL ni Deminni/fles flash//
Xuchai He, Weixi Feng. Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. Mmworld: Towards distiiiisei multi-faceted world model evaluation in videos. arXiv preprint 240x::2406608407,, 2024a.
Yuchen He, Zeqing Yuan, Yihong Wu, Liqi Cheng, Dazhen Deng, and Yingcai Wu. Vistec: Video modeling for sports technique recognition and tactical analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14900-498,, 2024b.
Jan Held, Anthony Cioppa, Silvio Giancola, Abdullah Hamdi, Bernard Ghanem, and Mare Van Droogenbroeck. Vars: Video assistant referee system for automated soccer decision mak- ing from multiple views. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5086-5097, 2023.
Jan Held, Anthony Cioppa, Silvio Giancola, Abdullah Hamdi, Christel Devue, Bernard Ghanem, and Marc Van Droogenbroeck. Towards ai-powered video assistant referee system for association football. arXiv preprint arXhv:2407.12483 2024a.
Jan Held, Hani Itani, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogen- broeck. X-vars: Introducing explainability in football refereeing with multi-modal large language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 3267-3279, 2024b.
Yebowen Hu, Kaiqiang Song. Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, and Fei Liu. Sportsmetrics: Blending text and numerical data to understand information fusion in llms. arXiv preprint 4rX9v:2402.10979, 2024a.
Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Wenlin Yao, Hassan Foroosh, Dong Yu, and Fei Liu. When reasoning meets information aggregation: A case study with sports narra- tives. arXiv preprint arXiv:2406. 2084, 2024b.
Kuan-Hao Huang, Chen Li, and Kai-We Wei Chang. Generating sports news from live commentary: A chinese dataset for sports game summarization. In Proceedings of the 1st Conference of the Asia- Sscaf-aciffic Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 609-615, 2020a.
12

---

Published as a conference paper at ICLR 2025
Kuan-Hao Huang. Chen Li, and Kai-Wei Chang. Generating sports news from live commentary: A chinese dataset for sports game summarization. Proceedings of the 1st Conference of the Asia-Pacifee Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 609-615, 2020b.
Pedro Calciolari Jardim, Leonardo Mauro Pereira Moraes, and Cristina Dutra Aguiar. Qasports: A
question answering dataset about sports. In Anais do V Dataset Showcase Workshop, pp. 1-12. SBC, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified vi- sual representation empowers large language models with image and video understanding, arXiv preprint arXiv:2311.08046 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint 1oXXvv:705.03300 2017.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances neural information processing systems, 35:22199 222219922221,. 2022.
Haopeng Li, Andong Deng, Qiuhong Ke, Jun Liu, Hossein Rahmani, Yulan Guo, Bernt Schiele, and Chen Chen. Sports-qa: large-scale video question answering benchmark for complex and professional sports. arXiv preprint S.XXi::24010050000 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2024b.
Qi Li, Tzu- Then Chiu, Hsiang-Was Wei Huang, Min-Te Sun, and Wei-Shinn Ku, Videobadminton: A video dataset for badminton action recognition. arXiv preprint arXiv:2403. 2024c.
Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. Multisports: A multi-person video dataset of spatio-temporally localized sports actions. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13536-13545. 2021.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74- 14-81,20 2004.
Haotian Liu, Chunyuan Li. Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
Llavavvi.qitan
Qianying Liu, Sicong Jiang, Yizhong Wang, and Sujian Li. Liveqa: A question answering dataset over sports live. Chinese Computational Linguistics: 19th China National Conference, CCL 2020, Hainan, China, October 30-November 2020, Proceedings 19, pp. 316-328. Springer, 2020.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge. Ying Shan, and Ge Li. St-llm: Large language models effective temporal learners. In European Conference on Computer Vision pp. 1-18. Springer, 2025.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16634, 2023.
13

---

Published as a conference paper at ICLR 2025
Adela Ljajić, Ertan Ljajić, Petar Spalevié, Branko Arsić, and Darko Vučkovié. Sentiment analysis of textual comments in field of sport. In 24nd International Electrotechnical and Computer Science Conference (ERK 2015), IEEE, Slovenia, 2015.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
Adrien Maglo, Astrid Orcesi, and Quoc-Cuong Pham. Efficient tracking of team sport players with few game-specific annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, pp- 3161-3447777 2022.
OpenAI. Hello gpt-4o, 2024a. URL http:///ss///ppaaaccooo/////a/aa
OpenAL. Hello gpt-4o, 2024b. URL httppp///bbee....ccoo//////lllla
OpenAl. Gpt-4 technical report, 2024c. UBL:ttts.//arx...r//bb///22
Nadav Oved, Amir Feder, and Roi Reichart. Predicting in-game actions from interviews of nba players. Computational Linguisties, 46(3):667-712, 2020.
Noah Sarfati, Ido Yerushalmy, Michael Chertok, and Yosi Keller. Generating factually consistent
sport highlights narrations. In Proceedings of the 6th International Workshop Multimedia Content Analysis in Sports, pp. 15-22, 2023.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text gener- ation. arXiv preprint arXiv:2004.04665 2020.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine- grained action understanding. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2616-2625, 2020.
Craig Thomson, Ehud Reiter, and Somayajulu Sripada. Sportsett: basketball-a robust and main- tainable data-set for natural language generation. In Proceedings of the Workshop on Intelligent Information Processing and Natural Language Generation, pp. 32 40, 2020.
Rhys Tracy, Haotian Xia, Alex Rasla, Yuan-Fang Wang, and Ambuj Singh. Graph encoding and neural network approaches for volleyball analytics: From game outcome to individual play pre- dictions. arXiv preprint 230. /142, 2023.
Renaud Vandeghen, Anthony Cioppa, and Marc Van Droogenbreeck. Semi-supervised training to improve player and ball detection in soccer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3481-3490, 2022.
Staša Vujičié Stanković and Miljana Mladenović. An approach to automatic classification of hate speech in sports domain on social media. Journal of Big Data, 10(1):109, 2023.
Jiaan Wang, Zhixu Li, Tingyi Zhang, Duo Zheng, Jianfeng Qu, An Liu, Lei Zhao, and Zhigang Chen. Knowledge enhanced sports game summarization. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1045 1053, 2022a.
Jiaan Wang, Zhixu Li, Tingyi Zhang, Duo Zheng, Jianfeng Qu, An Liu, Lei Zhao, and Zhigang Chen. Knowledge enhanced sports game summarization. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp.
2022b.
Jiawei Wang, Liping Yuan, and Yuchen Zhang. Tarsier: Recipes for training and evaluating large video description models, 2024. URL https://arxiv: org/ abs 2407700077404
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in 2022.
neural information processing systems,
Haotian Xia, Rhys Tracy, Yun Zhao, Erwan Fraisse, YuannFFan Wang, and Linda Petzold. Vren: Volleyball rally dataset with expression notation language. In 2022 IEEE International Confer- ence on Knowledge Graph (ICKG), pp. 337-346. IEEF, 2022.
14

---

Published as a conference paper at ICLR 2025
Haotian Xia, Rhys Tracy, Yun Zhao, Yuqing Wang, Yuan-Fang Wang, and Weining Shen. Advanced volleyball stats for all levels: Automatic setting tactic detection and classification with a single camera. In 2023 IEEE International Conference Data Mining Workshops (ICDMW), pp. 1407- 1416, 2023.
doi: 10.1 DL11099/CDDWW0887.22222223
Haotian Xia, Zhengbang Yang, Yuqing Wang, Rhys Tracy, Yun Zhao, Dongdong Huang, Zezhi Chen, Yan Zhu, Yuan-fang Wang, and Weining Shen. Sportqa: A benchmark for sports under- standing in large language models. arXiv preprint arXiv:2402. 15862, 2024a.
Haotian Xia, Zhengbang Yang, Yun Zhao, Yuqing Wang, Jingxi Li, Rhys Tracy, Zhuangdi Zhu, Yuan-fang Wang, Hanjie Chen, and Weining Shen. Language and multimodal models in sports: A survey of datasets and applications. arXiv preprint arXiv:2406. /2252, 2024b.
Jinglin Xu, Sibo Yin, Guohao Zhao, Zishuo Wang, and Yuxin Peng. Fineparser: fine- ree-grained spatio-temporal action parser for human-centric action quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14628-14637, 2024a.
Jinglin Xu, Guohao Zhao, Sibo Yin, Wenhao Zhou, and Yuxin Peng. Finesports: A multi-person hierarchical sports video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 21773-21782, 2024b.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint (001::v8000096000 2018.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models, 2024.
04840
Hangjie Yuan, Dong Ni, and Mang Wang. Spatio-temporal dynamic inference network for group activity recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7476-7485, 2021.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.088 02813,, 2024b.
Sania Zahan, Ghulam Mubashar Hassan, and Ajmal Mian. Learning sparse temporal video mapping for action quality assessment in floor gymnastics. IEEE Transactions on Instrumentation and Measurement, 2024.
Boyi Zeng, Jun Zhao, and Shantian Wen. A textual and visual features-jointly driven hybrid intelli- gent system for digital physical education teaching quality evaluation. Mathematical Biosciences and Engineering. 13581:1351 2023.
Jiawen Zhang, Dongliang Han, Shuai Han, Heng Li, Wing-Ko Kai Lam, and Mingyu Zhang. Chat- match: Exploring the potential of hybrid vasion-language deep learning approach for the intelli- gent analysis and inference of racket sports. Computer Speech & Language, 89:101694, 2025.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint 1eXiv:/904.0990 2019.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal thain-of-thought reasoning in language models. arXiv preprint arXiv:2302.0092.. 2023.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04266 2024.
15

---

Published as a conference paper at ICLR 2025
Kevin Zhu, Alexander Wong, and John McPhee. Fencenet: Fine-graine grained footwork recognition in fencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 2589-35999 2022a.
Kevin Zhu, Alexander Wong, and John McPhee. Fencenet: Fine-grained footwork recognition in fencing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- fion, pp. 3589-3598, 2022b.
CONTENTS
Introduction
2
Related Work
2.1
Multimodal Sports Analysis
2.2
Multimodal Sports QA
3
Sports Understanding Benchmark
3.1
Quality Control
3.2
SPORTU-text: Pure Text QA
3.3
SPORTU-video: Multimodal Video QA
Experiment
Models
4.2
Prompting Strategies
Evaluation Metries
5
Results
5.1
Error Analysis
6
Conclusion
10
More Related Works
19
Broader Impacts
19
Discussion
20
Multi-Angle Videos Result
D.I Model Performancel
21
D.2 Example of Multi-Angle Videos
22
Zero Shot Based Prompt Setting Results Across rules-, strategy-, and scenario-related Question on SPORT-text
23
Zero Shot Result
23
Zero Shot With CoT Result
23
16

---

Published as a conference paper at ICLR 2025
F Five Shot Based Prompt Setting Results Across rules-, strategy,, and scenario-related Question on SPORT-text
24
FT Five Shot Result
24
E2 Five Shot With CoT Result
24
Result of Three Prompt Settings on SPORTU-video across different levels of difficulty
25
Examples of Errors Across Prompt Strategies
26
Correlation Matrix of Evaluation Metries
29
Criteira of G-Eval
30
K SPORTU-text Prompt Templates
31
K. zero shot standard prompt
31
zero shot CoT prompt
K.3 five shot standard prompt
32
Five shot Co'I' prompt
32
SPORTU-video Prompt Templates
33
X YR Prompt Template
33
X RY Prompt Template
33
Prompt Template
34
Open-ended Template
34
M Additional Error Analysis
35
Question Template for Each Sport
36
SPORTU-ttet Examples
38
0.1
Rule-related Question
38
0.2
Strategy-relate Question
39
0.3
Scenario-related Question
40
SROUTU-video Examples
41
Basketball
41
Volleyball
42
P3 Soccer
44
Badminton
45
American Football
47
P.6
Ice Hockey
48
P.7
Baseball
50
17

---

Published as a conference paper at ICLR 2025
Examples of Each Error Type
52
Q.1
Question Understanding Error
52
Q.2
Visual Perception Error
53
Q.3
Hallucination Error
55
Q.4
Reasoning Error
56
Lack of Domain Knowledge
58
18

---

Published as a conference paper at ICLR 2025
A MORE RELATED WORKS
In this section, we provide a detailed comparison table of various QA datasets across multiple dimensions. This includes QA Type, whether explanations are provided, the presence of multilevel difficulty, and the coverage of domain knowledge in terms of both action and rule. Additional features compared include whether datasets incorporate slow motion, multi-camera angles, and the number of sports covered. Average explanation or answer length refers to the average word count for open-epeeeee ended explanations or answers.
Table 5: Comparison of various QA datasets across multiple dimensions. MC means Multiple Choice. OF means open-ended. Avg Exp. and Ans. Length refers to the average word count for open-ended explanations and answers.
Deachmark
QA byp
Qsediee Type Explazation Nattiimd INft
Dveaats Kavelidgy Skew Matkon Mutil Cameraa Angle No. of Spara Ang Exp, Longth
Acion
Ral
BiGkbeec head ware DnGS QASper
Conees-rree
30C GE MC
Casiat cumatre
SunDA Roree
Cooreciitee
Span-- of
Video
GE
2004
Miles
0F
B BROADER IMPACTS
Sports understanding is a critical domain for MLLMs to develop as they are increasingly applied to real-world tasks, such as supporting sports education. MLLMs with advanced reasoning capabilities can empower non-experts to quickly grasp the rules and dynamics of sports, enhancing their ability to enjoy and understand games. They also hold promise for supporting advanced applications like sports strategy analysis and real-time decision-making, which brings new chapters for the sports community.
The SPORTU benchmark addresses key gaps in existing datasets by systematically evaluating MLLMs' sports understanding across three difficulty levels: easy, medium, and hard. This ticred structure reveals how models perform well commonsense-based questions but face challenges as tasks demand deeper sports knowledge and reasoning. These findings highlight the current limita- tions of MLLMs in sports reasoning and underscore the need for further advancements address these gaps. As the timing when we write the paper, the MLLMs are still struggling with combin- ing actions with corresponding rules and explaining the rationale, so believe that SPORTU can meaningfully guide the development and benchmarking of future MLLMs in the sports domain.
19

---

Published as a conference paper at ICLR 2025
C DISCUSSION
At the time writing this paper, current SOTA MLLMs still perform poorly on challenging tasks that require combining sports knowledge with corresponding actions, particularly in connecting these actions to various rules. Through different prompting strategies, we found that the models performed even worse when required to provide a reasoning process before predicting the final answer. However, the sports domain necessitates that MLLMs have robust and reliable reasoning process, explaining the concept behind the questions, where people can be inspired and learn from the context, is often more important than simply providing the final result.
Additionally, we observed that better frame extraction strategies need be developed specifically for sports tasks to ensure that the actions critical for reasoning about the question are clearly provided to the model. A more effective grounding method could also enhance the model's ability to distin- guish actions more accurately. During the experiments, we noticed that while models sometimes captured and described most the correct movements, they often failed to infer the corresponding rule violations. This highlights significant gap in connecting recognized actions to specific rules, underscoring the need for models to not only identify movements but also to understand what a rule violation should look like.
We hope that our benchmark can positively contribute to the community, serving as foundational step bring more attention the sports domain and inspire the development of advanced models that can help people engage more deeply with sports.
20

---

Published as a conference paper at ICLR 2025
D MULTI-ANGLE VIDEOS ResULT
D.1 MODEL. PERFORMANCE
Table 6: Accuracy comparison across multiple camera angles of the question
Model (%)
P(All Correct)
P(All False)
P(At least One Correct At least One False)
Haude-3.0-Haikke
32.17
32.11
35.72
Claude-3.5-Somen
52.86
18.59
28.55
Gemini 1.5 Pro
43.24
25.41
31.35
Gemini 1.5 Flash
48.48
22.96
28.55
GPT-4ominii
36.89
28.40
34.72
GPT-4o
47,42
20.84
31.73
ChatUniVi
29.31
38.52
32.17
LLaVA-NeXT
53.67
22.26
24.07
mPLUG-OWB
42.77
27.91
29.31
ST-LLM
22.72
58.39
24.88
Tarsier
48.25
26.05
25.70
Video-ChalGPT
25.06
35.84
39.10
VideoChat2
47.84
21.27
30.89
Qwen2-VL. VL-72B
51.78
20.11
28.11 18.11
In this section, we analyze the performance of models when answering the same question from different camera angles within the same scene. The objective is to evaluate how consistently mod- els understand the same scenario when presented from multiple perspectives. For example, the question is "What color is the person's jersey?", the same question is asked across different camera angles for the same scene. The results are categorized into three distinct cases:
P(All Correct): This indicates the percentage of instances where the model answered cor- rectly for all camera angles of the same scene. A higher percentage reflects the model's ability to consistently interpret and understand the scene across multiple perspectives.
P(All False): This shows the percentage of instances where the model answered incorrectly for all angles of the same scene, highlighting consistent misunderstanding across different perspectives.
P(At least One Correct& At least One False): This category reflects situations where the model answered correctly for at least one camera angle but incorrectly for at least one other, indicating that the model's understanding varies depending on the perspective.
The analysis reveals that camera angle variation significantly affect the model's performance. While some models are more consistent across angles, others show noticeable drop in accuracy when the angle changes, suggesting challenges in maintaining robust understanding across multiple viewpoints. This highlights critical area for further research and improvement in multimodal models when handling multi-angle video inputs. We will present example in the next section where the model answers the question incorrectly when viewed from one camera angle but answers it correctly from another.
21

---

Published as a conference paper at ICLR 2025
D.2 EXAMPLE OF MULTI-ANGLA VIDEOS
In this section, we provide an example that shows that under the same question, the model got different answers from two different camera angles.
RURES
ep
Question: Based on this video, which the following descriptions apply to the situstion that occuree??
A: Other B: Handball Holding
D: Simulation
Ground Truth:
a player their am to block or control the ball inside the penalty area. This action is consistent with handball offense, which occurs player deliberately handles ball their
Model Answer:
answer Explanation:
striped jersey using
Figure 4: Under camera angle one, the model answers the question correctly.
Question: Based this video, which the following descriptions the situatin that occuree?
A: Other B: Handball Holding D: Simulation
Ground Truth: B
Model Answer:
The correct answer is: Explanation: The sequence of frames shows player in a yellow jersey attempting and white jersey jumping for the ball. There clear evidence handball,
the ball, with player halding, or simulation. The players are engaged typical merial challenge, and the player in red and white appears to fall naturally after the challenge. Therefore, the situation does not the specific descriptions of handball, holding, simulation, making \"Othen" the most aporopriate option.
Figure 5: Under camera angle two, the model answers the question wrong.
22

---

Published as a conference paper at ICLR 2025
E ZERO SHOT BASED PROMPT SETTING RESULTS ACROSS RULES-, STRATEGY-, AND SCENARIO-RELATED QUESTION ON SEXT
E.1 ZERO SHOT RESULT
Table 7: Zero Shot Performance of LLMs across different question types.
Model( %)
Acc. (overall)
Acc. (rule)
Acc. (scenario)
Acc. (strategy)
Claude-3.5-Sonnet
64.33
58.27
67.42
70.62
GPT-4o
70.22
68.56
69.69
74.58
1-05BB
66.67
62.33
69.97
68.93
Gemini-1.3 1.5 Pro
63.33
60.43
64.41
67.05
E.2
ZERO SHOT WITH CoT RESULT
Table 8: Zero Shot CoT Performance of LLMs across different Question Types.
Model
Aec.(%)
ROUGE-L
BERTScore
BLEURT CTC Presv,
G-Eval
Overall
Claude-3.5-Somne
64.67
0.26
0.65
0.57
0.43
3.78
Gemini-1.5
62.67
0.28
0.62
0.53
0.43
3.79
GPT--4o
68.78
0.27
0.66
0.57
0.43
4.16
Llama3.1-405B
64.44
0.25
0.64
0.55
0.43
3.89
Rule
Claude-3.5-Sone
64.67
0.26
0.65
0.57
0.43
3.79
Gemini-1.5
62.67
0.28
0.62
0.53
0.42
3.80
GPT-4o
68.78
0.27
0.66
0.57
0.43
4.16
1amma3.1-4057.
64.44
0.25
0.64
0.55
0.43
3.89
Strategy
Claude-3.5-Soma
67.43
0.25
0.65
0.57
0.43
3.89
Gemini-1.5
62.29
0.26
0.61
0.52
0.42
3.98
GPT-4o
73.14
0.27
0.66
0.56
0.43
4.23
Llama3.1-4057
65.71
0.24
0.63
0.54
0.43
3.98
Scenario
Slaude-3.5-Somee
66.20
0.26
0.65
0.57
0.44
3.86
Gemini-1.5
61.13
0.27
0.62
0.53
0.43
3.71
GPT-4o
68.73
0.27
0.66
0.56
0.43
4.10
1lama3.1-411 1-405B
66.57
0.25
0.64
0.55
0.43
3.91
23

---

Published as a conference paper at ICLR 2025
F FIVE SHOT BASED PROMPT SETTING RESULTS ACROSS RULES-, STRATEGY-, AND SCENARIO-RELATED QUESTION ON SPORT-TEXX
F.I
FIVE SHOT RESULT
Table Five Shot Performance LLMs across different question types.
Model (%)
Acc. (overall)
Acc. (rule)
Acc. (scenario)
Acc. (strategy)
Claude-3.5-Sonnet
68.44
65.04
70.82
70.62
GPT-4o
70.78
70.46
70.54
71.75
Llama3. 1lam5
66.33
62.6
68.84
68.93
Gemini-1.3 1.5 Pro
64.11
63.14
61.76
70.62
F.2
FIVE SHoT WITH CoT RESULT
Table 10: Five Shot CoT Performance of LLMs across different question types.
Model
Aec.(%)
ROUGE-L
BERTScore
BLEURT
CTC Presv,
G-Eval
Overall
Claudc-3.5-Somee
65.22
0.27
0.65
0.56
0.43
3.98
Gemini-1.5
61.22
0.30
0.62
0.53
0.43
3.73
GPT--0o
71.00
0.33
0.68
0.58
0.44
4.13
Llama3.1-405B
65.22
0.32
0.67
0.57
0.44
3.81
Rule
Claude-3.5-Sooe
64.77
0.27
0.65
0.56
0.43
3.95
Gemini-1.5
59.89
0.31
0.63
0.54
0.43
3.63
GPT-4o
70.46
0.34
0.69
0.58
0.44
4.09
10mma3.1-4057.
62.60
0.33
0.67
0.58
0.44
3.73
Strategy
Claude-3.5-Soma
66.86
0.26
0.65
0.56
0.43
4.15
Gemini-1.5
63.84
0.29
0.62
0.53
0.42
3.99
GPT-4o
71.43
0.33
0.68
0.58
0.44
4.23
1l-0aB.1-405B
66.29
0.32
0.67
0.57
0.44
3.89
Scenario
Slaude-3.5-Somee
64.79
0.27
0.65
0.57
0.43
3.95
Gemini-1.5
61.19
0.30
0.62
0.53
0.43
3.75
GPT-4o
71.27
0.32
0.68
0.58
0.44
4.12
1.ama3.1-411 1-405B
67.61
0.32
0.67
0.57
0.44
3.87
24

---

Published as a conference paper at ICLR 2025
G RESULT OF THREE PROMPT SETTINGS ON SPORTU-VIDEO ACROSS DIFFERENT LEVELS OF DIFFICULTY
Table 11: Overall performance of MLLMs on SPORTU-video for multiple-choice questions across three difficulty levels. The best results are bolded. The results highlight that models perform best with the X Y prompt (25/56 leading performances), followed by X - (21/56), and X YR (10/56).
Performance X-RY
Model
Difficulty
X-YR
X-Y
Easy
68.41
66.58
66.62
Medium
46.43
46.11
46.53
Claude-3.0-Haile
Overall
48.07
47.19
47.95
Easy
88.65
63.83
89.15
Medium
65.10
55.52
65.88
Claude-3.5-Sone
Hard
52.57
39.32
53.06
Overall
69.52
55.08
70.18
Fasy
85.85
85,
87.52
Medium
58.25
58.11
61.22
Gemini 1.5 Pro
Hard
43.53
42.75
39.98
Overall
65.13
63.04
64,93
Easy
85.99
$9.07
85.19
Gemini 1,5 Flash
Medium
53.38
50.85
Hard
38.73
12.89
38.26
Overall
59,97
46.68
62.52
Easy
66.09
59.02
66.68
Medium
55.69
42.25
58.12
GPT-4omini
Hurd
47.54
13.67
44.49
Overall
57.24
42.06
58.19
Easy
79.92
84.30
Medium
62.31
65.98
64.83
GPT-4o
Hard
57.84
40.51
56,20
Overall
65.56
68.79
Easy
94.86
84.16
95.11
Owen2-VL-72B
Medium
66.27
61.69
66.97
Hard
36.53
30.56
44.12
Overall
62.65
70.94
Easy
59,22
49.04
$3,99
ChatUniVi
Medium
36.95
28.55
35.63
Hurd
32.21
18:71
39.07
Overall
42.35
32.58
Easy
94.24
91.43
92.44
Medium
67.02
56.00
59.39
LLAVA-NeXT
Hard
33.44
34.21
30.78
Overall
68.89
62.16
63.72
Easy
87.28
88,51
87.11
Medium
55.58
58.75
57.37
mPLUUG-OW13
Overall
59.26
61.27
60.80
Easy
59.26
68.54
63.09
Medium
36.08
33.25
41.14
ST-LLM
Hard
30.51
22.10
36.08
Overall
41.59
40.09
46.39
Fasy
89.71
84.01
88.13 13
Medium
58.43
50.94
58.35
Tarsier
Hard
24.25
24.80
25.17
Overall
61.32
55.70
60.99
Easy
62.36
63.91
37.04
Medium
36.01
Video-ChuGPT
Hard
32.55
19.12
22.82
Overall
44.63
42.36
34.05
Easy
88.30
88.62
89.37
Medium
58.73
60.45
58.65
VideoChur2
Hurd
26.94
28.13
25.29
Overall
61.55
62.79
61.53
25

---

Published as a conference paper at ICLR 2025
H EXAMPLES OF ERRORS ACROSS PROMPT STRATEGIES
In this section, we show examples where the question was answered correctly in the X - Y prompt setting but failed in either the X RY, X + R, or both settings.
Question: What main color jersey does the player who committed the foul wear? white&red B: green C: white D: black&yellow
Ground Truth:
Y C
X - RY: To determine the main color jersey of the player who committed the foul, let's analyze the video frames step by step: 1. The video shows basketball game progress. In2. There are two teams the court: wearing white jerseys and the other wearing green jerseys.In3. In the later frames, we can see a player in a green jersey making contact with player in a white jersey near the center the court.Ind In4. The referee, visible black, is seen raising his arm, which typically indicates foul cn5.. The player the green jersey is the one who appears to be committing the action that leads to the foul call. Based on these observations, we can condlude that the player who committed the foul is wearing green jersey. Therefore, the correct answer is: The correct answer is:
X YR: The correct answer is: chose this option because in the video frames, clearly see that the player who commits the foul is wearing a green jersey. The sequence of images shows player in a green uniform making contact with a player a white uniform, which appears be the foul question. The other players the wearing green jerseys, confirming that this is one of the team colors. None of the other color options (white&red, white, or black&yellow) match the jersey color of the player committing the foul."
team also
Figure 6: Basketball Easy level Question
26

---

Published as a conference paper at ICLR 2025
Question: What main color jersey is the side team wearing?
Green B: Purple C: Red D: White
Ground Truth: D
X + Y:
+ To determine the main color jersey of far team, analyze video frames step step: Identify the players: There two players visible most frames. 2.
Determine which player is on the far side: The player wearing darker jersey is consistently positioned farther from the camera, making them the \"far sidel" player. 3. Observe the jersey color of the far side player: The player on the far side is wearing dark-colored jersey. Analyze the color options provided: The options Green, Purple, Red, and White. 5. Compare the observed color to the options: The dark jersey appears to be black or a very dark shade. Among the given options, the closest match would be option B: Purple, as it's the darkest color provided. Confirm consistency: Throughout all frames, the far side player's jersey remains consistently dark. 7. Consider lighting and video quality: While the jersey appears black, the lighting and video quality might affect color perception. Purple is the most plausible option among those given. Based on this analysis, the main color jersey of the far side team most likely to be Purple, it's the darkest color option provided and the closest match to the observed jersey color in the video frames. The correct answer is: B
X - YR: The correct answer is: D The far side team is wearing a white jersey. In the video frames, we can clearly see one player wearing a black jersey in the foreground, who appears from China based the on their jersey. The opponent, who on the far side the court, wearing predominantly white jersey with some pink red accents. This white and pink/red jersey is consistent across all the frames where the player is visible.
Figure 7: The same question is answered correctly the X - setting but incorrectly in both the X RY and X - YRR settings.
27

---

Published as a conference paper at ICLR 2025
Question: What kind of fault foul does this video show?
A: cross the center line B: double hit C: over reach D: ilegal attack Ground Truth: B
Y: B
- RY: Based the video frames provided, can analyze the situation step by step:1. The images show volleyball match at the CEV EuroVolley 2019 tournament.2. We players from two teams on either side of the net.3. In the later frames, we see a player reaching over the net to contact the ball.4. This action is known as reaching over the net or over-reaching in volleyball.5. Reaching over the net to interfere with the opponent's play is generally not allowed in volleyball, except under specific circumstances. 6. This action doesn't appear to be a center line violation, a double hit, or an illegal attack based on what we can see the Images.Givel these observations, the most likely fault or foul shown in this video is over reach or reaching over net. correct answer C"
The correct answer is: chose option C, over reach, because video frames show volleyball player reaching over the hit the ball. In volleyball, reaching over the net to play the ball the opponent's side is fault known as over reaching or reaching beyond the net. in the sequence of images, above and beyond the top of the net as they attempt to play the ball. This action is not allowed in volleyball as players must respect the opponent's playing space. The ball must be played on your own side of the net, except in specific situations like blocking. The other options don't fit what we see in the images:A) Cross the center line isn't visible in these frames.B) Double hit isn't apparent from a single contact shown.D) Illegal attack is too vague and doesn't specifically describe the over-reaching we can see. Therefore, based on the visual evidence in the video frames, the fault being committed is most likely an over reach, making option C the correct answer.
YR:
player's hand and extending
Figure 8: The same question is answered correctly in both the X + Y setting and X + YR settings but incorrectly in the X - RY setting.
28

---

Published as a conference paper at ICLR 2025
I CORRELATION MATRIX OF EVALUATION METRICS
We applied Pearson correlation to assess the relationship between the automatic evaluation metrics and human-annotated scores for the SPORTU-Video part. As shown in Figure D G-eval has the highest correlation with the human scores among all the automatic metrics. However, the correlation still weak, indicating the need for the development new evaluation metric in the future to improve the assessment process.
C-Eval CTC Preserv BLEURT BERTScore ROUGE- Human Rate
-10
1.00
-0.04
0.00
0.24
0.08
0.07
0.18
0.42
-0.04
1.00
0.77
0.12
0.72
0.12
0.18
-0.01
0.8
0.00
0.77
1.00
0.34
0.90
0.15
0.10
0.00
-.6
0.24
0.34
0.10
0.12
0.08
0.72
0.90
0.43
1.00
0.13
0.14
0.11
0.4
0.07
0.12
0.15
0.10
0.13
1.00
0.38
0.14
-02
GE-Eval
0.18
0.18
0.10
0.12
0.14
0.38
1.00
0.09
G-Eval
0.42
-0.01
0.00
0.22
0.11
0.14
0.09
1.00
0.0
Human Rate ROUGE-L
BERTScore
BLEURT CTC Preserv.
CI-Eval
GE-Eval
G-Eval
Figure 9; Examples of SPORTU-video
29

---

Published as a conference paper at ICLR 2025
J CRITEIRA OF G-EVAL
CRITERIA OF G-EVAL
This score assesses the everal accurscy, condiseness, and relevance of the modelggenertee exple- deanifying clearly
ground explention, explanatinn
Booring Breakdown:
-Boore (very poot):
explasation incornett
key might
inroevinnt details that
-Buore (poor)
The explanation containa same cornet elamants cleatty identity the bey
SSoore [adequrieg:
The explanation isentifisa the bey base but includea too much unrecessary information
clarfty, The key point is present could more
issus core esplunation correct und relevant but may include a tew extra, non-essential detalls. jeacellend;
explanation clearly identiiiss
minor unnecessary details.
necessary information dlarryy and offectively answers the queetion.
Destustion Stepe:
ground euplanation.
idantity specifis contrut
exptonation that reprosents leaus
(n a faul, mistske)
Read model cuppst explacation.
ground acptanation,
dinecly
and concisely the model outpurt identiftie the kay loss. explanation
Assign
rolevance
Model Generited Explanation: (MModel Generated Conteni)
Ground Trush Explanation: ([Broand Trush Content)) Evaluation ONLY)
Figure 10: Criteria and prompt used in G-Eval score evaluation. The same prompt template is applied to all four evaluator models.
30

---

Published as a conference paper at ICLR 2025
K SPORTU-TEXT PROMPT TEMPLATES
K.1 ZERO SHOT STANDARD PROMPT
Table 12: Prompt Template for Zero-shot Standard Prompt SPORTU-text
role: "system", "You are a sport experts answering sports-related questions. Please indicate the correct answer(s) clearly with letter(s))"
content:
role: "user",
content: "Question: {(question)) {foptions}} ). Only output the correct option letters."
K.2
ZERO SHOT COT PROMPT
Table 13: Prompt Template for Zero-shot CoT Prompt on SPORTU-text
role: "system", content: "You are a sport experts answering sports-related questions. Please indicate the correct answer(s) clearly with letter(s)."
ruser""ser",
content: Question: {questiion}} ((options)), Let's think step by step."
31

---

Published as a conference paper at ICLR 2025
K.3
FIVE SHOT STANDARD PROMPT
Table 14: Prompt Template for Five-shot Standard Prompt on SPORTU-text
role: "system", content: "You are a sport experts answering sports-related questions. Please indicate the correct answer(s) clearly with letter(s)."
shot examples)
role:"user",
content: "Question: ({question)) {(options)) Only output the correct option letters."
K.4
FIVE SHOT COT PROMPT
Table 15: Prompt Template for Five-shot CoT Prompt on SPORTU-text
role: "system", content: "You are a sport experts answering sports-related questions. Please indicate the correct answer(s) clearly with letter(s)."
(five-shot examples with explanations)). { role:"user",
content: "Question: {[question)} Options: {Loptions))."
32

---

Published as a conference paper at ICLR 2025
L SPORTU-VIDEO PROMPT TEMPLATES
L.1 X- YR PROMPT TEMPLATE
Table 16: Prompt Template for SPORTU-video X YR. setting
role: "system",
content: "You are sports expert analyzing series of video frames that form a con- tinuous short video clip. Your task is to answer questions based on the content of these frames."
role: "user",
content:
{(video fr
type: "text",
text: "Based on the video frames provided, answer this sports-related question:
{fquestion}) Options: {(options)]. }- Respond with the letter of the correct option, for- matted as "The correct answer is: and explain why you chose that option."
}
L.2 X RY PROMPT TEMPLATE
Table 17: Prompt Template for SPORTU-video X - RY setting
system",
content: "You are sports expert analyzing series of video frames that form tinuous short video clip. Your task is to answer questions based on the content of these frames."
role: "user"
content:
[video_frames])
type: "text",
text: "Based on the video frames provided, answer this sports-related question:
{question)} Options: {{options}). )- Let's answer this question step by step. add sen- tence formatted as 'The correct answer is: at the end of your thinking process.
33

---

Published as a conference paper at ICLR 2025
L.3 X Y PROMPT TEMPLATE
Table 18: Prompt Template for SPORTU-video X + setting
role: "system",
content: "You are a sports expert analyzing series of video frames that form a con- tinuous short video clip. Your task is answer questions based on the content of these frames."
role: "user",
content: {(video [(video.frames]]] ]rames)]
type: "text",
text: "Based the video frames provided, answer this sports-related question:
[qqestion)} Options: {Hoptions}} ). Answer with only the letter of the correct option."
1
}
L.4
OPEN-ENDED TEMPLATE
Table 19: Prompt Template for SPORTU-video Open-ended Questions
role: "system",
content: "You are sports expert analyzing series of video frames that form a con- tinuous short video clip. Your task answer questions based on the content of these frames."
role: "user",
content: (video (video.frames) }rames)}
type: "text",
text: "Based the video frames provided, answer this sports-related question: [question)}} Please provide a detailed explanation, focusing the sports aspects of the video."
34

---

Published as a conference paper at ICLR 2025
M ADDITIONAL ERROR ANALYSIS
This section presents the error analysis of the models we evaluated on SPORTU-video. For the error lypes, we evaluated the models with coarse granularity, dividing the errors into five categories as follows:
Question Understanding Error - The model misinterprets the intent or context the ques- tion, providing answer that does not align with what the question is asking.
Visual Perception Error The model incorrectly interprets the visual content, leading to faulty assumptions about the data presented in the video.
Hallucinations The model generates content or details that do not exist in the actual data, essentially 'hallucinating' information.
Reasoning Error The model exhibits poor logical reasoning, resulting in incorrect con- clusions based on the available data.
Lack of Domain Knowledge The model fails answer questions that require specific domain expertise, revealing a gap in its knowledge.
Cenooae wf
Figure 1I: Error type distribution across different MLLMs on SPORTU-video tasks: the left side represents errors from multiple-choice questions, while the right side represents errors from open- ended questions.
We observe that open-ended questions have the highest frequency of question understanding errors. For instance, when asked 'Why is it a foul the video?', the model might respond that there is no foul, which is question understanding error because the phrasing of the question already implies that a foul occurred. This issue is much less frequent in multiple-choice questions (as shown on the left side of the figure [T). When presented with four options, the model tends select of the provided answers, rather than completely misinterpreting the premise of the question. Examples of each Error are in Appendix
35

---

Published as a conference paper at ICLR 2025
N
QUESTION TEMPLATE FOR EACH SPORT
This section shows the questions we use for each sport. (Table 20 Toble26
Table 20: Question Template for Volleyball
Why fault in the video?
What sport does this video show?
What kind of fault or foul does this video show?
What main color jersey is the libero wearing?
the libero's 's jersey color does not count, what main color jerseys are the players on the right side wearing?
the libero's 's jersey color does not count, what main color jerseys are the players on the left side wearing?
the libero's 's jersey color does not count, what main color jerseys are the players on the far side wearing?
the libero's 's jersey color does not count, what main color jerseys are the players on the close side wearing?
What main color jersey is the libero on the close side wearing?
What main color jersey is the libero on the far side wearing? What main color jersey is the libero on the left side wearing? What main color jersey the libero on the right side wearing? Is there rule violation in the video?
Table 21: Question Template for Basketball
What sport does this video show?
What specific type foul, any, occurred the video? Choose the most appropriate one.
What main color jersey is the offensive team wearing?
What main color jersey is the defensive team wearing?
What main color jersey does the player who committed the foul wear?
What main color jersey is the player who was fouled wearing?
What main jersey colors do the teams wear in this video? Is there rule violation in the video?
Why foul in the video?
Table 22: Question Template for Badminton
Why rule violation the video?
How is the player making a technical error in the video?
What sport does this video show?
What kind of rule violation is in the video?
What main color jersey the left side team wearing?
What main color jersey is the right side team wearing?
What main color jersey is the close side team wearing?
What main color jersey is the far side team wearing?
What specific action does the player perform in the video?
How many players shown in total in this video?
Is the player making a technical error in the video?
there rule violation in the video?
36

---

Published as a conference paper at ICLR 2025
Table 23: Question Template for Baseball
What sport does this video show?
What kind of rule violation in the video?
Based on this video, which of the following descriptions best applies to the situation? As referee, what procedure would you follow in this situation the video?
What main color jersey is the fielder team wearing?
What main color jersey is the batting team wearing?
What main jersey colors do the two teams wear in this video?
there rule violation in the video?
Why did the rule violation occur in the video?
Table 24: Question Template for Soccer
Why is it a foul in the video?
Why not foul in the video?
Based on this video, which of the following descriptions apply to the situation that occurred?
What sport does this video show?
What kind of foul does this video show?
What main color jersey is the offensive team wearing?
What main color jersey is the defensive team wearing?
What main color jersey does the player who committed the foul wear?
What main color jersey is the player who was fouled wearing?
What main color jersey does the goalkeeper wear?
What main jersey colors do the two teams wear in this video?
How many players are shown in total in this video?
there rule violation in the video?
Table 25: Question Template for Ice Hockey
What sport does this video show?
What kind of foul is committed in the video?
What main color jersey does the player who committed the foul wear?
What main color jersey is the player who was fouled wearing?
What main jersey colors do the two teams wear this video? Is there rule violation in the video?
we consider the fight in the video to be a legit fight, defined as a fight between two willing participants who drop their gloves and helmets, with the fight ending when one player falls officials intervene, and this fight does not result in a penalty, is there any other foul in the video that will cause penalty?
Why is it foul in the video?
Table 26: Question Template for American Football
Why is it an error in the video?
Why foul in the video?
What sport does this video show?
What kind of foul does this video show?
What kind of error does the player make in this video?
What main color jersey is the offensive team wearing?
What main color jersey is the defensive team wearing?
What main jersey colors do the two teams wear this video?
Is there foul in this video?
Does any player make an error in this video?
37

---

Published as a conference paper at ICLR 2025
O SPORTU-TEXT EXAMPLES
O.1 RULE-RELATED QUESTION
Table 27: Rule Question
Question:
How does ball possession work in baskethall after the successful final free throw? A) The team that made the free throw retains possession.
B) The team that missed the free throw gains possession.
C) The opposing team gains possession.
D) The team with the most points gains possession.
Answer: C
Explanation: According to FIBA's rules on what should happen after a successful field goal or the last successful free throw, following the last successful free throw, any player from the non-scoring team should inbound the ball from any position behind their own endline; therefore, option A is incorrect. The team that missed the free throw is the same team the one executing the free throw during the game, but the question specifies that the team's last free throw was successful; therefore, option B is incorrect. Since the rule states that any player from the non-scoring team should inbound the ball from behind their own endline, and since the team that executed the last free throw scored, the non-scoring team, i.e., the opposing team, should regain possession of the ball and inbound it; therefore, option C is correct. The scoring situation of both teams does not affect the distribution of possession after successful free throw, so neither the number of points scored nor the point differential is determining factor for possession distribution; therefore, option D is incorrect. Hence, the correct answer is option C.
Table 28: Rule Question Which player is typically responsible for throwing the ball to the receivers American football?
Question: Question:
A) Linebacker.
B) Quarterback.
C) Running back.
D) Tight end.
Answer: B
Explanation: Linebacker is on the defensive side and does not often have a chance to touch the ball, therefore option A is incorrect. Quarterback usually throws the ball to running back, therefore option B is correct. Running back is the one who often receives the ball, therefore option C is incorrect. Tight end usually blocks the other player and not primarily meant grab the ball, therefore option D is incorrect. Hence, the correct answer is option B.
38

---

Published as a conference paper at ICLR 2025
O.2 STRATEGY-RELATHR QUESTION
Table 29: Strategy-related Question Sample 1
What are key considerations when implementing a successful blocking scheme volleyball defensive strategies?
Question:
A) The blocker's positioning in relation to the attacker's hitting arm.
B) The speed of the incoming serve.
C) The position of the setter on the opposing team.
D) The timing and coordination between the blockers.
Answer: ABCD
Explanation: The positioning in relation to the attacker's hitting arm is crucial for successful block and for the entire team's 's block and defensive formation. The side blocker usually posi- tions their head line with the hitter's arm and the ball to cover the straight line, or uses their right hand on the hitter's arm and the ball cover the cross-court angle. The back-row players will adjust their position based on the main angles that the blockers are covering. Therefore, option A is correct. Increasing the speed of the serve makes it harder for the opponent re- ceive, potentially putring them out of system. This reduces the opponent's 's attacking options compared when they are system, making blocking easier. Therefore, option B is correct. The setter's position directly determines the team's 's strategic options and makes it easier for blockers read and predict the play. Therefore, option C is correct. Timing and coordination between the blockers are also essential, as blockers aim to jump together without leaving gaps multi-blocker
they decide to execute a multi- blocker jump. Therefore, option is correct. Hence, the correct answer to this question is A, B, C, and D.
Table 30: Strategy-related Question Sample 2
Question: Question: Which of the following are common attack patterns in volleyball offensive
strategies?
A) Quick set
B) Slide attack
C) 5-1 formation
D) Triple quick
Answer: AB
Explanation: A quick set is an attacking pattern for the middle blocker. The middle blocker jumps close to the setter and jumps before the setter sets the ball. The setter then sets the ball low and quickly to the middle blocker's optimal attacking position. Therefore, option A is correct. A slide attack is another attacking pattern for the middle blocker, where the middle blocker approaches the right-side position, usually 2-3 meters from the setter, and swings with a one-foot jump. The setter will set a low are ball backward to the right side. Therefore, option B is correct. The 5-1 formation is team strategy concerning how many setters are on the court and is not related to attack patterns. Therefore, option C is incorrect. 'Triple quickis not a common attack pattern or strategy in standard volleyball. Therefore, option incorrect. Hence, the correct answers are A and B.
39

---

Published as a conference paper at ICLR 2025
O.3 CHENARIO-RELATER QUESTION
Table 31: Scenario- Question
Question:
Which of the following scenarios would most likely result in a red card offense in a football match?
A) A player politely disagrees with the referee's decision.
B) player accidentally trips another player while trying get the ball.
C) A player uses offensive language or gestures towards the referee.
D) player passes the ball back to his own goalkeeper.
Answer: C
Explanation: Disagreeing with the referee's decision, as long as it done reasonably and without abuse, will not result in red card, therefore option A is incorrect. Accidentally trip- ping a player will not result red card because not intentional, therefore option is incorrect. Using offensive language or gestures violates sportsmanship and will result in red card, therefore option is correct. Passing the ball back to the goalkeeper does not result red card; it will result in an indirect free kick, therefore option D is incorrect. Hence, the correct answer is option C.
Table 32: Scenario-Related Question 2
Question: In a competitive basketball game, Player A accidentally knocks the ball which then bounces off Player B's hand, rolls on the boundary line, touches Player C's foot while he is standing on the line, and finally goes out of the court. Which player is considered to have caused the ball go out of bounds?
A) A) Player A because accidentally knocked the ball first.
B) Player because the ball touched his hand before rolling the boundary line.
C) Player because the ball touched his foot while he was standing on the boundary line.
D) None of the players caused the ball to go out of bounds because the ball rolled its own.
Answer: C
Explanation: According to FIBA's rules regarding out-of-bounds, only the player who last touched or was touched by the ball before it went out of bounds is considered responsible for the out-of-bounds situation. Since Player A touched the ball before touched Player B, Player A was not the last player to touch the ball; therefore, option A is incorrect. Similarly, after Player touched the ball, it then touched Player C, so Player was not the last player to touch the ball; therefore, option B is incorrect. Because Player C was the last player to touch the ball before was deemed out of bounds, and the ball touched Player C's foot, causing to go out of bounds, Player C is considered responsible for the out-off-ounds situation; therefore, option correct. Since the ball touched least one player before going out of bounds, it cannot be considered that player caused the ball to go of bounds; therefore, option D is incorrect. Hence, the correct answer is option C.
40

---

Published as a conference paper at ICLR 2025
P SROUTU-VIDEO EXAMPLES P.I BASKETBALL
Question:In this video, what are the main jersey colors of the two teams?
A: One team wears purple jersey, the other team wears red jersey,
B: One team wears red & blue jersey, the other team wears red jersey,
C: One team wears black&orange jersey. the other team wears red blue jersey, One team black jersey, the other team wears white jersey
Answer:
Figure 12: Basketball Easy level Question
Question: Is there rule violation in the video? A: Yes B:No
Answer: A
Figure 13: Basketball Medium level Question
41

---

Published as a conference paper at ICLR 2025
Why is foul in the video? Answer: The player in the purple and yellow jersey elbowed the shooting player in white in their right armpit.
Figure 14: Basketball hard level Question
P.2 VOLLEYBALL
M
VM
VOUTTRAL
LEASEE
VOLLEYB
Question:What sport does this video show?
A: Other B: Indoor Volleyball C: Basketball
D: Badminton
Answer: B
Figure 15: Volleyball easy level Question
42

---

Published as a conference paper at ICLR 2025
Question:lf the libero's jersey color does not count, what main color jerseys are the players the close side wearing? A: white B: red C: yellow D: white&red Answer: A
Figure 16: Volleyball medium level Question
Question:Why is a fault in the video? Answer: The player in the red jersey the right side reached over the net with their right hand while hitting the ball
Figure 17: Volleyball hard level Question
43

---

Published as a conference paper at ICLR 2025
P.3 SOCCER
Question: How many players are shown in total this video?,
A: B: C: D: 4
Answer: B
Figure 18: Soccer easy level Question
Question:Based on this video, which of the following descriptions apply to the situation that occurred?
A: Penalty kick B: Other C: Offside D: DOGSO-SPA
Answer: D
Figure 19: Soccer medium level Question
44

---

Published as a conference paper at ICLR 2025
OW you
now you
000
now yo
can
L can
Question: What kind of foul does this video show? A: trip foul B: offside foul C: pull foul D: dive foul Answer: D
Figure 20: Soccer hard level Question
P.4 BADMINTON
2020
020
TOKYO
2020
TOKYO 2,20
g
TOKYO 2020
TOKYO 2020
2013
9
Question:What main color jersey is the far side team wearing? A: blue B: black C: purple D: yellow Answer: A
Figure 21: Badminton easy level Question
45

---

Published as a conference paper at ICLR 2025
Question: What specific action does the player perform in the video? A: Net Shot B: Net Kill C: Push Shot D: Lift
Answer: A
Figure 22: Badminton medium level Question
Question: What kind of rule violation is in the video?
A: Service Fault B: Over-the-net Fault C: Early Hit Fault D: None of the above Answer: A
Figure 23: Badminton hard level Question
46

---

Published as a conference paper at ICLR 2025
P.5 AMERICAN FOOTBALL
Question: What sport does this video show?
A: Baseball B: Handball Ice Hockey D: American Football
Answer: D
Figure 24: American Football easy level Question
48
48
Question: What main color jersey is the offensive team wearing? A: orange B: blue white D: green
Answer: C
Figure 25: American Football medium level Question
47

---

Published as a conference paper at ICLR 2025
Question: What kind of error does the player make in this video? A: Interception B: Missed tackle C: Missed catch D: Touchdown Answer: B
Figure 26: American Football hard level Question
P.6
ICE HOCKEY
Question: What sport does this video show?
A: Ringette B: Bandy C: Broomball D: Ice Hockey
Answer: D
Figure 27: Ice Hockey easy level Question
48

---

Published as a conference paper at ICLR 2025
SKIJ
THE S
EXIT AIG
HES
DIS
MBara
EES
If we consider the fight in the video to be legit fight, defined as fight between two willing participants who drop their gloves and helmets, with the fight ending when one player falls officials intervene, and this fight does not result in penalty, there any other foul the video that will cause a penalty?
Question:
A: No B: Yes Answer: A
Figure 28: Ice Hockey medium level Question
Co
ccO
iCO
MOCO
NOCO
MOCO
Question: What kind of foul is committed in the video? A: boarding B: third-man-in C: tripping D: elbowing
Answer: D
Figure 29: Ice Hockey hard level Question
49

---

Published as a conference paper at ICLR 2025
P.7 BASEBALL
Question: In this video, what the main jersey colors of the two teams?
A: One team wears dark blue jersey, the other team wears grey jersey white&red jersey, the other team wears dark red jersey C: One team wears dark blue jersey, the other team wears white&green jersey
B: One team
D: One team wears white jersey, the other team wears dark blue jersey
Answer:
Figure 30: Baseball easy level Question
on
ton
mton
onion
omion
Question: What main color jersey is the batting team wearing? A: tan B: green C: blue D: gray
Answer: C
Figure 31: Baseball medium level Question
50

---

Published as a conference paper at ICLR 2025
question: What kind of rule violation is in the video?
A: Balk B: Running Out of the Baseline C: Interference D: None of the above Answer: C
Figure 32: Baseball hard level Question
51

---

Published as a conference paper at ICLR 2025
Q EXAMPLES OF EACH ERROR TYPE
Q.1 QUESTION UNDERSTANDING ERROR
Question:Wy fault in the video?
steppped
foot, why player
right arm right side
neck
Answer: player
by player
ground
Error Category: :
Question Understanding
The model's answer focuses on explaining the reason for the player's fall, which is different from the question's intent. Although mentions faul, fails to identify key details provided question's purpose and eventually provides an explaratiio focused on the player's This shows the model misunderstands
Error Reason:
Figure 33: Examples of Question Understanding Type and Error Reason Explanations
Question:Wyy fault in the video?
Ground Truth: the process shooting, the defender's right hand offensive player's 's am before making contact with the ball.
Model answer: The video does not show foul being committed. The player in the black jersey successfully completes dunk, and there is infraction called by the referees.
Error category: Question Understanding Error
Error Reason:
The question assumes that fault has already occurred the video and asks for the reason behind it. However, the model recognise this presugpositin. Insleed,
occurre,, contradicting question's premise, misunderstanding of the question's intent and preconditions, leading to wrong answer.
Figure 34: Examples of Question Understanding Error and Error Reason Explanation
52

---

Published as a conference paper at ICLR 2025
IRAL-AD
APAL A60
Question:Why at teuit in the video?
Ground Truth: player white jersey
early
jersey
serve
any movement action determine foult. To volleyball game, need to see the actuel play. 1suit volleybell occurs when player violales the rule Net touch:A player touches the net with any part of their body while the ball is in Foot fault: server steps on over the end line player the cleenly, bounds: The ball lands outside the court boundaries. Please provide video clip or more context for me to accurately determine the reason for the fault.
Answer: Unfortunately, the provided videa still
answer your question about
Double
Carrying the player holds
the instead hitting
Error Category Question Underslanding Error
Reason Although question explicitly meunderstanding leads the model provide wrong output.
mistakenly assumes input image.
Figure 35: Examples of Question Understanding Error and Error Reason Explanation
Q.2 VISUAL PERCEPTION ERROR
Quastion:What kind rule violation is in video? Over-the-net
above
Ground Truth: A
Model Answer: The cornect answer Explanation: The frames show badminton match with players actively engaged in rally. There occurrin
over-the-net correctly, and the shuttlecock in play without any apparent infractions.
hit,
The players poellioned
Category: Perception
Error Reason: The model falls to recognize that the player on the right extends their racket over the net block the play the on the left, resulting wrong answer.
Figure 36: Examples of Visual Perception Error and Error Reason Explanation
53

---

Published as a conference paper at ICLR 2025
Question:What main color jersey the defensive team wearing? grean & white black pink blue white
Model Answer: The correct answer is: C. The defensive team is wearing pink jerseys.
Error Category: Visual Porception Error
jersey misinterprets
Figure 37: Examples of Visual Perception Error and Error Reason Explanation
M
VNL
VOLLEYS
Question:Why fault in the video?
Ground Truth: Players white jerseys hold ball their hand too long
Model Answer: The faull the video that player wearing shirt. This violation of rules of volleyball, players required wear shirts during game.
Error Category: Visual Perception Error Error Reason: The model feils distinguish the jersey color, causing incorrecty assume that the player not wearing which wrong answer.
Figure 38: Examples of Visual Perception Error and Error Reason Explanation
54

---

Published as a conference paper at ICLR 2025
Q.3 HALLUCINATION ERROR
Tim Ma
Tim
Time
Tim H
AMBRIK
AMMRIA H
AMARIA
AMARIA
Tim Ha
Tim
Ti
BAMBRIA
CAMBRIA WN
CAMBRIA
CAMBRIA
Question/What committed video?
A: boarding B: cross checking C: Bhird-man-in D: bind-side hit
Ground Truth:
Model Answer: The correct enswer is: boarding. In the video, player from the orange team is thrown into the air player from the teem,
boarding.
Error Category:
Hallucinations
Error Reason: There no player got thrown into air, which leads hallucinations.
Figure 39: Examples of Hallucinations Error and Error Reason Explanation
Ope
peeeee eeppppe
peps
pepspeessss pepsi
Question:How many players 7: B. C: D:9
Ground Truth:
Model Answor:The The correct answer C. 4 Based on the videa frames proviced, there are players shown the player the blue jersey. player jersey.
players ground engaged physical confrontation.
Error Category:
Hallucinations
Error Reason: The model hallucinates by inventing two non-existent players in the video, leading to an incorrect answer.
Figure 40: Examples of Hallucination Error and Error Reason Explanation
55

---

Published as a conference paper at ICLR 2025
RETERFN
RED
RFTEREN
TEREN
RFTERED
1o
REN
RETERFN
REN
RETERFD
FREN
RFTFLED
FRED
RFTERFT
Question:Why a the video?
Ground Truth: yellow blocks sideway player in black jersey advance.
Model Answer:
the videa, foul is called when soccer player slides into the goal and makes contact with another player. The foul sliding player's foot
standing player's body, causing standing player belence potentially suffer an injury. This type of called a "ssliiin tacklel" and generally considered unsportsmanikke and dangerous soccer games. The referee, who responsible for enforcing the rules and ensuring fair play, blows the whistle stop the game and award safety well-being players
upposing the game.
inlegrily of
Error Category: Hellucinstions
Error Reason: The model hallucinates extensive details not
video, such referee blowing the whistle and awarding appear the video, leading inacourate answer
None
Figure 41: Examples of Hallucination Error and Error Reason Explanation
Q.4 REASONING ERROR
Question: What does this video show?. American Footbell Cricket C: Floorball Basketball
Ground Truth:
Model Answer: The correct answer is: A. American Football. The video frames clearly show basketball court and players wearing baskatball uniforms, indicating the sport playod baskotball, American football, cricket, floorball.
Error Category Reasoning Error
correctly identifies it fails to lagically connect its observations to the correct conclusion, inesplicably choosing "American Footbal! instead of "Basketoall. This incone inconsistency istency démonstrates reasoning error integrating observed tacts with the given anewer options.
content, descrbing buskettzlll court players
However,
Figure 42: Examples of Reasoning Error and Error Reason Explanation
56

---

Published as a conference paper at ICLR 2025
Questionn:n this video, what are the main jarsey corors the two teams? One team wears blackkyellew jersey, the other team wears whiteggreen jensey.
One leam wears black&yellow jarsey, the other leamn wears black&ysllow jersey, white
One team jersey the other
jersey,
jarsey
answer option because frames show one player weelling white jersey with blue and orange accents, while the other player is a dark jersey that appears to be black with green accents. This matches the descr piton aplion A, where jersey (interpreted as
Answer:
Error Category: Reasoning Enor
The model eccurately de bescribes the jersey colors observed in the video, identifying one as white with blue and orange accents and the other as black with acpents. However, incorrecly
Error Reeson:
these observations the jersey aptions, loading logical incosistency (e.g. orange eccents" main jarsey oolor. This demonstratas visual percection or domein knowledge.
black and low accentsi green jersey (interpreiee orange accants).
descriptions
interoreling "white
green't
reasoning rather lack
Figure 43: Examples Reasoning Error and Error Reason Explanation
13
15
13
B10
BARIPP 13
15
13
BARPER
15
DASER
B40
Question:Why a foul the video?
Model Answer: video shows basketbal player white jersey with the number 15 and the word LLkkersi on it, who appeers to be erguing or dispuling with another player blue and player in the white jersay gesturing with his hands and seems to be expressing frustration disagreement. The player in the jersey. defensive
Ground Truth: The while elbowed the blue jersey player
the and jaw.
jersey
iHardeni
Error Category: Reasoning Error
delailed descriptions interaction beiween the players, including physical contact that could reasonably be Interpreted foul. However, it falls to logically conclude
the player which could be interpreted physical allercation move that mey have been perceived es foul by the player in the white jersey. However, without additional context information about the specific play, possible definiively datermina whether foul was committed.
jersey
demonsirating an incomplete massaing process despite having sufficient abservational details.
Figure 44: Examples of Reasoning Error and Error Reason Explanation
57

---

Published as a conference paper at ICLR 2025
Q.5
LACK OF DOMAIN KNOWLEDGE
Ame
Ame
Amer
Amer tel
Red Riv
Red River
Re
Amerita
Ameritel
Red Ri
Ree
Question: Is there rule violation video?
Ground Truth:
Madel answer: The correct answer is: B. No. There no rule volation the video. The players are engaged in a game hockey, and teams following
game.
Error category: Lack of Domain Knowledge
resognize rule depicted
player intentionally ogponent's
with their stick, constiluting slashing hockey. This indicates the model's lack understanding of hockey rules and its inaiiiity to identify common fouls, leading to an incorrect conclusion that no violation occurred.
Figure 45: Examples of Lack of Domain Knowledge and Error Reason Explanation
Parts
Paris
Paris
MiKASA
AAS
Question:ls there a rule the video?
Ground Truth:
Model Answer: The correct answer is: Besed on the video trames, there does not appear to be any rule violation visible. The players on court
engaged volleyball match, and their actions and
any
the game.
Error Category: : Lack of Domain Knowledge
recognize specific volleyball violation: allacking player not allowed complete allack above the net. This oversight demonstrates model's lack understanding detailed volleyball rules, leading to an incorrect conclusion that no rule violation occurred.
Reason:
sets
three-meter an
Figure 46: Examples of Lack of Domain Knowledge and Error Reason Explanation
58

---

Published as a conference paper at ICLR 2025
Question:What kind fault or foul does this video show?
fault,
Ground Truth:
Model Answer: model The correct
This video shows net faull. valleyball, net fault touches the net while attempting to play the ball. The frames depict players prepering for serve and the subsequent action, which suggests net during the play. common violation resuits point the opposing
player
player may
Category:
Domain Knowledge
Error Reason: model fails recognize the action the video players moving prematurely before the serve-which clear indicalor of a rolation fault. Instead, the model incorrectly idenlifies the faull as net faull, suggesling lack of understanding of volleyball rules and dielinct characteristios of different types faults.
Figure 47: Examples of Lack of Domain Knowledge and Error Reason Explanation
59


Reviews:
[{"review_id": "oIUOcvFoLE", "reviewer": "Reviewer_7Lbr", "summary": "This paper introduces SPORTU, a new benchmark designed to evaluate the capabilities of Multimodal Large Language Models (MLLMs) in sports understanding and reasoning. SPORTU consists of two components: SPORTU-text, focusing on text-based reasoning, and SPORTU-video, focusing on video-based reasoning. The authors evaluate various LLMs on both components, revealing limitations in complex reasoning tasks.", "questions": "Please check weakness for details.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The proposed dataset could be useful for the community.\n2. Both close and open-sourced models are evaluated.\n3. Metrics are studied with human verification.", "weaknesses": "1. The reviewer is concerned about the lack of diversity and coverage of the dataset because of the limited prompt templates and number of samples. \n\n2. Implementation could be possibly flawed. \n- The error in Figure 6 looks suspicious and makes the reviewer wonder whether the model is called correctly or not. \n- The reasoning prompt asks the model to first generate answer and then reasoning, which is not optimal since the model's final answer cannot benefit from the reasoning process.\n- It is known that LLM usually prefers its own answer so it is important to understand G-eval' quality with different LLMs as the rater.\n\nMinor:\nL821 typos of \"Section ??\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "buMbmAVZp6", "reviewer": "Reviewer_uNov", "summary": "The paper provides a multimodal dataset (text and slow-motion video) for evaluating (multimodal) LLM capabilities in the sports domain.", "questions": "- The video quality for the datasets\n- For each sports type, the video are biased to certain views or events?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- A multimodal new dataset for sports domain (with multiple sports) and well annotated by experts; the dataset should be helpful for the research communities\n\n- A well prompting capabilities to show the limitation of current LLM capabilities on the dataset. \n\n- Evaluating several reasonable public or private LLM models.", "weaknesses": "- Yet another vertical dataset for LLM\n- It's helpful but marginal to expand the technical depth for the community\n- not clearly identified what current models failed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pIrRI0H7m5", "reviewer": "Reviewer_gCvg", "summary": "In the AI+Sports area, existing works are limited to restricted kinds of sports, absence of explanations, or lack of reasoning on rules, and it proposes SPORTU consisting of SPORTU-text and SPORTU-video to boost understanding more sports with rules understanding. SPORTU-text evaluates models on rule comprehension and strategy understanding in the pure text domain and SPORTU-video evaluates models on understanding both video details and rules in the video domain.\nIt evaluates LLMs and MLLMs on SPORTU-text and SPORTU-video, revealing their limitations in complicated sports questions.", "questions": "1. How do you split the SPORTU-text questions into rules-related, strategy-related, and scenario-related? What is your basis?\n\n2. What are the results on rule/strategy/scenario, respectively, on sportu-text?\n\n3. How is the error analysis in 5.1 conducted? Is there a definition for each error type?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. It proposes SPORTU-text and SPORTU-video to boost understanding more sports with rules understanding in text and video domains.\n2. It analyzes the views, reasoning prompts, sport types, the error types, which are comprehensive.\n3. The writing is clear.", "weaknesses": "1. A benchmark aims to evaluate certain abilities and give some insights. The paper does not deeply discuss why the models have different performances and does not give advice on how to resolve the problem of understanding videos and reasoning on rules.\n\n2. Prompt strategy in LLM can also be tested on MLLM when evaluating on SPORTU video benchmark to see how the reasoning process influences MLLM.\n\n3. It's not very clear if the questions in this dataset can comprehensively detect the models' abilities to understand sports.\n\n4. The Pearson correlation between humans and the other metrics is low. Many are near 0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LtejoPtgHe", "reviewer": "Reviewer_rN5c", "summary": "This paper prensents SPORTU, a comprehensive Sports Understanding Benchmark that integrates both text-based and video-based tasks to evaluate models’ sports reasoning and knowledge application capabilities. Based on this benchmark, this paper tests the capability of existing open-source or close-source  models.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. As a sport domain understanding benchmark, the proposed  SPORTU combines text-based and video-based tasks to assess models' sports reasoning and knowledge application abilities.\n\n2. The evaluation setting is comprehensive including the direct prompting, chain-of-thought (CoT) prompting. In addition, few-shot promoting is also applied in SPORT-text evaluation.", "weaknesses": "1. Unclear motivation. The authors should clarify the differences between the proposed SPORTU and existing sport domain understanding benchmarks. Although discussions have been made in introduction and related work section together with Table 1, it is still unclear why the introduced features , for example, slow motion, multi-camera angles are important. More discussions and visualizations are needed.\n\n2. Missing details in dataset construction. There exist some unclear details in the dataset construction. For example, how to guarantee the multi-camera setting? Is it achieved simply by human annotator check? In addiation, the proposed SPORTU contains both the multi-choice and open-ended question, how are these two categories divided?\n\n3. More advanced evaluation methods should be applied. For example, ST-LLM [1], qwen-vl [2]\n\n4. The paper writing should be polished. Some references are missing, for example \"Section ??\" in Line 821. The quotation mark error in '”Why is it a foul in the video?”' in Linee 482.\n\n[1] ST-LLM: Large Language Models Are Effective Temporal Learners\n[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)