SportU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models

## Abstract

Abstract Multimodal Large Language Models (MLLMs) are advancing the ability to reason about complex sports scenarios by integrating textual and visual information. To comprehensively evaluate their capabilities, we introduce SPORTU, a benchmark designed to assess MLLMs across multi-level sports reasoning tasks. SPORTU comprises two key components: SPORTU-text, featuring 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models’ ability to reason about sports solely through question-answering (QA), without requiring visual inputs; SPORTU-video, consisting of 1,701 slow-motion video clips across 7 different sports and 12,048 QA pairs, designed to assess multi-level reasoning, from simple sports recognition to complex tasks like foul detection and rule application. We evaluate four prevalent LLMs mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting on the SPORTU-text part. We evaluate four LLMs using few-shot learning and chain-of-thought (CoT) prompting on SPORTU-text. GPT-4o achieves the highest accuracy of 71%, but still falls short of human-level performance, highlighting room for improvement in rule comprehension and reasoning. The evaluation for the SPORTU-video part includes 7 proprietary and 6 open-source MLLMs. Experiments show that models fall short on hard tasks that require deep reasoning and rule-based understanding. Claude-3.5-Sonnet performs the best with only 52.6% accuracy on the hard task, showing large room for improvement. We hope that SPORTU will serve as a critical step toward evaluating models’ capabilities in sports understanding and reasoning. The dataset is available at https://github.com/haotianxia/SPORTU .

## 1 Introduction

Figure: Figure 1: SPORTU consists of two parts: SPORTU-text, which evaluates sports understanding through text-based multiple-choice questions, and SPORTU-video, which assesses multi-level reasoning through video-based QA tasks. SPORTU provides a comprehensive sports understanding evaluation of multi-leveled reasoning beyond perception. Right side is a sample from the scenario-based question in SPORTU-text, along with examples of both hard-level and easy-level questions from SPORTU-video.
Refer to caption: x1.png

The sports domain has witnessed a surge in interdisciplinary research, combining Natural Language Processing (NLP) and computer vision (CV) to tackle a wide range of applications. For instance, NLP-based approaches have been leveraged for automated sports news generation, producing detailed summaries and news articles from game data (Huang et al., ; Wang et al., ). Concurrently, hate speech detection has been employed to mitigate the impact of toxic content on social media (Vujičić Stanković & Mladenović, ), enabling athletes to maintain focus on their game. In the realm of CV, action recognition (Zhu et al., ; Li et al., ), player detection (Maglo et al., ; Vandeghen et al., ), and tactical analysis (He et al., ; Xia et al., ) have been explored, enhancing visual content for analysis and fan engagement. The recent emergence of Large Language Models (LLMs) (OpenAI, ; AI@Meta, ; Jiang et al., ; Anil et al., ) and Multimodal LLMs (MLLMs) (OpenAI, ; Gemini Team, ; Anthropic, ; Lin et al., ) has further accelerated this trend, enabling researchers to develop novel tasks such as AI-assisted refereeing (Held et al., ; ), where models analyze game videos to identify fouls and violations, and interactive sports education (Zhang et al., ; Zeng et al., ), where users engage with LLMs to learn rules, strategies, and game-related content.

However, the effectiveness of these applications depends crucially on the model’s deep understanding of sports knowledge. While LLMs act as study guides, helping users learn rules and general strategies through text, MLLMs extend this knowledge to video-based tasks that require video and action perception, as well as the ability to connect movements with context-based rules. For example, when answering a question like “Why is it a rule violation in the video?”, a model must distinguish the series of actions performed by the players and understand the corresponding rules that define the fault. This capability underscores the deeper reasoning required for real-world sports comprehension. The challenges, ranging from general video recognition to deep sports knowledge reasoning, highlight the need for a dedicated sports-focused question-answering (QA) dataset to improve the model’s ability to comprehend and contextualize sports information effectively.

Existing sports QA datasets, either text-based or video-based, have limitations that hinder a comprehensive evaluation of sports understanding. Text-based datasets (bench authors, ; Liu et al., ; Xia et al., ; Jardim et al., ) primarily assess comprehension of numerical data, rules, and context extraction, but lack detailed explanations to evaluate the underlying reasoning processes. In addition, video-based datasets, such as SportsQA (Li et al., ) and SoccerNet-XFoul (Held et al., ), focus on action understanding and multi-view QA, but are constrained by their narrow scope, covering only a single sport or lacking multi-level reasoning. For example, in SportsQA, questions like “What does TEAM A do before/after TEAM B’s action?” mainly require recognizing sequences of actions and their outcomes, rather than connecting these actions to the underlying rules and game dynamics. This highlights the need for a comprehensive multimodal sports benchmark to evaluate the capabilities of MLLMs across a diverse range of sports, with varying levels of difficulty, to assess their ability to apply deep reasoning and rule-based understanding in real-world scenarios.

To address this gap, we introduce SPORTU, a comprehensive Sports Understanding Benchmark for sports knowledge and slow-motion multilevel video reasoning. As a multifaceted benchmark, SPORTU comprises both text and video components to facilitate a thorough assessment of models’ capabilities. As illustrated in Figure 1, our dataset includes two parts: SPORTU-text and SPORTU-video. The text component, SPORTU-text, features 900 multiple-choice questions with human-annotated explanations for rule comprehension and strategy understanding. This component focuses on testing models’ ability to reason about sports through question-answering, independent of visual inputs. The video component, SPORTU-video, comprises 1,701 slow-motion video clips, including 300 clips with varying camera angles and 12,048 QA pairs, categorized into three difficulty levels. The easy level is designed to be answerable without requiring sports domain knowledge, while the hard level demands in-depth rule comprehension and accurate video perception. This tiered structure allows SPORTU-video to evaluate the sports understanding capabilities of MLLMs in a more nuanced and progressive manner. The use of slow-motion clips is crucial, as most fouls involve brief and subtle actions that may be overlooked in real-time footage. By providing models with a better opportunity to perceive and interpret these critical moments accurately, we can more effectively evaluate their performance.

To gain deeper insights into SPORTU, we evaluate 4 LLMs on SPORTU-text and 13 MLLMs that handle videos or image sequences on SPORTU-video, including both open-source and proprietary models. We selected GPT-4o, Claude-3.5, LLaMA-3, and Gemini 1.5 Pro for SPORTU-text, while for SPORTU-video, we included a broader range such as GPT-4V, Gemini 1.5 Flash, and Video-ChatGPT. The models were evaluated using few-shot learning for SPORTU-text and multi-frame input for SPORTU-video. Key results show that GPT-4o performs best on SPORTU-text, achieving 71% accuracy.Claude-3.5-Sonnet leads the SPORTU-video evaluation with an overall accuracy of 69.52%, but it achieves only 52.57% on hard-level tasks, indicating challenges with more complex reasoning.

In summary, our contributions are threefold:

- •
We introduce SPORTU, a comprehensive Sports Understanding Benchmark that integrates both text-based and video-based tasks to evaluate models’ sports reasoning and knowledge application capabilities. SPORTU-text tests models’ understanding of facts, rules, strategies, and scenario-based sports knowledge, while SPORTU-video presents a three-tiered difficulty level, featuring challenging sports scenarios such as foul detection and complex rule application, with slow-motion footage to capture fine-grained details essential for understanding nuanced actions and rule violations.
- •
This research deepens our understanding of LLM and MLLM performance on sports tasks, showing GPT-4o achieved 71% accuracy on SPORTU-text, while Claude-3.5-Sonnet led SPORTU-video with 69.52% overall accuracy. However, Claude-3.5-Sonnet struggled on hard-level tasks, achieving only 52.57%, highlighting significant gaps in deep reasoning with rule comprehension.
- •
We will make the SPORTU benchmark publicly available to support advancements in sports analytics and applications.

## 2 Related Work

### 2.1 Multimodal Sports Analysis

Recent surveys, such as (Xia et al., ), have highlighted the importance of integrating sports with NLP and CV techniques to advance research in the sports community. Traditional text-based applications have have encompassed a range of tasks, including sentiment analysis (Baca et al., ; Ljajić et al., ), game predictions (Beal et al., ; Xia et al., ; Oved et al., ; Tracy et al., ), game statistics summarization (Thomson et al., ; Hu et al., ), and game news generation and narrative construction (Sarfati et al., ; Huang et al., ; Wang et al., ; Hu et al., ). In the CV domain, studies have focused on sports action recognition (Xu et al., ; Li et al., ; Yuan et al., ; Zhu et al., ), sports action quality assessment (Zahan et al., ; Xu et al., ), and tactic analysis (He et al., ). Notably, Chen et al. () has demonstrated the efficacy of multimodal integration by leveraging natural language input to enhance sports videos with visualizations.

While prior work has made significant strides in sports understanding, the advent of MLLMs offers a fresh perspective on this domain. In contrast to earlier multimodal approaches, MLLMs boast a distinctive blend of scalability, expressiveness, and flexibility, thereby enabling the tackling of intricate sports understanding tasks with unparalleled depth and nuance.

Table: Table 1: Comparison of various QA datasets across multiple dimensions. This table highlights key distinctions between SPORTU and other QA datasets, including QA Type (MC: Multiple Choice, OE: Open Ended), whether explanations are provided, the presence of Multilevel Difficulty, and coverage of Domain Knowledge in terms of both Action and Rule. Additional features compared include whether datasets incorporate Slow Motion, Multi-Camera Angles, and the number of sports covered. Avg Exp. or Ans. Length refers to the average word count for open-ended explanations or answers.

### 2.2 Multimodal Sports QA

Question answering (QA) has been widely adopted to evaluate the comprehension abilities of LLMs across various domains. However, existing QA datasets have limited coverage of sports understanding, as shown in 1. Several QA datasets have been introduced to evaluate models’ understanding of textual information, ranging from factual recall to multi-hop reasoning (Joshi et al., ; Yang et al., ; Clark et al., ). In the multimodal domain, QA Benchmarks have been extended to evaluate MLLMs’ image and video understanding by answering questions based on visual inputs (Fu et al., ; Yue et al., ; ; He et al., ; Zhou et al., ). However, sports-related QA is underrepresented in these general datasets, and even when sports topics are included, the questions often lack sufficient difficulty due to the datasets’ broader focus.
Existing general QA datasets tend to focus on surface-level aspects of sports, such as historical facts or well-known events in text-based tasks, and perception-level tasks, like object or action identification in video-based tasks. As a result, they often fail to assess the deep, domain-specific knowledge and reasoning required for a nuanced understanding of sports.

Existing sports-domain-specific QA benchmarks are limited in their ability to test models’ sports understanding. Text-based datasets, such as BIG-bench (bench authors, ), LiveQA (Liu et al., ), and QASports (Jardim et al., ), focus on factual recall, while SportQA (Xia et al., ) is a notable exception, introducing rule-based questions that require scenario-based reasoning. However, even SportQA lacks explanations, which are crucial for evaluating MLLMs’ reasoning processes, particularly for complex tasks. The absence of explanations limits the ability to fully assess a model’s reasoning capabilities.

On the other hand, sports-domain-specific Video QA(VQA) datasets that effectively combine video and text modalities to test comprehensive sports understanding are scarce. Sports-QA (Li et al., ) stands out by covering eight sports, including volleyball and basketball, with videos sourced from MultiSports (Li et al., ) and FineGym (Shao et al., ). However, although it provides detailed action recognition annotations, Sports-QA does not assess models on understanding sports rules, such as foul detection. This limitation arises because MultiSports does not include foul clips, and FineGym focuses solely on granular gymnastic actions, which do not encompass rule-based scenarios.

Foul detection requires models to recognize player actions and determine rule violations, making it a critical component of sports understanding. Another benchmark, SoccerNet-XFoul (Held et al., ), evaluates soccer understanding, including rule violation detection and explanation. However, its focus on a single sport limits its generalizability, as an MLLM’s performance in one sport may not extend to others with different rules and dynamics. To address the limitations of previous works, we introduce SPORTU, a comprehensive benchmark that spans multiple sports and incorporates both rule-based reasoning and foul detection across text and video modalities, offering a more diverse and in-depth evaluation of sports understanding.

## 3 Sports Understanding Benchmark

Figure: Figure 2: Examples of SPORTU-text (left) and SPORTU-video (right). Figure 1 also shows an example of an open-ended SPORTU-video question.
Refer to caption: x2.png

To address the limitations in the current sports-domain QA dataset and evaluations, we introduce SPORTU. This benchmark consists of two datasets:

SPORTU-text – the first pure-text-based sports QA dataset designed to evaluate models’ understanding of rules, factual knowledge, scenario-based situations, and strategies with human-annotated explanations. It contains 900 QA pairs, with each question having one or more correct answers.

SPORTU-video – the first slow-motion VQA dataset covering multiple sports, designed to evaluate MLLMs’ sports understanding abilities. It features a multilevel question design using slow-motion videos, with some clips offering multiple camera angles. The tasks range from easy-level tasks, such as sports recognition, to medium-level tasks, such as team role recognition, and hard-level tasks, such as rule violation explanations. The dataset contains 1,701 slow-motion video clips across 7 sports, with 12,048 QA pairs designed to test models’ multi-level video-text reasoning capabilities.

Overall, SPORTU provides a comprehensive evaluation of MLLMs’ ability to understand and apply sports knowledge. It fills the existing gap in current sports QA benchmarks by offering a detailed assessment across both text-based reasoning and multimodal video tasks.

To guarantee the accuracy of annotations in SPORTU, we employed a rigorous quality verification process. Our annotator team consisted of nine sports experts, including two intercollegiate athletes and seven players with over five years of training experience (the detailed annotation and verification process is described in Appendix A).

### 3.1 SPORTU-text: Pure Text QA

SPORTU-text is designed as the first pure-text-based sports QA dataset that provides detailed explanations for each question option, aiming to evaluate models’ understanding of rules, factual knowledge, scenario-based reasoning, and strategies. The dataset consists of 900 QA pairs, with each question having one or more correct answers and accompanied by human-annotated explanations to ensure a high-quality assessment of reasoning processes. SPORTU-text can also serve as a benchmark for model explainability, allowing researchers to compare models’ generated reasoning with human-provided explanations.

Dataset Construction
Among all sports-specific QA benchmarks, only SportQA (Xia et al., ) Level-3 questions assess models’ deep understanding of sports knowledge. We randomly selected 900 questions from five different sports: American football, basketball, soccer, volleyball, and tennis. Each question was carefully annotated, with detailed explanations provided for each option—whether correct or incorrect. Annotators explained why an option was correct or not, offering clear insights into the reasoning required for each answer. Examples can be found in the appendix E or in the actual SPORTU-text dataset.

### 3.2 SPORTU-video: Multimodal Video QA

SPORTU-video is the first multimodal video QA dataset designed to evaluate MLLMs’ sports understanding across various tasks, with a specific focus on integrating visual and textual reasoning. It is unique in its use of slow-motion video clips across multiple sports and includes a multilevel question design to test a range of model abilities, from simple recognition to complex rule-based reasoning.

SPORTU-video consists of 1,701 slow-motion video clips across 7 different sports, including soccer, basketball, volleyball, ice hockey, tennis, baseball, and badminton. We also ensured that for some sports, the videos featured multiple camera angles to challenge the models’ ability to capture consistent judgments across different perspectives. Specifically, 300 video scenes were selected to include multi-angle views. Each clip is accompanied by one or more QA pairs, for a total of 12,048 QA pairs (with 10,973 Multiple Choice Questions and 1,075 open-ended questions based on the explanations that the annotators labeled), with three levels of complexity: Easy: 25.36%, Medium: 50.22%, Hard: 24.42%.

Dataset Construction The construction of SPORTU-video began by identifying the types of tasks that could be asked based on the sports domain. Some questions, such as sports recognition and identifying the number of players, are common across all sports. Other questions are sports-specific, requiring knowledge of rules or strategies unique to each sport. The full question template can be found in the appendix D.

To classify the questions, we considered three different levels of difficulty based on the sports knowledge and reasoning required. Easy-level questions: These tasks rely on commonsense, such as basic sports recognition. For example, questions like ”What sport does the video show?” or ”How many players are shown?” do not require sports knowledge. Medium-level questions: These questions require sports knowledge beyond commonsense. For example, models are asked to identify which team is on offense based on the video or to recognize specific roles, such as a libero in volleyball, by identifying the libero’s jersey color. Hard-level questions: These tasks involve deep rule-based reasoning. For example, identifying rule violations or technical errors requires models to understand the sport’s rules in detail and apply them to the specific context of the video.

Once the question types were defined, the annotators collected the appropriate video clips and labeled the corresponding ground truths that the videos showed, as multiple questions could often be answered from a single video. These slow-motion clips were sourced from YouTube under the Creative Commons License, and each video was manually cropped to ensure high-quality footage suitable for detailed action analysis and rule comprehension. For each video, annotators carefully reviewed the content and annotated the answers for the designed questions. After the initial annotation, we used templates to generate both open-ended questions and multiple-choice questions based on the annotator-labeled ground truths. For multiple-choice questions, the ground truth answer was included as one of the options, while distractors were selected based on the question type. For example, distractors for jersey color questions were drawn from a set of plausible colors, while for ”what sport” questions, the distractors consisted of similar sports. In foul-related questions, the distractors included other types of fouls within the same sport. For open-ended questions, the annotators’ explanations were used directly as the ground truth.

Explanations One of the unique aspects of SPORTU-video is its emphasis on tasks involving foul detection and technical errors, where open-ended questions are accompanied by detailed human-annotated explanations. These explanations provide insights into why certain fouls or errors occurred, helping to evaluate models not only on their accuracy but also on their ability to explain the reasoning behind their answers. Due to limited annotation resources, these explanations are only provided for the most challenging tasks involving rule violations and technical errors, as these require models to combine action recognition with textual rule understanding to determine the violation. We believe that these tasks offer the most rigorous test of a model’s ability to connect sports knowledge with video comprehension.

## 4 Experiments

We compare MLLM’s performance on the SPORTU benchmark, We also evaluate the ability of models to produce explanations. We start by describing the MLLMs in our experiments and their Experimental Setup(§4.1), followed by evaluation metrics (§4.2).

### 4.1 Experimental Setup

SPORTU-text Evaluation: We evaluated several leading language models on the SPORTU-text, including open-source models like Llama-3.1-405B (Dubey et al., ), and closed-source models such as GPT-4o (2024-08-06 version) (OpenAI, ), Gemini-1.5 Pro(Gemini Team, ), and Claude-3.5-Sonnet (20240620 version) (Anthropic, ). Access to these models was facilitated through their respective APIs.

We primarily focused on the Chain-of-Thought (CoT) prompting method, based on evidence from Wei et al. (), which demonstrates the effectiveness of few-shot CoT in reasoning tasks, including sports understanding. In addition to CoT, we explored standard prompting (SP) for both zero-shot and few-shot settings (Brown, ) and zero-shot CoT (Kojima et al., ) as baselines. In the few-shot settings, five exemplars were provided for each task, with answers annotated by human experts. Exemplars were consistently sourced from the development set, with the temperature parameter set to 0 to ensure consistent response generation.

SPORTU-video Evaluation: We investigate a range of MLLMs, including 7 close source models and 6 open source models. For close source models, we evaluate GPT-4o (2024-08-06 version) and -4omini (OpenAI, ), GPT-4V (OpenAI, ), Gemini 1.5 Pro (Gemini Team, ), Gemini 1.5 Flash (Gemini Team, ) and Claude-3.5-Sonnet (20240620 version) (Anthropic, ) and-3.0-Haiku Anthropic ().For Open source models, we evaluate ChatUniVi (Jin et al., ), LLaVA-NeXT (Liu et al., ), mPLUG-Owl3 (Ye et al., ), Tarsier (Wang et al., ),
Video-ChatGPT (Maaz et al., ), and VideoChat2(Li et al., ). For the closed-source models, we adhered to the default settings provided by their official APIs. GPT and Claude family models processed ten image frames extracted from the video content as input. The Gemini family models processed the entire video, as their API supports video input. Due to computing resource limitations, we used the 7b versions of all open-source models in this evaluation. For VideoChat, we set ‘max_frames’ to 100, while for the other open-source models, we used 16 frames as input. Across all closed and open-source models, we set the temperature parameter to 0 to ensure consistent response generation.

Models were prompted to generate the answer first, followed by an explanation for why the model chose that answer. This method has been shown to be effective in generating interpretable results in reasoning tasks (Chen et al., ), and it also facilitates human error analysis. More details regarding the specific prompts used for both SPORTU-text and SPORTU-video can be found in Appendix B. All inferences are run on an NVIDIA RTX A6000.

### 4.2 Evaluation Metrics

We use accuracy (prediction compared to ground truth) to evaluate the predictions of each model in multiple-choice tasks. We explore several methods to evaluate different aspects of model-generated explanations and open-ended questions: ROUGE-L (Lin, ), BERTScore (Zhang et al., ), BLEURT (Sellam et al., ), CTC-Preservation (Deng et al., ), and G-Eval Score (Liu et al., ). ROUGE-L computes the surface-form similarity between model-generated explanations and
reference (human-annotated) explanations. BERTScore and BLEURT measure semantic similarity using pre-trained BERT (Devlin, ) and fine-tuned BERT models respectively. CTC metrics evaluate the information alignment of model-generated explanations. G-Eval is a framework that uses large language models with chain-of-thought reasoning and a form-filling paradigm to assess the quality of NLG outputs. We apply this framework to evaluate the accuracy, conciseness, and relevance of the model-generated explanations compared to the ground truth, reporting these as an overall score. We use GPT-4o to implement the G-Eval process. The detailed prompt can be found in the Appendix B. To ensure the reliability of the G-Eval scores, we conducted a human evaluation on a randomly selected set of 20 questions per sport across 7 sports, totaling 140 questions. This subset was used to compare the G-Eval scores with human-annotated scores across 13 models. By using the same set of questions for all models, we ensured consistency in comparison, allowing us to assess whether the G-Eval scores were in line with human judgment. The human annotators rated the model-generated content using the same criteria as G-Eval, providing a reference point for how well the automated evaluation aligns with human evaluation.

## 5 Results

We compared the performance of different models across both SPORTU-text and SPORTU-video.

**Table 2: Performance of LLMs with Standard Prompt Settings**
| Setting | Model | Acc.(%) |
| --- | --- | --- |
| Zero-Shot(SP) | Claude-3.5-Sonnet | 64.33 |
| gemini-1.5 Pro | 63.33 |  |
| GPT-4o | 70.22 |  |
| Llama3.1-405B | 66.67 |  |
| Five-Shot(SP) | Claude-3.5-Sonnet | 68.44 |
| gemini-1.5 Pro | 64.11 |  |
| GPT-4o | 70.78 |  |
| Llama3.1-405B | 66.33 |  |

The overall results for SPORTU-text are shown in Table 2 and Table 3 , while the multiple-choice QA results for SPORTU-video are presented in Table 4, and the open-ended QA results are displayed in Table 5. Among all models, GPT-4o performs the best in SPORTU-text, achieving the highest accuracy of 71% in the five-shot CoT prompt setting, along with a G-Eval score of 4.16 in the zero-shot CoT prompt setting. The G-Eval score confirms that GPT-4o can somewhat grasp sports-related rules, strategies, and scenario-based questions. However, there remains a notable gap compared to expert performance, which exceeds 90%, as mentioned in Xia et al. ().

Table: Table 3: Performance of LLMs on SPORTU-text evaluation across CoT settings, including accuracy for both standard and CoT prompting strategies. Explanation quality metrics (ROUGE-L, BERTScore, BLEURT, CTC Preservation, and G-Eval) are provided for CoT settings generated-explanation assessment.)

Table: Table 4: Overall performance of MLLMs on SPORTU-video for multiple-choice questions across three difficulty levels: easy, medium, and hard. The table shows a clear trend of decreasing accuracy as task difficulty increases, with both closed-source and open-source models demonstrating significant drops in performance on harder questions.

For the SPORTU-video multiple-choice task, Claude-3.5-Sonnet achieved the highest overall accuracy at 69.52%. However, performance dropped significantly from easy to hard questions, with only 52.57% accuracy on the hard problems. Among the open-source models, LLAVA-NeXT outperformed others, including some closed-source models. While it excelled on easy and medium-level questions, its performance on hard questions was only 33.44%, indicating a lack of domain knowledge, particularly in rule comprehension. For open-ended questions, GPT-4o again led with a G-Eval score of 1.84, a result further validated by human annotators. Even with human ratings, the score remained close to G-Eval’s assessment, with the model not exceeding a score of 3. A score of 1 indicates very poor performance, and 2 suggests poor performance based on the evaluation criteria. This highlights the model’s struggle to connect observed actions with relevant domain knowledge, such as identifying technical errors or specific rules. Overall, none of the MLLMs achieved an average score above 3, demonstrating a gap in deep domain knowledge required for video sports understanding. Among the evaluated metrics, G-Eval shows the closest alignment with human ratings, with a Pearson correlation coefficient of 0.41. However, this correlation is still relatively low, suggesting the need for a domain-specific metric for evaluating sports content. Additionally, we notice that the performance of models on multi-angle videos shows variability depending on the camera perspectives for the same scene, indicating that models struggle with consistent understanding across different angles. More details can be found in the appendix B.

Table: Table 5: Model performance on SPORTU-video open-ended tasks, evaluated using various metrics for generated-explanation quality assessment. The table also includes human ratings (denoted with a *), which were conducted to verify the reliability of the G-Eval scores as mentioned in 4.2.

### 5.1 Error Analysis

To gain deeper insights into the limitations of MLLMs, we applied the X→YR prompting method, where models first generated an answer and then explained their reasoning. This provided valuable information about how models approached reasoning, especially in complex tasks like foul detection. We analyzed errors across both open-ended and multiple-choice tasks, selecting 20 incorrect examples per sport for each model, resulting in a total of 3640 errors.

Figure 3 shows the radar chart representing the distribution of different error types. The most frequent error was Question Understanding Error, particularly in questions like “Why is it a foul in the video?” where models incorrectly responded that there was no foul despite the question’s clear presupposition. Another common issue was Hallucination Error, such as when a model mentioned a referee in its explanation, even though no referee was visible in the video. We also observed that the appearance of Question Understanding Error is much less frequent in multiple-choice questions. When presented with four options, the model tends to select one of the
provided answers rather than completely misinterpreting the premise of the question.

Detailed examples of these errors and further case studies are provided in Appendix C.

Figure: Figure 3: Error type distribution across different MLLMs on SPORTU-video tasks. The analysis reveals that Question Understanding Error is the most common issue, followed by Hallucination Error. Each error type highlights specific model limitations in comprehending the task.
Refer to caption: x3.png

## 6 Conclusion

In this paper, we introduced SPORTU, a comprehensive benchmark designed to evaluate the sports understanding capabilities of Multimodal Large Language Models (MLLMs). SPORTU consists of two key components: a text-based section, SPORTU-text, which focuses on evaluating models’ ability to comprehend rules, strategies, and scenario-based reasoning through multiple-choice questions, and a video-based section, SPORTU-video, which challenges models to apply these insights to real-world sports footage, including tasks like sports recognition, foul detection, and rule application. By combining both text-based and video-based tasks, SPORTU provides a holistic assessment of models’ reasoning abilities across multiple levels, from basic recognition tasks to complex scenario-based reasoning. Our evaluation results indicate that while leading models such as GPT-4o have made progress in understanding fundamental rules and strategies through the pure text task, they still struggle with deeper, scenario-based reasoning that requires connecting visual actions with domain-specific knowledge. Additionally, we observed that multi-angle video inputs introduce further challenges, as model performance varies significantly across different perspectives of the same scene, revealing limitations in their ability to maintain consistent understanding. The error analysis further highlights key challenges such as question misunderstanding and hallucination, revealing areas where future models need to improve. We hope SPORTU will serve as a critical resource to push the boundaries of multimodal understanding in real-world sports environments.

## Appendix A Dataset Quality Control

To guarantee the accuracy and consistency of annotations and question generation in SPORTU, we employed a rigorous quality verification process. Our team of annotators consisted of nine experts: two were intercollegiate student-athletes with over 12 years of experience, and seven were players who had undergone at least 5 years of training in their respective sports. It helped maintain the high standard of explanations and annotations for both SPORTU-text and SPORTU-video.

During the training phase, each team member worked with twenty examples per batch for both text-based questions and VQA tasks. They were asked to: Annotate explanations for the SPORTU-text dataset.
Collect videos for SPORTU-video and annotate the key information necessary for generating questions.
These key annotated variables were later used in a template-based system to generate questions. Once each annotator demonstrated full mastery of the annotation process, we officially launched the large-scale annotation phase.

As part of our verification protocol, annotators were required to double-review the videos they collected to ensure accuracy and quality. We prioritized the removal of controversial or hard-to-interpret clips, especially those where even human experts might disagree or feel unsure about the decision, to minimize the risk of mislabeling.

## Appendix B Supplementary Experiments and Results

In this section, we provide supplementary experiments and results of the SPORTU-video benchmark, covering model performance across multiple camera angles (Section B.1), accuracy across different sports and difficulty levels (Section B.2
), criteria for evaluating model-generated explanations (Section B.3), and the correlation between automatic evaluation metrics, human-annotated scores (Section B.4), and example of prompt in Section B.5

### B.1 Model Performance of Multi-Angle Videos

Table: Table 6: Accuracy comparison across multiple camera angles of the same question

In this section, we analyze the performance of models when answering the same question from different camera angles within the same scene. The objective is to evaluate how consistently models understand the same scenario when presented from multiple perspectives. For example, if the question is “What color is the person’s jersey?”, the same question is asked across different camera angles for the same scene. The results are categorized into three distinct cases:

- •
P(All Correct): This indicates the percentage of instances where the model answered correctly for all camera angles of the same scene. A higher percentage reflects the model’s ability to consistently interpret and understand the scene across multiple perspectives.
- •
P(All False): This shows the percentage of instances where the model answered incorrectly for all angles of the same scene, highlighting consistent misunderstanding across different perspectives.
- •
P(At least One Correct& At least One False): This category reflects situations where the model answered correctly for at least one camera angle but incorrectly for at least one other, indicating that the model’s understanding varies depending on the perspective.

The analysis reveals that camera angle variation can significantly affect the model’s performance. While some models are more consistent across angles, others show a noticeable drop in accuracy when the angle changes, suggesting challenges in maintaining robust understanding across multiple viewpoints. This highlights a critical area for further research and improvement in multimodal models when handling multi-angle video inputs

### B.2 Results Across Different Levels of Difficulty and Sports for Each Model

Table 7 shows the accuracy of 13 models across different sports and difficulty levels.

Table: Table 7: Model accuracy across different sports and difficulties.

### B.3 Criteira of G-Eval

This score assesses the overall accuracy, conciseness, and relevance of the model-generated explanation to the ground truth explanation. The explanation should focus on identifying and clearly explaining the key issue or relevant action.

Scoring Breakdown:

- •
Score 1 (very poor):
The explanation is incorrect or does not address the key issue at all. It might be filled with irrelevant details that distract from the main point.
- •
Score 2 (poor):
The explanation contains some correct elements but fails to clearly identify the key issue. There may be excessive irrelevant details that obscure the main point.
- •
Score 3 (adequate):
The explanation identifies the key issue but includes too much unnecessary information or lacks clarity. The key point is present but could be more concise.
- •
Score 4 (good):
The explanation clearly identifies the key issue with some minor unnecessary details. The core explanation is correct and relevant but may include a few extra, non-essential details.
- •
Score 5 (excellent):
The explanation is concise, accurate, and directly addresses the key issue without any unnecessary information. It clearly and effectively answers the question.

Evaluation Steps:

- 1.
Read the ground truth explanation.
- 2.
Identify the specific context in the ground truth explanation that represents the key issue (e.g., a foul, a mistake).
- 3.
Read the model output explanation.
- 4.
Compare the model’s explanation to the ground truth explanation, focusing on how directly and concisely the model output identifies the key issue.
- 5.
Assign a score for explanation of overall relevance on a scale of 1 to 5, based on the criteria.

Model Generated Explanation: {{Model Generated Content}}

Ground Truth Explanation: {{Ground Truth Content}}

Evaluation Form (scores ONLY):
Overall:

### B.4 Correlation Matrix of Evaluation Metrics

We applied Pearson correlation to assess the relationship between the automatic evaluation metrics and human-annotated scores. As shown in Figure 4, G-eval has the highest correlation with the human scores among all the automatic metrics. However, the correlation is still relatively weak, indicating the need for the development of a new evaluation metric in the future to improve the assessment process

Figure: Figure 4: Correlation matrix of different evaluation metrics in SPORTU-video
Refer to caption: x4.png

### B.5 Prompt Templates

Detailed templates for zero- and five-shot standard prompts, as well as zero- and five-shot CoT (Chain of Thought) prompts in the SPORTU-text task, are shown in table 8 - table 11. The template prompt for the SPORTU-video multiple-choice task is shown in table 12, and for the SPORTU-video open-ended task, it is shown in Table 13.

**Table 8: Zero-Shot Standard Prompt for SPORTU-text**
| {role: “system”, |
| --- |
| content: “You are a helpful assistant who is good at answering multiple-choice sports questions.” } |
| {role: “user”, |
| content: “**Question:** {{question}}. You are a sports expert who answers the above question. What are the correct answers to this question?” } |

**Table 9: Zero-Shot CoT for SPORTU-text**
| {role: “system’”, |
| --- |
| content: “You are a helpful assistant who is good at answering multiple-choice sports questions.”} |
| {role: “user”, |
| content: “**Question:** {{question}}. You are a sport expert who answers the above question. What are the correct answers for this question? Think step by step, and please explain why correct answers are correct while the rest options are incorrect. You should explain each option in detail. Add one more sentence after your detailed explanation at the end to show the correct answers’ option letters only with A, B, C, D. The correct answers: ” } |

**Table 10: Five-Shot Standard Prompt for SPORTU-text**
| {role: “system”, |
| --- |
| content: “You are a helpful assistant who is good at answering multiple-choice sports questions. Some examples are given below. {{five-shot examples}} ”} |
| {role: “user”, |
| content: “**Question:** {{question}}. You are a sports expert who answers the above question. What are the correct answers to this question?”} |

**Table 11: Five-Shot CoT for SPORTU-text**
| {role: “system”, |
| --- |
| content: “You are a helpful assistant who is good at answering multiple-choice sports questions. Some examples are given below. {{five-shot examples with human-annotated explanations }}” } |
| {role: ‘user’, |
| content: “**Question:** {{question}}. You are a sports expert who answers the above question. What are the correct answers to this question?” } |

**Table 12: Zero-Shot Standard Prompt for SPORTU-video Multiple Choice Questions**
| {role: “system”, |
| --- |
| content: “You are a sports expert analyzing a series of video frames that form a continuous short video clip. Your task is to answer questions based on the content of the video.”} |
| {role: “user”, |
| content: “**Question:** {{question}}. Answer this sports-related question based on the video{{frames}}, Respond with the letter of the correct option (A, B, C, or D), formatted as ‘The correct answer is: ’, and explain why you chose that option.”} |

**Table 13: Zero-Shot Standard Prompt for SPORTU-video Open-Ended Questions**
| {role: “system”, |
| --- |
| content: “You are a sports expert analyzing a series of video frames that form a continuous short video clip. Your task is to answer questions based on the content of the video.”} |
| {role: “user”, |
| content: “**Question:** {{question}}. Answer this sports-related question based on the video{{frames}}. Please provide a detailed explanation focusing on the sports aspects of the video.” } |

## Appendix C Additional Error Analysis

This section presents the error analysis of the models we evaluated on SPORTU-video. For the error types, we evaluated the models with coarse granularity, dividing the errors into five categories as follows:

- •
Question Understanding Error – The model misinterprets the intent or context of the question, providing an answer that does not align with what the question is asking.
- •
Visual Perception Error – The model incorrectly interprets the visual content, leading to faulty assumptions about the data presented in the video.
- •
Hallucinations – The model generates content or details that do not exist in the actual data, essentially ’hallucinating’ information.
- •
Reasoning Error – The model exhibits poor logical reasoning, resulting in incorrect conclusions based on the available data.
- •
Lack of Domain Knowledge – The model fails to answer questions that require specific domain expertise, revealing a gap in its knowledge.

### C.1 Error Analysis for SPORT-video

Figure: Figure 5: Error type distribution across different MLLMs on SPORTU-video tasks: the left side represents errors from multiple-choice questions, while the right side represents errors from open-ended questions.
Refer to caption: x5.png

We observe that open-ended questions have the highest frequency of question understanding errors. For instance, when asked ’Why is it a foul in the video?’, the model might respond that there is no foul, which is a question understanding error because the phrasing of the question already implies that a foul occurred. This issue is much less frequent in multiple-choice questions (as shown on the left side of the figure 5). When presented with four options, the model tends to select one of the provided answers, rather than completely misinterpreting the premise of the question.

### C.2 Examples of Each Error Type

Figure: Figure 6: Examples of Each Error Type
Refer to caption: extracted/5939216/7.jpg

Figure: Figure 7: Examples of Each Error Type
Refer to caption: extracted/5939216/8.jpg

Figure: Figure 8: Examples of Each Error Type
Refer to caption: extracted/5939216/9.jpg

Figure: Figure 9: Examples of Each Error Type
Refer to caption: extracted/5939216/10.jpg

Figure: Figure 10: Examples of Each Error Type
Refer to caption: extracted/5939216/11.jpg

## Appendix D Questions for Each Sport

This section shows the questions we use for each sport. (Table 14 - Table 20).

**Table 14: Question Template for Volleyball**
| What sport does this video show? |
| --- |
| What kind of fault or foul does this video show? |
| What main color jersey is the libero wearing? |
| If the libero’s jersey color does not count, what main color jerseys are the players on the right side wearing? |
| If the libero’s jersey color does not count, what main color jerseys are the players on the left side wearing? |
| If the libero’s jersey color does not count, what main color jerseys are the players on the far side wearing? |
| If the libero’s jersey color does not count, what main color jerseys are the players on the close side wearing? |
| What main color jersey is the libero on the close side wearing? |
| What main color jersey is the libero on the far side wearing? |
| What main color jersey is the libero on the left side wearing? |
| What main color jersey is the libero on the right side wearing? |
| Is there a rule violation in the video? |

**Table 15: Question Template for Basketball**
| What sport does this video show? |
| --- |
| What specific type of foul, if any, occurred in the video? Choose the most appropriate one. |
| What main color jersey is the offensive team wearing? |
| What main color jersey is the defensive team wearing? |
| What main color jersey does the player who committed the foul wear? |
| What main color jersey is the player who was fouled wearing? |
| What main jersey colors do the two teams wear in this video? |
| Is there a rule violation in the video? |
| Why is it a foul in the video? |

**Table 16: Question Template for Badminton**
| Why is it a rule violation in the video? |
| --- |
| How is the player making a technical error in the video? |
| What sport does this video show? |
| What kind of rule violation is in the video? |
| What main color jersey is the left side team wearing? |
| What main color jersey is the right side team wearing? |
| What main color jersey is the close side team wearing? |
| What main color jersey is the far side team wearing? |
| What specific action does the player perform in the video? |
| How many players are shown in total in this video? |
| Is the player making a technical error in the video? |
| Is there a rule violation in the video? |

**Table 17: Question Template for Baseball**
| What sport does this video show? |
| --- |
| What kind of rule violation is in the video? |
| Based on this video, which of the following descriptions best applies to the situation? |
| As a referee, what procedure would you follow in this situation in the video? |
| What main color jersey is the fielder team wearing? |
| What main color jersey is the batting team wearing? |
| What main jersey colors do the two teams wear in this video? |
| Is there a rule violation in the video? |
| Why did the rule violation occur in the video? |

**Table 18: Question Template for Soccer**
| Why is it a foul in the video? |
| --- |
| Why is it not a foul in the video? |
| Based on this video, which of the following descriptions apply to the situation that occurred? |
| What sport does this video show? |
| What kind of foul does this video show? |
| What main color jersey is the offensive team wearing? |
| What main color jersey is the defensive team wearing? |
| What main color jersey does the player who committed the foul wear? |
| What main color jersey is the player who was fouled wearing? |
| What main color jersey does the goalkeeper wear? |
| What main jersey colors do the two teams wear in this video? |
| How many players are shown in total in this video? |
| Is there a rule violation in the video? |

**Table 19: Question Template for Ice Hockey**
| What sport does this video show? |
| --- |
| What kind of foul is committed in the video? |
| What main color jersey does the player who committed the foul wear? |
| What main color jersey is the player who was fouled wearing? |
| What main jersey colors do the two teams wear in this video? |
| Is there a rule violation in the video? |
| If we consider the fight in the video to be a legit fight, defined as a fight between two willing participants who drop their gloves and helmets, with the fight ending when one player falls or officials intervene, and this fight does not result in a penalty, is there any other foul in the video that will cause a penalty? |
| Why is it a foul in the video? |

**Table 20: Question Template for American Football**
| Why is it an error in the video? |
| --- |
| Why is it a foul in the video? |
| What sport does this video show? |
| What kind of foul does this video show? |
| What kind of error does the player make in this video? |
| What main color jersey is the offensive team wearing? |
| What main color jersey is the defensive team wearing? |
| What main jersey colors do the two teams wear in this video? |
| Is there a foul in this video? |
| Does any player make an error in this video? |

## Appendix E Sample Questions

### E.1 SPORTU-text Examples

This section shows examples of our SPORTU-text. (Table 21 - Table 26)

**Table 21: Sample Question 1**
| Question: Which of the following scenarios would most likely result in a red card offense in a football match? |
| --- |
| A) A player politely disagrees with the referee’s decision. |
| B) A player accidentally trips another player while trying to get the ball. |
| C) A player uses offensive language or gestures towards the referee. |
| D) A player passes the ball back to his own goalkeeper. |
| Answer: C |
| Explanation: Disagreeing with the referee’s decision, as long as it is done reasonably and without abuse, will not result in a red card, therefore option A is incorrect. Accidentally tripping a player will not result in a red card because it is not intentional, therefore option B is incorrect. Using offensive language or gestures violates sportsmanship and will result in a red card, therefore option C is correct. Passing the ball back to the goalkeeper does not result in a red card; it will result in an indirect free kick, therefore option D is incorrect. Hence, the correct answer is option C. |

**Table 22: Sample Question 2**
| Question: How is a penalty awarded in football? |
| --- |
| A) When a player intentionally handles the ball. |
| B) When a player scores a goal. |
| C) When a player passes the ball to another player. |
| D) When a player runs with the ball towards the goal. |
| Answer: A |
| Explanation: Handling the ball in the penalty area may result in a penalty kick, therefore option A is correct. Scoring a goal does not result in a penalty kick, therefore option B is incorrect. Passing the ball does not result in a penalty kick, therefore option C is incorrect. Dribbling towards the goal does not result in a penalty kick, therefore option D is incorrect. Hence, the correct answer is option A. |

**Table 23: Sample Question 3**
| Question: Why might a team be penalized with a loss of 5 yards in American football? |
| --- |
| A) For a false start offense. |
| B) For scoring a touchdown. |
| C) For completing a pass. |
| D) For calling a timeout. |
| Answer: A |
| Explanation: A team is penalized with a loss of 5 yards in American football for a false start offense, therefore option A is correct. Option B is incorrect because it isn’t a stated rule in American football. Option C is incorrect because it isn’t a stated rule in American football. Option D is incorrect because it isn’t a stated rule in American football. Hence, the correct answer is option A. |

**Table 24: Sample Question 4**
| Question: Which of the following scores is considered a ”deuce” in a tennis match? |
| --- |
| A) 30-30 |
| B) 40-40 |
| C) 15-15 |
| D) 0-0 |
| Answer: B |
| Explanation: “Deuce” refers to a score of 40-40 in tennis. 30-30 is not a deuce, therefore option A is incorrect. 40-40 is a deuce, therefore option B is correct. 15-15 is not a deuce, therefore option C is incorrect. 0-0 is not a deuce, therefore option D is incorrect. Hence, the correct answer to this question is B. |

**Table 25: Sample Question 5**
| Question: How many sets does a player need to win for a Grand Slam men’s singles tennis match to be declared the winner? |
| --- |
| A) 1 set |
| B) 2 sets |
| C) 3 sets |
| D) 4 sets |
| Answer: C |
| Explanation: Grand Slam men’s singles matches are best of five sets, meaning the first player to win three sets wins the match. Winning one set is not enough to win the match, therefore option A is incorrect. Winning two sets is also not enough to win the match, therefore option B is incorrect. The first player to win three sets wins the match, therefore option C is correct. Winning four sets is not possible since winning three sets is the maximum in a best-of-five format. Therefore, option D is incorrect. Hence, the correct answer is C. |

**Table 26: Sample Question 6**
| Question: How can the player holding the ball be considered out of bounds? |
| --- |
| A) If the player’s foot is touching the boundary line |
| B) If the player is holding the ball while standing outside the boundary line |
| C) If the player jumps from outside the boundary line and catches the ball while in the air |
| D) All of the above |
| Answer: D |
| Explanation: A player is out of bounds if their foot touches the boundary line, but this is not the only factor determining if a player is out of bounds. Therefore, option A is incorrect. Standing outside the boundary line also is considered out of bounds, but again is not the only determining factor. Therefore, option B is incorrect. Catching the ball in the air while jumping from outside the boundary line also counts as out of bounds, but this is not the only determining factor. A player holding the ball is considered out of bounds if their foot touches the boundary line, if they are standing outside the boundary line while holding the ball, or if they jump from outside the boundary line and catch the ball while in the air, so all of the above answers include the factors that determine if a player is considered out of bounds. Therefore, option D is correct. Hence, the correct answer is D. |

### E.2 SROUTU-video Examples

This section shows examples of our SPORTU-video. (Figure 11 - Figure16)

Figure: Figure 11: Examples of SPORTU-video
Refer to caption: extracted/5939216/1.jpg

Figure: Figure 12: Examples of SPORTU-video
Refer to caption: extracted/5939216/2.jpg

Figure: Figure 13: Examples of SPORTU-video
Refer to caption: extracted/5939216/3.jpg

Figure: Figure 14: Examples of SPORTU-video
Refer to caption: extracted/5939216/4.jpg

Figure: Figure 15: Examples of SPORTU-video
Refer to caption: extracted/5939216/5.jpg

Figure: Figure 16: Examples of SPORTU-video
Refer to caption: extracted/5939216/6.jpg

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please check weakness for details.
soundness: 1
strengths: 1. The proposed dataset could be useful for the community.
2. Both close and open-sourced models are evaluated.
3. Metrics are studied with human verification.
summary: This paper introduces SPORTU, a new benchmark designed to evaluate the capabilities of Multimodal Large Language Models (MLLMs) in sports understanding and reasoning. SPORTU consists of two components: SPORTU-text, focusing on text-based reasoning, and SPORTU-video, focusing on video-based reasoning. The authors evaluate various LLMs on both components, revealing limitations in complex reasoning tasks.
weaknesses: 1. The reviewer is concerned about the lack of diversity and coverage of the dataset because of the limited prompt templates and number of samples. 

2. Implementation could be possibly flawed. 
- The error in Figure 6 looks suspicious and makes the reviewer wonder whether the model is called correctly or not. 
- The reasoning prompt asks the model to first generate answer and then reasoning, which is not optimal since the model's final answer cannot benefit from the reasoning process.
- It is known that LLM usually prefers its own answer so it is important to understand G-eval' quality with different LLMs as the rater.

Minor:
L821 typos of "Section ??"

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: - The video quality for the datasets
- For each sports type, the video are biased to certain views or events?
soundness: 2
strengths: - A multimodal new dataset for sports domain (with multiple sports) and well annotated by experts; the dataset should be helpful for the research communities

- A well prompting capabilities to show the limitation of current LLM capabilities on the dataset. 

- Evaluating several reasonable public or private LLM models.
summary: The paper provides a multimodal dataset (text and slow-motion video) for evaluating (multimodal) LLM capabilities in the sports domain.
weaknesses: - Yet another vertical dataset for LLM
- It's helpful but marginal to expand the technical depth for the community
- not clearly identified what current models failed.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. How do you split the SPORTU-text questions into rules-related, strategy-related, and scenario-related? What is your basis?

2. What are the results on rule/strategy/scenario, respectively, on sportu-text?

3. How is the error analysis in 5.1 conducted? Is there a definition for each error type?
soundness: 4
strengths: 1. It proposes SPORTU-text and SPORTU-video to boost understanding more sports with rules understanding in text and video domains.
2. It analyzes the views, reasoning prompts, sport types, the error types, which are comprehensive.
3. The writing is clear.
summary: In the AI+Sports area, existing works are limited to restricted kinds of sports, absence of explanations, or lack of reasoning on rules, and it proposes SPORTU consisting of SPORTU-text and SPORTU-video to boost understanding more sports with rules understanding. SPORTU-text evaluates models on rule comprehension and strategy understanding in the pure text domain and SPORTU-video evaluates models on understanding both video details and rules in the video domain.
It evaluates LLMs and MLLMs on SPORTU-text and SPORTU-video, revealing their limitations in complicated sports questions.
weaknesses: 1. A benchmark aims to evaluate certain abilities and give some insights. The paper does not deeply discuss why the models have different performances and does not give advice on how to resolve the problem of understanding videos and reasoning on rules.

2. Prompt strategy in LLM can also be tested on MLLM when evaluating on SPORTU video benchmark to see how the reasoning process influences MLLM.

3. It's not very clear if the questions in this dataset can comprehensively detect the models' abilities to understand sports.

4. The Pearson correlation between humans and the other metrics is low. Many are near 0.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the weakness part.
soundness: 2
strengths: 1. As a sport domain understanding benchmark, the proposed  SPORTU combines text-based and video-based tasks to assess models' sports reasoning and knowledge application abilities.

2. The evaluation setting is comprehensive including the direct prompting, chain-of-thought (CoT) prompting. In addition, few-shot promoting is also applied in SPORT-text evaluation.
summary: This paper prensents SPORTU, a comprehensive Sports Understanding Benchmark that integrates both text-based and video-based tasks to evaluate models’ sports reasoning and knowledge application capabilities. Based on this benchmark, this paper tests the capability of existing open-source or close-source  models.
weaknesses: 1. Unclear motivation. The authors should clarify the differences between the proposed SPORTU and existing sport domain understanding benchmarks. Although discussions have been made in introduction and related work section together with Table 1, it is still unclear why the introduced features , for example, slow motion, multi-camera angles are important. More discussions and visualizations are needed.

2. Missing details in dataset construction. There exist some unclear details in the dataset construction. For example, how to guarantee the multi-camera setting? Is it achieved simply by human annotator check? In addiation, the proposed SPORTU contains both the multi-choice and open-ended question, how are these two categories divided?

3. More advanced evaluation methods should be applied. For example, ST-LLM [1], qwen-vl [2]

4. The paper writing should be polished. Some references are missing, for example "Section ??" in Line 821. The quotation mark error in '”Why is it a foul in the video?”' in Linee 482.

[1] ST-LLM: Large Language Models Are Effective Temporal Learners
[2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

Meta-review: Summary: This paper introduced a benchmark to comprehensively evaluate the capabilities of MLLMs, which include multi-level sports reasoning tasks. Multiple baselines are considered for comprehensive evaluations.

Strengths: (1) The proposed benchmark could be helpful for further experiments for evaluation. (2) The evaluations are comprehensive. (3) The contribution to sports-related action understanding and reasoning area is clear.

The main remaining concerns (including the ones that reviewers did not reply to confirm with) are (1) the constructed dataset is not significantly different from previous methods, (2) the lack of diversity and coverage of the dataset, (3) implementation on reasoning prompt and rater selection could be possibly flawed.

This paper received diverse ratings, i.e., 6, 6, 5, 5. The AC checked the authors' responses, reviewers' comments, and the revised version. For the remaining concern (1), the authors claimed that "our dataset is the first to test MLLMs' sports understanding ability, particularly by combining action recognition with rule-based reasoning across different sports and varying difficulty levels", which AC agreed with. The novelty on sports-related QA with rule-based reasoning is clear and significant, which would be a good benchmark for sports-related action understanding and reasoning. For concern (2), the contributions to sports-related area are significant, and the goal is not a general benchmark. For concern (3), the authors provided results with reasoning prompt and rater selection. Also, for other concerns, the authors provided extensive results, which are comprehensive. Therefore, the AC think the major concerns have been addressed and thus recommend the paper as accept.

Decision: Accept (Poster)