SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports
Authors: Haotian Xia
Haonan Ge
Junbo Zou
Hyun Woo Choi
Xuebin Zhang
Danny Suradja
Botao Rui
Ethan Tran
Wendy Jin
Zhen Ye
Xiyang Lin
Christopher Lai
Shengjie Zhang
Junwen Miao
Shichao Chen
Rhys Tracy
Vicente Ordonez
Weining Shen
Hanjie Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models
Sports Understanding
Benchmark
Primary area: datasets and benchmarks

Abstract: Artificial Intelligence brings powerful new tools to sports, from automated officiating to tactical analysis, but these applications all depend on a core reasoning capability. 
Deeply understanding sports requires an intricate blend of fine-grained visual perception and rule-based reasoning—a challenge that pushes the limits of current multimodal models. 
To succeed, models must master three critical capabilities: perceiving nuanced visual details, applying abstract sport rule knowledge, and grounding that knowledge in specific visual evidence.
Current sports benchmarks either cover single sports or lack the detailed reasoning chains and precise visual grounding needed to robustly evaluate these core capabilities in a multi-sport context. 
To address this gap, we introduce SportR, the first multi-sports large-scale benchmark designed to train and evaluate MLLMs on the fundamental reasoning required for sports intelligence. 
Our benchmark provides a dataset of 4,789 images and 2,052 videos. To enable granular evaluation, we structure our benchmark around a progressive hierarchy of question-answer (QA) pairs designed to probe reasoning at increasing depths—from simple infraction identification to complex penalty prediction. 
For the most advanced tasks requiring multi-step reasoning, such as determining penalties or explaining tactics, we provide 6,841 high-quality, human-authored Chain-of-Thought (CoT) annotations.
In addition, our benchmark incorporates both image and video modalities and provides manual bounding box annotations to test visual grounding in the image part directly. Extensive experiments demonstrate the profound difficulty of our benchmark. State-of-the-art baseline models perform poorly on our most challenging tasks. While training on our data via Supervised Fine-Tuning and Reinforcement Learning improves these scores, they remain relatively low, highlighting a significant gap in current model capabilities. SportR presents a new challenge for the community, providing a critical resource to drive future research in multimodal sports reasoning. The dataset is available at https://github.com/chili-lab/SportR.

Published as a conference paper at ICLR 2026
SPORTR: A BENCHMARK FOR MULTIMODAL LARGE LANGUAGE MODEL REASONING IN SPORTS
Haotian Xia 1- Haonan Ge Junbo Zou Hyun Woo Choi Xuebin Zhang Danny Suradja' Botao Rui Ethan Tran Wendy Jin' Zhen Ye Xiyang Lin"
Christopher Lai Shengjie Zhang Junwen Miao Shichao Chen Rhys Tracy
Vicente Ordonez! 1.2 Weining Shen Hanjie Chen
'Department of Computer Science, Rice University
2 Ken Kennedy Institute, Rice University
3 Department of Statistics, University of California, Irvine
4 *College of Sciences, Georgia Institute of Technology
5 Department of Applied Mathematics and Statistics, Johns Hopkins University 6 Department of Computer Science, University of California, Santa Barbara [hx50, hanjie | @ rice.edu, weinings @ uci.edu
ABSTRACT
Artificial Intelligence brings powerful new tools to sports, from automated offi- ciating to tactical analysis, but these applications all depend on a core reasoning capability. Deeply understanding sports requires an intricate blend of fine-grained visual perception and rule-based reasoning a challenge that pushes the limits of current multimodal models. To succeed, models must master three critical ca- pabilities: perceiving nuanced visual details, applying abstract sport rule knowl- edge, and grounding that knowledge in specific visual evidence. Current sports benchmarks either cover single sports or lack the detailed reasoning chains and precise visual grounding needed robustly evaluate these capabilities in a multi-sport context. To address this gap, we introduce SportR, the first multi- sports large-scan scale benchmark designed to train and evaluate MLLMs the fun- damental reasoning required for sports intelligence. Our benchmark provides a dataset of 4,789 images and 2,052 videos. To enable granular evaluation. we structure our benchmark around a progressive hierarchy of question-answer (QA) pairs designed to probe reasoning at increasing depths-froed from simple infraction identification to complex penalty prediction. For the most advanced tasks re- quiring multi-step reasoning, such determining penalties or explaining tacties, we provide 6,841 high-quality, human- authore Chain-of- oh-Thuthh (CoT) anno- tations. In addition, our benchmark incorporates both image and video modal- ities and provides manual bounding box annotations to test visual grounding in the image part directly. Extensive experiments demonstrate the profound dif- ficulty of our benchmark. State-of-the-art baseline models perform poorly our most challenging tasks. While training on our data via Supervised Fine- Tuning and Reinforcement Learning improves these scores, they remain rela- tively low, highlighting a significant gap in current model capabilities. SportR presents challenge for the community, providing critical resource drive future research in multimodal sports reasoning. The dataset is available at https::/githuu: com chilii Lab/SportR
INTRODUCTION
The analysis of sports has rapidly evolved from traditional applications in game prediction and statis-
ties
Xia et al. 2022 Oved et al., (2020) into a sophisticated domain for artificial intelligence (Xia
et al.
2024a). While the recent advent of Multimodal Large Language Models (MLLMs) (Ope-
nAl
2024b AmazonAGl et al. 2025 Gemini Team 2024 Anthhopii 2024a) has unlocked new
"Equal Contribution.

---

Published as a conference paper at ICLR 2026
off tater
Aftecl Beceaves
Seraktig
Iefersive
Parting
Backibor
Peahing
Hove
sPd
SPORTR
Carrye of Pace
be of
Castorech
gr ovee tauny
offf faot
Sustatte
Gutt der-eerrta Tose
oftentvv toe
Pase Interteee cereeeeene rence
Too dany Hen
guppin seat
Uie Mattiin
Figure 1: Overview of the SportR benchmark. It consists of two parts, SportsImage and Sports Video, covering 50 rule infraction categories and 12 different kinds of tactics.
challenge of sports understanding by enabling systems to reason about why events happen (Gautam et al., 2025: Zhang al. 2025). The effectiveness of these applications hinges the capacity of MLLMs for deep rulc-base based visual reasoning which remains a significant bottleneck. For instance, correctly adjudicating subtle hand-check foul in baskethall demands not only recognizing the in- teraction but also precisely identifying the momentary illegal contact and connecting this visual evidence to sports knowledge.
To address the challenge of complex reasoning in sports understanding, we conceptualize the path sports intelligence pyramid that defines a progressive path for model evaluation. At its base lies perceptual understanding recognizing players, actions, and basic game states. On tasks at this level, such as identifying the sport being played, current models already achieve high accuracy (Xial et al. 2024b Chen et al. 2025b), indicating this foundation is primarily established. At the other extreme lie tasks involving the identification of elite professional-leve scenarios involving obscure rules or complex tactical sequences. However, the most critical and immediate challenge lies some- where in the middle: Mastering the identification fundamental and common fouls and tactics that form the core of sports comprehension for any experienced participant.
Question Answering (QA) has become an effective framework for evaluating MLLM understanding across various domains (Chen et al.) 2025a; Yue et al. 2024 Wu et al.) 2023 Shao al.) (024a). However, progress in the sports domain is fundamentally constrained by the limitations of existing benchmarks. While multi-sport benchmarks such SPORTU (Xia et al. 2024b) provide explana- tions for evaluation, their annotations are not in the form of fine-grained Chain-of-Thought (CoT) reasoning traces needed to explicitly train the reasoning capabilities of model. Conversely, spe- cialized benchmarks such as SoccerNet-XFoul (Held et al. (2024) for soccer or FSBench (Gao et al.) 2025) for figure skating, offer deep single-sport analysis but do not support the evaluation of multi- sport generalization which is essential for a robust reasoming model. Furthermore, critical limita- tion shared across these prior works the lack of precise visual grounding annotations, making difficult to verify whether model judgments are tied to specific visual evidence and whether there judgments assess the true source of their reasoning. Studies have shown that while models excel at basic perceptual tasks (c.g., sport identification), they struggle significantly with tasks requiring true comprehension [iaa et al. 2024b Chen et al. 2025b).
To address these critical gaps, we introduce SportR, new large-scale, multimodal benchmark de- signed to train and evaluate fine- fine-graine grained reasoning for fundamental sports understanding. We focus on diverse set of five globally popular ball and racket sports basketball, soccer, table tennis, badminton, and American football to provide a rich testbed for generalization. Our benchmark is based on corpus of 4, 4,789 images (from all five sports) and 2,052 videos (from four sports, exclud-
2

---

Published as a conference paper at ICLR 2026
ing badminton), encompassing 50 distinct foul types and 12 fundamental tactics. The comerstone of our work is a collection of 6,841 high-quality, fully human-annotated CoT. We also derive over 20,000 structured question-ansaen answer pairs.
Our contributions are threefold:
We introduce a novel benchmark for sports reasoning, featuring a progressive QA hierar- chy that enables granular evaluation of model capabilities. To provide gold standard for complex reasoning, we include fully human-annotated CoT traces that contain step- stp-by-step logic for the most challenging tasks.
2. We introduce the first multi-spon benchmark to feature an explicit visual grounding task, requiring models to output the precise bounding box of rule infraction, directly testing the ability of a model to ground abstract rule knowledge in precise visual evidence.
3. We demonstrate through experiments that SportR is a viable training resource and a rel- atively challenging benchmark. We show that state-of-the-art training methodologies, in- cluding Supervised Fine-Tuning and GRPO-based Reinforcement Learning, have clear per- formance gains. Also, observe that these reasoning skills generalize across modalities; a model trained exclusively on our image data exhibits improvement unseen video tasks. However, even tuned models struggle significantly on the grounding task, only improving from 4.61 % to 9.94% on average Intersection over Union (IoU), highlighting the difficulty our benchmark and the substantial room for future research.
2 RELATED WORK
2.1 MULTIMODAL SPORTS ANALYSIS
The analysis of sports has rapidly evolved from isolated computer vision (CV) and natural language processing (NLP) aaks-suchh as action recognition (Shao et al.] 2020 Li et al. (02T or news generation (Huang et al. (020) a domain ripe for the unifying power of MLLMs (Xia et al.) 2024a). The advent of foundation models has catalyzed a shift towards systems that can process raw game footage and text to produce nuanced, human-like insights, tackling complex applications from automated commentary (Rao et al. 2024 Baughman et al, 2024) to tactical analysis (Caron & M'uller 2023 Zhang et al.
2025). This progress has enabled "sports understanding" applications, where the goal is not merely to describe what is happening, but to reason about why it is happening in the context of rules (Held al. 2024), strategies, and player interactions (Gautam al. [025 Rao et al. 2025). As models demonstrate increasing proficiency in generating fluent and context-aware outputs, the community's focus has shifted towards evaluating and improving their capacity for deep. multi-step reasoning, which is essential for trustworthy and reliable sports analysis. Current MLLMMs, such as Gemini 1.5 Pro (Gemini Team, 2024), Claude-3.5-Somme (Anthropic) 2024b), and GPT-4o (OpenAl 2024a) have demonstrated sports understanding abilities. Subsequent research has continued push the boundaries. For instance, the recent work has employed new strategy and framework to achieve
state-of-the-at performance However, even with these advances, the accuracy on the most difficult, rule-based tasks remains modest, suggesting that the core challenge of connecting visual perception to abstract rules is far from solved.
benchmarks (Chen et al.) 2025b).
This necessitates the development of more challenging and fine- ine-grained benchmarks designed to probe the limits of their performance and reasoning capabilities.
2.2 MULTIMODAL SPORTS QA BENCHMARKS
Question answering (QA) has become the standard way for evaluating the reasoning abilities of MLLMs across different domains Chen et al. (2025a): Liu et al. Yue et al. (2024); Wu et al. (2023); Shao et al. (2024a). In the sports domain, early multimodal QA benchmarks primarily focused on assessing models' ability to recognize actions and answer descriptive or temporal questions. For instance, ActionAtlas (Salehi et al. 2024) which focuses action recognition via multiple-choice, or Ego-Exo4D (Grauman et al. 2024) which evaluates skill proficiency and pose estimation and Sports-QA (Li et al. 2024c), which leveraged existing action recognition datasets to create large-
3

---

Published as a conference paper at ICLR 2026
scale VQA benchmark, but its questions do not center on the complex, rule-based scenarios that define competitive play.
More recent benchmarks have begun to address this gap by focusing on deeper forms of reason- ing. Notably, SPORTU (Xia et al. 2024b) introduced a comprehensive evaluation benchmark with a multi-level difficulty design, explicitly including "hard" questions that require understanding game explanatons
rules and tactics. However, while SPORTU provides explanations for evaluation, the limitations are that there are not enough of this type of question, and these rationales were not designed as detailed and fine-grained, human-annotated reasoning processes suitable for training models to per- form explicit, step-by-step reasoning. Furthermore, lacks precise annotations for visual grounding, making it difficult to assess whether model's judgment is based on specific visual evidence, such as the subtle point of contact defining foul. addition, SPORTU primarily relies on multiple-choice questions and slow-motion replays, limiting its utility for training deep reasoning and evaluating true sports understanding. Recent studies suggest that the MCQ format may not accurately reflect an LLM's true capabilities due to misalignment with real-world generation tasks and sensitivity to option ordering (Li et al. 2024d). A detailed comparison is provided in Appendix A
There are also more recent benchmarks that appear to cover single sport. For example. SoccerNet- XFoul (Held et al. 2024) provides anpert-annotated explanations for fouls, FSBench (Gao et al.) 2025) delves into the fine-grained, artistic scoring criteria of figure skating, FineBadminton intro- duces a henchmark for badminton understanding (He et al., 2025), and SoccerBench (Rao et al.) 2025) offers thirteen distinct tasks for multifaceted soccer video analysis. These video-centric datasets focus on a single sport, limiting cross-domain evaluation. While these datasets are invalu- able, critical gap remains: the lack of a multi-sport benchmark that provides rich, human-written reasoning process annotations specifically designed to teach models how to reason, combined with the precise grounding annotations needed to verify that this reasoning is tied the correct visual evidence.
SPORTR BENCHMARK
Q8
play a dhoww INTI Image?
Q10
vien whorks thorl-theee merrtng porale?
Answe:: Themss
Keswert plaser invblis
inuee rnfecciinn
trrcugh
Jasoy Mthorane playent
canl.
www.frr,
Fa Intraction occumadion the speaticloo byor?
Elite Scanarios docralyy protensional lnvel sosraatce
Q7
What Sphrmbee formation
Eapla Why tule Intrccionn whille jonny slager
peronna foa.
Scoma: Phossa Skan 3158)
2013 Eloo proces
Q8
meraberr cocaree maarnage tattl meseressii bing parslyy?
Fundamantal
Antawers suppa crimmmmee
Sporta ta Russoning instttttrr
the honuss....
QB
Tottore in vise?
Q12
Weat etodeus eamaay ortensive formiiio video7
tockies undee drnng
Optar why arsle intraction oldor
Fasthind
Sport Perception
Krower the centaro the
Unddrrstanding
mage areckins
Monaly zenic sports plauess and sustoos
Fa unn,, whar
Q13
primaryy defersivv forme move showw Tas
locaaali coardiaaee
Hille Seevira Westor
Sporidmage
Sport/iidco
Anmon::
(1972, XXL 32/71, (2771
Too Our
Figure 2: SportR overview. Left: A three-level pyramid frames our evaluation scope- pprreepennnioon (base and well established), fundamental sports reasoning (our focus), and elite scenarios (out of scope). Right: We instantiate a 13-question hierarchy with concrete examples: QI-Q7 (SportsIm- age) cover infraction detection, type, penalty reasoning, explanation, grounding by box coordinates, and offensive/defeensie tactics; Q8- Q13 (Sports Video) mirror these tasks in the temporal domain.
To address the limitations in current sports benchmarks, we introduce SportR, a new large-scale, multimodal benchmark designed to train and evaluate the fine-grained visual reasoning capabilities of MLLMs. The cornerstone of our benchmark is a collection of high-quality, human buman-authoredCoo rationales, each providing a detailed, step-by-step explanation for fundamental foul or tactical sce- nario. Our work is guided by conceptual pyramid of sports understanding that defines a progressive path for model evaluation (Figure 2)
4

---

Published as a conference paper at ICLR 2026
At the base of this pyramid lies perceptual understanding recoonnzznngg players, actions, and basic game states. On tasks at this level, such as the easy questions in SPORTU, state- of-the-art models and recent work already achieve near- near-perfect accuracy (Xia et al. 2024b Chen et al. 2023b), indicating this foundation is largely established.
At the apex are elite, professional-level scenarios involving highly complex tactical com- binations or obscure, controversial edge-case rulings. While a benchmark at this level represents an ultimate goal for the community, it requires degree of annotation expertise and model capability that is currently beyond reach.
SportR is therefore positioned bridge this critical gap, focusing on the essential middle layer: the fundamental fouls and tactics that form the core of deep sports comprehension for any experienced participant. This domain where, as we will demonstrate in our experiments, even the most advanced MLLMs struggle profoundly, with grounding per- formance often failing to surpass a low threshold. By establishing that models must first master these fundamentals before aspiring to elite-level reasoning, we frame SportR as necessary and foundational next step for the community.
We first designed a suite of QA pairs to probe a model's reasoning at increasing levels of depth. This hierarchy ranges from foundational tasks like infraction identification (Is there foul?) and classification (What type of foul?), to more complex challenges like penalty prediction (What the penalty?) and culminating in precise visual grounding task. To provide high-quality ground truth for this hierarchy, especially for penalty prediction and tectics recognition tasks, each image or video is annotated with fully annan-annotated CoT.
The benchmark consists of two complementary components:
Sportslmage dataset of 4,789 images designed to test model's ability connect abstract rules to precise evidence in static scenes. Each image is paired with a CoT rationale focused on a specific foul or tactical play. This component's progressive QA suite culminates in a unique visual grounding task, where the model is required to output the exact bounding box coordinates of the rule infraction.
Sports Video dataset of 2,052 video clips created extend the reasoning challenge the tem- poral domain. This component addresses scenarios where understanding context is only possible by observing motion over time, such as a "traveling" violation in basketball. Each video is also annotated with a comprehensive COT, from which a similar suite of progressive QA pairs. This part aims to provide a more thorough evaluation benchmark in sports understanding.
Overall, SportR contains 6,841 unique, human- CoT rationales, from which we derive over 20,000 structured question-answe pairs. This benchmark fills critical gap by providing the re- sources to train models not just to answer what happened, but to reason about why it happened based on explicit, grounded visual evidence.
Our design prioritizes not only assessing a model's 's reasoning but also verifying that this reasoning matched to specific visual evidence. To this end, we introduce a novel explicit grounding task, where models must output the precise bounding box coordinates of a rule infraction. We focus this coordinate-base evaluation on SportsImage because fouls are often defined by discrete, localized events such specific point of physical contact-that can be unambiguously annotated and evaluated in a static frame. This provides a clean and tractable setting to evaluate a skill that, as later demonstrate, poses a significant challenge even for state-of-the-art models. Establishing reliable baseline for this explicit form of grounding is a critical first step before tackling the greater spatio-temporal complexity of localization in video, which remains an important direction for future work. Examples of our SportsImage and Sports Video can be found in Appendix
3.1 QUALITY CONTROL
To ensure the highest standard of data quality, our benchmark is fully human-authored and an- notated. Recognizing that state-of-the-art MLLMs still exhibit significant weaknesses in complex sports reasoning and grounding, as demonstrated by prior benchmarks (Xia et al. 2024b), we de- cided to discard any model-assisted generation for our annotations. We posit that creating a reli- able and expert-driven ground truth is a critical, pioneering step for domain where models are known to struggle.
5

---

Published as a conference paper at ICLR 2026
All CoT reasons and annotations were therefore created by team of 16 experts, all of whom authors of this paper. This team includes two former NCAA Division atudent-athletes with over 12 years of competitive experience, and 14 other members with least three years of dedicated training and engagement in their respective sports. This deep, practical expertise was foundational to maintaining the high standard of the benchmark.
A key principle of our data curation was to focus on fundamental and widely understood fouls and tacties, such as basketball blocking foul or a standard post move, rather than highly specialized or controversial professional-level edge cases. This focus ensures the scenarios, while challenging for AI systems, are grounded in the established, practical expertise of our annotation team. This allows the creation of clear, consistent, and authoritative rationales, minimizing ambiguity and subjectivity.
The process began with training phase where all annotators collaboratively developed and stan- dardized the guidelines for writing CoT rationales and annotating grounding boxes. Specifically, annotators were trained to follow strict -aaro-to-MMeroo logical flow. The reasoning process consists of three sequential steps: (1) Identify the specific court area and involved parties; (2) De- scribe the action details and dynamics leading up to the event; and (3) Pinpoint the precise point of contact or critical visual evidence. Each member then worked on small batches of examples, with the group reviewing the outputs to ensure a unified standard of quality and detail before commencing the full-scale annotation.
Following the initial annotation, we implemented a strict verification and filtering protocol. First, each annotator performed self-review of their work for accuracy. Critically, any instance where an annotator felt uncertain about the correct interpretation or the precise grounding location was flagged and passed still not be reached, or if both annotators remained unsure, the data point was discarded from the benchmark to minimize the risk of mislabeling.
second expert annotator for an independent review. If consensus could
3.2 SPORTSIMAGE: MULTIMODAL IMAGE QA
SportsImage is the first component of our benchmark, designed to rigorously evaluate an MLLM's ability to perform fine-grained reasoning and rule-based visual grounding in static scenes. This part covers five sports: basketball, soccer, table tennis, badminton, and American football. comprises 4,789 images, each capturing critical moment of gameplay across multiple sports.
Each image is paired with a comprehensive, human-authored CoT rationale. This rationale serves as the ground truth to generate a progressive QA suite of up to seven distinct questions, resulting in over 17,000 structured question-answer pairs for this component alone. These questions are structured to probe& model's reasoning at increasing levels of depth, covering: (QI) infraction identifica- tion, (Q2) foul classification, (Q3) penalty prediction, (Q4) free-form explanation, (Q6) offensive tactic identification, (Q7) defensive tactic identification, and culminating in (Q5) explicit visual grounding task requiring the model to output precise bounding box coordinates.
Explicit Visual Grounding: This is the most challenging and unique task in QA suite. This final question moves beyond conceptual understanding and requires the model to output the precise bounding box coordinates of the rule infraction. This novel task serves as a test of whether a model's reasoning is connected to specific, localized visual evidence. We focus this coordinate- uation on static images because fouls are often defined by discrete events-such of physical contact- that can be unambiguously annotated, providing clean and high- precision ground truth for this difficult skill. To the best of our knowledge, SportR is the first benchmark in the sports understanding domain to introduce such task, directly evaluating a model's ability to ground abstract rule knowledge in precise spatial evidence in sports.
eval- specific point
Dataset Construction Our annotation process was guided by our progressive QA hierarchy, with experts providing direct ground-truth answers for each question. We annotated a detailed CoT rea- soning process for specific tasks that demand more than simple answer. For the penalty prediction task (Q3), which requires a multi-step reasoning process, the CoT provides a gold-standard rea- soning path from visual evidence to rule application and final judgment. For the explanatory tactic identification tasks (Q6 and Q7), we also offered a human- annotated CoT to illustrate the tactical formation strategy.

---

Published as a conference paper at ICLR 2026
A key feature of our design is that this same expert-writted CoT also serves the definitive ground truth for the free-form explanation task (Q4). This approach allows the CoT to be leveraged flexibly: it can be used train or guide a model's 's reasoning process for the more complex tasks.
target for evaluating explanation generation,
chain-of-thouse prompt to
For the final visual grounding task (Q5), annotators manually drew the tightest possible bounding box around the critical visual evidence- for instance, the exact point illegal contact between two players' arms. This focus on static images allows for the creation of an unambiguous, high- precision ground truth for localization, ensuring that SportsImage provides a robust and multifaceted benchmark for assessing a model's sports understanding.
3.3 SPORTS MULTIMODAL VIDEO QA
Sports Video the second component of our benchmark, designed to extend the fine-grained rea- soning challenge into the temporal domain. Many fundamental fouls and tactics are inherently dynamic and can only be understood by observing the sequence and context of motion over time. For example, a "traveling" violation in basketball is impossible to judge from single static frame. Sports Video complements SportsImage by providing these crucial temporal scenarios, offering a more comprehensive evaluation of sports understanding.
The dataset consists of 2,052 video clips, from which we derive over 6,000 structured question- answer pairs. Like its image-based counterpart, each video is annotated with comprehensive, human-authored CoT rationale that explains the core foul or tactic. This CoT then serves as the ground truth for progressive QA suite designed evaluate spatio-temporal reasoning.
Dataset Construction The construction process for Sports is similar to that of SportsImage, adapting our progressive QA hierarchy to the temporal domain. Our annotation team collected short video clips that illustrate fundamental rule infractions or tactics requiring temporal understanding. Following the same protocol, experts provided direct, ground-truthh answers for each question in the suite. For tasks demanding detailed explanation of the sequence of events and reasoning cusionnnnnguhhhaeee such as penalty prediction and free-form explanation aanaatiion-nnntooooooo authored a comprehensive CoT rationale.
Unlike Sportslmage, this suite does not include the explicit, coordinate- based grounding task. The challenge of defning and consistently annotating precise spatial coordinates across multiple dy- namic frames is substantial research problem in its right. While this is important avenue for future research, our focus this component is to establish a strong baseline for a model's ability to perform complex temporal reasoning based on the provided CoT.
EXPERIMENT
To validate the challenge posed by SportR and to demonstrate its utility as a training resource, our experiments are designed with two primary objectives. First, we establish comprehensive baseline by evaluating a wide range of state-of-the-ar MLLMs a zero-shot setting. This demonstrates the difficulty of our benchmark for existing models. Second, we investigate the benchmark's s effective- ness for model improvement by performing supervised fine-tuning (SFT) and reinforcement learning (RL) on powerful open-source model. This serves as proof of concept that our dataset and its human- authored CoT reasons can be effectively used to enhance model capabilities in fine-graine sports reasoning.
4.1 BASELINE MODELS
We evaluate a diverse set of leading MILMSs to establish robust performance baseline on SportR. Our evaluation includes both proprictary and open-source models.
Proprictary Models: We evaluate the latest and most powerful closed-source models, including GPT- 5 (a20) 2025), Claude 0 (Anthropic 2025), and Gemini 2.5 Pro (Anthropic 2025). Access to these models was facilitated through their official APIs. of recently released, seghhrmeformig open-source MLLMs, including models from the LLaVA fam- ily (LLaVA-One Vision 7B, LLa LALVAA-eet (Li et al. 2024a b), the Qwen family (QwenVL-2.5 7B, 72B) (Bai et al. 2025), Deepseek-Vi VI. (Wu et al. 2024h, and Glm-4.5V (Team al. 2025
Models: We selected a broad range
7

---

Published as a conference paper at ICLR 2026
All baseline evaluations conducted in zero-shot setting. For each question in our test set, the model is provided with the image or video and the question text. We use consistent prompt template across all models and set the temperature 0.7.
4.1.1 MODEL TRAINING ON SPORTR
Beyond zero-shot baseline evaluations, we fine-tuned an open-source model to validate SportR as an effective training resource. We employed two-stage training process on the Qwen-2.5VL-7B model: an initial Supervised Fine-TTuning (SFT) phase followed by Reinforcement Learning (RI.) using the Group Relative Policy Optimization (GRPO) algorithm (Shao et al. 2024b). The training was conducted exclusively our SportsImage component to demonstrate its value for teaching fine- grained visual reasoning. Full details on our data preparation, SFT/RL methodology, and reward function design are deferred to Appendix B
4.2 EVALUATION
Our evaluation is conducted on the SportsImage test set and the entire EportsVided dataset, the latter of which is for zero-shot evaluation and cross-modal generalization testing.
We employ two primary metrics: Intersection over Union (IoU) (Everingham et al. 2010 for the vi- sual grounding task (Q5) and an LLMM-asJJudge framework for all other text-based QA tasks (detaile prompts are in Appendix C To mitigate the known issue of self-preference bias where models may unfairly favor their own outputs (Zheng et al. 2023). we use three proprietary models judges: GPT-5 (a20) 2025), Gemini 2.5 Pro (Comanici et al. 2025), and Claude 4.0 Sonnet (Anthropic) 2025 and report the average score. To validate the reliability of this metric, we conducted a human verification study on a stratified subset of the test set (660 samples) using the same instruction as we provided to L.Mss. We observed a strong correlation between the LLM average score and expert human judgments (Pearson > 0.65 across modalities), confirming that our "Average Score" ap- proach aligns well with human evaluation. The correlation map are shown in Appendix B For each generated response, report the scores from all three judge models as well as the average score to provide a comprehensive and robust assessment of performance.
RESULT
Our experiments are designed to demonstrate both the profound challenge posed by our benchmark and its utility as training resource. We present results for the SportsImage and Sports Video com- ponents separately to provide analysis of the model's 's capabilities in both static and video settings.
5.1 PERFORMANCE ON SPORTSIMAGE
We present the performance of all models the test of our SportsImage component in Table 1 The results are broken down by each of the seven progressive question types, providing a granular view of model capabilities. The evaluation clearly demonstrates both the profound challenge posed by our benchmark and its utility as training resource.
Baseline Performance. The zero-shot results underscore the difficulty of our benchmark. Even the most powerful proprictary model, GPT-5, struggles to achieve high accuracy on the core reason- ing tasks. Across all baselines, performance is particularly low tasks requiring deep, specialized knowledge, such as Foul Classification (Q2) and, most notably, the Explicit Visual Grounding task (Q5), where IoU scores are consistently below 7%. This confirms that existing models lack the sports fine-grained perception and rule-based reasoning abilities that SportR is designed to measure.
Effect of Training on SportsImage. After the SFT phase, we observe dramatic improvement across nearly all tasks. For instance, Foul Classification (Q2) accuracy leaps from 14.4% to 50.7%, validating that our numan-authored data provides strong learning signal. The subsequent Re- inforcement Learning (SFT+RL) phase further pushes performance on most tasks, achieving the highest scores on 5 out of 7 categories.
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison across models (v%) on SportsImage. Q1: Infraction identifi-
cation; Q2: Foul Classification; Q3: penalty prediction; Q4: Free-form Explanation. Q5: Visual
Grounding (loU); Q6: Offensive Tactic Identification; Q7: Defensive Tactic Identification.
Model
Q1
Q2
Q3
Q4
Q5
Q6
Q7
GPT-5
69.19
44.21
44.49
41.34
5.70
65.75
58.82
Claude 4 Sonnet
52.49
26.21
30.55
14.02
3.67
31.16
73.30
Qwen-2.5VL-72B
49.97
22.92
32.61
14.64
6.93
18.83
66.96
Llava-Next-8B
53.26
12.01
17.85
16.81
0.00
40.04
66.99
MiniCPM-V-4.5
49.97
17.57
27.36
11.19
4.13
37.86
57.09
Gemini 2.5 Pro
58.79
17.54
19.54
35.16
3.67
21.12
45.23
DeepScek-v12
47.56
23.09
30.41
21.19
0.91
32.04
31.70
GLM-4.5v
51.71
20.69
26.15
20.78
4,46
26.53
27.03
Qwen- VL-7B (Base)
48.29
14.43
21.69
12.32
4.61
24.66
21.81
Owen-VL--77 (SFT)
69.82
50.71
33.13
32.94
2.88
55.08
76.56
Qwen-VIL-7B (SFT+RL)
84.19
51.54
52.34
27.44
9.94
60.89
87.07
5.2 PERFORMANCE ON SPORTS VIDEO
Table 2: Performance comparison across models (%) Sports Q8: Video Infraction iden- tification; Q9: Video Foul Classification; Q10: Video Penalty Prediction; Q11: Video Free-form
Explanation. Q12: Offensive Tactic Identification; Q13: Defensive Tactic Identification.
Model
Q8
Q9
Q10
Q11
Q12
Q13
Owen-2.5VIL-72BB
30.19
13.84
15.90
8.67
31.86
13.10
GLM-4.5v
24.82
17.61
19.22
12.86
19.01
9.82
Video-R1-7B
25.15
14.56
11.54
8.14
30.60
3.75
MiniCPM-V-4.5
27.33
2.11
2.93
5.00
35.84
7.44
Qwen2.5- (Base)
25.49
15.06
11.63
8.27
33.41
3.64
Qwen2.5-V1L-77 (SFT)
70.65
17.20
12.08
17.76
8.04
12.13
Qwen2.5-VL--BB (SFT+RL)
59.52
17.53
19.71
14.89
9.37
12.88
Claude 4 Sonnet
36.93
21.37
16.99
8.15
38.08
8.25
Gemini 2.5
64.93
25.69
26.71
44.18
17.87
GPT-5
59.17
34.39
41.83
24.02
60.82
8.42
The performance on our VportsVideo part is presented in Table 2 The video tasks show a greater challenge than their image-based counterparts, with overall model performance being lower. As with Sportslmage, the SOTA proprietary model, GPT-5, achieves the highest scores on the majority of tasks (Q9-Q12) with a relatively low score, underscoring the difficulty of our benchmark. An- other finding of our work emerges from our Qwen-2.5VL-7B (SFT+RL) model, which was trained model
exclusively on SportsImage data. Despite having no exposure to videos during training, demonstrates promising degree of cross-modal generalization. Its performance Video Infrac- tion Identification (Q8) improved dramatically from 25. 55.49% to 59.52%, surpassing all other models, including GPT-5. We also observe performance gains most tasks, except for Q12. This find- ing underscores the value of SportsImage as a foundational resource for building generalized sports reasoning capabilities.
5.3 ERROR ANALYSIS (NEW SECTION ADDED BASED ON REVIEWER FEEDBACK)
To understand the specific challenges posed by SportR, we conducted a manual eтто analysis on 1,500 failure cases. We sampled 150 images and 150 videos from the test set for each of the six representative models. Errors were classified into five categories: (1) Visual Hallucination, fabricat- ing non-existent objects or events; (2) Domain Knowledge Gap, misapplying sports rules or failing to recognize standard penalty; (3) Reasoning Error, flawed logical derivation despite correct per- ception; (4) Format Violation, failing to follow output schema or refusing to answer, and (5) Visual Perception Error, missing critical visual evidence present in the input. The aggregate error distribu- tion is illustrated in Figure For a detailed breakdown by modality (Image vs. Video), please refer to Appendix D

---

Published as a conference paper at ICLR 2026
Visual
Clarde-4-Somet
Hallucination
GPT-5
50
Gemiini225-Pro
Qwen-255VL-72B
40
Qwen-25-VV-7BB
SFT+RL
Visual
20
Domain
Perception Error
Knowledge Gap
10
Format Violation
Reasoning Error
Figure 3: Error type distribution across different MLLMs on overall SPORTR tasks. The analysis reveals that Visual Perception Error is the most common issue, followed by Hallucination Error. Each error type highlights specific model limitations in comprehending the task.
We found in video tasks, errors are overwhelmingly dominated by Visual Perception Error and Visual Hallucination. Across all models, these two categories combined frequently exceed 60-70% of total errors. This indicates that the primary barrier in video the inability to parse fine-grained, temporal dynamics. Consequently, models rarely reach the stage of successful rule adjudication, resulting in an artificially low "Domain Knowledge Gap."
In contrast, static image tasks-whers temporal ambiguity is removed and visual perception is in- herently easier-reveal the true extent of the models' limitations in sports logic. We observe a sharp spike Domain Knowledge Gaps when shifting from video to image. This trend particularly evident in proprietary SOTA models. For instance, GPT-5's Domain Knowledge Gap rises from 20.00% video to 36.00% images. This confirms that the decrease visual perception errors, expected due to the static nature of images in sports, effectively unmasks the underlying deficiency in reasoning: even when SOTA models successfully perceive the visual evidence, they struggle significantly to map that evidence to the correct abstract sports rule.
CONCLUSION
In this paper, we introduced SportR, a large-scale benchmark designed to train and evaluate the fine-grained, rule-based visual reasoning of Multimodal Large Language Models. Built upon foundation of over 6,500 fully human-authored Chain-of-Thought rationales, we aim to provide benchmark that can used both for training and evaluation. By integrating tasks across both images and videos, our benchmark provides a holistic assessment of a model's 's ability in sports understand- ing. Our experiments demonstrate that while our dataset an effective training resource, it presents profound challenge to current models. For example, even after fine-tuning and reinforcement learning, performance on the explicit grounding task remains modest, underscoring the difficulty of connecting abstract rules to precise visual evidence. Crucially, our error analysis reveals that current MLLMs suffer from a fundamental shortage in fine-gratd mne-grained visual perception for dynamic events and significant gap in aligning visual evidence with abstract domain knowledge. We hope SportR will serve as a critical tool for the community, inspiring advancements in MLLMs and contributing more robust and reliable real-world sports understanding.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENTS
This project is supported in part by the Rice Ken Kennedy Institute Research Award #108 1025 and the U.S. National Science Foundation under Grant No. 2429680. We thank the anonymous reviewers for their thoughtful feedback and constructive suggestions, which helped improve this work.
REFERENCES
GPT-5 System Card
OpenAI, 2025.
URL
https://cdn.openee com/
get 5- cestem-card.por
AmazonAGI, Aaron Langford, Aayush Shah, Abhanshu Gupta, Abhimanyu Bhatter, Abhinav Goyal, Abhinav Mathur, Abhinav Mohanty, Abhishek Kumar, Abhishek Sethi, Abi Komma, al. The Amazon Nova Family of Models: Technical Report and Model Card. arXiv preprini arXiv:2506. 12103, 2025.
Anthropic. Claude 3.5 Sonnet Model Card Addendum, 2024a. URL hwwp::///wwww//// cdn.
anthropic. com fed9cc1 comffe99cc Card Btt_ccllauu____dddddiieee 25344488888777777775777888550523
Anthropic. Introducing the next generation of Claude, 2024b.
URL https: //ww.
mnttroppic..coo//ews//eee aude- 3- fami ly
Anthropic. System Card: Claude Opus 4 & Claude Sonnet 4, 2025. URL https: www odn.
4 6aa0 S.r.://DDDD.00000222222222622220
anthropic.
com wttp:opp...ccmaa..
pdd
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong. Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report. arXiv preprint ar Xrxvv22502. 13923, 2025.
Aaron Baughman, Eduardo Morales, Rahul Agarwal, Gozde Akay, Rogerio Feris, Tony Johnson, Stephen Hammer, and Leonid Karlinsky. Large scale generative ai text applied to sports and music. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4784-4792, 2024.
Matthew Caron and Oliver Mouller. Tacticalgpt: uncovering the potential of iims for predicting tactical decisions in professional football. In StatsBomb Conference, pp. 2023.
Hanjie Chen, Zhouxiang Fang. Yash Singla, and Mark Dredze. Benchmarking large language mod- els on answering and explaining challenging medical questions. In Proceedings of the 2025 Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3563-3599, 2025a
Haodong Chen, Haojian Huang, XinXiang Yin, and Dian Shao. Finequest: Adaptive knowledge- assisted sports video understanding via agent-of-thought. reasoning, 2025b. URL https://
arxiv. owxiv.org//bs. /250 11796
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Rong Gao, Xin Liu, Zhuozhao Hu, Bohao Xing, Baiqiang Xia, Zitong Yu, and Heikki Kälviäinen. Fsbench: figure skating benchmark for advancing artistic sports understanding. arXiv preprint arXiv:2504. 19514, 2025.
11

---

Published as a conference paper at ICLR 2026
Sushant Gautam, Cise Midoglu, Vajira Thambawita, Michael A Riegler, Pål Halvorsen, and Mubarak Shah. Soccerchat: Integrating multimodal data for enhanced soccer game understand- ing. arXiv preprint arXiv:2505.16630 2025.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text, 2024.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang. Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1290 2025.
Xusheng He, Wei Liu, Shanshan Ma, Qian Liu, Chenghao Ma, and Jianlong Wu. Finebad- minton: multi-level dataset for fine-grained badminton video understanding. arXiv preprint vrXiv:2508.07555 2025.
Jan Held, Hani Itani, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogen- broeck. X-vars: Introducing explainability in football refereeing with multi-mode large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- lion, pp. 3267-3279, 2024.
Kuan-Hao Huang, Chen Li, and Kai-Wei Chang. Generating sports news from live commentary: A chinese dataset for sports game summarization. In Proceedings of the 1st Conference of the Assa-Paccifa Chapter the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 609-615, 2020.
Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl: Reinforce- ment learning for generalizable medical reasoning in vision-language models. arXiv preprint DrXiv:2503.13933 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li. Ziwei Liu, and Chunyuan Li. LLa -neeonnnn Easy Visual Task Transfer, 2024a. htthhtts://
http://xxvvvv////. org/ abs/ 2408 03326
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang. Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLaVe--eeT-tnnerleavee: Tackling Multi-image, Video, and 3D in Large Multimodal Models, 2024b. URL ltttts::/aaxiii..t org/ abs/ /2007 07895
Haopeng Li, Andong Deng, Qiuhong Ke, Jun Liu, Hossein Rahmani, Yulan Guo, Bernt Schiele, and Chen Chen. Sports-qa: A large-scale video question answering benchmark for complex and professional sports. arXiv preprint arXiv:2401.0f 5055 2024c.
Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. Can multiple-choice questions really be useful in detecting the abilities of LLMs? In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 2819-2834, Torino, Italia, May 2024d. ELRA and ICCL. URL http:e///aallaattaoooocooooo//o
Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. Multisports: A multi-person video dataset of spatio-temporally localized sports actions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13536-13545. 2021.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Zhaowei Liu, Xin Guo, Haotian Xia, Lingfeng Zeng. Fanggi Lou, Jinyi Niu, Mengping Li, Qi Qi, Jiahuan Li, Wei Zhang, Yinglong Wang, Weige Cai, Weining Shen, and Liwen Zhang. Visfineval: A scenario-driven chinese multimodal benchmark for holistic financial understanding.
12

---

Published as a conference paper at ICLR 2026
Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang. Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, et al. Fin-rl: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXIv:2503.16252, 2025.
OpenAl. Hello gpt-4o, 2024b, http:/ss///ppaaa..cc/////e////ss gpt-
OpenAl. Hello gpt-4o, 2024a. URL
Nadav Oved, Amir Feder, and Roi Reichart. Predicting in-game actions from interviews of nba players. Computational Linguistics, 4663)667712,, 2020.
Jiayuan Rao, Haoning Wu, Chang Liu, Yanfeng Wang, and Weidi Xie. MatchTime: Towards Auto- matic Soccer Game Commentary Generation. In Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, 2024.
Jiayuan Rao, Zifeng Li, Haoning Wu, Ya Zhang, Yanfeng Wang. and Weidi Xie. Multi-agent system for comprehensive soccer understanding. arXiv preprint arXiv:2505.0377 2025.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.
Mohammadreza Reza Salehi, Jae Sung Park, Aditya Kusupati, Ranjay Krishna, Yejin Choi, Hanna Hajishirzi, and Ali Farhadi. Actionatlas: A videoqa benchmark for domain--ppccialized action recognition. Advances in Neural Information Processing Systems, 37:137372-137400 2024.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hicrarchical video dataset for fine- grained action understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2615-2625, 2020.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for chain-of thouhhtttt reasoning. Advances in Neural Information P Processing Systems, 11.8612-862,, 2024a.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv.2402.03300, 2024b.
Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi. Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng. Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang. Chenhui Zhang. Da Yin, Fan Yang. Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan. Mingdao Liu, Mingde Xu. Mingzhi Zhang. Qinkai Zheng, Sheng Yang. Shi Zhong, Shiyu Huang. Shuyuan Zhao. Siyan Xue. Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang. Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. GLM-4.5V and GLM-4. 14-4..iiThinningg Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning, 2025. hRtpstts::///xx
org/ abs abs/2507. 01006
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint 23ti:::200..188... 2023.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu. Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai
13

---

Published as a conference paper at ICLR 2026
Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. DeepSeck-VL2 Mixture- of-Experts Vision-Language Models for Advanced Multimodal Understanding, 2024. URL
https: org/ abs /wws/21222
Haotian Xia, Rhys Tracy, Yun Zhao, Erwan Fraisse, Yuan-Fang Wang, and Linda Petzold. Vren: volleyball rally dataset with expression notation language. 2022 IEEE International Conference Knowledge Graph (ICKG), pp. 337-346. IEEE, 2022.
Haotian Xia, Zhengbang Yang, Yun Zhao, Yuqing Wang, Jingxi Li, Rhys Tracy, Zhuangdi Zhu, Yunn-fang Wang, Hanjie Chen, and Weining Shen. Language and multimodal models in sports: A survey of datasets and applications. arXiv preprint arXiv:2406. 1t..6.222555 12252, 2024a.
Haotian Xia, Zhengbang Yang, Junbo Zou, Rhys Tracy, Yuqing Wang, Chi Lu, Christopher Lai, Yanjun He, Xun Shao, Zhuoqing Xie, et al. Sportu: A comprehensive sports understanding benchmark for multimodal large language models. arXiv preprint 2rXiv:2410.08477 2024b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Jiawen Zhang, Dongliang Han, Shuai Han, Heng Li, Wang-Kai Lam, and Mingyu Zhang. Chat- Match: Exploring the potential of hybrid vision-language deep learning approach for the intelli- gent analysis and inference of racket sports. Computer Speech & Language, 89:101694, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 04446595-46662 2023.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
During the preparation of this manuscript, we utilized large language models (LLMs) to assist with grammar correction and improve the clarity of the writing.
A DETAILED COMPARISON WITH EXISTING BENCHMARKS
While benchmarks like SPORTU Xia et al. 2024b) pioneered multi--port evaluation, SportR rep- resents a structural evolution designed to support training and fine-grained reasoning. The key distinctions are summarized below:
1. Training vs. Evaluation Utility. SPORTU's "hard" tasks rely heavily on Multiple Choice Questions (MCQs). While effective for low-cost evaluation, MCQs allow models to guess answers and cannot be used to train generative reasoning capabilities. In contrast, SportR provides over 6,500 human-authored Chain-of-Thouuhh (CoT) annotations. This transforms the benchmark from pure evaluation set into rich training resource that explicitly teaches models how to reason step-by-step.
2. Visual Fidelity (Normal vs. Slow Motion). A critical limitation of SPORTU's video subset is its reliance on slow-motion replays, which simplifies temporal perception. SportR utilizes normal- speed videos, presenting significantly more realistic challenge. Models must capture fleeting visual cues without temporal assistance, testing true temporal understanding.
Task Scope: Adjudication vs. Perception. Prior benchmarks often include basic perception tasks (e.g.. counting players, identifying colors). SportR filters these out to focus exclusively the "Reasoning Gap": complex rule-based adjudication and visual grounding. This ensures the benchmark measures deep sports understanding rather than basic visual recognition.
14

---

Published as a conference paper at ICLR 2026
B DETAILED MODEL TRAINING METHODOLOGY
Here we provide a comprehensive description of the training process used to validate SportR as training resource.
B.1 MODEL TRAINING ON SPORTR
Beyond zero-shot evaluation, investigate whether SportR serve effective resource for model improvement through two-stage training process: supervised fine-tuning followed by re- inforcement learning. We select Qwen2.5-VL (7B) as backbone model. The training process exclusively uses the SportsImage component, with the data images partitioned into an SFT set, an RL set, and
based on Image ID.
B.2 TRAINING DATA PREPARATION
For questions, such as infraction identification and foul classification, the target output for the SFT phase is the concise, ground-truth answer enclosed in janswer; and j/answer, tags to train the model for direct question answering. To explicitly teach the model the reasoning process, we apply targeted supervision for the penalty prediction task (q4) only. For this question, the training target includes the full human-authores CoT as supervised reasoning steps within <think>... reason ing steps....... ... </think>, <answer>... final answer... < </answer>, For the distinct task of free-form explanation (q6), we use separate format where the target output is the full human rationale enclosed within <Explanations
</Explanat on>, sig- naling an explanation task. This approach enables a targeted, multi-task learning process. allowing the model to simultaneously learn direct answering, step-by-step reasoning, and detailed explanation generation.
explanations
B.3 SUPERVISED TUN-INGNIGG (SFT)
Following the methodology of recent work in large-model alignment (Guo et al. 2025 Liu et al. 2025). we perform a "cold-start" SFT phase. This initial step serves two primary objectives in our experimental design.
First, it directly validates our benchmark's 's efficacy as a resource for supervised learning. By fine- tuning the model 10% of the SportsImage data with 6 epochs, demonstrate that high- quality annotations can lead to performance gains over the zero-shot baselines, as shown our results (Table 2).
Second, this SFT phase used for preparing the model for the more intensive RL training. This aims teach the model the format of our questions and the fundamental patterns present in the CoT rationales before the more intensive RL phase.
B.4 REINFORCEMENT LEARNING
Group Relative Policy Optimization: During the reinforcement learning phase, we employ the Group Relative Policy Optimization (GRPO) Shao al. (2024b), which has been commonly used in reinforcement learning for model training across multiple domains Guo et al. (2025). Liu et al.
2025
Lai et al. (2025).
For each prompt y. sample group of G responses {0 frx Ir
Tota(- 4) and obtain rewards
We define the advantage A, as follows:
Pr
A
Or
15

---

Published as a conference paper at ICLR 2026
To lg) (o l
where Mr, o, are the mean and standard deviation of fri}, and let p.
GRPO maxi-
mizes the following objective:
(0) E
G min (piA,, clip(e,, 1 - 1++) B DKI( (4) Tret( 19))
Reward Function Design: Our reward function R(olg) a standard weighted combination of correctness reward Reorect and format incentive Rfomat:
R(olg) = (oure + 0.5 Ro)
(2)
The correctness reward, Reorrect desk-dependent. For the standard QA tasks (qI-q5), we use an LLM-as-Judge DeepSeek-V22 V3.1 (Liu et al.) 2024)) to provide a binary reward (The template can be found Appendix B). the output within the <answer> </answer>> is semantically consistent with the standard answer, reward of 1 is assigned; otherwise, the reward is O. For the explicit visual grounding task (q7), the reward derived from the Generalized Intersection over Union (GIoU) (Rezatolighi standard IoU, it provides a meaninglul gradient even for non-overlapping boxes, serving as a more effective training signal (Rezatofighi et al. 2019). Since the GloU metric ranges from -1 to map it to a valid reward signal between and by applying the transformation. (GIoU + 1)/2. The second component, Riormats provides format incentive for the reasoning task. A score of is awarded if the output contains exactly one <thi nk> block, one < </thing hi> block, one <answer> applied to other tasks.
al. 2019). We opt for GloU as the reward signal because, unlike
block, and one < answer>, and otherwise. This reward
We did not include the Explanation questions (Q4) in the RL training phase, as their results are difficult evaluate due lack of specific metrics for sports.
We conducted the GRPO-based reinforcement learning on 4xH20 GPUs.
C PROMPT
C.I INFERENCE PROMPT
You are an expert sports assistant with advanced multi-sport knowledge and image/video anal- ysis capabilities for detecting tactics, fouls, and infractions.
Figure 4: System Prompt format for Answer Generation
You will receive a sport video. You need to answer the question. {$Question] You need output your response based on the following instructions. Instruction: Valid output schemas (choose exactly ONE based on the question intent):
1) Explanation-on
(when the question explicitly asks "why/explain"):
3Explanate i onn.... TEExpanattin
2) All other types of questions:
<think>.. </tti/hhhh>h>.. <answer>... <......///vvvi. </answer>
<
Figure System Prompt format for Answer Generation
C.2 TRAINING PROMPT
You are an expert sports assistant with advanced multi sport knowledge and image/video anal- ysis capabilities for detecting tactics, fouls, and infraction.
Figure 6: System Prompt format for Q1 Q7 SFT training set
16

---

Published as a conference paper at ICLR 2026
Answer the question after <answer> and end in </answer>, You don't need output any reasoning process. Imaging you are an expert in sport, you will receive a picture of the sport scene. You need to answer the question. Is there rule infraction (foul or infraction)? you can only answer "yes" or "no".
Output Example for answer part: <answer> yes </answer> Output Example 2 for answer part: <answer> no </answer>
Figure 7: User Prompt format for Q2 SFT training set
Answer the question after <answer> and end in </answer>. You don't need output any reasoning process. Imaging you are an expert in sport, you will receive a picture of the sport scene. You need to answer the question. If rule infraction (foul infraction) occurred, what was the specific type? you only need to output the specific rule infraction (foul or infraction) type.
You need to output your answer after <answer> and end in </answer>.
Figure 8: User Prompt format for Q3 SFT training set
First, think between <think> and </think>, answer the question after <answer> and end in < /aasswrr>, If a rule infraction (foul or violation) occurred, what the resulting penalty? You need to output your answer after <answer> and end in </answer>
Figure 9: User Prompt format for Q4 SFT training set
Answer the question after <answer> and end in < /nswer>,> You don't need output any reasoning process. Imaging you are an expert in sport, you will receive picture of the sport scene. You need answer the question.
If rule infraction occurs in the image, find and locate it, and output the coordinates. If there no rule infraction, locate the area makes the most sense to determine whether foul has been committed.
The coordinates must written in the form: [x1, y1, x2, y2].
x1 left boundary (minimum value, in pixels)
y1 = top boundary (minimum y value, in pixels)
x2 right boundary (maximum value, in pixels)
y2 = bottom boundary (maximum value, in pixels)
This bounding box uniquely defines rectangle region.
From these four values, the four corners can be derived as:
(x1, yl) left (x2, yl) top-right, (x2, y2) bottom right, (x1, y2) bottom-left.
You can only output reasoning process between <think> and </think>. You need to output
your answer after <answer> and end in </answer>.
Output Example for answer part: (do NOT copy the numbers, just follow the structure):
<answer> [x1, yl, x2, y2] </answer>
Figure 10: User Prompt format for Q5 SFT training set
17

---

Published as a conference paper at ICLR 2026
Answer the question after <answer> and end in </answer>, You don't t need output any reasoning process.
What offensive formation play is shown in this image? You need to output your answer after <answer> and end in </answer>.
Figure 11: User Prompt format for Q6 SFT training set
Answer the question after <answer> and end in </answer>, You don't 't need to output any reasoning process.
What defensive formation or play is shown in this image? You need to output your answer after <answer> and end in </ /answer>.
Figure 12: User Prompt format for Q7 SFT training set
{{ content trim }}
You are an expert sports assistant with advanced multi-sport knowledge and image/video analysis capabilities for detecting tacties, fouls, and infractions.
Answer the question only between <answer> and < </answer>. Do not output any internal reasoning, chain-of thouthtt or content between <think> tags. helpful, you may include single-sentence clarification between <expl anation> and </explamat after the answer; otherwise omit it.
What defensive formation or play is shown in this image? Output your final answer only be- tween <answer> and </ answer>.
Figure 13: System Prompt format for RL training set
Please think about this question as if you were human pondering deeply. Engage in an internal dialogue using expressions such "let me think", "wait", "Hmm". "oh, I see", "let's break down", etc., or other natural language thought expressions. It's encouraged to include self- reflection verification in the reasoning process. Provide your detailed reasoning between the <think> and </think> tags, and then give your final answer between the <answer> and </ </answer> tags.
Figure 14: User Prompt format for RL training set
18

---

Published as a conference paper at ICLR 2026
Below are two answers to question. [Question] is the question, [Standard Answer] is the standard answer, and [Model answer] is the answer extracted from a model's output.
Determine whether these two answers are consistent in meaning. they express essentially the same conclusion, treat them as consistent. Synonymous wording counts consistent (e.g., "pink" and "it is pink"). Focus on the final English conclusion; ignore length, style, or reasoning details.
Output exactly one line:
Judgement:
or Judgement: 0
Acceptance criteria (0/1):
Example:
[Question]: Is there a rule violation?
[Standard Answer]: Yes, there rule violation in the image. [Model answer]: Yes.
Judgement:
[Question]: a violation occurred, what was the specific type? [Standard Answer]: Free Hand Touches the Playing Surface Violation. [Model answer]: Free hand touching the playing surface.
Judgement: 133999
[Question]: violation occurred, what the resulting penalty? [Standard Answer]: Point to the opponent.
[Model answer]: The opponent is awarded the point.
Judgement:
[Question]: Is there rule violation?
[Standard Answer]: No, there is no rule violation in the image.
[Model Landera: No.
Judgement:
[Question]: If a violation occurred, what was the specific type?
[Standard Answer]: N/A
N/A
Judgement:
19

---

Published as a conference paper at ICLR 2026
[Question]: a violation occurred, what the resulting penalty? (Standard Answer]: No penalty.
[Model Lnnswrr:: No penalty.
Judgement: 339999
[Question]: Is there rule violation?
[Standard Answer]: Yes, there is a rule violation in the image. [Model Lanser]: No.
Judgement: 0
[Question]: If violation occurred, what was the specific type? [Standard Answer]: Net Touch.
Lnswer:: Free Hand Touches the Playing Surface Violation.
Judgement: 0
[Question]: If violation occurred, what is the resulting penalty? [Standard Answerj: Point to the opponent.
[Model_answer]] answer]: Replay (let). Judgement: 0
[Question]: Is there a rule violation?
[Standard Answer]: Yes, there is rule violation in the image. [Model answer]: Yes.
Judgement:
[Question]: a violation occurred, what was the specific type? [Standard Answer]: Service Let.
[Model answer]: Let serve.
Judgement: 339955
339997
[Question]: violation occurred, what the resulting penalty?
[Standard Answer]: Let (replay).
[Model answer]: Re-serve; the point replayed.
Judgement:
20

---

Published as a conference paper at ICLR 2026
[Question]: Is there a rule violation?
(Standard Answer]: Yes, there is rule violation in the image. [Model Lnnswrr:: Yes.
Judgement: 339999
[Question]: violation occurred, what was the specific type? [Standard Answer]: Service Fault.
[Model :anser]: Illegal serve (fault).
Judgement:
[Question]: If violation occurred, what the resulting penalty?
[Standard Answer]: Point to the opponent.
[Model :nswer]: The receiver scores the point.
Judgement:
[Question]: Is there rule violation?
[Standard Answer]: No, there no rule violation in the image. [Model_answer]] answer]: Yes.
Judgement: 0
[Question]: If a violation occurred, what was the specific type? [Standard Answer]: N/A
[Model answer]: Net Touch.
Judgement: 0
[Question]: a violation occurred, what the resulting penalty? [Standard Answer]: No penalty.
[Model answer]: Point to the opponent.
Judgement: 0
333955
Figure 15: Reward Prompt format for RL training set
21

---

Published as a conference paper at ICLR 2026
C.3 EVALUATION PROMPT TEMPLATE
You are a scoring assistant for sports questions.
Rules
Compare the semantic meaning of "ground truth" and "model response" and assign similarity score between 0 (completely different) and (identical).
Output
Return ONLY one line:(replace the score to the real number) <ansve>>scoe<</nsser>>
Figure 16: System Prompt format for LLM-as-judge answer
Now is your turn
spprt:: (SSport)
questtin: (SQuestion}
grruhh {STruth}
respones (SModelAnswer)
Figure 17: User Prompt format for LLM-as-judge answer
22

---

Published as a conference paper at ICLR 2026
D
ERROR ANALYSIS
Visual Helludiaation
Charb-SSorr
GPT-5
Gemini-25PPoo
Qwen--25VVL-728
(-en-25-VLL77
Visaal Peroption
21
Domain
Knowledge Gap
10
Format Violtion
Receaning
Error
Figure 18: Error type distribution across different MLLMs on overall SportsImage tasks.
Visual
Chazlo-4Somet
Helloctration
GPT-5
30
Gamini-2.-PPo
Quen-25-V1-78
SET-RL
Peroption Eroor
Knowledee Gap
Reesoning
Viokaton
Error
Figure 19: Error type distribution across different MILLMs on overall Sports Video tasks.
To support the findings in Result Section, we provide the detailed statistical breakdown of error types separated by modality. Figure 18 and Figure 19 present the error distribution for SportsImage and Sports Video, respectively.
D.1 MODALITY-SPECIHIC OBSERVATIONS
VportsVideo: The Perception Bottleneck. As shown in Figure 19 video tasks are overwhelm- ingly dominated by Visual Perception Errors and Visual Hallucinations. For instance, the Qwen- 2.5-VL-7B Base model exhibits a 49.33% perception error rate in video. compared to 36.00% in images. Across all models, the combined rate of perception and hallucination errors frequently exceeds 60-70%. This indicates that the primary bottleneck in video is the inability to process fine- grained temporal dynamics. Models fail to correctly perceive the event, often preventing them from reaching the stage of rule application, resulting in artificially low domain knowledge gap.
SportsImage: The Knowledge Mapping Gap. As shown in Figure 18 static images, where temporal ambiguity is removed, the error distribution shifts significantly. Stronger models show sharp spike in Domain Knowledge Gaps. For example, Qwen-2.5VL--7235 knowledge gap rises from 10.67% in video 43.33% in images. Similarly, Claude-4-Sonne'ss knowledge gap increases from 23.33% (Video) to 32.00% (Image). This implies that when models can perceive the visual evidence more clearly, they still fail map that evidence to the correct sports rule, validating the difficulty of the reasoning task itself.
D.2 CASE STUDY
23

---

Published as a conference paper at ICLR 2026
accurred, specific type?
Ground Truth: none
Model answer: <think>.. wttp:///ww.....ssseeeoo/saeaaaaa
Error category: Formet Violation
model produce valid thinking
Figure 20: Example of Format Violation
Question: Explein infraction did not this play?
Ground Truth:n this scene, the camera is facing the table head-on Balow the table the player wearing blue shirt his body leaning forward. ready
CARD
Ni aku
rica
S
Above the body leaning
the table extended straight and delermined that serving.
table
svinging racket,
From player's i actions
ZAI
However, the trajectory the ball cennot be seen around the server on the tabil, which is abriomnll The server's is left arm, extended straight and hortzontal, will block the opponents line sight.
Model answer: No rule infraciion. The player black reaching wide hitting ball the
the post. Under ITTF opponent's court mackttiamm may cross infraciion would pocur only the player
legal
during follow-thooggh touched the natitable or hit the bail before had neachedhoouncee on his side. In thes frame there's 's's no exidenze of contact with the net or table and the strike from side, the play legal.
category: : Perosption
Error Reason: The model missed the black-jersyy playor's serving molion. The key visusl evidence in the image were ignored.
Figure 21: Example of Visual Perception Error
Question: Explain Intraction did
Ground Truth: defender attacker hand. the the foul, the offence attacker commoling
defendars goslsaoper addiion
attacking player delfand. This adtion
opportunity
Model answer: At the and of the video, the playar in yellow usad his hand to push the attacking playar, cisnpting clear scoring opportunity. Player white atlempts shot on goal. but the pookeeper makes save. The players are aclively engaged in the game, and theme are no visible fouls or offside postions. No rule infrection eccurred.
Error category: Ressoning Eroor
Error Reason: The model cornectly desects the pish that abeihurts clear scoring chanow. but stil produces the incomest concuusion that no Ruul occurred. demondraaing Reasoning Emor.
Figure 22: Example of Reasoning Error
24

---

Published as a conference paper at ICLR 2026
Question: rule infrectinn (foul violations?
Fe
Ground Truth:
Model answer: -thinkein the middle the picture, the player wearing the red jersey with number attompted onbble the ball. At same time. the player the black jersey thee to defend from behind the red jersey player. During this process. the black
player player's back. According to footbal lllggal dangorous
hand made ifa pushing picture, player ended up contacting the player
contact with
player fiold
oppanent
jarsey attempled the rad jersey. The phys iall contact was only minar and should bo deemed as an offenie <itheko <answar-No, it's not foul.
Error category: : Visual Halluciation
Error Resson: The model mistakerly belleves that the black jersey player right hand made contact with the red jersey player's back. fictionalzzs non-existent contact.
Figure 23: Example of Visual Hallucination
E CORRELATION MAP BETWEEN HUMAN AND LLMS
Pearson Correlation (IMAGE)
GPT-5
1.
1.0000
0. 0.8594
0.8122
0.9461
0.61.8
1.0
Gemini 2.5
0.6
Claude
0.6264
LLM Average
0.9461
0.9648
0.9406
1.0000
0.6502
0.2
Human
0.6189
0.6030
0.6264
0.6502
1.0000
cPt.s
Geminn
Claude
Average
Human
Figure 24: Pearson Correlation in image tasks.
To assess the reliability of our LLM-as-Judge framework, conducted Human Verification Study on a stratified subset of the test set. This study involved 660 randomized samples (360 from SportsImage and 300 from SportsVideo), covering explanation tasks. Expert human annotators
25

---

Published as a conference paper at ICLR 2026
Pearson Correlation (VIDEO)
GPT3S
1.0000
0.8456
0.7849
0.9437
0.6540
1.0
Gemin 2.5
0.8456
1.0000
0.8607
0.9508
0.6522
0.6
Claude
LLM Auerage
1.0000
0.7043
0.2
Human
0.6540
0.6522
0.6837
0.7043
1.0000
GPT.S
Gemini
Average
Humen
Figure 25: Pearson Correlation in video tasks.
scored the model responses following the exact same instructions provided to the LLM judges, operating blindly without knowledge of the model sources.
E.1 CORRELATION ANALYSIS
We calculated the Pearson correlation coefficient between the human scores and the scores assigned by the three LLM judges (GPT-5, Gemini 2.5 Pro. Claude 4 Sonnet) as well their ensemble
average. The results are visualized in Figure 24 and Figure 25
Internal Consistency. First, we observe high internal consistency among the three proprietary models (Pearson correlation ranging from 0.81 to 0.96). This indicates that frontier MLLMs share robust, stable internal standard for evaluating sports reasoning, reducing concerns about randomness or model-specific hallucinations in grading.
Alignment with Human Judgment. Crucially, the LLM Average Score demonstrates a stronger correlation with human experts (r 0.65 for Image, judge. For instance, while individual models occasionally diverge from human ratings due to spe- cific biases, the ensemble approach effectively mitigates this variance. This confirms that averaging scores from multiple top-tier models serves as a reliable proxy for human evaluation in this bench- mark.
0.70 for Video) than any single LLM
E.2 SCOPE OF EVALUATION METRIC
It is important clarify the objective of this evaluation setup. Our primary goal is to validate the relative difficulty of the SportR benchmark and the performance hierarchy of current models, rather than to propose novel, perfect sports evaluation metric. The analysis confirms that our metric is sufficiently robust to distinguish between model capabilities and to establish a reliable comparison, which fulfills the purpose of this benchmarking study.
26

---

Published as a conference paper at ICLR 2026
F EXAMPLES
F.1
SPORTSIMAGE EXAMPLES
d
dEX
25
Question: there rule infraction in the image? there rule violation the image
Figure 26: Example of Q!
25
Question:
If rule infraction occurs the image, find and locate it, and output the coordinates.
[2072, 703, 2271, 833]
Figure 27: Example of Q5
27

---

Published as a conference paper at ICLR 2026
F.2 VIDEO EXAMPLES
Quitton
Explar why rute eftecton cnd in ald nM ocear tharpta??
Assarr The player crabilggt
to The narher lethrowing hall
wakt dues obrisen algitia weanee This Seoud choie incetiwally inmane paier is conide slend wrthiat oxnphiien lecauc pro
takme..
Figure 28: Example Q10
F.3 SPORTSVIDEO EXAMPLES
Question: infraction occurred Anewan
resulting penally?
It. the video, the grey reon stacking of the duee biue playors is is offiale poskion, as he neser to the asponcass goal line than both the sall and the player is Mus jeneyy who considerod as the scuand-l lot opponces. Acconting to foorball mlces player in efisidc pesstion when the bell played by toanzate is only penal and taey become activly unvoved the game by inter foring with play. inter foring with opponcni, gaining advaniage from heing the video, otttuee
the bloe team is dedendng Decing uttdds the stayer the grey which abead
position with as exception beiig made the player receives the hall from player ollide posétion his teaommate the Cherotoro,
who made delibenne cieerb atterapts auciret fize lick be awuidol.
Since
Figure 29: Example of QII
28


Reviews:
[{"review_id": "RYEATDSGEC", "reviewer": "Reviewer_AVw9", "summary": "This paper introduces SportR, a large-scale multimodal benchmark for evaluating and training MLLMs on fine-grained sports reasoning. The benchmark comprises 5,017 images and 2,101 videos across five sports (basketball, soccer, table tennis, badminton, American football), covering 50 foul types and 12 tactics. The key innovation is 7,118 fully human-annotated Chain-of-Thought (CoT) rationales and a novel visual grounding task requiring precise bounding box prediction. Experiments show that state-of-the-art models struggle significantly, with visual grounding IoU scores below 7% for baseline models.", "questions": "- What specific instructions were given to annotators for drawing bounding boxes? How was consensus reached on box boundaries? Examples in Fig. 18 show a single box - what about fouls involving multiple players?\n\n- For Q4 (free-form explanation), how is semantic similarity measured by LLM judges? What's the correlation with human judgment?\n\n- What's the split ratio? How many images/videos in test set? How was stratification across sports/foul types ensured?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper clearly articulates the gap between current sports benchmarks - either single-sport with detailed annotations OR multi-sport without fine-grained reasoning chains. SportR addresses both limitations simultaneously.\n\n- Fully human-authored CoT by 16 domain experts (including 2 NCAA Division I athletes) with rigorous quality control is a significant strength.\n\n- The coordinate-based grounding evaluation (Q5) is a novel visual grounding task.", "weaknesses": "- Missing human evaluation: No human agreement studies or inter-annotator reliability metrics for the evaluation itself\n\n- Visual grounding (Q5) only applies to SportsImage, not SportsVideo, and no temporal grounding annotations at all.\n\n- The evaluation use close source LLM as judge, the paper acknowledges self-preference bias but doesn't adequately address it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vCQlrO8cSx", "reviewer": "Reviewer_1rJ3", "summary": "This paper introduces SportR, a new large-scale multimodal benchmark designed to evaluate and train multimodal large language models on fine-grained, rule-based reasoning in sports. The benchmark covers both images and videos across five major sports, providing over 7,000 human-authored Chain-of-Thought (CoT) rationales and 20,000 structured question-answer pairs. It introduces hierarchical QA tasks that range from infraction identification to penalty prediction and visual grounding. The authors also evaluate a range of open and closed source MLLMs (e.g., GPT-5, Gemini 2.5 Pro, Qwen2.5-VL) and show that the benchmark is both challenging and effective as a training resource.", "questions": "1. How consistent were the CoT rationales across annotators? Even for two athletes with similar levels of experience, it seems likely their thought processes would heavily diverge. How important is this for training data quality?\n2. How sensitive are the results to the LLM-as-judge choice? Would human evaluation yield similar model rankings?\n3. Is there any evidence that models trained on SportR improve performance on non-sports multimodal reasoning benchmarks? This would support the idea that valuable skills are gained from learning this task, rather than being a case of sports related tasks not being common in existing training datasets.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Human-authored CoT rationales: The manual, expert-driven CoT annotations enhance the dataset’s reliability and interpretability, avoiding the noise of model-generated explanations. The listed procedures and training for annotators are rigorous and well justified.\n2. Comprehensive multimodal design: Covering both image and video modalities allows for evaluation of both spatial and temporal reasoning.\n3. Hierarchical QA framework: The progressive question design (from simple classification to reasoning and grounding) is well thought out and probes different layers of model understanding.\n4. Strong empirical evaluation: The authors present clear baselines across leading proprietary and open-source MLLMs and demonstrate meaningful improvements from fine-tuning and reinforcement learning.", "weaknesses": "1. Limited ablation and error analysis: The paper would benefit from deeper analysis of failure cases or qualitative examples that illustrate where models still fall short and potential reasons why. The authors successfully show that their tasks are difficult, but they don't provide insights into why the tasks are difficult for current MLLMs.\n2. Visual Localization True Difficultly: For the visual grounding tasks that require predicting bounding box coordinates, it is unclear whether the low accuracy stems from the model’s inherent difficulty in producing precise coordinates or from the underlying reasoning complexity of the task itself. It would be informative to know how performance changes if the prompt focuses on spatial relations instead of direct coordinate prediction, for example, asking “near which body part is the foul occurring?” rather than requesting explicit bounding boxes. Additionally, could the authors report bounding box prediction accuracy for simpler localization tasks that do not involve reasoning, such as identifying the player’s position on the field?\n3. LLM Evaluator Reliability: It remains unclear how reliable and stable LLM-as-a-judge evaluation is in this setting. How often do frontier models disagree with each other when serving as evaluators? Even for a small subset of samples, it would be helpful to report how frequently the LLM’s judgments diverge from expert human annotations. While perfect alignment is not expected, providing this comparison would give readers a sense of the evaluation’s variance and help contextualize the reported results with an appropriate error margin.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Aeno4tw1rV", "reviewer": "Reviewer_7hxG", "summary": "This paper introduces SportR, a new multi-sports benchmark designed to evaluate the reasoning capabilities required for sports intelligence. \nIt provides a dataset of images and videos with question-answer pairs and detailed reasoning chains to test a model's ability to perceive visual details, apply sports rules, and ground knowledge in visual evidence.\nExperiments show that current state-of-the-art models struggle significantly with the benchmark's most challenging tasks, highlighting a major gap in multimodal reasoning capabilities.", "questions": "See the Weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper features a progressive hierarchy of questions that systematically test reasoning depth—from simple identification to complex, multi-step tasks like penalty prediction.\n\n- The benchmark consists of 7,118 human-authored Chain-of-Thought (CoT) annotations for the most complex tasks, providing models with explicit examples of the required reasoning process.\n\n- Extensive experiments show that state-of-the-art models perform poorly on SportR's most difficult tasks.", "weaknesses": "- The author states that the paper is \"the first large-scale, multi-sport benchmark specifically designed to evaluate core reasoning capabilities.\" However, I know that there are actually some existing sports-domain datasets and benchmarks, such as \"Sportsu\". Therefore, what are the key differences between the dataset proposed in this paper and the existing datasets? Is it merely the addition of Chain-of-Thought (CoT) and grounding annotations?\n\n- As shown in Table 1 and Table 2, after SFT and SFT+RL, the model's performance significantly improves, even surpassing the zero-shot results of Gemini-2.5. This raises a question: is the model memorizing specific patterns from the training data, or has it acquired the ability to generalize to new scenarios (for example, a new match/game)? Furthermore, if a simple SFT+RL pipeline can enable the model to achieve a performance score of over 80, does this suggest that the task might not be challenging enough?\n\n- There seems to be no mention of the video attributes, such as duration, resolution, and FPS, which are crucial for video understanding tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WCOgAkS2xL", "reviewer": "Reviewer_m3qH", "summary": "(I tend to write shorter reviews and the length of the review does not reflect the quality of paper or the time spend on reviewing it).\n\nNote that I am not an expert in dataset creation, nor action benchmarks. I am reviewing this from a PoV of a deep learning research. \n\nThis paper introduces SportR, a large-scale, multi-sport benchmark designed to train and evaluate the fine-grained, rule-based reasoning of Multimodal Large Language Models (MLLMs). The authors argue that existing benchmarks lack the detailed reasoning chains and precise visual grounding needed to assess deep sports understanding. To address this gap, SportR provides a dataset of 5,017 images and 2,101 videos covering five sports, 50 foul types, and 12 tactics. The primary contributions are: (1) a progressive question-answering hierarchy complemented by 7,118 high-quality, human-authored Chain-of-Thought (CoT) annotations for complex tasks like penalty prediction; (2) the introduction of a novel explicit visual grounding task, the first for a multi-sport benchmark, which requires models to output precise bounding box coordinates for infractions ; and (3) extensive experiments showing that while state-of-the-art models perform poorly, training on SportR (via SFT and RL) yields significant gains and even demonstrates cross-modal generalization from image-based training to video task.", "questions": "See above. I defer to other reviewers for more questions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The dataset is solid along with the wide range of benchmarking across various current models. I am sure the benchmark will have a lot of utility for the community.", "weaknesses": "While I am not an expert, I see a few relevant papers that are missing from RW like. \n\nhttps://mrsalehi.github.io/action-atlas/\n\nhttps://ego-exo4d-data.org/\n\nAdding these things and on going general skill set understanding benchmarking to the background will help the readers contextualize the work better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)