Venue: NAACL 2024

# SportQA: A Benchmark for Sports Understanding in Large Language Models

Haotian Xia1, Zhengbang Yang1, Yuqing Wang2, Rhys Tracy3, Yun Zhao4, Dongdong Huang5, Zezhi Chen1, Yan Zhu5*, Yuan-fang Wang5, Weining Shen1*

Corresponding authors. _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 5061-5081 June 16-21, 2024 0204 Association for Computational Linguistics

###### Abstract

A deep understanding of sports, a field rich in strategic and dynamic content, is crucial for advancing Natural Language Processing (NLP). This holds particular significance in the context of evaluating and advancing Large Language Models (LLMs), given the existing gap in specialized benchmarks. To bridge this gap, we introduce SportQA, a novel benchmark specifically designed for evaluating LLMs in the context of sports understanding. SportQA encompasses over 70,000 multiple-choice questions across three distinct difficulty levels, each targeting different aspects of sports knowledge from basic historical facts to intricate, scenario-based reasoning tasks. We conducted a thorough evaluation of prevalent LLMs, mainly utilizing few-shot learning paradigms supplemented by chain-of-thought (CoT) prompting. Our results reveal that while LLMs exhibit competent performance in basic sports knowledge, they struggle with more complex, scenario-based sports reasoning, lagging behind human expertise. The introduction of SportQA marks a significant step forward in NLP, offering a tool for assessing and enhancing sports understanding in LLMs. The dataset is available at [https://github.com/haotianxia/SportQA](https://github.com/haotianxia/SportQA)

languageresourceLanguageLanguage

experience.

As shown in Figure 1, humans' understanding of sports can be divided into three levels of difficulty: factual and historical knowledge (Level-1), rules and tactics comprehension (Level-2), and complex sports-scenario analysis (Level-3). In our SportQA, level-1 understanding, covering 21,385 problems, allows humans to answer by recalling facts without needing additional sports expertise, such as knowing who won a specific Olympic event. Level-2, with 45,685 problems, suitable for sports enthusiasts, requires some expertise to understand diverse sports strategies and rules, like understanding soccer's offside rules. Level-3, encompassing 3,522 problems, is designed for sports specialists with years of experience, featuring more complex scenarios. One example under this level would be determining the best technique for a volleyball player facing three blockers during a spike in an amateur game. Level-1 and Level-2 problems are presented as multiple-choice questions with one correct answer each, while Level-3 problems follow a 'Multiple Select' format, allowing for one to four correct answers, and include both single and multi-hop questions in two levels of difficulty. Overall, three levels of questions collectively provide a comprehensive set to evaluate LLMs' sports understanding across a spectrum of expertise.

To gain deeper insights into the sports understanding challenges posed by SportQA, we extensively evaluated several popular recent LLMs, including Llama2 Touvron et al. (2023), PaLM2 Anil et al. (2023), GPT-3.5, and GPT-4 OpenAI (2023). The models were assessed through few-shot standard prompting, as well as chain-of-thought prompting Wei et al. (2022). Our findings show that GPT-4 outperforms in all levels, achieving an average accuracy of 82.16% in Level-1, 75% in Level-2, and 47.14% in level-3. However, in Level-3, the most challenging level, GPT-4's performance is about 45% inferior compared to human experts' accuracy in the same test questions, indicating that its understanding of sports significantly lags behind human capabilities, suggesting considerable scope for improvement in this area.

In summary, our contributions are threefold:

* We introduced SportQA, the first comprehensive dataset tailored for sports understanding in LLMs. It encompasses three levels of difficulty, aligned with human comprehension of sports. The dataset covers a diverse range of questions, from basic sports history to complex, scenario-based queries, thereby establishing itself as an essential tool for assessing the LLMs' sports understanding capabilities.
* We conducted extensive experiments on SportQA, evaluating recent LLMs' sports comprehension capabilities and gained insights into their strengths and areas for improvement through manual error analysis. This research has contributed to a better understanding of LLM performance within the context of sports NLP.

Figure 1: Illustration of three levels of understanding in sports.

 

3. We explored new directions for NLP in sports, underscoring its potential to enhance sports journalism and facilitate communication between athletes and coaches. This work not only broadens the use of NLP technologies but also lays the groundwork for the future integration of AI in sports-related fields. Furthermore, we made our datasets publicly available to benefit the research community.

## 2 Related Work

**Sports NLP.** Sports NLP is an emerging field, increasingly capturing interest due to its diverse applications, which range from sentiment analysis Baca et al. (2023); Ljajic et al. (2015) to game outcome predictions Beal et al. (2021); Xia et al. (2022); Oved et al. (2020); Tracy et al. (2023), and from generating game summaries Thomson et al. (2020); Huang et al. (2020) to augmenting sports videos with computer vision techniques Chen et al. (2022). Despite these advancements, a critical dimension remains underexplored: an in-depth understanding of sports in LLMs. Current applications primarily focus on analytics and do not delve into the complexities of sports understanding. Ensuring a deeper comprehension of sports in LLMs is crucial, as it can significantly broaden the scope and impact of NLP and LLM applications in the sports domain. This advancement is not only essential for enhancing current applications but also pivotal for exploring new avenues in Sports NLP.

**QA in Sports.** There have been several QA datasets involving testing LLMs reading and knowledge-based comprehension. Popular datasets include Trivia QA Joshi et al. (2017), HotpotQA Yang et al. (2018), QUASAR Dhingra et al. (2017); Talmor and Berant (2018), KQA Pro Cao et al. (2022) , and BoolQ Clark et al. (2019). However, among these datasets, sports-related QA is scarce. When sports topics are included, they usually focus on historical facts or well-known events, rather than the intricacies of sports rules, strategies, and real-time decision-making. This focus on factual recall rather than a deeper, strategic understanding of sports limits LLMs in demonstrating nuanced comprehension of this special domain.

Recent work Wei et al. (2022) has examined LLMs' sports understanding using BIG-bench's dataset on sports subtask (bench authors, 2023), focusing on distinguishing plausible from implausible sports statements. This dataset contains 986 questions that require knowledge of both the names of athletes and actions common in particular sports. However, understanding a sport involves more than merely matching names and activities. QASports Jardim et al. (2023) is a context extractive question-answering dataset focusing on soccer, American football, and basketball. It primarily involves answering questions based on specifically provided paragraphs focused on fact-based questions. Similarly, LiveQA Liu et al. (2020) challenges LLMs with basketball-related questions derived from play-by-play broadcast passages, focusing on understanding basketball from given paragraphs. Typical question types include timeline assessment, event tracking, and mathematical computations, such as 'Which team will make a three-pointer first in the second quarter?" While this tests comprehension within a specific context, it does not extensively cover other crucial aspects of sports understanding, such as strategy comparisons. The common drawback of these sports QA datasets is their focus on surface-level insights, such as basic facts and famous activities. True sports understanding involves rules, gameplay, and tactics. In contrast, our SportQA benchmark provides a comprehensive scope, covering a wide range of sports subjects, from facts and history to rules and complex scenarios. Moreover, our dataset is able to assess LLMs' intrinsic sports knowledge beyond just finding information in the provided paragraphs. We aim to address the limitations of existing datasets, fostering a deeper understanding of sports.

**Paradigms in LLMs Training.** Pre-trained language models on diverse texts, exemplified by BERT Devlin et al. (2019) has been a cornerstone in NLP. These models have been effectively used in tasks ranging from disease prediction Zhao et al. (2021) to text classification Wang et al. (2022b). However, the introduction of GPT-3 Brown et al. (2020) marked a paradigm shift from extensive task-specific fine-tuning to zero-shot and few-shot learning approaches, enabling adaptation to new tasks with minimal training. This transition has led to the development of advanced prompting techniques to enhance LLMs' understanding and reasoning abilities. Representative examples include chain-of-thought (CoT) prompting Wei et al. (2022), zeroshot CoT prompting Kojima et al. (2022), selfconsistency Wang et al. (2022a), Tree-of-Thought prompting Yao et al. (2023), and metacognitive prompting Wang and Zhao (2023). This evolution has necessitated a focus on the latest advancements in LLMs. Consequently, our study concentrates on 

evaluating recent LLMs, such as Llama2 (Touvron et al., 2023), PaLM2 (Anil et al., 2023), GPT-3.5, and GPT-4 (OpenAI, 2023) in diverse sports understanding tasks to explore their capabilities.

## 3 Sports Understanding Benchmark

We introduce SportQA, a benchmark specifically designed to evaluate the understanding of sports for LLMs. Developed in close collaboration with sports experts, questions in SportQA consist of three main levels of difficulty, each focusing on a distinct aspect of sports comprehension.

Level-1: Foundational Sports Knowledge.Questions in this level assess basic sports knowledge, focusing on factual recall and historical information. It provides a foundation for understanding the broader context of sports within LLMs.

Level-2: Rules and Tactics Comprehension.This level's questions mainly evaluate LLMs' understanding of sports rules and strategies. Covering 35 distinct sports types, it tests the models' ability to interpret game rules and employ strategic thinking, offering deeper insights into sports comprehension. Additionally, it includes more complex fact-based historical questions to ensure a well-rounded assessment of sports knowledge.

level-3: Advanced Scenario-Based Understanding.In this level, questions focus on evaluating the synthesis and integration of diverse sports knowledge. These involve complex, scenario-based questions that replicate real-world sports situations, requiring deep comprehension and advanced analytical thinking from LLMs.

Overall, the benchmark comprises a total of 70,592 questions. For each dataset and level, we provide a few-shot development set consisting of 5 questions per task, along with a separate test set for a thorough evaluation. Table 1 provides a detailed overview of these levels and tasks, with more extensive details available in Appendix A. We employ accuracy as the evaluation metric across all tasks, ensuring a clear and objective assessment of LLMs' sports understanding capabilities. Specifically, we employ holistic accuracy for the Level 3 multi-hop experiment, where an entire question is marked as incorrect if the model fails to answer any component, main or sub-question, correctly.

### Dataset Construction and Quality Verification

Dataset Construction.We constructed the SportQA dataset through a hybrid approach. For Level-1 and -2 problems, we combined automated templates with expert-driven modifications to create a diverse range of questions, ensuring both consistency and comprehensive coverage. For the more complex level-3 questions, we exclusively used manual question creation by experienced sports experts. This dual approach, integrating both automated and manual processes, was instrumental in crafting questions that accurately reflect the multifaceted and intricate aspects of sports knowledge.

Quality Verification.To ensure the integrity and accuracy of SportQA, a meticulous manual review process was essential due to the complexity of sports knowledge. This task was carried out by a highly skilled team of 36 intercollegiate student-athletes from both the US and China, each with a minimum of 8 years of sports training experience. Their extensive experience and expertise in various sports were crucial in evaluating the context and accuracy of each question. Their deep understanding of sports rules, strategies, and contexts ensured that every question in the dataset was not only factually accurate but also relevant and challenging.

During the recruitment phase, each potential team member was interviewed using ten examples for every level of questions and asked to compose example Q&A pairs. After hiring, each member first goes through a training session to learn the task and the annotation process. Once they fully master the annotation process, we launch the official batches for them to work on.

### Level-1: Foundational Sports Knowledge

Level-1 problems aim to assess LLMs' proficiency in foundational sports knowledge, focusing on factual recall and historical information. This level includes a total of 21,385 multiple-choice questions, sourced from a variety of QA datasets. As these datasets have different Q&A formats (true/false, multiple choices, and open-ended narratives), efforts were spent to normalize the formats and check for correctness and relevance.

Questions from Trivia QA Joshi et al. (2017), QUASAR Dhingra et al. (2017), and Hotpot QA Yang et al. (2018) were originally in open-answer format and have been adapted for this benchmark. For BoolQ Clark et al. (2019), which 

primarily featured true/false questions, we adapted these into 2-way multiple-choice formats. KQA Pro Cao et al. (2022) questions, already in a multiple-choice format, were also included and checked for accuracy and relevance. The transformation of open-answer questions into multiple-choice format followed by automated template generation or manual refinement.

**Step 1: Automated Template Generation.** For each answer type (sport, time, country/location, person, numbers, etc.), we utilized different semantic libraries. For example, (1) Sports: If the answer was a sport, other sports were automatically selected as distractors. (2) Time: Adjacent periods were chosen for time-based answers. (3) Country/Location: Similar geographical locations were used as distractors. (4) Person: A specialized semantic library of names was created for each sport to gather information about athletes, and this library was then used to select distractors.

**Step 2: Manual Refinement.** Questions requiring more nuanced distractors or falling outside these categories were further refined by our review team of 36 student-athletes, who manually crafted additional misleading options.

Level-1 questions establish the foundation for sports knowledge. However, due to the limited scope and specificity of sports in existing datasets, level-2 is introduced to expand this scope. It addresses the need for professional knowledge of sports, such as rules and strategies.

### level-2: Rules and Tactics Comprehension

level-2 problems are designed to conduct an indepth assessment of LLMs' understanding of sports rules, tactics, and an extended range of historical and factual knowledge. This level comprises 45,685 questions, covering a broad spectrum of sports disciplines.

**Step 1: Content Categorization and Annotation.** The review team categorized content from Wikipedia related to 35 distinct sports tasks, including the 28 Olympic sports and their variation, the four new sports (Breaking, sport climbing, skateboarding, and surfing) debuting in the 2024 Paris Olympics, and popular non-Olympic sports like baseball, ice hockey, and American football. The focus was on compiling sources primarily about rules and tactics, supplemented with historical and factual content for comprehensive coverage. They performed categorization and annotation of content from Wikipedia, identifying and annotating critical information points within each sport's rules, tactics, and historical contexts.

**Step 2: Hybrid Question Development and Templates.** Our approach to question development was twofold. First, we employed a set of predefined templates to ensure consistent coverage across various sports, and examples of these templates are illustrated in Appendix A. Furthermore, for scenarios that required deeper exploration beyond these templates, we manually crafted questions, ensuring a rich and diverse set of queries based on the annotation of content and knowledge points.

**Step 3: Distractor Generation and Categoriza 

tion.** The distractor generation for SportQA was a nuanced process. Automated distractors for template-based questions were created using a specialized, sport-specific library, ensuring each was relevant and challenging. This library, organized by sport, included diverse categories such as athletes' names and tactics. For questions crafted manually, our team of experts meticulously developed plausible yet incorrect distractors, adding depth and complexity to the dataset.

**Step 4: Expert Review and Alignment.** The review team ensured each question's alignment with the source content, verifying the efficacy of questions and eliminating outdated or irrelevant information.

By following the above steps, level-2 problems not only offer a comprehensive evaluation of sports rules and tactics but also enrich the dataset with an expansive array of sports history and factual knowledge, enhancing the overall utility of SportQA.

### level-3: Scenario-Based Questions

level-3 problems are a comprehensive and nuanced component of our benchmark, comprising 3,522 scenario-based questions across six key sports. This level includes football (soccer), basketball, volleyball, tennis, table tennis, and American football. Each sport features a set of both multi-hop and single-hop questions with multiple-select types (one to four correct answers), meticulously categorized into easy and hard difficulty levels. The categorization is based on the complexity of the scenarios and the number of integrated knowledge points required to answer each question. This results in four distinct tasks per sport, 24 tasks in total, closely mirroring the intricacies and complexities of real-world sports situations.

The requirement for manual question creation in level-3 is rooted in the sophisticated and deep understanding of each sport needed to craft these questions. This level's complexity demands not just a surface-level acquaintance with the sports but an in-depth, experiential knowledge, often only possessed by those who have actively participated or been deeply involved in the sport. Such profound insight is crucial for developing questions that accurately capture the nuances, strategic elements, and practical applications inherent in real-world sports scenarios. This elevates level-3 above Levels 1 and 2, making it a more advanced and comprehensive test of sports understanding and analysis.

**Step 1: Expert Proposed Assessment Angle.** We first invited coaches to propose the assessment angles, which are further detailed in Appendix A, for these sports based on their expertise. Their extensive coaching experience and mastery understanding of sports ensure that the knowledge points behind each question are meaningful and effective for the sports.

**Step 2: Manual Questions Generation.** After coaches proposed the assessment angles across all 6 sports, these angles were used by our review team, combining their training and competition experience with their unique advanced understanding and expertise in sports, to develop questions. This ensures a comprehensive assessment of LLMs' sports understanding and analytical skills.

The distribution of questions per sport, shown in Appendix A, is directly influenced by the availability and expertise of our review team. Sports with more expert reviewers are represented with a larger number of questions, ensuring the authenticity and depth of the content. This alignment guarantees that the questions are not only challenging but also reflect the dynamic nature of each sport.

The manual creation process of level-3 emphasizes the professional rigor and subject-matter expertise that underpin SportQA, making it an exceptional measure of sports knowledge.

## 4 Experiments

This section evaluates the performance of prevalent LLMs on our SportQA benchmark. We aim to assess the efficacy of these models in understanding complex sports scenarios across multiple tasks. We present the optimal result, derived from multiple iterations for each experimental condition.

### Experimental Setup

We evaluated several leading language models on the SportQA benchmark, including open-source models like Llama2-13b-chat Touvron et al. (2023), and closed-source models such as PaLM-boson-chat Anil et al. (2023), GPT-3.5-turbo, and GPT-4 OpenAI (2023). Access to these models was facilitated through their respective APIs.

For Level-1, we randomly selected 2000 questions from the test set. For level-2, our sampling strategy varied based on the number of questions per sport: 30% of all questions for sports with fewer than 200 questions, 15% for 200-800, 5% for 800-1500, 2.5% for 2500-10,000, and 1.5% for more than 10,000 questions, totaling 2243 questions. In 

level-3, the test sample size was determined based on the total questions of sports, with 20% for soccer, basketball, and tennis, 30% for volleyball, and 50% for table tennis and American football, a total of 980 questions.

In our study, we primarily focused on the CoT prompting method for model evaluation. This decision was informed by the evidence Wei et al. (2022), which highlights the effectiveness of few-shot CoT in sports understanding contexts. The CoT approach, involving a step-by-step reasoning process, is particularly suitable for complex sports understanding tasks. We also considered the zero-shot CoT method Kojima et al. (2022) and few-shot standard prompting (SP) Brown et al. (2020) as additional prompting baselines for evaluation. In the few-shot setting (CoT or SP), we take 5 exemplars for each task, where the answers to these demonstrations are annotated by human experts. Exemplars were consistently drawn from the development set, with the temperature parameter set to 0 to ensure consistent response generation. Additional details about the prompts we used are available in Appendix B.

In addition to model performance evaluation, we engaged student-athletes not part of the review team, each specializing in one of the sports covered in level-3. These athletes were tasked with manually answering the test set of level-3. This effort, involving each expert in their respective sport, was undertaken to better understand human performance on these complex, scenario-based questions. Their contributions provide valuable insights into the realistic expectations of human expertise in sports understanding and serve as a standard for expert-level accuracy.

### Overall Performance Comparison

We compared the performance of different models across all tasks in three different levels, and the overall result is shown in Table 2. Results for individual tasks are shown in Appendix C. There are several key takeaways. First, GPT-4 consistently outperforms other models across all tasks, demonstrating a performance advantage of over 15% compared to other models on average. Regarding prompting effectiveness, we note that CoT often results in performance enhancements, which corroborates the findings from Wei et al. (2022), emphasizing the efficacy of step-by-step prompting with few exemplars in augmenting LLMs' performance in intricate reasoning tasks. Each model, including GPT-4, consistently exhibited the best performance in Level-1, with a gradual decline in accuracy moving to Levels 2 and 3. This trend aligns with the increasing complexity and sophistication designed for each successive level of our benchmark. Level-1, focusing on foundational knowledge, proved to be the most accessible, while level-3, with its advanced scenario-based tasks, posed the greatest challenge.

Finally, while GPT-4 leads among all the models, human expertise still exceeds it by roughly 30% - 65% in different tasks of level-3, highlighting the complexity of these sports understanding tasks and indicating room for future improvements in LLMs.

### Error Analysis

To better understand the mistakes made by models, we randomly chose and manually analyzed 20 instances from each level where a model made an incorrect choice. We prompted the model to explain its decisions, then reviewed these explanations to identify errors, understand the reasons behind them,

\begin{table}
\begin{tabular}{l c c c c c c} \hline

- & 96.63 & 96.02 & 94.90 & 91.84 \\ \hline \end{tabular}
\end{table}
Table 2: Performance comparison of each model across three levels in SportQA. GPT-4 consistently outperforms other models under both zero-shot (0S) and 5-shot (5S) settings across all levels. Human performance serves as an upper bound, illustrating that there still exists room for improvement in LLMs on sports understanding tasks.
 

and categorize them into specific error types. The error analysis for Levels 1 and 2 in SportQA is closely related to the fundamental nature of these levels, which focus on factual knowledge and basic rule comprehension. We identify the following error types which align with the core challenges of recalling specific sports facts, understanding basic rules, and applying simple strategies: Deficiency in Conceptual Understanding, Misuse of Known Information, and Inaccuracies in Factual Recall.

For level-3 of SportQA, given its unique question types involving scenario-based, multi-knowledge point questions with both single and multi-hop reasoning, we devised specific error categories: Conceptual Misunderstanding, Logical Reasoning Error, and Contextual Misinterpretation.

For this analysis, we focused on GPT-4. Figure 2 shows the error types and their respective proportions for each level group. Within level-1 and level-2, "deficiency in conceptual understanding " was the most frequent error, accounting for 40% of all mistakes. In level-3 of the SportQA analysis, During the analysis process, we noticed that 82.5% of incorrect questions have inadequacy in multifaceted answer identification. Within this, the greatest error category appears to be conceptual misunderstanding, prevalent in both single-hop and multi-hop scenarios, constituting 55% and 50% of the errors, respectively. For instance, there is an error in distinguishing the difference between "referee" and "mediator" in the scenario-based questions. This illustrates the model often fails to grasp the concepts that are combined with scenarios necessary for these complex questions. Detailed descriptions of each error type, along with examples and case studies of CoT incorrect outputs and their error analysis, are available in Appendix D.

## 5 Conclusion

This study has introduced SportQA, a benchmark for assessing the understanding of sports in LLMs. Unlike previous benchmarks that primarily focused on basic fact-recall or simplistic sports-related queries, SportQA delves deeper into the intricacies of sports knowledge, spanning historical facts, rules, strategies, and scenario-based reasoning. Our evaluations reveal that while current LLMs like GPT-4 show promising capabilities in foundational knowledge and rule comprehension, their performance in complex, scenario-based reasoning remains a challenge and still lags behind human expertise. The results underscore the need for ongoing advancements in NLP and AI to achieve a deeper and more nuanced understanding of sports. Improvements to logical reasoning and in-depth understanding in the context of sports will be a gateway toward allowing Large Language Models to improve their performance and adaptability across a large range of real-world, diverse, and everchanging domains. SportQA serves as an essential tool for future research in this area, offering a structured framework to measure and enhance LLMs' capabilities in sports understanding.

### Limitation

While SportQA presents a broad assessment of sports understanding, we acknowledge its limitations. The primary limitation lies in the complexity of creating scenario-based questions for level-3. Due to the intricate nature and high standards required for these questions, the volume of questions and the range of sports it covers at this level is limited compared to other levels. We are dedicated to continually updating and expanding the dataset to enhance its scope and depth.

Another significant limitation of SportQA is its

Figure 2: Error type distributions across different levels of SportQA analysis, with level-3 separated into ’SingleHop’ and ‘Multi-Hop’ reasoning to highlight specific error trends. In levels 1 and 2, the model often struggles with the details of concepts, and these issues become amplified in level-3. This amplification of errors leads to more significant problems, resulting in worse performance of models in the more complex scenarios of level-3.

 

focus predominantly on rules and gameplay aspects of sports understanding. Critical elements such as sports medicine and psychology, which are integral to a comprehensive understanding of sports, are not currently covered in the benchmark. These areas require specialized medical and psychological knowledge, making their integration into the benchmark complex and demanding. This gap in content underscores the need for a more interdisciplinary approach in future iterations of SportQA, to encompass a wider spectrum of sports knowledge and understanding and the need to recruit team members with a more diverse background.

Furthermore, while we aimed to include a variety of LLMs in our analysis, budgetary constraints limited our ability to evaluate certain high-capacity models like the open-source Llama2-70b-chat (Touvron et al., 2023). This represents a limitation in our current study, as including a wider range of models could potentially offer more insights into their respective capabilities in sports understanding tasks. In future work, we plan to extend our model evaluation to encompass a broader spectrum of LLMs, ensuring a more comprehensive analysis.

## Acknowledgements

We would like to thank Weixi Feng for his helpful discussion. Moreover, we thank anonymous reviewers for their valuable suggestions.

## References

* Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. _arXiv preprint arXiv:2305.10403_.
* Baca et al. (2023) Luis Baca, Natali Ardiles, Jose Cruz, Wilson Mamani, and John Capcha. 2023. Deep learning model based on a transformers network for sentiment analysis using nlp in sports worldwide. In _Advances in Computing and Data Sciences_, pages 328-339. Springer Nature Switzerland.
* Beal et al. (2021) Ryan Beal, Stuart E Middleton, Timothy J Norman, and Sarvapali D Ramchurn. 2021. Combining machine learning and human experts to predict match outcomes in football: A baseline model. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 15447-15451.
* BIG bench authors (2023) BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_.
* Brown et al. (2022) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantam, Franav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Cao et al. (2022) Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022. KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6101-6119, Dublin, Ireland. Association for Computational Linguistics.
* Chen et al. (2022) Zhutian Chen, Qisen Yang, Xiao Xie, Johanna Beyer, Haijun Xia, Yingcai Wu, and Hanspeter Pfister. 2022. Sportshesia: Augmenting sports videos using natural language. _IEEE transactions on visualization and computer graphics_, 29(1):918-928.
* Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2924-2936, Minneapolis, Minnesota. Association for Computational Linguistics.
* Cong et al. (2023) Xin Cong, Bowen Yu, Mengcheng Fang, Tingwen Liu, Haiyang Yu, Zhongkai Hu, Fei Huang, Yongbin Li, and Bin Wang. 2023. Universal information extraction with meta-pretrained self-retrieval. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4084-4100, Toronto, Canada. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dhingra et al. (2017) Bhuwan Dhingra, Kathryn Mazatiis, and William W. Cohen. 2017. Quasar: Datasets for question answering by search and reading.
* Ding et al. (2023) Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11173-11195, Toronto, Canada. Association for Computational Linguistics.
* Fei et al. (2023) Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, and Tat-Seng Chua. 2023. Reasoning implicit sentiment 

with chain-of-though prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Trento, Canada, pp. 1171-1182. External Links: Link Cited by: SS4.
* K. Huang, C. Li, and K. Chang (2020)Generating sports news from live commentary: a chinese dataset for sports game summarization. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 690-615. Cited by: SS3.
* P. Calciolari Jardim, L. Mauro Pereira Moraes, and C. Dutra Aguiar (2023)Qasports: A question answering dataset about sports. In Anais do V Dataset Showcase Workshop, SBC, Cited by: SS4.
* M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer (2017)TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 1601-1611. External Links: Link Cited by: SS4.
* T. Kojima, S. Shane Gu, M. Reid, Y. Matsuo, and Y. Iwasawa (2022)Large language models are zero-shot reasoners. Advances in neural information processing systems35, pp. 22199-22213. External Links: Link, Document Cited by: SS4.
* T. Li, X. Ma, A. Zhuang, Y. Gu, Y. Su, and W. Chen (2023)Few-shot in-context learning on knowledge base question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 6966-6980. External Links: Link, Document Cited by: SS4.
* Q. Liu, S. Jiang, Y. Wang, and S. Li (2020)Liveqa: A question answering dataset over sports live. In Proceedings of the 19th Chinese National Conference on Computational Linguistics, Trento, Canada, pp. 1057-1067. External Links: Link, Document Cited by: SS4.
* A. Ljajic, F. Ljajic, P. Spalevic, B. Arsic, and D. Vuckovic (2015)Sentiment analysis of textual comments in field of sport. In 24nd International Electrotechnical and Computer Science Conference (ERK 2015), Slovenia, Cited by: SS4.
* OpenAI (2023)Gpt-4 technical report.. Cited by: SS4.
* N. Oved, A. Feder, and R. Reichart (2020)Predicting in-game actions from interviews of nba players. Computational Linguistics46 (3), pp. 667-712. External Links: ISSN 0021-909X, Link, Document Cited by: SS4.
* A. Talmor and J. Berant (2018)The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 641-651. External Links: Link, Document Cited by: SS4.
* C. Thomson, E. Reiter, and S. Sripada (2020)Sportsett: basketball-a robust and maintainable data-set for natural language generation. In Proceedings of the Workshop on Intelligent Information Processing and Natural Language Generation, pp. 32-40. External Links: Link, Document Cited by: SS4.
* H. Tourvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023)LIMA 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: SS4.
* R. Tracy, H. Xia, A. Rasla, Y. Wang, and A. Singh (2023)Graph encoding and neural network approaches for volleyball analytics: From game outcome to individual play predictions. arXiv preprint arXiv:2308.11142. Cited by: SS4.
* X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdherry, and Denny Zhou (2022)Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Cited by: SS4.
* Y. Wang and Y. Zhao (2023)Metacognitive prompting improves understanding in large language models. arXiv preprint arXiv:2308.05342. Cited by: SS4.
* Y. Wang, Y. Zhao, R. Callcut, and L. Petzold (2022)Integrating physiological time series and clinical notes with transformer for early prediction of sepsis. arXiv preprint arXiv:2203.14469. Cited by: SS4.
* J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022)Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems35, pp. 24824-24837. External Links: Link, Document Cited by: SS4.
* H. Xia, R. Tracy, Y. Zhao, E. Fraisse, Y. Wang, and L. Petzold (2022)Vren: Volleyball rally dataset with expression notation language. In 2022 IEEE International Conference on Knowledge Graph (ICKG), pp. 337-346. External Links: ISBN 978145036611, Link, Document Cited by: SS4.
* Zhilin Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018)HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2369-2380. External Links: Link, Document Cited by: SS4.
* S. Yao, D. Yu, J. Zhao, Izhak Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan (2023)Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. Cited by: SS4.
* R. Zhao, X. Li, S. Joty, C. Qin, and L. Bing (2023)Verify-and-edit: A knowledge-enhanced chain-of-thought framework. In Proceedings of the 61st Annual Meeting of the 

Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5823-5840, Toronto, Canada. Association for Computational Linguistics.
* Zhao et al. [2021] Yun Zhao, Yuqing Wang, Junfeng Liu, Haotian Xia, Zhenni Xu, Qinghang Hong, Zhiyang Zhou, and Linda Petzold. 2021. Empirical quantitative analysis of covid-19 forecasting models. In _2021 International Conference on Data Mining Workshops (ICDMW)_, pages 517-526. IEEE.

## Appendix A Detailed Question Distribution and Sample Questions for SportQA

This appendix provides an in-depth look at the question distribution across different sports and question types within SportQA, as well as illustrative examples of the types of questions featured in the dataset.

### level-2 Question Distribution Across Sports

A comprehensive table and corresponding figures illustrate the number of questions per sport, categorized by the type of knowledge they assess -- historical facts, rules, or strategic understanding. The distribution of the number of questions between 'Rule or Strategy' and 'History or Fact' is shown in Figure 3. The detailed distributions for each task in level-2 are shown in Figure 5 and Table 4.

### Question Templates

The example of templates we used to generate level-2 questions is shown in Table 3.

### level-3 Assessment Angles

This section presents an example of assessment angles, shown in Table 7, for level-3, as proposed by sports experts. These angles are designed to challenge Large Language Models (LLMs) with realistic and complex sports scenarios that require an advanced understanding of game dynamics, rules, and strategies. For a detailed exposition of these angles within the context of basketball, please refer to Table 7.

### Sample Questions

The sample questions for each level are shown in Figure 4.

## Appendix B Prompting Methods

We utilize both standard Prompt (SP) and CoT in our experiments with LLMs. For SP, questions are presented without additional steps in the prompt. For CoT, zero-shot prompting is inspired by Kojima et al. [2022], instructing the model to "Let's think step by step". For few-shot CoT, we manually craft the step-by-step process for 5-shot exemplars in the development set. Examples of Few-shots and CoT prompts in our experiments are included in Table 8 - 12.

## Appendix C Additional Results on GPT-4

The GPT-4 Performance of individual tasks in level-2 and level-3 are shown in Table 5 and Table 6. We notice 5-Shot CoT prompting has the best performance in most of the tasks.

 

but provides incorrect explanations or conclusions or applies a general rule or concept to a specific sports scenario, leading to inaccurate answers. Incorrect Fact Recall: The model recalls and uses a fact that is either incorrect or not applicable to the given context, despite understanding the general concept.

**level-3 error Categories.** Conceptual Misunderstanding: Errors where the model fails to grasp essential concepts required for answering the questions. Logical Reasoning Error: Mistakes in linking multiple steps of reasoning (For example, score calculation). Contextual Misinterpretation: Instances where the model misinterprets the scenario or context, leading to incorrect conclusions.

Figure 4: Example questions for SportQA





\begin{table}
\begin{tabular}{c c c c c c} \hline  & **0-Shot** & **5-Shot** & **5-shot** \\
2-3 & **CoT (\%)** & **SP (\%)** & **CoT (\%)** \\ \hline Water Polo & 66.67 & 78.26 & **81.15** \\ Baseball & 68.24 & 75 & **80.26** \\ Rugby or American Football & 69.51 & **75.6** & 73.17 \\ Athletics & 71.05 & 73.91 & **78.26** \\ Cycling & 70.37 & 79.62 & **81.48** \\ Table Tennis & 72.5 & 82.5 & **82.56** \\ Baltimore & 65.11 & 76.74 & **79.07** \\ Football (Soccer) & 67.16 & 85.07 & **85.07** \\ Sailing & 61.9 & 74.6 & **79.36** \\ Ice Hockey & 66.66 & **80** & 76 \\ Diving & 51.78 & 76.78 & **78.57** \\ Tennis & 77.42 & 82.25 & **88.7** \\ Handball & 74.13 & **87.93** & 84.48 \\ Weightlifting & 84.37 & 87.5 & **87.5** \\ Equestrian & 36.95 & **63.04** & 50 \\ Surring & 70.27 & 75.67 & **81.08** \\ Hockey & 64.51 & 67.74 & **74.19** \\ Judo & 72.5 & 77.5 & **77.5** \\ Sport Climbing & 85 & 95 & **95** \\ Volleyball & 69.35 & **74.19** & 72.58 \\ Gymnastics & 60.97 & 73.17 & **75.6** \\ Modern Pentathion & 77.77 & 66.67 & **94.44** \\ Takekwood & 71.69 & 79.24 & **86.79** \\ Archery & 80.55 & **83.33** & 80.55 \\ Canoeing & 80.52 & 72.73 & **80.52** \\ Shooting & 70 & 76.67 & **84.44** \\ Wrestling & 74 & 78 & **78** \\ Boxing & 71.42 & **84.28** & 80 \\ Triathlon & 72 & **80** & 78 \\ Breaking & 76.74 & 83.72 & **86.46** \\ Golf & 64.61 & **84.61** & 83.07 \\ Basketball & 69.72 & 71.55 & **72.47** \\ Fencing & 71.42 & 71.42 & **80.35** \\ Skateboarding & 60.78 & 60.78 & **66.67** \\ Swimming & 69.52 & 69.52 & **70.47** \\ \hline \end{tabular}
\end{table}
Table 4: Total Number of Questions for Each Sports Task in level-2

\begin{table}
\begin{tabular}{l|c c c} \hline Sport & Total Questions & **S-Shot** & **5-shot** \\ Canoeing & 415 & \\ Fencing & 382 & \\ Sport Climbing & 72 & \\ Surfing & 257 & \\ Breaking & 149 & \\ Cycling & 1102 & \\ Equestrian & 159 & \\ Golf & 2605 & \\ Boxing & 1423 & \\ Ice Hockey & 3025 & \\ Wrestling & 170 & \\ Archery & 127 & \\ Swimming & 710 & \\ Water Polo & 465 & \\ Hockey & 417 & \\ Athletics & 1398 & \\ Basketball & 4400 & \\ Baseball & 512 & \\ Taekwondo & 182 & \\ Table Tennis & 273 & \\ Badminton & 149 & \\ Modern Pentathlon & 65 & \\ Shooting & 606 & \\ Tennis & 2526 & \\ Judo & 810 & \\ Hundball & 391 & \\ Diving & 191 & \\ Sailing & 430 & \\ Tirathlon & 172 & \\ Vilebyball & 1263 & \\ Weightlifting & 115 & \\ Football (soccer) & 8062 & \\ Rugby \& American football & 10956 & \\ \hline \end{tabular} 
\begin{tabular}{c c c c c} \hline \multirow{2}{*}{**Sports**} & **Prompt** & **Easy** & **Hard Single- Easy Multi-Multi-hop** \\  & **Setting** & **Single-hop** & **hop Tasks** & **hop Tasks** & **Task** \\  & **(\%)** & **(\%)** & **(\%)** & **(\%)** \\ \hline \multirow{6}{*}{American Foodhall} & Zero-shot CoT & 75 & 56.25 & 39.39 & 21.88 \\  & 5-Shot SP & 81.25 & **71.88** & 39.39 & 28.13 \\  & 5-Shot CdTe CP & **84.38** & 68.75 & **42.42** & **31.25** \\  & Zeno-shot CoT & 77.42 & 73.33 & 33.87 & 22.95 \\  & 5-Shot SP & 80.65 & 78.33 & 37.1 & 24.59 \\  & 5-Shot CdTe CP & **83.87** & **81.67** & **37.1** & 22.95 \\  & 5-Shot CP & 58.54 & 42.86 & 22.73 & 21.62 \\  & 5-Shot SP & 73.17 & 57.14 & 20.45 & 18.92 \\  & 5-Shot CPT & **75.61** & **59.52** & **27.27** & **21.62** \\  & 5-Shot CDF & 65.91 & 56.62 & 22.72 & 22.92 \\ Basketball & 5-Shot SP & 61.36 & 59.09 & 27.27 & **25** \\  & 5-Shot CC & **65.91** & **59.09** & **29.55** & 20.83 \\  & 5-Shot CP & 67.86 & 48.15 & 42.86 & 26.92 \\  & 5-Shot SP & 71.43 & 59.26 & 32.14 & **34.62** \\  & 5-Shot CCT & **71.43** & **62.96** & **32.14** & 30.77 \\  & 5-Shot CC & 53.85 & 42.5 & 30.77 & **20** \\ Tennis & 5-Shot SP & 53.85 & **47.5** & **46.15** & 20 \\  & 5-Shot CoT & **56.41** & 45 & 38.46 & 14.29 \\ \hline \end{tabular}
\end{table}
Table 5: GPT-4 Performance of each task in Level-2

* [15] M. C. Bader, J. A. Harvey, and R. H. S. Heath, "The role of the game in a game: The game's guide to games," _Journal of Game Theory_, vol. 34, no. 2, pp. 70-86, Jan 1984.

[MISSING_PAGE_POST]

 



**Question:** The 1932 German football championship Final was played at a stadium that opened in what year?

A) 1912 B) 1928 C) 1930 D) 1932

**Answer:** B

**Question:** How many events did the Nordic Combined consist of in the 1948 Olympics? A) Two events B) Three events C) One event D) Four events

**Answer:** C

**Question:** Who succeeded David Stern to become the Commissioner of the NBA? A) Richard Parsons B) Adam Silver C) Gary Bettman D) Roger Goodell

**Answer:** B

**Question:** Which sport is played on a variable ground ranging from 50x100yd minimum to 100x130yd maximum?

A) Baseball B) Association Football (Soccer) C) Basketball D) Volleyball

**Answer:** B

**Question:** In American Football, how many points does a touchdown score?

A) Six B) Three C) Four D) Five

**Answer:**

**Q:** The 1932 German football championship Final was played at a stadium that opened in what year?

A) 1912 B) 1928 C) 1930 D) 1932

**Answer:** Choices A, C, D are wrong since these years are irrelevant to the stadium where German football championship is. Hence, the correct answer is B.

**Q:** How many events did the Nordic Combined consist of in the 1948 Olympics?

A) Two events B) Three events C) One event D) Four events

**Answer:** In 1948 Olympics, the Nordic Combined consisted of only one event. Hence, the correct answer is C.

**Q:** Who succeeded David Stern to become the Commissioner of the NBA? A) Richard Parsons B) Adam Silver C) Gary Bettman D) Roger Goodell

**Answer:** David Stern served as the Commissioner of the NBA from 1984 until February 1, 2014. Adam Silver succeeded David Stern to become the Commissioner of the NBA. Hence, the correct answer is B.

**Q:** Which sport is played on a variable ground ranging from 50x100yd minimum to 100x130yd maximum? A) Baseball B) Association Football (Soccer) C) Basketball D) Volleyball

**Answer:** Choices A, C, D are wrong because their court size is all less the 50*100 yards. Hence, the correct answer is B.

**Q:** In American Football, how many points does a touchdown score?

A) Six B) Three C) Four D) Five

**Answer:**Based on the fact of the rules of American football, a touchdown scores 6 points. Hence, the correct answer is A.

\begin{table}
\begin{tabular}{l l}
**Question:** The 1932 German football championship Final was played at a stadium that opened in what year?

A) 1912 B) 1928 C) 1930 D) 1932

**Answer:** Choices A, C, D are wrong since these years are irrelevant to the stadium where German football championship is. Hence, the correct answer is B.

**Q:** How many events did the Nordic Combined consist of in the 1948 Olympics?

A) Two events B) Three events C) One event D) Four events

**Answer:** In 1948 Olympics, the Nordic Combined consisted of only one event. Hence, the correct answer is C.

**Q:** Who succeeded David Stern to become the Commissioner of the NBA? A) Richard Parsons B) Adam Silver C) Gary Bettman D) Roger Goodell

**Answer:** David Stern served as the Commissioner of the NBA from 1984 until February 1, 2014. Adam Silver succeeded David Stern to become the Commissioner of the NBA. Hence, the correct answer is B.

**Q:** Which sport is played on a variable ground ranging from 50x100yd minimum to 100x130yd maximum? A) Baseball B) Association Football (Soccer) C) Basketball D) Volleyball

**Answer:** Choices A, C, D are wrong because their court size is all less the 50*100 yards. Hence, the correct answer is B.

**Q:** In American Football, how many points does a touchdown score?

A) Six B) Three C) Four D) Five

**Answer:** Based on the fact of the rules of American football, a touchdown scores 6 points. Hence, the correct answer is A.

\end{table}
Table 8: 5-Shot Prompt for Few-Shot Prompting in Level-1 Experiment



**Question:** What is the result of a bout when enough rounds have passed after an accidental foil that causes injury?

A) The injured fighter loses B) The fight goes to a decision C) The fight is restarted D) The uninjured fighter is disqualified

**Answer:** B

**Question:** What defensive maneuver involves rotating the body to allow an incoming punch to pass harmlessly by?

A) Bobbing B) Blocking C) Slipping D) Clinching

**Answer:** C

**Question:** What is the Filipino martial art that includes boxing known as?

A) Karate B) Suntukan C) Taekwondo D) Kung Fu

**Answer:**B

**Question:** The "door policy" at events is used to decide:

A) Who will perform at the event B) The theme of the event C) Who will be admitted and at what price D)

The duration of the event

**Answer:** C

**Question:** Which boxer is NOT widely recognized as one of the best bolo punchers in boxing history?

A) Coferino Garcia B) Roy Jones Jr. C) Floyd Mayweather D) Sugar Ray Leonard

**Answer:** C

\begin{table}
\begin{tabular}{l l}
**Q:** What is the result of a bout when enough rounds have passed after an accidental foil that causes injury?

A) The injured fighter loses B) The fight goes to a decision C) The fight is restarted D) The uninjured fighter is disqualified

**Answer:** Choices A and D are incorrect because injuries do not matter for the final fight outcome and Choice C is incorrect because boxing matches are never restarted. Hence, choice B is correct.

**Q:** What defensive maneuver involves rotating the body to allow an incoming punch to pass harmlessly by?

A) Bobbing B) Blocking C) Slipping D) Clinching

**Answer:** Slipping rotates the body slightly so that an incoming punch passes harmlessly next to the head. Hence, the correct answer is C.

**Q:** What is the Filipino martial art that includes boxing known as?

A) Karate B) Suntukan C) Taekwondo D) Kung Fu

**Answer:** Choices A, C, and D are incorrect as these martial arts are not from the Phillippines and have very specific and distinct moves from boxing. Hence, the correct answer is B.

**Q:** The “door policy” at events is used to decide:

A) Who will perform at the event B) The theme of the event C) Who will be admitted and at what price D) The duration of the event

**A:** Choices A, B, and D are incorrect because these options do not involve general admittance. Hence the correct answer is C.

**Q:** Which boxer is NOT widely recognized as one of the best bolo punchers in boxing history?

A) Coferino Garcia B) Roy Jones Jr. C) Floyd Mayweather D) Sugar Ray Leonard

**Answer:** Choices A, B, and D are incorrect because these three are widely known for popularizing and being excellent users of the bolo punch. Hence, the correct answer is C.





**Main Question:** Why might a coach call a timeout late in the fourth quarter when the team is on defense and the opposing team is on the third down?

A. To rest the defensive players and prevent a score B. To discuss strategy to force a turnover C. To argue with the referee about a previous play D. To allow a television commercial break

**Sub-Question 1:** What is the purpose of a timeout in American football?

A. To stop the clock B. To make strategic adjustments C. To replace the football D. To challenge a referee's decision

**Sub-Question 2:** What can happen on a third down in American football that would prompt a defensive coach to call a timeout?

A) The defense needs to prevent the offense from scoring B) The defense wants to conserve time for their offense C) The offense is likely to punt on the next down D) The offense might attempt a field goal

**Answer:** In American Football, timeouts are usually called to stop the clock, make lineup or strategy adjustments, or allow players to rest at an important moment. Hence the correct answers to the Main Question are A and B. And the correct answers to Sub-Question 1 are A and B. A pivotal third down late in the fourth quarter could make the difference between the opponent scoring a touchdown, a field goal, or nothing, so this play is very important to potentially prevent a score or decrease the amount scored by the opponent. Additionally, if a score is unavoidable or if a defensive stop is likely but your team is still trailing, a timeout on this play can save time for your team's offense to score in return and potentially take back the lead. Hence the correct answers to Sub-Question 2 are A and B.

**Main Question:** How do the responsibilities of an American Football team's offensive and defensive coordinators differ during a game?

A) Both design and call plays for the team B) One focuses on scoring strategies while the other focuses on preventing the opponent from scoring C) Both coordinate player substitutions and adjustments on their side of the ball D) One oversees the development of the game plan, while the other implements it during the game

**Sub-Question 1:** How does an offensive coordinator contribute to a team's performance in a game?

A) By calling offensive plays B) By managing the defense's tactics C) By designing scoring strategies D) By coaching the special teams unit

**Sub-Question 2:** How does a defensive coordinator contribute to a team's performance in a game?

A) By implementing the head coach's overall strategy B) By calling defensive plays C) By managing the offense's tactics D) By designing strategies to prevent the opponent from scoring

**Answer:** The Offensive Coordinator and Defensive Coordinator in American Football both design and coordinate plays and strategies as well as decide substitutions and lineup or strategy adjustments in game for their respective sides of the ball. Hence the correct answers to the Main Question are A, B, and C. And the correct answers to Sub-Question 1 are A and C. And the correct answers to Sub-Question 2 are B and D.

**Main Question:** Why might a high school football coach enforce strict adherence to tackling techniques during practice sessions?

A) To ensure the team wins more games. B) To minimize the risk of injury to players. C) To make practices more challenging. D) To comply with state sporting regulations.

**Sub-Question 1:** What is one of the primary concerns that safe tackling techniques aim to address during football games?

A) Improving player coordination. B) Reducing the risk of concussions. C) Enhancing the entertainment value of the game. D) Increasing the speed of the players.

**Sub-Question 2:** Why are concussions a significant concern in contact sports like American football?

A) They can lead to long-term health issues. B) They often require expensive equipment to diagnose. C) They result in penalties during the game. D) They are not detectable until days after the game.

**Answer:** Strict tackling techniques in American Football are typically used to prevent injuries, primarily concussions from helmet-to-helmet contact. Although sport regulations typically ban helmet-to-helmet contact, these regulations are not typically very strict about tackling form, so a coach enforcing strict tackling technique is not likely to be for following regulations. Hence the correct answer to the Main Question is A. And the correct answer to Sub-Question 1 is B. Concussions are a large concern because of the potential long-term and severe issues that you can come from them. Hence the correct answer to Sub-Question 2 is A



**Main Question:** What play should the coach call next if they need to maintain possession to run down the clock at the end of the 4th quarter, considering the team is currently ahead, it's 3rd down with 2 yards to go, and the opposing team still has 2 timeouts?

A) A deep passing play B) A quarterback kneel C) A short-yardage running play D) A punt

**Sub-Question 1:** What is the primary objective for the team that is ahead near the end of the 4th quarter?

A) To score as quickly as possible B) To maintain possession and run down the clock C) To allow the other team to score D) To stop the game clock

**Sub-Question 2:** On a 3rd down with 2 yards to go, which type of play is most likely to achieve a first down and continue possession?

A) A deep passing play B) A quarterback kneel C) A short-yardage running play D) A punt

**Answer:** In order to run down the clock, American Football teams typically run the ball or do a quarterback kneel because incomplete passes will stop the clock. In a situation where there is a 3rd down with 2 yards to go and the opponent has two timeouts, a qb kneel will not help get a first down to continue running off the clock. Hence the correct answer to the Main Question is C. A team that is ahead will want to run down the clock and maintain their lead as best as possible. Hence the correct answer to Sub-Question 1 is B. On a 3rd down with 2 yards to go, a short run play or short pass play will be most likely to acheive a first down. Hence the correct answer to Sub-Question 2 is C.

**Main Question:** Why might a defensive coordinator instruct a linebacker to shift his positioning from a standard 4-3 defense alignment to a position directly over the opposing team's tight end before the snap in a 3rd-and-long situation?

A) To better defend against a possible run play to the outside. B) To apply pressure on the quarterback by blitzing through the gap. C) To cover the tight end, anticipating a pass in a likely passing down. D) To confuse the offensive line's blocking scheme.

**Sub-Question 1:** Why would a defensive coordinator anticipate a pass in a 3rd-and-long situation?

A) The offensive team needs to gain a significant number of yards. B) Running plays are more effective in long-yardage situations. C) The defense has been successful in stopping the run all game. D) The offensive team's star running back is injured.

**Sub-Question 2:** Why would a linebacker be tasked with covering the tight end instead of a defensive back in this scenario?

A) The linebacker is typically faster than the defensive back. B) The tight end is a less skilled receiver than the wide receivers. C) The defensive back is occupied with covering a wide receiver. D) The tight end is known for exceptional blocking rather than receiving.

**Answer:** In American Football, linebacks lining up directly over a player typically means they will be covering them in the case they run a passing route. To blitz or contain run plays, linebackers will usually line up over a gap, and to confuse the offensive line they will usually fake blitzes. Hence the correct answer to the Main Question is C. In a 3rd-and-long situation, the offense has a lot of space to make up to get a first down. The most likely plays to gain large yardages in a single play are deeper passing plays, though they are more risky. Hence the correct answer to Sub-Question 1 is A. Tight ends are typically check-down passing options, meaning the primary option wide receivers are usually more important to have good coverage on. Hence defensive backs will usually cover wide receivers or more important zones, leaving the less important tight end receiver to be covered by the linebacker who is not as good in pass coverage. Hence the correct answer to Sub-Question 2 is C.





**Main Question:** How might the officiating crew determine that a double foil scenario occurred during a down where a defensive pass interference was also suspected?

A) By reviewing if both teams committed fools independent of each other during the same down. B) By assessing if the defensive team committed pass interference after the offense had already committed a holding foul. C) By identifying if the offensive team committed a false start before the defensive player interfered with a receiver. D) By determining if the defensive pass interference occurred simultaneously with an offensive holding.

**Sub-Question 1:** How can the officials confirm that the defensive team committed pass interference?

A) By checking if the defender made contact with the receiver before the ball was touched. B) By determining if the defender was playing the ball and not the receiver. C) By observing if the receiver was beyond the line of scrimmage when contact was made. D) By ensuring that the pass was catchable and the contact impeded the receiver's opportunity to catch the ball.

**Sub-Question 2:** How would the officials ascertain that the offensive team committed a holding foul?

A) By establishing if an offensive player used his hands outside the defender's frame to restrict movement.

B) By verifying if the offensive player's hands were inside the defender's shoulders while the defender was attempting to move past. C) By determining if the offensive player tackled the defender to the ground.

D) By observing if the offensive lineman maintained a blocking position without grabbing the defender's jersey or pads.

**Correct Answers:**

Main Question: A and B

Sub-Question 1: A, B, and D

Sub-Question 2: A, C, and D

**GPT-4 Answer:** In American Football, a double foul scenario occurs when both teams commit fools independent of each other during the same down. Hence the correct answer to the Main Question is A. Defensive pass interference is confirmed when the defender makes contact with the receiver before the ball is touched, and the pass was catchable and the contact impeded the receiver's opportunity to catch the ball. Hence the correct answers to Sub-Question 1 are A and D. A holding foul by the offensive team is ascertained when an offensive player uses his hands outside the defender's frame to restrict movement. Hence the correct answer to Sub-Question 2 is A.

**Explanation:**

Main Question: GPT-4 correctly understands that separate fools are independently reviewed, but fails to reason that in this dual foul scenario, this potential defensive pass interference could have occurred after an offensive holding penalty. This is a Logical Reasoning Error.

Sub-Question 1: GPT-4 fails to include the fact that incidental contact when a defender attempts to play the ball does not constitute pass interference in it's reasoning. This is a Logical Reasoning Error.

Sub-Question 2: GPT-4 shows clear understanding of the rules regarding offensive holding, but fails to reason that tackling a defensive player also follows this definition and that analyzing the inverse of a definition logically follows as analyzing the definition. This is a Logical Reasoning Error.

**Question:** In a regular basketball game, which of the following options correctly represents the number of players in a team on the court and the number of substitutes that can be made?

A) 5 players on the court, 7 substitutes B) 6 players on the court, 5 substitutes C) 7 players on the court, 3 substitutes D) 4 players on the court, 6 substitutes

**Correct Answers:** A

**GPT-4 Answer:** In a regular basketball game, each team has five players on the court at a time. The number of substitutes can vary depending on the league and level of play, but there is typically no limit on the number of substitutions that can be made during a game. Therefore, none of the options provided are correct.

**Explanation:**

GPT-4 misunderstands the context of what "substitutes" are referring to. Though it is correct that a team can make infinite substitutions in a basketball game, "substitutes" in this question are referring to the substitute players a team currently has on the bench. This is a Contextual Misinterpretation Error.

\begin{table} \end{table}
Table 13: GPT-4 5-shot CoT Incorrect Outputs and Error Analysis



**Question:** Why might a mediator be necessary during a football match?

A) To ensure the players follow the rules of the game B) To resolve disputes or conflicts between players or teams C) To ensure the spectators are entertained D) To keep track of the score

**Correct Answers:** B

**GPT-4 Answer:** A mediator, in this case, the referee, is necessary during a football match to ensure the players follow the rules of the game and to resolve any disputes or conflicts that may arise between players or teams. Hence, the correct answers are A and B.

**Explanation:**

 GPT-4 misunderstands the word mediator. By definition a mediator is an intermediary party in a conflict between 2 or more other parties. A referee will not need to be a "mediator" for any of their other responsibilities. This is a Conceptual Misunderstanding Error.

**Question:** During the final set of a Grand Slam tennis match, Player A decided to adopt a more aggressive playing style, frequently rushing to the net after his serve. Why might Player A have made this strategic decision?

A) Player A noticed that Player B struggles to return low volleys effectively. B) Player A wanted to use his strong net game to put pressure on Player B. C) Player A was trying to conserve energy by reducing the length of the rallies. D) Player A was attempting to intimidate Player B by showing off his physical prowess.

**Correct Answers:** A, B, and C

**GPT-4 Answer:** In tennis, a player might adopt a more aggressive playing style and frequently rush to the net after serving to put pressure on the opponent, exploit the opponent's weakness in returning low volleys, reduce the length of the rallies to conserve energy, or to intimidate the opponent. Hence the correct answers are A, B, C, and D.

**Explanation:**

 GPT-4 either misunderstands why an aggressive playing style is used in tennis or confuses the concepts of intimidation and applying pressure to an opponent. In non-contact sports such as tennis, aggressive playstyles are typically not used for intimidation, though they are frequently used to put pressure on an opponent to make them cause errors or break down mentally. Thus GPT-4 may be confusing the concepts of intimidation and applying pressure to cause the opponent to make errors. This is a Conceptual Misunderstanding Error.

\begin{table} \end{table}
Table 14: GPT-4 5-shot CoT Incorrect Outputs and Error Analysis

Figure 5: Proportional Distribution of Question Categories Across Sports. This bar chart illustrates the comparative distribution of ‘Rule or Strategy’ and ’History or Fact’ questions in various sports, highlighting the dominant category in each sport and indicating the variability in focus across different sports disciplines.