DOI: 10.5281/zenodo.15516887
title: SportsOri

description: # Public Sports Controversy Analysis with LLMs


## Overview


This repository contains code, datasets, and analysis for studying public opinion on controversial sports events using Large Language Models (LLMs). The project focuses on three major sports controversies:


1. **Maradona's "Hand of God"** (1986 FIFA World Cup)

2. **Frank Lampard's "Ghost Goal"** (2010 FIFA World Cup)

3. **Luis Suarez's Handball** (2010 FIFA World Cup)


The project applies advanced NLP techniques to analyze public sentiment and stance detection on these controversial sporting events, fine-tuning LLMs for improved accuracy in detecting stance in social media comments.


## Datasets


### Frank Lampard Ghost Goal


The Frank Lampard Ghost Goal dataset contains annotated social media comments about the controversial moment in the 2010 FIFA World Cup match between England and Germany. The incident occurred when Frank Lampard's shot crossed the goal line but was not awarded as a goal by the officials.


Each comment is labeled with one of four stances:

- **Favor**: Supporting that the goal should have been awarded

- **Against**: Arguing against the importance of the disallowed goal

- **Neutral**: Acknowledging the event without taking a stance

- **Irrelevant**: Comments unrelated to the controversy


Example usage:


```python

import pandas as pd


# Load the original dataset

df = pd.read_excel('data/Dataset/Frank Lampard Ghost Goal Labels.xlsx')


# Access comments and their labels

comments = df['Comments']

labels = df['Label']


# Distribution of stances

stance_distribution = df['Label'].value_counts()

print(stance_distribution)

```


### Maradona Hand of God


This dataset contains annotated comments about Diego Maradona's infamous "Hand of God" goal during the 1986 FIFA World Cup quarterfinal between Argentina and England.


### Luis Suarez Handball


This dataset contains annotated comments about Luis Suarez's handball incident in the 2010 FIFA World Cup quarterfinal between Ghana and Uruguay, where he deliberately blocked a goal-bound header with his hand.


## Fine-Tuning Scripts


The repository includes Jupyter notebooks for fine-tuning Llama 3.1 8B models using the Unsloth library:


- `Fine Tuning using Unlsoth script.ipynb`: General fine-tuning pipeline

- `frank-lampard-reasoning-unslo.ipynb`: Fine-tuning specifically for the Frank Lampard controversy

- `luis-saraez-fine-tune-unsloth.ipynb`: Fine-tuning specifically for the Luis Suarez controversy

- `maradona-fine-tuning-unsloth.ipynb`: Fine-tuning specifically for the Maradona controversy


## Analysis Tools


The `LLMsKnow` directory contains tools for analyzing LLM behavior and interpretability, based on the research paper "LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations."


Key scripts include:

- `analyze_frank_lampard.py`: Analyze the Frank Lampard dataset with balanced class handling

- `probe_all_layers_and_tokens.py`: Probe model layers to understand internal representations

- `probe.py`: General probing utilities for analyzing model behavior


## Prompts


The `data/Prompts/prompts.json` file contains carefully designed prompts for stance detection on each controversy. These prompts include:

- Background information about the controversy

- Example comments with labels

- Detailed definitions of stance categories


## Usage Examples


### Loading and Exploring a Dataset


```python

import pandas as pd


# Load the Frank Lampard dataset

df = pd.read_excel('data/Dataset/Frank Lampard Ghost Goal Labels.xlsx')


# Print basic statistics

print(f"Total comments: {len(df)}")

print(f"Stance distribution: {df['Label'].value_counts()}")


# Sample comment

print(f"Sample comment: {df['Comments'][0]}")

print(f"Stance: {df['Label'][0]}")

```


## Acknowledgements


This project builds upon the research framework presented in "LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations" by Orgad et al. (2024) [arXiv:2410.02707](https://arxiv.org/abs/2410.02707). The project utilizes code from the [LLMsKnow](https://github.com/technion-cs-nlp/LLMsKnow) repository for model probing and analysis tools, which has been adapted for sports controversy stance detection analysis.