---
license: cc-by-sa-4.0
---
This model contains the checkpoints of paper **"Towards Universal Soccer Video Understanding"**: https://arxiv.org/abs/2412.01820/.

[Project Page](https://jyrao.github.io/UniSoccer/) | [Paper](https://arxiv.org/abs/2412.01820/) | [Codes](https://github.com/jyrao/UniSoccer) | [Dataset]() (Soon) | [Checkpoints](https://huggingface.co/Homie0609/UniSoccer) 

<div align="center">
   <img src="./teaser.gif">
</div>


## Requirements
- Python >= 3.8 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))
- [PyTorch >= 2.0.0](https://pytorch.org/) (If use A100)
- transformers >= 4.42.3
- pycocoevalcap >= 1.2

A suitable [conda](https://conda.io/) environment named `UniSoccer` can be created and activated with:

```
conda env create -f environment.yaml
conda activate UniSoccer
```

## Train

<div align="center">
   <img src="./architecture.png">
</div>

#### Pretrain MatchVision Encoder
As described in paper, we have two methods for pretraining MatchVision backbone (supervised classification & contrastive commentary). You can train both this two methods as following shows:


First of all, you should prepare textual data as the format in `train_data/json`, and preprocess soccer videos into 30 second clips (15s before and after timestamps) for pretraining.

**Supervised Classification**
```
python task/pretrain_MatchVoice_Classifier.py config/pretrain_classification.py
```
**Contrastive Commentary Retrieval**
```
python task/pretrain_contrastive.py config/pretrain_contrastive.py
```

Also, you could finetune MatchVision with
```
python task/finetune_contrastive.py config/finetune_contrastive.py
```
To be noted, you should replace the folders in task and config files.

#### Train Downstream Tasks

You could train the commentary task by several different methods:

1. Use mp4 files
```
python task/downstream_commentary_new_benchmark.py 
```
For this method, you might train the commentary model MatchVoice with open visual encoder or language decoder, so you should crop the videos as 30s clips named as json files shows.

2. Use *.npy* files
```
python task/downstream_commentary.py
```
For this method, you cannot open the visual encoder, so you can extract features of all video clips and change ".mp4" by ".npy" as file names.

**To be noted,** folder `words_world` records the token ids of all words in LLaMA-3(8B) tokenizer of different datasets as

- *`match_time.pkl`*: MatchTime dataset ([Link here](https://huggingface.co/datasets/Homie0609/MatchTime))
- *`soccerreplay-1988.pkl`*: SoccerReplay-1988 dataset. (Not released yet)
- *`merge.pkl`*: Union set of MatchTime & SoccerReplay-1988


## Inference

<div align="center">
   <img src="./inference.png">
</div>

For inference, you could use the following codes, be sure that you have correctly crop the video clips, which is in the same format as before.
```
python inference/inference.py
```
Then, you could test the metrics for output `sample.csv` by:
```
python inference/score_single.py --csv_path inference/sample.csv
```

## Citation
If you use this code and data for your research or project, please cite:

@misc{rao2024unisoccer,
      title   = {Towards Universal Soccer Video Understanding},
      author  = {Rao, Jiayuan and Wu, Haoning and Jiang, Hao and Zhang, Ya and Wang, Yanfeng and Xie, Weidi},
      journal = {arXiv preprint arXiv:2412.01820},
      year    = {2024},
}