---
license: gpl-3.0
language:
- en
metrics:
- wer
---
# LipNet Phonemes Predictors

Project was developed on using python3.8, in a Linux Ubuntu 24.04
run `python -m pip install -r requirements.txt` to make sure your dependencies are the same as mine  
the list of video files to be used for training and validation when training normal LipNet (not phonemes prediction) 
are in unseen_train.txt and unseen_test.txt respectively.
the datasets are zipped in lip/*.zip, unzip them into the same location and run `python main.py` to start training  
hyperparamters are found in options.py  

Project Setup
1. pull this repo using `git pull https://huggingface.co/SilentSpeak/torchnet phonemes`
2. initialize a python virtualenv for this project using `python3.8 -m venv venv`
3. initialize the virtualenv using `source venv/bin/activate`
4. run `python -m pip install -r requirements.txt` to get dependencies
5. install git LFS using `git lfs install`
6. pull the GRID dataset and saved tensorboard runs using `git lfs pull`

Following the project setup, you can run training as follows:  
To run training for the LipNet phonemes predictor, run `python main.py`  
To run training for the LipNet phonemes to text transformer predictor, run `python TransformerTrainer.py`  
To run training for the LipNet-to-BiGRU-to-text transformer predictor, run `python TranslatorTrainer.py`  
To run evaluation for the lipnet phonemes predictor + phonemes-to-text transformer end-to-end pipeline, 
run `cd tests && python lipnet-pipeline.py`. The model weights used in `lipnet-pipeline.py` are included in the repo as 
LFS files in the `saved-weights` folder.  

The LRS2 dataset was too large to include in the repo, and access to the LRS2 dataset is conditional on accepting 
the non-commercial usage license. However, the config file for training on the LRS2 dataset can be found in `options_lrs2.py` 
, and the preprocessing code for the LRS2 dataset can be found in `scripts/extract_crop_lips_v2.py` and `scripts/generate_lsr2_train.py`.
The LRS2 dataset itself can be be found at [https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs2.html](https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs2.html)