diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..bc42bce204a063b6389a6383218e354efd2409de 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,13 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +checkpoints/pretrained.state filter=lfs diff=lfs merge=lfs -text +examples/kennedy_hd.mkv filter=lfs diff=lfs merge=lfs -text +examples/mona_hd.jpg filter=lfs diff=lfs merge=lfs -text +examples/mona_hd.mkv filter=lfs diff=lfs merge=lfs -text +input_audios/part_000_RVC_1.wav filter=lfs diff=lfs merge=lfs -text +input_videos/MC.mp4 filter=lfs diff=lfs merge=lfs -text +output_videos_hd/kennedy.mkv filter=lfs diff=lfs merge=lfs -text +output_videos_hd/mona.mkv filter=lfs diff=lfs merge=lfs -text +temp/result.avi filter=lfs diff=lfs merge=lfs -text +temp/temp.wav filter=lfs diff=lfs merge=lfs -text diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..369998379bc763a60255d30cd076a3f14fb65899 --- /dev/null +++ b/LICENSE @@ -0,0 +1,23 @@ +MIT License + +Copyright (c) 2023 Saif Hassan + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +NOTE: PLEASE READ LICENSE REQUIREMENTS (COPYRIGHTS INFORMATION) FROM `WAV2LIP OFFICIAL REPO`, MENTIONED ON MAIN PAGE OF THIS REPOSITORY. diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..951ece0691b2c0d599dc9959cd57df018e9cddad --- /dev/null +++ b/README.md @@ -0,0 +1,121 @@ +# Wav2Lip-HD: Improving Wav2Lip to achieve High-Fidelity Videos + +This repository contains code for achieving high-fidelity lip-syncing in videos, using the [Wav2Lip algorithm](https://github.com/Rudrabha/Wav2Lip) for lip-syncing and the [Real-ESRGAN algorithm](https://github.com/xinntao/Real-ESRGAN) for super-resolution. The combination of these two algorithms allows for the creation of lip-synced videos that are both highly accurate and visually stunning. + +## Algorithm + +The algorithm for achieving high-fidelity lip-syncing with Wav2Lip and Real-ESRGAN can be summarized as follows: + +1. The input video and audio are given to `Wav2Lip` algorithm. +2. Python script is written to extract frames from the video generated by wav2lip. +3. Frames are provided to Real-ESRGAN algorithm to improve quality. +4. Then, the high-quality frames are converted to video using ffmpeg, along with the original audio. +5. The result is a high-quality lip-syncing video. +6. The specific steps for running this algorithm are described in the [Testing Model](https://github.com/saifhassan/Wav2Lip-HD#testing-model) section of this README. + +## Testing Model + +To test the "Wav2Lip-HD" model, follow these steps: + +1. Clone this repository and install requirements using following command (Make sure, Python and CUDA are already installed): + + ``` + git clone https://github.com/saifhassan/Wav2Lip-HD.git + cd Wav2Lip-HD + pip install -r requirements.txt + ``` + +2. Downloading weights + +| Model | Directory | Download Link | +| :------------- |:-------------| :-----:| +| Wav2Lip | [checkpoints/](https://github.com/saifhassan/Wav2Lip-HD/tree/main/checkpoints) | [Link](https://drive.google.com/drive/folders/1tB_uz-TYMePRMZzrDMdShWUZZ0JK3SIZ?usp=sharing) | +| ESRGAN | [experiments/001_ESRGAN_x4_f64b23_custom16k_500k_B16G1_wandb/models/](https://github.com/saifhassan/Wav2Lip-HD/tree/main/experiments/001_ESRGAN_x4_f64b23_custom16k_500k_B16G1_wandb/models) | [Link](https://drive.google.com/file/d/1Al8lEpnx2K-kDX7zL2DBcAuDnSKXACPb/view?usp=sharing) | +| Face_Detection | [face_detection/detection/sfd/](https://github.com/saifhassan/Wav2Lip-HD/tree/main/face_detection/detection/sfd) | [Link](https://drive.google.com/file/d/1uNLYCPFFmO-og3WSHyFytJQLLYOwH5uY/view?usp=sharing) | +| Real-ESRGAN | Real-ESRGAN/gfpgan/weights/ | [Link](https://drive.google.com/drive/folders/1BLx6aMpHgFt41fJ27_cRmT8bt53kVAYG?usp=sharing) | +| Real-ESRGAN | Real-ESRGAN/weights/ | [Link](https://drive.google.com/file/d/1qNIf8cJl_dQo3ivelPJVWFkApyEAGnLi/view?usp=sharing) | + + +3. Put input video to `input_videos` directory and input audio to `input_audios` directory. +4. Open `run_final.sh` file and modify following parameters: + + `filename=kennedy` (just video file name without extension) + + `input_audio=input_audios/ai.wav` (audio filename with extension) + +5. Execute `run_final.sh` using following command: + + ``` + bash run_final.sh + ``` + +6. Outputs + +- `output_videos_wav2lip` directory contains video output generated by wav2lip algorithm. +- `frames_wav2lip` directory contains frames extracted from video (generated by wav2lip algorithm). +- `frames_hd` directory contains frames after performing super-resolution using Real-ESRGAN algorithm. +- `output_videos_hd` directory contains final high quality video output generated by Wav2Lip-HD. + + +## Results +The results produced by Wav2Lip-HD are in two forms, one is frames and other is videos. Both are shared below: + +### Example output frames + + + + + + + + + + + + + + + + + + + +
Frame by Wav2LipOptimized Frame
+ + + ### Example output videos + + | Video by Wav2Lip | Optimized Video | +| ------------- | ------------- | +|