Virtual Try-On Considering Temporal
Consistency for Videoconferencing
Daiki Shimizu1, and Keiji Yanai1
1The University of Electro-Communications, Tokyoⓒ UEC Yanai Lab. Tokyo.

Introduction• Real-time appearance change in videoconferencing.
– Style transformation.
– Virtual backgrounds.
– Virtual makeup.
– 3D Avatars.
– Virtual fitting.

Snap Cameraⓒ UEC Yanai Lab. Tokyo.

2

Objective

Real-time virtual try-on for videoconferencing
considering temporal consistencyⓒ UEC Yanai Lab. Tokyo.

3

Related work - PF-AFN -1Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo.

Parser-free virtual try-on via distilling appearance fows. In CVPR, 2021.

• PF-AFN1 is a real-time virtual try-on network.
– Student model takes only two images as input.

• This work does not take into account temporal information.

We extend PF-AFN for real-time video virtual try-on.
ⓒ UEC Yanai Lab. Tokyo.

4

Contribution• Propose a method to learn a virtual try-on network consideringtime-consistency.
• Develop virtual try-on system using virtual camera.

ⓒ UEC Yanai Lab. Tokyo.

5Proposed method• Base model: PF-AFN• Model inputs:– Current image.
– Template cloth.
– Previous generated images.
(additional input)• Additional loss:
temporal consistency loss ℒ!
ℒ! = 𝜆! (𝜆"! ℒ" 𝑆! , 𝑆!#$ + 𝜆% ℒ&$ 𝑆! , 𝑆!#$ +𝜆'" ℒ' 𝐶! , 𝐶!#$ + 𝜆( ℒ&$ 𝐶! , 𝐶!#$ +𝜆) ℒ&$ 𝑀! , 𝑀!#$ )ⓒ UEC Yanai Lab. Tokyo.

6System architecture
1. Take one frame from the camera stream.
2. Remove background using a pre-trained segmentation model.
3. Provide the foreground image to the proposed try-on model.
4. Obtain the cloth-changed image from the model.
5. Provide it to the virtual camera.
①

②⑤

④

ⓒ UEC Yanai Lab. Tokyo.

③
7

System architecture
1. Take one frame from the camera stream.
2. Remove background using a pre-trained segmentation model.
3. Provide the foreground image to the proposed try-on model.
4. Obtain the cloth-changed image from the model.
5. Provide it to the virtual camera.
①

②

⑤

④

ⓒ UEC Yanai Lab. Tokyo.

③8

System architecture
1. Take one frame from the camera stream.
2. Remove background using a pre-trained segmentation model.
3. Provide the foreground image to the proposed try-on model.
4. Obtain the cloth-changed image from the model.
5. Provide it to the virtual camera.
①

②

⑤

④

ⓒ UEC Yanai Lab. Tokyo.

③9

System architecture
1. Take one frame from the camera stream.
2. Remove background using a pre-trained segmentation model.
3. Provide the foreground image to the proposed try-on model.
4. Obtain the cloth-changed image from the model.
5. Provide it to the virtual camera.
①

②

⑤

④

ⓒ UEC Yanai Lab. Tokyo.

③10

System architecture
1. Take one frame from the camera stream.
2. Remove background using a pre-trained segmentation model.
3. Provide the foreground image to the proposed try-on model.
4. Obtain the cloth-changed image from the model.
5. Provide it to the virtual camera.
①

②

⑤

④

ⓒ UEC Yanai Lab. Tokyo.

③
11System architecture
1. Take one frame from the camera stream.
2. Remove background using a pre-trained segmentation model.
3. Provide the foreground image to the proposed try-on model.
4. Obtain the cloth-changed image from the model.
5. Provide it to the virtual camera.
①

②

⑤

④

ⓒ UEC Yanai Lab. Tokyo.

③12DemonstrationPF-AFN

Oursⓒ UEC Yanai Lab. Tokyo.

13

Limitation• Difficult to correct failed frame like initial frame or quick movement.
PF-AFNOursⓒ UEC Yanai Lab. Tokyo.

14

Conclusion• Extended PF-AFN by adding temporal consistency loss.
– Suppressed the flicker to some extent.
– Still difficult to:• Respond to quick movements.
• Correct failed pixels.

• Created a virtual fitting system for videoconferencing.

ⓒ UEC Yanai Lab. Tokyo.

15

ⓒ UEC Yanai Lab. Tokyo.

16