Virtual Try-On Considering Temporal
Consistency for Videoconferencing
Daiki Shimizu1 and Keiji Yanai1
The University of Electro-Communications, Tokyo, Japanshimizu-d@mm.inf.uec.ac.jp, yanai@cs.uec.ac.jp

Abstract. Virtual fitting, in which a person’s image is changed to anarbitrary clothing image, is expected to be applied to shopping sites andvideoconferencing. In real-time virtual fitting, image-based methods using a knowledge distillation technique can generate high-quality fitting
images by inputting only the image of arbitrary clothing and a person
without requiring the additional data like pose information. However,there are few studies that perform fast virtual fitting from arbitrary
clothing images stably with real person images for situations such asvideoconferencing considering temporal consistency. Therefore, the purpose of this demo is to perform robust virtual fitting with temporalconsistency for videoconferencing. First, we created a virtual fitting system and verified how effective the existing fast image fitting method isfor webcam video. The results showed that the existing methods do not
adjust the dataset and do not consider temporal consistency, and thus
are unstable for input images similar to videoconferencing. Therefore,
we propose to train a model that adjusts the dataset to be similar to avideoconference and to add temporal consistency loss. Qualitative evaluation of the proposed model confirms that the model exhibits less flicker
than the baseline. Figure 1 shows an example usage of our try-on system
which is running on Zoom.
Keywords: virtual try-on · image translation · temporal consistency ·videoconferencing

1Introduction

With the spread of social networking services, the use of filters to transform aperson’s appearance into a desired one has become a popular practice. In recentyears, there has been a growing demand for real-time appearance transformationin videoconferencing as well, especially in virtual fitting for clothing transformation.
Conventional virtual fitting in videoconferencing is generally performed bypositioning garments using poses and body meshes based on 3D data of the garment created by the developer. However, because garments are non-rigid, simple
positioning often results in wrinkles and other discomfort, and is computationally expensive in order to improve quality. In addition, users without expertise2D. Shimizu and K. Yanai

must choose from a limited set of clothes created by the developer, making it
difficult for them to freely change clothes.
On the other hand, image-based virtual fitting methods using deep learning
have been studied to convert a person image from an arbitrary clothing image
into a fitting image. In the past, it was difficult to collect data because it required images of the person after the fitting. However, by removing the clothingregions of the person using a region decomposition model and using a personrepresentation that is not limited by clothing, it is now possible to learn virtual fitting from only the clothing and the image data of the person wearing it
without the try-on image. However, since the pose and region information of theperson must also be inferred, the generation accuracy depends on the accuracyof these inferences, and the inference speed is reduced.
However, fast image-based virtual fitting methods [5,3] has emerged that
does not require pose information and can generate fitting images from only theclothing and person images.
In recent years, there has been active research on video transformation, starting with Few-shot vid2vid [8]. Among them, several models [2,6,9,10] for virtual
fitting of videos have appeared. However, even these models require pose information, making real-time generation difficult.
The purpose of this research is to apply image-based virtual fitting to perform
stable virtual fitting from arbitrary clothing in actual video images, assuming
videoconferencing. In this demo, we first created a virtual fitting system using
a web camera, and verified how effective it is in actual person videos using
existing fast image-based virtual fitting methods. We also improved the video
virtual fitting dataset and proposed learning of a model with an additional lossof temporal consistency that reduces the difference from the previous frame, andqualitatively verified the results.

2Related Work

2.1

Virtual Try-On

The development of image-based virtual fitting has been remarkable, and various
methods have been devised. In recent years, many models [4,7,3] have emergedthat use deep learning to generate fitting images from pose and segmentation inaddition to person and clothing images.
VITON [4], in particular, consists of two stages, clothing deformation andsynthesis, and has become a mainstream method today. These image-based virtual fitting models require pose and body region information in addition toclothing and person images, making them dependent on the accuracy of poseor segmentation and requiring additional inference time. Therefore, the fast image virtual fitting model WUTON, which uses knowledge distillation to inferdirectly from only the clothing and person images, and its improved version
PF-AFN [3], which generates high quality images, have been introduced. In thisdemo, we adopt PF-AFN as a base method, which is a fast model and does not
require guides such as poses.

Virtual Try-On Considering Temporal Consistency for Videoconferencing

3PF-AFN [3] enabled direct clothing and person generation without the needfor person masks by using the generated results of a parser-based teacher modelwith person pose and segmentation as input for the student model. These innovations have made it possible to perform virtual fitting generation at high speed
and high quality. In this demo, we adopt PF-AFN as baseline and perform learning that considers temporal consistency.
In recent years, research on video generation has developed with the adventof Few-shot vid2vid [8], which enables conversion of faces, poses, etc. in videos.
Starting with this, a video-based virtual fitting method using Optical Flow as
a guide was proposed, and others [6,9,10] have appeared. FW-GAN [2] was thepioneering work in video-based virtual fitting and also provided the only video
virtual fitting dataset, VVT. the network consists of a module that takes as
input the previous frame’s generated image and pose sequence, the previousgenerated frame, the person image, and the clothing image, and generates adeformation network for the clothing image and a flow for warping the previousgenerated frames. ShineOn [6] added DensePose as an input and introduced selfattention calculated from DensePose and person images. They also investigated
the activation function bottom-up. However, this method also requires guides
such as Optical Flow and DensePose, making real-time inference difficult.

ForegroundExtraction

VirtualCamera

Virtual Try OnNetworkFig. 2. Virtual Try-On System
Fig. 1. Our try-on system running on Zoom

3

Proposed Method

3.1

Virtual fitting system using a webcam

We developed a real-time virtual try-on system using a webcam and trained avirtual try-on model on the VITON [4] dataset. The virtual try-on system isshown in Fig.2. First, in order to apply the virtual try-on model to videoconferencing, it is necessary to remove the background from the webcam video. The
foreground is extracted using the pre-trained segmentation model input to thevirtual try-on model. The resulting fitting image is input to a virtual camera,4D. Shimizu and K. Yanai

and the user selects the virtual camera to display the fitting image in the actual
videoconferencing application.
In our investigation, we observed flickering in PF-AFN, as shown in Fig.4. To
solve this problem, we propose a learning method and Temporal PF-AFN. We
also use a video-based virtual fitting dataset, VVT [2], as the training dataset.
By using a video dataset, we can learn frames that are close in time continuously,
and can generate a stable PF-AFN even when there is a slight moving. The modelis called Temporal PF-AFN, in which the composite result of the previous frame
is added to PF-AFN as an input.
3.2

Learning considering temporal consistency

Our network structure flowchart is shown in Fig.3. Our model takes as input aperson image with the background removed, a fitting image generated in the previous frame, and a reference clothing image. The output is a deformed clothing
image, its mask, and an image containing the person and background. During
training, a loss function is added to the existing loss function so that the results
generated in the previous frame and the current frame are close. Temporal try-on
loss function Lt is as follows:
Lt (St , St−1 ) = λt (λp1 Lp (Ct , Ct−1 ) + λc LL1 (Ct , Ct−1 )+λp2 Lp (It , It−1 ) + λi LL1 (It , It−1 ) + λm LL1 (Mt , Mt−1 ))(1)where C, I, and M represent the generated clothing image, the try-on image, andthe mask, respectively, and t represents the frame number. λs are hyperparameters to weight the loss functions. The total loss consists of two loss functions.
The first is the VGG Loss, which is visually close, and the second is the L1Loss, which is pixel close. Each loss is calculated between the two frames of thegenerated clothing image and the final fitting image, and then added together.
The mask M of the generated clothing image is also calculated using L1 Lossbetween the two frames and then added together.

TeacherframesourceTemporal
PF-AFN

warpedFig. 3. Temporal PF-AFN

Virtual Try-On Considering Temporal Consistency for Videoconferencing

4Experiments

4.1

Implementation Details

5First, we describe the implementation of a virtual fitting system in videoconferencing. To apply the system, the background must be excluded from thewebcam video. The pre-trained segmentation model is extracted from the foreground using DeepLab V3+ [1] and input to the virtual fitting model. Next,
we describes the training method with Temporal PF-AFN. We used VVT as atraining dataset. The first 5 epochs were trained to generate the initial frame.
A single image and a concatenated zero-filled image were used as input. After
that, we trained 35 epochs by adding temporal losses to the input frames andpreviously generated frames. We employed Adam optimizer for training. Thelearning rate was set to 5 × 10−5 , β1 = 0.5, β2 = 0.999. The parameters of theproposed method were trained as λp1 = 0.2, λi = 1, λp2 = 0.2, λj = 1, λm = 2.
λt was trained as 0.01 in the Warping Module, and 1.0 in the Generation Moduleof PF-AFN. Other parameters were set following to PF-AFN.
4.2

Experimental Results

In the upper two rows of Fig.4, the method is better at generating rotations
around 8 sec, and there is less flickering. However, there is a quick sitting motion between 14 and 16 seconds, which is not generated well by the proposed
method because the pixel remains. In addition, because the initial frame was notgenerated well, the image was not corrected immediately and slowly changed tothe correct fitting image.In the movie in the lower row, both methods were able
to generate images between 13 and 19 seconds, but the baseline method caused
flickering.The results of each movie showed that the scene was generated moresmoothly with less flickering compared to PF-AFN, which processed each frame.

Fig. 4. Comparison between Temporal PF-AFN and PF-AFN. The center two columns
represent Temporal PF-AFN results and the right two columns of Temporal PF-AFNresults. (Click to play)65

D. Shimizu and K. Yanai

Conclusion

In this demo, we demonstrate a virtual fitting system for videoconferencing employing an improved PF-AFN. From the experimental results, PF-AFN showed
an uncomfortable flickering. Therefore, we proposed a temporal PF-AFN thatconsiders temporal consistency in order to suppress the flickering. Qualitativeevaluation confirmed that the fluctuation in the frontal plane could be suppressed. However, it was found that there is a problem that pixels remain inquick movements. For the future, we plan to further improve the stability of thegeneration process. The current method uses a foreground extractor to remove
the background, which is dependent on the accuracy of foreground extraction
and reduces the overall processing speed. We will make it possible to generate
images of people with actual backgrounds in real time by providing them into
the model as an input without a foreground extractor.

References
1. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoderwith atrous separable convolution for semantic image segmentation. In: Proc. ofEuropean Conference on Computer Vision (2018)
2. Dong, H., Liang, X., Shen, X., Wu, B., Chen, B.C., Yin, J.: FW-GAN: Flownavigated warping gan for video virtual try-on. In: Proc. of IEEE International
Conference on Computer Vision (2019)
3. Ge, Y., Song, Y., Zhang, R., Ge, C., Liu, W., Luo, P.: Parser-free virtual try-on
via distilling appearance flows. In: Proc. of IEEE Computer Vision and PatternRecognition. pp. 8485–8493 (2021)
4. Han, X., Wu, Z., Wu, Z., Yu, R., Davis, L.S.: VITON: An image-based virtual tryon network. In: Proc. of IEEE Computer Vision and Pattern Recognition (2018)
5. Issenhuth, T., Mary, J., Calauzènes, C.: End-to-end learning of geometric deformations of feature maps for virtual try-on. arXiv:1906.01347 (2019)
6. Kuppa, G., Jong, A., Liu, V., Liu, Z., Moh, T.: ShineOn: Illuminating design
choices for practical video-based virtual clothing try-on. In: Proc. of IEEE WinterConference on Applications of Computer Vision Workshops. pp. 191–200 (2021)
7. Minar, M.R., Tuan, T.T., Ahn, H., Rosin, P., Lai, Y.K.: CP-VTON+: Clothing
shape and texture preserving image-based virtual try-on. In: Proc. of IEEE Computer Vision and Pattern Recognition (2020)
8. Wang, T.C., Liu, M.Y., Tao, A., Liu, G., Kautz, J., Catanzaro, B.: Few-shot videoto-video synthesis. In: Proc. of Advances in Neural Information Processing Systems(2019)
9. Wei, D., Xu, X., Shen, H., Huang, K.: C2F-FWN: Coarse-to-fine flow warping network for spatial-temporal consistent motion transfer. Proc. of the AAAI Conferenceon Artificial Intelligence 35(4), 2852–2860 (May 2021)
10. Zhong, X., Wu, Z., Tan, T., Lin, G., Wu, Q.: MV-TON: Memory-based video
virtual try-on network. In: Proc. of ACM International Conference Multimedia. p.
908–916 (2021)