Arbitrary Virtual Try-on Network: Characteristics Representation and Trade-off between Body and Clothing
Authors: Yu Liu
Mingbo Zhao
Zhao Zhang
Jicong Fan
Yang Lou
Shuicheng YAN
Venue: ICLR 2023 poster (2023)
Keywords: Deep Learning
Virtual Try-on
Generative Adversarial Networks
Artificial Intelligence in Fashion

Abstract: Deep learning based virtual try-on system has achieved some encouraging progress recently, but there still remain several big challenges that need to be solved, such as trying on arbitrary clothes of all types, trying on the clothes from one category to another and generating image-realistic results with few artifacts. To handle this issue, we propose the Arbitrary Virtual Try-On Network (AVTON) that is utilized for all-type clothes, which can synthesize realistic try-on images by preserving and trading off characteristics of the target clothes and the reference person. Our approach includes three modules: 1) Limbs Prediction Module, which is utilized for predicting the human body parts by preserving the characteristics of the reference person. This is especially good for handling cross-category try-on task (e.g., long sleeves \(\leftrightarrow\) short sleeves or long pants \(\leftrightarrow\) skirts, etc.), where the exposed arms or legs with the skin colors and details can be reasonably predicted; 2) Improved Geometric Matching Module, which is designed to warp clothes according to the geometry of the target person. We improve the TPS-based warping method with a compactly supported radial function (Wendland's \(\Psi\)-function); 3) Trade-Off Fusion Module, which is to trade off the characteristics of the warped clothes and the reference person. This module is to make the generated try-on images look more natural and realistic based on a fine-tuning symmetry of the network structure. Extensive simulations are conducted and our approach can achieve better performance compared with the state-of-the-art virtual try-on methods.

Published as a conference paper at ICLR 2023
ARBITRARY VIRTUAL TRY-ON NETWORK: CHAR- ACTERISTICS REPRESENTATION AND TRADE-OFF BE- TWEEN BODY AND CLOTHING
Yu Liu, Mingbo Zhao"
Zhao Zhang"
Donghua University, China {2191408, mzhao 4 dhuu. edu, cn
Hefei University of Technology, China cs cszzhang@gmail com.
Jicong Fan
The Chinese University of Hong Kong, Shenzhen, China fan wwejjnicoccuulle edu. en
Yang Lou
Osaka University, Japan felix. www.llueeeee..e
Shuicheng Yan
Sea AI Lab, Singapore yansc@sea com
ABSTRACT
Deep learning based virtual try-on system has achieved some encouraging progress recently, but there still remain several big challenges that need to be solved, such as trying on arbitrary clothes of all types, trying on the clothes from category to another and generating image-realistic results with few artifacts. To handle this issuc, we propose the Arbitrary Virtual Try-On Network (AVTON) that is utilized for all-type clothes, which synthesize realistic try-on images by preserving and trading off characteristics of the target clothes and the reference person. Our approach includes three modules: 1) Limbs Prediction Module, which is utilized for predicting the human body parts by preserving the characteristics of cosss-catggoyy try-on
the reference person. This is especially good for handling task (c.g.. long sleeves + short sleeves or long pants + skirts, etc.). where the exposed arms legs with the skin colors and details can be reasonably predicted: 2) Improved Geometric Matching Module, which is designed to warp clothes ac- cording to the geometry of the target person. We improve the TPS- TPS-based warping method with a compactly supported radial function (Wendland's v/ V-function)) 3) Trade-Off Fusion Module, which is to trade off the characteristics of the warped clothes and the reference person. This module is to make the generated try-on images look more natural and realistic based on a fine-tuning symmetry of the net- work structure. Extensive simulations are conducted and our approach can achieve better performance compared with the state-of-the-ee virtual try-on methods.
INTRODUCTION
During the past few years, computer vision technology has been widely utilized in the extensive applications of artificial fashion. These applications include clothes detection Liu et al. (2016); Ge et al. (2019), clothes parsing ei al. (2019); Gong et al. (2017), clothes attributions and categories
"Corresponding authors. This work is supported by the National Natural Science Foundation of China (61971121, 61672365, 62106211). Our code and dataset available at
lauYuZzz/AVTON

---

Published as a conference paper at ICLR 2023
Reterence
Target Clothes
CP-VTON Results
AvTON Wanilla)
AVIUN (Pull)
Reference Parsnn
Target Clothes
CPVTTON Results
AVTON (Vamila)
AVIUN (Pull)
Person
Bubban
Whale
Figure I: We propose AVTON that is trained with an all-type clothing dataser. It can be adapted to all-type clothing try-on tasks and get image-realistic results. The types of clothes are divided into the top, bottom, and whole, and we apply CP-VTON [Wang et al. (2018a) and AVTON on them. The CP-VTON retained with the all-type clothing dataset, and the AVTON (Vanilla) indicates the AVTON trained without LPM and Wendland's 's Vi V-function Wendland (1995), while the AVTON (Full) is just the opposite.
(2016); Ge et al. (2019); | Gao et al. (2022) Zeng et al. (2022), clothes collocation Iwata et al. (2011) Zhao al. (2020): He et al. (2016); Shih et al. (2018); Li et al. 2017); Han et al. (201); : Cui et al. (0019): Zeng et al. (022). etc. These applications are all merited from recently developed technology. namely deep learning, due to its powerful feature extraction ability to capture rich mid-level image representations. Motivated by this technology, deep baased virtual try-on methods have been extensively studied and achieved considerable results recently.
recognition Wang et al. 201 2018a); Liu et al.
Many deep baased methods for virtual try-on have been developed during the past decade. VITONHHan et al. (2018) and CP-VTON Wang et al. (20T Sa) are the two first works. The key idea of VITON is to exploit Thin-Plate Spline (TPS) Duchon (1977) based method to warp the clothes images with texture mapped on it, while CP- has extended VITON by developing neural network layers to learn the transformation parameters. VTNFP Yu et al. (2099) and ACGPN Yang et al. (2020) are another two popular methods that have improved the performance of the try-on task with more characteristics of clothes and human body preserved. Recently, many methods have been developed that focus on certain aspect to improve performance. For example, the PFAFN Ge jet al. (202T) adopts the knowledge distillation from large complex network to achieve the faster reference time well remove the human parsing procedure; VITON-HD method Choi et al. (2021): Lee et al. (2022) is specially developed to handle high- resolutioo try-on task. Dong et al. 2022 further proposes wFlow for handling video try-on tasks in the wild by adopting the optical flow from videos, while Jiang et al.) (2022) has firstly utilized transformer for taming video try-on tasks. All the above methods have verified the effectiveness of deep learning- virtual try-on.
Although these works have made some progress, most of them only focus on top cloth try-on tasks and cannot handle arbitrary try-on tasks (the top, bottom, or the whole clothes). This is of great practice in real-world applications and currently, there are few works focused on this try-on task. In addition, still remains some ongoing challenges and limitations: 1) most benchmark datasets utilized for training virtual try-on methods mainly contain top clothes. As a result, the trained model can only handle the top clothing try-on task while cannot be adapted work on the other bottom or whole clothing try-on task; 2) cross-category try-on task (e.g., long sleeves+>short sleeves or long skirss, etc.) is another challenge in the try-on task. A case in point is that when people aim to try on from long sleeves to short sleeves, some parts of people's 's arms will be exposed. Therefore, it is necessary to preserve the characteristics of the reference person and predict such an exposed human body when generating the image-realistic try-on results. However, most current methods Han et al. (2018); Wang et al, (2018a); Pandey & Savakis (2020); Yu et al. (2019) do not consider this issue. As result, some bad try-on performances, e.g., the limbs of human beings covered by clothes,
2

---

Published as a conference paper at ICLR 2023
the color of the skin is wrongly painted and the hand details cannot be properly generated, may be appeared; 3) current methods are not good at trading off characteristics of the warped clothes and the reference person. Although it preserve the characteristics of the warped cloth and reference person as much as possible, its generated images are not realistic enough, such as some artifacts near the neck regions. The reason for these results is that the final fusion module prefers preserve the warped clothing characteristics and cannot correct these errors.
In this paper, we propose a new image-based virtual try-on method, called AVTON, to achieve bitrary clothing try-on and image-realistic results. The proposed method contains three modules: a) Limbs Prediction Module, which developed for predicting limbs, and keep the head and the non non-target human body parts, to preserve the characteristics of the reference person. This module especially suitable for handling cross-category try-on task, such as long sleeves <+ short sleeves or long pants ++ skirts, etc., where the exposed arms or legs (including their skin colors and details) can be reasonably predicted. This good to help the try-on system for formulating a realistic result in the following modules; b) Improved Geometric Matching Module, which is designed to warp clothes according to the geometry of the reference person. By carefully analyzing the basic concept of Thin-Plate Spline (TPS) Duchon (977) based methods, we argue that the selection of radial ba- sis function a key point to affect the performance of image warping. Motivated by this end, we then have adopted Wendland's V-function Wendland (1955) the compactly supported radial basis function. Both theoretical analysis and simulation have verified that the proposed method is able to characterize the local geometrical structure of images, which is good for warping the clothes image, especially with the complex texture; c) Trade-Off Fusion Module, which is to trade off the charac- teristies of the warped clothes and the reference person, this module images looks more natural and realistic based on a fine-tune symmetry of the network structure (a pair of UNet Ronneberger et al. (2015)). Experiments show that AVTON significantly outperforms the state-of-the-an methods for virtual try-on et al. (2018); Wang et al. (2018a); Yu et al. (2019); Yang et al. (020), and can generate realistic try-on images all-type clothing try-on task (Fig.
make the generated try-on
2 ARBITRARY VIRTUAL TRY-ON NETWORK
Input:
Step II
IGMM
Step III TOFM
14
Step I LPM
coodder
deooder
noiching
funebox
With
3dd72cn
Mock
Figure 2: An overview of our AVTON. Step Limbs Prediction Module takes the target clothes and the human body information H as the input to predict the exposed limbs and preserve the primary human body information, and output the predicted human body i; Step II: Improved Ge- ometric Matching Module takes the target clothes and the refined human body information Å as input, and output the warped clothes é; Step III: Trade-Off Fusion Module firstly takes the warped clothes è and the refined human body information / as the input to predict the composition mask in and the rendered person r, and then compose the outputs with the warped clothes generate the try-on image 1t.
Our goal to learn an arbitrary virtual try-on model that can be adapt to all-type clothing and cross-category try-on tasks and generate more realistic try-on images than prior arts. The proposed AVTON contains three modules, as shown in Fig. 2 First, the limbs Prediction Module (LPM) is utilized to predict the limbs, head, and non-target human body. This is especially useful for
3

---

Published as a conference paper at ICLR 2023
handling cross-category try-on task. Second, the Improved Geometric Matching Module (IGMM) uses Wendland's ' V-function Wendland (1995) to improve the TPS based method Duchon (1977) for warping the clothes. Third, the Trade-Off Fusion Module (TOFM) takes the warped clothes and the human body information as input, and then generates composition mask and rendered person by a pair of UNet Ronneberger al. (2013) and fusion part. Especially, the TOFM takes advantage of GAN Goodfellow et al. (2020).
2.1 LIMBS PREDICTION MODULE (LPM)
The purpose of designing the Limbs Prediction Module (LPM) is predict the exposed limbs and preserve the primary human body information (i.e., head, hand details, and non-target human body parts). Most earlier methods generate exposed limbs try-on steps but neglected primary human body information, which may generate unreasonable color of skin and occlusion. This can usually happen when handling cross-category try-on task, i.c. long sleeve + short sleeve, or long pant + short pant, some limbs originally covered by the clothes will be exposed. We thereby propose LPM to address these issues.
Given a reference person image 4,, LPM takes the target clothes the human body information (i.e., the pose and shape information extracted by DensePose Alp Güler et al. (2018), the head and non-target human body parts information p as mentioned in Section ??) imputs to predict exposed limbs (. In detail, the target clothes e and the human body information H/ d (#: concatenation) are firstly encoded as the input features. They are then formed as single tensor by a correlation layer and input to the decoder, which is to calculate the feature correlation by M M M and M. are the normalized encoder features of body information H and clothes information respectively. Finally, the exposed limbs are predicted by the decoder. The encoder and correlation layer are similar to CP-VTON GMM step Wang et al. (2018a), while the human body information d's encoder-decoder layers are similarly to UNet Ronneberger et al. (2015) structure shown in Fig. 2 All this leads to preserve the primary body information.
The LPM trained under a combination of the pixel-wise LI loss and VGG perceptual loss between predicted result and ground truth 4, where I, includes head, non-target human body parts and exposed limbs in the reference person I,:
LIPM = >uill - 1,111 + (vog
where
Lvva = (1.)) - (z
the VGG perceptual loss, where ÀLI and Auga are the trade-off parameters for two loss terms in Eq. [ which all set to in our experiments, and (I) denotes the feature map of limbs' image of the i-th layer in the visual perception network Φ. which a VGG19 pre-trained on ImageNet.
2.2 IMPROVED GEOMETRIC MATCHING MODULE (IGMM)
The old way of warping clothes is based Thin-Plate Splines (TPS) Duchon (1977) with r2 log as Radial Basis Functions (RBFs). This method yields minimal bending energy properties measured over the whole image. But since it the regions where all control points are located. It is advantageous for yielding an overall smooth deformation and preserving geometrical characteristics. but is problematic when only small part of the image is desired to be deformed. This will lead to unreasonable deformation when warps clothes. To address this issue,we this work has adopted f-function of Wendland Wendland (1995) as RBFs. As mentioned in Fornefett et al.) (1999 (001). it is more compactly support for the registration of images so that the bending region can be narrowed down when minimizing the bending energy. Here, we first give its formulation follows:
compactly supported RBFs, the deformation will cover
WenndlandWendland
(r) 1/: (1 (1-r) (1-1)14/)++++ La/at+e+1 (r)
(3)
where
(1 05r<l
(1 -)-
-10
r r21 1
4

---

Published as a conference paper at ICLR 2023
w() = S. (t)dt r r 0.
The equation also holds for different spatial supports as (r( - (r//a). We apply the V0,3,1 function as RBFs to replace the TPS's RBFs:
(r) = - (01-r/a) (4r/a+++)
(4)
where ar is a learnable parameter as same as the spatial transformation parameters 0. Inspired by CP-VTON Wang et al. (2018a), we use the same structure learn these parameters. As shown in Fig. 2 our IGMM firstly extract high-level features of the target clothes and the refined human body information f d h (h represents hand details in reference person 1,) respectively. Then a correlation layer to combine two features into as single tensor as input to the regression network that predicts 0 and a. Finally, a transformation V. based on Eq. 4 for warping the target clothes into the result é (c).
To learn the module above, make some derivations and experiments study the size of spatial supports a, which shows a significant positive correlation between spatial warping range and a, and we concluded that the best should meet the condition: >> D. where D is the maximum distance among control points in Delaunay triangles Delaunay et al. (1934). In our experiments, we set D = v + B (a and is the vertical distance and the horizontal distance among nearest control points, respectively). Consequently, the final regression network and Xa is set to 6 in our experiments.
r + D, where á is the sigmoid's output of the
To train the module, we conducted it under the pixel-wise L1 loss between the warped clothes è and ground truth G, where ep is the clothes worn on the reference person in I,:
LIGMM = 22 c, = V. c,
(5)
2.3 TRADE-OFF FUSION MODULE (TOFM)
Try-on image synthesis from the warped elothes and the reference person is many-to-one mapping problem. It aims at not only preserving the characteristics of the warped clothes and the reference person, but also trading off them to make images realistic. One of the common methods Han et al. 2018) is to produce composition mask for fusing UNet Ronneebrerr et al. (2015) rendered person with warped clothes and finally to produce a refined result. Although it can refine the course try-on image, it lacks preserving characteristics of the warped clothes. Another common method WWagg ct al. (2018)) is to utilize an UNet to render a person image and predict a composition mask si- multaneously, and then synthesizing the try-on image by fusing the rendered person and the warped clothes via the composition mask. But this way failed to preserve characteristics of reference person on account of using a single UNet structure, this structure prefers preserve characteristies of the warped clothes. And other methods Yu et al. (2019); Yang et al. (2020) have some problems with trading off characteristies between the warped clothes and the reference person (e.g., artifacts near the neck regions).
In this paper, we adopt a GAN Goodfellow et al. 2020 based method for generating the realistic try-on image. In detail, we formulate the generator G in GAN by a pair of UNet, where the inputs are the warped clothes and refined human body information Å, while the outputs are the generated the warped
composition mask rn and rendered person r. However, making the characteristics clothes and the reference person contribute equally when generating the try-on image is still prob- lematic, as this will cause occlusion and artifact problems. To make images more realistic, we add a fusion part in our try-on module (Fig. 2 residual blocks) for handling the above problems. Specifi- cally, in the proposed module, the warped clothes' features F, and the refined human body features are firstly summed element-wisely, so that the fused features can be obtained, i.c. F (F&+F + F). Then the fused features F are concatenated with Fa and F, and decoded respectively to get predicted mask in and predicted rendered person r. Finally, similarly to CP-VTON Wang al. (2018)). and r are fused together using m to synthesize try-on image I,:
4 = m 3 e (1 m) r
(6)
where @ represents element-wise matrix multiplication. Additionally, we use the multi-scale dis- criminators D that is similar to piX2pixHD Wang et al. (2018b).
5

---

Published as a conference paper at ICLR 2023
At the training phase, the generator's loss is the combination loss scheme of CP-VTON and pix2pixHD, includes LI loss, VGG perceptual loss (Eq. 2 and LSGAN loss:
LG = - fglla I + Xegg (v 1g)+
(7)
Amask ||m m-ll |1 + Asgan (D(C H, It) 1)2
while the discirminator's 's loss is:
CD (DD(e, H. 4))² (D(c, Ll)) 1
where Ig is the ground truth image, and Ig = 1, in training stage, my is the mask of c, In our experiments, we set AL1. Aegg and Amakk to 10, while set Alsgon to 1.
EXPERIMENTS
3.1 DATASET DESCRIPTION
In this work, we evaluate the performance of our proposed work based on two datasets: the VITON dataset (Han et al. (2018) that used VITON Han et al. (2018). CP-VTON [aang et al. (2018a) and ACGPN Yang et al. (2020) et al., and the newly-collected Zalando dataset. In this paper we call them VITON-Datase and Zalando-Dataset respectively.
The VITON-Dataset contains 16253 frontal-view woman and top clothing image pairs, which is split into a training set and testing set with 14221 and 2032 pairs respectively. We also use the strategy in ACGPN Yang et al. (2020) to score the complexities of images in dataset and divide them into Easy, Medium, and Hard levels, which is used to further evaluate the proposed method and other state-of-the art methods for handling different levels of try-on task.
Though VITON-Dataset only contains top clothes while lacks of bottom and whole clothes. cannot be utilized to train the model for handling arbitrary try-on tasks. We thereby in this subsection introduce the newly-collected Zalando-Dataset. The Zalando-Dataset is crawled from http://wwwwwalanooooooooo
It contains 34928 frontal-view human (include man and woman) and clothing (include top, bottom, and whole) image pairs. In our study, we split it into training set and testing set with 32746 and 2182 image pairs, respectively. In detail, the training set contains 19185 tops, 10587 bottoms, and 2974 whole clothes, while the testing set contains 1310 tops, 692 bottoms, and 180 whole clothes.
3.2 IMPLEMENTATION DETAILS
The experiments are conducted on the VITON-Datasct and Zalando-Dataset respectively, and the results are entirely independent each other. Followed by steps in Fig. 2 we first train the LPM and then use the LPM's trained results to train the IGMM, followed by training the TOFM with the trained results from LPM and IGMM. On the VITON-Datase training setup, each module is trained for 400K steps with batch size 4, while on the Millndo-Datasee training setup, each module is trained for 800K steps with batch size 4. Both training setups use the Adam optimizer with 81 = 0.5 and 32 0.999, and the learning rate is fixed 0.0001. Additionally, the resolution for all input and output images is 256 x 192, and we use single NVIDIA 2080Ti GPU in our experiments: we also use the same steps the training stage to test the modules. The qualitative results in the easy, medium, and hard cases respectively in the Milando-Dattsse testing stage can be seen in Fig. 4 and Fig.
3.3 QUALITATIVE RESULTS
On the VITON-Dataset. Fig. shows visual comparisons of our proposed method with VITON Han et al. 22118), CP-VTON Wang et al. (2018a), VTNFP Yu et al. (2019) and ACGPN Yang et al. (202)). To save a lot of work reproducing them (VTNFP has official code), we refer to the results from the paper of ACGPN. Note that we validate the official code of ACGPN qualitatively and quantitatively and receive the same results with ACGPN.
In comparison to VITON and CP- CTO,, VTNEP preserves more characteristics by using segmen- tation representation preserve the non-target parts, but it does not contain enough details. This

---

Published as a conference paper at ICLR 2023
Jnfenens Poam
VEDON
CIVTON
VINe
PRAIN
ACGES
AVIUN
2X Zareiniin
FUTIR
Figure 3: On the VTTON-Datased Qualitative comparisons of VITON Han et al. 2018), CP- VTON Wang et al. (2018a), VTNFP Yu et al. (2019), ACGPN Yang al. (2020) and AVTON in easy to hard levels (from top to bottom). Our method preserves more characteristics of the reference person with the LPM, and it also preserves more characteristics of the target clothes with the IGMM. What's more, AVTON can generate more realistic try-on images with the TOFM, which is good at trading off characteristics of the warped clothes and the reference person.
happens because of unawareness of the semantic layout and the relationship within the layout. ACGPN performs better than VTNFP, in that it can preserve hand details, but it also fails to generate try-on details. This is because ACGPN uses TPS to warp clothes (mentioned in Section 2.2). and uses simple UNet to fuse the features, which makes it difficult to trade off characteristics of the target clothes and the reference person (mentioned Section 2.3
However, AVTON does better both in preserving characteristics and trading off characteristics. Ben- efiting from the LPM, it predicts limbs first and then provides limb information the TOFM, which helps solve occlusion problems(e.g. the blue box on the fourth row, the arm is clearer than others). What's more, the IGMM warps clothes more reasonably to preserve styles and patterns (e.g., the blue box on the third row, sleeves the same length and patterns clear), and the TOFM makes the try-on image more realistic due to trade-off (e.g., the yellow box on the fourth row, the inner collar should be ignored). In a nutshell, AVTON can generate more realistic try-on images than VITON, CP-N VTNFP, and ACGPN.
On the GollnddoDDatased For fair comparisons, retrain CP-VTON, Outfit-VITON Neuberger ct al. (2020) and ACGPN using the Zalando-Dataset and select the best-trained models. We put on different types of clothes for person (Fig. 4 and put one type of clothes for persons of different complexity levels (Fig. 5). It is evident from the results that CP-VTON is not suitable for the all-type clothing try-on task, as CP- EP-VTON uses TPS to warp clothes that we motioned in Section 2.2 and uses a single UNet to generate limbs in the try-on step that we motioned in Section 2. O VITON performs better than CP-VTON, but the content of the generated images are incomplete (the first row in Fig. 44. This can be reasonable because the Outfit- VITON first decouples the features of the clothes and human body, and then decodes them to generate the try-on image. This procedure does not consider the perservation of spatial information, making the image incomplete. In comparison to CP-VTON and Outfit- VITON, ACGPN generates more realistic try-on results, but there are still some issues mentioned in the -TTON-Datase... However, AVTON can deal with these issues by allowing the LPM to accurately predict the color of skin, the IGMM to warp clothes accurately, and the TOFM to make results more realistic.
7

---

Published as a conference paper at ICLR 2023
Figure 4: On the Talando-Dataset. Qualitative comparisons of CP-VTON, Outfit-VITON, ACGPN and AVTON with different types of clothing. Our AVTON adapts successfully the all-type cloth- ing try-on task.
Corpe
ONNTNN
AEGPS
Vistes
CINTOS
OUEVVITIN
ACOPS
CENTON
QUEATTITS
SEEPS
IN TIT
Figure 5: On the Zalando-Dataset. Qualitative comparisons CP-VTON, Outfit ACGPN and AVTON at different complexity levels. Our AVTON adapts successfully to cross-category try- on task.
3.4 QUANTITATIVE RESULTS
We employ Structure SIMilarity (SSIM) Wang et al. (2004) and Learned Perceptual Image Patch Similarity (L.PIPS) Zhang et al.] (20T8) to measure the similarity between try-on images and groundtruths, and Inception Score Salimans et al. (2016) to measure the visual quality of try-on images. Specifically, we measure SSIM and LPIPS at different complexity levels on the VITON- Dataset, and measure them with different types of clothes the Zalando-Datinn
On the VITON-Dataset. TABLE shows quantitative comparisons of our AVTON with VITON, CP-VTON, VTNEP and ACGPN. In experiments, AVTON obtains significant lead in all these metrics over baseline methods. Specifically, the SSIM of our method improves by 0.010, 0.014, and 0.023 respectively over that of the best baseline method (i.e., ACGPN) each complexity level. For LPIPS, our method beats the best baseline method (i.e., ACGPN) by 0.028, 0.030, and 0.036 respectively at each complexity level. And our method surpasses the best baseline method (i.e.. ACGPN) by 0.195 in terms IS.
Methods VITON
Mean
/SIMS/
Medium
0.783 70.13
0.757
0.779/0.185
0.779/0.199
2,650
CRAVION
00.338
0.758/0.227
1.42//2243
0.229/0.361
2,757
VTNEPP
0.503/0.155
0880//0.155
0.501 70.158
0.788 0.170
2.764
ACGPN
0.815/0.107
0884/00101
0,861 20.110
0.82870.11
1,829
PF-AFN
0.549/0.100
0.85700005
0.345 70.091
0.521/0.110
1,883
AVTON (Vanilla)
0.513/0.135
0.120/0.128
0..10
0.151
1,880
AVTON (wio 93
0.819/0.123
0X36/00.1 0.111
0..16/0.126
/.137
2.983
AYTON LPM)
00.00
0.851 /0,084
0.852/0.093
0.649 10.101
2,859
(Full)
0.859/0,877
0.644/0.03
/0.500
0,851
Table 1: On the VITON-Dataset. SSIM and LPIPS are measured at different complexity levels.
AVTON (Vanilla), AVTON (w/o v) and AVTON (wlo LPM) are used for ablation studies.
On the Zalando-Dataset. As shown in TABLE 2 present quantitative comparisons of our AVTON with CP- VTON, Outfit-VITON, and ACGPN, in which CP-N,ON, Outfit-VITOR and ACGPN retrained as mentioned in Section
3.3
The SSIM of method improves by 0.007.
8

---

Published as a conference paper at ICLR 2023
Mediods
Mean
10p
SSIMI /LFIPS
Dotomn
IST
CP CI-V
0.758/0.880
0,732/0.195
0.392 20.170
0.812/0.131
3,556
OutI-VITON
0.733001172
0.792/0.337
ACTIPN
017870.115
0.360/0.105
0.810/0.126
0.665/0090
(Vanilia)
0.775/00152
70.126
rwia
0.81770.100
0.729
4,023
(Rull)
0.81970.115
6.788
3,976
Table On the Zalando-Datased SSIM and LPIPS are measured with different types of clothes. AVTON (Vanilla), AVTON (w/ V) and AVTON (w/o LPM) are used for ablation studies.
0.020, and 0.001 respectively over that of the best baseline method (i.e., ACGPN) for each type of clothing. For LPIPS, our method beats the best baseline method (i.e., ACGPN) by 0.014, 0.030, and 0.015 respectively for each type of clothing. And our method surpasses the best baseline method (i.c., ACGPN) by 0.099 in terms of IS.
1ind
Llottes
Merso
Clorbes
PeTtz
Reterence
Target Cloths
'Tt--nn (wM)
Try-On (Fil)
Person
Olidn
175 (60-))
V-uunton
Figure 6: Comparison of TPS and Wendland's 's Figure 7: Comparison of the results with and V-function.
without the LPM.
3.5 ABLATION STUDY
Similarly to the guantitative comparisons, evaluate the effectiveness of both the LPM and Y (Wendland's ' y f-function) using SSIM, LPIPS and IS. As shown in TABLE and TABLE 2 Wend- land's V-function plays important role, where AVTON (Full) surpasses the AVTON (w/o V)) by 0.046 and 0.008 in terms of the mean of SSIM, respectively. Here we show the visual comparison Fig. 6 where Wendland's V--functon can warp clothes locally and smoothly, while TPS warps clothes globally. Additionally, the LPM has a significant impact on LPIPS and IS. The LPIPS of AVTON (Full) reduced by 0.013 (TABLE I and the IS of AVTION (Full) is increased by 0.165 (TABLE T. As shown in Fig. 7 the results (w/o LPM) have broken arms. Note that in TABLE the of AVTON (Full) lower than that of AVTON (w/o LPM). It can be explained that the testing set contains Bottom and Whole cases, most of which do not have occlusion problems. Therefore, the prediction error caused by LPM can avoided, and the IS of the results without the LPM higher. In summary, the LPM is necessary for the cross-category try-on task in experiments.
4 CONCLUSION
Deep learning based virtual try-on system has achieved some encouraging progress recently, but there still remain several big challenges that need be solved, such as trying on arbitrary clothes of all types, trying on the clothes from one category to another and generating image-realistic results with few artifacts. In this paper, we collect a new dataset to enhance the robustness and adaptiveness of the virtual try-on model. Based on this dataset, we then propose novel virtual try-on network for handling all-type clothing try-on task (tops, bottoms, and whole clothes) and cross cross-category try-on task (e.g., long sleeves +> short sleeves or long pants t skirts, etc.). Extensive simulations and ablation study are conducted. Simulation results based on Quantitative, qualitative evaluation and user study illustrate the great superiority of our AVTON over the sfate-of-the-ar methods.

---

Published as a conference paper at ICLR 2023
REFERENCES
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7297-7306, 2018.
Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolution virtual try-on via misalignment-awwe aware normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14131-14140, 2021.
Zeyu Cui, Zekun Li, Shu Wu, Xiao Yu Zhang, and Liang Wang. Dressing as whole: Outfit compat- ibility learning based on node-wise graph neural networks. In The World Wide Web Conference, pp. 307-317, 2019.
B Delaunay, S Vide, A Lamémoire, and V De Georges. Bulletin de l'académie des sciences de l'urss. Classe des sciences mathématiques et naturelles, 1773--80,, 1934.
Xin Dong, Fuwei Zhao, Zhenyu Xie, Xijin Zhang, Daniel K Du Min Zheng, Xiang Long, Xiaodan Liang, and Jianchao Yang. Dressing in the wild by watching dance videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3480-3489, 2022.
Jean Duchon. Splines minimizing rotation-invariant semi-norms in sobolev spaces. In Constructive theory of functions of several variables, pp. 85-100. Springer, 1977.
Mike Fornefett, Karl Rohr, and Siegfried Stiehl. Elastic registration of medical images using radial basis functions with compact support. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), volume 1, pp. 402-407. IEEE, 1999.
Mike Fornefett, Karl Rohr, and H Siegfried Stichl. Radial basis functions with compact support for elastic registration medical images. Image and vision computing, 19(1-2):87-96, 2001.
Shanchuan Gao, Fankai Zeng, Lu Cheng, Jicong Fan, and Mingbo Zhao. Fashion image search via anchor-free detector. In Proceedings of the 2022 International Conference on Multimedia Retrieval, pp. 416-425, 2022.
Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo. Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5337- 5345, 2019.
Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8485-8493, 2021.
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self- supervised structure-sensitvvv learning and new benchmark for human parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 932-940, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(114113,114, 2020.
Xintong Han, Zuxuan Wu. Yu-Gang Jiang, and Larry S Davis. Learning fashion compatibility with bidirectional Istms. In Proceedings of the 25th ACM international conference on Multimedia, pp- 1078-1086, 2017.
Xintong Han, Zuxuan Wa, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 7543-7552, 2018
Ruining He, Charles Packer, and Julian McAuley. Learning compatibility across categories for het- erogeneous item recommendation. In 2016 IÉEE 16th International Conference on Data Mining (ICDM). pp. 937-942, IEFE, 2016.
10

---

Published as a conference paper at ICLR 2023
Tomoharu Iwata, Shinji Wanatabe, and Hiroshi Sawada. Fashion coordinates recommender system using photographs from fashion magazines. In IJCAI, volume 22, pp. 2262. Citeseer, 2011.
Jianbin Jiang, Tan Wang. He Yan, and Junhui Liu. Clothformer: Taming video virtual try-on in all module. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 10799-10808, 2022.
Sangyun Lee, Gyojung Gu, Sunghyun Park, Seunghwan Choi, and Jaegul Choo. High-resolution virtual try-on with misalignment and occlusion-handled conditions. In Computer Vision-ECC ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pp. 204-20422 219. Springer, 2022.
Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. arXiv preprint arXiv: 190000971 2019.
Yuncheng Li, Liangliang Cao, Jiang Zhu, and Jiebo Luo. Mining fashion outfit composition using an end-to-end deep learning approach on set data. IEEE Transactions on Multimedia, 19(8): 1946-1955, 2017.
Ziwei Liu, Ping Luo. Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Assaf Neuberger, Eran Borenstein, Bar Hilleli, Eduard Oks, and Sharon Alpert. Image based virtual try-on network from unpaired data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pultern Recognition, pp. 5184-5193, 2020.
Nilesh Pandey and Andreas Savakis. Poly-gan: Multi-conditioned gan for fashion synthesis. Neu- recomputing, 4 41::356-364, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-24-4 241. Springer, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29: 2234-2242, 2016.
Yong-Sian Shih, Kai-Yueh Yueh Chang, Hsuan-Tien Lin, and Min Sun. Compatibility family learn- ing for item recommendation and generation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic-pressre image-based virtual try-on network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 589-604, 2018a.
Ting-Chun Wang, Ming-Yi Yu Liu, Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798 80088, 2018b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13(4))600- 612. 2004.
Holger Wendland. Piecewise polynomial, positive definite and compactly supported radial functions of minimal degree. Advances in computational Mathematics, 4( 4(1):389-396, 1995.
Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photo- realistic virtual try-on by adaptively senerating-preseevvinn image content. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7850-7859, 2020.
Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xie. Vtnfp: An image-based virtual try-on network with body and clothing feature preservation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10511-10520, 2019.
11

---

Published as a conference paper at ICLR 2023
Fankai Zeng, Mingbo Zhao, Zhao Zhang, Shanchuan Gao, and Lu Cheng. Joint clothes detection and attribution prediction via anchor-free framework with decoupled representation transformer. In Proceedings of the ACM International Conference on Information & Knowledge Manage- ment, pp. 2444-2455 2022.
Wei Zeng, Mingbo Zhao, Yuan Gao, and Zhao Zhang. Tilegan: category-oriented attention-based high-quality tiled clothes generation from dressed person. NEURAL COMPUTING APPLICA- TIONS, 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Pruceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018
Mingho Zhao, Yu Liu, Xianrui Li, Zhao Zhang, and Yue Zhang. An end-to-end framework for clothing collocation based on semantic feature fusion. IEEE MultiMedia, 12-4312 2020.
12

---

APPENDIX
A THE DETAILED ANALYSIS OF THE COLLECTED ZALANDO-DATASET
In order show the superiority of the collected Zalando-Dataset we first illustrate the categories of the dataset, which can be shown in Fig. In detail, the division of dataset based on the categories and genders shown in Fig. [(a) From Fig. aa) we can see that the collected try-on dataset has involved almost all categories of clothes including both top and bottom clothes for man well as top, bottom and whole clothes for woman. Fig. [ further show the statistics and distribution of all categories of clothes according to gender and tyoes, where the images of top clothes take up 58.68% while those of bottom and whole clothes take up the remaining 41.32%. Since the conventional VITON-Dataset only involves images with top clothes while lacks of those of bottom and whole clothes, the collected Zalando-Datane is an extension to VITON-Datee by involving more clothing categories, which is good for handling real-world arbitrary try-on task.
In order to further show the superiority, we then analyze the characteristics of dataset and compare with those of VITON-Dataset Here, choose two measurements for comparisons: 1) the size of the dressed clothes taking up the whole referenced image, and 2) the complexity scores for the referenced image (only for top clothes), which are firstly defined in ACGPN Yang al. (2020) and are divided into easy, medium and hard cases. The characteristics analysis of proposed dataset and VYTON-Dataset based on two measures are shown Fig.
Sendor
Type
Top
14:00
Men
Bottom
Categaii
Zalando
Top
Hettom
Whole
(a)
(b)
Figure 1: The category division and statistics of Zalando-Dataset: (a) division of dataset; (b) statis- tics and distribution of dataset.
From Fig. a) we can see that the size of the dressed clothes taking up the whole referenced image has wide range from 0.2 to 0.6 in the collected dataset. While for VITON-Dataset, such range mainly focus on about 0.45. This means that the collected dataset has different sizes of the dressed clothes and is more extensive in data selection. This can also be more realistic and closer to the real-world cases. As a result, a try-on method that can handling different size of the dressed clothes is more useful and practical, which is good to satisfy the requirement of user. From Fig. bb6 we can see that the distribution of the score curve of the collected dataset is on the left of that of VITON-Dataset. This indicates that the complexity of the proposed dataset is higher than that of VIToN-Dataset. Therefore, method trained on such dataset can handle more complex try-on task such as limb intersections and torso occlusions, which is good enhance the robustness and adaptiveness of the model.
B ANALYSIS OF V-FUNCTION OF WENDLAND
The di-function of Wendland utilized in our work is to replace the conventional Radial basis function (RBF) in Thin-Plate Splines (TPS) Duchon (1977) for handling virtual try-on task, and we admit

---

8-
Zalando Dataset
Zalando Dataset
Viton Dataset
0.05
Viton Dataset
Hard
Medium
0.04
Easy
Percentage
5-
Percentage
4-
0.03
3-
0.02
2-
0.01
1-
0.00
0.0
0.2
0.1
0.6
1.0
20
40
60
80
100
120
140
Size (a)
Score
(b)
Figure 2: (a) the distribution of the size of the dressed clothes taking up the whole referenced image; (b) the distribution of the complexity scores for the referenced image (only for top clothes).
that it can indeed enhance the performance of try-on task. The reason for the improvement is that dh- function of Wendland is compactly supported for local deformation due to its locality, solvability and efficiency. Here, will explain by two issues: 1) why conventional RBF in TPS cannot work well compared with the fi-fuccionn of Wendland; 2) why we choose 4/,,1 -function of Wendland in work.
First, we will say the key technique of clothes warping for handling try-on task is Thin-Plate Splines (TPS) Duchon (1977); Fornefett al. (2001), which has been extensively utilized in VITON Han et al. 2018), CP-VITON Wang et al. 2018). ACGPN Yang et al. (2020) and other state-of-the-a methods. In detail, denote x the source point in the tiled clothes, y is the corresponding target point in the warped clothes, the goal of TPS is to fit nonlinear mapping function f (æ) between the source points x and the target warped points y by minimizing the following energy function:
Etps (f)
Σx
1=1 1/
K
f(x))11_
To solve the above problem, Radial basis function (RBF) is typically involved since it has natural representation of (a) for detailed introduction). To calculate the optimal f (xr), a set of controlled points in the original space {cz, i 1,2, ........ K} (the locations of which have already been fixed both in the original space and warped space) should be given. Then, RBF is to defines f(x) as follows:
K f(x) Σ wpp([ x - cs II)
(2)
where I|-I| is the usual Fuclidean norm, (p(r) is the Radial basis kernel in the formulation of yp(r) 2² log r, is a set of mapping coefficients to be learned. The optimal f(x) can be calculated by close form solution via nonlinear regression.
Here, as mentioned in Fornefett et al. (2001), the RBF is good for yielding an overall smooth de- formation and preserving geometrical characteristics, but it is compactly supported for local deformation. In detail, following the formulation of Radial basis kernel (rr) = logr,, we can see the further is from C;, the larger =|| x-c; is, then the larger Radial basis kernel (p(r) r log is (which can be shown as Fig. 3 In such case, f(x) in Eq. equation 2 tends to be very large causing that x will be unnecessarily deformed (we will show an example as follows). On the other hand, good f(x) should be compactly supported, i.c., f(z) is small given is far away from e, that the far points will not be affected by deformation. This is good for the case that small part of image is desired to be deformed, especially for handling the try-on task that the clothes part in the image is with small size. To the contrast, 4/s,6 funccion of Wendland is compactly supported, which can be shown Fig. 3
In order to further show why va..a,1 function of Wendland is better RBF based TPS, we will conduct visual example for comparison, which can be shown Fig. 4
2

---

Different
0.04
k=
0.00
20-
0.08
4.1
Ploght)
0.112
0.14
0.16
0.18
-02
0.2
0.4
0.6
0.8
(a)
(b)
Figure 3: The curve of RBF in TPS and ffnnctinn of Wendland with varied r: (a) sp(r) r² logr (b) fus.effnnttonn of Wendland with different &
TPS
Original image
Original grid
Warped image
Warped grid
(a)
Na. 1 function
Original image
Original grid
Warped image
Warped grid
(b)
Figure 4: the warped image and gird learned by the TPS and v4.3,11 a -function of Wendland: (a) TPS;
(b) 3..3,1 -function -ffunccnn of Wendland
3

---

The first and third columns in Fig. 4 the second and the fourth columns represent the original and warped grid, respectively, and the green "+" represents the controlled points both in the original space and warped space. From the above simulation results, we can see that by fixing the same controlled points, the warped grid learned by TPS is more deformed than that learned by 1a,3,11-uun -function of Wendland. This can be obviously observed that the second and third lines in the top grids of TPS has been deformed while those of 3,, -functinn of Wendland do not make any change. This has verified that the 1. of Wendland is better than RBF based TPS, which is good for handling the try-on task that the clothes part the image is with small size.
represent the original and warped images, respectively, while
C PARAMETER SELECTION FOR FUNNCFUN OF WENDLAND
We next explain why we choose 1h,, i-functnnnonn of Wendland in our work. The el-function of Wend- land is firstly developed for biomedical image registration Fornefett et al. (200T) which is adopted to replace RBF due to its locality, solvability and efficiency. Its general formulations are as follows:
[ [p(r) 0<r<1 ((r) 1
(3)
r r21
where p(r) a univariately polynomial. Let (r)) denote the univariate function, then & Rdd RR (([rl]) ((lrll)is the corresponding multivariate function in the space of dimension d. The math- ematical property of positive definiteness of e depends on the space dimension d. If a/ is positive definite on Rd, then v is also positive definite on I2 with 0 g S d. It has been proven in [e] that for given space dimension and smoothness C (R) there exists uu to a constant factor- only one function (r(r of the above formulation which is positive definite on Id and has a polynomial of minimal degree [d/2] + 3k + 1,where xx] is the floor function returning the largest integer of x. We neglect some math derivation in Fornefett et al. 2001) and give the final function follows:
vba = 12
(1 (1-r3 (2-13 (2/3)++411 (r)
with
(1- (1-r)"
0 05 <1 r21
(5)
as the truncated polynomial and
Twir) = 1Tth (r)dt)t
r 0
(6)
as the integral operator.
Following the above analysis, we can see that there are two key parameters in gi-function of Wend- land: 1) d: the space dimension to guarantee taa, positive definite; 2) k: the times for performing integral operator. Here, the dimension is 2 and the floor function [d/2] is equal no matter d = 3 or
2, we will choose different k, which can be shown as follows:
for simplicity. Then, by fixing = 3, we can list v/a -funccioo with
(r) (1 r
1 (r) (1 (4r
(7)
(r))(() -11 (1-r){(35r) + +
(r)) = (1 (1--)., (1-r) (32r*
3 + 25r + 8r +
To further compare the deformation performance with different k. we will also show the deformation grid by fixing some controlled points as in Fig. 5 From Fig. 5 the larger k is, the smoother the deformation grid is. But when > 1, the polynomial will become more complex while their deformations relatively smaller, we thereby choose à 1. Finally, analyzed above, will finally determine d = 3 and & = 1.
For the choice of the parameter in (r) of Eq. equation 4 and use Delaunay triangulation to analyze the distances between the control points, as shown in the
we first select 25 controlled points
Fig. 6
4

---

k=0 0
k- 2
k
(a)
(b)
(c)
(d)
Figure 5: the deformation grid learned with different Ir
Delaunay Triangulation
Delaunay side
Control points
Figure 6: Delaunay triangulation for analyzing the distances of the control points
a=a a
a= D
a
(a)
(b)
(c)
(d)
Figure the deformation grid learned with different a
5

---

In the Delaunay triangle, the distances between the nearest controlled points a, b, D, where D avaa + b2. In order to ensure the interaction between the controlled points, the value of must be greater than one of the three nearest neighbor distances, so we separately analyze the three cases when a, b, will be formed between two nearest points in the vertical or horizontal direction, and when a = D, it is flatter between two nearest points in vertical and horizontal directions. Finally, analyzed above, we will finally determine a 2 D.
D, as show in Figure 7 Obviously, when
6, saddle point
D ABLATION STUDY OF QUALITATIVE RESULTS
Some more simulations for the comparision between TPS and Wendland's V-function on VITON and the collected dataset can be seen in Fig. and Fig.
Here, order to further show why LPM is good for handling cross-category try-on task (e.g., long sleeves/pants <+ short slceves/pants, etc.), give another ablation study for analysis, where aim to show the full LPM can well predict some exposed arms or legs when long sleeves/pants t> short sleeves/pants, etc.) compared with LPM w/o correlation layer, LPM w/o U-Net, LPM. The simulation results are shown in Fig. TO and Fig. [1 From simulation results, we can that the performances of w/o LPM is the worst, since the features of human body and clothes are not well fused. LPM w/o correlation layer and LPM w/o U-Net can achieve relatively better performance. By meriting from the advantages of both correlation layer and U-Net structure, the full LPM can achieve the best performance of limb prediction, since the correlation layer is good to keep the correlated information of human body and clothes, while U-Net structure can well preserve the detailed information of original human body by utilizing shallow features.
w/o
E ARBITRARY CLOTHING COLLOCATION
In real life, a single clothing virtual try-on cannot meet people's needs, and clothing collocation can improve people's preference for virtual try-on. Hence, conduct additional experiment show the results of arbitrary clothing collocation (Fig. [2) During the experiment, we first try tops with our AVTON and got the intermediate results, then try on bottoms based on the intermediate results, and finally get the clothing collocation results. It can be seen from the experimental results that due to the characteristics- preeeervii function of the LPM and IGMM, the characteristics of the target clothes and the reference person can still be retained after two try-on steps. And benefit from the characteristic trade-off function of TOFM, the final try-on images are natural and realistic.
F USER STUDY
As shown in TABLE and TABLE 2 we conduct two user studies on the VITON- -ataastt and Zalando-Dataset. Within these two studies, we compare the ACGPN Yang et al. (2020) and our proposed method AVTON on the VITON-Dataset in easy, medium, and hard cases, respectively. And we compare the retrained ACGPN and our porposed method AVTON on the Solando-Datasee in the top, bottom, and whole cases, respectively. Note that we only choose the ACGPN as the baseline, as it is by far the best baseline of all the VITON-Dataset based methods. Specifically, invite 40 volunteers to complete the experiment. In each study, each volunteer is assigned 50 image pairs and is asked to select the most realistic image out of two virtual try-on results. Both studies show that the AVTON performs better than other methods in all-type clothing and cross-category try-on tasks.
REFERENCES
Jean Duchon. Splines minimizing rotation-invarian semi-norms in sobolev spaces. In Constructive theory of functions of several variables, pp. 85-100. Springer, 1977.
Mike Fornffett, Karl Rohr, and Siegfried Stichl. Radial basis functions with compact support for elastic registration of medical images. Image and vision computing, 19(1-2):87-96, 2001.

---

Target Person and Clothes
TPS
Wendland's V-function
UNE TEMME LIBRE
TENAMRE ORIING
TUMANE LIBRE
Figure 8: Compasion of image warping between TPS and Wendland's V-function on the VITON dataset

---

Target Person and Clothes
TPS
Wendland's V-function
Figure 9: Compasion of image warping between TPS and Wendland's V-function on the Zalando dataset
8

---

LPM w/o
LPM w/o U-Net
Target Clothes and Person
w/o LPM
Full LPM
Correlation
T
Figure 10: Comparison of limb prediction results with full LPM, w/o LPM, LPM w/o correlation
layer, LPM w/o U-Net structure on the AITON-Datase

---

LPM w/o
LPM w/o U-Net
Target Clothes and Person
w/o LPM
Full LPM
Correlation
Figure 11: Comparison of limb prediction results with full LPM, w/o LPM, LPM w/o correlation
layer, LPM w/o U-Net structure on the Zalando-Dater
10

---

Figure 12: The clothing collocation results in two parts. The upper left image of each part represents the reference person, the upper column each part represents the target tops, and the left column of each part represents the target bottoms. It can be seen that our AVTON can match clothes arbitrarily.
Methods
Mean
Easy
Medium
Hard
Methods
Mean
Top
Bottom
Whole
ACGPN
40.9%
39.0%
40.6%
43.0% 57.0%
ACGPN
36.1%
32.1%
40.5%
35.6%
AVTON (Pull) 59.1%
61.0%
59.4%
AVTON (Fall) 63.9%
67.9%
59.5%
64.4%
Table 1: On the VITON-Datased User study Table 2: On the Zalando-Dataser User study compares ACGPN and our proposed method compares the retrained ACGPN and our pro- posed method AVTON with different types of clothes.
AVTON different complexity levels.
Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference computer vision and pattern recogni- fion, pp. 7543-7552, 2018
Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic--rreervng image-based virtual try-on network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 589-604, 2018.
Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photo- realistic virtual try-on by adaptively generating senerating-preservingg image content. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7850-7859, 2020.
11


Reviews:
[{"review_id": "nzCO-HfEDNy", "reviewer": "Reviewer_QoKk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a method to handle arbitrary clothing in Virtual try on models. The papers argue that recent approaches in virtual try-on lack coverage in cases where the clothing styles change vastly, for example from skirt to pants or long sleeve to short sleeves. To cover such cases they proposed a limb prediction module which takes as input human and target clothing representations and produces the exposed limbs. \nSecondly, to cover the clothing deformation better, in their geometric deformation module, they modified previously used Thin-Plate Splines (TPS), by replacing TPS RBF functions with the Phi function of Wendland Wendland. The module takes as input human representations from the Limb prediction module, in addition to clothing and human representations and produces deformed clothing.\n\nFinally, in their Trade-off fusion module (TOFM) they adopt a GAN-based method which uses a pair of UNet as a generator and a multi-scale discriminator D as in pix2pixHD.  The authors argue that to preserve the details of clothing and body parts in the final render image, they use the wrapped clothed features from the geometric deformation module and refined human features ( from LPM) are fused by element-wise sum and concatenated with warped clothed features and human representation features to predict mask and rendered person. \n\n\n", "summary_of_the_review": "Even though the paper is written clearly, unfortunately it lacks coverage of state of the arts, specifically the works that have been done in 2021, where most of them address the same issues proposed in this work. Comparing this work with last year's works I find the novelty of the paper limited, and the results are not superior. \n\nSome references to a section on how the body parts were generated are also missing. \n\n\n", "strength_and_weaknesses": "Strength: \nThe paper is well written,  is  easy to read and follow, and the descriptions of each module is clear .\n\nThe model is evaluated on two different dataset, and it shows improvement compared to VTON, CPVTON, and VTNF.  While showing marginal improvement to ACGPN approach which uses similar ideas as described in this paper. \n\nTheir LPM uses a new input type, that represents the person's body better, and does not include clothing items in it, resulting in better final output. \n\nWeakness: \nThe paper related works are outdated and did not cover several important works which have been done in 2021. For example it would have been good to compare their approach specifically with :\n\n1- VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization, Choi, Seunghwan and Park, Sunghyun and Lee, Minsoo and Choo, Jaegul, CVPR 2021, \n\n2- Parser-Free Virtual Try-on via Distilling Appearance Flows, CVPR 2021\n\n3- Dressing in the Wild by Watching Dance Videos, CVPR2022\n\n4- High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions, Lee, Sangyun and Gu, Gyojung and Park, Sunghyun and Choi, Seunghwan and Choo, Jaegul, ECCV2022\n\nMost of the  work that has been done during the last year covers the same issues raised in this paper and achieved superior results compared to this paper.  For example in 5- ZFlow: Gated Appearance Flow-based Virtual Try-on with 3D Priors, ICCV2021, uses similar representations as input to their model as in LPM.\n\n\nIt is not clear how the segmentation of the body parts(p) in LPM are generated. The reference to the section is missing. I guess the paper has a missing section explaining how the masks are generated?\n\nnot sure if their collected dataset from zalando is also anything new as similar dataset was used in VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization\n\nCan the model generate a person in pants when only the upper part of the person is given as input?\nIt would be also good to see the result of the model on diverse clothing, for example a person in tight fitting clothing to a loose dress.\n\nIn figure 6 it would be good to compare their cloth deformation module with the work in ACGPN.\n\nIt is preferable to report the results with generated image resolution in the tables for better comparisons across different works. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7FIefE6ZLX", "reviewer": "Reviewer_NW3C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel virtual try-on network with a newly collected dataset for handling the all-type clothes try-on task and the cross-category try-on task. It consists of three sub-modules: 1) Limbs Prediction Module, which is designed for predicting the human body parts by preserving the characteristics of the reference person; 2) Improved Geometric Matching Module, where the radial basis function of TPS is replaced by the Wendland’s ψ-function to warp clothes more reasonably; 3) Trade-Off Fusion Module, which aims to trade off the characteristics of the warped clothes and the reference person. Extensive experiments are conducted to demonstrate the superior performance of the proposed method.", "summary_of_the_review": "Virtual try-on has great commercial value and application prospects. In this paper, three sub-modules are designed to handle the all-type clothes try-on task and the cross-category try-on task. It is relatively novel to make an improvement in the mathematical formulation of TPS. However, the architecture basically follows the previous methods, and it is not clear which part is explicitly designed for all-type clothes. Besides, it lacks comparisons with the latest methods in recent two years.", "strength_and_weaknesses": "Paper Strengths:\n1. The functions and principles of the three sub-modules are clearly described and illustrated.\n2. This paper makes an improvement in the mathematical formulation of TPS, and there are extensive analysis contents to verify its effectiveness.\n3. This paper provides a new dataset, which contains a richer variety of clothes.\nPaper Weakness:\n1. Compared with the top clothes, what is the main difficulty of the bottom clothes and whole clothes? In addition to the newly collected dataset, whether the algorithm part is specially designed to handle different clothes categories?\n2. The dataset collected by this paper is not introduced in detail, such as the number, resolution, and proportion of clothes categories.\n3. From 2021 to 2022, there are a lot of works on image-based virtual try-on, however, this method is not compared with them, the latest work compared in this paper is ACGPN in 2020.\n4. Most extensive methods use FID to measure the similarity of data distributions between the generated results and the referenced images, which does not appear in this paper.\n5. Several grammar and writing errors can be found in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5bvTEFC88Yh", "reviewer": "Reviewer_rQV8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new architecture for virtual try-on and a new method to warp the clothing image. To improve the performance on a variety of different clothes categories (tops, lower-body clothes as skirts and trousers, and dresses) they collect a new dataset. This dataset, called Zalando dataset, is richer than the viton dataset in terms of variety and thus provides a preferable training and testing ground. Based on the dataset, they introduce Wendland functions as RBFs to warp clothes and an Arbitrary Virtual Try-On Network (AVTON) for synthesizing try-on image on all-type clothes. In the experiment, they conduct both qualitative and quantitative comparisons to evaluate their approach against state-of-art methods. Also, a user study is conducted to evaluate the result.\n\n", "summary_of_the_review": "There are some minor issues should be addressed:\n1.\tThe author should provide a more detailed introduction of the dataset.\n2.\tInclude some failure cases may help to evaluate the limitation of the system.\nBased on the evaluation of the paper, I will make accept decision for this work.\n", "strength_and_weaknesses": "Strengths:\n1.\tThe collected all-type cloth data set will contribute to other researcher work in the field.\n2.\tThe rich experimental section. Good numerical results in terms of existing metrics (LPIPS, SSIM, IS) and on the user study indicate better performance in the virtual try-on task. Also, the rich qualitative analysis helps appreciating the model qualities.\n3.\tThe proposed system includes modules for limbs prediction, geometric matching, and trade-off fusion. Their limb prediction module can help to preserve body characters for synthesizing cross-category try-on images. \n4.\tThe introduction of the Wendland's Ψ-function in the warping module. To the best of my knowledge, this is the first work to propose to employ this support after the well-established TPS-based warping.\n5.\tThe overview figure is clear enough to help user understand the structure of the system.\nWeakness:\nInclude some failure cases may help to evaluate the limitation of the system.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0cfi4M1QJ1T", "reviewer": "Reviewer_C9dv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops an arbitrary virtual try on network by preserving characteristics representation and trade-off between body and clothes. In general, this work is an 2d virtual try-on task, which aims to handle three challengeable issues in GAN based try-on task. How to solve cross-category try-on task, how to well warp the targeted clothes according to the pose and body of targeted human, and how to merge the clothes and body information into the final try-on figures.  After careful reading, the main contributions of the proposed work can be summarized as follows: 1) to develop a new framework with three stages for well solving the above challengeable problems. In detail, the LPM mod\tule is to predict the exposed limbs when long pants are changed to short pants, which tends to be confronted in cross-category try-on task; 2) the IGM module is to designed to warp clothes according to the targeted person based on the locally linear information; 3) trade-off fusion module is to fusion the targeted body and targeted clothes information into final virtual try-on performance by replacing artifact parts. All the three module is targeted and practical. The simulation results including ablation study and comparison work based on new collected dataset are also convincing to verify the effectiveness of the proposed work.\n", "summary_of_the_review": "Although there are some issues need to be addressed, based on the quality and novelty of this paper, I believe it can be accepted.", "strength_and_weaknesses": "Strengths:\n1. The work is easy to understand and well structured.\n2. The novelties of the work are sufficient. The work has developed three modules in the try-on network framework, which are targeted to solve the problems confronted in GAN based 2D virtual try-on task. Specifically, the LPM module is the most important novelty developed in the proposed work. It is to predict the exposed or hidden limbs when perform cross-category try-on task, i.e., long pants<->short pants, this can be the first work pointed out, through which the arbitrary virtual try-on can be realized. \n3. Extensive simulations have been conducted to verify the effectiveness of the proposed work both in regular paper and appendix parts. The ablation study and compared work show the superiority of the proposed method.\n4. The authors have also collected a new dataset with more categories to train the network for cross-category and arbitrary-category try-on tasks. Based on the dataset, more types of simulations have been conducted to further analyze the proposed method, including arbitrary clothing collocation, user study, qualified analysis on LPM and IGMM.\n\nWeaknesses:\n1. The final conclusion is not well written. Some future work can be involved to show how the proposed work can handle real-world application, instead of just writing some sentences from introduction or abstract.\n2. The structure of each module can be more detailed. If possible, such structure can be illustrated in the appendix. In addition, some data processing strategy should also be provided.\n3. How to get ground truth $c_r$ and $l_r$. Does it need additional annotation? I suggest the authors can explain this point.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster