Towards Hard-pose Virtual Try-on via 3D-aware Global Correspondence Learning
Authors: Zaiyu Huang
Hanhui Li
Zhenyu Xie
Michael Kampffmeyer
qingling Cai
Xiaodan Liang
Venue: NeurIPS 2022 Accept (2022)

Abstract: In this paper, we target image-based person-to-person virtual try-on in the presence of diverse poses and large viewpoint variations. Existing methods are restricted in this setting as they estimate garment warping flows mainly based on 2D poses and appearance, which omits the geometric prior of the 3D human body shape.
Moreover, current garment warping methods are confined to localized regions, which makes them ineffective in capturing long-range dependencies and results in inferior flows with artifacts.
To tackle these issues, we present 3D-aware global correspondences, which are reliable flows that jointly encode global semantic correlations, local deformations, and geometric priors of 3D human bodies. Particularly, given an image pair depicting the source and target person, (a) we first obtain their pose-aware and high-level representations via two encoders, and introduce a coarse-to-fine decoder with multiple refinement modules to predict the pixel-wise global correspondence. (b) 3D parametric human models inferred from images are incorporated as priors to regularize the correspondence refinement process so that our flows can be 3D-aware and better handle variations of pose and viewpoint. (c) Finally, an adversarial generator takes the garment warped by the 3D-aware flow, and the image of the target person as inputs, to synthesize the photo-realistic try-on result. Extensive experiments on public benchmarks and our selected HardPose test set demonstrate the superiority of our method against state-of-the-art try-on approaches.

Towards Hard-pose Virtual Try-on via 3D-aware Global Correspondence Learning
Zaiyu Huang Hanhui Li Zhenyu Xie Michael Kampffmeyer', Qingling Cai' Xiaodan Liang
Shenzhen Campus of Sun Yat-Sen University
BByteDance, UiT The Arctic University Norway
*Peng Cheng Laboratory
(huangzy225, xiezhy6} myaul edu. cn {lihh77, caiqingl}
@mail. @mail.sysu sysu. edu. cn
michael. c. wwwppffmyyrrrutt.commm no, xdliang32@gaii.coom
Abstract
In this paper, we target image-based person-to-person virtual try-on in the presence diverse poses and large viewpoint variations. Existing methods are restricted in this setting as they estimate garment warping flows mainly based on 2D poses and appearance, which omits the geometric prior of the 3D human body shape. Moreover, current garment warping methods are confined to localized regions, which makes them ineffective in capturing long-range dependencies and results inferior flows with artifacts. To tackle these issues, we present 3D-aware global correspondences, which are reliable flows that jointly encode global semantic corre- lations, local deformations, and geometric priors of 3D human bodies. Particularly, given an image pair depicting the source and target person, (a) we first obtain their pose-aware and high-level representations via two encoders, and introduce coarse-to- decoder with multiple refinement modules to predict the pixel-wise global correspondence. (b) 3D parametric human models inferred from images are incorporated as priors to regularize the correspondence refinement process so that our flows can be 3D-aware and better handle variations of pose and viewpoint. (c) Finally, an adversarial generator takes the garment warped by the 3D-aware flow, and the image of the target person inputs, synthesize the photo-realistic try-on result. Extensive experiments on public benchmarks and our HardPose test set demonstrate the superiority our method against the SOTA try-on approaches.
Introduction
Current image-based virtual try-on (VTON) methods typically include two key modules, namely warping module that transfers a given garment to the desired pose and a blending module/generator that synthesizes the try-on result given the target person and the warped garment. To facilitate high-quality VTON, it is therefore crucial to accurately preserve garment details (e.g., color, texture, difficult problem, which most existing
shape, pattern) in the warping stage. This, however, methods try to address by relying on thin plate splines (TPS) 1 or appearance flows 2
While TPS is well-studied and efficient method that performs the warping based on set of control points and has a closed-form solution, is prone to inaccurate transformations under large geometric deformations B. To further improve the warping quality, there has been shift towards directly
Bott authors contributed equally. "ooresppnniigg authors. Our code will be available at 3D-GCL
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Appasrance
Trysan
& Apprammn
Try-an
Appasrance
Tryan
Apprsrannc
Try-an
Figure 1: Visual examples of the proposed 3D-aware global correspondence learning method for person-to-person virtual try-on. Our method can accurately preserve garment textures and generate photo-realistic results under diverse poses and viewpoints.
estimating appearance flow fields by leveraging the capacity of deep neural networks. These methods perform well in relatively casy try-on scenarios, such as transferring flat, in-shop garments with pure colors simple patterns to desired poses, but still suffer from severe artifacts in unconstrained scenes like person-to-person VTON, where varying poses and viewpoints commonly exist.
These artifacts are caused by the following two reasons: First, due to the lack of 3D geometric guidance in current methods, flows estimated merely based on appearance and 2D poses are unable to represent the deformable, non-rigid characteristics of garments, which results in texture distortions and improper warping. Second, for the purpose of efficiency, mainstream flow-based networks are usually constructed by stacking modules with limited receptive fields, e.g., ClothFlow 4 simply uses convolutional layer to predict its initial cloth flow. As the effective area of a network module may be small fraction of its theoretical receptive field (5), flows estimated by existing methods are de facto restricted to local areas and cannot leverage long-range dependencies effectively to tackle pose variations and occlusions.
that depiet cross- intaann/teneemainnn semantic mappings could be potential alternative to appearance flows for tackling pose and viewpoint variations, because they exploit correlations among high-level concepts, instead of appearance-oriented pixel-wise offsets. However, without extra guidanceeregularizztion we observe that these approaches tend to focus on large and apparent structures (e.g., human pose) while ignoring subtle details and thus have detrimental effect the VTON user experience. Moreover, the complexity of calculating high- resolution correspondences directly is expensive (O(2²), where is the resolution), which further hinders its subsequent processes and downstream applications.
Visual correspondences
feasible solution is to leverage 3D human models (e.g. SMPL dences that are less ambiguous. However, a naive implementation that simply replaces the 2D pose representation in a network with its 3D counterpart is ineffective. This is because in our hard-poss VTON task, these 3D human models fail to predict the body mesh correctly and inevitably introduce noise to the estimated flow. Besides, employing 3D human models will lead considerably lower inference speed [9.
g to generate 3D 3Daaware correspon-
Therefore, we propose a 3D-aware global correspondence learning (3D-GCL) method to tackle the aforementioned issues. The key idea of our 3D-GCL method is to leverage geometric priors of 3D human bodies guide the correspondences, so that modelling non-rigid deformations of garments can be more effective. Unlike previous methods that incorporate 3D information into their inputs explicitly, we consider the implicit supervision of 3D priors our correspondence learning process. In this way, not only our correspondences can be 3D-aware, but also the side effects of improper 3D information inferred from images can alleviated. Specifically, given a source image and target image, we first obtain their high-level and pose-aware feature maps via two parallel encoder networks, as well as the global correspondence via computing the correlation between the feature maps. After that, multiple geometry-awe aware correspondence refinement modules (GACRMs) are introduced to

---

upsample and refine the global correspondence stage-by-stage, thereby efficiently constructing the final full-resolution correspondences. More importantly, for each of the GACRMs, an elaborately designed 3D regularization loss is adopted to provide the GACRMs with the ability to leverage 31) geometric priors. At last, we warp the garment with the final correspondence, combine it with the image of the target person, and synthesize the try-on result via an adversarial generator Experiments on public benchmarks and our manually selected HardPose test set demonstrate that our 3D-GCL network synthesizes photo-realistic virtual try-on results that accurately preserve garment structures and textures even for hard-pose cases,
Our contribution can be summarized as follows:
We elaborately design a novel 3D-aware global correspondence learning framework for virtual try-on in the presence of diverse pose variations and viewpoint changes.
We present a geometry-aware correspondence refinement module and a 3D correspondence regu- larization loss, which together facilitate the understanding of geometric information and ensure the preservation of garment textures in our network.
We conduct extensive experiments to validate the superiority of our method against state-of-the-art approaches, terms of both garment warping and try-on synthesis.
2
Related Work
Virtual Try-on The most challenging aspect of VTON is the preservation of diverse textures and logos, which is addressed in current approaches by heavily relying on warping modules. One popular technique to warp the in-shop garment that has been employed by several early approaches [12 13 to utilize the TPS []. However, while these models can maintain garment details, they tend to introduce undesired distortions and artifacts under large pose deformations. To alleviate these artifacts, extra constraints and visual cues have been exploited [14] 15] to refine the control points of the TPS. However, these approaches are still limited by the sparse control-point-bssee representation of the TPS. Hence, there has been shift to utilize networks to predict dense flow fields directly 4 [6]. which has considerably improved in-shop VTON performance. Nevertheless, as we have emphasized, existing methods are unable to precisely represent long-range and local deformations via a single flow, and hence they are restricted in the hard-pose VTON task.
3D-aware Synthesis Models Recent advances in 3D pose estimation and reconstruction (c.g..
[1]]. Despite the effectiveness of 3D human models in modeling local dependencies, current 3D-aware synthesis models struggle to predict long-range deformations due to their unstable gradient propagation. This might be caused by inaccurate 3D model fitting, intertwined flow and feature gradients, and the use of bilinear sampling (22). Contrary to existing methods, we first predict global correspondence, and then augment it via the regularization provided by 3D human models. Hence we can circumvent the difficulty of learning long-range flows merely based on 3D human models.
88 (77788) facilitate the use of 3D human models in synthesis methods 3 9 19
20
Correspondence Learning Several cross-domain image synthesis methods based on correspondence learning 77 20 23 24] have been proposed recently, among which [20] adopts the same pose representation as in the proposed method. However, its learned flows are restricted to predefined UV space, which limits its scalability and performance. The proposed method instead does not include such restrictions, as its correspondences refined by 3D human models are robust to various poses and body shapes.
3 3D-awars Global Correspondence Learning
Our goal is to overcome the limitations of current try-on methods in handling large pose and viewpoint variations. This requires to estimate flows that 1) can model long-range correspondences to tackle occlusions, rare poses, and novel views; and 2) can preserve textures and local structures of garments to ensure accurate try-on results. To this end, given a pair of person images, our proposed 3D-GCL
$ Note, while [0] incorporates the 3D prior into the training the network, the intention and derivation of their 3D prior are different from our 3D-GCL. Similarly, while [1I] utilizes the global information for flow estimation, it does not explicitly model the correspondence between the source and target feature. A detailed discussion of these two methods is included in the supplementary.
3

---

Stage-I:
Comespondence Estimation
Source Person
TUY
Clothes
Festure Pyramid
Flow
Ouiput Flow
Flow
Target
IUV
Feature Pyramid
Cocrelation
GACRM
GACRM
Enceder
Stage-1I II: Try-on Synthesis
Try-on Output
Genemter
Warping
Figure 2: Overview of the proposed two-stage 3D-GCL framework. In the first stage, the 3D-aware global correspondence is predicted by the correlation between the pose-aware features extracted from the input images and is refined by multiple GACRMs. In the second stage, the target garment warped by the 3D-aware global correspondence, and fed into generator along with the image of the target person to generate the try-on result.
network (Sec. 3.1 first estimates a global flow via the correlation between their high-level 3D pose-aware features (Sec. 3.2 and then adopts multiple GACRMs to guide the global flow in depicting local deformations (Sec. 3.3) Our training objectives are provided in Sec. 3.4
3.1 Network Architecture
Given an image Ig of the source person wearing garment G and a target person image I, our task is to generate photo-realistic try-on image where the target person is wearing G. As shown in Fig. [2] our 3D-GCL network tackles this task in a two-stage manner: In the first stage, we employ an efficient encoder-decoder architecture [(a,, I,) estimate the 3D-aware global correspondence for warping IR x3 x R x
- IR H x x2, where H and W denote the image height and width. In the second stage, we adopt conditional generator g to synthesize the final try-on result I/- where G' E R " x3 is the warped garment. Choices for realizing g are flexible, and we simply employ modified StyleGAN2 [25). where replace the original style normalization strategy with spatially-adaptive normalization [26] to better maintain the spatial structures in garments. The details of our network architecture are provided the supplementary material.
G, i.e., f
3.2 Pose-aware Global Correspondence Estimation
Without the proper guidance of 3D pose information, existing appearance flows often fail to distinguish body parts (e.g., torso and arms) under heavy occlusions and pose variations. Moreover, full-resolution visual correspondences are computationally expensive. Therefore, we propose to exploit features that are pose-aware and compact for correlation learning, so that these two issues can be tackled simultaneously.
Specifically, we first employ DensePose 17 on I, and I to obtain their corresponding IUV maps P. xxxx3 which are the pixel-wise estimations of 3D parametric human body (i.c.. SMPL [8) projected on images. In contrast to previous appearance flow methods 2 we then extract features directly from P., and P. rather than from the input images segmentation masks This is reasonable, as the I channel and the UV channels of P. and P: have encoded semantic parts and positions of the 3D human body, respectively. Furthermore, this allows our network disentangle irrelevant appearance information (e.g., faces and skin tones) and focus on pose and shape variations. Following the same motivation, we also leverage a human parser (27 to obtain the segmentation of the target garment G. Two parallel fully convolutional encoders are adopted for feature extraction, ie., a source branch that takes the concatenation of P. and G as input and a target branch with input P.. Let x, y I NCC denote the output features of the source branch and the target branch,
'The ablation study for this choice is included in the supplementary material.

---

(b)
Cal
Conv
Warp
Correlation
(d)
Warp
PatchMatch
ConvGRU
Warp
COnvGRU
Supervise
Caxtilac
SMPS
Office
Flow
Figure 3: Comparison of flow estimation and refinement strategies. (a) Feature concatenation and convolution 4 (b) Feature correlation [28 13]. (c) GRU-assisted PatchMatch [24]. (d) The proposed GACRM. Flows computed from SMPL are exploited as the geometric prior for supervising the GACRM during the training phase, and consequently can obtain precise flows without computationally expensive operations like high-resolution feature correlations and PatchMatch.
W is the scaled spatial resolution and is the number feature channels. where x & throughout this paper, that N < H W, which ensures that the estimation of the correspondence computationally efficient. We calculate the correlation between x and y as follows:
X1y1
XIYN
o
XNY1
XNYN
where Oij, S Lij. N measures the semantic and pose similarity between the i-th feature vector in x and the j-th feature vector in y. As contains the similarities between all possible spatial combinations of feature vectors in x and it is able to capture long-range variations. Hence we consider our global correspondence. A coarse warping result be obtained via the linear combination of G weighted by o directly, i.e., we can down-sample G and reshape it to N x 3, and then obtain G' as G' oG. However. the sampl-nnling process will damage the quality of G severely. Therefore, we present the GACRM in the next section, which generates the full-resolution and accurate flow based on the global correspondence.
Comparison with other correlation based methods. Several recently proposed methods 7 24 also consider feature correlation for correspondence estimation. However, they target image translation across different domains (e.g., from sketches to photos) while our focus is on estimating global correspondences warp garments (from poses poses). Besides, they rely 2D keypoint-based representation of poses [29] that is sparse and cannot represent body shapes, while we adopt the 3D robust estimated correspondences. More importantly,
dense representation of poses that leads our estimated global correspondences serve as the base flows and are refined by GACRMs to tackle both long-range and local pose variations.
3.3 Geometry-aware Correspondence Refinement
A potential issue of correlation based global appearances is that they ignore the spatial structures of features (as in Eq. L Consequently, is hard to ensure local consistency in the global appearances. Even worse, the correlation between high-level semantic features tends to generate over-smooth flows that cannot preserve garment details such as patterns and logos. Therefore, we present the GACRM and 3D regularization loss leverage the geometric prior of 3D human bodies and tackle these issues.
To begin with, we convert o into flow fo of size x W via applying the argmax function. To incorporate local details into fo, we exploit the etrly-stage feature pyramids both the source branch and the target branch. Assume both feature pyramids consist of L layers (as shown in Fig. 2 we utilize a GACRM for each layer, which can formulated as follows:
g
f f" + h(x(+1 Y1+1, f()),
where < I < is the layer index. x; and y: denote feature maps of the I-th layer of the source and target feature pyramids and f" is the uc-scaled flow from the previous layer. h is responsible for
5

---

predicting a residual flow to enhance fi", which is realized simply by convolutional gated recurrent unit 22 (ConvGRU) taking x warped by f and y: as inputs, as shown Fig.
To incorporate the geometric prior of 3D human bodies into our correspondence refinement process, we propose novel 3D regularization loss for the global correspondence and its corresponding refined flows. Specifically, we adopt an off-the-shell 3D mesh regressor [30] to obtain the SMPL models of the source person and the target person. Following [9] render the two SMPL models to obtain visibility mask, and calculate a SMPL flow measuring the disparity between the source person mesh and the target person mesh projected on the 2D image plane. The SMPL flow utilized as the pseudo ground-trun in our 3D regularization loss, which is defined as follows:
C= Cy:
(3)
where Lo and y are loss terms for the global correspondence and the refined flows. In the remainder
of this section, for the conciseness of notation and presentation, we assume that all related variables in the 3D regularization loss are resized accordingly. To define L., note that it is common practice to conduct interpolations (e.g.. bilinear sampling) flow based warping, which are similar to the aforementioned coarse warping based on the global correspondence. Therefore, we generate pscudo ground-truth correspondence by scattering the interpolation weights of each pixel to its corresponding position in the correlation matrix, making the image warped by pscudo ground- truth correspondence visually close to its counterpart warped by the SMPL flow. Formally, we obtain the ground-truth correspondence g NxN follows:
Wj if (E(( and G = Σ GrGK
gf 1j
KER(1)
(4)
0,
otherwise
where (i)) denotes the reference points given i as the target interpolation point, wy is the interpolation weight of the reference point j. and G denotes the ground-trun warped garment. We then define Co as follows:
c. |I o (o -)GG||,
(5)
where M E {0, 1)N is binary mask with Mij = [o > 0j. [+] and @ denote the Iverson bracket and element- wise product, respectively. For L.,, we consider the 1 loss between the refined flows and the SMPL flow follows:
f₁= VO -6))
(6)
1=1
where V E {0, 13 and fot denote the visibility mask and the SMPL flow, respectively.
The above 3D) regularization loss encourages network to predict flows that agree well with
geometric deformations of 3D human bodies. This avoids the need of introducing complicated modules for imposing local constraints on flows, such as PatchMatch (24]. local attention [22), or multi-stage correlations [15]. Consequently, we can adopt an elegant and efficient architecture for the GACRM, while obtaining pixel-wise and precise global correspondences.
3.4 Objective Functions
Since the 3D-GCL network consists of two stages, we train its correspondence estimation subnetwork in the first stage and the generator for try-on synthesis in the second stage separately. In addition to the proposed 31) regularization loss Ly. we follow (20 25 and use the l loss L1, the perceptual loss Lepere and the adversarial loss Ludo for training. The total loss for the correspondence estimation stage C. is therefore defined as follows:
L. - C. + + Aperefpere:
where Ay. X9,- and A Aere are weighting hyperparameters, Li, and Lpere are applied to the garments warped by fo..., fr. The total loss for the generator Cg defined as follows:
Cg Nady Lado N/, f7, + Al Coeree
(8)
where Nodv. >8, and Agere are hyperparameters.

---

Guemett
Persoa
CoCorNe-12
wilow
OCM
AFRM
Oun
Gammen
Prnon
willow
OCM
AFWM
Ourr
Figure 4: Visual comparison of clothes warped by our 3D-aware global correspondences and other state-of-the-art warping methods. Our method better preserves garment details and structures under diverse poses and viewpoints.
Gomment
WPllor
PRAPN
WFoo
Figure 5: Visual comparison of our method and state-of-the-art VTON approaches. Our method is more robust to hard-pose cases and synthesizes photo-realistic try-on results.
4 Experiments
In this section, conduct extensive experiments to validate the effectiveness of the proposed 3D- GCL network. Due to the page limitation, the interested reader is refer to the supplemental material for more experimental settings and results.
Datasets Our experiments are conducted on two open-source datasets, DeepFashion 31 and MPV (32), which contain 52,712 and 37,723 fashion images, respectively. To ensure fair compari- son, we follow the train/test split of 20 22 [K] on the DeepFashion dataset and use the original split of MPV. Additionally, pairs without properly detected IUV maps SMPL models filtered out. In the test phase, query persons on the test list are further shuffled to match the task setting VTON. In this way, get 201,622/8,564 train/test pairs for DeepFashion and 22.236/10,544 train/test pairs for MPV. Furthermore, as this paper targets hard-pose VTON, we manually select test image pairs from DeepFashion and MPV to construct the HardPose test set. The HardPose test set consists of 2,406 image pairs, where each test pair contains diverse pose changes or large viewpoint variations. The details and visual examples of this HardPose test set can be found in the supplemental material.
Evaluation Metrics We evaluate the performance of both garment warping and final try-on synthesis. For the warping stage, we apply LPIPS 34 to evaluate the similarity between the warped garments and the corresponding ground-trute truths extracted by the human parser [27). We also adopt the mean Intersection over Union (mloU) metric evaluate the semantie-correetness of the estimated flows by comparing the segmentation maps of the warped garments with their ground-truths. Furthermore, following [44]] [5 21 35), we perform human evaluation by inviting 40 volunteers to select the warped clothes with the best visual quality among our method and the baseline approaches.
For the try-on stage, note that ground-truie truth try-on results are not available. Thus we select the widely used unpaired Fréchet Inception Distance (FID) to quantify the quality of the synthesis results. We
also perform a human evaluation study following recent virtual try-on methods [4 15 21 40
volunteers are invited to fill questionnaire, in which they are presented with the try-on results of all methods and are asked to select the most realistic one.

---

FullSet
HardPose
Method
DeepFashion
MPV
DeepFashion
MPV
mloUt
LPIPS
mloU 1
LPIPS
mloU f
HE1 f
LPIPS
mloU
HE 1
LPIPS 4
AFWM 153
71.24
0.1656
74,56
0.1012
63.36
25.63%
0.1648
66.63
32.50%
0.1014
CoCosNet-v2 24
0.2269
0.1873
2.375%
0.2251
3.750%
0,1800
CCM 20
62.74
0.1997
66.75
0.1655
65.17
16.75%
0.1865
63.92
8.625%
0.1575
wFlow
0.1686
0.1314
13.38%
0.1699
10.00%
0.1302
3D-GCL((Our)
75.76
0.1725
75.46
0.1318
74.22
41.88%
0.1732
71.89
45.13%
0.1345
Table 1: Comparison of warped clothes produced by different methods.
FallSet
HardPose
Method
DeepFnshion
MPV
DeepFashion
MPV
FID 4
HE* f
FID 4
HET
FID 4
HE* 7
FID 4
HE*
PRAFN 15
21.49
7.625%
10.62
12.50%
22.34
7.833%
14.35
14.67%
Djor 30
26.58
5.375%
26.62
3,875%
27.49
3.500%
30.70
4,500%
Pose with style
20
16.35
27.38%
6.78
20.88%
18,50
26.17%
9.808
19.83%
wFlow
19.81
11.63%
12.81
13.25%
22.97
9.667%
20.41
11.33%
3D GCL (OCl.OOu) (Ours)
10.58
48.00 %
6.00
49.50%
11.37
52.83%
8,776
49.67 %
Table 2: Comparison of final try-on results produced by different methods.
Implementation Details The proposed 3D-GCL network is implemented in PyTorch and trained with 4 Tesla V100 GPUs We first train the correspondence estimation subnetwork for 20 epochs with batch-size of 8, and then follow the settings of [20 25 to train the try-on generator.
Baselines We compare the warping and try-on performance of our model with publicly-available state-of-the-art methods. For the warping stage, we leverage the FlowNet backbone [28] for all methods and consider the warping models of AFWM [15), CCM (20), and wFlow [21] as the baseline methods. For the evaluation on VTON, we select four cutting-edge methods, including PBAFN [15]
Dior (36] Pose-with-Style
20 and wFlow 21
4.1 Qualitative results
As demonstrated in Fig. conventional warping strategies adopted in PBAFN 15] and Dior 36 fail
to capture correct correspondences in images. Although they employ extra smoothness constraint [5]. their warped garments are often violently distorted to fill the target silhouette facing complex poses and view-point changes. Moreover, they also ignore 3D information such orientations and body geometry, and thus harm the visual quality of their warped garments. The hybrid approaches Pose-with-Style [20] and wFlow 21 struggle to fuse different flow-based warping methods in a harmonious way. wFlow produces considerably blurry results since its composition masks are unreliable in low confidence regions. Pose-with-Styyf on the other hand only warps pixels in the pre-defined UV space of DensePose, which limits its application for diverse garment types such as skirts and dresses. On contrary, our method produces favourable results even in challenging scenarios with large pose and viewpoint changes. Note that with the help of global correspondence learning, our method has a larger receptive field and able to also generate more compelling results in front view transformations. Fig. 5 iilluuttases the visual comparison of our try-on results with the baseline methods. As mentioned in Section [] the try-on results are significantly affected by the quality of the deformed garments. Our method, with more accurate and harmonious estimated correspondences, outperforms the other four baseline methods visual quality considerably.
4.2 Quantitative results
The quantitative comparison between our correspondence estimation method against other baselines is reported in Table I Our method outperforms existing advanced warping methods such CoCosNet-
v2 (24] and CCM 20], and achieves comparable performance to wFlow (2I] on the LPIPS metric. Note that AFWM 15 obtains considerably lower LPIPS scores. This is mainly because AFWM adopts the expensive multi-stage feature correlations and encourages extremely smooth flows. Yet such a strategy tends to ignore the spatial and semantic structures garments, and consequently yields inferior visual quality in complex scenarios. To validate this, we also measure the mloU metric and assess the human evaluation (HE) score. As reported in Table our method obtains the highest mIoU scores, indicating that our warped garments more semantically accurate. Our method also
8

---

Ponum
Guement
ST-OCL+
3D-GCT+
3D-OCL
3D-CII
Penor
Gament
MDOOCT+
Figure 6: Visual comparison the proposed 3D-GCL network with different configurations.
Method
Global Correspondence
3D Regularization Loss
mloU
FID
3D-GCL
X
X
50.21
17.25
3D-GCL
X
57.36
11.53
3D-GCL
62.06
10.98
3D-GCL
74.22
10.58
Table 3: Ablation study of the proposed 3D-GCL network on DeepFashion [31].
obtains the highest human evaluation percentage, which indicates that it produces the best visual quality for warped garments under large viewpoint changes and pose variations.
As shown in Table 2 our method also outperforms other statc-of-the-ar VTON methods persistently across different test sets. Our method obtains the lowest FID scores, which indicates that method can produce high-fideliyy try-on results. This is also validated through the human evaluation. Note that most of the presented methods perform better the MPV dataset than the DeepFashion dataset. This is because images on the MPV dataset contain less pose variations and viewpoint changes. On the DeepFashion dataset, our method still surpasses the second best method (Pose-with-Style) by considerable margin, which suggests that our method can generalize well in complex scenarios.
4.3 Ablation Study
To validate the effectiveness of the global correspondence estimation strategy and the 3D regulariza- tion loss, we implement three variants of the proposed method and evaluate them DeepFashion. The experimental results of this ablation study is reported Table 3
Effect of Global Correspondence. Compared with the vanilla variant of the proposed 3D-GCL network (3D-GCL *), adopting global correspondences (3D-GCL t) provides us with the performance gains persistent on two metrics. This is expected, global correspondences are effective in capturing long-range deformations.
Effect of 3D Regularization Loss. Compared with global correspondences, the advantages of the 3D regularization loss (3D-GCL x) are even more significant, especially according the mloU metric. As discussed in Sec. 3.3 correspondence learning only matches pose features semantically, which means that, on their own, they are unable to preserve fine-grained structures and textures. This result suggests that the 3D regularization loss leveraging SMPL flows to provide geometrie guidance, is necessary and effective component of our network.
Finally, our full model obtains the best performance compared with other variants, which further supports that flows encoding both long-range and local deformations critical in producing compelling try-on results in challenging scenarios. The visual comparison of different variants of our method is shown in Fig. 6
Conclusion
In this paper, we propose 3D-aware global correspondence learning (3D-GCL) network tackle hard-pose person-to-person visual try-on. Our 3D-GCL network adopts efficient architecture for global correspondence estimation and refinement, and introduces a novel 3D regularization loss to exploit geometric priors of 3D human bodies to guide the refinement process of the global

---

correspondences. In this way, our full-resolution 3D-aware global correspondences can represent both long-range and local deformations precisely, and hence our method can handle large pose and viewpoint variations, while preserving garment textures and structures effectively. Quantitative and qualitative comparison validate the effectiveness of the proposed 3D-GCL. network. Due to its effectiveness and flexibility, the proposed 3D-GCL network constitutes new baseline for hard-pose try-on, and more advanced components, like attention models, can be integrated into the network in the future for further improvements.
6 Acknowledgement
This work was supported in part by National Key R&D Program of China under Grant No. 2020AAA0109700, National Natural Science Foundation of China (NSFC) under Grant No.U19A2073, No.U1811400 No.61902088 and No.61976233, Shenzhen Fundamental Research Program (Project No. RCYX202007 141 14642083, No. JCYJ20190807 1177715411113 1365) and CAAI- Huawei MindSpore Open Fund. We thank MindSpore for the partial support of this work, which is a new deep learning computing framwork
References
[1] Fred L. Bookstein. Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Transactions on pattern analysis and machine intelligence, 11(6):567-585, 1989.
[2] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. View synthesis by appearance flow. In Proceedings of the European Conference Computer Vision, 2016.
[3] Ayush Chopra, Rishabh Jain, Mayur Hemani, and Balaji Krishnamurthy. Zflow: Gated appear- ance flow-based virtual try-on with 3d priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5433-5442, 2021.
[4] Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew Scott. Clothflow: A flow-based model for clothed person generation. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 10471-10488 2019.
[5] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
[6] Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d 3d-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 117-126, 2016.
[7] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar- based image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5143-5153, 2020.
[8] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16. 2015.
[9] Yining Li, Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In Proceedingy of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3693-3702, 2019.
[10] Ayush Chopra, Rishabh Jain, Mayur Hemani, and Balaji Krishnamurthy. Zflow: Gated appear- ance flow-based virtual try-on with 3d priors. arXiv: Computer Vision and Pattern Recognition, 2021.
[11] Sen He, Yi-Zhe Song, and Tao Xiang. Style-based global appearance flow for virtual try-on. In CVPR, 2022.
"naps//www.mindsoor..tl
10

---

[12] Xintong Han, Zuxuan Wu, Zhe Wu. Ruichi Yu, and Larry S. Davis. Viton: An mmage-based virtual try-on network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7543-7552, 2018
[13] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward setracttristic-ppeeeviii image-based based virtual try-on network. In Proceedings of the European Conference on Computer Vision, pages 389-604, 2018.
[14] Han Yang. Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photo-realistic virtual try-on by adaptively generating- preseervin image content. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7850-7859, 2020.
[15] Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free virtual try-on via distilling appearance flows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8485-8493, 2021.
[16] Ruiyun Yu, Xiaoqi Wang, and Xiaohui Xic. Vinfp: An image-based virtual try-on network with body and clothing feature preservation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10511-10520, 2019.
[17] Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7297-7306, 2018.
[18] Benjamin Biggs, David Novotny, Sebastien Ehrhardt, Hanbyul Joo, Ben Graham, and Andrea Vedaldi. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. Advances in Neural Information Processing Systems, 33:20496-20507, 2020.
[19] Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: unified framework for human motion imitation, appearance transfer and novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5904-5913, 2019.
[20] Badour AlBahar, Jingwan Lu, Jimei Yang. Zhixin Shu, Eli Shechtman, and Jia-Bin Huang. Pose with style: Detail-presservng pose-guided image synthesis with conditional stylegan. ACM Transactions on Graphics, 40(6):1-11, 2021.
[21] Xin Dong, Fuwei Zhao, Zhenyu Xie, Xijin Zhang, Daniel K. Du, Min Zheng, Xiang Long, Xiaodan Liang, and Jianchao Yang. Dressing the wild by watching dance videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[22] Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H. Li, and Ge Li. Deep image spatial transformation for person image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages (690-7699, 2020.
[23] Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. 3d pose transfer with correspondence learning and mesh refinement. Advances in Neural Information Processing Systems, 34, 2021.
[24] Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11465-11475, 2021.
[25] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8110-8119, 2020.
[26] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEFF/VVF Conference on Computer Vision and Pattern Recognition, pages 2337-2346, 2019.

---

[27] Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, and Liang Lin. Graphon- omy: Universal human parsing via graph transfer learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7450-7459, 2019.
[28] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, pages 2758-2766, 2015.
[29] Zhe Cao, Gines Hidalgo. Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(01): 2021.
[30] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749-1759, 2021.
[31] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1096-1104, 2016.
[32] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, and Jian Yin. Towards multi-pose guided virtual try-on network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9026-9035, 2019.
[33] Yifang Men, Yiming Mao, Yuning Jiang, Wei- Ying Ma. and Zhouhui Lian. Controllable person image synthesis with strribute-dcomppon gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5084-5093, 2020.
[34] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 386-595,10 2018.
[35] Zhenyu Xie, Zaiyu Huang. Fuwei Zhao, Haoye Dong, Michael Kampffineee, and Xiao- dan Liang. Towards scalable unpaired virtual try-on via patch-routed spatially-adaptin gan. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages Associates, Inc., 2021.
Curran
[36] Aiyu Cui, Daniel McKee, and Svetlana Lazebnik. Dressing in order: Recurrent person image generation for pose transfer, virtual ury-on and outfit editing. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We have discussed the limitations in the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have
discussed the potential negative societal impacts in the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
12

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes] Our code and data are provided in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All training details are specified the supplementary material. (c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A] We fix the random seed so that our model runs stably and our results are reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Computational resources used are included in the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/relcasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] All assets used in our work are publicly released under an MIT License or a license restricted non-commercial
research and educational purposes.
(c) Did you include any assets either in the supplemental material or URL? [Yes] We have constructed the HardPose subset from DeepFashion and MVP, and we include the list of this subset in the supplemental material.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [Yes] The data and models used in this work are publicly released. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Faces in our visual examples are masked out, therefore our data do not contains identifiable information offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] The instructions given to the participants are provided in the supplementary material.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [N/A] The participants consist of volunteers.
13


Reviews:
[{"review_id": "nh6sd9Bn5qH", "reviewer": "Reviewer_AD5s", "summary": "The paper present a 3D-aware Global Correspondence Learning to tackle the virtual try-on problem. The method injects 3D priors onto the feature learning step via SMPL-based approach. The method shows benefits on using such an approach on two widely used dataset by the community", "questions": "- If someone has a one-to-one correspondance from the DensePose, why would there be a need for the Stage I network . This is not to trivialize Stage I model, but rather to make sure that the simple waping filed obtained from DensePose are not enough and hence justifies its role. A baseline with only the one-to-one correspondance from the DensePose is much appreciated I think\n- How the method deals with wrong semantic segmentation obtained from the human parser?", "limitations": "- As mentioned above, SMPL (DensePose) models are known to fail on hard posture especially on dance ones. How would the method deal on such cases? Relying solely on SMPL might give very wrong prediction, it would be good if the authors would develop on this", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written and the ideas are easy to follow\n- Conducted experiments on two datasets, DeepFashion and MPV\n- Ablation study on the proposed 3D-GCL framework\n- Good qualitative results on the try-on clothes\n\n**Weaknesses**\n- The model depends heavily on the SMPL model estimation. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YtBLrnUN1rV", "reviewer": "Reviewer_7wqS", "summary": "This work presents a 3D-aware global correspondence learning (3D-GCL) framework to tackle the image-based person-to-person virtual try-on problem. The core idea is to incorporate the geometric prior of 3D human body to guide the correspondence learning between the source and the target person, aiming at preserving detailed garment textures. To circumvent the difficulty of learning long-range correspondence, the authors introduce a coarse-to-fine framework. The garment warping flow is initialized via the global correlation between high-level image features and then progressively refined. In addition, the SMPL flow estimated from the source and the target person is introduced as the 3D-aware supervision signal to guide the refinement process. Empirical studies on the DeepFashion and MPV dataset demonstrate superior performance over previous methods, especially on hard pose samples.", "questions": "\n- Typos: Line 76 summarize --> summarized\n- What are the selection criteria for the HardPose test set? Is there a quantitative analysis of how hard the HardPose test set is compared to the public benchmarks?\n- Abalation study Table 3. What is the framework like for 3D-GCL w/o global correspondences?", "limitations": "Yes. The authors addressed the limitations and potential negative societal impact of their work. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- The proposed method is simple and effective. It demonstrates solid improvement over previous methods in terms of FID and human evaluation scores.\n\n## Weakness\n\n- Missing comparisons. Some recent works also focus on incorporating 3D priors [1] or global correspondence [2] into image-based virtual try-on. There is no deep discussion on the difference between this work and [1] [2]. Quantitative comparisons with [1][2] are also missing. Comparisons with the above works are needed to elucidate the main contributions of this work.\n\n- Unfair experimental setup. Line 224. \"pairs without properly detected IUV maps or SMPL models are filtered out.\" I am concerned whether this is a fair comparison with other methods, as filtering out failure detections is beneficial for testing the proposed method. This makes the empirical results less convincing.\n\n[1] ZFlow: Gated Appearance Flow-based Virtual Try-on with 3D Priors. ICCV 2021\n\n[2] Style-Based Global Appearance Flow for Virtual Try-On. CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QXhiYjxyfm_", "reviewer": "Reviewer_nqeh", "summary": "The paper proposes to reconstruct 3D human mesh from multiview images by fusing features with a transformer and assisting the training by pose alignment between different views.", "questions": "See weakness ", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The writing of the paper is clear. \n- The motivation of the paper and the effectiveness of the proposed 3D-aware global correspondence is well supported by extensive experiments. \n\nWeaknesses\n\nMore ablation studies could be added to analyze the effectiveness of the components of the method.\n- How does the multiscale/coarse-to-fine structure influence the final result?\n- Can the off-the-shelf 3D mesh regressor be applied to (a)(b)(c) in Figure 3?  Will the SMPL flow supervision benefit these strategies as well? How does the module structure affect the warping compared with other module structures?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "81jZfgvdPYV", "reviewer": "Reviewer_bUCs", "summary": "This work proposes a virtual try-on technique to transfer garment from a given source image to target image/pose. A main difference to existing works is the use of dense correspondences between source and target poses and also using SMPL based regularization loss functions. Experiments on DeepFashion and MVP datasets with the proposed test splits demonstrate better results than existing techniques.", "questions": "- The results reported in Tables 1 and 2 do not match to those reported in earlier works. I have checked some of the compared papers and their reported numbers seem different. Does this work use different test split compared to existing works?", "limitations": "There is no clear discussion of limitations of the proposed technique. For instance, the proposed technique fails when the dense pose estimation on either source or target images fail.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- SMPL based regularization losses for dense correspondence estimation.\n- Better results on two different datasets compared to existing works on the proposed test splits.\n\nWeaknesses:\n- The main technical novelty of this work compared to prior art is the use of SMPL estimates for correspondence loss functions. As such the technical novelty is somewhat incremental. \n- Line 224 says that \"image pairs without properly detected IUV maps or SMPL models are filtered out\". This seems unfair filtering of test set as the proposed method relies on IUV map and SMPL model estimation. As a result, the reported numbers could be biased towards the proposed technique. Also it is not clear what it means by 'properly' here.\n- The details on how the 'HardPose' dataset is constructed is not clear. What constitutes a hard pose? Having some visual examples that distinguishes normal and HardPose test sets would be good. And, I do not find any extreme poses in any of the result images. So, it is not clear to me whether any of the images can be considered a hard pose.\n\n\nPost rebuttal:\nAuthors addressed some of the concerns, especially w.r.t. experimental setup, in their response.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept