Recovering Geometric Information with Learned Texture Perturbations

1 INTRODUCTION Since neural networks are trained to generalize to unseen data , regularization is important for reducing overfitting , see e.g . Goodfellow et al . ( 2016 ) ; Scholkopf & Smola ( 2001 ) . However , regularization also removes some of the high variance characteristic of much of the physical world . Even though high-quality ground truth data can be collected or generated to reflect the desired complexity of the outputs , regularization will inevitably smooth network predictions . Rather than attempting to directly infer highfrequency features , we alternatively propose to learn a low-frequency space in which such features can be embedded . We focus on the specific task of adding highfrequency wrinkles to virtual clothing , noting that the idea of learning a low-frequency embedding may be generalized to other tasks . Because cloth wrinkles/folds are high-frequency features , existing deep neural networks ( DNNs ) trained to infer cloth shape tend to predict overly smooth meshes Alldieck et al . ( 2019a ) ; Daněřek et al . ( 2017 ) ; Guan et al . ( 2012 ) ; Gundogdu et al . ( 2019 ) ; Jin et al . ( 2020 ) ; Lahner et al . ( 2018 ) ; Natsume et al . ( 2019 ) ; Santesteban et al . ( 2019 ) ; Wang et al . ( 2018 ) ; Patel et al . ( 2020 ) . Rather than attempting to amend such errors directly , we perturb texture so that the rendered cloth mesh appears to more closely match the ground truth . See Figure 1 . Then given texture perturbations from at least two unique camera views , 3D geometry can be accurately reconstructed Hartley & Sturm ( 1997 ) to recover high-frequency wrinkles . Similarly , for AR/VR applications , correcting visual appearance from two views ( one for each eye ) is enough to allow the viewer to accurately discern 3D geometry . Our proposed texture coordinate perturbations are highly dependent on the camera view . Thus , we demonstrate that one can train a separate texture sliding neural network ( TSNN ) for each of a finite number of cameras laid out into an array and use nearby networks to interpolate results valid for any view enveloped by the array . Although an approach similar in spirit might be pursued for various lighting conditions , this limitation is left as future work since there are a great deal of applications where the light is ambient/diffuse/non-directional/etc . In such situations , this further complication may be ignored without significant repercussion . 2 RELATED WORK . Cloth : While physically-based cloth simulation has matured as a field over the last few decades Baraff & Witkin ( 1998 ) ; Baraff et al . ( 2003 ) ; Bridson et al . ( 2002 ; 2003 ) ; Selle et al . ( 2008 ) , datadriven methods are attractive for many applications . There is a rich body of work in reconstructing cloth from multiple views or 3D scans , see e.g . Bradley et al . ( 2008b ) ; Franco et al . ( 2006 ) ; Vlasic et al . ( 2008 ) . More recently , optimization-based methods have been used to generate higher resolution reconstructions Huang et al . ( 2015 ) ; Pons-Moll et al . ( 2017 ) ; Wu et al . ( 2012 ) ; Yang et al . ( 2016 ) . Some of the most interesting work focuses on reconstructing the body and cloth separately Bălan & Black ( 2008 ) ; Neophytou & Hilton ( 2014 ) ; Yang et al . ( 2018 ) ; Zhang et al . ( 2017 ) . With advances in deep learning , one can aim to reconstruct 3D cloth meshes from single views . A number of approaches reconstruct a joint cloth/body mesh from a single RGB image Alldieck et al . ( 2019a ; b ) ; Natsume et al . ( 2019 ) ; Onizuka et al . ( 2020 ) ; Saito et al . ( 2019 ; 2020 ) , RGB-D image Yu et al . ( 2019 ) , or video Alldieck et al . ( 2018a ; b ) ; Habermann et al . ( 2019 ) ; Xu et al . ( 2018 ) . To reduce the dimensionality of the output space , DNNs are often trained to predict the pose/shape parameters of human body models such as SCAPE Anguelov et al . ( 2005 ) or SMPL Loper et al . ( 2015 ) ( see also Pavlakos et al . ( 2019 ) ) . Habermann et al . ( 2019 ) ; Natsume et al . ( 2019 ) ; Varol et al . ( 2018 ) leverage predicted pose information to infer shape . When only the garment shape is predicted , a number of recent works output predictions in UV space to represent geometric information as pixels Daněřek et al . ( 2017 ) ; Jin et al . ( 2020 ) ; Lahner et al . ( 2018 ) , although others Gundogdu et al . ( 2019 ) ; Santesteban et al . ( 2019 ) ; Patel et al . ( 2020 ) define loss functions directly in terms of the 3D cloth vertices . Wrinkles and Folds : Cloth realism can be improved by introducing wrinkles and folds . In the graphics community , researchers have explored both procedural and data-driven methods for generating wrinkles De Aguiar et al . ( 2010 ) ; Guan et al . ( 2012 ) ; Hahn et al . ( 2014 ) ; Müller & Chentanez ( 2010 ) ; Rohmer et al . ( 2010 ) ; Wang et al . ( 2010 ) . Other works add real-world wrinkles as a postprocessing step to improve smooth captured cloth : Popa et al . ( 2009 ) extracts the edges of cloth folds and then applies space-time deformations , Robertini et al . ( 2014 ) solves for shape deformations directly by optimizing over all frames of a video sequence . Recently , Lahner et al . ( 2018 ) used a conditional Generative Adversarial Network Mirza & Osindero ( 2014 ) to generate normal maps as proxies for wrinkles on captured cloth . Geometry : More broadly , deep learning on 3D meshes falls under the umbrella of geometric deep learning , which was coined by Bronstein et al . ( 2017 ) to characterize learning in non-Euclidean domains . Scarselli et al . ( 2008 ) was one of the earliest works in this area and introduced the notion of a Graph Neural Network ( GNN ) in relation to CNNs . Subsequent works similarly extend the CNN architecture to graphs and manifolds Boscaini et al . ( 2016 ) ; Maron et al . ( 2017 ) ; Masci et al . ( 2015 ) ; Monti et al . ( 2017 ) . Kostrikov et al . ( 2018 ) introduces a latent representation that explicitly incorporates the Dirac operator to detect principal curvature directions . Tan et al . ( 2018 ) trains a mesh generative model to generate novel meshes outside an original dataset . Returning to the specific application of virtual cloth , Jin et al . ( 2020 ) embeds a non-Euclidean cloth mesh into a Euclidean pixel space , making it possible to directly use CNNs to make non-Euclidean predictions . 3 METHODS . We define texture sliding as the changing of texture coordinates on a per-camera basis such that any point which is visible from some stereo pair of cameras can be triangulated back to its ground truth position . Other stereo reconstruction techniques can also be used in place of triangulation because the images we generate are consistent with the ground truth geometry . See e.g . Bradley et al . ( 2008a ) ; Hartley & Sturm ( 1997 ) ; Seitz et al . ( 2006 ) . 3.1 PER-VERTEX DISCRETIZATION . Since the cloth mesh is discretized into vertices and triangles , we take a per-vertex , not a per-point , approach to texture sliding . Our proposed method ( see Section 4.1 ) computes per-vertex texture coordinates on the inferred cloth that match those of the ground truth as seen by the camera under consideration . Then during 3D reconstruction , barycentric interpolation is used to find the subtriangle locations of the texture coordinates corresponding to ground truth cloth vertices . This assumes linearity , which is only valid when the triangles are small enough to capture the inherent nonlinearities in a piecewise linear sense ; moreover , folds and wrinkles can create significant nonlinearity . See Figure 2 . 3.2 OCCLUSION BOUNDARIES . Accurate 3D reconstruction requires that a vertex of the ground truth mesh be visible from at least two cameras and that camera projections of the vertex to the inferred cloth exist and are valid . However , occlusions can derail these assumptions . First , consider things from the standpoint of the inferred cloth . For a given camera view , some inferred cloth triangles will not contain any visible pixels , and we denote a vertex as occluded when none of its incident triangles contain any visible pixels . Although we do not assign perturbed texture coordinates to occluded vertices ( i.e . they keep their original texture coordinates , or a perturbation of zero ) , we do aim to keep the texture coordinate perturbation function smooth ( see Section 4.2 ) . In addition , there will be so called non-occluded vertices in the inferred cloth that do not project to visible pixels of the ground truth cloth . This often occurs near silhouette boundaries where the inferred cloth silhouette is sometimes wider than the ground truth cloth silhouette . These vertices are also treated as occluded , similar to those around the back side of the cloth behind the silhouette , essentially treating some extra vertices near occlusion boundaries as also being occluded . See Figure 3a . Next , consider things from the standpoint of the ground truth cloth . For example , consider the case where all the cameras are in the front , and vertices on the back side of the ground truth cloth are not visible from any camera . The best one can do in reconstructing these occluded vertices is to use the inferred cloth vertex positions ; however , care should be taken near occlusion boundaries to smoothly taper between our texture sliding 3D reconstruction and the inferred cloth prediction . A simple approach is to extrapolate/smooth the geometric difference between our texture sliding 3D reconstruction and the inferred cloth prediction to occluded regions of the mesh . Once again , the definition of occluded vertices needs to be broadened for silhouette consideration . Not only will vertices not visible from at least two cameras have to be considered occluded , but vertices that don ’ t project to the interior of an inferred cloth triangle with valid texture coordinate perturbations will also have to be considered occluded . See Figure 3b . 4 DATASET GENERATION . Let C = { X , T } be a cloth triangulated surface with n vertices X ∈ R3n and texture coordinates T ∈ R2n . We assume that mesh connectivity remains fixed throughout . The ground truth cloth mesh CG ( θ ) = { XG ( θ ) , TG } depends on the pose θ . Given a pre-trained DNN ( we use the network from Jin et al . ( 2020 ) ) , the inferred cloth CN ( θ ) = { XN ( θ ) , TG } is also a function of the pose θ . Our objective is to replace the ground truth texture coordinates TG with perturbed texture coordinates TN ( θ , v ) , i.e . to compute C ′N ( θ , v ) = { XN ( θ ) , TN ( θ , v ) } where TN ( θ , v ) depends on both the pose θ and the view v. Even though TN ( θ , v ) is in principle valid for all v using interpolation ( see Section 6.3 ) , training data TN ( θ , vp ) is only required for a finite number of camera views vp . For each camera p , we also only require training data for finite number of poses θk , i.e . we require TN ( θk , vp ) , which is computed from TG using XG ( θk ) , XN ( θk ) , and vp .

Summary: The paper proposes a method to correct high-frequencies details in the textures of animated clothes. The main idea is to train a network to learn the 2D offset in the UV space for a given pose and view. The paper shows results using a t-shirt, on interpolating to novel views and 3D reconstruction by correcting the output of a state of the art method.