Learning Surface Parameterization for Document Image Unwarping

1 INTRODUCTION . Reconstructing 3D shapes from images is a core problem in computer vision and graphics research . With the progress in differentiable rendering ( Sitzmann et al. , 2019b ; Kato et al. , 2018 ; Niemeyer et al. , 2020 ; Li et al. , 2018 ; Liu et al. , 2019b ) , recent learning-based 3D reconstruction approaches have achieved impressive results using 2D supervision from single image ( Chen & Zhang , 2019 ; Groueix et al. , 2018 ; Choy et al. , 2016 ; Mescheder et al. , 2019 ; Wang et al. , 2018 ) or multi-view images ( Tang & Tan , 2018 ; Yariv et al. , 2020 ) . These methods achieve high quality 3D reconstruction using differentiable rendering with various 3D representations such as 3D mesh ( Wang et al. , 2018 ) , volumetric representation ( Mildenhall et al. , 2020 ) , or implicit functions ( Mescheder et al. , 2019 ) . In recent neural rendering methods such as NeRF ( Mildenhall et al. , 2020 ) and IDR ( Yariv et al. , 2020 ) , continuous representations such as volume or implicit functions achieve significantly better reconstruction results than meshes or voxels because they do not discretize the 3D surface a priori . However , these continuous representations usually do not encode explicit surface parameterization , allowing 3D shape re-texturing , editing the existing texture in the 2D texture space , or recovering 2D texture from 3D surfaces . One of the most direct applications of 2D texture recovery in a geometrically constrained manner , is document unwarping , i.e. , inference of a document ’ s flatbed-scanned version from a casual photo of a potentially creased document . Whereas 2D texture recovery could be equally valuable for other domains such as garments , or faces , the existing datasets are not directly applicable to our method . Our novel texture mapping approach learns surface parameterization for document unwarping by learning continuous bijective functions between 3D surface positions and 2D texture-space coordinates . We use a signed distance function ( SDF ) ( Chan & Zhu , 2005 ) to represent geometry and model the appearance as a function of the 2D texture coordinates . By utilizing implicit differentiable rendering ( IDR ) , ( Yariv et al. , 2020 ) we can reconstruct 3D shape and learn the corresponding UV parameterization of the surface simultaneously using a per-pixel rendering loss and appropriate geometric regularizations . We utilize two fully connected multi-layer perceptrons ( MLPs ) to learn a bijective mapping between 3D shapes and 2D texture space . More specifically , the forward MLP maps the 3D surface coordinates to 2D texture coordinates and the backward MLP maps the 2D texture coordinates to corresponding 3D surface coordinates . Following IDR ( Yariv et al. , 2020 ) , we obtain the 3D surface coordinates by sphere-tracing along the ray , cast through each pixel . Our appearance rendering is formulated as a function of the 3D and the texture coordinates . Therefore , the forward and backward MLPs can be trained with a 2D pixel-wise loss between the rendered image and the given ground truth image . To the best of our knowledge , this is the first neural rendering method that can learn effective UV parameterization for implicit surfaces . As a corollary , our method is also the first method which utilizes implicit surface based neural rendering for document unwarping . It is a challenging task due to the presence of geometric and photometric distortions in the document . For this particular problem we introduce a prior for shapespecific texture mapping to initialize the forward MLP ( 3D to 2D mapping ) . This prior is learned from a large dataset of UV mapped document meshes , assuming that document texture space maps to a 2D equiangular quadrilateral . This assumption regularizes the forward MLP to output a highquality texture space that avoids degenerate solutions ( see Fig . 3 ) . Moreover , we introduce a conformality constraint in the backward MLP , which is consistent with how paper folds happen in the physical world , i.e. , without any stretch or tear . This constraint also ensures that the backward function is bijective and smooth ( Petrini et al. , 2018 ) . The main contributions of our paper are the following : First , we propose an efficient way to learn texture parameterization for implicit neural representations using a differentiable rendering framework . Without 3D supervision , it only requires multi-view images as ground-truth and a texture mapping prior . Second , we show that our method can be effectively used for document unwarping tasks by learning a prior for explicit texture mapping on the document shape . We show that this prior can be learned from a dataset of texture-mapped meshes . Third , we show that our method is effective for document image unwarping and texture editing ( see Fig . 1 ) . We achieve a 52 % relative improvement over the publicly available state-of-the-art1 ( Das et al. , 2019 ) in terms of mean local distortion across 500 views from ten synthetic scenes . Additionally , we achieve a ∼25 % improvement in optical character recognition ( OCR ) in terms of character and word error rate . 2 PREVIOUS WORK . Neural Rendering . Neural rendering generates images and videos by integrating conventional computer graphics rendering pipelines into deep neural networks ( Tewari et al. , 2020 ) . It enables explicit or implicit control of scene properties , including illumination , geometry , texture , etc . Neural rendering can synthesize semantic photos ( Park et al. , 2019b ; Bau et al. , 2019 ) , novel views ( Hedman et al. , 2018 ; Sitzmann et al. , 2019a ) , relighting ( Xu et al. , 2018 ; Meka et al. , 2019 ) , facial/body reenactment ( Chan et al. , 2019 ; Wei et al. , 2019 ) , estimate scene properties etc . Kato ( Kato et al. , 2018 ) proposed a differentiable neural renderer using an approximate gradient for rasterization . Liu ( Liu et al. , 2019a ) proposed SoftRas , which extended differentiable rasterization . Li ( Li et al. , 2018 ) further demonstrated the feasibility of integrating ray-tracing in deep neural networks . More recently , implicit surface or volume rendering has become mainstream in neural rendering approaches such as IDR ( Yariv et al. , 2020 ) and NeRF ( Mildenhall et al. , 2020 ) . These approaches are based 1A more recent approach , CREASE ( Markovitz et al. , 2020 ) , data and models are not publicly available . on multi-view surface reconstruction to associate the scene geometry to the appearance in different views . NeRF is extended to lot of variants including PixelNeRF ( Yu et al. , 2020 ) , MVSNeRF ( Chen et al. , 2021 ) , dynamic NeRF ( Li et al. , 2020 ; Pumarola et al. , 2020 ) , GRAF ( Schwarz et al. , 2020 ) and so on . Texture Mapping . Texture mapping is an essential step in the computer graphics rendering pipeline . It defines a correspondence between a vertex on the 3D mesh and a pixel in the 2D texture image . To find such a mapping , FlexiStickers ( Tzur & Tal , 2009 ) required users to specify a sparse set of correspondences . Bi ( Bi et al. , 2017 ) proposed a patch-based texture mapping method using the 3D shape and images from multiple views . Morreale ( Morreale et al. , 2021 ) used networks to represent 3D surfaces/shapes . Besides the above general texture mapping methods , some approaches focus on a specific object categories such as faces ( Deng et al. , 2018 ; Chen et al. , 2019 ) and human bodies ( Mir et al. , 2020 ; Zhao et al. , 2020 ) . Recently , AtlasNet ( Groueix et al. , 2018 ) represented a 3D mesh as a collection of parametric surfaces ; thus , texture mapping is trivial to obtain from a 2D parametric surface . A similar idea was adopted by Bednarik ( Bednarik et al. , 2020 ) where they introduced geometric constraints when learning the decomposition . More recently NeuTex ( Xiang et al. , 2021 ) aims to recover the texture of a subject using NeRF ( Mildenhall et al. , 2020 ) . However , NeuTex uses a spherical UV domain without any geometric constraints . Therefore , it is not suitable for document unwarping . Moreover , since NeRF ( Mildenhall et al. , 2020 ) doesn ’ t learn an explicit geometry , NeuTex requires a coarse point-cloud to initialize the backward MLP . With an SDF based ( Yariv et al. , 2020 ) approach , our approach does not require such an initialization routine . We jointly learn the texture mapping and the geometry from scratch . Document Unwarping . Document unwarping is a special application of texture mapping : the 3D object is usually a rectangular piecewise-developable surface , and the texture is well-structured , containing straight text lines , ( usually ) rectangular text blocks and figures , etc . Previous work usually adopted a two-step methodology : 1 ) 3D surface estimation and 2 ) deformed surface flattening . The 3D surface of a deformed document can be estimated from shading ( Wada et al. , 1997 ) , multi-view images ( Ulges et al. , 2004 ) , text lines ( Tian & Narasimhan , 2011 ) , local character orientations ( Meng et al. , 2018 ) , document boundaries ( Koo et al. , 2009 ) , and learning-based strategies ( Pumarola et al. , 2018 ) . Flattening the obtained 3D surface always involves an expensive optimization process under certain geometry constraints such as conformality ( You et al. , 2017 ) or isometries ( Bartoli et al. , 2015 ) . Flattening could be easier if the obtained 3D shape had a low dimensional parameterization like Generalized Cylindrical Surface ( GCS ) ( Kil et al. , 2017 ) . Some studies ( Das et al. , 2017 ; Liang et al. , 2008 ; Meng et al. , 2015 ) proposed to unwarp each patch on the surface individually and then stitch the unwarped patches together . In recent years , data-driven methods ( Ma et al. , 2018 ; Das et al. , 2019 ; Li et al. , 2019 ; Markovitz et al. , 2020 ; Das et al. , 2021 ) have addressed document unwarping by leveraging large-scale synthetic datasets . These datasets contain deformed document images and their corresponding ground-truth UV coordinates . Methods trained on synthetic images often suffer from generalization performance due to the domain gap between synthetic and real data . In this paper , we utilize neural rendering techniques to learn a surface parameterization of a deformed document . We simultaneously estimate both 3D shapes and UV coordinates with a cycle consistency loss and geometric constraints . By leveraging the information from multi-view images , the proposed method demonstrates better document unwarping performance compared to a previous state-of-the-art , Das et al . ( 2019 ) . 3 METHOD . A schematic diagram of the proposed approach is shown in Fig . 2 . We utilize a recent differentiable rendering method , IDR ( Yariv et al. , 2020 ) for surface reconstruction and jointly learn the texture mapping of the learned implicit surface using two MLPs . In Sec . 3.1 we first describe some preliminaries about surface parameterization and IDR . 3.1 PRELIMINARIES . Surface Parameterization . The problem of surface parameterization focuses on finding a bijective mapping F between a surface Z ∈ R3 and a polygonal domain Ω ∈ Rn . For a parametric or discrete surface representation , we can explicitly compute this mapping ( Tzur & Tal , 2009 ) using constrained optimization . In contrast , implicit surfaces are represented as continuous functions and can not be readily parameterized . In this paper , we propose to learn such bijective mapping between a learned implicit surface and a 2D planar domain Ω ∈ R2 using our proposed forward and backward MLPs . Ω is the texture space or UV space , parameterized using 2D UV coordinates t = ( u , v ) . We can use any continuous parameterization function as the UV space . Since this work particularly focuses on document unwarping , we choose the UV space to be a regular 2D grid . Implicit Differentiable Rendering . Implicit Differentiable Rendering ( Yariv et al. , 2020 ) reconstructs the geometry of an object from multi-view images as the zero level set , Zθ of an MLP S , Zθ = { z ∈ R3 | S ( z ; θ ) = 0 } ( 1 ) where θ are the learnable parameters . To render the surface Zθ , IDR uses another MLP to model the radiance ( RGB color ) as a function of the surface point ( zp ) , corresponding surface normal ( np ) , view direction ( vp ) and a global geometry feature vector ( gp ) : Cp = A ( zp , np , vp , gp ) ( 2 ) Here , Cp denote the predicted color at pixel p and A denotes the appearance MLP . The surface point is obtained by a sphere-tracing method ( Hart , 1996 ) along the ray rp ( τ ) through pixel p. τ ∈ Rk denotes camera parameters of the scene . Additionally , IDR also presents a differentiable way to obtain a ray and geometry intersection point ( ẑp ) as a function of the camera ray . Although , the IDR can disentangle geometry and appearance , it only allows to re-texture a new geometry with a learned appearance MLP , A . Editing a texture or extracting a surface texture map is not possible in a vanilla IDR framework since no explicit texture mapping is learned .

Summary: **High-Level Overview**  This paper proposes a neural rendering technique to learn how to estimate an implicit surface and its UV-parameterization from a set of 2D images. The proposed approach is applied to the problem of document unwarping, wherein a wrinkled, folded, or bent document (e.g. a piece of paper with text on it) can be digitally flattened with its contents preserved. For good measure, the paper also shows that learning the texture mapping enables the contents of the unwarped document to be edited in the texture space. The proposed approach is evaluated against a prior work, DewarpNet (Das et al., 2019) on the basis of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) as well as the auxiliary task of OCR.  **Key Contributions**  * Builds on IDR (Yariv et al., 2020) to learn an explicit bijective texture mapping for a learned implicit surface * A loss function formulation for the framework * Application of learned implicit surfaces for document unwarping with reduction in texture distortion and improvements in subsequent OCR metrics  **How It Works**  * Starts with the same problem formulation as IDR: an MLP to model the surface (Eq. 1) and an appearance MLP to model the color (Eq. 2) * Modifies the appearance MLP to also be a function of texture coordinates (Eq. 3) * Adds an additional forward and backward MLP to learn the bijective UV mapping * Jointly trains all MLPs using a pixelwise rendering loss