Learning Surface Parameterization for Document Image Unwarping

1 INTRODUCTION . Reconstructing 3D shapes from images is a core problem in computer vision and graphics research . With the progress in differentiable rendering ( Sitzmann et al. , 2019b ; Kato et al. , 2018 ; Niemeyer et al. , 2020 ; Li et al. , 2018 ; Liu et al. , 2019b ) , recent learning-based 3D reconstruction approaches have achieved impressive results using 2D supervision from single image ( Chen & Zhang , 2019 ; Groueix et al. , 2018 ; Choy et al. , 2016 ; Mescheder et al. , 2019 ; Wang et al. , 2018 ) or multi-view images ( Tang & Tan , 2018 ; Yariv et al. , 2020 ) . These methods achieve high quality 3D reconstruction using differentiable rendering with various 3D representations such as 3D mesh ( Wang et al. , 2018 ) , volumetric representation ( Mildenhall et al. , 2020 ) , or implicit functions ( Mescheder et al. , 2019 ) . In recent neural rendering methods such as NeRF ( Mildenhall et al. , 2020 ) and IDR ( Yariv et al. , 2020 ) , continuous representations such as volume or implicit functions achieve significantly better reconstruction results than meshes or voxels because they do not discretize the 3D surface a priori . However , these continuous representations usually do not encode explicit surface parameterization , allowing 3D shape re-texturing , editing the existing texture in the 2D texture space , or recovering 2D texture from 3D surfaces . One of the most direct applications of 2D texture recovery in a geometrically constrained manner , is document unwarping , i.e. , inference of a document ’ s flatbed-scanned version from a casual photo of a potentially creased document . Whereas 2D texture recovery could be equally valuable for other domains such as garments , or faces , the existing datasets are not directly applicable to our method . Our novel texture mapping approach learns surface parameterization for document unwarping by learning continuous bijective functions between 3D surface positions and 2D texture-space coordinates . We use a signed distance function ( SDF ) ( Chan & Zhu , 2005 ) to represent geometry and model the appearance as a function of the 2D texture coordinates . By utilizing implicit differentiable rendering ( IDR ) , ( Yariv et al. , 2020 ) we can reconstruct 3D shape and learn the corresponding UV parameterization of the surface simultaneously using a per-pixel rendering loss and appropriate geometric regularizations . We utilize two fully connected multi-layer perceptrons ( MLPs ) to learn a bijective mapping between 3D shapes and 2D texture space . More specifically , the forward MLP maps the 3D surface coordinates to 2D texture coordinates and the backward MLP maps the 2D texture coordinates to corresponding 3D surface coordinates . Following IDR ( Yariv et al. , 2020 ) , we obtain the 3D surface coordinates by sphere-tracing along the ray , cast through each pixel . Our appearance rendering is formulated as a function of the 3D and the texture coordinates . Therefore , the forward and backward MLPs can be trained with a 2D pixel-wise loss between the rendered image and the given ground truth image . To the best of our knowledge , this is the first neural rendering method that can learn effective UV parameterization for implicit surfaces . As a corollary , our method is also the first method which utilizes implicit surface based neural rendering for document unwarping . It is a challenging task due to the presence of geometric and photometric distortions in the document . For this particular problem we introduce a prior for shapespecific texture mapping to initialize the forward MLP ( 3D to 2D mapping ) . This prior is learned from a large dataset of UV mapped document meshes , assuming that document texture space maps to a 2D equiangular quadrilateral . This assumption regularizes the forward MLP to output a highquality texture space that avoids degenerate solutions ( see Fig . 3 ) . Moreover , we introduce a conformality constraint in the backward MLP , which is consistent with how paper folds happen in the physical world , i.e. , without any stretch or tear . This constraint also ensures that the backward function is bijective and smooth ( Petrini et al. , 2018 ) . The main contributions of our paper are the following : First , we propose an efficient way to learn texture parameterization for implicit neural representations using a differentiable rendering framework . Without 3D supervision , it only requires multi-view images as ground-truth and a texture mapping prior . Second , we show that our method can be effectively used for document unwarping tasks by learning a prior for explicit texture mapping on the document shape . We show that this prior can be learned from a dataset of texture-mapped meshes . Third , we show that our method is effective for document image unwarping and texture editing ( see Fig . 1 ) . We achieve a 52 % relative improvement over the publicly available state-of-the-art1 ( Das et al. , 2019 ) in terms of mean local distortion across 500 views from ten synthetic scenes . Additionally , we achieve a ∼25 % improvement in optical character recognition ( OCR ) in terms of character and word error rate . 2 PREVIOUS WORK . Neural Rendering . Neural rendering generates images and videos by integrating conventional computer graphics rendering pipelines into deep neural networks ( Tewari et al. , 2020 ) . It enables explicit or implicit control of scene properties , including illumination , geometry , texture , etc . Neural rendering can synthesize semantic photos ( Park et al. , 2019b ; Bau et al. , 2019 ) , novel views ( Hedman et al. , 2018 ; Sitzmann et al. , 2019a ) , relighting ( Xu et al. , 2018 ; Meka et al. , 2019 ) , facial/body reenactment ( Chan et al. , 2019 ; Wei et al. , 2019 ) , estimate scene properties etc . Kato ( Kato et al. , 2018 ) proposed a differentiable neural renderer using an approximate gradient for rasterization . Liu ( Liu et al. , 2019a ) proposed SoftRas , which extended differentiable rasterization . Li ( Li et al. , 2018 ) further demonstrated the feasibility of integrating ray-tracing in deep neural networks . More recently , implicit surface or volume rendering has become mainstream in neural rendering approaches such as IDR ( Yariv et al. , 2020 ) and NeRF ( Mildenhall et al. , 2020 ) . These approaches are based 1A more recent approach , CREASE ( Markovitz et al. , 2020 ) , data and models are not publicly available . on multi-view surface reconstruction to associate the scene geometry to the appearance in different views . NeRF is extended to lot of variants including PixelNeRF ( Yu et al. , 2020 ) , MVSNeRF ( Chen et al. , 2021 ) , dynamic NeRF ( Li et al. , 2020 ; Pumarola et al. , 2020 ) , GRAF ( Schwarz et al. , 2020 ) and so on . Texture Mapping . Texture mapping is an essential step in the computer graphics rendering pipeline . It defines a correspondence between a vertex on the 3D mesh and a pixel in the 2D texture image . To find such a mapping , FlexiStickers ( Tzur & Tal , 2009 ) required users to specify a sparse set of correspondences . Bi ( Bi et al. , 2017 ) proposed a patch-based texture mapping method using the 3D shape and images from multiple views . Morreale ( Morreale et al. , 2021 ) used networks to represent 3D surfaces/shapes . Besides the above general texture mapping methods , some approaches focus on a specific object categories such as faces ( Deng et al. , 2018 ; Chen et al. , 2019 ) and human bodies ( Mir et al. , 2020 ; Zhao et al. , 2020 ) . Recently , AtlasNet ( Groueix et al. , 2018 ) represented a 3D mesh as a collection of parametric surfaces ; thus , texture mapping is trivial to obtain from a 2D parametric surface . A similar idea was adopted by Bednarik ( Bednarik et al. , 2020 ) where they introduced geometric constraints when learning the decomposition . More recently NeuTex ( Xiang et al. , 2021 ) aims to recover the texture of a subject using NeRF ( Mildenhall et al. , 2020 ) . However , NeuTex uses a spherical UV domain without any geometric constraints . Therefore , it is not suitable for document unwarping . Moreover , since NeRF ( Mildenhall et al. , 2020 ) doesn ’ t learn an explicit geometry , NeuTex requires a coarse point-cloud to initialize the backward MLP . With an SDF based ( Yariv et al. , 2020 ) approach , our approach does not require such an initialization routine . We jointly learn the texture mapping and the geometry from scratch . Document Unwarping . Document unwarping is a special application of texture mapping : the 3D object is usually a rectangular piecewise-developable surface , and the texture is well-structured , containing straight text lines , ( usually ) rectangular text blocks and figures , etc . Previous work usually adopted a two-step methodology : 1 ) 3D surface estimation and 2 ) deformed surface flattening . The 3D surface of a deformed document can be estimated from shading ( Wada et al. , 1997 ) , multi-view images ( Ulges et al. , 2004 ) , text lines ( Tian & Narasimhan , 2011 ) , local character orientations ( Meng et al. , 2018 ) , document boundaries ( Koo et al. , 2009 ) , and learning-based strategies ( Pumarola et al. , 2018 ) . Flattening the obtained 3D surface always involves an expensive optimization process under certain geometry constraints such as conformality ( You et al. , 2017 ) or isometries ( Bartoli et al. , 2015 ) . Flattening could be easier if the obtained 3D shape had a low dimensional parameterization like Generalized Cylindrical Surface ( GCS ) ( Kil et al. , 2017 ) . Some studies ( Das et al. , 2017 ; Liang et al. , 2008 ; Meng et al. , 2015 ) proposed to unwarp each patch on the surface individually and then stitch the unwarped patches together . In recent years , data-driven methods ( Ma et al. , 2018 ; Das et al. , 2019 ; Li et al. , 2019 ; Markovitz et al. , 2020 ; Das et al. , 2021 ) have addressed document unwarping by leveraging large-scale synthetic datasets . These datasets contain deformed document images and their corresponding ground-truth UV coordinates . Methods trained on synthetic images often suffer from generalization performance due to the domain gap between synthetic and real data . In this paper , we utilize neural rendering techniques to learn a surface parameterization of a deformed document . We simultaneously estimate both 3D shapes and UV coordinates with a cycle consistency loss and geometric constraints . By leveraging the information from multi-view images , the proposed method demonstrates better document unwarping performance compared to a previous state-of-the-art , Das et al . ( 2019 ) . 3 METHOD . A schematic diagram of the proposed approach is shown in Fig . 2 . We utilize a recent differentiable rendering method , IDR ( Yariv et al. , 2020 ) for surface reconstruction and jointly learn the texture mapping of the learned implicit surface using two MLPs . In Sec . 3.1 we first describe some preliminaries about surface parameterization and IDR . 3.1 PRELIMINARIES . Surface Parameterization . The problem of surface parameterization focuses on finding a bijective mapping F between a surface Z ∈ R3 and a polygonal domain Ω ∈ Rn . For a parametric or discrete surface representation , we can explicitly compute this mapping ( Tzur & Tal , 2009 ) using constrained optimization . In contrast , implicit surfaces are represented as continuous functions and can not be readily parameterized . In this paper , we propose to learn such bijective mapping between a learned implicit surface and a 2D planar domain Ω ∈ R2 using our proposed forward and backward MLPs . Ω is the texture space or UV space , parameterized using 2D UV coordinates t = ( u , v ) . We can use any continuous parameterization function as the UV space . Since this work particularly focuses on document unwarping , we choose the UV space to be a regular 2D grid . Implicit Differentiable Rendering . Implicit Differentiable Rendering ( Yariv et al. , 2020 ) reconstructs the geometry of an object from multi-view images as the zero level set , Zθ of an MLP S , Zθ = { z ∈ R3 | S ( z ; θ ) = 0 } ( 1 ) where θ are the learnable parameters . To render the surface Zθ , IDR uses another MLP to model the radiance ( RGB color ) as a function of the surface point ( zp ) , corresponding surface normal ( np ) , view direction ( vp ) and a global geometry feature vector ( gp ) : Cp = A ( zp , np , vp , gp ) ( 2 ) Here , Cp denote the predicted color at pixel p and A denotes the appearance MLP . The surface point is obtained by a sphere-tracing method ( Hart , 1996 ) along the ray rp ( τ ) through pixel p. τ ∈ Rk denotes camera parameters of the scene . Additionally , IDR also presents a differentiable way to obtain a ray and geometry intersection point ( ẑp ) as a function of the camera ray . Although , the IDR can disentangle geometry and appearance , it only allows to re-texture a new geometry with a learned appearance MLP , A . Editing a texture or extracting a surface texture map is not possible in a vanilla IDR framework since no explicit texture mapping is learned .

Summary: The paper presents a multi-view method for undistorting documents. The method makes use of an existing method (IDR) for reconstructing distorted documents in 3-D, with an additional bijective neural network to parameterize distorted surfaces by rectangles. The method significantly outperforms SOTAs in the experiments.