Year: 2021

Abstract:
In this paper, we present a novel approach to learn texture mapping for a 3D surface and apply it to document image unwarping. We propose an efficient method to learn surface parameterization by learning a continuous bijective mapping between 3D surface positions and 2D texture-space coordinates. Our surface parameterization network can be conveniently plugged into a differentiable rendering pipeline and trained using multi-view images and rendering loss. Recent work on differentiable rendering techniques for implicit surfaces has shown high-quality 3D scene reconstruction and view synthesis results. However, these methods typically learn the appearance color as a function of the surface points and lack explicit surface parameterization. Thus they do not allow texture map extraction or texture editing. By introducing explicit surface parameterization and learning with a recent differentiable renderer for implicit surfaces, we demonstrate state-of-the-art document-unwarping via texture extraction. We show that our approach can reconstruct high-frequency textures for arbitrary document shapes in both synthetic and real scenarios. We also demonstrate the usefulness of our system by applying it to document texture editing.

heading: 1 INTRODUCTION
text: Reconstructing 3D shapes from images is a core problem in computer vision and graphics research. With the progress in differentiable rendering (Sitzmann et al., 2019b; Kato et al., 2018; Niemeyer et al., 2020; Li et al., 2018; Liu et al., 2019b), recent learning-based 3D reconstruction approaches have achieved impressive results using 2D supervision from single image (Chen & Zhang, 2019; Groueix et al., 2018; Choy et al., 2016; Mescheder et al., 2019; Wang et al., 2018) or multi-view images (Tang & Tan, 2018; Yariv et al., 2020). These methods achieve high quality 3D reconstruction using differentiable rendering with various 3D representations such as 3D mesh (Wang et al., 2018), volumetric representation (Mildenhall et al., 2020), or implicit functions (Mescheder et al., 2019). In recent neural rendering methods such as NeRF (Mildenhall et al., 2020) and IDR (Yariv et al., 2020), continuous representations such as volume or implicit functions achieve significantly better reconstruction results than meshes or voxels because they do not discretize the 3D surface a priori. However, these continuous representations usually do not encode explicit surface parameterization, allowing 3D shape re-texturing, editing the existing texture in the 2D texture space, or recovering 2D texture from 3D surfaces. One of the most direct applications of 2D texture recovery in a geometrically constrained manner, is document unwarping, i.e., inference of a document’s flatbed-scanned version from a casual photo of a potentially creased document. Whereas 2D texture recovery could be equally valuable for other domains such as garments, or faces, the existing datasets are not directly applicable to our method.
Our novel texture mapping approach learns surface parameterization for document unwarping by learning continuous bijective functions between 3D surface positions and 2D texture-space coordinates. We use a signed distance function (SDF) (Chan & Zhu, 2005) to represent geometry and model the appearance as a function of the 2D texture coordinates. By utilizing implicit differentiable rendering (IDR), (Yariv et al., 2020) we can reconstruct 3D shape and learn the corresponding UV parameterization of the surface simultaneously using a per-pixel rendering loss and appropriate geometric regularizations.
We utilize two fully connected multi-layer perceptrons (MLPs) to learn a bijective mapping between 3D shapes and 2D texture space. More specifically, the forward MLP maps the 3D surface
coordinates to 2D texture coordinates and the backward MLP maps the 2D texture coordinates to corresponding 3D surface coordinates. Following IDR (Yariv et al., 2020), we obtain the 3D surface coordinates by sphere-tracing along the ray, cast through each pixel. Our appearance rendering is formulated as a function of the 3D and the texture coordinates. Therefore, the forward and backward MLPs can be trained with a 2D pixel-wise loss between the rendered image and the given ground truth image. To the best of our knowledge, this is the first neural rendering method that can learn effective UV parameterization for implicit surfaces.
As a corollary, our method is also the first method which utilizes implicit surface based neural rendering for document unwarping. It is a challenging task due to the presence of geometric and photometric distortions in the document. For this particular problem we introduce a prior for shapespecific texture mapping to initialize the forward MLP (3D to 2D mapping). This prior is learned from a large dataset of UV mapped document meshes, assuming that document texture space maps to a 2D equiangular quadrilateral. This assumption regularizes the forward MLP to output a highquality texture space that avoids degenerate solutions (see Fig. 3). Moreover, we introduce a conformality constraint in the backward MLP, which is consistent with how paper folds happen in the physical world, i.e., without any stretch or tear. This constraint also ensures that the backward function is bijective and smooth (Petrini et al., 2018).
The main contributions of our paper are the following: First, we propose an efficient way to learn texture parameterization for implicit neural representations using a differentiable rendering framework. Without 3D supervision, it only requires multi-view images as ground-truth and a texture mapping prior. Second, we show that our method can be effectively used for document unwarping tasks by learning a prior for explicit texture mapping on the document shape. We show that this prior can be learned from a dataset of texture-mapped meshes. Third, we show that our method is effective for document image unwarping and texture editing (see Fig. 1). We achieve a 52% relative improvement over the publicly available state-of-the-art1 (Das et al., 2019) in terms of mean local distortion across 500 views from ten synthetic scenes. Additionally, we achieve a ∼25% improvement in optical character recognition (OCR) in terms of character and word error rate.

heading: 2 PREVIOUS WORK
text: Neural Rendering. Neural rendering generates images and videos by integrating conventional computer graphics rendering pipelines into deep neural networks (Tewari et al., 2020). It enables explicit or implicit control of scene properties, including illumination, geometry, texture, etc. Neural rendering can synthesize semantic photos (Park et al., 2019b; Bau et al., 2019), novel views (Hedman et al., 2018; Sitzmann et al., 2019a), relighting (Xu et al., 2018; Meka et al., 2019), facial/body reenactment (Chan et al., 2019; Wei et al., 2019), estimate scene properties etc. Kato (Kato et al., 2018) proposed a differentiable neural renderer using an approximate gradient for rasterization. Liu (Liu et al., 2019a) proposed SoftRas, which extended differentiable rasterization. Li (Li et al., 2018) further demonstrated the feasibility of integrating ray-tracing in deep neural networks. More recently, implicit surface or volume rendering has become mainstream in neural rendering approaches such as IDR (Yariv et al., 2020) and NeRF (Mildenhall et al., 2020). These approaches are based
1A more recent approach, CREASE (Markovitz et al., 2020), data and models are not publicly available.
on multi-view surface reconstruction to associate the scene geometry to the appearance in different views. NeRF is extended to lot of variants including PixelNeRF (Yu et al., 2020), MVSNeRF (Chen et al., 2021), dynamic NeRF (Li et al., 2020; Pumarola et al., 2020), GRAF (Schwarz et al., 2020) and so on.
Texture Mapping. Texture mapping is an essential step in the computer graphics rendering pipeline. It defines a correspondence between a vertex on the 3D mesh and a pixel in the 2D texture image. To find such a mapping, FlexiStickers (Tzur & Tal, 2009) required users to specify a sparse set of correspondences. Bi (Bi et al., 2017) proposed a patch-based texture mapping method using the 3D shape and images from multiple views. Morreale (Morreale et al., 2021) used networks to represent 3D surfaces/shapes. Besides the above general texture mapping methods, some approaches focus on a specific object categories such as faces (Deng et al., 2018; Chen et al., 2019) and human bodies (Mir et al., 2020; Zhao et al., 2020). Recently, AtlasNet (Groueix et al., 2018) represented a 3D mesh as a collection of parametric surfaces; thus, texture mapping is trivial to obtain from a 2D parametric surface. A similar idea was adopted by Bednarik (Bednarik et al., 2020) where they introduced geometric constraints when learning the decomposition. More recently NeuTex (Xiang et al., 2021) aims to recover the texture of a subject using NeRF (Mildenhall et al., 2020). However, NeuTex uses a spherical UV domain without any geometric constraints. Therefore, it is not suitable for document unwarping. Moreover, since NeRF (Mildenhall et al., 2020) doesn’t learn an explicit geometry, NeuTex requires a coarse point-cloud to initialize the backward MLP. With an SDF based (Yariv et al., 2020) approach, our approach does not require such an initialization routine. We jointly learn the texture mapping and the geometry from scratch.
Document Unwarping. Document unwarping is a special application of texture mapping: the 3D object is usually a rectangular piecewise-developable surface, and the texture is well-structured, containing straight text lines, (usually) rectangular text blocks and figures, etc. Previous work usually adopted a two-step methodology: 1) 3D surface estimation and 2) deformed surface flattening. The 3D surface of a deformed document can be estimated from shading (Wada et al., 1997), multi-view images (Ulges et al., 2004), text lines (Tian & Narasimhan, 2011), local character orientations (Meng et al., 2018), document boundaries (Koo et al., 2009), and learning-based strategies (Pumarola et al., 2018). Flattening the obtained 3D surface always involves an expensive optimization process under certain geometry constraints such as conformality (You et al., 2017) or isometries (Bartoli et al., 2015). Flattening could be easier if the obtained 3D shape had a low dimensional parameterization like Generalized Cylindrical Surface (GCS) (Kil et al., 2017). Some studies (Das et al., 2017; Liang et al., 2008; Meng et al., 2015) proposed to unwarp each patch on the surface individually and then stitch the unwarped patches together. In recent years, data-driven methods (Ma et al., 2018; Das et al., 2019; Li et al., 2019; Markovitz et al., 2020; Das et al., 2021) have addressed document unwarping by leveraging large-scale synthetic datasets. These datasets contain deformed document images and their corresponding ground-truth UV coordinates. Methods trained on synthetic images often suffer from generalization performance due to the domain gap between synthetic and real data. In this paper, we utilize neural rendering techniques to learn a surface parameterization of a deformed document. We simultaneously estimate both 3D shapes and UV coordinates with a cycle consistency loss and geometric constraints. By leveraging the information from multi-view images, the proposed method demonstrates better document unwarping performance compared to a previous state-of-the-art, Das et al. (2019).

heading: 3 METHOD
text: A schematic diagram of the proposed approach is shown in Fig. 2. We utilize a recent differentiable rendering method, IDR (Yariv et al., 2020) for surface reconstruction and jointly learn the texture mapping of the learned implicit surface using two MLPs. In Sec. 3.1 we first describe some preliminaries about surface parameterization and IDR.

heading: 3.1 PRELIMINARIES
text: Surface Parameterization. The problem of surface parameterization focuses on finding a bijective mapping F between a surface Z ∈ R3 and a polygonal domain Ω ∈ Rn. For a parametric or discrete surface representation, we can explicitly compute this mapping (Tzur & Tal, 2009) using constrained optimization. In contrast, implicit surfaces are represented as continuous functions and cannot be readily parameterized. In this paper, we propose to learn such bijective mapping between a learned
implicit surface and a 2D planar domain Ω ∈ R2 using our proposed forward and backward MLPs. Ω is the texture space or UV space, parameterized using 2D UV coordinates t = (u, v). We can use any continuous parameterization function as the UV space. Since this work particularly focuses on document unwarping, we choose the UV space to be a regular 2D grid.
Implicit Differentiable Rendering. Implicit Differentiable Rendering (Yariv et al., 2020) reconstructs the geometry of an object from multi-view images as the zero level set, Zθ of an MLP S,
Zθ = {z ∈ R3 | S(z; θ) = 0} (1) where θ are the learnable parameters. To render the surface Zθ, IDR uses another MLP to model the radiance (RGB color) as a function of the surface point (zp), corresponding surface normal (np), view direction (vp) and a global geometry feature vector (gp):
Cp = A(zp,np, vp, gp) (2) Here, Cp denote the predicted color at pixel p and A denotes the appearance MLP. The surface point is obtained by a sphere-tracing method (Hart, 1996) along the ray rp(τ) through pixel p. τ ∈ Rk denotes camera parameters of the scene. Additionally, IDR also presents a differentiable way to obtain a ray and geometry intersection point (ẑp) as a function of the camera ray. Although,the IDR can disentangle geometry and appearance, it only allows to re-texture a new geometry with a learned appearance MLP, A. Editing a texture or extracting a surface texture map is not possible in a vanilla IDR framework since no explicit texture mapping is learned.

heading: 3.2 LEARNING SURFACE PARAMETERIZATION
text: To learn a meaningful parameterization of the implicit surface Zθ, we represent the radiance at pixel p as a function of the UV space. To this end, we modify the IDR model (Eq. 2):
Cp = Auv(tp, zp,np, vp, gp) (3) The texture parameterized appearance MLP is modeled as a function of the texture coordinate tp at surface point zp, corresponding to a pixel p. We can jointly train the surface MLP (S) and texture parameterized appearance MLP (Auv) using a pixel wise rendering loss between the predicted radiance (Cp) and ground-truth radiance (Cgtp ) at pixel p.
Forward and backward texture parameterization. We represent the mapping between the 3D surface and 2D texture space using the forward function Fuv:
z→ t. (4) The Fuv is modeled as an MLP. It is trained by mapping a ray-surface intersection point ẑp to its corresponding texture coordinate tp corresponding to a pixel p. Now to establish the bijective mapping (discussed in Sec. 3.1) between the surface and texture space we utilize a backward function Fz: t→ z. (5) Fz is an MLP that learns an inverse mapping between the texture and the 3D space. It is trained by mapping a texture coordinate tp to its corresponding ray-surface intersection point ẑp.
Shape specific prior for Fuv . Jointly training the forward, backward and rendering network leads
to the wrong UV mapping with local minima (see Fig. 3) where multiple ẑp map to a single texture coordinate. To avoid such degenerate cases, we initialize Fuv with a texture mapping prior, learned from a large dataset of UV mapped meshes. This learned prior (F̂uv) makes the learned texture mapping suitable for document unwarping. We assume the document shape to be a deformed quadrilateral and the corresponding UV space to be a regular grid (∈ [0.0, 1.0]). The top leftmost and the bottom rightmost 3D coordinate of the shape maps to (u, v) = (0, 0) and (u, v) = (1, 1) respectively. To learn F̂uv we utilize a collection of UV mapped document meshes from the Doc3D (Das et al., 2019) dataset and train an MLP with the same parameters as Fuv . For each scene, we use F̂uv to initialize the weights of Fuv and train jointly with S and Auv .
Deformation constraints for Fz. Conformal map (Haker et al., 2000) allows a 3D domain to be mapped to a texture domain with low distortion satisfying the bijective property between domains. We use a conformality constraint for Fz to ensure the deformation properties mentioned above. We define the conformality constraint in terms of the metric tensor, J>J of the Fz, where J is the jacobian of Fz (Eq. 6):
J = [ δFz δu δFz δv ] = [Du Dv] J>J = [ D>uDu D > uDv
D>uDv D > v Dv
] = [ E F
F G
] (6)
The conformality constraint is defined as J>J = βI. Here β is a unknown local scaling function and I is the identity matrix. For developable surfaces which can be physically flattened without any stretch e.g. papers, β doesn’t vary across the parameterization space. Therefore, we consider a fixed global scale ([βgu, β g v ]) for the conformality constraint.
Unwarping by sampling Fz. To unwarp an input image, we determine the pixel at p = (x, y) in the input image should be projected to (u, v) in the unwarped image. Here the unwarped image refers to the texture space. The coordinates (u, v) and p are associated by Fz and τ : For a (u, v) coordinate, its corresponding point in 3D is obtained by ẑ′p = Fz(u, v). Given the camera parameter τ , ẑ′p is projected to p in the input image. Thus for each pixel in the unwarped image, we can find its corresponding pixel in the input image which is all we need for unwarping.

heading: 3.3 LOSS FUNCTIONS
text: We use the rendering losses on the predicted color, Cp, and predicted document mask Mp at pixel p to train the geometry S. Here Mp ∈ {0, 1} refers to whether the pixel p is occupied (Mp = 1) by the shape or not (Mp = 0). We assume masks are provided as input. Additionally, we employ appropriate regularization losses to jointly train S, Auv , Fuv and Fz .
Loss for S. Following IDR (Yariv et al., 2020), for each p we apply a sphere tracing (Hart, 1996) algorithm to find the intersection point of the ray rp(τ) and the surface Zθ. Given the ground-truth RGB color Cgtp and the predicted RGB color Cp, the RGB loss is defined as:
Lrgb = 1 |P | ∑ p∈Pin ∥∥Cgtp − Cp∥∥1 (7) Where P is the set of pixels in the minibatch. The pixels Pin ⊂ P for which ray surface intersection has been found and Mp = 1. The mask loss is defined as:
Lmask = 1 α|P | ∑
p∈Pout
CE(Mgtp ,Mp) (8)
Here Pout = P \Pin, alpha is a tunable parameter and CE(.) is the cross-entropy loss. The value of Mp =Mp,α(θ, τ) is a differentiable function of the learned Zθ (Yariv et al., 2020). Additionally, to force Zθ to be a approximate signed distance function we use Eikonal Regularization (Gropp et al.,
2020): Lek = Ez(‖∇zS(z; θ)‖ − 1)2 (9) where z denotes uniformly sampled points within a bounding box of the 3D domain.
Loss for Fuv. Although we initialize Fuv with learned prior parameters, we constrain the predicted 2D texture coordinates during training in order to avoid non-uniform mapping of the 3D and the UV domain which can squeeze or stretch the warped texture (example in supplementary). We employ a Chamfer distance between the tp and uniformly sampled 2D points T ∈ [0, 1] to ensure Fuv approximately outputs U ∼ [0, 1]. This regularization term is defined as:
Luv = CDp∈Pin(T , tp) (10) here CD(.) denotes the Chamfer distance and tp the predicted texture coordinates corresponding to ray-surface intersection points ẑp.
Loss for Fz. ẑ′p is the output of Fz. Fz is trained with weighted regression loss between ẑp and ẑ′p:
Lz = 1 |Pin| ∑ p∈Pin wp(ẑp − ẑ′p)2 (11)
wp is a pre-calculated per-pixel weight based on the document mask (M ) which assigns higher value to the pixels at the boundary of the document. (More weight calculation details in Supplementary).
Additionally, to constrain Fz to be a fixed scale conformal mapping (Bednarik et al., 2020). We employ three constraints on the elements of the metric tensor E, F and G defined in Eq. 6.
LE = 1 |Pin| ∑ p∈Pin (Ep − Ẽ)2 LG = 1 |Pin| ∑ p∈Pin (Gp − G̃)2 LF = 1 |Pin| ∑ p∈Pin (Fp) 2
Here Ẽ and G̃ is the mean of E and G.
Our combined loss function is defined as: L = (Lrgb + γ1Lmask + γ2Lek)︸ ︷︷ ︸
LS
+ρLuv + (δ1Lz + δ2LE + δ3LG + δ4LF )︸ ︷︷ ︸ LT
(12)
Here γ, ρ and δ denote the hyperparameters associated with the losses.

heading: 3.4 TRAINING DETAILS
text: The surface MLP S(z, θ) consists of 8 layers with a hidden layer dimension of 128, with a skip connection to the middle layer (Park et al., 2019a). Following IDR (Yariv et al., 2020), S is initialized to produce an approximate SDF of a unit sphere. The rendering network Auv has 4 layers with hidden layer dimension of 512 and uses a sine activation function (Sitzmann et al., 2020) at each layer. Fuv and Fz share identical architecture with 8 layers with 512 dimensional hidden units and sine activation (Sitzmann et al., 2020). Following NeRF (Mildenhall et al., 2020), we use a k dimensional Fourier mapping (χk : R → R2k) to learn high frequency details in the shape, RGB and the UV space. For S, Auv we follow the setting of (Yariv et al., 2020), and set k = 6 and k = 4 respectively. For Fuv and Fz we empirically set number of Fourier bands k = 10. We start with an initial learning rate of 1e×−5 and train for 150K iterations by halving the learning rate after every 50K iterations. Initially, α is set to 50 and doubled during the training after every 50K iterations. We set γ1 = 100.0, γ2 = 0.1 and ρ = 0.001. δ1 is set to 0.001 for the initial 30K iterations. Afterward, δ1 is multiplied by a factor 2 at every 10K iterations for a maximum of 7 times. δ2, δ3 and δ4, are set to zero for the initial 100K iterations. Only Lz is sufficient to achieve a good texture to 3D mapping during the shape optimization phase. Afterwards we set δ2 = δ3 = 0.001 and δ4 = 0.01. The metric tensor calculation is implemented using auto-differentiation.

heading: 4 EXPERIMENTAL RESULTS
text: First, we quantitatively compare the proposed method with state-of-the-art document unwarping method DewarpNet (Das et al., 2019). Our quantitative and qualitative experiments are performed on 10 synthetic scenes and 10 real scenes. Second, we apply our method to texture editing. Last, we conduct ablation studies to demonstrate the effectiveness of our proposed loss functions.

heading: 4.1 EVALUATION DATASET AND METRICS
text: Our synthetic evaluation data consists of 10 scenes rendered using Blender following a rendering pipeline similar to Doc3D. Each scene consists of 50 random views sampled from a 45o solid angle in the upper hemisphere. The real-world evaluation data consists of 3 scenes from the dataset of (You et al., 2017), and 9 scenes captured by us. Each scene consists of 5-20 images per scene. We manually annotate the masks for each scene. To obtain camera poses for the real-world data, we utilize the COLMAP (Schönberger & Frahm, 2016) multi-view reconstruction pipeline. Both synthetic and real data include the document scan, as the unwarping ground-truth.
We use image-based evaluation metrics for quantitative evaluation, including Local Distortion (LD) and Multi-Scale Structural Similarity (MS-SSIM). These are standard metrics used for document unwarping evaluation (Das et al., 2019; Ma et al., 2018). LD is based on dense SIFT flow (Liu et al., 2011) between the unwarped and scanned images. Image similarity metric, MS-SSIM (Wang et al., 2003) is based on local image statistics (mean and variance) of the unwarped and scanned (groundtruth) images calculated over multiple Gaussian pyramid scales. We use the same settings as (Das et al., 2019; Ma et al., 2018) for fair comparison.

heading: 4.2 DOCUMENT UNWARPING
text: The primary application of our learned forward and backward MLP is document unwarping. The quantitative comparison with the state-ofthe-art model (Das et al., 2019) is shown in Table 1 for the synthetic and real scenes. In terms of average performance of all the views (all views col. in Table 1) we improve the LD by ∼ 52% compared to (Das et al., 2019). Since we use multi-view images for training, our results are more consistent across all the views compared to DewarpNet, which is also a key reason for the significant improvement. We conjecture that (Das et al., 2019) as a single image unwarping method should perform well on simpler deformations and frontal view images. However, it is not always the case. In qualitative comparisons in Fig. 4, DewarpNet often gen-
erates artifacts even for reasonably frontal views and simple deformations. Comparatively, our results are qualitatively superior.
We also report in a stricter evaluation scenario (best view column of Table 1) where we compare our results with the best possible numerical results achieved by DewarpNet from a single view in a scene. We perform better than DewarpNet in 91.2% of all views, however when the best view can be selected our method do slightly worse in 7 scenes. This ‘stricter’ setting shows quantitatively competitive results compared to DewarpNet with a oracle (practically challenging) view selector.The choice of the best unwarped result is often subjective. For a more comprehensive comparison, we qualitatively compare the best results of DewarpNet with our results across 6 scenes in Fig. 5. These 6 scenes are chosen among the 7 scenes for which DewarpNet achieves a better quantitative result than the proposed approach for at least one view. In Fig. 5 our results are clearly better than the DewarpNet in all cases, with straighter lines and better rectified structure. The evaluation scores do not accurately reflect the improvement due to the sensitivity of LD and MSSIM to subtle perceptually unimportant global transformations, such as translation of the image by few pixels. However, such transformations do not affect the visual quality or readability of the unwarped results. More discussion and qualitative comparison is available in supplementary material.
DewarpNet Proposed
ED ↓ CER (std) ↓ WER (std) ↓ ED ↓ CER (std) ↓ WER (std) ↓ Mean 798.30 0.2827 (0.12) 0.4646 (0.17) 600.78 0.2122 (0.10) 0.3568 (0.11)
Table 2: Comparison of OCR error metrics: We improve the OCR performance of Das et al. (2019) by∼25% in terms of Edit Distance (ED), Character Error Rate (CER), and Word Error Rate (WER).
The quantitative comparison for real scenes are reported in Table 1 (bottom). We achieve better results in terms of mean and best evaluation score than DewarpNet in 2 out of 3 scenes. We notice that the evaluation results are a little worse for the real scenes than synthetic scenes due to the fewer available views (5-10 compared to 50). Moreover, there are cases like Real 6, which do not have sufficient texture. Such data are a failure case of IDR since there is insufficient information to reconstruct the 3D shape. As a result of the poor 3D shape, our texture parameterization network produces an inferior unwarping result (More details are available in Supplementary). We also report qualitative comparisons with You et al. (2017) and Das et al. (2019) on additional real documents in supplementary.
OCR Evaluation. We also evaluated the OCR performance on 5 real scenes across 77 images in Table 2. We use Edit Distance (ED) (Miller et al., 2009), Character Error Rate (CER) and Word Error Rate (WER) as our evaluation metrics. ED is defined as the total number of substitutions (s), insertions (i) and deletions (d) required to obtain the reference text, given the recognized text. The reference text is obtained by running the OCR algorithm on the scanned ground-truth image of each document. CER is defined as: (s+ i+ d)/N where N is the number of characters in the reference text. We use Tesseract 4.1.1 based LSTM OCR engine for this experiment. Our unwarped results reduce the ED, CER and WER by ∼25%. This improvement proves our unwarped results are more suitable for downstream applications tasks like OCR.
Texture Editing. In addition to document unwarping, our proposed forward and backward MLP can also be used for high quality texture editing. We show two texture editing examples in Fig. 1. We use the backward MLP to unwarp the texture from the input image, then we edit the texture and warp it back to image space using the learned forward MLP. (More details in Supplementary).
Ablation Study. We ablate how loss terms Lz , LE , LF , and LG affect the unwarping results. We train Fuv and Fz with different combinations of these loss terms and report the mean MSSIM and LD in Table 3 (appendix). Qualitative results for one scene are shown in Fig. 6 (appendix).

heading: 5 TRAINING TIME, GENERALIZABILITY AND FUTURE WORK
text: Our proposed method for a scene can be trained in approximately 18 hours for 448× 448 resolution images using a single Titan Xp GPU. The current training time per scene is very high compared to DewarpNet’s inference time which makes it unsuitable for real time applications. However, this is a fast growing field and there are multiple other works that are focusing on improving the speed and generalization abilities (Garbin et al., 2021; Bergman et al., 2021) of neural rendering. Therefore, obtaining a faster training scheme is considered as a future work.
Our method can be applied to fabrics, which are very similar to papers and lead to practical applications of texture editing. However, none of the current 3D garment/fabric datasets (Patel et al., 2020) can be easily adapted to train the F̂uv prior. For more complex UV spaces (e.g., texture atlas), learning the prior may require decomposing the shape to multiple simple UV maps. The proper way to do this is beyond the scope of this paper, however we believe it’s an exciting future work. As importantly, in this paper, we have introduced a number of domain specific strong constraints that suit the rectangular paper shape. These constraints improve empirical results. More general objects will require different constraints e.g., spherical UV domain, local scaling of the conformal map etc.

heading: 6 CONCLUSIONS
text: We have introduced an end-to-end trainable architecture that can simultaneously learn texture parameterized 3D shapes from multi-view images. This is the first work to learn surface parameterization of an implicit neural representation to the best of our knowledge. We have demonstrated the applicability of our approach on multiple synthetic and real scenes for the task of document unwarping and document texture editing. We want to extend this method to learn surface parameterization for more complex shapes such as faces or general 3D objects in future work.

heading: 7 ETHICS STATEMENT
text: Texture editing application of our proposed approach can have both positive and negative societal impact. On the positive side, real document images can be gracefully redacted to protect sensitive information. On the contrary, it can be potentially used for editing real documents and change the content to commit fraud and spread misinformation.

heading: 8 REPRODUCIBILITY STATEMENT
text: We believe our results are reproducible by following the training details in Section 3.4 of main submission and Section 2, 3, 4, 5 from the supplementary material.

heading: A APPENDIX
text: A.1 ABLATION STUDY
We ablate how loss terms Lz , LE , LF , and LG affect the unwarping results. We train Fuv and Fz with different combinations of these loss terms and report the mean MSSIM and LD in Table 3. Qualitative results for one scene are shown in Fig. 6 (appendix). In the basic version (listed as b), no conformality constraints (LE , LF and LG) are used and wp = 1 in Lz . The b+w version introduces a weighting function that assigns a higher value to wp if a pixel is closer to the document boundary. Introducing this loss improves the boundary; notice the white margin at the top and bottom in the second column of Fig. 6. Introducing the conformality constraints alleviates the unusual stretch in the texture and improves smoothness (Fig. 6, col. 3). Using both improves the boundary and the texture smoothness (Fig. 6, col. 4) and achieves the best result.
(a) (b) (c)
(d) (e) (f)

Review summaries:
The authors present a method to jointly learn a shape and UV parameterization of a creased document observed from multiple views. They use an SDF based implicit shape representation in combination with a recent neural volumetric renderer IDR, and optimize the energy composed of the main photometric term (a discrepancy between the observed and rendered image) and regularization terms enforcing well behaved UV space and a fixed scale conformal mapping which corresponds to a behaviour of a real paper material. Since the implicit field representation does not explicitly model a surface, the authors cannot directly optimize the 3D->2D mapping using standard graphics techniques and rather adapt the IDR renderer to be conditioned on the locations sampled from the UV space and train it jointly with the MLPs defined for the bijective mapping 2D->3D and 3D->2D. The authors show that they can get better results both quantitatively and qualitatively when compared to one of the SotA works DewarpNet.

**High-Level Overview**  This paper proposes a neural rendering technique to learn how to estimate an implicit surface and its UV-parameterization from a set of 2D images. The proposed approach is applied to the problem of document unwarping, wherein a wrinkled, folded, or bent document (e.g. a piece of paper with text on it) can be digitally flattened with its contents preserved. For good measure, the paper also shows that learning the texture mapping enables the contents of the unwarped document to be edited in the texture space. The proposed approach is evaluated against a prior work, DewarpNet (Das et al., 2019) on the basis of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) as well as the auxiliary task of OCR.  **Key Contributions**  * Builds on IDR (Yariv et al., 2020) to learn an explicit bijective texture mapping for a learned implicit surface * A loss function formulation for the framework * Application of learned implicit surfaces for document unwarping with reduction in texture distortion and improvements in subsequent OCR metrics  **How It Works**  * Starts with the same problem formulation as IDR: an MLP to model the surface (Eq. 1) and an appearance MLP to model the color (Eq. 2) * Modifies the appearance MLP to also be a function of texture coordinates (Eq. 3) * Adds an additional forward and backward MLP to learn the bijective UV mapping * Jointly trains all MLPs using a pixelwise rendering loss

This paper presents document unwrapping based on optimization of neural networks. Four MLPs are defined to model the implicit surface of the target, the colors of the target, the mapping function from 3D points to 2D texture coordinates, and the inverse mapping from 2D to 3D. Similarly to NERF, the target example is approximated by training the MLPs using differentiable rendering.   

The paper presents a multi-view method for undistorting documents. The method makes use of an existing method (IDR) for reconstructing distorted documents in 3-D, with an additional bijective neural network to parameterize distorted surfaces by rectangles. The method significantly outperforms SOTAs in the experiments.