Learning Surface Parameterization for Document Image Unwarping
Authors: Sagnik Das
Ke Ma
Zhixin Shu
Dimitris Samaras
Venue: ICLR 2022 Submitted (2022)
Keywords: implicit functions
texture mapping
surface parameterization

Abstract: In this paper, we present a novel approach to learn texture mapping for a 3D surface and apply it to document image unwarping. We propose an efficient method to learn surface parameterization by learning a continuous bijective mapping between 3D surface positions and 2D texture-space coordinates. Our surface parameterization network can be conveniently plugged into a differentiable rendering pipeline and trained using multi-view images and rendering loss. Recent work on differentiable rendering techniques for implicit surfaces has shown high-quality 3D scene reconstruction and view synthesis results. However, these methods typically learn the appearance color as a function of the surface points and lack explicit surface parameterization. Thus they do not allow texture map extraction or texture editing. By introducing explicit surface parameterization and learning with a recent differentiable renderer for implicit surfaces, we demonstrate state-of-the-art document-unwarping via texture extraction. We show that our approach can reconstruct high-frequency textures for arbitrary document shapes in both synthetic and real scenarios. We also demonstrate the usefulness of our system by applying it to document texture editing.

Under review as a conference paper at ICLR 2022
LEARNING SURFACE PARAMETERIZATION FOR DOCU- MENT IMAGE UNWARPING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, present novel approach to learn texture mapping for 3D sur- face and apply it to document image unwarping. We propose an efficient method to learn surface parameterization by learning a continuous bijective mapping be- tween 3D surface positions and 2D texture-space coordinates. Our surface param- eterization network can be conveniently plugged into differentiable rendering pipeline and trained using multi-view images and rendering loss. Recent work on differentiable rendering techniques for implicit surfaces has shown high-quality 3D scene reconstruction and view synthesis results. However, these methods typi- cally learn the appearance color as a function of the surface points and lack explicit surface parameterization. Thus they do not allow texture map extraction or texture editing. By introducing explicit surface parameterization and learning with re- cent differentiable renderer for implicit surfaces, we demonstrate state-of-the-art document wnwaeennnn via texture extraction. We show that our approach can re- construct
textures for arbitrary document shapes in both synthetic and real scenarios. We also demonstrate the usefulness of our system by applying it to document texture editing.
INTRODUCTION
Reconstructing 3D shapes from images is core problem in computer vision and graphics research.
With the progress in differentiable rendering (Sitzmann et al.) 2019b Kato et al. 2018; Niemeyer al.] [2200] Li et al., 2018) Liu et al. 2019b), recent learning based 3D reconstruction approaches have achieved impressive results using 2D supervision from single image (Chen & Zhang 2019
Groueix et al., 2018, Choy et al. 2016 Mescheder et al. 2019 Wang et al. 2018) or mulri-view images Tang & Tan 2018: Yariv et al. 2020). These methods achieve high quality 3D reconstruc- tion using differentiable rendering with various 3D representations such as 3D mesh (Wang et al., 2018), volumetric representation (Mildenhall et al. (020), or implicit functions (Mescheder et al. 2019). recent neural rendering methods such as NeRF (Mildenhall et al. 2020) and IDR ( (Yariv iet al. 2020), continuous representations such volume implicit functions achieve significantly better reconstruction results than meshes or voxels because they do not discretize the 3D surface a priori. However, these continuous representations usually do not encode explicit surface parame- terization, allowing 3D shape re-texturing, editing the existing texture in the 2D texture space, or recovering 2D texture from 3D surfaces. One of the most direct applications of 2D texture recov- cry in geometrically constrained manner, is document unwarping, i.c., inference of a document's ' version from a casual photo of a potentially creased document. Whereas 2D tex- ture recovery could be equally valuable for other domains such as garments, or faces, the existing datasets are not directly applicable to our method.
Our novel texture mapping approach learns surface parameterization for document unwarping by learning continuous bijective functions between 3D surface positions and 2D texture-space coor- dinates. We use signed distance function (SDF) (Chan & Zhu| (005) to represent geometry and model the appearance as a function of the 2D texture coordinates. By utilizing implicit differentiable rendering (IDR), (Yariv et al. 2020) parameterization of the surface simultaneously using a per-pias pex-p rendering loss and appropriate ge- ometric regularizations.
reconstruct 3D shape and learn the corresponding UV
We utilize two fully connected multi-layer perceptrons (MLPs) to learn a bijective mapping be- tween 3D shapes and 2D texture space. More specifically, the forward ML. maps the 3D surface

---

Under review as a conference paper at ICLR 2022
AMAZNGLYY ACCURATE QUANILUM COMPUTING
Input
Unwarped
Edited
Texture edited images
Figure 1: Proposed forward- serard-backwaad backward network can be utilized in unwarping or editing the surface texture: The flattened texture can be edited and warped back to produce a texture edited image.
coordinates to 2D texture coordinates and the backward MLP maps the 2D texture coordinates to corresponding 3D surface coordinates. Following IDR (aariv et al. 2020), we obtain the 3D surface coordinates by sphere-tracing tracing along the ray, cast through each pixel. Our appearance rendering is formulated function of the 3D and the texture coordinates. Therefore, the forward and backward MLPs can be trained with a 2D pixel-wise loss between the rendered image and the given ground truth image. To the best of our knowledge, this is the first neural rendering method that can learn effective UV parameterization for implicit surfaces.
As a corollary, our method is also the first method which utilizes implicit surface based neural rendering for document unwarping. It is challenging task due the presence of geometric and photometric distortions in the document. For this particular problem we introduce a prior for shape- specific texture mapping to initialize the forward MLP (3D to 2D mapping). This prior is learned from a large dataset of UV mapped document meshes, assuming that document texture space maps to 2D equiangular quadrilateral. This assumption regularizes the forward MLP to output a high- quality texture space that avoids degenerate solutions (see Fig. 33. Moreover, we introduce a con- formality constraint in the backward MLP, which consistent with how paper folds happen in the physical world, i.c., without any stretch or tear. This constraint also ensures that the backward function is bijective and smooth (Petrini et al. 2018).
The main contributions of our paper are the following: First, we propose an efficient way to learn texture parameterization for implicit neural representations using differentiable rendering frame- work. Without 3D supervision, it only requires multi-view images as ground-truth and texture mapping prior. Second, we show that our method can be effectively used for document unwarping tasks by learning prior for explicit texture mapping the document shape. We show that this prior can be learned from a dataset of texture-mapped meshes. Third, we show that our method is effective for document image unwarping and texture editing (see Fig. T]. We achieve 52% relative improvement over the publicly available state-of-the-art (Da et al. 2019) in terms of mean local distortion across 500 views from ten synthetic scenes. Additionally, ment in optical character recognition (OCR) in terms of character and word error rate.
achieve 25% improve-
2 PREVIOUS WORK
Neural Rendering. Neural rendering generates images and videos by integrating conventional com- puter graphics rendering pipelines into deep neural networks (Tewari et al.) 2020). It enables explicit or implicit control of scene properties, including illumination, geometry, texture, etc. Neural render- ing can synthesize semantic photos (Park al.) [219b]] Bau al. 2019 novel views (Hedman al.,
2018 Sitzmann et al. 2019a), relighting (Xu et al. 2018: Meka et al.) 2019). facial/body reenact- ment (Chan et al.) 2019 Wei et al., (019)) estimate scene properties etc. Kato (Kato et al., 2018) proposed differentiable neural renderer using approximate gradient for rasterization. Liu (Liu ctt al. 2019a) proposed SoftRas, which extended differentiable rasterization. Li (Li et al. 2018) further demonstrated the feasibility of integrating ray-tracing in deep neural networks. More cently, implicit surface or volume rendering has become mainstream in neural rendering approaches such IDR Yariv et al.] 2020) and NeRF (Mildenhall et al. 2020). These approaches are based
recent approach, CREASE Markovitz al. 2020), data and models publicly available.
2

---

Under review as a conference paper at ICLR 2022
on multi-view surface reconstruction to associate the scene geometry to the appearance in different views. NeRF is extended to lot of variants including PixeINeRF Yu et al. 2020), MVSNeRF (Chen
et al. 2021), dynamic NeRF (Li et al. 2020; Pumarola et al. 2020). GRAF (Schwarz et al. 2020)
and 80 on.
Texture Mapping. Texture mapping is essential step in the computer graphics rendering pipeline. defines a correspondence between a vertex on the 3D mesh and a pixel in the 2D texture image. To find such mapping, FlexiStickers (Tzur & Tal 2009) required users to specify sparse set of correspondences. Bi (Bi et al., 2017) proposed a patch-based texture mapping method using the 3D shape and images from multiple views. Morreale (Morreale et al. (021] used networks to rep- resent 3D surfaces/shapes. Besides the above general texture mapping methods, some approaches
focus on a specific object categories such as faces (Deng al. 2018 Chen et 2019) and human
bodies (Mir et al., 2020, /Zhao et al. 2020). Recently, AtlasNet (Grooueix et al.
2018) represented
3D mesh as a collection of parametric surfaces; thus, texture mapping trivial obtain from a
2D parametric surface. A similar idea was adopted by Bednarik (Bednarik et al.) 2020) where they introduced geometric constraints when learning the decomposition. More recently NeuTex (Xiang et al. 2021) aims to recover the texture of a subject using NeRF (Mildenhall et al. 2020). How- ever, NeuTex uses a spherical UV domain without any geometric constraints. Therefore, it is not suitable for document unwarping- Moreover, since NeRF (Mildenhall al., (020) doesn't learn an explicit geometry, NeuTTex requires coarse point-cloud to initialize the backward MLP. With an SDF based (Yariv al., (020) approach, our approach does not require such an initialization routine. We jointly learn the texture mapping and the geometry from scratch.
Document Unwarping. Document unwarping is special application of texture mapping: the 3D object is usually a rectangular piecewise-develope
surface, and the texture is well-structured, containing straight text lines, (usually) rectangular text blocks and figures, etc. Previous work usually adopted a two-step methodology: 1) 3D surface estimation and 2) deformed surface flattening. The 3D surface of deformed document can be estimated from shading (Wada al., 1997). multi-view images (Ulges et al.) 2004), text lines (Tian & Narasimhan] 2011), local character orientations (Meng) ct al. 2018), document boundaries (Koo et 22.9), and learning- based strategies (Pumarola et al., 2018 Flattening the obtained 3D surface always involves expensive optimization process under
certain geometry constraints such as conformality You et al. 2017) or isometries (Bartoli et al.,
2013). Flattening could be easier if the obtained 3D shape had a low dimensional parameterization
like Generalized Cylindrical Surface (GCS) (Kil et al.] (007).. Some studies (Das et 2017 Liang al. 2008 Meng et al.) 2015) proposed to unwarp each patch on the surface individually and then stitch the unwarped patches together. In recent years, data-driven methods (Ma et al. 2018 Das
2019; Li et al. 2019 Markovitz et al., 2020; Das et al., 2021) have addressed document unwarping by leveraging large-scale synthetic datasets. These datasets contain deformed document images and their corresponding ground-truth UV coordinates. Methods trained on synthetic images often suffer from generalization performance due to the domain gap between synthetic and real data. this paper, we utilize neural rendering techniques to learn a surface parameterization of a deformed document. We simultaneously estimate both 3D shapes and UV coordinates with a cycle consistency loss and geometric constraints. By leveraging the information from multi-view images, the proposed method demonstrates better document unwarping performance compared to a previous state-of-the-aa Das et al. (2019).
3 METHOD
A schematic diagram of the proposed approach is shown in Fig. 2 We utilize a recent differen- tiable rendering method, IDR (Yariv et al. (2200) for surface reconstruction and jointly learn the texture mapping of the learned implicit surface using two MLPs. In Sec. [.]l we first describe some preliminaries about surface parameterization and IDR.
3.1 PRELIMINARIES
Surface Parameterization. The problem of surface parameterization focuses on finding a bijective mapping F between a surface Z E R23 and a polygonal domain Ω2 E I.".. For a parametric or discrete surface representation, we can explicitly compute this mapping (Tzur& & Tal 2009) using constrained optimization. In contrast, implicit surfaces are represented as continuous functions and cannot be readily parameterized. In this paper, we propose to learn such bijective mapping between learned
3

---

Under review as a conference paper at ICLR 2022
IDR
F
hp
Pp Cp
Za
1.5
x
Figure 2: Proposed surface parameterization learning using the forward (Fuv) and backward MLP (F,): Given camera pose T, and a pixel we jointly learn the geometry represented by a SDF Zg. the Fuvs and the Fg. 2p is the ray-surface intersection point in 3D domain and tp is the corresponding texture coordinate in UV domain. The yellow arrows denote the input and output of the IDR Yariv et al. 2020), and Cp is the predicted RGB color. Triangles denote the losses defined in Eq. 12
implicit surface and 2D planar domain Ω R using proposed forward and backward MLPs. is the texture space or UV space, parameterized using 2D UV coordinates 1 = (u. v). We can use any continuous parameterization function the UV space. Since this work particularly focuses document unwarping, we choose the UV space to be regular 2D grid.
Implicit Differentiable Rendering. Implicit Differentiable Rendering YYariv al. 2020 recon- structs the geometry of an object from multi-view images as the zero level set, Ze of an MLP S,
Zo (z R2 S(z; = 0}
where 0 are the learnable parameters. To render the surface Ze. IDR uses another MLP to model the radiance (RGB color) as a function of the surface point (Z,). corresponding surface normal (np), view direction (vp) and a global geometry feature vector (g,)):
Cp = A(Zp, np. Vp. gp))
(2)
Here, Cp denote the predicted color at pixel p and A denotes the appearance MLP. The surface point obtained by sphere-tracine method (Hart 1996) along the ray rT(T) through pixel p. т E g denotes camera parameters of the scene. Additionally, IDR also presents a differentiable way to obtain a ray and geometry intersection point (2,r) as a function of the camera ray. Although,the IDR can disentangle geometry and appearance, it only allows to re-texture new geometry with learned appearance MLP, A. Editing a texture or extracting a surface texture map is not possible a vanilla IDR framework since explicit texture mapping is learned.
3.2 LEARNING SURFACE PARAMETERIZATTION
To learn a meaningful parameterization of the implicit surface Zo. we represent the radiance at pixel p as a function of the UV space. To this end, we modify the IDR model (Eq. 2
Cp Aue (tp: Zp. Dp. Yp. Kp)
The texture parameterized appearance MLP is modeled as a function of the texture coordinate t, surface point zp. corresponding to a pixel p. We can jointly train the surface MLP (S) and tex- ture parameterized appearance MLP (A using pixel wise rendering loss between the predicted radiance (C₂) and ground-truth radiance (C%)) at pixel p.
Forward and backward texture parameterization. We represent the mapping between the 3D surface and 2D texture space using the forward function F.v:
t.
(4)
The Fav is modeled as an MLP. It is trained by mapping a ray-surface intersection point 2, to its corresponding texture coordinate tp corresponding to pixel p. Now to establish the bijective map- ping (discussed in Sec. 3.I between the surface and texture space we utilize backward function F,:
(5)
F, is an MLP that learns an inverse mapping between the texture and the 3D space. trained by mapping texture coordinate t, to its corresponding ray-surface intersection point ž,,.
Shape specific prior for Fu.. Jointly training the forward, backward and rendering network leads
4

---

Under review as a conference paper at ICLR 2022
to the wrong UV mapping with local minima (see Fig. where multiple 2p map to single texture coordinate. To avoid such degenerate cases, we initialize Fuv with a texture map- ping prior, learned from a large dataset of UV mapped meshes. This learned prior (Fuv) makes the learned texture mapping suitable for document unwarping. We assume the document shape to be a deformed quadrilateral and the corresponding UV space be a regular grid (E [0.0, 1.0D. The top leftmost and the bottom rightmost 3D coordinate of the shape maps to (u, v) (0, 0) and (u, v) (1, respectively. To learn Fur we utilize a collection of UV mapped document meshes from the Doc3D (Das et al.] 2019) dataset and train MLP with the same parameters as Fue. For each scene, we use Fuv to initialize the weights Fuv and train jointly with and Auv.
Figure Without prior the for- ward network, Fav leads to degen- erate cases: multiple 3D points, & are mapped to the same texture co- ordinate 1p-
Deformation constraints for Fr. Conformal map (Haker et al. 2000) allows 3D domain to be mapped to a texture domain with low distortion satistying the bijec- tive property between domains. We the deformation properties mentioned above. We define the conformality constraint in terms of the metric tensor, JJ J of the Ez. where J is the jacobian of F, (Eq. 6
conformality constraint for F, to ensure
8F, FF, = 8u
J D Da DT D.D,D, D. D.
E
F G
= [Du Del
(6)
dv
F
The conformality constraint is defined J BI Here 8 is unknown local scaling function and I the identity matrix. For developable surfaces which can be physically flattened without any stretch e.g. papers, 3 doesn't 't vary across the parameterization space. Therefore, consider fixed global scale ([3g, (g] for the conformality constraint.
Unwarping by sampling Fz. To unwarp input image, determine the pixel at p (a, y) in the input image should be projected to (u, v) the unwarped image. Here the unwarped image refers to the texture space. The coordinates (u, u) and p associated by F, and r: For (u, v) coordinate, its corresponding point in 3D is obtained by 2 fz v). Given the camera parameter & is projected to p in the input image. Thus for each pixel in the unwarped image, we can find its corresponding pixel in the input image which all need for unwarping.
3.3 LOSS FUNCTIONS
We the rendering losses the predicted color, Cp. and predicted document mask Mp at pixel to train the geometry S. Here Mp E (0, 1) refers to whether the pixel is occupied (Mp = 1) by the shape or (M, 0). We assume masks are provided as input. Additionally, we employ appropriate regularization losses to jointly train S, Aue Fuv and F.-
Loss for S. Following IDR YYariv al. 2020), for each p we apply sphere tracing (Hart 1996) algorithm to find the intersection point of the ray (T) and the surface Zg. Given the ground-truth RGB color Cs and the predicted RGB color Cp, the RGB loss is defined as:
Lrgb = P
Σ PEP.
|| -Crll1
Where P is the set of pixels in the minibatch. The pixels P.n. C. P for which ray surface intersection has been found and M, = The mask loss is defined as:
I Timask
Σ PEP
CE(MBL p Mp)
a P
Here Pout P Pins alpha is tunable parameter and CE(.) is the cross cotso-y loss. The value of
Mr. = Mp.a (8, 7) is a differentiable function of the learned Z, Yariv et al. 2020). Additionally, to force 7n to be approximate signed distance function we Eikonal Regularization (Gropp al.)
5

---

Under review as a conference paper at ICLR 2022
2020
Lek = ( ((; - -)2
(9)
where: 2 denotes uniformly sampled points within a bounding box of the 3D domain.
Loss for Fuv. Although we initialize Fuv with learned prior parameters, constrain the predicted 2D texture coordinates during training in order to avoid non-uniform mapping of the 3D and the UV domain which can squeeze or stretch the warped texture (example in supplementary). We employ a Chamfer distance between the l and uniformly sampled 2D points T E [0, to ensure Fur approximately outputs U [o, 1]. This regularization term is defined as:
Luv = CD (T.tp)
(10)
here (.) denotes the Chamfer distance and ip the predicted texture coordinates corresponding to ray-surface intersection points 2p.
Loss for Fz. žy is the output of Fz. Fz trained with weighted regression loss between εp and #/2
Σ wp(áp - 2/22²
(11)
P
PEP...
wp is pre-caleulated per-pixel weight based on the document mask (M) which assigns higher value to the pixels at the boundary of the document. (More weight calculation details in Supplementary).
Additionally, to constrain F2 to a fixed scale conformal mapping (Bednarik al., 2020p. We employ three constraints on the elements of the metric tensor E, F and G defindd in Eq. 6
LE
Σ (Ep E)² PEPmn.
LG
Σ EG, (G,-G) LF PPon
1
Σ (F
Pin
PPan
PEP.n.
PEPi
Here É and G is the mean of E and G. Our combined loss function is defined as:
L = (Lrg + 71 Lmakk + bebc tol + (61Lz + L2LE + 8gLa + +gbF)
(12)
Ls
LT
Here 7, p and 8 denote the hyperparameters associated with the losses.
3.4 TRAINING DETAILS
The surface MLP S(z, 8) consists of 8 layers with a hidden layer dimension 128, with a skip connection to the middle layer (Park et al.) 2019a). Following IDR Yarri al.) 2020), S is ini- tialized to produce an approximate SDF of a unit sphere. The rendering network A.o has 4 layers with hidden layer dimension of 512 and uses sine activation function (Sitzmann al. 2020) at each layer. Fav and Fz share identical architecture with 8 layers with 512 dimensional hidden units and sine activation (Sitzmann et al. 2020). Following NeRF (Mildenhall et al. 2020), we use a dimensional Fourier mapping (Xx R + 2k)) to learn high frequency details in the shape, RGB and the UV space. For Aue we follow the setting of (Yariy al., 2020), and set k = and = respectively. For Fur and Fg empirically set number of Fourier bands k = 10. We start with an initial learning rate -ex and train for 150K iterations by halving the learning rate after every 50K iterations. Initially, is set to 50 and doubled during the training after every 50K iterations. We set 71 = 100.0, 722 0.1 and p = 0.001. 8 is set to 0.001 for the initial 30K iterations. Afterward, 5, is multiplied by factor 2 every 10K iterations for maximum of 7 times. 82, ōg and 54, are set to zero for the initial 100K iterations. Only J.z is sufficient to achieve a good texture to 3D) mapping during the shape optimization phase. Afterwards we set d3 0.001 and d4 0.01. The metric tensor calculation is implemented using auto-differentiation.
EXPERIMENTAL RESULTS
First, we quantitatively compare the proposed method with state-of-the-arr document unwarping method DewarpNet (Das et al., 2019). Our quantitative and qualitative experiments are performed on 10 synthetic scenes and 10 real scenes. Second, we apply our method to texture editing. Last, we conduct ablation studies demonstrate the effectiveness of our proposed loss functions.

---

Under review as a conference paper at ICLR 2022
4.1 EVALUATION DATASET AND METRICS
Our synthetic evaluation data consists 10 scenes rendered using Blender following a rendering pipeline similar to Doc3D. Each scene consists of 50 random views sampled from 45" solid an- gle in the upper hemisphere. The real-world evaluation data consists of 3 scenes from the dataset of (You a 2017), and 9 scenes captured by us. Each scene consists of 5-20 images per scene. We manually annotate the masks for each scene. To obtain camera poses for the real-world data, utilize the COLMAP (Schönberger & Frahm 2016) multi-view reconstruction pipeline. Both synthetic and real data include the document scan, as the unwarping ground-trute
We use image-based evaluation metrics for quantitative evaluation, including Local Distortion (LD) and Multi-Scale Structural Similarity (MS-SSIM) These are standard metrics used for document unwarping evaluation (Das et al., 2019 Ma al. 2018). LD based on dense SIFT flow (Liu al. 2011 between the unwarped and scanned images. Image similarity metric, MS-SSIM (Wang et al.
2003
is based on local image statisties (mean and variance) of the unwarped and scanned (ground- images calculated over multiple Gaussian pyramid scales. We use the same settings as (Das)
truth)
ct al. 2019: Ma et al., (0188) for fair comparison.
4.2
DOCUMENT UNWARPING
The primary application of our learned forward and backward MLP document unwarping. The quan- titative comparison with the state-of- the-art model (Das et al., 2019) is shown in Table the synthetic and real scenes. In terms of average per- formance of all the views (all views col. in Table [ we improve the LD by ~ 52% compared
DewarpNet (all views)
DewarpNet (best view)
Proposed (all views)
Soene
MSSIM f
LD |
MSSIM
LD J
MSSIM
LD
Synth
0.42
9.54
3.29
0.74
2.59
Synth 2
0.75
5.68
0.83
2.59
0.76
4.40
Synth 3
0.73
7,80
0.85
2.94
0.78
5.44
Synth
0.59
6.88
0.63
2,53
0.64
2.85
Synth 5
0.48
7.11
0.64
3.13
0.61
4.55
Synth
0.50
6.34
0.62
2.53
0.47
3.92
Synth
0.52
7.99
0.76
2.64
0.74
2.55
(Das et al. 2019). Since we use multi-view im- ages for training, our results are more consistent across all the views com- pared DewarpNet, which also a key reason for the significant im- provement. We conjecture that (Das et al.] 2019) as single image un- warping method should perform well on simpler deformations and frontal not al-
Synth 8
0.56
10.05
0.70
3.44
0.64
5.31
Synth
0.49
7.48
0.73
1.87
0.78
1.56
Synth
0.52
8.07
0.78
2.78
0.73
3.13
Syn. Mean
0.56
7.69
0.70
2.82
0.69
3.63
Real I
0.26
9.77
0.39
5.78
0.37
5.68
Real 12
0.24
12.94
0.24
10.98
0.35
8.38
Real
0.44
9.15
0.48
7.78
0.37
16.80
Real Mean
0.31
10.62
0.37
8.18
0.36
10.28
Table 1:
Comparison with (Das et al. 2019) on synthetic scenes: all views refers to the mean error metric on all scene images, best view refers to the lowest possible error from an image in scene.
view images. However, ways the case. In qualitative compar- isons in Fig. DewarpNet often gen- erates artifacts even for reasonably frontal views and simple deformations. Comparatively, our sults are qualitatively superior.
We also report in a stricter evaluation scenario (best view column of Table I where compare our results with the best possible numerical results achieved by DewarpNet from a single view in scene. We perform better than DewarpNet in 91.2% of all views, however when the best view can be selected our method do slightly worse in 7 scenes. This "stricter' setting shows quantitatively competitive results compared to DewarpNet with a oracle (practically challenging) view selector: The choice of the best unwarped result is often subjective. For a more comprehensive comparison, we qualitatively compare the best results of DewarpNet with our results across scenes in Fig. 5 These 6 scenes are chosen among the scenes for which DewarpNet achieves better quantitative result than the proposed approach for at least one view. In Fig. 5 our results are clearly better than the DewarpNet in all cases, with straighter lines and better rectified structure. The evaluation scores do not accurately reflect the improvement due to the sensitivity of LD and MSSIM to subtle perceptually unimportant global transformations, such translation of the image by few pixels. However, such transformations do not affect the visual quality or readability of the unwarped results. More discussion and qualitative comparison is available in supplementary material.
7

---

Under review as a conference paper at ICLR 2022
AMONELY LRANUM
SEATER COMPTIN)
METIEST BANTUM
TRADING
TRADING
TRADING
N (c)
a
Figure 4: Qualitative comparison with DewarpNet (Das et al. 2019):
: (a) Input image, (b) Dewarp- Net unwarping, (c) proposed unwarping, (d) GT scanned image, (e) enlarged regions: DewarpNet (top), and proposed (bottom). We use reasonable frontal view of the document for a fair comparison.
DewarpNet
Proposed
ED +
CER (std)
WER (std) 4
ED 4
CER (std)
WER (std) 4
Mean 798.30
0.2827 (0.12) 0.4646 (0.17)
600.78
0.2122 (0.10) 0.3568 (0.11)
Table 2: Comparison of OCR error metrics: We improve the OCR performance of Das et al. (2019)
by 25% in terms of Edit Distance (ED), Character Error Rate (CER), and Word Error Rate (WER).
The quantitative comparison for real scenes reported in Table I (bottom). We achieve better results in terms mean and best evaluation score than DewarpNet in 2 out scenes. We notice little worse for the real scenes than synthetic scenes due to the
that the evaluation results fewer available views (5-10 compared to 50). Moreover, there are cases like Real 6, which do not have sufficient texture. Such data reconstruct the 3D shape. As result of the poor 3D shape, our texture parameterization network produces inferior unwarping result (More details available in Supplementary). We also report qualitative comparisons with You et al. (0017) and Das et al. (2019) on additional real documents in supplementary.
failure case of IDR since there insufficient information to
We also evaluated the OCR performance on real scenes across 77 images in Table We use Edit Distance (ED) (Miller et al. 2009). Character Error Rate (CER) and Word Error Rate (WER) as our evaluation metrics. ED is defined as the total number of substitutions (s), insertions (i) and deletions (d) required to obtain the reference text, given the recognized text. The reference text is obtained by running the OCR algorithm on the scanned ground-truth image of each document. CER is defined as: (s + + d)/N where N the number of characters the reference text. We use Tesseract 4.1.1 based LSTM OCR engine for this experiment. Our unwarped results reduce the ED, CER and WER by 25%. This improvement proves our unwarped results are more suitable for downstream applications tasks like OCR.
OCR Evaluation.
8

---

Under review as a conference paper at ICLR 2022
Mild Hybrn
Mild Hyyrnd
"Pad
te Ferrari 812
Fe Ferrari 8 812 Superfase
Superfas fast
(8)
(b)
(d)
(c)
Figure 5: Comparison of DewarpNet (a,c.e) with the proposed unwarped result (b,d.f) for the view that yields the best LD with DewarpNet. Proposed results are clearly better, however this improve- ment is not captured by LD. Follow the blue dashed boxes for discrimitative regions.
Texture Editing. In addition to document unwarping. our proposed forward and backward MLP can also be used for high quality texture editing. We show two texture editing examples in Fig, We use the backward MLP to unwarp the texture from the input image, then we edit the texture and warp it back to image space using the learned forward MLP. (More details in Supplementary).
Ablation Study. ablate how loss terms T.z, Lx, Lr, and La affect the unwarping results. We train F.v and F, with different combinations of these loss terms and report the mean MSSIM and LD in Table 3 (appendix). Qualitative results for one scene are shown in Fig. 6 (appendix).
5 TRAINING TIME, GENERALIZABILITT AND FUTURE WORK
Our proposed method for scene can be trained in approximately 18 hours for 448 x 448 resolution images using a single Titan Xp GPU. The current training time per scene is very high compared to DewarpNet's inference time which makes it unsuitable for real time applications. However, this is fast growing field and there are multiple other works that focusing on improving the speed and generalization abilities (Garbin et al. 2021 Bergman et al. (021) of neural rendering. Therefore, obtaining a faster training scheme is considered as a future work.
Our method can be applied to fabrics, which are very similar to papers and lead to practical appli- cations of texture editing. However, none of the current 3D garment/fabric datasets (Patel et al.) 2020) can be easily adapted to train the Fuv prior. For more complex UV spaces (e.g.. texture atlas), learning the prior may require decomposing the shape multiple simple UV maps. The proper way to do this is beyond the scope this paper, however we believe it's an exciting future work. As importantly, in this paper, have introduced number of domain specific strong constraints that suit the rectangular paper shape. These constraints improve empirical results. More general objects will require different constraints e.g., spherical UV domain, local scaling of the conformal map etc.
CONCLUSIONS
We have introduced an end-to-end trainable architecture that can simultancously learn texture pa- rameterized 3D shapes from multi-view images. This is the first work to learn surface parameteri- zation an implicit neural representation to the best of our knowledge. We have demonstrated the applicability of our approach multiple synthetic and real scenes for the task of document unwarp- ing and document texture editing. We want to extend this method to learn surface parameterization for complex shapes such faces general 3D objects in future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Adrien Bartoli, Yan Gerard, Francois Chadebecq, Toby Collins, and Daniel Pizarro. Shape-from template. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10):2099-21 2118, 2015.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Yun Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics (TOG), 38(4), 2019.
Jan Bednarik, Shaifali Parashar, Erhan Gundogdu, Mathieu Salzmann, and Pascal Fua. Shape re- construction by learning differentiable surface representations. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, 2020.
Alexander W. Bergman, Petr Kellnhofer, and Gordon Wetzstein. Fast training of neural lumigraph representations using meta learning, 2021.
Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. Patch-based optimization for image-based texture mapping. ACM Transactions on Graphics (TOG), 36(4):106-1, 2017.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Pro- ceedings of the International Conference on Computer Vision, 2019.
Tony Chan and Wei Zhu. Level set based shape prior segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2005.
Anpei Chen, Zhang Chen, Guli Zhang, Kenny Mitchell, and Jingyi Yu. Photo-realistic facial details synthesis from single image. In Proceedings of the International Conference Computer Vision, 2019.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. arXiv preprint arX5v:2103.15595, 2021.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision, pp. 628 644. Springer, 2016.
Sagnik Das, Gaurav Mishra, Akshay Sudharshana, and Roy Shilkrot. The Common Fold: Utilizing the Four-Fold to Dewarp Printed Documents from Single Image. In Proceedings of the 2017 ACM Symposium on Document Engineering, DocEng '17, pp. 125-128, 2017. ISBN 978-1-4503- 4689-4. doi: 10.1 145/3103000031222
Sagnik Das, Ke Ma, Zhixin Shu, Dimitris Samaras, and Roy Shilkrot. DewarpNet: Single-image document unwarping with stacked 3D and 2D regression networks. In Proceedings of the Inter- national Conference on Computer Vision, 2019.
Sagnik Das, Kunwar Yashraj Singh, Jon Wa, Erhan Bas, Vijay Mahadevan, Rahul Bhotika, and Dimitris Samaras. End-to-end piece-wise unwarping of document images. In Proceedings of the IEEE/CVF International Conference Computer Vision pp.
2021.
Jiankang Deng. Shiyang Cheng. Niannan Xue, Yuxiang Zhou, and Stefanos Zafeiriou. Uv-gan:
Adversarial facial uv map completion for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. 2021.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regu- larization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
10

---

Under review as a conference paper at ICLR 2022
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-måché approach to learning 3d surface generation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
Steven Haker, Sigurd Angenent, Allen Tannenbaum, Ron Kikinis, Guillermo Sapiro, and Michael Halle. Conformal surface parameterization for texture mapping. IEEE Trunsactions on Visualiza- tion and Computer Graphics, 6(2):181-188) 2000.
John C Hart. Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces. The Visual Computer, 12(10))527-555, 1996.
Peter Hedman, Julien Philip. True Price, Jan Mun-Micheel Frahm, George Drettakis, and Gabriel Bros- tow. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics
(TOG), 37(6):1-15, 2018.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Taeho Kil, Wonkyo Seo, Hyung Il Koo, and Nam Ik Cho. Robust Document Image Dewarping Method Using Text-Lines and Line Segments. In Proceedings of the International Conference on Document Analysis and Recognition, pp. 865-870. IEEE, 2017.
Hyung II Koo, Jinho Kim, and Nam IK Cho. Composition of a dewarped and enhanced document image from two view images. IEEE Transactions on Image Processing, 18(7)1551-1166, 2009,
Tzu-Mao Li, Miik Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG), 1-11, 2018.
Xiaoyu Li, Bo Zhang, Jing Liao, and Pedro V. Sander. Document Rectification and Illumination Correction using Patch-based CNN. ACM Transactions on Graphics (TOG), 2019.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. arXiv preprint 1rX1 2020.
Jian Liang, Daniel DeMenthon, and David Doermann. Geometric rectification of camera-captured document images. IEEE Transactions on Puttern Analysis and Machine Intelligence, 30(4):591- 605, 2008
Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5))978-994 2011.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the International Conference on Computer Vision, 2019a.
Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning infer implicit surfaces without 3d supervision. arXiv preprint arXiv:1911.00766 2019b.
Ke Ma, Zhixin Shu, Xue Bai, Jue Wang, and Dimitris Samaras. DocUNet: Document Image Un- warping via A Stacked U-Net. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
Amir Markovitz, Inbal Lavi, Or Perel, Shai Mazor, and Roee Litman. Can you read me now? Content aware rectification using angle supervision. In Proceedings of the European Conference on Computer Vision. Springer, 2020.
Abhimitra Meka, Christian Haene, Rohit Pandey, Michael Zollhöfer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu, Jay Busch, Jason Dourgarian, al. Deep reflectance fields: High- quality facial reflectance field inference from color gradient illumination. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Gaofeng Meng, Zuming Huang, Yonghong Song, Shiming Xiang, and Chunhong Pan. Extraction of virtual baselines from distorted document images using curvilinear projection. In Proceedings of the International Conference Computer Vision, 2015.
11

---

Under review as a conference paper at ICLR 2022
Gaofeng Meng, Yuanqi Su, Ying Wu, Shiming Xiang, and Chunhong Pan. Exploiting Vector Fields for Geometric Rectification of Distorted Document Images. In Proceedings of the European Conference Computer Vision, 2018.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision, 2020.
Frederic P. Miller, Agnes Vandome, and John McBrewster. Levenshtein Distance: Information Theory, Computer Science, String (Computer Science), String Metric, Damerau? Levenshtein Dis- tance, Spell Checker, Hamming Distance. Alpha Press, 2009, ISBN 6130216904.
Aymen Mir, Thiemo Alldieck, and Gerard Pons-Moll. Learning transfer texture from clothing images to 3d humans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Luca Morreale, Noam Aigerman, Vladimir Kim, and Niloy J. Mitra. Neural surface maps. 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019a.
Taesung Park, Ming-Yi Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with ppatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019b.
Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. Tailoret: Predicting clothing in 3d as a function of human pose. shape and garment style. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR). IEEE, jun 2020.
Michela Petrini, Gianfranco Pradisi, and Alberto Zaffaroni. Guide To Mathematical Methods For Physicists, Advanced Topics And Applications. World Scientific, 2018
Albert Pumarola, Antonio Agudo, Lorenzo Porzi, Alberto Sanfeliu, Vincent Lepetit, and Francesc Moreno-Noguer. Geometry-Aware Network for Non-Rigid Shape Prediction from Single View. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc -oouno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint arXh::2011.13961 2020.
Johannes Lutz Schönberger and Jan-Michael Frahm. Stracturefrrom-motion revisited. In Confer- Computer Vision and Pattern Recognition (CVPR), 2016.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems, 2020.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, 2019a.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awart neural scene representations. arXiv preprint 191.019906.0118,, 2019b.
12

---

Under review as a conference paper at ICLR 2022
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. In arXiv, 2020.
Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. arXiv preprint
arXiv:1806. 1806.04877 2018.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pp. 701-727. Wiley Online Library, 2020.
Yuandong Tian and Srinivasa G Narasimhan. Rectification and 3D reconstruction of curved docu- ment images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, 2011.
Yochay Tzur and Ayellet Tal. FlexiStickers: Photogrammetric texture mapping using casual images. In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, 2009.
Adrian Ulges, Christoph H. Lampert, and Thomas Breuel. Document Capture Using Stereo Vision. In Proceedings of the 2004 ACM Symposium on Document Engineering, DocEng '04, pp. 198- 200, 2004. ISBN 1-58113-955 139111--
doi: 10.1145/1030097 1030434.
Toshikazu Wada, Hiroyuki Ukida, and Takashi Matsuyama, Shape from shading with interreftec- tions under proximal light source: Distortion-free free copying of an unfolded book. International Journal of Computer Vision, 24(2):125-135, 1997.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 52-67, 2018.
Zhou Wang, Ecro Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thirty-Seentth Asilomar Conference Signals, Systems and Computers, 2003.
Shih-En Wei, Jason Saragih, Tomas Simon, Adam W Harley, Stephen Lombardi, Michal Perdoch, Alexander Hypes, Dawei Wang, Hernan Badino, and Yaser Sheikh. Vr facial animation via mul- tiview image translation. ACM Transactions on Graphics (TOG), 38(4):1-16, 2019.
Fanbo Xiang, Zexiang Xu, Miloš Hašan, Yannick Hold Hold-Geoffroy, Kalyan Sunkavalli, and Hao Su. NeuTex: Neural texture mapping for volumetric neural rendering. arXiv preprint arXiv:2103.00762, 2021.
Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image- based relighting from optimal sparse samples. ACM Transactions on Graphics (TOG), 37(4):1-13, 2018
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances Neural Information Processing Systems, 33, 2020.
Shaodi You, Yasuyuki Matsushita, Sudipta Sinha, Yusuke Bou, and Katsushi Ikeuchi. Multiview Rectification of Folded Documents. IEEE Transactions Pattern Analysis and Machine Intelli- gence, 2017.
2017
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one few images. arXiv preprint arXiv:2012.02190, 2020.
Fang Zhao, Shengcai Liao, Kaihao Zhang, and Ling Shao. Human parsing based texture transfer from single image to 3D human via cross-view consistency. In Advances in Neural Information Processing Systems, 2020.
13

---

Under review as a conference paper at ICLR 2022
Mean
b+w
b+c
b+w+c
LD 11.2
10.50
6.24
3.63
MSSIM
0.4632
0.4622
0.5556
0.6888
Table 3: Weighted Lg, and conformality effects: for the model trained without conformality constraints and with wp 1; w is for weighted L, and is for the use of conformality constraints.
ETHICS STATEMENT
Texture editing application of our proposed approach have both positive and negative societal impact. On the positive side, real document images can be gracefully redacted to protect sensitive information. On the contrary, it can be potentially used for editing real documents and change the content to commit fraud and spread misinformation.
8 REPRODUCIBILITY STATEMENT
We believe our results are reproducible by following the training details Section 3.4 main submission and Section 2, 3, 4, 5 from the supplementary material.
A APPENDIX
A.1 ABLATION STUDY
We ablate how loss terms L,, Lg. Lr, and LG affect the unwarping results. We train Fav and F. with different combinations of these loss terms and report the mean MSSIM and LD in Table 3 Qualitative results for one scene are shown in Fig. (appendix. In the basic version (listed as b), no conformality constraints (LE, LF and La) are used and wp in Lz. The b+ version introduces weighting function that assigns higher value to up if a pixel is closer to the document boundary. Introducing this loss improves the boundary; notice the white margin at the top and bottom in the second column of Fig. 6 Introducing the conformality constraints alleviates the unusual stretch in the texture and improves smoothness (Fig. 6 col. 3). Using both improves the boundary and the texture smoothness (Fig. [6] col. 4) and achieves the best result.
b (11.01)
b (9.83)
bbc (6.59)
b+w+c (2.71)
GT
Figure 6: Illustration of weighted L., and conformality effects: is for the model trained without conformality constraints and with wp 1; w is for weighted L, and is for the of conformality constraints. GT is the ground-truth scan. The number in parenthesis denote the respective LD values.
14


Reviews:
[{"review_id": "uP4Ivd4aYfR", "reviewer": "Reviewer_Cc2n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method seems to be primarily based on IDR with following similarities:\n- 3D surface is represented by a Signed Distance Field (SDF)\n- Appearance is learned from a neural network that maps 3D surface geometry (location and normal) and viewing angle to radiance values\n- The networks are trained with an overall loss including an RGBA rendering term and an Eikonal term for regularising SDF\n\nIDR can handle specular materials but at the cost of introducing so-called 'shape-radiance' ambiguities. This means geometry cannot be possibly recovered due to view-dependency and is only regularised by neural nets' inherent priors (c.f. NeRF++). In this paper though, it seems document pages are all diffuse and I think it would make more sense to drop the view direction term $\\mathbf{v}_p$ from Eq. 3. This would disambiguate geometry and could improve robustness.\n\nThe authors propose a forward-backward network to learn the square parameterisation of surface with a conformal loss on the forward network. While this design encourages bijective mapping through minimising $L_z$, there is no guarantee the resulting networks are truly bijective. Why not use structures that guarantee invertibility (e.g. Invertible Residual Networks, Behrmann et al.)?  \n\nIn paragraph \"Deformation constraints for Fz\" the author wrote $\\mathbf{J}^\\top\\mathbf{J}=\\beta I$. Shouldn't this be $\\mathbf{J}^\\top\\mathbf{J}=diag(\\alpha,\\beta)$ since you don't know the aspect ratio of document (which is what authors are actually doing with $L_E$,$L_G$)?\n\nThe results show clear improvments over existing methods and look very promising. Though personally I'm not familiar with the specific topic of document unfolding or methods that are compared against. \n\n\n\n", "summary_of_the_paper": "The paper presents a multi-view method for undistorting documents. The method makes use of an existing method (IDR) for reconstructing distorted documents in 3-D, with an additional bijective neural network to parameterize distorted surfaces by rectangles. The method significantly outperforms SOTAs in the experiments.\n\n", "summary_of_the_review": "I like the overall idea of this paper and the results look really convincing. However I would not recommend acceptance to ICLR given its significant overlap with an existing technique (IDR).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XZIJqFJWbk1", "reviewer": "Reviewer_K4gj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- The paper is well structured and clearly written which makes it easy to follow and reproduce the method.\n- The authors creatively combine ideas from different domains, namely the implicit field representation with a volumetric renderer with the parametric mapping functions and distortion regularizers which allow them to model a real physically valid paper material behaviour.\n- The authors experiment on multiple tasks including dewarping, texture editing and OCR and the qualitative results look compelling.\n- The Fig. 2 is useful for understanding the pipeline.\n- Good ablation of the loss term L_T both quantitatively and qualitatively.\n\nWeaknesses\n\n- Table 1. - It is not clear why the authors did not include the \"best view\" setting of the proposed method too, it would be a more fair comparison to the \"best view\" of DewarpNet, as now both methods would be able to use the oracle. Was there a reason for not including this comparison?\n- There exist more recent document rectification works which are as recent as DewarpNet or even newer, specificially [1] and [2] (see below), both seem to provide code publicly. Is there a reason why the authors excluded these works for comparison?\n- The authors claim that LD and MSSIM are sensitive to perceptually unimportant global transformations such as global translation. Could the authors then first align the GT and the unwarped image by optimizng for the best translation and then evaluate the LD and MSSIM to show that the drop in precision is indeed due to the transformations?\n- In Eq. 8, it is not clear how M_p is computed and the description could be more detailed.\n- The last paragraph of  page 8: What is \"Data 6\", does it correspond to the line \"Real6\" in the Table 1?\n\nMinor problems:\n- Fig. 3 is first referenced from the 5th paragraph of the Intro, but at this point the variables used in the caption are not defined and it is hard to interpret the figure. I would suggest either referencing the figure later from section 3.2, or giving more comprehensive self-contained caption.\n- Section 3.4 Training Details: In the last sentence, the authors probably meant \\delta_2, \\delta_3, \\delta_4 instead of \\delta_1, \\delta_2, \\delta_3.\n- Typos:\n\t- Section 3.2, paragraph Deformation constraints for F_z: \"conformal map allow\" -> \"conformal map allows\"\n\t- Fig. 5: \"best\" -> \"the best\"\n\nA question for the authors:\nIs L_{uv} of Eq. 10 really needed? The loss terms L_E, L_F and L_G should exactly such behaviour where the mapping does not introduce strong local distortions. The authors explain that if L_{uv} is not employed, the effective learned UV space does not span the full rectangle [0, 1]^{2}. However, at the same time, the authors show that the distortion regularizers L_E, L_F, L_G are only turned on later during the training. Could the authors comment on what happens if the the terms L_E, L_F, L_G are turned on from the beginning and the term L_{uv} is not used at all?\n\nReferences\n[1] X. Li et al. Document Rectification and Illumination Correction using Patch-based CNN. TOG 2019.\n[2] X. Liu et al. Geometric rectification of document images using adversarial gated unwarping network. Pattern Recognition 2020.", "summary_of_the_paper": "The authors present a method to jointly learn a shape and UV parameterization of a creased document observed from multiple views. They use an SDF based implicit shape representation in combination with a recent neural volumetric renderer IDR, and optimize the energy composed of the main photometric term (a discrepancy between the observed and rendered image) and regularization terms enforcing well behaved UV space and a fixed scale conformal mapping which corresponds to a behaviour of a real paper material. Since the implicit field representation does not explicitly model a surface, the authors cannot directly optimize the 3D->2D mapping using standard graphics techniques and rather adapt the IDR renderer to be conditioned on the locations sampled from the UV space and train it jointly with the MLPs defined for the bijective mapping 2D->3D and 3D->2D. The authors show that they can get better results both quantitatively and qualitatively when compared to one of the SotA works DewarpNet.", "summary_of_the_review": "The paper introduces a creative way to combine an implicit shape representation with parametric functions and a neural volumetric renderer to deal with document unwarping. The authors further employ deformation regularizers which reflect the real behaviour of a material on which the documents are usually printed. The paper is written in a clear way, I believe it should be easy to reproduce, and the qualitative results seem compelling. I see a couple minor problems, namely (i) exclusion of two seemingly relevant approaches for comparison (see above), (ii) a little confusion about the column \"best view\" in the table not being used for the proposed method too and (iii) unsupported claim about the sensitivity of LD and MSSIM and I would like the authors to comment on these. In general, I find this submission to be compelling overall and upon clarification of some of the issues mentioned above I lean towards acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RCoLQFtyseC", "reviewer": "Reviewer_hfPz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \nThis paper seems to be the first paper where implicit differentiable rendering is applied to document unwrapping. \n\nThe proposed method is reported to be par with the baseline method (DewrapNet) in quantitative comparisons and outperformed the baseline in qualitative comparisons. \n\nWeakness:\nThe proposed method is similar to NERF, where MLPs are trained for each subject. However, DewrapNet is trained once and prediction does not include training. Thus, I am not sure the comparisons are fair. At least, I think the performance issues should be discussed.  \n\nBased on the above reason, the practicality of the proposed method is not clear.", "summary_of_the_paper": "This paper presents document unwrapping based on optimization of neural networks. Four MLPs are defined to model the implicit surface of the target, the colors of the target, the mapping function from 3D points to 2D texture coordinates, and the inverse mapping from 2D to 3D. Similarly to NERF, the target example is approximated by training the MLPs using differentiable rendering. \n\n", "summary_of_the_review": "I think this paper is interesting for an experimental study, where implicit differential rendering is applied for document unwrapping. Although I doubt the practical value of the method. \n\nAs an experimental study, the supplemental material seems to include much information about the study that is missing in the paper itself. My concern is that the paper deeply depends on the supplemental document. \n\nOverall, I would like to positively evaluate the trial of implicit differential rendering for document unwrapping, and showing detained results in supplemental document.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FaCl-TpmqwA", "reviewer": "Reviewer_QH85", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Advancing IDR**\n\nAlthough the IDR technique that this paper builds upon is able to render textures, these textures must be learned via a MLP trained for appearance information, which is not sufficient to enable texture editing after the fact. In other words, while textures can be rendered an implicit surface via IDR, no explicit surface parameterization is created, so the appearance is fixed. In contrast, the propose method learns the surface parameterization explicitly, enabling the original texture of the implicit surface to be both rendered and subsequently edited. Enabling this subsequent manipulation is a key contribution of this work that suggests real opportunity for further development.\n\n**Simplified Problem**\n\nThis paper shows the proposed approach to work quite well for document unwarping. However, document unwarping provides a fairly simplified version of the more general problem of learning explicit texture maps for implicit surfaces.\n\n1. The implicit surface in question (a piece of paper) does not need to be stretched to be unwrapped, so the problem of distortion in the texture space is significantly reduced. Furthermore, no unwarping needs to occur as the texture space actually defines the unwarped document exactly.\n2. The implicit surface is already a topological disc, which circumvents the problem of defining the UV space on a more complex surface\n\nWhile the paper is forthright about these simplifying circumstances, it does somewhat limit the evaluation of the proposed approach in general. It would be helpful for the authors to articulate how this method would generalize beyond documents unwarping and what potential pitfalls may exist in trying to do so.\n\n**Experimental Evaluation**\n\n*Quantitative Results*\n\nThe auxiliary metric of OCR performance is apt for document unwarping, and the improvements here are pretty clear (Table 1, supplementary). However, the improvements in Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) highlighted in Table 1 are not quite as obvious. It seems that while the proposed method outperforms DewarpNet on average, the best view from DewarpNet is  still a little better. Table 1 is missing best view statistics for the proposed method, making it difficult to judge what to make of this evaluation. Additionally, it is not entirely clear what to make of an improvement of 0.05 MS-SSIM. As this is a fairly perceptual measure, it would be helpful for the paper to describe what this improvement means in terms of actual perceptual difference.\n\nRuntime improvement should not be considered \"out-of-scope of this paper\" (Sec 12, supplementary). This is especially true when the 18-hour training time per scene is significantly higher than the inference time for the primary comparison work, DewarpNet (Sec 14, supplementary). This evaluation belongs in the main paper and needs to be discussed. It is well known that neural volume rendering approaches like IDR are quite slow, and their study is still nascent. However, delegation to the supplementary without any reference in the main paper comes across as evasive. This discussion ought to be present in the main paper to provide a complete and fair evaluation of the proposed technique.\n\n*Qualitative Results*\n\nThe qualitative results highlighted in Figure 4 are not particularly discriminative. For example, the DewarpNet result shown in the top row, column (e) seems noticeably sharper than the corresponding result of the proposed approach. Similarly, in the second row, the proposed approach (column (c)) seems to have a less distorted unwarping than DewarpNet (column (b)), but the effects seem pretty marginal in the close up comparison given in column (e). Figure 5 does a better job showing where the proposed approach outperforms DewarpNet.\n\n**Other Notes**\n\n* The OCR evaluation (Sec 10, supplementary) is very relevant for the document unwarping task and is the quantitative result that best distinguishes the proposed approach. The authors ought to consider moving it to the main paper.\n* Given the tightly constrained problem, the discussion of limitations (Sec 12, supplementary) and failure cases (Sec 13, supplementary) ought to be part of the main paper.\n", "summary_of_the_paper": "**High-Level Overview**\n\nThis paper proposes a neural rendering technique to learn how to estimate an implicit surface and its UV-parameterization from a set of 2D images. The proposed approach is applied to the problem of document unwarping, wherein a wrinkled, folded, or bent document (e.g. a piece of paper with text on it) can be digitally flattened with its contents preserved. For good measure, the paper also shows that learning the texture mapping enables the contents of the unwarped document to be edited in the texture space. The proposed approach is evaluated against a prior work, DewarpNet (Das et al., 2019) on the basis of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) as well as the auxiliary task of OCR.\n\n**Key Contributions**\n\n* Builds on IDR (Yariv et al., 2020) to learn an explicit bijective texture mapping for a learned implicit surface\n* A loss function formulation for the framework\n* Application of learned implicit surfaces for document unwarping with reduction in texture distortion and improvements in subsequent OCR metrics\n\n**How It Works**\n\n* Starts with the same problem formulation as IDR: an MLP to model the surface (Eq. 1) and an appearance MLP to model the color (Eq. 2)\n* Modifies the appearance MLP to also be a function of texture coordinates (Eq. 3)\n* Adds an additional forward and backward MLP to learn the bijective UV mapping\n* Jointly trains all MLPs using a pixelwise rendering loss", "summary_of_the_review": "This paper does a good job demonstrating the applicability of the proposed neural rendering technique to the problem of document unwarping, both in the formulation of the problem (learning the UV grid *is* unwarping the document) as well as in the empirical evaluation (OCR). Design decisions are well-motivated and clearly articulated, and the merits of the proposed technique are backed up by relevant experiments. However, the improvements over prior work seem marginal at best in the key metrics of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD). Furthermore, the qualitative results do not make the improvements especially obvious. Lastly, the 18 hour training time per scene (Sec 12, supplementary) is pretty prohibitive for the document unwarping application explored by this paper. Given these caveats, it is hard to see how substantially this work might impact the research community.\n\nAdditionally, although document unwarping is an interesting application of this neural rendering technique, it is also very niche. Given the potential this paper suggests for learning editable texture-mapped surfaces, it would have been nice to see a more general application (e.g. 3D shapes, objects, or structures) flirted with, even if only as a teaser experiment or discussion of future direction. Despite this wish, it would be inappropriate to knock the submission for not exploring every possible application. However, it does leave an open question as to whether the proposed approach is more generally applicable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an architecture for learned surface parameterization, with application to image unwarping, which can be coupled with differentiable rendering, multi-view data, and other modern objective terms.  The shape of the document is parameterized using an SDF technique, coupled with neural rendering and objective terms inspired by classical geometry processing.  This machinery is quite "heavy," leading to slow training times.

As pointed out by reviewer QH85, there were some experimental discrepancies---rightfully acknowledged by the authors---which make comparisons to DewarpNet less favorable for the new method, at least from a quantitative perspective.  Visual inspection makes the comparison more favorable, although it would be preferable for the quantitative quality metrics and qualitative examples to align.  

Runtime measurements here are also not favorable and severely limit applicability of this technique in real-world scenarios, as pointed out by reviewers hfPz and QH85.

While the mistaken quantitative results are forgivable, the AC agrees that the scope of this work is quite narrow; it is not clear where this architecture would be applied relative to the motivating application.
Decision: Reject