Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method to jointly learn a shape and UV parameterization of a creased document observed from multiple views. They use an SDF based implicit shape representation in combination with a recent neural volumetric renderer IDR, and optimize the energy composed of the main photometric term (a discrepancy between the observed and rendered image) and regularization terms enforcing well behaved UV space and a fixed scale conformal mapping which corresponds to a behaviour of a real paper material. Since the implicit field representation does not explicitly model a surface, the authors cannot directly optimize the 3D->2D mapping using standard graphics techniques and rather adapt the IDR renderer to be conditioned on the locations sampled from the UV space and train it jointly with the MLPs defined for the bijective mapping 2D->3D and 3D->2D. The authors show that they can get better results both quantitatively and qualitatively when compared to one of the SotA works DewarpNet.
main_review: Strengths

- The paper is well structured and clearly written which makes it easy to follow and reproduce the method.
- The authors creatively combine ideas from different domains, namely the implicit field representation with a volumetric renderer with the parametric mapping functions and distortion regularizers which allow them to model a real physically valid paper material behaviour.
- The authors experiment on multiple tasks including dewarping, texture editing and OCR and the qualitative results look compelling.
- The Fig. 2 is useful for understanding the pipeline.
- Good ablation of the loss term L_T both quantitatively and qualitatively.

Weaknesses

- Table 1. - It is not clear why the authors did not include the "best view" setting of the proposed method too, it would be a more fair comparison to the "best view" of DewarpNet, as now both methods would be able to use the oracle. Was there a reason for not including this comparison?
- There exist more recent document rectification works which are as recent as DewarpNet or even newer, specificially [1] and [2] (see below), both seem to provide code publicly. Is there a reason why the authors excluded these works for comparison?
- The authors claim that LD and MSSIM are sensitive to perceptually unimportant global transformations such as global translation. Could the authors then first align the GT and the unwarped image by optimizng for the best translation and then evaluate the LD and MSSIM to show that the drop in precision is indeed due to the transformations?
- In Eq. 8, it is not clear how M_p is computed and the description could be more detailed.
- The last paragraph of  page 8: What is "Data 6", does it correspond to the line "Real6" in the Table 1?

Minor problems:
- Fig. 3 is first referenced from the 5th paragraph of the Intro, but at this point the variables used in the caption are not defined and it is hard to interpret the figure. I would suggest either referencing the figure later from section 3.2, or giving more comprehensive self-contained caption.
- Section 3.4 Training Details: In the last sentence, the authors probably meant \delta_2, \delta_3, \delta_4 instead of \delta_1, \delta_2, \delta_3.
- Typos:
	- Section 3.2, paragraph Deformation constraints for F_z: "conformal map allow" -> "conformal map allows"
	- Fig. 5: "best" -> "the best"

A question for the authors:
Is L_{uv} of Eq. 10 really needed? The loss terms L_E, L_F and L_G should exactly such behaviour where the mapping does not introduce strong local distortions. The authors explain that if L_{uv} is not employed, the effective learned UV space does not span the full rectangle [0, 1]^{2}. However, at the same time, the authors show that the distortion regularizers L_E, L_F, L_G are only turned on later during the training. Could the authors comment on what happens if the the terms L_E, L_F, L_G are turned on from the beginning and the term L_{uv} is not used at all?

References
[1] X. Li et al. Document Rectification and Illumination Correction using Patch-based CNN. TOG 2019.
[2] X. Liu et al. Geometric rectification of document images using adversarial gated unwarping network. Pattern Recognition 2020.
summary_of_the_review: The paper introduces a creative way to combine an implicit shape representation with parametric functions and a neural volumetric renderer to deal with document unwarping. The authors further employ deformation regularizers which reflect the real behaviour of a material on which the documents are usually printed. The paper is written in a clear way, I believe it should be easy to reproduce, and the qualitative results seem compelling. I see a couple minor problems, namely (i) exclusion of two seemingly relevant approaches for comparison (see above), (ii) a little confusion about the column "best view" in the table not being used for the proposed method too and (iii) unsupported claim about the sensitivity of LD and MSSIM and I would like the authors to comment on these. In general, I find this submission to be compelling overall and upon clarification of some of the issues mentioned above I lean towards acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: **High-Level Overview**

This paper proposes a neural rendering technique to learn how to estimate an implicit surface and its UV-parameterization from a set of 2D images. The proposed approach is applied to the problem of document unwarping, wherein a wrinkled, folded, or bent document (e.g. a piece of paper with text on it) can be digitally flattened with its contents preserved. For good measure, the paper also shows that learning the texture mapping enables the contents of the unwarped document to be edited in the texture space. The proposed approach is evaluated against a prior work, DewarpNet (Das et al., 2019) on the basis of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) as well as the auxiliary task of OCR.

**Key Contributions**

* Builds on IDR (Yariv et al., 2020) to learn an explicit bijective texture mapping for a learned implicit surface
* A loss function formulation for the framework
* Application of learned implicit surfaces for document unwarping with reduction in texture distortion and improvements in subsequent OCR metrics

**How It Works**

* Starts with the same problem formulation as IDR: an MLP to model the surface (Eq. 1) and an appearance MLP to model the color (Eq. 2)
* Modifies the appearance MLP to also be a function of texture coordinates (Eq. 3)
* Adds an additional forward and backward MLP to learn the bijective UV mapping
* Jointly trains all MLPs using a pixelwise rendering loss
main_review: **Advancing IDR**

Although the IDR technique that this paper builds upon is able to render textures, these textures must be learned via a MLP trained for appearance information, which is not sufficient to enable texture editing after the fact. In other words, while textures can be rendered an implicit surface via IDR, no explicit surface parameterization is created, so the appearance is fixed. In contrast, the propose method learns the surface parameterization explicitly, enabling the original texture of the implicit surface to be both rendered and subsequently edited. Enabling this subsequent manipulation is a key contribution of this work that suggests real opportunity for further development.

**Simplified Problem**

This paper shows the proposed approach to work quite well for document unwarping. However, document unwarping provides a fairly simplified version of the more general problem of learning explicit texture maps for implicit surfaces.

1. The implicit surface in question (a piece of paper) does not need to be stretched to be unwrapped, so the problem of distortion in the texture space is significantly reduced. Furthermore, no unwarping needs to occur as the texture space actually defines the unwarped document exactly.
2. The implicit surface is already a topological disc, which circumvents the problem of defining the UV space on a more complex surface

While the paper is forthright about these simplifying circumstances, it does somewhat limit the evaluation of the proposed approach in general. It would be helpful for the authors to articulate how this method would generalize beyond documents unwarping and what potential pitfalls may exist in trying to do so.

**Experimental Evaluation**

*Quantitative Results*

The auxiliary metric of OCR performance is apt for document unwarping, and the improvements here are pretty clear (Table 1, supplementary). However, the improvements in Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD) highlighted in Table 1 are not quite as obvious. It seems that while the proposed method outperforms DewarpNet on average, the best view from DewarpNet is  still a little better. Table 1 is missing best view statistics for the proposed method, making it difficult to judge what to make of this evaluation. Additionally, it is not entirely clear what to make of an improvement of 0.05 MS-SSIM. As this is a fairly perceptual measure, it would be helpful for the paper to describe what this improvement means in terms of actual perceptual difference.

Runtime improvement should not be considered "out-of-scope of this paper" (Sec 12, supplementary). This is especially true when the 18-hour training time per scene is significantly higher than the inference time for the primary comparison work, DewarpNet (Sec 14, supplementary). This evaluation belongs in the main paper and needs to be discussed. It is well known that neural volume rendering approaches like IDR are quite slow, and their study is still nascent. However, delegation to the supplementary without any reference in the main paper comes across as evasive. This discussion ought to be present in the main paper to provide a complete and fair evaluation of the proposed technique.

*Qualitative Results*

The qualitative results highlighted in Figure 4 are not particularly discriminative. For example, the DewarpNet result shown in the top row, column (e) seems noticeably sharper than the corresponding result of the proposed approach. Similarly, in the second row, the proposed approach (column (c)) seems to have a less distorted unwarping than DewarpNet (column (b)), but the effects seem pretty marginal in the close up comparison given in column (e). Figure 5 does a better job showing where the proposed approach outperforms DewarpNet.

**Other Notes**

* The OCR evaluation (Sec 10, supplementary) is very relevant for the document unwarping task and is the quantitative result that best distinguishes the proposed approach. The authors ought to consider moving it to the main paper.
* Given the tightly constrained problem, the discussion of limitations (Sec 12, supplementary) and failure cases (Sec 13, supplementary) ought to be part of the main paper.

summary_of_the_review: This paper does a good job demonstrating the applicability of the proposed neural rendering technique to the problem of document unwarping, both in the formulation of the problem (learning the UV grid *is* unwarping the document) as well as in the empirical evaluation (OCR). Design decisions are well-motivated and clearly articulated, and the merits of the proposed technique are backed up by relevant experiments. However, the improvements over prior work seem marginal at best in the key metrics of Multi-Scale Structural Similarity (MS-SSIM) and Local Distortion (LD). Furthermore, the qualitative results do not make the improvements especially obvious. Lastly, the 18 hour training time per scene (Sec 12, supplementary) is pretty prohibitive for the document unwarping application explored by this paper. Given these caveats, it is hard to see how substantially this work might impact the research community.

Additionally, although document unwarping is an interesting application of this neural rendering technique, it is also very niche. Given the potential this paper suggests for learning editable texture-mapped surfaces, it would have been nice to see a more general application (e.g. 3D shapes, objects, or structures) flirted with, even if only as a teaser experiment or discussion of future direction. Despite this wish, it would be inappropriate to knock the submission for not exploring every possible application. However, it does leave an open question as to whether the proposed approach is more generally applicable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents document unwrapping based on optimization of neural networks. Four MLPs are defined to model the implicit surface of the target, the colors of the target, the mapping function from 3D points to 2D texture coordinates, and the inverse mapping from 2D to 3D. Similarly to NERF, the target example is approximated by training the MLPs using differentiable rendering. 


main_review: Strengths: 
This paper seems to be the first paper where implicit differentiable rendering is applied to document unwrapping. 

The proposed method is reported to be par with the baseline method (DewrapNet) in quantitative comparisons and outperformed the baseline in qualitative comparisons. 

Weakness:
The proposed method is similar to NERF, where MLPs are trained for each subject. However, DewrapNet is trained once and prediction does not include training. Thus, I am not sure the comparisons are fair. At least, I think the performance issues should be discussed.  

Based on the above reason, the practicality of the proposed method is not clear.
summary_of_the_review: I think this paper is interesting for an experimental study, where implicit differential rendering is applied for document unwrapping. Although I doubt the practical value of the method. 

As an experimental study, the supplemental material seems to include much information about the study that is missing in the paper itself. My concern is that the paper deeply depends on the supplemental document. 

Overall, I would like to positively evaluate the trial of implicit differential rendering for document unwrapping, and showing detained results in supplemental document.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a multi-view method for undistorting documents. The method makes use of an existing method (IDR) for reconstructing distorted documents in 3-D, with an additional bijective neural network to parameterize distorted surfaces by rectangles. The method significantly outperforms SOTAs in the experiments.


main_review: The method seems to be primarily based on IDR with following similarities:
- 3D surface is represented by a Signed Distance Field (SDF)
- Appearance is learned from a neural network that maps 3D surface geometry (location and normal) and viewing angle to radiance values
- The networks are trained with an overall loss including an RGBA rendering term and an Eikonal term for regularising SDF

IDR can handle specular materials but at the cost of introducing so-called 'shape-radiance' ambiguities. This means geometry cannot be possibly recovered due to view-dependency and is only regularised by neural nets' inherent priors (c.f. NeRF++). In this paper though, it seems document pages are all diffuse and I think it would make more sense to drop the view direction term $\mathbf{v}_p$ from Eq. 3. This would disambiguate geometry and could improve robustness.

The authors propose a forward-backward network to learn the square parameterisation of surface with a conformal loss on the forward network. While this design encourages bijective mapping through minimising $L_z$, there is no guarantee the resulting networks are truly bijective. Why not use structures that guarantee invertibility (e.g. Invertible Residual Networks, Behrmann et al.)?  

In paragraph "Deformation constraints for Fz" the author wrote $\mathbf{J}^\top\mathbf{J}=\beta I$. Shouldn't this be $\mathbf{J}^\top\mathbf{J}=diag(\alpha,\beta)$ since you don't know the aspect ratio of document (which is what authors are actually doing with $L_E$,$L_G$)?

The results show clear improvments over existing methods and look very promising. Though personally I'm not familiar with the specific topic of document unfolding or methods that are compared against. 




summary_of_the_review: I like the overall idea of this paper and the results look really convincing. However I would not recommend acceptance to ICLR given its significant overlap with an existing technique (IDR).
review_title: null

Meta-review: This paper proposes an architecture for learned surface parameterization, with application to image unwarping, which can be coupled with differentiable rendering, multi-view data, and other modern objective terms.  The shape of the document is parameterized using an SDF technique, coupled with neural rendering and objective terms inspired by classical geometry processing.  This machinery is quite "heavy," leading to slow training times.

As pointed out by reviewer QH85, there were some experimental discrepancies---rightfully acknowledged by the authors---which make comparisons to DewarpNet less favorable for the new method, at least from a quantitative perspective.  Visual inspection makes the comparison more favorable, although it would be preferable for the quantitative quality metrics and qualitative examples to align.  

Runtime measurements here are also not favorable and severely limit applicability of this technique in real-world scenarios, as pointed out by reviewers hfPz and QH85.

While the mistaken quantitative results are forgivable, the AC agrees that the scope of this work is quite narrow; it is not clear where this architecture would be applied relative to the motivating application.
Decision: Reject