From Transparent to Opaque: Rethinking Neural Implicit Surfaces with $\alpha$-NeuS

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 14:58:06

Abstract: Traditional 3D shape reconstruction techniques from multi-view images, such as structure from motion and multi-view stereo, primarily focus on opaque surfaces. Similarly, recent advances in neural radiance fields and its variants also primarily address opaque objects, encountering difficulties with the complex lighting effects caused by transparent materials. This paper introduces $\alpha$-NeuS, a new method for simultaneously reconstructing thin transparent objects and opaque objects based on neural implicit surfaces (NeuS). Our method leverages the observation that transparent surfaces induce local extreme values in the learned distance fields during neural volumetric rendering, contrasting with opaque surfaces that align with zero level sets. Traditional iso-surfacing algorithms such as marching cubes, which rely on fixed iso-values, are ill-suited for this data. We address this by taking the absolute value of the distance field and developing an optimization method that extracts level sets corresponding to both non-negative local minima and zero iso-values. We prove that the reconstructed surfaces are unbiased for both transparent and opaque objects. To validate our approach, we construct a benchmark that includes both real-world and synthetic scenes, demonstrating its practical utility and effectiveness. Our data and code are publicly available at https://github.com/728388808/alpha-NeuS.

Main paper: 
  
 Appendix:  


Decision: This paper was reviewed by four experts in the field.  The paper received the following recommendations: Strong Accept, Weak Accept, Weak Accept, and Strong Reject.  All the reviewers agreed that the results are promising and the new dataset will contribute to the community.  On the theoretical contribution, however, the reviewers could not reach a consensus unfortunately.  The three positive reviewers highlight the theoretical contribution in their reviews, while the other reviewer strongly disagrees. Having read the submission, the reviews, and the author feedback, the AC decided to accept the paper.  Although Theorem 1 might have room to improve its presentation, it is acceptable to present it in a plain English in the main text with providing a detailed proof in the appendix as the authors did in the original submission. In addition to this point, the reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Reviews: review: paper_summary: paper_summary: The paper presents a theoretically justified method for surface reconstruction of transparent and opaque objects. The authors prove the theorem that the NeuS density function is unbiased even for transparent surfaces. They propose a method based on DCUDF for extracting unbiased surfaces. The method is validated on synthetic Blender datasets and real datasets captured by the authors. The method is compared quantitatively with NeuS at different threshold levels and qualitatively with NeUDF.
strength_weakness: strength_weakness: 1. There are no comparisons of the method on publicly available data with transparent and opaque objects, see Question 2.
2. My concern is that it is not clear whether the method can reconstruct scenes with only transparent objects. Comparison with transparent object methods is limited, see Question 1. The method hasn't been tested on transparent object datasets that have been used previously in the literature, see Question 2.
3. Another concern is the number of competitors. In fact, there is only one quantitative evaluation with NeuS. A quantitative comparison of the method with NeUDF is not presented.
questions: questions: Questions and suggestions:
1. Related work on the method of 3D reconstruction of transparent objects is limited and should be reviewed. I suggest that the authors review the following papers: "NEMTO: Neural Environment Matting for Novel View and Relighting Synthesis of Transparent Objects", ICCV 2023; "Ray Deformation Networks for Novel View Synthesis of Refractive Objects", WACV-2024; "Volumetric Rendering with Baked Quadrature Fields", 2023. All of these papers have applications to transparent object reconstruction and I would like to see how they relate to considered setup. 
2. While some of the previously mentioned papers do not consider datasets with opaque objects, your method is expected to be applicable to datasets with only transparent objects, so I suggest the authors experiment with this setup on publicly available datasets from "Through the Looking Glass": Neural 3D Reconstruction of Transparent Shapes" and NEMTO. It should also be noted that some works consider publicly available data with both transparent and opaque objects, see Fig. 7 and Fig. 9 in the paper "Volumetric Rendering with Baked Quadrature Fields" (https://arxiv.org/pdf/2312.02202v1) from the NeRF-Synthetic and Mip-NeRF-360 datasets respectively.
3. It looks like the authors of "$\alpha$Surf" have released their code (https://github.com/ChikaYan/alphasurf), so I suggest the authors to check it again. 
4. It would be better to have an illustration for Eq. 8.
5. Can you clarify training hyperparameters of your method like lambda coefficients, etc and explain their choice?

Typos:
1. Line 44, .e.g -> e.g.
limitations: limitations: The limitations of the method are adequately addressed in the Limitations section.
paper_summary: paper_summary: The paper proposed a extended version of NeuS that can reconstruct both transparent and opaque surfaces at the same time with unbiasedness. The unbiasedness is guaranteed by theoretical proof and a companying surface extraction method is also proposed. The authors evaluated their method on a small benchmark and demonstrated the methods’ effectiveness and superiority over other methods.
strength_weakness: strength_weakness: 1. The evaluation set is relatively small, adding more synthetic cases would be fine.
questions: questions: I don’t have any question for this submission.
limitations: limitations: 1. Is it possible to include participating media in your pipeline? e.g. if there’s some smog inside a jar, will the unbiasedness still apply? Or can the pipeline be generalized into a stochastic geometry perspective?
2. If possible, can GT of real-world test cases be constructed with some ‘destroy’ to the object, like what NeRO did?
paper_summary: paper_summary: This paper proposes an adaptation to NeuS to extract semi-transparent surfaces. The method relies on training the original NeuS but with a modification to the mesh extraction. Instead of using marching cube; this paper proposes to extract the 0 level of the SDF field and also its local extrema that represent semi-transparent surfaces.  

The contributions are:
- A theoretical derivation to show that the proposed method is sound
- A new method to extract mesh

The method is evaluated on a custom dataset with both synthetic and real scenes of semi transparent objects.
strength_weakness: strength_weakness: - Experiments and data: 
  - It would be nice to have an additional experiment showing the geometric accuracy of the new mesh extraction procedure with usual data, like DTU. I expect this method to perform worse than NeuS on DTU since this method can extract many spurious local minima of the SDF that are ignored with Marching Cube. 
  - While the proposed dataset is interesting, it may be too easy compared to real world. The synthetic data looks like specularities are disabled. However specularities are most present on transparent objects (glass, water) and this may be a harder challenge to solve. As stated in the limitation section, the proposed method cannot deal with reflections. 



- Presentation
  - There could be a discussion about the role of the temperature parameter $s$ in equation (1). This parameter is used to map a SDF value to an opacity. It is introduced in Neus as a method to have a smooth optimization, starting from fuzzy surfaces and finishing with sharp ones. However in this paper s is also used to make sure nonzero SDF still produce nonzero opacity. 
  - From my understanding, this method follows NeuS training but this is never clearly stated
  - What are the normal maps in Fig 4, 6, it looks like normals from the proposed method but it is not written in the caption.
  - Lack of explanations about the optimization (Section 3.2). The paper shows two objective functions (l 167 and 176) it is not clear which one is used. Also there are no explanation about the optimization itself, the solver, stopping criterion, etc.
questions: questions: See limitations: the paper could benefit from an improved presentations. 

I expect from the rebuttal:
- Results and discussion on opaque scene (DTU)
- More explanations about the optimization of section 3.2
limitations: limitations: The limiations are correctly stated
paper_summary: paper_summary: The paper proposes an extension of NeuS to also support transparent surfaces. It states a proposition that claims that NeuS is also unbiased for transparent surfaces, proposes an approach for simultaneous extraction of opaque and transparent surfaces, and presents a benchmark dataset.
strength_weakness: strength_weakness: (1) The current paper organization and dependency severely hinders its appreciation. It should have at least a background / overview section with a minimum of presentation and intuition behind the core concepts. I understand that this paper is an extension of NeuS, but assuming that reading NeuS is an unavoidable prerequisite to conceive an overview of the method is not a gentle choice regarding readers or reviewers. For example, the Abstract, Introduction, and Method sections refer to the concept of unbiased surfaces, which is only introduced at the second paragraph of section 3. Moreover, that is a loose definition since it depends on the concept of rendering weights, which is not introduced at all. The expression rendering weights is ambiguous since it could means a plethora of concepts in the context of the paper. Presenting the equation beforehand and pointing to $\omega(t)$ would remove the ambiguity.

(2) Also in the ambiguity realm, Theorem 1 is not a theorem per se in its current presentation. To be a theorem, a proposition should be a clearly stated assertion that is precise and unambiguous. To be precise, all symbols must be precisely defined beforehand or in the theorem body. Theorem 1 is based on the weight function $\omega(t)$ presented in NeuS, but its current assertion is plain English, without any precise definitions provided. The proposition is not currently verifiable in that form. Overall the paper has a significant problem stating its definitions and symbols.

(1) and (2) are the core reasons behind my current rating and, unfortunately, I do not think they could be solved without substantial rewriting and in rebuttal time. I think the problem addressed is important, and also that the paper has good results, but it is based on a core theorem that is not correctly stated. I advise the authors to rewrite the paper, following the guidelines:

* Please take a more didactic approach. In practice, at least a background / overview section should be present. The paper should be self-contained in some extent. That would broaden the public for the paper, since readers could understand the paper in different levels and in increasingly difficulty. It would also ease the review process, possibly earning more favor in future reviews.

* Please precisely present the symbols and equations before the theorem statement to remove ambiguities. The usual form of a theorem is "Let {definitions and symbols}. {conditions using the symbols} then {conclusions using the symbols}." In case the logic is bidirectional, the form is "Let {definitions and symbols}. {conditions / conclusions using the symbols} if and only if {conditions / conclusions using the symbols}". Following those standards helps to smooth the reading flow and helps the reader to verify whether the theorem holds. Since the theorem is a core component of the paper, including a mathematician author or making one reviewing the paper before the submission could also be a good idea.

* Another approach would be to remove the theorem and make a discussion based on empirical evidence. The current results are good, arguably sufficient for acceptance if the paper is written in such standpoint.

(3) The paper criticizes $\alpha$Surf in a few locations, saying it introduces noise and artifacts. I can see such phenomena in that reference, however I do not think it is good practice to criticize another work without showing comparison experiments. Since the work is still unpublished and the code source is unavailable I would like to politely ask the authors to remove such critics. A more polite way to contextualize would be to say that $\alpha$Surf is a concurrent work, without judging its results.
questions: questions: (1) I would like to see a quantitative evaluation against NeUDF too. Even though NeUDF seems to smooth the surface, $\alpha$-NeuS also seems to introduce noise. I am curious about which one is closer to the ground truth surface.
limitations: limitations: Yes, the paper has a paragraph about limitations.

Comments: title: Thank you for your reviews
comment: Dear Reviewers,

We would like to express our gratitude for your efforts in reviewing our work and providing us with invaluable feedback. We would like to draw attention to the contributions made by our work.

The reconstruction of thin, transparent objects presents a significant challenge. Our objective is to develop a unified framework for reconstructing thin transparent and opaque objects simultaneously. Our key contributions are as follows:

(1) We generalize NeuS to reconstruct transparent and semi-transparent objects and demonstrate, through a mathematically rigorous proof, that unbiased surfaces lie on either the local minimum distance surface (if the local minimum distance is non-negative) or the zero level set.

(2) We propose a unified computational framework to reconstruct 3D objects from fully transparent to fully opaque.

(3) We construct a benchmark that includes both synthetic and real-world models for evaluation, demonstrating that our method outperforms existing methods.

We thank you again for your invaluable feedback.

Sincerely,
The Authors
The mathematically rigorous proof of the theorem is in the technical appendices of our original submitted paper. Since NeuS is a well-established work in the field, we assume familiarity among our readership. We believe that the fewer symbols the easier it is to understand, so the theorem was originally presented in plain language in our original paper. As requested by the Reviewer GRsP, we revise the theorem symbolically, and present it in a mathematically rigorous way. Here we also outline the proof briefly here for clarity:

(1) Opacity-Distance Relationship: Utilizing the volume rendering equation and NeuS's distance to density function, we establish a correlation between opacity and learned distance. Specifically, we demonstrate that opacity $\leq 0.5$ when the learned distance reaches a non-negative local minimum, and opacity $> 0.5$ when the local minimum is negative.

(2) Color Weight Maximum: We analyze the derivatives of the color rendering weight relative to the ray parameter $t$. We establish that the maximum color weight coincides with the local minimum distance when it is non-negative, and with the zero iso-surface when the local minimum distance is negative.

Reviewer GRsP raised concerns about what he/she described as "ambiguous" concepts, including rendering weights and unbiasedness. Notice that the rendering weight $w(t)$ is fundamental to the volume rendering equations used in prominent models such as NeRF and NeuS. Similarly, "unbiasedness" is a pivotal concept in NeuS, now considered standard in neural rendering of implicit surfaces. Since these terms are well-established within the field, we chose not to redefine them in our initial submission. Recognizing that these concepts might be challenging for readers who are not familiar with NeuS, we plan to include a detailed overview of NeuS in the revised manuscript, as indicated in our rebuttal.

We respectfully disagree with Reviewer GRsP on the notion that machine learning papers neglect rigorous theorem treatment. Contrary to this view, all other reviewers have acknowledged the importance of our theoretical contribution. As pointed by Reviewer 6LF5, our paper "provides a rigorous theoretical analysis, extending the unbiasedness proof of NeuS to cover a range of material opacities from fully transparent to fully opaque. This theoretical work completes and expands the framework of NeuS."

Beyond theoretical contributions, our work includes practical advances. We have developed an algorithm that outperforms existing methods and the concurrent $\alpha$Surf approach. We also introduce a real-world dataset featuring semi-transparent objects, showcasing the practical effectiveness of our approach on these realistic scenes.
The main claims of the paper are theoretical, the entire Method section is based on Theorem 1. When the authors organize the paper that way, they should be aware of the burden of rigorous Math checking in the review. As I said before, Theorem 1 was not stated as a theorem in the original manuscript, making it impossible to check. It was highly ambiguous, with several core definitions missing. My original rating was Strong Reject because that is a major technical and methodological flaw, turning it impossible to adequately evaluate the paper. I turned it a Reject because I had hope the other reviews and rebuttal could point out other perspectives that could make me reevaluate the rating. After carefully reading them and reflecting, that was not the case.

I appreciate that the authors provided a new statement for the theorem, however it is not possible for me to rigorously do that checking in the rebuttal discussion time frame because I have other five reviews in NeurIPS, and submissions. Unfortunately, there was nothing the authors could have done to increase the rating, given the situation. I believe it is risky for NeurIPS to accept a theoretical paper without properly checking the core theorem.

I maintain my advice that the authors should create a new version based on a rigorously stated and proven theorem with adequate context or change the paper structure to not focus on theoretical claims. For example, the original NeuS has a different structure, with the theorem not being protagonist in the claims. Even though I believe every theorem should be rigorously checked (even the ones in supplementary), there is a sad tendency in Machine Learning papers to not treat theorems with proper attention. In some extent, I consider that a disrespect to the field of Mathematics and to mathematicians. Unfortunately, the manuscript crossed the boundary of what is acceptable in a theoretical standpoint.
Dear Reviewer GRsP,

Thank you for dedicating time to review our manuscript. We appreciate your feedback regarding the overview of NeuS and the symbolic expression of Theorem 1. We have carefully addressed these points in our rebuttal and believe that the necessary revisions are straightforward.

We noted that subsequent to our rebuttal submission, the evaluation of our paper shifted from 'Reject' to 'Strong Reject.' Could you please provide further clarification or specific reasons for this change in the assessment? We are eager to understand your concerns more fully and to address them appropriately in our revisions.

Thank you. 

The authors
Thank you for considering the review and for the responses.