Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a Fourier Occupancy Field representation to address the problem of monocular human mesh reconstruction from an image. In FOF, the 3D mesh is essentially converted into a multi-channel image that also possesses sampling scalability, stable reconstruction, and low-complexity reconstruction. Experiments demonstrate the proposed method outperforms existing methods quantitatively and qualitatively, while also enabling real-time applications.
strengths_and_weaknesses: The strengths of the paper are:
- The paper is well-written. The discussions and explanations are not only informative but also insightful.
- The proposed FOF representation is simple and effective. The methodology is general purpose and could benefit other related tasks.

The weaknesses of the paper are:
- Minor improvements on paper writing: 1) A typo in Line 236 on the symbol subscript. 2) Table 1 would be much more informative if references could be added for each type of method. 3) It would be nice to specify if any post-processing techniques are used such as smoothing or connected components.
- There is a significant performance gap, both quantitatively and qualitatively, between vanilla FOF and the enhanced version FOF-SMPL. However, in the main methodology section, the SMPL part was somehow downplayed. If this was done to emphasize FOF as the main novelty, it would be more convincing to apply FOF to other relevant tasks. Otherwise, if the focus is on 3D human reconstruction, the SMPL part should be included as an important subsection with more details.
- The experimental section would be further richened. For instance, a quantitative ablation study on the number of FOF components would be great to help understand how it works. A runtime break-down would be also nice to shed light on how fast each component is.
questions: - Regarding the number of FOF components: It would be great to have a quantitative analysis of how this affects the performance. Also, it seems this number doesn't significantly impact run time since only tensor multiplication is needed. It would be great if the authors could explain why a maximum of 15 is used, i.e., what would happen if N>100?
- Would the FOF representation be used in other related tasks such as general single image 3D prediction?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed an occupancy field representation for monocular real-time human reconstruction. The proposed representation represents a 3D object with a 2D field orthogonal to the view direction. Each 2D position stores the 1D occupancy field of the object along the view direction, and the 1D occupancy field is represented by real-valued fourier series. The proposed method is able to achieve 30+FPS while achieving comparable accuracy to other monocular methods.
strengths_and_weaknesses: Strength: The paper proposes a real time monocular human reconstruction method with implicit occupancy representation. The occupancy field is represented as a pixel aligned 2D grid. The novel idea of this paper is, in each grid the 1D occupancy signal is represented by real-valued fourier series.

The paper shows that 1D occupancy signals tend to be low frequency, thus can be effectively represented by the first few fourier basis functions. This reduces the output dimension of the inference model, thus effectively improves computational efficiency. 

Weakness: approximating occupancy with a very small number of fourier basis is still quite counter intuitive. I think the author could do a better job in analyzing the property of this approximation. It would be nice to see what is the theoretical bound of the distance to the surface given Equation 6. It would also be nice if the authors visualize the reconstructed 1D occupancy curve given different number of fourier basis. In addition, representing signals with a discrete set of uniformly sampled fourier basis is equivalent to applying sinc filter to regular-grid delta signals. In other words, there are different ways of representing a band-limited signal (e.g. spatial vs frequency representation), thus it would be helpful if the author provides some insight what is the benefits of representing signals in the frequency v.s. spatial domain. 

In general, I recognize the practical advantage of the proposed method, but feels the technical / theoritical contribution of the paper is a bit thin. The paper in general follows the engineering details of other methods such as PIFU and ICON and the difference is mainly the simplified way of representing occupancy signals with fourier transforms. Such strategy has been seen in use in other problems, and it inhirits the defects of bandlimited approximations.
questions: (1) With the fourier transformation of 1D the occupancy field given in Equation 6, what is the error bounds of the distance between recovered v.s. groundtruth surface point? 

(2) More visualization of the recovered 1D occupancy curves.

(3) Discussion of different ways to represent bandlimited signals and its implication in the problem of interest.

(4) In the discussion, the authors mentioned future works could incorporate wavelet transforms to improve results for thin objects. But it seems that would blow up the dimension of the representation with an naive implementation, and its computational advantage over voxel-based representation may diminish. I would be curious to learn if the idea of utilizing basic signal processing techniques advocated by the authors could indeed be generalized to more complex scenarios. 


limitations: As pointed out by the author, the bandlimited approximation of the 1D occupancy field is insufficient to recover fine-details as well as thin structures. The poor results in hand and limb regions reveal such defects.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new representation for human reconstruction. Specifically, for a pixel in the input image, it approximates the occupancy along the casted ray as the Fourier series. The proposed approach learns the representation from data by regressing to the coefficients of the Fourier basis functions. The submission shows the efficacy and flexibilities of the representation on real captured data.
strengths_and_weaknesses: ## Strengths

1. Originality: the proposed representation of FOF is novel. Approximating the occupancy with Fourier series is interesting and principled.
2. Quality: the presented reconstruction is of high-quality.
3. Clarity: the paper is well-written and easy to follow.
4. Significance: it is an important problem of developing an efficient and powerful 3D representation to enable real-time application. The submission demonstrates a real-time pipeline with the FOF representation.

## Weakness

I do not see some major weakness. But it would be great if authors can discuss more about the quality of the reconstruction. For example, in Fig. 4, Fig. 5, and video in the supplementary, I observe that PIFuHD usually maintains more details even though FOF-Normal already utilizes the inferred normal maps. It would be beneficial to discuss why FOF representations are not detailed than PIFuHD.
questions: Essentially, the training of FOF will be a per-pixel regression problem. Namely, the network needs to regress to GT coefficients $C(x, y)$ for Fourier basis functions. Since the per-pixel regression is a hard problem, I am curious about how robust the FOF representation is to the inaccuracy of such regression. Specifically, I think showing the following results would be beneficial:
1. The training loss of relative error (absolute error may not be that informative as the magnitude matters)
2. Manually add noises to the GT coefficients and plot a curve to show the correlation between the perturbed coefficients and the quantitative quality of the reconstructed meshes.
limitations: I appreciate the authors's discussions about limitations of the proposed approach in Sec. 5, which helps the understanding of the suitable scenarios.
review_title: null

Meta-review: metareview: This paper received 3 positive reviews: 2xBA + A. All reviewers acknowledged that this work introduces meaningful and non-trivial contributions, it is well presented, and the claims are supported by strong empirical performance. The remaining questions and concerns were addressed in the authors' responses, which seemed convincing to the reviewers.
The final recommendation is therefore to accept.
confidence: Certain
Decision: Accept