FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction
Authors: Qiao Feng
Yebin Liu
Yu-Kun Lai
Jingyu Yang
Kun Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Monocular human reconstruction
Representation
Real-time
Single image

Abstract: The advent of deep learning has led to significant progress in monocular human reconstruction. However, existing representations, such as parametric models, voxel grids, meshes and implicit neural representations, have difficulties achieving high-quality results and real-time speed at the same time. In this paper, we propose Fourier Occupancy Field (FOF), a novel, powerful, efficient and flexible 3D geometry representation, for monocular real-time and accurate human reconstruction. A FOF represents a 3D object with a 2D field orthogonal to the view direction where at each 2D position the occupancy field of the object along the view direction is compactly represented with the first few terms of Fourier series, which retains the topology and neighborhood relation in the 2D domain. A FOF can be stored as a multi-channel image, which is compatible with 2D convolutional neural networks and can bridge the gap between 3D geometries and 2D images. A FOF is very flexible and extensible, \eg, parametric models can be easily integrated into a FOF as a prior to generate more robust results. Meshes and our FOF can be easily inter-converted. Based on FOF, we design the first 30+FPS high-fidelity real-time monocular human reconstruction framework. We demonstrate the potential of FOF on both public datasets and real captured data. The code is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/FOF.  

FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction
Qiao Feng Tianjin University f fengqiaoOt.j..o edu en
Yebin Liu
Yu-Kun Lai Cardiff University laiy40cardiff ac uk
Tsinghua University
liuyebin@mail. tsinghua. edu. cn httpimmaaa.tttstitiitee
Jingyu Yang Tianjin University yjy@tju.ede edu.
Kun Li"
Tianjin University likētju edu. cn
Abstract
The advent of deep learning has led to significant progress in monocular human reconstruction. However, existing representations, such as parametric models, voxel grids, meshes and implicit neural representations, have difficulties achieving high-quality results and real-time speed at the same time. In this paper, we propose Fourier Occupancy Field (FOF), a novel, powerful, efficient and flexible 3D geom- etry representation, for monocular real-time and accurate human reconstruction. A FOF represents a 3D object with a 2D field orthogonal to the view direction where at each 2D position the occupancy field of the object along the view direction is compactly represented with the first few terms of Fourier series, which retains the topology and neighborhood relation in the 2D domain. A FOF can be stored as multi-channel image, which is compatible with 2D convolutional neural networks and can bridge the gap between 3D geometries and 2D images. A FOF is very flexible and extensible, e.g., parametric models can be easily integrated into FOF prior to generate more robust results. Meshes and our FOF can be easily inter-converted. Based on FOF, we design the first 30+FPS high-fidelity real-time monocular human reconstruction framework. We demonstrate the potential of FOF on both public datasets and real captured data. The code is available for research purposes http://cceeuueelaccccccu/////aap
Figure I: Our novel 3D geometry representation (FOF) enables monocular real-time human recon- struction with high-quality results at 30+FPS.
"ooresponning author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Comparison with existing representations. topologies and geometries.)
(Generality: ability to deal with complex
Computational Efficiency
Representation
Aligned with Input
High--Quaitty
Generality
Purametric Model 17.20
Voxel Grid [T
X
Mesh 531 22 [4]
Implicit Function 22 23 P
X
Our FOF
V
1 Introduction
3D human reconstruction from a single image is very popular in computer vision and graphics, which can be widely used in various applications, such as mixed reality, virtual try-on, and "metaverse". However, high-fidelity monocular human reconstruction in real-time remains a challenging task. 3D representation is at the core of this problem and determines the design and performance of approach. 3D geometry representation is also the key for 3DTV and Holographic Telepresence systems, which need to fulfill the requirements of accuracy, efficiency, compatibility and generality []]
Parametric human models, e.g., SMPL 17 and SMPL-X 20] represent 3D human body with parameters. Their results are robust but they are unable to reconstruct the clothes. Many classic representations, such as voxel grids 31 or meshes 33 2 14, have been exploited for monocular human reconstruction. However, the representation of voxel grids has to store discrete spatial grid samplings in memory. has a space complexity of (n (where n the number of grids in each dimension), which limits the achievable spatial resolution. Meshes on the other hand are a compact representation for surface geometry. However, it challenging for meshes to cope with change of connectivity, and as result, they may produce unstable or over-smoothed results when there are change of topology and/or substantial deformations.
Recently, implicit neural representations have emerged and been widely used in monocular human reconstruction 222 23 9 27 The 3D space is treated as a continuous field, such as an occupancy field and signed distance field, which can be formulated as R - R F(x. y. z). Rather than sampling the field and store it as a voxel grid, neural network is used to fit the field. Thus, the results can be produced at any resolution. However, human geometry is not explicitly reconstructed, and the network has to infer values for a huge number of spatial grid sample points to extract the geometry. The number of sampled points in inference is limited by available GPU memory, and the sampled points have to forwarded several batches, which very time-consuming. Therefore, it is very difficult implement high frame rate real-time reconstruction based implicit neural representations. Although Monoport (I5), as a real-time method, renders the mesh-free implicit field at 15FPS with an efficient sampling scheme, it does not reconstruct the geometry explicitly. Thus, it is not compatible with traditional graphics applications.
In this paper, we propose a new representation, Fourier Occupancy Field (FOF), for monocular real-time and accurate human reconstruction, which an expressive, efficient and flexible 3D object representation in the form of 2D map aligned with the input image. The key idea of FOF is to represent 3D object with a 2D field by representing the occupancy field along z-axis into Fourier series and only keeping the first few terms, which is memory-effiiiet and can preserve most of the geometry information. With grid sampling on the z-axis and y-axis, the FOF can be stored explicitly multi-channel image, which is compatible with the existing CNN frameworks and suitable for 3DTV applications. Unlike depth maps, our FOF can represent the complete geometry rather than just the visible part, and it is very efficient to produce resolution meshes and our FOF can be easily cnter-converted. Detailed comparisons with existing representations are given in Table
human shapes. Moreover, high-
Based on our FOF, we design a simple and efficient pipeline for monocular real-time human recon- struction, which is end-to-end and easy to train. Some live captured examples are shown in Fig. I. Experimental results on both public dataset and real captured data show that our approach can reconstruct human meshes accurately and robustly in real-time.

---

Input Image
ab22
Reconstructed mesh
Eqpaaii storp vicr drotion
Emgereergge
Occupancy
Figure 2: Overview of our reconstruction pipeline. With our FOF, only image-to-image network is needed for high quality monocular human reconstruction.
To summarize, the main contributions of our work include:
We propose Fourier Occupancy Field, a novel representation for 3D humans, which can represent a high-quality object geometry with 2D map aligned with the image. It can bridge the gap between 2D images and 3D geometries.
We present simple and efficient pipeline for monocular real-time 3D human reconstruction, which end-to-end and easy to train. Parametric models can be optionally transformed into Fourier occupancy field as prior produce robust results.
Compared with SOTA methods based other representations, our approach can produce more realistic results real-time. Moreover, it is the first 30+FPS pipeline, which 2 times faster than Monoport [15] with better results.
2
Related work
We briefly overview literature on monocular human reconstruction using a single color camera here.
Surface-based reconstruction. This kind of approaches focuses the interface between geometry and empty space and can be broadly classified into three categories: parametric models, UV-map- based methods, and graph-based methods. Parametric models, such SMPL [17], SMPL-X 20 and SRAR [19 are popular representations for 3D human. They collect a large amount of naked human shapes, and get analytical model on them with statistical methods, which generate a human body mesh with dozens of parameters. Based on this, a lot of works [12 13 estimate 3D human shapes from RGB image by predicting the parameters of the parametric model, but these methods cannot reconstruct the clothes or hair. An overview of 3D human shape estimation from a single image be found in Tian el al. (25). Other methods try estimate clothed humans by adjusting vertices of the parametric models. Alldieck et al. [2] warp the input image to align with the UV map, and estimate the displacements on the T-pose SMPL mesh. Their results are always T-pose and cannot match the input images well. Li et al. [4] reconstruct the geometry of a clothed human body with graph neural networks, but the recovered mesh tends be smooth and lacks fine details. Zhu et al. [32 33 use four stages constrained by joints, silhouettes and shading, but their results are sometimes inconsistent with the images. It is very difficult to represent 2D manifold surfaces of different 3D objects with a fixed structure, and hence most methods above rely heavily on the topology defined by SMPL. and cannot produce detailed human geometries with complex topology. Habermann ef al. 7 g propose a deep learning approach for monocular dense human performance capture, but this method needs to scan the actor with a 3D scanner as the template mesh. Graph- methods cannot get exact features aligned with the results, and hence the results are unstable and over-smoothed.
Volume-based reconstruction. Volume-based methods estimate the attributes, e.g., occupancy and signed distance, for each point in the space and can represent 3D shapes with arbitrary topology. Voxel girds and implicit neural networks are two main representations used in volume-base based methods.

---

Coufficients
Approximated f(x)
Line 3D space
((s,))
Fourier series
((x,))
bies functions
view direction
Occupancy
outside
inside
outside
Figure 3: The proposed Fourier Occupaney Field (FOF). Each pixel the image I corresponds to line in 3D space and can be described with a occupancy function f(z), which can be expanded as Fourier series and represented with vector C(ar, v)- The 3D human model can be encoded as 2D vector field, our FOF.
Zheng et al. [3 regress the voxel grids of 3D human using convolutional neural networks, but this kind of methods requires intensive memory and has limited resolution of the model. Unlike voxel grids, implicit neural representation can represent detailed 3D shapes without resolution limitations.
Saito er al. [22] pixel-aligned function reconstruct 3D human from single RGB image. They further use normal maps to significantly improve the 3D geometric details (23). Zheng et al. [30]
use the voxelized SMPL mesh as a prior to make the results more robust. Xiu al. [27] propose
method to correct SMPL model estimated by other methods (29 13 6 according to the input image,
and then regress occupancy field from the normal map and the signed distance field of the corrected SMPL. Their results are robust to human poses, but the geometric details do not match the input images and are noisy. The common problem of the methods based on implicit neural representation is that they are computationally heavy and very mine-consuming. Although Li et al. [IS] propose an efficient sampling scheme to speed up the inference, they can only achieve mesh-free rendering at 15FPS. When generating a complete mesh, their method will be less than 10FPS.
In this paper, we contribute the first real-time high-quality human reconstruction system working 30+FPS with a single RGB camera, benefiting from the proposed efficient and flexible 3D geometry representation FOF.
3 Method
The goal of our work is to reconstruct a high-fidelity 3D human model from single RGB image in real-time. To this end, we propose an efficient and flexible 2D representation of 3D geometry (Sec. B. named Fourier Occupancy Field (FOF), as shown in Fig. 2 In the proposed reconstruction framework, we first learn the FOF representation for the input image by an image-to- FOF mapping (Sec. 3.2), and then reconstruct the 3D human model by the efficient Fourier inversion with low complexity (Sec. 3.3
3.1 Fourier Occupancy Field
Without loss of generality, 3D objects to be reconstructed are normalized into []1 cube. Denoted by (z, y, 2) [-1, the 3D coordinates, and by S the surface of 3D object. The 3D object can be described by a 3D occupancy field F: [-1,1 + (.,0.5, 1}, where the occupancy function F(x, y, 2) is defined as
1, 0.5, 0, (z, 2) is outside the object.
(z, M, 2) is inside the object,
F(x, y, 2) =
(x, y, 2) E S,
(I)
Such 3D representation highly redundant, because only small subset, i.e., iso-surface of F(x,, y, =) with value of 0.5, is sufficient to represent the surface of the object. Direct inference of the 3D occupancy field F from single image not only confronts with the curse of dimensionality, but also requires more computation and memory in handling high dimensional feature maps. Note

---

that the occupancy field defined in the 3D cube can be regarded as the collection of a large number of ID signals defined on the lines along the view direction. Without loss generality. we assume that the view direction is the same as the z-axis of the 3D cube. The occupancy function, as ID signal, along the line passing through a particular point (x". ) E [-1, on the xyy-plnne can be written as f(z) [-1,1] + F(x", y" ,)) We explore Fourier representation for such family of 1D occupancy signals, and propose a more compact 2D vector field orthogonal to the view direction, namely FOF, for more efficient representation of the 3D occupancy field, which is detailed in the following subsections.
Fourier series on single occupancy line. ID occupancy signals {f(=)) are essentially on-off signals switching the boundaries of human bodies, which lie low-dimensional manifold the ambient signal space. For accurate and fast 3D reconstruction, the compact representation should have the following merits:
Sampling Scalability: The representation should be able to adapt to different sampling rates in the inference stage without introducing systematic mismatch besides the inherent approximation error due to sampling. Without retraining, systems with such a representation would support diverse applications with different resolution, quality, and speed requirements.
2. Stable Reconstruction: The representation should provide stable reconstruction quality for diverse humans with various parameters such garments, heights, weight, and poses.
3. Low-complexity Reconstruction: The representation should allow low-complexity recon- struction support emerging real-time applications such Holographic Telepresence.
However, most popular representation do not enjoy all the above merits. For instance, discrete Fourier transform and discrete wavelet transform have stable and low-complexity reconstruction, but only support the sampling rate used in preparation of the training set due to their discrete nature. Learned dictionaries are able to provide sparse representation, but require compuration-demaniigg iterative reconstruction, leaving alone the sampling scalability. Without doubt, sampling scalability is the most difficult property to meet. Pushing the concept of sampling scalability to the limit, one representation would be scalable to arbitrary sampling rates in theory if it allows continuous reconstruction along the 2-axis. Note that the representation should stay discrete and compact for efficient inference. Such a bridge of discrete-representttion and muntinuous-reconstrue for the representation of 1D occupancy signals {f()). J. By nature, Fourier series would have infinite number of discrete coefficients, which impractical. Note that most energy of 1D occupancy signals concentrate on only few low-frequency terms, which yields compact representation by subspace approximation.
motivates us use Fourier series
Formally, let (=) be periodic extension of [=)). Note that [p(() satisfies the Dirichlet conditions: 1) fa(=) is absolutely integrable over one period; 2) fp(=) has a finite number of discontinuities within one period; 3) fz(z) has finite number of extreme points within one period. Thus, fz( can be expanded as convergent Fourier series:
fx(2) = ag
(an cos(122T) + b, sin(n27)),
(2)
2
where {a,}, (b..) E IR are coefficients of basis functionn (cos(ner) and (sin(nz)). respectively. Since [() is a specific period of fp(z). Eq. ( also holds for f(=). Note that by defining the occupancy function as Eq. (T) Eq. (2) holds Tor discontinuity at z" on the surface S because f(=*) = (f(2-))
+))7 = 0.5, where - and +(=++) are the left hand limit and right hand limit of f(2) at 2. For compact representation, we approximate ID occupancy signals by a
subspace spanned by the first 2N + 1 basis functions:
b (z)c,
where b(2) [1/2, cos(2), sin())..., cos sin, (Nn) is the vector of the first 2N + 1 basis functions spanning the approximation subspace, and e (ao, a1 bi....... uN, bx] is the coefficient vector which provide a more compact representation of the 1D occupancy function f(z). In our implementation, N is chosen as 15, which is accurate enough for most 3D human geometries.
5

---

Fourier occupancy field. Then, such Fourier subspace approximation is applied to all ID occupancy signals over the zy-plane, which is shown in Fig. E
F(z, y. 2) = b%6 (z)C v),
(4)
where (1,, y) is the (2N + 1) Fourier coefficient vectors for the 1D occupancy signals at (a, y). In this way, we obtain the Fourier Occupancy Field C 1-1, 22 for 3D occupancy field F.
Note that 3D coordinates (a, y, 2) in Eq. (4) are continuous, and can be sampled as needed. Particu- larly, unlike the discrete Fourier transform discrete wavelet transform, the subspace dimension for 1D occupancy signals {f())) is not coupled with the sampling grid of 2-axis. This would avoid potential sampling mismatch between the sampling grid used in the training data and that in the testing stage. The reconstruction of F from the FOF C' is sampling scalable in the sense that sampling rate along the z-axis can be chosen as necessary, e.g., according to the requirement of reconstruction quality and speed. 3D models any resolution can be readily obtained from the FOF representation by simply alter the sampling rate along the z-axis. Moreover, with sampling on zy-plane, e.g., aligned with the sampling grid of the input image of size H x W, the FOF C' a multi-channel image of size H x W x (2N + 1), which is more compact than the 3D occupancy field and is friendly for network prediction as detailed in the following subsection.
3.2 Learning FOF with neural networks and variants
Network. Thanks to the efficient dimensionality reduction with subspace approximation, the task of FOF learning from input image is essentially to learn an image-to-iman network. To this end, we exploit the network standing on the HR-Net [26] framework for its outstanding fitting capability in various vision tasks. We use the weak perspective camera model in implementation, in which FOF (and thus the reconstructed geometry) is naturally aligned with the input RGB image. Note that perspective camera model can also be used by using the normalized device coordinate space (NDC space). Note that the proposed FOF representation is scalable in terms of sampling along not only the z-axis (discussed in Sec. 3.1), but also the x- and y-axes. For example, for the input image of size W x H, we learn the FOF of size W x H x (2N + 1). To reconstruct a 3D shape of size W' x H' x K, we simply resize the learned FOF to the size of W x H' (2.N +1) along the æy-plane, and specify the number of sampling points K for the z-axis at the Fourier inversion. Such a scalable reconstruction of 3D shapes a significant departure from aalling-afee--rcconstttee that involve complex 3D manipulation, and is more analogous to efficient image resizing. Without requiring complex accommodation, our method can be readily deployed in various applications with heterogeneous computation resources. Our FOF is also highly extensible. Here we introduce two variants of the reconstruction pipeline, named
strategies
and FOF-Normal.
FOF-SMPL. In FOF-SMPL, we use the SMPL parametric model as a prior to achieve more robust reconstruction, inspired by PaMIR B0]. Specifically, we transform SMPL mesh to a FOF with the algorithm in Sec. 3.3 and concatenate it with the RGB image as the input together. Thus, the input of FOr-SMPL is a (31 + -channnl image. The exploit of smpl provides a reference for the result, and eliminates the depth ambiguity which makes the network more focused on the local relative details rather than the global absolute 3D positions. This makes the network more robust different poses and improves reconstruction quality for cases with high ambiguity.
FOF-Normal. In FOF-Normal, we normal maps enhance the results. Specifically, infer the front and back normal maps with the pix2pix network [1T] for the input image, and also concatenate them with the RGB image together as input. Thus, the input of FOr-Normal is (3 + 3 3)-channel image. Note that we use the inferred normal maps, instead of ground-truth, in the training stage, which makes the network more robust for the input normal maps.
Loss. We use the LI loss is to train our FOF baseline and variants. To make the network more focused on the human geometry, we only supervise the human foreground region of the image. Given a training sample (T, C), the loss function L is formulated as:
1
Σ
[[(,,,) C(r,))),
(5)
M
(K.M)EM
where M is the set of foreground pixels of C, and Ô is the FOF predicted by the network.

---

3.3 Transformation between FOF and mesh
FOF to mesh. Extracting meshes is necessary for applications such as mesh-based rendering and animation. Recent methods based on implicit neural representations need to run network on a 3D sampling grid. Instead, as described in Eq. 4 the reconstruction of 3D occupancy field from FOF is simply a multiplication of two tensors. Then, the 3D mesh is extracted from the iso-surface of the occupancy at the threshold of 0.5 with the Marching Cubes algorithm [8]. Both steps can be parallelized on GPUs for fast mesh generation. In this paper, all our results are produced directly by the marching cubes algorithm and no post-prose post-processing technique used.
Mesh to FOF. To prepare training data, 3D meshes should be transformed into FOF as training labels. By definition Eq. O we are to calculate the first 2N + coefficients of the Fourier series for each ID occupancy signal fz)). For generic signals, this would require numerical integration over a sampled version of f=), which, however, may introduce numerical errors and demands considerable computation. Fortunately, thanks to the particular form of occupancy signals, the Fourier coefficients can be derived analytically with exact solutions. Note that the 1D occupancy signal f=) is actually defined by discontinuities, where the line goes inside outside the mesh. We extract the discontinuous points of f f() by designing a rasterizer-like procedure. Formally, suppose the line associated with f=) passes though the mesh for & times. For the 7H traversing, let and 2 be the
locations going inside and outside the mesh, respectively. Then, the set 2 - {(2,, =f)1 collects those intervals of the line that are inside the mesh. According to the definition of the occupancy signal, we have: fz) = 1, E 2, and f(2) 0 otherwise except for a finite number of discontinuities with measure zero. Therefore, we have the following analytical solution for the Fourier coefficients.
{ an _-1 fz) cos(n2z)dz
Σ - (sin(nn=2) - sin(ntz)))/ (nx), bn = [_ (z) (iss)sin(mz)(d = E.= (cos(11T2,) (oo=m)))))/)
(6)
The FOF representation is obtained by calculating the Fourier coefficients for all the ID occupancy signals. Since the number of intervals & is generally small, e.g., 2~6 for human meshes, the calculation of an or b, is of (11) complexity. Thus, the transformation from discontinuities to FOF has computational complexity of OWN)W, and can be parallelized on GPUs for acceleration.
4 Experiments
4.1 Datasets and metrics
We collect 2038 high-quality human scans from Twindom and THuman2.0 (28) to train and evaluate our method. Twindom contains 1512 scans with various clothing, but their poses simple and lack diversity. THuman2.0 0 contains 526 scans with more variety in body poses. We randomly select 1059 from Twindom and 368 from THuman2.0 as the training set, and 302 from Twindom and 105 from THuman2.0 the test set. The remaining subjects used as the validation set. We render 256 views images with 4 lights randomly selected from 240 lights for each scan. All the images are rendered with weak perspective camera and pre-computed radiance transfer (PRT(24) renderer. To obtain the corresponding SMPL model, which is used for FOF-SMPL. and ICON 27 we OpenPose (5) to detect 2D keypoints the rendered images and generate SMPL model with multi-view version of SMPLify-X (20).
Datasets.
Metrics. We use Chamfer distance, P2S (point-to susrfcc) distance, and normal image error for evaluation. Chamfer distance is defined as:
dcp(Spr: Sqx)
Σ min
1|x y
v112
Σ min | vll2. ZES,t
Spr
Syt
yESpr
SES
where Sp. is set of points on the predicted mesh surface, and Sgt is set of points on the ground- truth mesh surface. We average the Chamfer distances of all test samples, and report the square root of half of it. We also measure the average point-to-surface distance from the vertices on the reconstructed surface to the ground truth. To measure the visual quality, we render the reconstructed geometries normal maps from the view direction, and compute L1 loss with the ground truth.
'ntts://web.wwindom.com
7

---

Figure Real-time reconstruction results by our method.
Table 2: Comparison with the state-of-the-att methods.
FOF
FOF-SMPI.
FOF-Normal
PIFu
PIFuHD
ICON
ICON fitter
Chamfer
2.693
1,580
2.601
3.688
2.983
2.831
2.856
P2S
1,482
0.878
1,435
2.604
1.773
1.586
1.544
Normal
2.807
2.652
2.837
3.265
2.648
3.723
3.628
4.2
The real-time pipeline
To perform real-time monocular human reconstruction, we design three-stage pipeline. In the first stage, get image from the video stream with OpenCV (4, and use RVM [T6] get a human mask to remove the background. RVM is originally a matting method, and we use the threshold of 0.5 obtain a binary mask. the second stage, the FOF of the human is estimated by our network in Sec. 3.2. Then the FOF is resized to a proper resolution (256 x 256 in our implementation), and transformed a mesh introduced Sec. 3.3 the final stage, we render the mesh into images with PyTorch3D [21 to preview the reconstructed geometry in real time. Our three stages are all implemented with Py PyTorc and running on single RTX-3090 GPU. Our pipeline works at 30FPS which is limited by the camera. With high-frame-rate camera, TensorRT and more GPUs, our frame rate can be further improved. Fig. gives some real-time reconstruction results. More results are htt.://dww..iicillllll/////////e
available at
4.3 Comparison
We compare our method with three state-of-art approaches, PIFu (22), PIFuHD 23 and ICON (27] together with our two variants, FOF-Normal and FOr-SMPL. We retrain PIFu on our dataset. Other methods are used with the pre-trained checkpoints, because they do not provide the training codes. We use 512 x 512 x 512 resolution for all these methods. We do not compare our method with B and [10] because they do not release their codes. For simplicity. we also ignore comparisons with some works that have already been compared like PaMIR [30 We compare our method with two versions ICON. In Table 2 ICON is the original version proposed in their paper, (CONfater is provided in their code, which is visually better but may have slightly worse metrics.
Quantitative results. Table 2 gives Chamfer distances, P2S distances, and normal image errors on the test set. Ground-truth SMPL meshes are used in FOF- and ICON (27]. Because of the weak perspective camera, the results of all the methods may not be aligned with the ground-truie -oound-truths on the view direction. Thus, we randomly sample 400000 points the estimated mesh and the ground-truth mesh, and align them by making their z-coordinates have the same mean. Our results have the lowest reconstruction ervors compared with others, which are slightly worse than PIFuHD in terms of the normal image error, coming the second place.
Qualitative results. Fig. 5 and Fig. show some visual results for challenging poses and various shapes, respectively. As shown
figures, our method is robust to various poses and shapes, and achieves detailed and reasonable reconstruction. FOF uses bandlimited approximation (keeping only the first 31 frequencies our implementation), inevitably discarding high-frequency components of detailed geometries. However, this also makes FOF capable of avoiding high- figh-frequency artifacts

---

Ground tratis
POF- FOF-Bese (Cuus)
FOF-SMPL (Ouss)
PIFaHD
PIFa
ICON
Figure 5: Comparison with other methods from single image with challenging human poses.
Inpor
(Oun)
(Oums)
1CON
Figure 6: Comparison with other methods from single image with various clothes.
as shown Fig. and Fig. This also consistent with the quantitative results in Sec. 4.3 PIFuHD uses a network with larger capacity in coarse-to-fie pipeline and visually better, which is more computation and memory demanding. These strategies can also used for our FOF-based reconstruction to enhance the results but would inevitably make the pipeline more cumbersome to some extent and might sacrify the merit of real-time implementation.
4.4 Ablation study
We show the effect of different N on the accuracy of the approximation in Fig. 7 When N < 12, obvious artifacts appear on the meshes. While the setting of N 15 preserves most geometric details, which is used in our implement. More details can be found in the supplementary document,
5 Conclusion and discussion
Conclusion. We propose novel efficient and flexible representation, Fourier Occupancy Field (FOF), for monocular real-time human reconstruction. FOF is aligned with the input image and

---

N
N = 12
N 15
Original Mesh
Figure 7: FOF with different N
can be estimated with simple image-to-image network. Moreover, FOF can represent a complex 3D shape in the form of multi-channel image, while preserving most geometric details, which is computationally efficient to extract the mesh and suitable for 3DTV applications. Experimental results indicate that our method can infer high-fidelity human models in real-time with state-of-the-art reconstruction quality.
Limitations. The FOF representation is based on the Fourier series expansion of the square-wave- like function. When the duty eycle of the square-wave function is small, the spectrum of the function will contain many high-frequency components. In such cases, we need more terms in the Fourier series to approximate the function. The number of terms need is roughly inversely proportional to the duty cycle. Therefore, FOF cannot represent the objects that are too thin. In future work, we will try wavelet transformation for better representation capability and apply common objects. Moreover, our FOF representation has great potential for other related tasks, such as 3D shape generation, completion of 3D human bodies and monocular 3D reconstruction for other shapes.
representation to
Broader Impact. The proposed method will promote the development of avatar generation and 3DTV, which is useful for applications that require accurate and efficient 3D reconstruction. However, this efficient acquisition may cause privacy and ethical problems. We suggest policymakers to establish efficient regulatory system and inform users about potential risks to avoid the disclosure of personal information.
6 Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (62122058. 62171317 and 62231018).
References
[1] A Aydin Alatan, Yucel Yemez, Ugur Gudukbay. Xenophon Zabulis, Karsten Muller, Çig- dem Eroglu Erdem, Christian Weigel, and Aljoscha Smolic. Scene representation technologies for 3DTV A survey. IEEE Trans. Circuits and Systems for Video Technology, 2007.
[2] Thiemo Alldieck, Gerard Pons-Moll, Christian Theobalt, and Marcus Magnor. Tex2shape: Detailed full human body geometry from single image. In Proc. ICCV, 2019.
[3] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. Photorcalistic monocular 3D reconstruction of humans wearing clothing. In Proc. CVPR, 2022.
[4] G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
10

---

[5] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. Openpose: Realtime multi- person 2d pose estimation using part affinity fields. IEEE Trans. Pattern Analysis and Machine Intelligence, 2019.
[6] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Collabo- rative regression of expressive bodies using moderation. In Proc. 3DV, 2021.
[7] Mare Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. DeepCap: Monocular human performance capture using weak supervision. Proc. CVPR, 2020.
[8] Mare Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. deeper look into deepcap. IEEE Trans. Pattern Analysis and Machine Intelligence, 2021.
[9] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto. Geo-PIFu: Geometry and pixel aligned implicit functions for single-view human reconstruction. In Proc. NeurlPS, 2020.
[10] Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung. Arch++: Animation-ready clothed human reconstruction revisited. In Proc. ICCV, 2021.
[11] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proc. CVPR, 2017.
[12] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proc. CVPR, 2018.
[13] Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black. PARE: Part attention regressor for 3D human body estimation. In Proc. ICCV, 2021.
[14] Kun Li, Hao Wen, Qiao Feng, Yuxiang Zhang, Xiongzheng Li, Jing Huang, Cunkuan Yuan, Yu-Kun Lai, and Yebin Liu. Image-guided human reconstruction via multi-scale graph transfor- mation networks. IEEE Trans. Image Processing, 2021.
[15] Ruilong Li, Yuliang Xiu, Shunsuke Saito, Zeng Huang, Kyle Olszewski, and Hao Li. Monocular real-time volumetric performance capture. In Proc. ECCV, 2020.
[16] Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robust high-resolution video matting with temporal guidance. In Proc. WACV, 2021.
[17] Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics, 2015.
[18] William Lorensen and Harvey Cline. Marching cubes: A high resolution 3d surface construction algorithm. Pro. SIGGRAPH, 1987.
[19] Ahmed A A Osman, Timo Bolkart, and Michael J. Black. STAR: A sparse trained articulated human body regressor. In Proc. ECCV, 2020.
[20] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proc. CVPR, 2019.
[21] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. #rXiv:2007.08501 2020.
[22] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PlFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. ICCV, 2019.
[23] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. PIFuHD: Multi-level pixel- aligned implicit function for high-resolution 3d human digitization. In Proc. CVPR, 2020.
[24] Peter-Pike Sloan, Jan Kautz, and John Snyder. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In Proc. SIGGRAPH, 2002.
11

---

[25] Yating Tian, Hongwen Zhang, Yebin Liu, and Limin Wang. Recovering 3d human mesh from monocular images: A survey. 0tt1::20030092... 2022.
[26] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE Trans. Puttern Analysis and Machine Intelligence, 2019.
[27] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black. ICON: Implicit Clothed humans Obtained from Normals. In Proc. CVPR, 2022.
[28] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In Proc. CVPR, 2021.
[29] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. PyMAF: 3D human pose and shape regression with pyramidal mesh alignment feedback loop. In Proc. 2021.
[30] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. PaMIR: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE Trans. Pattern Analysis and Machine Intelligence, 2021.
[31] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3D human reconstruction from single image. In Proc. ICCV, 2019.
[32] Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang. Detailed human shape estimation from a single image by hierarchical mesh deformation. In Proc. CVPR, 2019.
[33] Hao Zhu, Xinxin Zuo, Haotian Yang, Sen Wang, Xun Cao, and Ruigang Yang. Detailed avatar recovery from single image. IEEE Trans. Puttern Analysis and Machine Intelligence, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section4.1
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] In the supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
12

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [Yes] In the supplemental material.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] In the supplemental material.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "gvHTgxYFTWh", "reviewer": "Reviewer_mKrW", "summary": "The paper proposes a Fourier Occupancy Field representation to address the problem of monocular human mesh reconstruction from an image. In FOF, the 3D mesh is essentially converted into a multi-channel image that also possesses sampling scalability, stable reconstruction, and low-complexity reconstruction. Experiments demonstrate the proposed method outperforms existing methods quantitatively and qualitatively, while also enabling real-time applications.", "questions": "- Regarding the number of FOF components: It would be great to have a quantitative analysis of how this affects the performance. Also, it seems this number doesn't significantly impact run time since only tensor multiplication is needed. It would be great if the authors could explain why a maximum of 15 is used, i.e., what would happen if N>100?\n- Would the FOF representation be used in other related tasks such as general single image 3D prediction?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper are:\n- The paper is well-written. The discussions and explanations are not only informative but also insightful.\n- The proposed FOF representation is simple and effective. The methodology is general purpose and could benefit other related tasks.\n\nThe weaknesses of the paper are:\n- Minor improvements on paper writing: 1) A typo in Line 236 on the symbol subscript. 2) Table 1 would be much more informative if references could be added for each type of method. 3) It would be nice to specify if any post-processing techniques are used such as smoothing or connected components.\n- There is a significant performance gap, both quantitatively and qualitatively, between vanilla FOF and the enhanced version FOF-SMPL. However, in the main methodology section, the SMPL part was somehow downplayed. If this was done to emphasize FOF as the main novelty, it would be more convincing to apply FOF to other relevant tasks. Otherwise, if the focus is on 3D human reconstruction, the SMPL part should be included as an important subsection with more details.\n- The experimental section would be further richened. For instance, a quantitative ablation study on the number of FOF components would be great to help understand how it works. A runtime break-down would be also nice to shed light on how fast each component is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dHBS8kLaRHP", "reviewer": "Reviewer_AEcD", "summary": "This paper proposed an occupancy field representation for monocular real-time human reconstruction. The proposed representation represents a 3D object with a 2D field orthogonal to the view direction. Each 2D position stores the 1D occupancy field of the object along the view direction, and the 1D occupancy field is represented by real-valued fourier series. The proposed method is able to achieve 30+FPS while achieving comparable accuracy to other monocular methods.", "questions": "(1) With the fourier transformation of 1D the occupancy field given in Equation 6, what is the error bounds of the distance between recovered v.s. groundtruth surface point? \n\n(2) More visualization of the recovered 1D occupancy curves.\n\n(3) Discussion of different ways to represent bandlimited signals and its implication in the problem of interest.\n\n(4) In the discussion, the authors mentioned future works could incorporate wavelet transforms to improve results for thin objects. But it seems that would blow up the dimension of the representation with an naive implementation, and its computational advantage over voxel-based representation may diminish. I would be curious to learn if the idea of utilizing basic signal processing techniques advocated by the authors could indeed be generalized to more complex scenarios. \n\n", "limitations": "As pointed out by the author, the bandlimited approximation of the 1D occupancy field is insufficient to recover fine-details as well as thin structures. The poor results in hand and limb regions reveal such defects. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper proposes a real time monocular human reconstruction method with implicit occupancy representation. The occupancy field is represented as a pixel aligned 2D grid. The novel idea of this paper is, in each grid the 1D occupancy signal is represented by real-valued fourier series.\n\nThe paper shows that 1D occupancy signals tend to be low frequency, thus can be effectively represented by the first few fourier basis functions. This reduces the output dimension of the inference model, thus effectively improves computational efficiency. \n\nWeakness: approximating occupancy with a very small number of fourier basis is still quite counter intuitive. I think the author could do a better job in analyzing the property of this approximation. It would be nice to see what is the theoretical bound of the distance to the surface given Equation 6. It would also be nice if the authors visualize the reconstructed 1D occupancy curve given different number of fourier basis. In addition, representing signals with a discrete set of uniformly sampled fourier basis is equivalent to applying sinc filter to regular-grid delta signals. In other words, there are different ways of representing a band-limited signal (e.g. spatial vs frequency representation), thus it would be helpful if the author provides some insight what is the benefits of representing signals in the frequency v.s. spatial domain. \n\nIn general, I recognize the practical advantage of the proposed method, but feels the technical / theoritical contribution of the paper is a bit thin. The paper in general follows the engineering details of other methods such as PIFU and ICON and the difference is mainly the simplified way of representing occupancy signals with fourier transforms. Such strategy has been seen in use in other problems, and it inhirits the defects of bandlimited approximations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "b76tx2saz3", "reviewer": "Reviewer_mkLu", "summary": "This paper proposes a new representation for human reconstruction. Specifically, for a pixel in the input image, it approximates the occupancy along the casted ray as the Fourier series. The proposed approach learns the representation from data by regressing to the coefficients of the Fourier basis functions. The submission shows the efficacy and flexibilities of the representation on real captured data.", "questions": "Essentially, the training of FOF will be a per-pixel regression problem. Namely, the network needs to regress to GT coefficients $C(x, y)$ for Fourier basis functions. Since the per-pixel regression is a hard problem, I am curious about how robust the FOF representation is to the inaccuracy of such regression. Specifically, I think showing the following results would be beneficial:\n1. The training loss of relative error (absolute error may not be that informative as the magnitude matters)\n2. Manually add noises to the GT coefficients and plot a curve to show the correlation between the perturbed coefficients and the quantitative quality of the reconstructed meshes.", "limitations": "I appreciate the authors's discussions about limitations of the proposed approach in Sec. 5, which helps the understanding of the suitable scenarios.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Originality: the proposed representation of FOF is novel. Approximating the occupancy with Fourier series is interesting and principled.\n2. Quality: the presented reconstruction is of high-quality.\n3. Clarity: the paper is well-written and easy to follow.\n4. Significance: it is an important problem of developing an efficient and powerful 3D representation to enable real-time application. The submission demonstrates a real-time pipeline with the FOF representation.\n\n## Weakness\n\nI do not see some major weakness. But it would be great if authors can discuss more about the quality of the reconstruction. For example, in Fig. 4, Fig. 5, and video in the supplementary, I observe that PIFuHD usually maintains more details even though FOF-Normal already utilizes the inferred normal maps. It would be beneficial to discuss why FOF representations are not detailed than PIFuHD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept