PORF: POSE RESIDUAL FIELD FOR ACCURATE NEURAL SURFACE RECONSTRUCTION
Authors: Jia-Wang Bian
Wenjing Bian
Victor Adrian Prisacariu
Philip Torr
Venue: ICLR 2024 poster (2024)
Keywords: Neural Surface Reconstruction
Camera Pose Estimation
Neural Radiance Field
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Neural surface reconstruction is sensitive to the camera pose noise, even when state-of-the-art pose estimators like COLMAP or ARKit are used. Existing Pose-NeRF joint optimisation methods have struggled to improve pose accuracy in challenging real-world scenarios. To overcome the challenges, we introduce the pose residual field (PoRF), a novel implicit representation that uses an MLP for regressing pose updates. Compared with the conventional per-frame pose parameter optimisation, this new representation is more robust due to parameter sharing that leverages global information over the entire sequence. Furthermore, we propose an epipolar geometry loss to enhance the supervision that leverages the correspondences exported from COLMAP results without the extra computational overhead. Our method yields promising results. On the DTU dataset, we reduce the rotation error of COLMAP poses by 78\%, leading to the reduced reconstruction Chamfer distance from 3.48mm to 0.85mm. On the MobileBrick dataset that contains casually captured unbounded 360-degree videos, our method refines ARKit poses and improves the reconstruction F1 score from 69.18 to 75.67, outperforming that with the provided ground-truth pose (75.14). These achievements demonstrate the efficacy of our approach in refining camera poses and improving the accuracy of neural surface reconstruction in real-world scenarios.

Published as a conference paper at ICLR 2024
PORF: POSE RESIDUAL FIELD FOR ACCURATE NEU- RAL SURFACE RECONSTRUCTION
Mia-aangaa Bian Department of Engineering Science University of Oxford
Wenjing Bian
Department of Engineering Science University of Oxford
Victor Adrian Prisacariu
Philip Torr
Department of Engineering Science University of Oxford
Department of Engineering Science University of Oxford
ABSTRACT
Neural surface reconstruction is sensitive to the camera pose noise, even if state- of-the-art pose estimators like COLMAP or ARKit are used. More importantly, existing Pose-NeRF joint optimisation methods have struggled to improve pose accuracy in challenging real-world scenarios. To overcome the challenges, in- troduce the pose residual field (PoRF), a novel implicit representation that uses MLP for regressing pose updates. This is more robust than the conventional pose parameter optimisation due to parameter sharing that leverages global infor- mation over the entire sequence. Furthermore, we propose an epipolar geometry loss to enhance the supervision that leverages the correspondences exported from COLMAP results without the extra computational overhead. Our method yields promising results. On the DTU dataset, we reduce the rotation error by 78% for COLMAP poses, leading to the decreased reconstruction Chamfer distance from 3.48mm to 0.85mm. On the MobileBrick dataset that contains casually captured unbounded 360-degree videos, our method refines ARKit poses and improves the reconstruction FI score from 69.18 to 75.67, outperforming that with the dataset provided ground-truth pose (75.14). Moreover, we integrate our method into the Nerfstudio library, leading consistently improved performance in diverse chal- lenging scenes. These achievements demonstrate the efficacy of our approach in refining camera poses in real-world scenarios.
INTRODUCTION
Object reconstruction from multi-view images is fundamental problem in computer vision. Re- cently, neural surface reconstruction (NSR) methods have significantly advanced in this field Wang let al. 20)]aa); Wu al. (2023). These approaches draw inspiration from implicit scene represen- lation and volume rendering techniques that were used in neural radiance fields (NeRF) Mildenhall| (2020). In NSR, scene geometry is represented by using signed distance function (SDF) field, learned by multilayer perceptron (MLP) network trained with image-based rendering loss. Despite achieving high-quality reconstructions. these methods sensitive camera pose noise, common issue in real-world applications, even when state-of-the-ar pose estimation methods like SchhnerSchönberger
Frahm (2016) or ARKit are used. An example of this sensitivity is evident in Fig. D where the reconstruction result with the COLMAP estimated poses shows poor quantitative accuracy and visible noise on the object's 's surface. In this paper, we focus on refining the inaccurate camera pose improve neural surface reconstruction.
Recent research has explored the joint optimisation of camera pose and NeRF, while they are not designed for accurate neural surface reconstruction in real-world scenes. As shown in Fig. [ exist- ing methods such as BARF Lin et al. (2021) and SPARF Truong et al. (2023) are hard to improve reconstruction accuracy the DTU dataset Jensen al. (2014) via pose refinement. We regard the challenges as stemming from independent pose representation and weak supervision. Firstly, existing methods typically optimise pose parameters for each image independently. This approach

---

Published as a conference paper at ICLR 2024
Reference Image
GT Pose (0.65)
COLMAP Pose (3.60)
TEUE TT
BEECE
BARF Pase (3.95)
SPARF Pose (3.34)
Our Pose (0.64)
FUNEE
BEUKE!
BERRE
Figure I: Reconstruction results on the DTU dataset (scan24). All meshes were generated by us- ing Voxurf Wu et al. (2023). The Chamfer Distance (mm) is reported. BARF Lin et al. (2021)
SPARF Truong et al. (023)) , and our method all take the COLMAP pose as the initial pose. More results are illustrated in the supplementary material.
disregards global information over the entire sequence and leads to poor accuracy. Secondly, the colour rendering loss employed in the joint optimisation process exhibits ambiguity, creating many false local minimums. To overcome these challenges, we introduce a novel implicit pose represen- tation and a robust epipolar geometry loss into the joint optimisation framework.
The proposed implicit pose representation is named Pose Residual Field (PoRF), which employs an MLP network to learn the pose residuals. The MLP network takes the frame index and initial camera pose inputs, as illustrated in Fig. [ Notably, as the MLP parameters are shared across all frames, it is able to capture the underlying global information over the entire trajectory for boosting performance. As shown in Fig. 3 the proposed PoRF shows significantly improved accuracy and faster convergence than the conventional pose parameter optimisation approach.
Moreover, we use feature correspondences in the proposed epipolar geometry loss to enhance su- pervision. Although correspondences have been used in SPARF Truong et al. (2023), it depends on expensive dense matching. As shown in Fig. 3 our method demonstrates similarly excellent per- formance with both sparse matches by handeraft methods (SIFT Lowe [Lwe by deep learning methods (LoFTR Sun et al. (202I)). Besides, in contrast to SPARF which fuses correspondences and NeRF-rendered depths to compute reprojection loss, approach epipolar geometry loss that solely involves poses and correspondences, without relying on the ren- dered depths by NeRF that can be inaceurate and limit pose accuracy. Notably, as NeRF rendering is time-consuming, SPARF is constrained in the number of correspondences it can use in each iter- ation, while our method allows for the utilisation of an arbitrary number of matches.
(2004) and dense matches
the
We conduct a comprehensive evaluation on both DTU Jensen et al. (2014) and MobileBrick Li et al. (2023a) datasets. Firstly, on the DTU dataset, our method takes the COLMAP pose as initialisation and reduces the rotation eтror by 78%. This decreases the reconstruction Chamfer Distance from 3.48mm to 0.85mm, where we use Voxurf Wu et al.) (2023) for reconstruction. Secondly, on the MobileBrick dataset, our method is initialised with ARKit poses and increases the reconstruction Fl score from 69.18 to 75.67. slightly outperforms the provided imperfect GT pose in reconstruc- tion (75.14) and achieves state-of-the-art performance. Moreover, we integrate our method into the Nerfstudio library/ Tancik et al. (2023), and the resulting method outperforms the original pose opti- misation method in different challenging scenes. These results demonstrate the effectiveness of our approach in refining camera poses and improving reconstruction accuracy, especially in real-world scenarios where the initial poses might not be perfect.
2

---

Published as a conference paper at ICLR 2024
In summary, contributions are follows:
1. We introduce novel approach that optimises camera pose within neural surface recon- struction. It utilises the proposed Pose Residual Field (PoRF) and robust epipolar geom- etry loss, enabling accurate and efficient refinement of camera poses.
2. The proposed method demonstrates its effectiveness in refining both COLMAP and ARKit poses, resulting in high-quality reconstructions on the DTU dataset and achieving state-of- the-art performance the MobileBrick dataset.
RELATED WORK
Neural surface reconstruction. Object reconstruction from multi-view images is fundamen- tal problem in computer vision. Traditional multi-view stereo (MVS) methods Schönberger et al.) (0016); Furrkawa et al. (2009) explicitly find dense correspondences across images to compute depth maps, which are fused together to obtain the final point cloud. The correspondence search significantly boosted by the deep learning based approaches Yao et al. (2018) 2019); Zhang et al. (2020a). Recently, Neural Radiance Field (NeRF) Mildenhall et al. (2020) has been proposed to implicitly reconstruct the scene geometry, and it allows for extract- ing object surfaces from the implicit representation. To improve performance, VoISDF Yariv et al.) (202T) and NeuS Wang al. (2021a) proposed to an implicit SDF field for
and depth estimation processes
representa- tion. MonoSDF Yu et al. (2022) and NeurlS Wang et al. (2022a) proposed to leverage the external monocular geometry prior. HF-NeuS Wang et al. (2022b) introduced an extra displacement network for learning surface details. NeuralWarph Darmon DarpDooon et al. (2022) used patch warping to guide surface optimisation. Geo-NeuS Fu al. (2022) and RegSDF Zhang al. (2022) proposed to leverage the sparse points generated by SiM. Neuralangelo Li et al. (2023b) proposed coarse-to-fine method on the hash grids. Voxurf et al. (203) proposed voxel-based representation that achieves high accuracy with efficient training.
Joint NeRF and pose optimisation. NeRFmm Wang et al. (2021b) demonstrates the possibility of jointly learning or refining camera parameters alongside the NeRF framework Mildenhall et al. (2020). BARF Lin et al. (2021) introduces a solution involving coarse-to-line positional encoding to enhance the robustness of joint optimisation. In the case of SC-NeRF Deong et al. (202T), both intrinsic and extrinsic camera parameters are proposed for refinement. In separate approaches, SiN- eRF et al. (0022) and GARF et al. (2022) suggest employing distinct activation functions within NeRF networks to facilitate pose optimiation. Nope-NeRI Bian et al. (2023) employs an ex- ternal monocular depth estimation model to assist refining camera poses. 1.2G Chen et al. (2023) puts forth local-to-glote -olobblll scheme, wherein the image pose is computed from learned multiple lo- cal poses. Notably similar to our approach SPARF Truong et al. (2023), which also incorporates correspondences and evaluates its performance the challenging DTU dataset et al.) (2014).
However, it is tailored to sparse-view scenarios and lacks the accuracy achieved by our method when an adequate number of images is available.
PRELIMINARY
Volume rendering with SDF representation We adhere to the NeuS Wang et al. (2021a) method- ology to optimise the neural surface reconstruction from multi-view images. The approach involves representing the scene using an implicit SDF field, which is parameterised by an MLP. To render an image pixel, a ray originates from the camera centre o and extends through the pixel along the viewing direction
+ tult 2 0}. By employing volume rendering (Max) 1995), the colour for the image pixel is computed by integrating along the ray using N discrete sampled points (p, o tæuli 1, I,..., N,t, < t:4)):
described as {p(t)
((r) Σ Tacc T. II (1-az),
(I)
3

---

Published as a conference paper at ICLR 2024
In this equation, ag represents the opacity value, and T, denotes the accumulated transmittance. The primary difference between NeuS and NeRF Mildenhal et al. (2020) lies in how a, is formulated. In NeuS, a; is computed using the following expression:
a; = max ( - -0 (.(((((++11)) ,0)
(x(((((t)))
In this context, f() is the SDF function, and (x) (1 te - denotes the Sigmoid function, where the parameter is automatically learned during the training process.
Neural surface reconstruction loss. In line with NeuS Wang al. (2021a), our objective is to minimise the discrepancy between the rendered colours and the ground truth colours. To achieve this, we randomly select batch of pixels and their corresponding rays in world space P {Cx, Ok, ) from an image in each iteration. Here, C represents the pixel colour, o is the camera centre, and Uk denotes the viewing direction. We set the point sampling size and the batch size as m.
The neural surface reconstruction loss function is defined follows:
NNR Lootour + Xrreg
(3)
The colour loss Lestour is calculated as:
Σ Cr Cill,
Leatour
(4)
The term Lreg incorporates the Eikonal term Gropp et al. (2020) applied to the sampled points to regularise the learned SDF, which can expressed as:
Lreg
Σ [(VV/Ox))) 1)?.
(5)
Here, f is the learned SDF, and PA. represents the sampled points for the k-th pixel in the i-th ray. controls the influence of the regularisation term in the overall loss function.
Joint pose optimisation with NSR. Inspired by the previous work Wang et al. (2021b): Lin al. (202I) that jointly optimises pose parameters with NeRF Mildenhal et al. (2020). we propose a naive joint pose optimisation with neural surface reconstruction and Eqn. 3 aas the loss function.
This is denoted as the baseline method in Fig. 3 and in the next section, we present the proposed method for improving performance.
METHOD
Pose residual field (PoRF). We approach camera pose refinement as a regression problem using an MLP. In Fig. 2 we illustrate the process where the input to the MLP network is a 7D vector, consisting of 1D image index (i) and a 6D initial pose (r, t). The output of the MLP a 6D pose residual (Ar, At). This residual is combined with the initial pose to obtain the final refined pose (r','). t¹). In this context, rotation is represented by 3D axis angles.
To encourage small pose residual outputs at the beginning of the optimisation, we multiply the MLP output by a fixed small factor (a). This ensures an effective initialisation (the refined pose is close to the initial pose), and the correct pose residuals can be gradually learned with optimisation. Formally, the composition of the initial pose and the learned pose residual expressed as follows:
f - Ar i=tta At
(6)
4

---

Published as a conference paper at ICLR 2024
with poisy poses
Correspondences
LBG
i-N
Fe
G
Volums Buonketing Pereeeeee SDR
--. Lectour
6,,
(f, 4)
D Leee
PoRF
NSR
Figure 2: Joint optimisation pipeline. The proposed model consists of a pose residual field Fg and neural surface reconstruction module Gg. PoRF takes the frame index and the initial camera pose as input and employs an MLP to learn the pose residual, which composited with the initial pose to obtain the predicted pose. The output pose is used to compute the neural rendering losses with the NSR module and the epipolar geometry loss with pre-computed 2D correspondences. Parameters and @ updated during back-propagee procc-prppggtionn
The resulting final pose (f, t) is utilised to compute the loss, and the gradients can be backpropagated shallow MLP with to the MLP parameters during the optimisation process. In practice, hidden layers, which we find sufficient for object reconstruction.
Intuition on the PoRF Our method draws inspiration from the coordinates-ML.3 Similar to NeRF Mildenhall et al. 2020). in which the position and view direction of a point are taken as inputs by the MI.P to learn density and colour, our method takes the initial position (translation) and view direction (rotation) of the camera inputs to learn the camera's pose residual. Also, approach incorporates the frame index as an additional input, serving as temporal coordinate.
The MLP parameters are shared across all frames, which enables our method to extract global infor- mation from the entire sequence. The inclusion of the frame index allows our method to learn local information for each framee Overall, the utilisation of both global and local information empowers our method to effectively handle various challenges and learn accurate camera poses.
The parameter sharing in method leads to better robustness than the conventional pose refine- ment method. Specifically, the previous method directly back-propagates gradients to per-frame pose parameters, so the noisy supervision and inappropriate learning rates can make certain pose parameters trapped in false local minima, causing the optimisation to diverge. In contrast, our method can leverage information from other frames to prevent certain frames from being trapped in false local minima. A quantitative comparison is illustrated in Fig. where the performance of L2 (utilising PoRF) notably surpasses that LI (excluding PoRF).
Epipolar geometry loss. To improve the supervision, introduce epipolar geometry loss that makes use of feature correspondences. The correspondences can be either sparse or dense, and they be obtained through traditional handerafted descriptors, like Lowe (2004), deep learning techniques, like LoFTR Sun et al. (202I). As a default option, we utilise SIFT matches, exported from COLMAP pose estimation results, to avoid introducing extra computational cost.
In each training iteration, we randomly sample matched image pairs and calculate the Sampson distance (Eqn. ( for the correspondences. By applying threshold 6, can filter outliers and obtain m inlier correspondences. We represent the inlier rate for each pair as and the Sampson error for each inlier match as eg. The proposed epipolar geometry loss is then defined as:
LEG =
Σ wi
Σ ck
(7)
m

---

Published as a conference paper at ICLR 2024
Table 1: Absolute pose accuracy evaluation the DTU dataset. For all methods, the COLMAP
pose is used as the initial pose. The upper section presents rotation errors in degrees, while the lower section displays translation errors millimetres (mm).
Scan
mean
COLMAP (Initial)
0.68
0.67
0.63
0.62
0.75
0.67
0.64
0.68
0.56
0.66
0.62
0.69
0.66
0.64
0.63
0.65
BARF Tin al.
2021
0.67
0.73
0.67
0.79
0.67
0.83
0.47
0.59
0.55
0.62
0.75
0.67
0.67
0.70
0.84
0.69
L20 hen el m 2023
0.67
0.65
0.69
32.04
0.80
0.68
0.49
0.66
0.76
0.61
29.02
6.41
0.69
2.63
2.47
5.29
SPART Triinng Ours
2023
0.40
0.36
0.39
0.09
0.13
0.10
0.13
0.13
0.16
0.13
0.12
0.26
0.12
0.12
0.16
0.15
0.11
0.13
0.14
COLMAP (Initial)
0.79
1,06
0.81
0.81
0.80
0.89
0.88
1.07
2.07
1.07
1.05
1.22
1.03
0.95
1.02
1.03
BARF et al 202
1.53
2.19
1.80
2.50
2.32
2.31
2.06
2.65
3.45
2.27
2.29
2.59
3.29
3.15
3.13
2.50
Che
2025
1.30
1,90
115
2.07
2.45
172
21.1
28.9
SPARF
Troong
a 2023
3.66
4.76
3.89
2.22
5.68
2.57
2.32
2.96
4.88
3.21
2.94
2.85
1.96
2.23
2.65
3.25
0.77
008
1.03
1.02
Table Reconstruction accuracy evaluation on the DTU dataset. The evaluation metric is Chamfer distance, expressed in millimetres (mm).
Method
Pose
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
NoRE Mildenhall
2020
1.83
2.39
1.79
0.66
1.79
1.44
1.20
1.96
1.44
2.61
1.04
1.13
0.99
1.54
IDR Yary et a. 2020
1.63
1.87
0.63
0:48
1.04
0.79
0.77
1.33
1.16
0.76
0.67
0.90
0.42
0.51
0.53
0.90
VotSD
2021
GT
0.70
1.29
0.86
PointNaKI Xu of
2172
0.87
2.06
1.20
1:01
1.01
1.39
0.80
1.04
0.92
0.74
0.97
0.76
0.56
0.90
1.05
1.02
NeuS Wang et
21214
0.83
0.98
0.56
0.37
1.13
0.59
0.60
1.45
0.95
0.52
1.43
0.36
0.45
0.45
0.77
Voxur Wa
0.65
0.74
0.39
0.35
0.96
0.64
0.85
1.58
1.01
0.68
0.60
1.11
0.37
0.45
0.47
0.72
COLMAP
3.60
4.36
3.62
3.00
4.44
3.52
2.64
4.11
2.85
3,86
2.98
3,86
2.76
3.26
3.34
3.48
Voxurf
BARF
3.95
4.35
3,84
3.02
4.38
4.09
2.02
3.68
2.56
3.62
3.61
4.06
2.63
3.04
4.14
3.53
SPARF
1.35
6.17
2.81
1.85
3.24
2.17
2.05
1.03
1.62
2.45
2.89
Ours
0.64
0.84
0.59
0,48
1.10
1.02
0.90
1.43
1.10
0.83
0.86
1.11
0.58
0.65
0.67
0.85
Here, WN pi is the loss weight that mitigates the influence of poorly matched pairs. The com- putation of the Sampson distance between two points, denoted as z and x', with the fundamental matrix F obtained from the (learned) camera poses and known intrinsies, expressed as:
Fz)) dsampson (I, F),F)= (Fx) + (Fx)² (FT +
(8)
Here, subscripts and 2 refer to the first and second components of the respective vectors.
Training objectives. The overall loss function in our proposed joint optimisation method com- bines both Eqn. and Eqn. 7 follows:
C + BLEG,
where hyperparameter that controls loss weights. The first term LNSR represents the loss related to neural surface reconstruction, which backpropagates the gradients to both PoRF and ren- dering network parameters. The second term LEG represents the loss associated with the epipolar geometry loss, which backpropagates the gradients to PoRF only.
EXPERIMENTS
Experiment setup. We perform our experiments on both the DTU Jensen al. (2014) and Mo- bileBrick Li et al. (2023a) datasets. Following previous methods such as Wu et al. (0023 and Li al. (203aa), our evaluation uses the provided 15 test scenes from DTU and 18 test scenes from MobileBrick. We assess both the accuracy of camera poses and the quality of surface reconstruc- tions. The baseline methods for comparison includes BARF Lin et al. (02TI). L2G Chen et al. 2023), and SPARF Truong al. (2023). For fair comparison, all methods, including ours, take the same initial pose inputs- namely, COLMAP poses on the DTU dataset and ARKit poses on the MobileBrick dataset. Here, consider all methods to be operating in two-stage manner, only the pose refinement is conducted by these methods, and we utilise Voxurf et al. (2023) with the refined poses for the reconstruction step for all methods.

---

Published as a conference paper at ICLR 2024
Table 3: Absolute pose accuracy the MobileBrick dataset. The ARKit pose the initial pose for all methods. Note that the provided GT pose in this dataset is imperfect.
Metries
Rotation Error (Deg.)
Translation Error (mm)
Scene
ARKit
BARF
L2G
SPARF
Ours
ARKit
BARF
L2G
SPARF
Ours
aston audi
0,43
0.22
0.22
0.22
0.14
2,49
1.21
1.21
1.01
0.70
0.64
0.46
0.52
1.10
0.28
3.38
2.25
2.24
4.35
1.11
beetles
0.49
0.52
0.57
0.17
0.14
1.29
2.06
1.21
1.23
big_ben
0.44
0.33
0.49
0.27
0.26
4.23
3.03
2.13
2.38
1.82
boat
0.61
0.24
0.38
0.26
0.22
1.51
0.97
1.29
1.12
0.98
bridge
0.44
0.22
0.18
0.32
0.22
1.08
1.29
1.12
1.26
0.78
cabin
0.61
0.34
0.32
0.30
0.26
1.34
1.44
1.55
1.09
0.93
camera
0.30
0.23
0.26
0.17
0.18
1.06
1.07
0.67
0.61
castle
0.55
4.77
4.79
4.77
0.30
2.42
17.1
21.1
24.12
2.55
colosscum
0,46
0.49
0.35
0.95
0.28
3.67
4,21
4.41
11.26
3.25
convertible
0.40
0.27
0.25
0.23
0.13
1,65
1.18
1.21
1.31
0.92
ferrari
0.43
0.20
0.20
0.23
0.13
1.89
1.10
1.19
0.80
0.59
jeep
0.34
0.24
0.22
0.28
0,23
0.98
1.13
1.08
0.65
0.50
london_bux
0.72
0.70
0.73
0.50
0.42
3.62
2.97
2.98
3.61
2.63
motorcycle
0.34
0.26
0.29
0.27
0.20
0.97
1.01
1.10
0.82
0.60
porsche
0.28
0.19
0.20
0,20
0.15
1.29
1.16
1.18
0.88
0.71
satellite
0.41
0.23
0.26
0.15
0.17
0,92
1.11
0.97
0,98
space_shuttle
0.31
0.28
0.26
0.58
0.18
1.45
1.81
1.76
3.43
1.30
mean
0.46
0.57
0.58
0.61
0.22
1.90
2.50
2.84
3.39
1.23
Table Reconstruction accuracy evaluation on the MobileBrick dataset. Note that the provided GT pose in this dataset is imperfect. The results are averaged over all 18 test scenes.
Methods
2.5mm Rec. (%r 1
SwATn Rec. (%) 1
Channer J
Accu. (%)
Accu. (%)
(mm)
TSDD-Fusion
BNV Fusion Cret al Ki1
41.77
25.96
33.27
71.20
47.09
55.11
9.60
Neural-RGBD Azanove
2032
20.61
10.66
13.67
39.62
22.06
27.66
22.78
COLMAP Schomberper
2016
74.89
68.20
71.08
93.79
84.53
88.71
5.26
Mildenhall N
NeRF
2040
47.11
40.86
43.55
78.07
69.93
73.45
7.98
NeuS Wang
77.35
70.85
73.74
93.33
86.11
89.30
4,74
Voxui Wu 2023
78.44
72,41
75.13
93.95
87.53
90.41
4.71
ARKit
71.90
66.98
69.18
91.81
85.71
88.43
5,30
Voxurl
BARF
77.12
71.36
73.96
93.47
87.56
90.22
4.83
SPARF
76.10
70.44
72.99
92.73
86.66
89.39
4,95
Ours
79.01
72.94
75.67
94.07
87.84
90.65
4.67
Implementation details. The hyperparameters used for training the surface reconstruction align with those utilised in NeuS Wang et al. (2021a). To calculate the proposed epipolar geometry loss, we randomly sample 20 image pairs each iteration and distinguish inliers and outliers by using threshold of 20 pixels. The entire framework undergoes training for 50,000 iterations, which takes 2.5 hours in one NVIDIA A40 GPU. However, it's noteworthy that convergence is achieved rapidly, typically within the first 5,000 iterations. More details are in the supplementary materials.
COMPARISONS
Results on DTU. The pose refinement results are presented in Table T It shows that BARF Lin et al. (02T) and L2G Chen et al. (023) are unable to improve the COLMAP pose on DTU, and several scenes. While SPARF Truong al. (2023) shows slight improvement in rotation, it remains limited, and results in worse translation. Besides, we also evaluated Nope- NeRFBBan Bian et al. (2023), it diverged on DTU and MobileBrick scenes due the bad initialisation of the depth scale. Overall, our proposed method achieves substantial reduction of 78% in rotation error and shows minor improvement translation error.
L2G diverges
The reconstruction results are presented in Tab. where we run Voxurf Wu et al. 2023 for recon- struction. It shows that the reconstruction with the COLMAP pose falls behind previous methods that use the GT pose. BiRFLin al. (202T) 1 hard to improve performance. While SPARFTruong jet al. (2023) can slightly improve performance, it is limited. When employing the refined pose by method, the reconstruction accuracy becomes comparable most of the previous methods that

---

Published as a conference paper at ICLR 2024
Table 5: Novel view rendering results the Nerfstudio dataset. We replace the pose optimisation module in the Nerfacto with our PoRF MLP, leading to consistently improved novel view rendering accuracy. Note that we do not use the proposed loss function here.
Ours
Nerfacto
Scenes
PSNR
SSIM T
LPIPS +
PSNR
SSIM
LPIPS
Egypt
23.08
0.743
0.245
20.75
0.623
0.305
person
25.87
0.742
0.261
24.33
0.643
0.299
kitchen
20.72
0.796
0.271
19.60
0.73
0.298
plane
22.44
0.672
0.356
19.72
0.557
0.440
dozer
21.16
0.760
0.266
20.26
0.716
0.279
floating tree
20.82
0.769
0.204
20.67
0.733
0.211
aspen
17.16
0.344
0.611
16.48
0.292
0.657
stump
22.62
0.820
0.194
21.96
0.753
0.213
sculpture
22.28
0.692
0.296
21.74
0.652
0.315
Giannini Hall
19.29
0.590
0.406
17.66
0.491
0.444
mean
21.54
0.693
0.311
20.32
0.619
0,346
the GT pose, although there is still minor gap between our method and Voxurf with the GT pose, i.e., 0.85mm vs. 0.72mm.
The pose results are summarised in Tab. We initialise all baseline methods, including ours, with the ARKit pose. Note that the provided GT pose in this dataset is imperfect, which obtained by pose refinement from human labelling. The results show that our method is closer the GT pose than others. More importantly, our method can consistently improve pose accuracy in all scenes, while other methods cannot.
Results on MobileBrick.
The reconstruction results presented in Tab. It shows that the refined pose by all methods can lead to an overall improvement in reconstruction performance. Compared with BARF Lin et al. (202I) and SPARF Truong et al. (2023), our method shows more significant improvement. As result, our approach surpasses the provided GT pose in terms of reconstruction Fl score, i.e. 75.67 vs 75.13, achieving state-of-the-an -fettherrt performance on the MobileBrick dataset.
5.2 GENERALISATION
The proposed MLP-based pose optimisation method can be readily plugged into different NiRF-like systems to improve performance. To demonstrate the universal use of our method, integrate it into the Nerfstudio library Tancik et al.] (2023), which is modular framework for neural radiance field development. We compare method with the recommended baseline method (Nerfacto), which integrates multiple methods into one, including pose optimisation. For a fair comparison, we replace the original pose parameter optimisation module with our proposed PoRF MI.,, and do not use the proposed loss function. Tab. provides the evaluation results on the Nerfstudio dataset. The dataset comprises ten in-the-wild, 360-degree captures obtained using either mobile phone or a mirror-less camera with fisheye lens. The data was processed using COLMAP or the Polycam app to obtain camera poses and intrinsic parameters. It shows that our method can consistently boost novel view rendering performance in diverse scenes.
5.3 ABLATION STUDIES
We conduct extensive ablation studies the DTU dataset Jensen et al. (2014). The pose errors during training are shown in Fig. where we compare six different settings and we can draw some key observations from the ablation studies:
PoRF vs pose parameter optimisation: L2 surpasses LI in terms of accuracy, thereby high- lighting the benefits of utilising the Pose Residual Field (PoRF) contrast to conventional pose parameter optimisation within the same baseline framework, where solely the neural rendering loss employed. Upon the integration of the proposed epipolar geometry loss L with the baseline approach, 1.4 outperforms L.3, substantiating the consistency of the observed outcomes.
8

---

Published as a conference paper at ICLR 2024
Reference image
Initial pose
Refined pose
Chamfer (mm) 1=
4.36
0.84
F1 f:
83.28
89.01
Figure 4: Reconstruction results on DTU (top) and MobileBrick (bottom) datasets. The initial pose denotes the COLMAP pose DTU, and the ARKit pose MobileBrick. We use the standard evaluation metrics, i.e., Chamfer distance (mm) for DTU and score for MobileBrick.
Efficacy of the epipolar geometry loss: L3 demonstrates faster convergence when compared to L11 offering substantiation of the efficacy of the proposed T.E loss within the context of pose parameter optimisation. Upon incorporating the proposed Pose Residual Field (PoRF), L4 surpasses L2, thus reinforcing the same conclusion.
Ablation over correspondences: we compare handcrafted matching methods (L.4 representing
1. Baseline (D)
B4
SIFT Lowe (2004) with deep learning- based methods (L5 denoting LoFTR Sun et al. (202I)).
Rotation emoor (dagreas)
0.5
Be PoRF+LL (Ours)
On average, SIFT yields 488 matches per im-
LoTTR
age pair, while LoFTR generates a notably higher count of 5394 matches. The results indicate that both methods achieve remarkably similar levels of accuracy and convergence. It's worth noting that the SIFT matches are derived from
0.3
COLMAP results, so their utilisation does not en- tail additional computational overhead.
10k
ning iterations
Ablation over scene representations: We sub- stitute the SDF-based representation (L4) with NeRF representation MMileenhall al. (2020), denoted L6. This adaptation yields comparable
Figure 3: Pose
during training the DTU
dataset. The results are averaged over 15 scenes. Baseline (B) denotes the naive joint optimisation of NSR and pose parameters.
performance, highlighting the versatility our method across various implicit representations.
CONCLUSION
This paper introduces the concept of the Pose Residual Field (PoRF) and integrates with an epipo- lar geometry loss to facilitate the joint optimisation of camera pose and neural surface reconstruction. Our approach leverages the initial camera pose estimation from COLMAP or ARKit and employs these newly proposed components to achieve high accuracy and rapid convergence. By incorporat- ing our refined pose in conjunction with Voxurf, a cutting-ede surface reconstruction technique, we achieve state-of-tt oftthe-rrtttttt reconstruction performance the MobileBrick dataset. Furthermore, when evaluated on the DTU dataset, our method exhibits comparable accuracy to previous approaches that rely ground-truth pose information.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
The authors gratefully acknowledge the financial support provided by Apple. This work is also supported by the UKRI grant: Turing AI Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1. We also thank the Royal Academy of Engineering and FiveAl.
REFERENCES
Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In CVPR, pp. 6290-6301, June 2022.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Wenjing Bian, Zirui Wang, Kejie Li, Jia- Via-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. In CVPR, pp. 4160-4169, 2023.
Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. tocal-to-global registration for bundle-adjusting neural radiance fields. In CVPR, pp. 8264-8273, 2023.
Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. Garf: gaussian activated radiance fields for high fidelity reconstruction and pose estimation. arXiv e-prints, pp. arXiv- 2204, 2022.
François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR, pp. 6260 2022.
Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geomeery-cconiistent neural implicit surfaces learning for multi-view reconstruction. NeurlPS, 35:3403-3416 2022.
Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski. Reconstructing building interiors from images. In ICCV, pp. 80 80-87. IEEE, 2009.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regu- larization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In CVPR, pp. 406-413, 2014. doi: 10. 10.11099CVR.22044..59
Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In ICCV, pp. 5846-5854, 2021.
Kejie Li, Yansong Tang, Victor Adrian Prisacariu, and Philip HS Torr. Bnv-fusion: Dense 3d recon- struction using bi-level neural volume fusion. CVPR, 2022.
Kejie Li, Jia-Wang Wang Bian, Robert Castle, Philip HS Torr, and Victor Adrian Prisacariu. Mobilebrick: Building lego for 3d reconstruction mobile devices. In CVPR, pp. 4892449944444 4901, 2023a.
Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In CVPR, 2023b.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey, Barf: Bundle-adjusting neural radiance fields. ICCV, pp. 5741-5751, 2021.
David G Lowe. Distinctive image features from ocale-invariant keypoints. IJCV, 60:91-110, 2004.
Nelson Max. Optical models for direct volume rendering. IEEE TVCG, 1995.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, pp. 405-421. Springer, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Trans. Graph., 41(4), July 2022.
10

---

Published as a conference paper at ICLR 2024
Johannes Lutz Schönberger and Jan- AinMieh Frahm. Structure- from-motion revisited. CVPR, 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, 2021.
Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A modular frame- work for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-12, 2023.
Prune Truong, Marie-Julic Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In CVPR, pp. 4190-42000 2023.
Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. Neuris: Neural reconstruction of indoor scenes using normal priors. In ECCV, pp. 139-155. Springer, 2022a.
Peng Wang, Lingjic Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021a.
Yiqun Wang. Ivan Skorokhodov, and Peter Wonka. Hf-neus: Improved surface reconstruction using high-frequency details. NeurIPS, 35:1966-1978, 2022b.
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF--- Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07066 2021b.
Tong Wu, Jiaqi Wang, Xingang Pan, Xudong Xu, Christian Theobalt, Ziwei Liu, and Dahua Lin. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. In ICI.R, 2023.
Yitong Xia, Hao Tang, Radu Timofte, and Luc Van Gool. Sinerf: Sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. arXiv preprint arXiv:22/0.0455.0 2022.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neu- mann. Point-nerf: Point-based neural radiance fields. In CVPR, pp. 5438-5585 1488-5448,44 2022.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi view stereo. In ECCV, pp. 767-783, 2018.
Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In CVPR, pp. 3525-5534, 2019.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurIPS, 33, 2020.
Lior Yariy, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Thirty-Fifhh Conference on Neural Information Processing Systems, 2021.
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Ex- ploring monocular geometric cues for neural implicit surface reconstruction. NewrIPS, 2022.
Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang. Visibility-aware multi-view stereo network. BMVC, 2020a.
Jingyang Zhang, Yao Yao, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. Critical regularizations for neural surface reconstruction in the wild. In CVPR, pp. 6270-6279. 2022.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020b.
Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv: 10Xiv 1801108844 2018
11

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL RESULTS
A.1 DATASETS
The DTU Jensen al. (2014) dataset contains variety of object with 49 or 64 posed multi- view images for each scan. It covers different materials, geometry, and texture. We evaluate our method DTU with the same 15 test scenes following previous work, including IDR Yariv et al. (2020), NeuS Wang et al.] (2021a), and Voxurf Wu et al. (2023). The results are quantitatively compared by using Chamfer Distance, given the ground truth point clouds.
The MobileBrick Li et al. (023a) contains iPhone-captured 360-degree videos of different objects that are built with LEGO bricks. Therefore, the accurate 3D mesh, obtained from the LEGO website, is provided as the ground truth for evaluation. The ARKit camera poses and the manually refined ground-truth poses by the authors are provided, however, the accuracy is not comparable the poses obtained by camera calibration such as that in the DTU dataset. We compare our method with other approaches on 18 test scenes by following the provided standard pipeline.
A.2 EVALUATION METRICS
To assess the accuracy of camera poses, we employ a two-step procedure. First, we perform a 7- degree-of-freedon offffreooomm (7-DoF) pose alignment to align the refined poses with the ground-truth poses. This alignment process ensures that the calculated poses are in congruence with the ground truth. Subsequently, we gauge the alignment's effectiveness by utilising the standard absolute trajectory error (ATE) metric, which quantifies the discrepancies in both rotation and translation between the aligned camera poses and the ground truth. Moreover, we utilise these aligned camera poses to train Voxurf Wu et al. (2023) for object reconstruction. This ensures that the reconstructed objects accurately aligned with the ground-trute 3D model, facilitating a meaningful comparison between the reconstructed and actual objects.
A.3 IMPLEMENTATION DETAILS
Coordinates normalisation for unbounded 360-degree scenes. In the DTU Jensen et al. (2014) and MobileBrick Li et al. (203a) datasets, images include distant backgrounds. The original NeuS Wang et al. 2022 la) either uses segmentation masks to remove the backgrounds or recon-
structs them using an additional NeRF model Mildenhall et al. (2020), following the NeRF++ method Zhang et al. (2020b).
In contrast, we adopt the MipNeRF-360 approach Barron et (0022) which employs a normalisa- tion to handle the point coordinates efficiently. The normalisation allows all points in the scene to be modelled using single MLP network. The normalisation function is defined as:
x ((2
X |x|
1
contract(x)
(10)
[x
[
> 1
By applying this normalisation function, all points in the scene effectively mapped to the range (0-2) in space. This normalised representation is used to construct an SDF field, which then utilised for rendering images. Therefore, the proposed method can process images captured from 360-degree scenes without the need for segmentation masks.
Rendering network architectures. Our neural surface reconstruction code is built upon NeuS Wang et al. (2021a), which means that the network architectures for both SDF representa- tion and volume rendering hidden layers, with each layer housing 256 nodes, and it employs the ELU activation function for nonlinearity. Similarly, the rendering network consists of 4 layers and utilises the same nonlinear activation functions. This consistency in architecture ensures that the SDF and rendering processes are aligned within our implementation.
identical. To provide more detail, the SDF network comprises 8
PoRF MLP architectures. Refinement) module, consisting of 2-layer shallow MI.. Each layer in this MIP comprises 256
We employ a simple neural network structure for our PoRF (Pose
12

---

Published as a conference paper at ICLR 2024
nodes, and the activation function employed is the Exponential Linear Unit (ELU) for introducing nonlinearity. The input to the PoRF MLP consists of 7-dimensional vector. This vector is com- prised of 1-dimensional normalised frame index and 6-dimensional set of pose parameters. The PoRF MLP's output is a 6-dimensional vector, which represents the pose residual.
Correspondence generation. By default, we export correspondences from COLMAP, which are generated using the SIFT descriptor Lowee (204) and the outliers are removed by using a RANSAC- based two-view geometry verification approach. These correspondences, while reliable, tend to be sparse and may contain noise. However, method is designed automatically handle outliers in such data. Additionally, we conduct experiments with correspondences generated by LoFTR [Sun] al. (2021). These correspondences dense, providing denser point cloud representation. We adhere to the original implementation's 's guidelines to select high-confidence correspondences and further employ a RANSAC-based technique eliminate outliers. Consequently, the final set of correspondences obtained through LoFTR remains denser than SIFT matches.
Training details. Our approach follows a two-stage pipeline. In the first stage, we undertake the training of the complete framework, conducting total of 50,000 iterations using all available im- ages. This comprehensive training enables the refinement of pose information for each image. In the second stage, employ Voxurf et al. (2023) for the purpose of reconstruction. It's impor- tant to note that this reconstruction is executed solely using the training images. This step ensures fair comparison with prior methods, encompassing both surface reconstruction and novel view rendering. Regarding the hyperparameters used in our joint training of neural surface reconstruction and pose refinement, we have adopted settings consistent with those detailed in NeuS Waan et al. (2021a). Specifically, we randomly sample 5 512 rays from a single image in each iteration, with 128 points sampled per ray.
A.4 DETAILS FOR BASELINE METHODS
BARF Lin et al. 2021 and L2G Chen et al. (2033) are designed for LLFF and NeRF-Synthetic datasets. We find that naively running them with the original setting DTU and MobileBrick scenes leads to divergence. Therefore, we tuned hyper- parameters for them. Specifically, for BARF, we reduce the pose learning rate from the original 1e-3 le-4. For L2G, we multiply the output of their local warp network by a small factor, i.e., 0.01. As SPARF et al. (203)) was tested DTU scenes, we do not need to tune parameters. However, as it runs on sparse views (e.g., 3 or 6 views) default, they build all-to-all correspondences. When running SPARF in dense views (up to 120 images), we build correspondences between neighbouring co-visible views (up to 45 degrees in the relative pose) considering the memory limit.
A.S DETAILS FOR EXPERIMENTAL RESULTS
Quantitative ablation study results on DTU. Tab. RT shows the quantitative results for different settings illustrated as curves in Fig. 3 Firstly, the comparison spanning from L.1 to L4 serves to validate the effectiveness of the proposed components, which encompass critical elements like the PoRF (Pose Refinement) module and the epipolar geometry loss. This analysis highlights the sig- nificance of these components in enhancing the overall performance of the method. Secondly, the comparison encompassing L4 to L6 is intended to showcase the versatility and adaptability of the proposed method across different correspondence sources and scene representations. This demon- stration underlines the method's capacity deliver robust results regardless of variations in input data, thereby emphasising its universal applicability.
Table RI: Quantitative ablation study results on DTU. The results are averaged over test scenes.
Baseline (B) denotes naive joint optimisation of pose parameters and NSR.
Methods
COLMAP
Baseline (B) 2. B-PoRF
3. B+LEG
B+PoRF+Leo (Ours)
Ours-LoPTR 6. Ours-NeRF
(Deg.)
0.14
Tran. (mm)
1.03
2.67
1.06
1.37
1.02
1.03
1.00
13

---

Published as a conference paper at ICLR 2024
Scan40
COLMAP
BARF
SPARF
OURS
Scan63
COLMAP
BARF
SPARF
OURS
Figure S1: Qualitative reconstruction results on the DTU dataset. meshes were generated by using Voxurf Wu et al. (2023). All refinement method takes the COLMAP Schönberger & Frahm (2016) pose as the initial pose.
Qualitative reconstruction comparison. We include additional qualitative reconstruction re- sults for both the DTU dataset in Fig. [T and the MobileBrick dataset Fig. S2 These sualisations serve to illustrate key points. It is evident that the reconstruction quality using DOLMMA(SShönberrg
& Frahm (2016) pose information marred by noise, making it challenging to enhance even when employing BARF Lin et al. (2021). SPARF Truong al. (2023) demonstrates improvements in visual quality, but it lags behind our method in terms the precision and fidelity of
the reconstructed object surfaces. These supplementary qualitative results underscore the superior performance and capabilities of our method in comparison to alternative approaches, particularly when it comes to achieving accurate and detailed object surface reconstructions.
14

---

Published as a conference paper at ICLR 2024
Ferrari
ARKit
BARF
SPARF
OURS
Colosseum
ARKit
BARF
SPARF
OURS
Figure S2: Qualitative reconstruction results on the MobileBrick dataset. All meshes were generated by using Voxurf al. (2023). All refinement method takes the ARKit pose as the initial pose.
B ADDITIONAL EXPERIMENTAL RESULTS
B. NOVEL VIEW SYNTHESIS
We have also provided additional novel view synthesis results on both the DTU dataset (Table R2 and the MobileBrick dataset (Table R3). In these evaluations, we utilise Voxurf et al. (2023) for image rendering. It's essential to note that Voxurf was not originally designed for image rendering purposes, and it may face challenges, particularly when dealing with images that contain distant backgrounds, as observed in the MobileBrick dataset. These additional results offer a comprehensive view of our method's performance across different datasets, emphasising its strengths and limitations
15

---

Published as a conference paper at ICLR 2024
in novel view synthesis tasks. The qualitative comparison results illustrated in Fig. S6 and
Fig. S7
respectively.
Table R2: Quantitative novel view synthesis results on the DTU dataset. The Voxurf Wu et al. is used for reconstruction and image rendering.
2023
Poses
GT
COLMAP BARF Lin al. (2021)
SPARF SPARFTruong Truong al. (2023)
OURS
PSNR 1
31.84
27.63
27.50
29.59
31.67
SSIM t
0.931
0.861
0.856
0.892
0.929
LPIPS
0.142
0.208
0.211
0.167
0.143
Table R3: Quantitative novel view synthesis results on the MobileBrick dataset. The Voxurf Wu
al. 2023) is used for image rendering. The images in this dataset encompass 360-degree distant
backgrounds, and it's worth noting that Voxurf was not specifically designed to render images and
handle challenges in such scenarios.
Poses
GT
ARKit
BARF Lin et al. 2021
SPARF Truong et al. 2023)
OURS
PSNR
23.65
17.94
19.45
18.87
18.97
SSIM f
0.719
0.561
0.634
0.607
0.619
LPIPS $
0.447
0.541
0.501
0.506
0.501
B.2 ADDITIONAL ABLATION STUDIES
Ablation over the inputs PoRE In our PoRF module, the inputs comprise both the frame index and the initial camera pose. In this experiment, conducted an ablation study to dissect the impact each component on the re- sults. Figure [S3 presents the outcomes of this study, demonstrating that the removal of the ini- tial camera pose results in a significant drop in performance, while the removal of the frame in- dex causes minor decrease. It's crucial note that although the frame index appears to have relatively low impact on accuracy, remains an essential input. This is because the frame in- dex is unique to each frame and serves criti- cal role in distinguishing frames that are closely situated space.
1. PoRF
Rotation error (degrees)
0.6
2. PoRF w/o Pose
3. PoRF w/o Index
0.5
0.4
0.3
0.2
0.1
10k
15k
40k
45k
50k
Training Iterations
Figure S3: Ablation study results of the PoRF in- puts on Scan37.
B.3 ADDITIONAL ANALYSIS
Impact of the pose noise reconstruction. To understand how pose noises affect reconstruction accuracy, we make an empirical analysis by introducing different levels of pose noise on DTUDensen al. (0044). The results shown in Fig. S5 where we use Voxurf WW et al. (2023) to do recon- struction with different poses. To facilitate a comparison between rotation and translation, we use the COLMAP pose еггоr as a reference, i.e., using its pose eтrоr as unit. It reveals that recon- struction accuracy is highly sensitive to rotation eтrоrs, whereas small perturbations in translation have minimal impact on reconstruction accuracy. Therefore, in the optimisation process, the recon- struction loss cannot provide strong supervision for optimising translation. This explains why our method shows significant improvement in rotation but minor improvement translation.
Robustness to the initial pose noise. To assess the robustness of our method, we introduce varying degrees of noise to the ground truth (GT) pose during initialisation and subsequently apply our pose refinement process. The outcomes of this analysis are illustrated in Fig. S4] displaying quantitative results for the DTU dataset Jensen et al. (2014). In line with the preceding analysis, we utilise the COLMAP pose error reference point for comparison. The findings underscore that method
16

---

Published as a conference paper at ICLR 2024
Final Rot. Error
diagonal after refinement
Final Trans. Error
dagenall after refinement
1
Pose Rotation Noise (1 unit 0.67 deg.)
Pose Translation Noise (1 unit 1.06 mm)
Figure S4: Robustness of method the initial pose noise DTU (Scan37). The dashed lines
and solid denote the pose error before and after refinement, respectively.
Chamfer (mm)
Chamfer (mm)
6
*
4-
recon. with colmap poses
2-
2
recon. pertubed poses
0
0.0
0.5
1.0
1.5 2.0
2.5 3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
+ Rotation Noise (1 unit 0.67 deg.)
Pose + Translation Noise (1 unit 1.06 mm)
Figure S5: Reconstruction errors with pose errors on DTU (Scan37). We add noise to the GT pose and adopt the COLMAP pose error as one unit for comparison.
consistently reduces pose error across different levels of pose noise, revealing its capacity for robust performance. Additionally, the method exhibits greater resilience translation errors in comparison to rotation errors.
C LIMITATIONS
First, our method builds upon the foundation of an existing technique, NeuS Wang et al. (2021a). for the joint optimisation of neural surface reconstruction and camera poses. Consequently, the training process is relatively slow, taking approximately 2.5 hours for 50,000 iterations. In future work, we plan to explore techniques like instant Müller al. (2022) methods to accelerate this training phase further.
Second, it's essential note that our method, unlike SPARF Truong al. (0023, is primarily designed for high-accuracy pose refinement. As result, it requires nearly dense views and a robust camera pose initialisation such dense views and quality camera pose initialisation can often be accomplished using existing methods like ARKit and COLMAP.
optimal performance. In practical real-world applications, achieving
17

---

Published as a conference paper at ICLR 2024
Reference Image
SPARF
Figure S6: Qualitative novel view synthesis results on the DTU dataset. Images were rendered by
using Voxurf Wu al. 2023
18

---

Published as a conference paper at ICLR 2024
Reference Image
ARKIT
SPARF
Figure S7: Qualitative novel view synthesis results on the MobileBrick dataset. Images were ren- dered by using Voxurf Wu al. (202))
19


Reviews:
[{"review_id": "6lh0WRGo4f", "reviewer": "Reviewer_7s1S", "summary": "The authors propose a novel approach in refining the inaccurate camera pose and thereby improving neural surface reconstruction. \n\nThe paper presents two key innovations: Pose Residual Field (PoRF) and a robust epipolar geometry loss. PoRF, an implicit pose representation, uses an MLP network to learn pose residuals, considering global information across all frames for enhanced performance. The epipolar geometry loss, used for better supervision, relies on feature correspondences.\n\nThe method shows significant improvements in accuracy and efficiency in camera pose refinement. It outperforms existing methods like BARF and SPARF, particularly in datasets like DTU and MobileBrick.", "questions": "Considering that reference [1] is already available, I would appreciate it if you could provide some insight into how the PoRF Epipolar Geometry loss presents a novel approach compared to the 3D loss mentioned in [1].", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The utilization of global data across all frames by PoRF marks a notable advancement compared to techniques that individually optimize each image. This approach significantly enhances the refinement of camera poses, thereby achieving greater accuracy and efficiency in reconstructions. \n- Additionally, the method proves to be highly effective in adjusting poses from various sources, such as COLMAP and ARKit, demonstrating top-tier performance in practical, real-world dataset applications.", "weaknesses": "I've noticed that the concept of applying Epipolar Constraints in situations involving inaccurate or absent poses has been previously examined in [1]. It would be interesting to see if there's any discussion or comparison of this aspect in the context of PoRF's approach.\n  \n[1] Chen, S., Zhang, Y., Xu, Y., & Zou, B. (2022). Structure-Aware NeRF without Posed Camera via Epipolar Constraint. arXiv preprint arXiv:2210.00183.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFLP2GHgh4", "reviewer": "Reviewer_7mvY", "summary": "The manuscript introduces the use of an additional small MLP to refine the poses given to neural reconstruction methods jointly with the standard neural reconstruction objective. An additional epipolar loss derived from sparse feature matching (like SIFT matches from COLMAP) is used to effectively supervise this residual pose network.\nThis small addition if a learned shared pose residual update has big impacts on the quality of the reconstruction as demonstrated in the experiments. Notably it outperforms considerably the direct optimization of the poses with the standard reconstruction loss formulation.", "questions": "The axis angle formulation for pose is a common way to parameterize updates to poses. It would be good to also clarify how the algorithm goes back from these to the full 4x4 (or 3x4) pose matrix used for transformations in the rest of the method.\n\nIn figure 4 there is a clear point at which the errors start growing linearly where before they were sub-linear. I am curious why the authors think this happens.\n\nIs any position encoding used for the time/frame id input to PORF?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The addition of the pose residual learning with a small MLP is a straight forward addition to any neural representation learning method. \nThe fact that sparse correspondences needed for the epipolar loss are available from COLMAP without additional cost is an added benefit that makes it easy to incorporate into existing methods.\n\nThe quantitative experiments are high quality and compare the proposed approach against a diverse set of existing methods. They clearly show the improvements in reconstruction quality and pose estimation.\n\nThe additional ablations wrt to different algorithm settings are illustrative and help clarify that both PORF and epipolar line loss are needed to achieve high performance (Fig 5). As well as that the method can be used together with NERF (instead of just NeuS) and that SIFT matches are sufficient for the epipolar loss (vs more expensive deep learned LOFTR).\n\nThe qualitative results in Fig 1 and 6 very clearly show the ability of the poses residuals to correct for very noisy surfaces.\n\nThe manuscript is clear and well written and easy to follow except some details (see weaknesses).", "weaknesses": "The use of L1-L6 is unclear and not well introduced. To my knowledge this notation is not common to denote different experiments or ablations. I highly recommend introducing it upfront or using short acronyms to denote different configs. This would improve readability of the manuscript.\n\nThe effects of training the additional MLP and computing the additional epipolar loss is likely not significant when compared to training without them but it would still be good to quantify any difference in terms of overall time to convergence. If this additional time is negligible it could further support the use of PORF as a no-brainer. \n\nIts a minor thing but I am curious as to how much the time/frame id input is used by PORF. Ablating with and without time input would show whether PORF mostly compensates for global shifts or whether it can learn to fix individual poses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j7kz30523p", "reviewer": "Reviewer_jgNN", "summary": "This paper targets the optimization of camera pose for neural surface reconstruction. For handling the camera pose noise and intensive-view scenarios, this paper introduces PoRF, which learns the pose residuals (i.e., offset) by the frame index and initial camera pose. Moreover, the epipolar geometry loss is used to enhance supervision for camera pose estimation.  Along with the volume rendering and reconstruction loss in NeuS as the baseline, the proposed method shows high accuracy and robustness in intensive-view scenarios and a noisy initial camera pose. The proposed method is validated on the DTU and MobileBrick datasets, respectively. It uses the COLMAP and ARKit poses as the initial camera poses and achieves better camera poses and reconstruction results compared to other SOTA.", "questions": "See Weaknesses. My main concern is the novelty.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The proposed method is simple yet effective for joint optimization of camera pose and neural surface reconstruction.\n- The results are impressive. Tabs. 1-4 show the proposed method achieves high-quality camera pose estimation and neural surface reconstruction. Moreover, Figs. 3-5 show the proposed method is robust to noise.\n- The paper is well organized and easy to follow.", "weaknesses": "[Fig.2] Based on Eq.6, $\\alpha$ should multiply the residuals. However, Fig.2 shows $\\alpha$ multiply the input.\n\n[Ablation of PoRF]\n- [Shared MLP] The paper claims that the shared MLP captures the underlying global information and therefore boosts performance. However, this is not discussed in Fig.S5 or ablation study.\n- [$\\alpha$] It would be better to discuss the use of $\\alpha$ like w/ and w/o the fixing factor $\\alpha$ and its value. This is not discussed in Fig.S5.\n\n[Design of PoRF] The PoRF formulates camera pose estimation as a pose refinement procedure, which takes initial poses as input and predicts the offsets. The final prediction is the initial poses plus the offsets. This paper claims the PoRF was inspired by Nerf and can be treated as one of its main contributions. However, similar modules like recurrent pose refinement or iterative offset regression are a standard module in pose estimation, including 6D object[1,4] or human pose estimation[2,3]. Compared to existing modules, PoRF only introduces an additional index as input. This makes the novelty incremental. It would be better to highlight the difference or advantage of PoRF compared to those existing modules.\n\n\n[Epipolar] Similar Epipolar geometry contraints are also used in multi-view tasks like [5,6]. They all extract keypoints and then use Sampson distance. The framework is like a combination, which limits its novelty.\n\n\nReferences\n- [1] CRT-6D: Fast 6D Object Pose Estimation with Cascaded Refinement Transformers. WACV2023.\n- [2] End-to-end Hand Mesh Recovery from a Monocular RGB Image. ICCV2019.\n- [3] Human Pose Estimation with Iterative Error Feedback. CVPR2016.\n- [4] RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimization. CVPR2022.\n- [5] Deep Keypoint-Based Camera Pose Estimation with Geometric Constraints. IROS2020.\n- [6] PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JVXUiUz7LB", "reviewer": "Reviewer_dcYV", "summary": "This paper proposes an approach called PoRF for robust neural surface reconstruction with noisy camera pose initialization. The method leverages an MLP for regressing pose updates, which is more robust and accurate than conventional pose parameter optimization. The proposed method also includes an epipolar geometry loss to enhance supervision, which leverages correspondences exported from COLMAP results without limited computational overhead. The paper presents experiments on the DTU and MobileBrick datasets, demonstrating the efficacy of the proposed approach in refining camera poses and improving the accuracy of neural surface reconstruction in real-world scenarios.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is claimed to be more robust than conventional methods due to parameter sharing that leverages global information over the entire sequence.\n2. The paper presents extensive experiments on the DTU and MobileBrick datasets, demonstrating the efficacy of the proposed approach in refining camera poses and improving the accuracy of neural surface reconstruction in different scenarios.\n3. This paper is well-written and easy to understand, with clear explanations of the proposed method and experimental results.", "weaknesses": "1. It is a little unclear to me why using a single MLP for pose regression can improve pose optimization performance. Did you try to use multiple MLPs (one MLP for one frame)? Although it is claimed that one single MLP can leverage the global context information, it is not very convincing to me.\n2. Why should your method even outperform NeuS in Table 2 with GT poses? Does it mean your optimized pose is better than GT's or GT poses are also erroneous?\n3. The proposed method was only evaluated on two ideal object-centric datasets, which cannot demonstrate the generalization ability to different types of real-world data. I suggest the authors consider testing their performance on datasets like KITTI and mipnerf 360's dataset.\n4. In Eq. (7), the square of the inlier rate is used to weigh the loss terms. Why did you just simply use squares rather than other forms, e.g., cubic? Is this just an intuitive design or a careful selection after experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)