Title: Learning Depth-regularized Radiance Fields from Asynchronous RGB-D Sequences
Venue: NeurIPS 2023
Accepted: no
Keywords: Neural Radiance Fields
Abstract: Recently it is shown that learning radiance fields with depth rendering and depth supervision can effectively promote the view synthesis quality and convergence. But this paradigm requires input RGB-D sequences to be synchronized, hindering its usage in the UAV city modeling scenario. To this end, we propose to jointly learn large-scale depth-regularized radiance fields and calibrate the mismatch between RGB-D frames. Although this joint learning problem can be simply addressed by adding new variables, we exploit the prior that RGB-D frames are actually sampled from the same physical trajectory. As such, we propose a novel time-pose function, which is an implicit network that maps timestamps to SE(3) elements. Our algorithm is designed in an alternative way consisting of three steps: (1) time-pose function fitting; (2) radiance field bootstrapping; (3) joint pose error compensation and radiance field refinement. In order to systematically evaluate under this new problem setting, we propose a large synthetic dataset with diverse controlled mismatch and ground truth. Through extensive experiments, we demonstrate that our method outperforms strong baselines. We also show qualitatively improved results on a real-world asynchronous RGB-D sequence captured by drones. Codes, data, and models will be made publicly available.

Abstract

Recently it is shown that learning radiance fields with depth rendering and depth supervision can effectively promote the view synthesis quality and convergence. But this paradigm requires input RGB-D sequences to be synchronized, hindering its usage in the UAV city modeling scenario. To this end, we propose to jointly learn large-scale depth-regularized radiance fields and calibrate the mismatch between RGB-D frames. Although this joint learning problem can be simply addressed by adding new variables, we exploit the prior that RGB-D frames are actually sampled from the same physical trajectory. As such, we propose a novel time-pose function , which is an implicit network that maps timestamps to SE(3) elements. Our algorithm is designed in an alternative way consisting of three steps: (1) time-pose function fitting; (2) radiance field bootstrapping; (3) joint pose error compensation and radiance field refinement. In order to systematically evaluate under this new problem setting, we propose a large synthetic dataset with diverse controlled mismatch and ground truth. Through extensive experiments, we demonstrate that our method outperforms strong baselines. We also show qualitatively improved results on a real-world asynchronous RGB-D sequence captured by drones. Codes, data, and models will be made publicly available.

1 Introduction

Incorporating depth rendering and depth supervision into radiance fields has been demonstrated as a helpful regularization technique in several recent studies [2, 21, 32, 20]. However, this technique has not yet been successfully introduced into radiance field learning from UA V (Unmanned Aerial Vehicle) images, despite it's a typical choice in city modeling. A closer look at the aforementioned works reveals that they assume synchronized RGB and depth signals, which is hard to guarantee in UAV vision due to the lack of suitable synchronized sensors for long sensing ranges. So we study the problem of learning depth-regularized radiance fields from asynchronous RGB-D sequences.

As a recap, the canonical radiance field [15] learns a neural network parameterized by θ that represents a 3D scene, from input images I and their intrinsic/extrinsic parameters T I . To alleviate the reliance on T I , some works [30, 11, 8] aim to resolve a different problem that self-calibrates T I . In other words, they jointly learn θ and T I from input images I . Similarly, the formulation considered here is to learn scene representation θ , camera parameters T I and T D from inputs images I and depths D .

It is natural to develop the joint learning formulation

to resolve the problem

, as it amounts to adding new parameters to existing methods [30, 11, 8]. However, an important prior is ignored that RGB-D

frames are actually sampled from the same physical trajectory. As conceptually shown in Fig. 1-a/b, 33

T I and T D can be considered as samples from a function that maps timestamps to SE(3) elements. 34

We name this function as time-pose function and model it with a neural network parameterized by 35

ϕ . As such, we address the problem with a new formulation that learns scene representation θ and 36

RGB

D

RGB

D

Problem of

Interests

t

1

t

2

t

3

t

4

(c)

(d)

Our

Solution

(a)

(b)

(e)

time-pose function ϕ from inputs RGB images I and depths D . An interesting fact is that both θ and 37 ϕ are implicit neural representation networks (or say coordinate-based networks) that allow fully 38 differentiable training. To our knowledge, this new formulation has not been proposed before. 39

We also propose an effective learning scheme designed in an alternative manner. In the first stage, we 40 fit the time-pose function ϕ using one modality (e.g., RGB images) and infer the poses of the other, 41 using a balanced pose regression loss and a speed regularization term. Secondly, we bootstrap a large42 scale radiance field θ based upon Mega-NeRF [29] using the outputs of the trained time-pose function. 43 Thanks to the first step, depth regularization can be imposed here in spite of RGB-D misalignment. 44 Finally, thanks to the cascade of two fully differentiable implicit representation networks, we jointly 45 optimize the 3D scene representation θ and compensate pose errors by updating ϕ . 46

47

48

49

50

51

52

53

54

55

56

57

Since the problem considered is new, we contribute a synthetic dataset (named AUS) for systematic evaluation. Using six large-scale 3D scenes, realistic drone trajectories of different difficulty levels are generated. Specifically speaking, simple trajectories are heuristically designed with a zig-zag pattern while complicated ones are generated by manual control signals in simulation. We also control the mismatch between RGB-D sequences using different protocols, to cover as many as possible scenarios that the algorithm may encounter in reality. Through a set of comprehensive experiments, we show the proposed method outperforms several state-of-the-art counterparts and our design choices contribute positively to performance. Last but not least, we present a real-world evaluation using asynchronous sensors on drones. Our depth rendering results (on unseen viewpoint) is shown in Fig. 1-d, which is much better than the result of Mega-NeRF shown in Fig. 1-e. This success is credited to the usage of depth regularization as made possible by our novel algorithm.

58

59

60

61

62

63

64

65

66

67

To summarize, we have the following contributions in this paper: (1) We formalize the new problem of learning depth-regularized radiance fields from asynchronous RGB-D sequences, which is rooted in UAV city modeling. (2) We identify an important domain-specific prior in this problem: RGB-D frames are sampled from the same underlying trajectory. We instantiate this prior into a novel timepose function and develop a cascaded fully differentiable implicit representation network. (3) In order to systematically study the problem, we contribute a photo-realistically rendered synthetic dataset that simulates different types of mismatch. (4) Through a comprehensive benchmarking on this new dataset and real-world asynchronous RGB-D sequences, we demonstrate that our method can promote performance over strong prior arts. Anonynous code: https://anonymous.4open.science/r/async-nerf

2 Related Works

Large-scale Radiance Fields. Neural Radiance Field (NeRF) [15] has shown impressive results in 68 neural reconstruction and rendering. However, its capacity to model large-scale unbounded 3D scenes 69 is limited. Several strategies [29, 26, 32, 14, 35] have been proposed to address this limitation, with a 70 common principle of dividing large scenes into blocks or decomposing the scene into multiple levels. 71 Block-NeRF [26] clusters images by dividing the whole scene according to street blocks. Mega-NeRF 72 [29] utilizes a clustering algorithm that partitions sampled 3D points into different NeRF submodules. 73 BungeeNeRF [32] trains NeRFs using a growing model of residual blocks with predefined multiple 74 scales of data. Switch-NeRF [14] designs a gating network to jointly learn the scene decomposition 75 and NeRFs without any priors of 3D scene shape or geometric distribution. However, these prior 76 works fail to leverage the rich geometric information in depth images for effective regularization. 77

Depth-regularized Radiance Fields. Volumetric rendering requires extensive samples and sufficient 78 views to effectively differentiate between empty space and opaque surfaces. Depth maps can 79 serve as geometric cues, providing regularization constraints and sampling prior, which accelerates 80 NeRF's convergence towards the correct geometry. DS-NeRF [3] enhances this process using depth 81 supervision from 3D point clouds, estimated by structure-from-motion, and a specific loss for rendered 82 ray termination distribution. Mono-SDF [38] and Dense-Depth Prior [21] further supplement this 83 with a pretrained dense monocular depth estimator for less-observed and textureless areas. To adapt 84 NeRF for outdoor scenarios, URF [20] rasterizes a pre-built LiDAR point cloud map to generate 85 dense depth images and alleviates floating elements by penalizing floaters in the free space. Moreover, 86 S-NeRF [34] completes depth on sparse LiDAR point clouds using a confidence map, effectively 87 handling street-view scenes with limited perspectives. However, those methods are not readily 88 applicable to UAV captured images due to the lack of suitable synchronized sensors for long ranges. 89

90

91

92

93

94

95

96

97

Broader UAV Vision and Synchronization. Like autonomous driving, UAV vision is drawing increasing attention due to its unique characteristics. Broader UA V vision covers many topics like counting [31][7], trajectory forecasting [18], intention prediction [33], object tracking [16], physics understanding [39], next-best-view prediction [6], 3D reconstruction [41], and calibration [19]. Sensor synchronization is challenging for UAV vision (and other settings) and several works address the problem from an algorithmic perspective. One possibility is to adopt tailored hardware designs or software protocols [1] to synchronize all the devices. Another branch of sensor-agnostic methods utilizes temporal priors by using Sum-of-Gaussians [4] or parametric interpolation functions [36].

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

(i)

(ii)

(iii)

3 Problem Formulation and Optimization Pipline

Problem & Challenge. Our goal is to learn a neural radiance field parameterized by θ for large-scale scene representation from UAV images as done in prior works [29, 32]. However, these prior works fail to leverage depth supervision, which is known [3, 21] as useful for training floater-less NeRFs. To our knowledge, there are no easily accessible synchronized RGB-D sensor suites for large-scale outdoor scenes, and trivially synchronizing them according to timestamp 1 cannot fully address the misalignment issue. Instead of using expensive hardware, we take an algorithmic perspective.

Input & output. There are some prior works on large-scale scene modeling using aerial images [32, 29, 5, 6]. In this study, we assume an input RGB-D stream captured by drones: a set of RGB camera images { I ( i ) } N I i =1 and a set of depth maps { D ( j ) } N D j =1 (shown in Fig. 1-a) and we aim to recover the spatiotemporal transformations between them. Given that our focus is on relative transformation, it is viable to consider either the RGB or the depth stream as the reference without compromising generality. For convenience, we assume a set of camera poses {T ( i ) I } N I i =1 for color images are obtained by an SfM algorithm. The neural scene representation parameterized by θ outputs an image ˆ I as well as a depth map ˆ D at a given perspective camera pose T .

Observation. Note that all the sensor data are captured with a drone on the same trajectory 2 , we can model the relationship between capture time t and sensor poses T with an implicit time-pose function as ϕ : t → ˆ T = [ ˆ x , ˆ q ] , where t is the timestamp of capture, and the estimated pose ˆ T is represented by a translation vector ˆ x ∈ R 3 and a quaternion ˆ q ∈ R 4 .

Pipeline overview. We formulate our method as a 3-step optimization problem (as shown in Fig. 2). First, since the time-pose relationship for the RGB captures are given, we can train a time-pose

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

MLP

Pose

Color Loss

Depth Loss

Translation

Loss

Rotation

Loss

Speed

Loss

Missing modality

1-D Multi-resolution

Feature Grid

MLP

Hash & Interp

TIME POSE

FUNCTION

Error Compensation by Joint Optimization

GT/Predicted Trajectory

Bootstrapping

function on the RGB sequence (Fig. 2-(i)). Then, to train the neural radiance field, we first bootstrap the network with pure RGB supervision (Fig. 2-(ii)). To further enable training with RGB-D supervision, we can use the previously trained time-pose function to estimate the corresponding depth camera poses {T ( j ) D } of the depth timestamps { t ( j ) D } . Since both of the networks are differentiable, we jointly optimize the networks in an end-to-end manner in the third stage (Fig. 2-(iii)).

4 Method

We introduce in Section 4.1 the details of learning an implicit time-pose function. In Section 4.2, we describe our neural scene representation networks and the bootstrapping strategy. In Section 4.3, we adopt depth supervision and jointly train the time-pose function with RGB-D pairs.

4.1 Time-Pose Function

We represent the camera trajectory as an implicit time-pose function ϕ whose input is a timestamp t , and whose output is a 6-DoF pose T that consists of a 3-D translation x i and a 4-D quaternion q i .

Network Overview The time-pose function (shown in the left part of Fig. 3) is approximated with a compact 1-D multi-resolution hash grid {G ( l ) } L l =1 , followed by an MLP decoder. The hash grid consists of L levels of separate feature grids with trainable hash encodings [17]. The reason why we choose this architecture is as follows: The time-pose function is a coordinate-based function that may contain coarse and fine-level feature components 3 [27], and this architecture allows us to sample the hash encodings from each grid layer with different resolutions and perform quadratic interpolation on the extracted encodings to obtain a feature vector V i when querying a specific timestamp t that is in the range of all timestamps. This design choice is also empirically validated in Table. 3.

After obtaining the interpolated feature vector, an MLP with two separated decoder heads is used to predict the output translation ˆ x i and rotation ˆ q i vectors respectively. The forward pass can be expressed in the following equations:

where interp denotes interpolation, h is the hash function parameterized by π l , F MLP , l trans , l rot are the MLP networks and the decoder heads, with Φ MLP , Φ trans , Φ rot representing their parameters.

Depth-pose Prediction Since both the depth maps and the RGB images are collected by the same drone on the same flight, they cover the same spatial-temporal footprints except for the difference in the placement of the two sensors on the aircraft. For every depth frame, we first predict the RGB camera pose using the capture timestamps of the depth sensor with the time-pose function then transform the predicted RGB camera pose to the depth sensor pose with a pre-calibrated pose transformation T RGB → D between sensors.

To optimize the Time-Pose Function, we propose the following objective function:

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

where L trans , L rot , L speed are translation, rotation and speed losses respectively as shown in the left panel of Fig. 3. and λ trans , λ rot , λ speed are the weighting parameters. Note that λ trans and λ rot are automatically adjusted as explained in a later paragraph.

Pose Representation. There are some common choices to represent rotation for optimizing camera poses like rotation matrices [37] or Euler-angles [25, 28]. However, they are not continuous for representing rotation [42] due to their non-homeomorphic representation space to SO(3) . We choose to use unit quaternion as our raw rotation representation because arbitrary 4-D vectors can be easily mapped to legitimate rotations by normalizing them to the unit length [10].

Optimization of Translation and Rotation. We optimize the translation and the rotation vectors by minimizing the mean square error (MSE) between the estimated and ground-truth camera poses:

Since x and q are in different units, the scaling factor λ trans and λ rot play an important role in balancing the losses. To prevent translation and rotation from negatively influencing each other in training and to tap into possible mutual facilitation, we make the weighting factors learnable by using homoscedastic uncertainty [9] as L σ = L trans exp( -ˆ s trans ) + ˆ s trans + L rot exp( -ˆ s rot ) + ˆ s rot, where ˆ s are learnable parameters, thus the loss terms are balanced during training course 4 .

Optimization of Motion Speed. Observing that the time-pose function is essentially a function of translational displacement and angular displacement with respect to time, we can use the average linear velocity 5 to supervise the gradient of the network output, with regard to the input vectors. Since the linear velocity variation is small and the angular velocity variation is relatively larger in the scenes captured by the drone, only the average linear velocity is used to supervise the neural network and the latter is not supervised in our method:

4.2 Bootstrapping Large-scale Neural Radiance Fields

In this part, we introduce our proposed scene representation (right half of Fig. 3) that is bootstrapped in the second phase of the optimization process (Fig. 2-(ii)). Due to the limited capacity of MLPs, we follow Mega-NeRF[29] and partition the scene map into a series of equal-sized blocks in terms of spatial scope, and each block learns its individual scene representation with an implicit field. In this stage, we optimize the scene representation with pure RGB data. Specifically, the radiance field is denoted as { f ( i ) NeRF } N x × N y i =1 , where N x , N y denotes the spatial grid size. Each implicit function represents a geographic region with x centroid i as its centroid. The k th scene model can be written as:

where k = arg min j || x pts -x centroid j || 2 and γ is the positional encoding function.

For view synthesis, we adopt volume rendering techniques to synthesize color image ˆ I and depth map ˆ D . To be specific, we sample a set of points for each emitted camera ray in a coarse-to-fine manner [15] and accumulate the radiance and the distance along the corresponding ray to calculate the rendered color ˆ I and depth ˆ D . To obtain the radiance of a spatial point x pts, we use the nearest scene model for prediction. A set of per-image appearance embedding [13] is also optimized simultaneously in the training.

where o and d denote the position and orientation of the sampled ray, x ( t ) = o + t d represents the sampled point coordinates in the world space, and T ( t ) = exp ( -∫ t near σ ( k ) ( x ( s )) d s ) is the

accumulated transmittance. We optimize the scene representation model with only the photometric 190 error as L bootstrarp = MSE ( I, ˆ I ) . We empirically observe that this bootstrapping is critical to the 191 challenging third stage which jointly learns θ and ϕ using asynchronous RGB-D data. 192

193

194

195

196

197

198

199

200

201

202

4.3 Joint Optimization

While the time-pose function learns a good initialization from the RGB sequence, there are still errors to be compensated. In this section, we describe how we perform simultaneous mapping and pose optimization, which compensates for the initial error of the time-pose function.

We jointly optimize the inaccurate camera poses and the implicit maps: when fitting parameters Θ ( k ) NeRF of the scene representation, the estimated depth camera poses ˆ T ( j ) D ∈ SE(3) (where t ∈ R 3 and q ∈ SO(3) ) will be simultaneously optimized on the manifold:

where L is the objective function we demonstrate in the next paragraph.

To train the implicit representation to obtain photo-realistic RGB rendering maps and accurate depth map estimation, we update the mapping losses as:

where λ color and λ depth ( α ) are weighting hyper-parameters for color and depth loss, in which the 203 depth loss weight starts to grow from zero gradually with the training process α . 204

To compensate for the error from the time-pose function extracted poses, we jointly optimize two 205 implicit representation networks thanks to the end-to-end differentiable nature. 206

207

5 Asynchronous Urban Scene (AUS) Dataset

YOSVRK

NGXJ

SGT[GR

YOSVRK

NGXJ

SGT[GR

GLYPH<c=9,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>BGLYPH<c=10,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>GLYPH<c=1,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>"64GLYPH<c=14,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>/FXGLYPH<c=1,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> :PSL

GLYPH<c=9,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>CGLYPH<c=10,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>GLYPH<c=1,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>"64GLYPH<c=14,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>4BOGLYPH<c=1,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>'SBODJTDP

GLYPH<c=9,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>DGLYPH<c=10,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>GLYPH<c=1,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>"64GLYPH<c=14,font=/AJEIAL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>4NBMM

(XOJMK

:U]T

9INUUR

)GYZRK

RGBD sequence

Sampled RGB

Sampled Depth

with fixed offset

GLYPH<c=18,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=17,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=17,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>GLYPH<c=14,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>YGLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>

YGLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>

YGLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>

RGBD sequence

Sampled RGB

Sampled Depth

with random offset

YGLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>

ZGLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>

GLYPH<c=18,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=17,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=17,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>GLYPH<c=14,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>YGLYPH<c=6,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>

GLYPH<c=9,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>EGLYPH<c=10,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>3F TB NQMFGLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>XJUI GLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>GJYFEGLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>PGGTFU

GLYPH<c=9,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>F GLYPH<c=10,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> GLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H> 3F TBNQ MFGLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>XJUIGLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>SBOEPNGLYPH<c=1,font=/AJEIGL+FZYANZQKSJF--GBK1-0-GBK-EUC-H>PGGTFU

Dataset Collection. Our Asynchronous Urban Scene (AUS) dataset as illustrated in Fig.4 is generated 208 using Airsim [23], a simulator plug-in for Unreal Engine. With 3D city models loaded in Unreal 209 Engine, the simulator can output photorealistic and high-resolution RGB images with synchronized 210 depth images (resampled later) according to the a drone trajectory and a capture framerate. We choose 211 Airsim as it strikes a good balance between rendering quality and dynamics modeling flexibility. 212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

3D City Scene Models. To generate the AUS dataset, we exploit a total of six scene models, covering two large-scale ones shown in Fig. 4-a/b and four (relatively) small-scale ones shown in Fig. 4-c. The former uses the New York and San Francisco city scenes provided by Kirill Sibiriakov [24], in which AUS-NewYork covers a 250 × 150 m 2 area with many detailed buildings and AUS-SanFrancisco consists of a 500 × 250 m 2 area near the Golden Gate Bridge. The latter uses four model files provided in the UrbanScene3D dataset [12]. As such, at the scene level, AUS features a good coverage of both large-scale modern cities and smaller cultural heritage sites.

Trajectory Generation. Trajectory complexity matters for our problem. On one hand, in many real-world applications, photographers may manually control drones to capture a city. On the other hand, simple trajectories can be modeled by simple functions, rendering the neural time-pose function unnecessary. To build a meaningful and comprehensive benchmark, we use three types of trajectories: a trivial Zig-Zag trajectory (simple in Fig. 4), a more complex randomly generated trajectory (hard in Fig. 4), and a very complex manually controlled trajectory (manual in Fig. 4). In AUS-Small, we only provide manually controlled trajectories, since the scene sizes are relatively small and using the former two trajectory strategies leads to an unrealistically large overlap between frames.

Mismatch Resampling. We first sample synchronized RGB-D sequences in the simulator at a high frequency (50fps) then re-sample RGB and depth images with various offsets to create asynchronous RGB-D sequences. As shown in Fig. 4-d/e, we exploit two settings for the AUS dataset. In Fig. 4-d, every RGB-D pair is resampled according to a fixed offset denoted by x% . For example, we sample the RGB image at 5fps or say every 10 frames and x = 30 means every depth image is 3 frames later than the RGB counterpart. In Fig. 4-e, the offset between an RGB-D pair is randomly selected, simulating a challenging real-world asynchronous sequence. Offset ablation will be shown later.

6 Experiments

In this section, we show the effectiveness of our 3-step optimization pipeline by qualitatively and quantitatively evaluating our proposed methods and comparing with baseline methods.

6.1 Results

We evaluate our proposed method against NeRF-W [13] and city-scale Mega-NeRF [29] and present the quantitative results in Table 1 and Table 2. NeRF-W is the baseline from which we borrow the aforementioned idea of per-image appearance embedding and Mega-NeRF is a state-of-the-art (SOTA) large-scale scene modeling framework which our network is built upon.

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

Ground-Truth

Ours

Mega-NeRF

NeRF

Castle

School

Town

Bridge

SF manual

SF hard

SF simple NY manual NY hard NY simple

Castle

School

Town

Bridge

SF manual

SF hard

SF simple NY manualNY hard NY simple

11

RGB-D View Synthesis The standard metrics for novel view synthesis and depth estimation are used for evaluation. For RGB view synthesis, metrics including PSNR, SSIM, and the VGG implementation of LPIPS [40] are used. For depth estimation, RMSE, RMSE log, δ 1 , 2 , 3 are used. In Table 1, we quantitatively show on the RGB view synthesis task that, together with depth supervision, our method can generate more photo-realistic images than two SOTA baselines, and show on the depth estimation tasks that our method significantly improves the learned geometry of the scene representation network. We also present the RGB-D view synthesis results qualitatively in Fig.5 in which our method synthesize photo-realistic images and accurate depth maps, while baseline methods fail at predicting reliable depth maps (e.g. in the School scene, they mispredict the void space as a dense surface; in NY hard , the depth values around glasses are obviously inaccurate).

Depth Pose Estimation We evaluate the performance of our time-pose function in Table 2, or say specifically the accuracy of our method to localize depth sensor poses. As shown quantitatively, our method can achieve an average pose error of 1 . 04 m and 1 . 07 ◦ in the first stage, where only time-pose pairs from the RGB sequence are used to optimize the network. After joint optimization in the third stage, our method cuts half the errors to 0 . 53 m and 0 . 41 ◦ .

Real-world Evaluation. In the real-world experiments, we use the DJI M300 UAV (equipped with a high-definition RGB camera and LiDAR to collect real data, where the RGB camera collects images at the frame rate of 30fps and the LiDAR collects depth information at 240Hz. The poses of the RGB images are provided by COLMAP [22]. The fixed transformations between sensors are provided by the producer or can be calibrated manually. A qualitative comparison is provided in Fig. 1 and more results are in the supplementary.

6.2 Ablation Studies

Time-Pose Function Network Structure. We compared our proposed time-pose function implementation with several commonly used implicit representation architectures on the localization accuracy

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

(Tab. 3). In order to highlight the gap between different methods, we downsample the dataset to increase the difficulty. (a) Pure MLP architecture : processing the positional-encoded [15] input timestamps with an MLP; (b) 1-D Feature Grid : storing a feature vector for each second in the timestamp span and performing linear feature interpolation in the query's neighborhood. (c) Ours : our proposed 1-D multi-resolution hash grid with different resolution layers. The results (Table 3) show that our proposed multi-resolution architecture outperforms other network architectures in accuracy. The network structures are further detailed in the supplementary materials.

Speed Loss. We compared our method's localization accuracy with and without the optimization of motion speed (see the comparison of 'Ours' and 'Ours w/o speed' in Tab. 3). The results show that minimizing the gradient error (i.e., speed loss) help a lot in improving the accuracy of the translation (from 20 m to 6 . 3 m ).

Different Sampling Offsets. To show the robustness of our proposed time-pose function, we compare the localization accuracy of implicit trajectory representations under different sampling offsets, whose definition is described in Fig. 4-d/e. The quantitative results (Table. 4) indicate that the network's output exhibits a controllable margin of error as the data offset increases.

Joint Optimization for Pose Error Compensation. To demonstrate the importance of rectifying erroneous poses of depth images in asynchronous RGB-D sequences using the time-pose function, we train a Mega-NeRF[29] with depth supervision but disabled the joint optimization stage. From the evaluation results (Table 5), we observe its substantial impact on the rendering quality (PSNR for RGB and RMSE for depth). Due to limited space, qualitative results are in the supplementary.

7 Conclusion

In this paper, we present a method to learn depth-supervised neural radiance fields from asynchronous RGB-D sequences. We leverage an important prior that the sensors cover the same spatial-temporal footprints and propose to utilize this prior with an implicit time-pose function. With a 3-staged optimization pipeline, our method calibrates the RGB-D poses and trains a large-scale implicit scene representation. Our experiments on a newly proposed large-scale dataset show that our method can effectively register depth camera poses and learns the 3D scene representation for photo-realistic novel view synthesis and accurate depth estimations. Broader impact and limitations: Large-scale scene modelling can be used for potential military use, which the method is not intended for.

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

References

Sameer Ansari, Neal Wadhwa, Rahul Garg, and Jiawen Chen. Wireless Software Synchronization of Multiple Distributed Cameras. In 2019 IEEE International Conference on Computational Photography (ICCP) , pages 1-9, May 2019. ISSN: 2472-7636.

Haowen Deng, Mai Bui, Nassir Navab, Leonidas Guibas, Slobodan Ilic, and Tolga Birdal. Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation. International Journal of Computer Vision , pages 1-28, 2022.

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12882-12891, 2022.

A. Elhayek, C. Stoll, N. Hasler, K. I. Kim, H.-P. Seidel, and C. Theobalt. Spatio-temporal motion tracking with unsynchronized cameras. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 1870-1877, June 2012. ISSN: 1063-6919.

Antoine Guedon, Pascal Monasse, and Vincent Lepetit. Scone: Surface coverage optimization in unknown environments by volumetric integration. In Advances in Neural Information Processing Systems , 2022.

Antoine Guédon, Tom Monnier, Pascal Monasse, and Vincent Lepetit. Macarons: Mapping and coverage anticipation with rgb online self-supervision. arXiv preprint arXiv:2303.03315 , 2023.

Meng-Ru Hsieh, Yen-Liang Lin, and Winston H Hsu. Drone-based object counting by spatially regularized regional proposal network. In Proceedings of the IEEE international conference on computer vision , pages 4145-4153, 2017.

Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5846-5854, 2021.

Alex Kendall and Roberto Cipolla. Geometric loss functions for camera pose regression with deep learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5974-5983, 2017.

Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In Proceedings of the IEEE international conference on computer vision , pages 2938-2946, 2015.

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5741-5751, 2021.

Liqiang Lin, Yilin Liu, Yue Hu, Xingguang Yan, Ke Xie, and Hui Huang. Capturing, reconstructing, and simulating: the urbanscene3d dataset. In ECCV , 2022.

Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7206-7215, June 2021. ISSN: 2575-7075.

Zhenxing Mi and Dan Xu. Switch-nerf: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In International Conference on Learning Representations (ICLR) , 2023.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , Lecture Notes in Computer Science, pages 405-421, Cham, 2020. Springer International Publishing.

Matthias Mueller, Neil Smith, and Bernard Ghanem. A benchmark and simulator for uav tracking. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 445-461. Springer, 2016.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics , 41(4):1-15, July 2022. titleTranslation:.

Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling 346 social behavior for multi-target tracking. In 2009 IEEE 12th international conference on computer vision , 347 pages 261-268. IEEE, 2009. 348

349

350

351

Amir M Rahimi, Raphael Ruschel, and BS Manjunath. Uav sensor fusion with latent-dynamic conditional random fields in coronal plane estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4527-4534, 2016.

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12932-12942, 2022.

Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12892-12901, 2022.

Johannes L. Schönberger and Jan-Michael Frahm. Structure-from-Motion Revisited. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4104-4113, June 2016. ISSN: 1063-6919.

Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics: Results of the 11th International Conference , pages 621-635. Springer, 2018.

Kirill Sibiriakov. Artstation page https://www.artstation.com/vegaart, 2022.

Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proceedings of the IEEE international conference on computer vision , pages 2686-2694, 2015.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8248-8258, 2022.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, June 2020. arXiv:2006.10739 [cs].

Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1510-1519, 2015.

Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12912-12921, June 2022. ISSN: 2575-7075.

Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 , 2021.

Longyin Wen, Dawei Du, Pengfei Zhu, Qinghua Hu, Qilong Wang, Liefeng Bo, and Siwei Lyu. Detection, tracking, and counting meets drones in crowds: A benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7812-7821, 2021.

Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In The European Conference on Computer Vision (ECCV) , 2022.

Dan Xie, Sinisa Todorovic, and Song-Chun Zhu. Inferring" dark matter" and" dark energy" from videos. In Proceedings of the IEEE International Conference on Computer Vision , pages 2224-2231, 2013.

Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-neRF: Neural radiance fields for street views. In The Eleventh International Conference on Learning Representations , 2023.

Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin. Grid-guided neural radiance fields for large urban scenes, 2023.

Anqi Joyce Yang, Can Cui, Ioan Andrei Bârsan, Raquel Urtasun, and Shenlong Wang. Asynchronous Multi-View SLAM. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 5669-5676, May 2021. ISSN: 2577-087X.

Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin. 397 iNeRF: Inverting Neural Radiance Fields for Pose Estimation. In 2021 IEEE/RSJ International Conference 398 on Intelligent Robots and Systems (IROS) , pages 1323-1330, September 2021. ISSN: 2153-0866. 399

400

401

402

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

403

404

405

406

407

408

409

410

411

Kuo-Hao Zeng, Roozbeh Mottaghi, Luca Weihs, and Ali Farhadi. Visual reaction: Learning to play catch with your drone. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11573-11582, 2020.

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 586-595, June 2018. ISSN: 2575-7075.

Runze Zhang, Siyu Zhu, Tian Fang, and Long Quan. Distributed very large scale bundle adjustment by global camera consensus. In Proceedings of the IEEE International Conference on Computer Vision , pages 29-38, 2017.

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representa412 tions in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 413 Recognition , pages 5745-5753, 2019. 414

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explained and analysed the issue of unsynchronous RGB and depth measurement under the UAV city modelling scenario, which makes it challenge to incorporate depth supervision for NeRF optimisation under such scenerio. The authors proposed a novel solution to it by modelling the continuous RGB camera trajectory as an implicit time-pose function. Under the prior knowledge that both RGB and depth are generated on the same trajectory, poses of depth images can be queried from the time-pose function. Based on this time-pose function, the authors further proposed a 3-stage optimisation pipeline to train the NeRF model with depth supervision. Qualitative and quantitative results show the proposed method achieves better results compared with previous methods that use RGB inputs only.\n\nSTRENGTHS:\n1. This paper studied a very interesting and important problem of synchronising RGB and depth measurements, especially under large scale UAV scenerios, which posed challenges to incorporating depth supervision for NeRF optimisation under such scenerios. \n\n2. The authors proposed a novel solution by modelling the continuous trajectory as a time-pose function, and designed a 3-stage optimisation pipeline to leverage the synchronised RGB and depth meaurement for NeRF training.\n\n3. Experimental results showed the effectiveness of proposed methods compared to previous RGB-only methods. Ablation studies also show  naively using unsynchronised depth image could hinder the performance. \n\n4. The paper is very well written and easy to follow. The storyline and motivation are very clear.\n\nWEAKNESSES:\n1. Although the experimental results in Tab. 1 and 5 showed the proposed method achieves better results than Mega-NeRF and NeRF-W, it's not compeltely fair as the baseline methods only take in RGB images. The authors need to show that the problem could not be easily solved by trivial efforts such as jointly optimising the poses of depth images. For example, in tab. 5 the author showed the results of Mega-NeRF with depth but the joint optimisation was switched off. Also in line 104, the authors mentioned that soft synchronization cannot fully address the misalignment issue, but it would be good to also show how good can all the methods perform with this simple alignment. \n\n2. Most of the experiments and results are shown in synthetic dataset. Only Fig. 1 and the supplementary video showed results from real-world scenes. It would be good to show more results on real world sequences.\n\n3. It would be good to show some run-time comparison and analysis.\n\nQUESTIONS:\nAll my questions are stated in the Weakness section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
The paper proposes a way to train NeRF with asynchronous RGBD videos. Specifically, three technical contributions have been made:
1. New problem formulation for NeRF training from async RGBD video.
2. Propose a time-pose function to use async RGB and Depth stream, resulting in better pose estimation and NeRF training.
3. Propose a new synthetic dataset for this task.

---
**After rebuttal**: I have read authors' rebuttal and it addresses my concerns.

STRENGTHS:
1. The paper is well-written and easy to follow.
2. It is novel to formulate the pose estimation issue of an RGBD stream in a time-pose function, which constrains the challenging issue better. Since ground truth poses for RGB stream is known, this paper essentially proposes a novel way to use an async depth stream to 
    1. improve NeRF quality;
    2. refine RGB poses; 
    3. optimise poses for depth images from interpolated RGB poses.
3. Although the method requires poses for the RGB video, which is a strong assumption for un-posed NeRF training, the formulation is still novel and smart to me.

WEAKNESSES:
In real datasets, I think there should be a self-motion distortion caused by the motion of the mounted drone. I am wondering how would this LiDAR depth distortion affect the performance?

QUESTIONS:
See the weakness section.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to reconstruct aerial scenes with Neural Radiance Fields that are supervised with RGB images and asynchronous depth images. To address the asynchronicity they propose a novel time-pose function that provides a prior to optimise the poses of the depth images. To validate their method a new synthetic dataset is introduced. They outperform one relevant baseline and show that asynchronous depth captures are an issue worth considering in NeRF reconstructions.\n\nSTRENGTHS:\n* S1. Relevancy. Depth supervision is an important method to improve 3D NeRF reconstruction and asynchronous depth has not been considered before. The Problem formulation is novel and interesting with significance in the field of UAV reconstruction.\n* S2. Method Idea. The proposed time-pose function is an original approach to exploit the prior that depth and rgb captures lie on the same trajectory and it is shown that jointly optimising the NeRF reconstruction and time-pose function can significantly improve RGB and Depth reconstruction.\n* S3. Presentation. The paper is written in a cohesive manner with a clear structure.\n\nWEAKNESSES:\n* W1. Missing baselines for pose optimisation. The authors ablate design decisions for their proposed time-pose function but do not provide baselines for pose optimisation, which is, in this reviewers opinion, their main contribution. A qualitative comparison to methods like BARF (https://github.com/chenhsuanlin/bundle-adjusting-NeRF) or other state-of-the-art methods (https://nope-nerf.active.vision/, https://prunetruong.com/sparf.github.io/) would support the decision to use the time-pose function.\n* W2. Questionable assumptions for poses. As far as this reviewer understands the poses for the depth are assumed unknown and only initialised by the time-pose function that was trained on the rgb capture timesteps. This reviewer questions this assumption in general, as in both the synthetic and real-world setting poses for the depth can either be obtained from the simulation or from GPS measurements. A quantitative comparison between GT/GPS poses, BARF optimised poses and the time-pose function would strongly support the authors decision to use an implicit function to represent the drones trajectory. \n* W3. Missing Implementation details for reproducibility. Some important implementation details are missing from both the paper and supplementary, to be precise:\n   * The number of images & depth images in the generated and real datasets.\n   * Is the time-pose function using positional encoding / fourier features ? This is not clear. \n   * No training parameters are given for the proposed method and baselines. \n* W4. Missing Discussion of related work. Related work in the field of representing signals with MLPs is not discussed (e.g. SIREN) and not considered in the design of the time-pose function.\n\nQUESTIONS:\n* Q1. How many images and depth images are used for training ? \n* Q2. Why is a comparison to BARF, related methods and a GPS baseline not necessary for the proposed method - or why has it been omitted ? \n* Q3. What is the intuition for the fully MLP-based baseline for the time-pose function ? The model capacity seems very large to represent a rather simple data distribution.\n* Q4. Is the NeRF example in Figure 5 the NeRF-W baseline mentioned in the quantitative results or the standard NeRF ? This might be a typo but makes that part unclear. \n\nUPDATE: After the rebuttal most of these questions have been addressed (see discussion for details). The score is adjusted and the authors are asked to include these additional clarifications into the final paper version.\n\nLIMITATIONS:\nThe authors do not discuss limitations of their method in the paper but include a very small discussion in the supplementary. Overall limitations cannot really be discussed since there is no objective baseline to compare the method to. For broader impact only potential military uses are mentioned, whereas the method also has implications for surveillance. Environmental impacts of large-scale neural network training are also not mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to include depth supervision in NeRF in the UAV city modeling scenario. The key problem is that the images and depth maps are asynchronous. This paper exploits a prior that RGB-D frames are sampled from the same physical trajectory. It fits a time-pose function to the available RGB cameras and computes depth map cameras by this function and a pre-calibrated pose transform between sensors. Then it trains a NeRF with RGB loss and optimizes it further with RGB-D supervision. A new synthetic dataset is proposed for evaluation.\n\nSTRENGTHS:\n1. This paper tackles a new problem in NeRF modeling on UAV datasets. it defines a scenario where RGB and depth frames are asynchronous. \n\n2. Instead of directly adding new parameters to predict depth camera poses together with the NeRF optimization, it identifies a prior of the relation of RGB poses and depth poses.\n\n3. It designs a network with 1D hash encoding to fit a time-pose function for RGB cameras.\n\n4. It generates a new synthetic dataset to evaluate the proposed method.\n\n5. It designs a new speed loss in pose learning.\n\nWEAKNESSES:\n1. A key contribution of this method is the learned time-pose function. It actually can be seen as an interpolation function to interpolate the RGB poses on depth timestamps. Since camera interpolation is a common practice in 3D software such as Blender, I am curious about whether other simple interpolation methods such as linear interpolation or the interpolation in Blender could get worse or better accuracy.\n\n2. As stated above, on easier datasets, the simple interpolation methods may still get good results. It will be better to construct more challenging datasets to demonstrate the effectiveness of the proposed time-pose function. This situation can be considered when generating synthetic datasets.\n\n3. Directly adding extra parameters to estimate depth poses are also an option as stated in the paper. What is the accuracy like if we directly use this method? It would be not surprising if it gets worse results because the poses are not well initialized.\n\n4. The prior that \"RGB-D frames are actually sampled from the same physical trajectory\" can be explained more in the main paper such as in the Introduction section. This can make us understand clearly that there is a fixed and known pose transformation between the depth and image sensors so we can estimate the depth poses by interpolating image poses.\n\nQUESTIONS:\n1. What is the performance using simple linear interpolation or the Blender to get depth poses?\n\n2. If the simple interpolation methods can still get good results, can we consider this and generate more challenging datasets to validate the effectiveness of the proposed time-pose function?\n\n3. What is the performance if we directly add extra parameters to estimate the depth pose?\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a pipeline to build depth-supervised neural radiance fields (NeRF) using asynchronous RGB-D sequences. Since the task is novel, the paper also contributes a synthetic dataset and demonstrates that it outperforms certain baselines in the experiments.

STRENGTHS:
## Originality
The task is novel and practical for lots of real-world settings.

WEAKNESSES:
## Quality
- The most obvious baseline is missing. What if authors simply initialize the depth frames with their (misaligned) poses from the sensor and perform BARF [1] style joint optimization on the depth frames’ poses (note that depth frames can have different poses from the RGB images after optimization)?


- Since the above baseline is missing, it’s hard to understand why the time-pose network is necessary.

- In my opinion, the baselines authors compared against are in the wrong direction. NeRF-W and Mega-NeRF are not solving the problem of inaccurate camera poses. Authors should look into works that try to solve pose estimation and neural radiance fields simultaneously such as BARF, NeRF–, and iNeRF [1, 2, 3].

- Line 156-157’s motivation for using Quaternion since other rotation representations are not continuous is weird. Quaternion is also not continuous and is known to not be the best representation for rotation regression. See this paper [4]


### References
- [1] BARF: Bundle-Adjusting Neural Radiance Fields, Lin et al.
- [2] NeRF--: Neural Radiance Fields Without Known Camera Parameters, Wang et al.
- [3] INeRF: Inverting Neural Radiance Fields for Pose Estimation, Yen-Chen et al.
- [4] On the Continuity of Rotation Representations in Neural Networks, Zhou et al.

QUESTIONS:
Same as Weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-08