UP-NeRF: Unconstrained Pose Prior-Free Neural Radiance Field
Authors: Injae Kim
Minhyuk Choi
Hyunwoo J. Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: neural radiance field
pose estimation

Abstract: Neural Radiance Field (NeRF) has enabled novel view synthesis with high fidelity given images and camera poses. Subsequent works even succeeded in eliminating the necessity of pose priors by jointly optimizing NeRF and camera pose. However, these works are limited to relatively simple settings such as photometrically consistent and occluder-free image collections or a sequence of images from a video. So they have difficulty handling unconstrained images with varying illumination and transient occluders. In this paper, we propose **UP-NeRF** (**U**nconstrained **P**ose-prior-free **Ne**ural **R**adiance **F**ields) to optimize NeRF with unconstrained image collections without camera pose prior. We tackle these challenges with surrogate tasks that optimize color-insensitive feature fields and a separate module for transient occluders to block their influence on pose estimation. In addition, we introduce a candidate head to enable more robust pose estimation and transient-aware depth supervision to minimize the effect of incorrect prior. Our experiments verify the superior performance of our method compared to the baselines including BARF and its variants in a challenging internet photo collection, *Phototourism dataset*. The code of UP-NeRF is available at https://github.com/mlvlab/UP-NeRF.

UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fields
Injae Kim* Korea University dna90410korea. ac.kr
Minhyuk Choi* Korea University sodlqnf www.korea.ccc..... ac.kr
Hyunwoo Kim Korea University
byunwoojkim@korea.cc ac.
Abstract
Neural Radiance Field (NeRF) has enabled novel view synthesis with high fidelity given images and camera poses. Subsequent works even succeeded in eliminating the necessity of pose priors by jointly optimizing NeRF and camera pose. How- ever, these works are limited to relatively simple settings such as photometrically consistent and occluder-free image collections or a sequence of images from a video. So they have difficulty handling unconstrained images with varying illumi- nation and transient occluders. this paper, we propose UP-NeRF (Unconstrained Pose-prite Pose-prior-fiee prior-frree Neural Radiance Fields) to optimize NeRF with unconstrained image collections without camera pose prior. We tackle these challenges with sur- rogate tasks that optimize color-insensitive feature fields and separate module for transient occluders to block their influence on pose estimation. In addition, we in- troduce candidate head to enable more robust pose estimation and transient-aware depth supervision to minimize the effect incorrect prior. Our experiments verify the superior performance of our method compared to the baselines including BARF and its variants in a challenging internet photo collection, Phototourism dataset. The code of UP-NeRF is available at hhtps:: //githtt..../. www.m/wwaa/ppua......... NeRF
Introduction
Neural Radiance Fields (NeRF) opened up new chapter in novel view synthesis by demonstrating
its capacity to generate high-quality novel view images given only set of 2D images. Its powerful performance has enabled many practical applications including virtual/augmenter reality (VR/AR) (2]
31, autonomous systems 4 5). and robotics (68]. Various follow-up studies have developed NeRF
by addressing its limitations, such as the necessity of dense input views ( iI and long training time [7] Also, there have been efforts [8 19 to deal with unconstrained images with varying illuminations and transient occluders to relieve the burden collecting clean images or establishing specialized setting to capture them. For instance, NeRF-W W [18] introduces an appearance embedding to handle varying illumination and transient head to filter transient occluders.
One of the popular research topics in NeRF is joint optimization for pose and neural scene repre- sentation. Originally, NeRF requires accurate camera pose priors which are generally obtained with classic methods like structure from motion (SIM) [20]. Although the poses obtained from such method are treated as ground truth in many works, does always give optimal results or even fails to converge. Hence recent works 213 have proposed method so-called umposed-NeRF that simultaneously trains camera poses and NeRF. This eliminates the necessity pose priors and burdensome preprocessing. While these methods have only succeeded in forward-facing scenes, NoPe-NeRF 24 further improved the method to even outdoor scenes with continuous images from videos. These previous works, however, are based on the photometric loss which is not reliable for
First two authors have an equal contribution.
i Corresponning author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Color
Feature
Surrogate
of densely captured 2D images and corresponding camera parameters of each image. Despite its light require- ments, it demonstrates quality reconstruction of the complex geometry scenes. The key idea of NeRF is that even though given are only 2D images, an implicit function represented by a multilayer perceptron can be utilized to parameterize the color and density of each 3D point on ray, which is subsequently synthesized to decide the color of pixel from which the ray was shot. Thanks to its have followed the trail of Figure I:
Deep feature focuses semantic information of objects rather than their colors, which makes an effective descriptor for images with photometric inconsistency.
unconstrained images, so they have difficulty in training with unconstrained images. In addition, the unconstrained images have complex camera poses, which is hard optimize if prior like continuity of images from a video is not available. Therefore, training unposed-NeRF with unconstrained images causes contradiction: To estimate camera poses accurately, images must be photometrically consistent without transient occluders; to treat unconstrained images without photometric consistency and transient occluders, accurate camera poses needed. Previous unposed-NeRI's in the literature are not capable of learning in an unconstrained setting, leading to suboptimal pose estimation and poor rendering quality.
To tackle the problem, we propose UP-NeRF (Unconstrained Pose-prior-free Neural Radiance Field) that jointly optimizes pose and neural scene representations while minimizing the influence of photometric inconsistency and transient occluders on pose estimation. Our contributions are as follows: First, we introduce the novel architecture that enables robust pose optimization with images of complex camera poses. Second, we propose Seature-surroger the deep feature descriptor. Third, we enable transient-free pose optimization with the isolated network. Lastly, our transient-aware depth loss suggests an effective way to impose depth prior only to the static objects. Consequently, our model enables successful training of unposed-NeRF with unconstrained images, showing its effectiveness over a naive combination of current methods.
bundle adjustment by adopting
2 Related Work
Neural Radiance Fields (NeRF). With the advent of techniques adopting neural fields recon- struct 3D representations 25| Neural Radiance Field (NeRF) [ proposed by Mildenhall er al. showed promising result without the requirement of exact 3D geometry of each object in the scene. Compared to previous approaches which require pre-defined representation of the scene by meshes [30 31) voxels 25 26], or point clouds 27 32], NeRF only requires
simple but effective architecture, many other researches 110 12 NeRF to ameliorate its weakness and improve performance.
14 33
Joint optimization of camera pose. Among a lot of interesting NeRF-based works, one major group of following works deals with the requirement of accurate camera pose of each image. Accurate camera pose including intrinsic and extrinsic parameters of camera is indispensable for NeRF in that sampling accurate 3D coordinates in the world space heavily depends on them. However, such an accurate camera pose is not easily equipped unless the dataset is custom-made like a synthetic dataset, so structure from motion (SfM) (36] has been a classical solution obtaín such parameters. Although lot of works have postulated camera poses from SfM to be ground truth, they can be sub-optimal depending on properties of scenes like monotonous texture. solve this problem, other methods (22 37 propose joint optimization of neural field and poses.
dealt with unconstrained outdoor images where color consistency is not guaranteed owing to various environments (sunshine, time, weather, etc.) and so-called transient entities (pedestrian, cars, sundries, etc.) appear in images.
Expansion to unconstrained scenes. Several researches 18 19

---

S-NeRF [4] and CF-NeRF 42 proposed Bayesian network and normalizing flow for measuring the uncertainty of transient objects. DeN [43] and RobustNeRF 44 demonstrated that transient objects can be sharply decoupled from static ones without blurry floaters. Some works 45 46 utilized mask supervision to jointly optimize camera pose with object-centric images of different illumination. In contrast to previous works which require either color consistency or the absence of transient objects to jointly optimize the pose, our method is capable of optimizing camera pose without both requirements.
3 Method
We present a framework dubbed UP-NeRF to optimize unposed-NeRF under challenging conditions. We first briefly summarize Neural Radiance Fields (NeRF) M followed by joint optimization of camera poses and NeRF. In Section 3.1 we propose new architecture that handles different convergence rates of camera poses, which can enable robust training of unposed-Nee Next, we explain our methods in Sectiom3.2 3.2 3.3 that handle the difficulty of unconstrained image collections like different light conditions (e.g. weather, time, etc.) and transient occluders. addition, we will discuss how monocular depth priors can be used to optimize unposed-NeRF with unconstrained images Section 3.4.
Neural Radiance Fields (NeRF). Given N images [1:) and camera parameters, Neural Ra- diance Field (NeRF) Ng learns a continuous volumetric radiance field which maps 3D point (z, y, 2) and viewing direction d = (dg, (y. de) to volume density and color (r, g, b). The mapping functions parameterized by simple neural networks, e.g., multi-layer perceptrons (MLPs), are defined as
(1), x(0) To, (Tx(r())), <(1) = TT, (z(1), ra(d)),
where 75x is a positional encoding function for 3D coordinates of each sample point on the ray r(() = o + (d, which is parameterized by the origin point and the direction d. Color value c(I) is encoded with an intermediate feature z(t) and positional encoding function for view directions na()). The synthesized RGB value C(r) is calculated by volume rendering as
Ö(r) Σ ((t)))((((())))((t)))
k-1
--1 Σ
where T(4x) = II (1 - a(44:)) exp
(1w )ou )0w))
==1
where 6; fx+1 te is the distance between adjacent points for approximated rendering via quadrature on discrete points and attx) NeRF employs coarse and fine models for hierarchical sampling. The coarse model is optimized using stratified sampling, and then the fine model optimized with samples biased toward the relevant parts of the volume. The parameters are optimized by minimizing photometric loss Ligb = C(r C() ll2 with given ground-truth color C(r). Note that even if is a color map. any information that represents the scene can be used for training a neural field. In this paper, we exploit deep feature map F well color map for training, which have been studied by Kobayashi et al. 47 to learn selectively editable neural fields.
exp(-o(tx))a)) To improve sampling efficiency,
NeRF with camera pose estimation. Some prior works optimize NeRF with joint estimation of camera pose. We parameterize the camera poses {p; ) as 6 degrees of freedom and assume that Then, the problem can be
the camera intrinsics are known as several prior works 23 24 formulated as
0" p' = arg min Lig ( C(C) U.
where is learnable camera pose. BARF 23 shows that a coarse-to-fine positional encoding strategy facilitates more stable joint optimization of NeRF and camera pose. Following BARF, we parameterize camera poses with the sc(3) Lie algebra and use a coarse-to-fine strategy for pose optimization.
3

---

3.1 Candidate head for robust pose optimization
Training unposed-NeRI a joint optimization process with a chick-and-egg problem; estimation of camera pose P requires accurate scene representation II and vice versa. During the initial phase of joint training, we observed that inaccurate scene representation, which is represented by predicted density and color in Eq. T causes specific images be optimized in the wrong poses. We refer to these images as 'hard-pose' images, which may include images that capture the details of the scene or have little overlap with other images like the image in Fig. To mitigate this problem, introduce an additional MLP head called candidate head. It yields image-depended (and tentative) scene representations by density (c) and color c, given candidate embeddings ) Ti=1 as:
l0 (t), c (+) 11g, (z(t), (
Note that (-)() denotes that is related to the candidate embeddings. The synthesized RGB value C(f) is generated by joint volumetric rendering of the shared representation (a and c) and the image dependent representation (o()) and color c(+)):
c) = Σ T (ta) ((a((t)6))o(ta) +
k=1
(5)
where (t)) exp Σ (o(tw) o {
The intuition is consider easy pieces first to complete a large jigsaw puzzle and handle hard pieces later. In the initial stages of training, shared representation is predominantly learned by 'easy- pose" images. Mcanwhile, candidate representation is primarily trained by hard-pose images. This remedy effectively prevents the shared representation from being distracted by the wrong supervision introduced by hard-pose images.
As the shared representation becomes accurate enough to facilitate the pose estimation of hard- images, the influence of the candidate head is gradually reduced that hard-post images can be assimilated into the shared representation, and the final model uses only the shared representation, and c. To achieve that, we propose loss scheduling given as:
C = Wu,v Ligh (CC) +
+(1 (1- wu,v) CoblC
(6)
where wa. is defined as:
if 1 <
1 - coscooss (v))
(1)
if usl<v if 1.
(7)
Here 1 [0, 1] denotes training progress and u, e E [0, 1] are hyperparameters. Then, Wayr tunes the influence of c( and Ć. Specifically, for learning during the initial stage of training (1 < u), only O(() is utilized, whereas only Č is employed in the later stage (1 > v). With the loss scheduling, a high-quality final NeRF model can be trained. In final training pipeline,
will use features
instead of colors, as shown in Fig. 2 which will be covered in Section 3.2
The volume rendering method shown in Eq. (5) is similar to NeRF-W 18] in that it has additional head for color and density. In our architecture, however, we do not have a regularization term and uncertainty field that hinder the candidate head from handling hard-pose images. The size of embedding ((() is adjusted by how challenging the poses of the given images are, which can be viewed as adjusting the regularizers for "slack" variables in the constrained optimization problem, e.g., support vector machine (SVM). The quantitative analysis of the candidate embedding size provided in T

---

(c)
7/6
Volumetric rendering
Depth rondering Alpha blending
D
D
Mrno-depth degeh 8
Lo
MLLP(1)
MILP(7)
f
7x(x)
(a(d)
(4)
L
Ligh
Feature Encoder
F
C
Exp
Figure 2: Overall training pipeline of our UP-NeRF (Il is based on Section 3.2). To learn the poses robustly, image-dependent scene representation is learned by the candidate head Tl, (Section 3.1) The depth loss doppt provides transient-aware depth prior to the model (Section 3.4). In the early stages of training. the deep feature F is used for ielo-inndependent surrogate optimization (Sec- tion 3.2). After the poses are roughly learned, the NeRF is optimized with a separated TransientNet Ts to prevent unnecessary gradient flowing the pose parameters (Section 3.3
3.2 Feature-surrogate bundle adjustment
The motivation of this section is shown in Fig. I Given images with diverse appearances due to different weather and times, unposed-NeRF cannot be stably trained with photometric loss since RGB values of corresponding pixels between images
guaranteed to be the same. We adopt the deep features from ViT [49] as a surrogate, which are less sensitive to appearance changes, and we train the neural field with these features achieve better robustness under challenging conditions.
With a slight change in Eq. our feature field is defined as:
f(t) I1,, (z(()),
(8)
(t), (t) Iee et(
Unlike the color c(l) encoding in Eq. (. the feature f(() encoding is performed without viewing direction because the features ought to be independent of viewing directions. Furthermore, p(o) (r) rendered similarly to Eq. (5) by volume rendering with f(t), (t),
p(((()) and of (()) is then optimized by minimizing Crear (r) (r()) (r) with 2D deep feature F((r) from pretrained model. We replace the loss Cogo (C()) C.) in Eq. 6 with the loss Leeat (F F.) for the early training.
Meanwhile, the unconstrained images can have different colors even though they have the same semantic object. As a result, the feature f(t) can be used to encode image-dependent static color c( along with viewing direction d; and appearance embedding (e(()) liml for each image as:
(0)(1) Ie, (T(1), (d(), e(a)).
Note that (.)(0) means it is related to the appearance embeddings. The predicted RGB value ((((() calculated by volume rendering Eq. 2 with c, c((((t) and (t). This static color ( is optimized jointly with the transient color (()) that will be introduced in the next section, Section 3.3 In summary, through candidate scheduling Eq. (6). the feature field trained only for the early stage
(1 < u), Ho, is not involved in learning in the later stage of training (1 2 v), and the radiance field starts training as feature field gradually stops training (u S 1 < v). See Fig. 2 for the final NeRF model II architecture.
5

---

3.3 Isolated network for transient-free pose optimization
The next challenge is the occlusion of transient objects. To optimize the poses accurately, their influence must be minimized in the training process as much as possible. The previous work NeRF- W [8]. which tackles this problem, models in 3D space by adding transient head. If naively adopt it in our model, it causes noisy gradient flowing into the learnable pose parameters, with the pose being learned in the wrong direction due to transient occluders. So we mitigate this problem by handling transients occluders at the separated TransientNer Ta in 2D image level. It maps deep feature (r(r) to opacity (t)), which is intended to filter out the transient based semantic information. Then, along with transient embedding (4/-) and uncertainty (
{ Ji=1- it encodes image-dependent transient color C(7)
[al) 2 (r)) - To. (F.(r)), (r),,, ( (r) = To. (z (r).)
(10)
Note that (.)() means it is related to the transient objects. The final predicted color Cr(r) is obtained
by alpha blending with (( (r) from Section 3.2
(r) (1 - a (T)))
+ ( (()))))))))
(11)
The RGB loss for ray with given color C(r) is:
(r) [[C,()) - C.(r) log 10g (r)
2 + a
(12)
(r)
8f (r) + Amin- As in NeRF-W, the uncertainty (x) is modeled as the where = =
(r) variance of an isotropic normal distribution with mean (r),, resulting the negative log- likelihoooo loss of (r). It reduces the impact of transient occluders as noise when training static objects. Omin > 0 is a hyperparameter for ensuring a minimum importance. The last term suppresses (+) as possible to prevent TransientNet from rendering static objects with Aa > O. We set Amn to 0.1 and Xa to 1.0.
3.4 Transient-awars depth prior
This section is about incorporating geometric prior from monocular depth into NeRF, which is highly inspired by NoPe-NeRF [24]. The mono-depth prior provides strong geometry cues for jointly training pose and NeRF, Even though NoPe-NeRF proposes inter-frame loss to leverage depth prior, it can only be applied to the successive images from a video so we cannot adopt such loss for the unconstrained images. So, we apply only the depth loss Eq. (13). The predicted inverse depth N==1 is acquired from depth network DPT [50]. Since this is a monocular prediction, is not
[DiT>] N an absolute inverse depth value. Therefore learnable parameters { and { which denote scale and shift factor, were introduced for each mono-depth to learn undistorted depth map D It
optimized with rendered NeRF depth D,:
K
(T) = 7h D (T( (r) Es
Di(r) = Σ ((t))(o((W)6)))
(13)
By optimizing the L.1 loss Laepth = (r) (r)ll. NeRF learns a strong geometry prior. However, since NeRF should exclude transient objects, need to apply depth prior only to the areas without transient objects. In Section 3.I the uncertain parts are generated from the candidate
density ((()) of the candidate head, while certain parts are generated from the shared density This implies that the portion of candidate density also filters the transient parts as in Fig. 5 Based on this observation, we can measure transient confidence weight welepth (r) E [0, 1] from the ratio of candidate density o(") to shared density o. In consequence, the depth loss is defined as follows:
Leepth (r) (1 (.lop(())) (r)) [r) - D.(r)
K
(14)
where (!Nepl)(r)

---

Table 1: Novel view synthesis results Photo Tourism Dataset.
PSNR
SSIM 1
LPIPS 4
ref.
BARF
BARF
BARF
UP- NeRF
ref. NeRF-W
BARF
BARF
BARF
UP. NeRF
ref. NeRF-W
BARF
BARF
BARF
UP.
NeRF-W
W
-W
-W
WD
-W
WD
NeRF
Brandenburg Gate
26.50
8.94
11.98
12.45
26.29
0.887
0.221
0.629 0.619 0.884
0.116
1.027
0.549 0.630 0.126
Trevi Fountain
22.26
11.59
14.34
16.10
22.17
0.706
0.152
0.442 0.576 0.691
0.211
1.0%6
0.746 0.487 0.235
Taj Mahal
25.15
7.73
14.13
12.90
24.87
0.858
0.124
0.633 0.624 0.834
0.165
1.053 0.677 0.659 0.209
Sacre Cocur
22.33
8.25
12.51
9.33
21.59
0.827
0.229 0.679 0.651
0.791
0.123
1.027 0.571 0.453 0.157
Mean
24.06
9.13
13.24
12.70 23.73
0.820
0.182 0.596 0.618 0.800
0.154
1.051
0.636
0.557 0.182
Table 2: Camera pose estimation on Photo'Tourish Dataset.
Rotation (*)
Translation J
BARF
BARF-W
BARF-WD
UP-NeRF
BARF
BARF-W
BARF-WD
UP-NeRF
Brandenburg Gate
128.32
21.42
159.00
0.621
4.479
2.890
3.334
0.070
Trevi Fountuin
114.63
33.09
23.08
1.590
6.627
4.763
4.292
0.105
Taj Mahal
94.06
143.30
51.28
0,549
6.054
5.252
3.875
0.120
Sacre Coeur
99.33
112.19
113.50
1.452
12.353
9.875
9.073
0.222
Mean
109.09
77.50
86.72
1.053
7.378
5.695
5.144
0.129
3.5 Training pipeline
The overall training pipeline is shown in Fig. 2 Integrating the depth loss into Eq. (6 modified from Section 3.2 and Section 3.3 the overall loss follows:
Way + (1 (Fre) Ad
(15)
The loss implies that depth loss Edepth decays the training progresses as feature loss Lreat does. This is because we exploit depth supervision only for initial pose estimation. and using it further can degenerate the rendering quality in that the geometry prior we utilize not ground truth but like pseudo-label from depth network. A hyperparameter Xa for depth loss is set to 0.001.
We hierarchical sampling strategy, and there is slight difference in the coarse model loss with Eq. (12). We optimize the TransientNet with the fine model alone, and the coarse model uses detached a and cf render C (r) in Eq. 11 and does not uncertainty. Thus, the final RGB loss for the coarse model is as follows.
1 [ -rC
Lig
Ceourse
(16)
In addition, the way of composing sample set for the fine model is slightly different from NeRF []. which only uses the density a of the coarse model. Along with a, we also need to use candidate density (() until scheduling eliminates the effect of candidate head. Depending on the scheduling factor ways both density a and (()) are used at the beginning. and only is used in the end.
4 Experiments
Datasets. To demonstrate our methods work well in the unconstrained images. We report results on the Phototourism dataset. It consists of internet photo collections of famous landmarks and we select 4 scenes, Brandenburg Gate, Sacre Coeur, Taj Mahal, and Trevi fountain, which are also used N. We follow the split used by NeRF-W 118 and downsample each image by 2 times. All the initial camera poses are set to the identity transformation.
Implementation details. All the models are trained for 600K iterations with randomly sampled 2048 pixel rays at each step with learning rate of 5 10-4 decaying to 5 x 10-5 for NeRF and transient network T, and x 10 decaying to 10-3 for pose p and two factors s; and L, of depth. We use Adam optimizer [51] across all the experiments except test-time appearance optimization, where AdamW [52] is used instead. The number of sampling points each ray for volumetric rendering is set to 128 for both coarse and fine models. We use the default coarse-to-fine strategy
7

---

ground-truth
BARF
BARF-W
BARF-WD
Ours
ref. NeRF-W
Taj Taj-Mahal
N/A
Trevi-Fountain
N/A
Figure 3: Qualitative results of baselines and model, where (*) denotes BARF-variant baselines we implement. As shown the figure (columns 2-4), a naive combination of BARF and other methods failed to converge, resulting in poor synthesis quality. In contrast, model achieves comparable quality to reference NeRF with perfect camera poses.
of BARF which starts from training progress 0.1 to 0.5. We the scheduling parameters and v to be 0.1 and 0.5, respectively, the same as the parameters of coarse-to-fine. Note that both play the same role in helping robust pose estimation in the early stages. The transient confidence weight Wilept is manually detached to prevent those parameters from affecting NeRF directly. We extract deep features from the DINO model 49 whose powerful 2D correspondence representation has
been proven in various works 47 54 55,, and monocular depth from DPT [50) offline. The detailed process of extracting features follows Amir et [49] Full implementation details are provided in the supplementary material.
Evaluation. We implement the evaluation process which consists of two stages, test-time pose optimization, and appearance optimization. In NeRF-W, they do just appearance optimization since trains with given poses. But in our case, we need optimize pose either evaluate novel view synthesis. Therefore, we optimize both pose and appearance on test images for pose test time optimization and then initialize appearance again and optimize it with the optimized pose. In conclusion, the results of novel view synthesis are evaluated using PSNR, SSIM [56), and LPIPS [57]. The train camera poses are trocrustes-alignedd for comparison with ground truth poses in the same way as BARF 23
4.1 Results
To verify the difficulty of our task, compare our model with BARF and its variants. First, we implement BARF- W baseline by applying latent appearance modeling and the transient head of N to BARF The other baseline is relevant NoPe-NeRF 24. NoPe-NeRF succeeded in training NeRF with pose prior in the outdoor scene. But it uses sequence of images which is from video, so most of their principal components are using near-frame images. is strong prior to estimate camera poses, but in-the-wild images do not have such prior so only the mono-depth loss is applicable to our setting. Therefore, we add the loss term Eq. (14) without dodoph to BARF-W (BARF-WD) to check whether the depth prior alone enough to guide the model to robust pose estimation with unconstrained photos.
Table shows the quantitative result of camera pose estimation, and translation errors are scaled by 10. As expected, BARF fails to estimate the poses due to the lack of photometric consistency in the unconstrained image collection. BARF-W also fails optimize, which means that a hard problem that cannot be solved simply by adding appearance modeling and transient head. Although

---

3,000 iterations
15,000 iterations
45,000 iterations
Results
Ground Truth
w/o candid head
Ours
Figure 4: Visual comparison between our model with and without candidate head for a hard pose image during the training process. The small patch images compare the ground -ouund-truth pose (black) with the optimized poses (red).
Wedepth
C
Table 3: Quantitative results of ablation study. All metrics are an average of scene results.
Pose
Novel View Synthesis
Rot (
Trans.
PSNR f SSIM
LPIPS 4
UP-NeRF
1.053
0.129
23.73
0.800
0.182
w/o Lieat
143.4
6.193
13.01
0.568
0.644
w/o To
2.329
0.314
22.51
0.781
0.224
w/o Weleph
1.226
0.170
23.60
0.795
0.215
w/o Weleph efc)
4.972
1.042
21.23
0.744
0.283
Figure Visualization tran-
sient confidence weight.
BARF-WD showed slightly better results, can be seen that the depth prior alone not enough to solve the photometric inconsistency. In contrast, our model succeeds in pose estimation and shows quality result.
Table I and Fig. 3 show the novel view synthesis results. As shown in the figure, baselines have poor rendering quality, which must be rooted in a fallure pose estimation. Even though variants of BARF (columns 3-4) show a slight improvement over BARF, its sub-optimal quality and pose estimation gives strong evidence that mere combination current methods or imposing depth supervision not enough to resolve the problem. In contrast, our model shows comparable performance against reference NeRF-W and a clear appearance that implies its accurate pose estimation.
4.2
Ablation study
In this section, we verify the efficacy of each component in pose estimation with wild photos and provide a visualization of how these help the model optimize pose correctly.
we optimize the radiance field by minimiz- ing Ciea for the early training since the unconstrained images not guaranteed color consistency. When training only with photometric loss Ligb without feature-surrogate it can be seen from Table that it fails to find accurate camera poses and render the novel views. This result shows that the feature-surrogate method is important for the early stage of training.
Feature-surrogate bundle adjustment. In Section 3.2
Transient-free pose optimization. We demonstrate the effect of separating the transient network from NeRF by comparing our model with a baseline where the transient network remains the same as NeRF-W, meaning that transient occluders are rendered from 3D space as static objects. Table (column 2) clearly shows that gradient flows of transient objects adversely affect both pose estimation and NeRF optimization.

---

Table 4: Quantitative analysis on the candidate embedding size.
Table 5: Overall learning time comparison with NeRF- -W
NcRF-W
UP-NeRF
size of g1)
8 16
32
Preprocessing (COLMAP)
Preprocessing (DINO&DPT)
Training
Training
Rotation (*) 4. 4.972 1.286
1.053
3.162
Translation
1.042 0.184 0.129 0,474
Brandenburg Gate
29h 29m
50h 34m
Oh 16m
02m
PSNR t
21.23 23.54
23.73 23.11
Trevi Fountain
150h 06m
51h 24m
Oh 36m
42h 15m
SSIM 1
0.744 0.802
0.800 0.779
Taj Mahal
21h 33m
49h 22m
Oh 20m
42h 13m
LPIPS
0.283 0.184 0.182 0.201
Sacre Cocur
25h 27m
50h 34m
Oh 18m
42h 51m
Transient-aware depth prior. As shown in Fig. 5 confidence score filters transient occluders while imposing depth prior to the static correctly as we intended. Even though the static (e.g.. the gate) is slightly masked, it is justifiable in that the figure is visualized at the early of the training, meaning that the pose not fully optimized and the model is still being trained to discern the static from the transient.
Candidate head. The last row of Table shows that candidate head plays crucial role in optimizing camera pose accurately. Additionally, we provide quantitative analysis on the candidate embedding g size in Table 4 We observed that setting the candidate embedding to an appropriate size is of great help to camera pose estimation, and eventually succeeded in synthesizing novel view images.
4.3 Analysis
Candidate head. Fig. 4 shows that candidate embedding facilitates appropriate pose estimation for hard pose image, inducing the model not to vainly struggle to overfit the scene without updating the pose. In contrast the baseline without the candidate head which stuck in local minima, our model successfully estimates the pose. The figure implies that if the rendered scene at the early stage similar the ground truth with an incorrect pose, the baseline tries to fit the scene by optimizing NeRF, rather than pursuing accurate pose optimization. However, our model is more robust against being trapped into pose local minima for the hard pose images.
Overall training time We compared overall learning time between NeRF-W W [8 and UP-NeRF in Table 5] Before training NeRF-W, it is necessary to run COLMAP to get camera poses. The time required for COLMAP depends on the number of images, and the execution time becomes non-negligible as the number increases. For example, Trevi fountain with about 3200 images, takes nearly I week to finish. However, UP-NeRF takes just minutes to prepare training, which includes extracting DINO feature and DPT depth maps. The training time of UP-NeRF is also faster than NeRF-W because it renders transient objects directly from the feature maps without a volume rendering process. UP-NeRF only requires about 30 minutes of preprocessing time, so we expect using faster models such as Instant-NGP [88 to reduce the training time greatly.
5 Conclusion
We propose UP-NeRF, robust unposed-NeRF that can learn with unconstrained image collection with variable illumination and transient occluders. Thanks to the color-insensitive feature field, separated transient network, candidate head for robust pose optimization, and transient-aware depth prior, our model has less difficulty in estimating poses even in such challenging conditions. The experiment shows the promising results of our model compared to baselines, even comparable to the reference model with accurate poses.
Acknowledgments and Disclosure of Funding
This work was partly supported by ICT Creative Consilience program (ITTP-2023-2020-00001 supervised by the IITP, and the National Research Foundation of Korea (NRF) grant funded by the Korea govermment (MSIT) 22222222R1222200500
10

---

References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[2] Nianchen Deng, Zhenyi He, Jiannan Ye, Praneeth Chakravarthula, Xubo Yang, and Qi Sun. Foveated neural radiance fields for real-time and egocentric virtual reality. arXiv preprint arXiv:2103. 16365, 2021.
[3] Sixu Li, Chaojian Li, Wenbo Zhu, Boyang Yu, Yang Zhao, Cheng Wan, Haoran You, Huihong Shi, and Yingyan Lin. Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction. In ISCA, 2023.
[4] Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. ICLR, 2023.
[5] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
[6] Jeffrey Ichnowski, Yahav Avigal. Justin Kerr, and Ken Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. In CoRL, 2021.
[7] Michal Adamkiewicz. Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. IEEE Robotics and Automation Letters, 2022.
[8] Kunlong Hong. Hongguang Wang, and Bingbing Yuan. Inspection nerf:: Rendering multi-type local images for dam surface inspection task using climbing robot and neural radiance field. Buildings, 2023.
[9] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[10] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on diet: Semantically consistent few-shot view synthesis. ICCV, 2021.
[11] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes. In CVPR, 2021.
[12] Sara Kridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa, Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[13] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
[14] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In ICCV, 2021.
[15] David B. Lindell, Julien N.P. Martel, and Gordon Wetzstein, Autoint: Automatic integration for fast neural volume rendering. In CVPR, 2021.
[16] Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In ICCV, 2021.
[17] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
[18] Ricardo Martin-Brualla Brualla, Noha Radwan, Mehdi M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[19] Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang. Hallucinated neural radiance fields in the wild. In CVPR, 2022.
11

---

[20] Johannes L Schonberger and Jan- MincMel Frahm. mottoon revisited. In CVPR, 2016.
[21] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF--- Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021.
[22] Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In ICCV, 2021.
[23] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In ICCV, 2021.
[24] Wenjing Bian, Zirui Wang, Kejie Li, Jia- Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. In CVPR, 2023.
[25] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In CVPR, 2020.
[26] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In NeurIPS, 2016.
[27] Riccardo Roveri, A Cengiz Öztireli, Ioana Pandele, and Markus Gross. Pointpronets: Con- solidation of point clouds with convolutional neural networks. Computer Graphics Forum, 2018.
[28] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. In NeurlPS, 2019.
[29] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020.
[30] Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. Paparazzi: surface editing by way of multi-view image processing. ACM Trans. Gruph., 2018.
[31] Chia- Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. Beyond volumetric albedo-a surface optimization framework for non-line-of-sight imaging. In CVPR, 2019.
[32] Eldar Insafutdinov and Alexey Dosovitskiy. Unsupervised learning of shape and pose with differentiable point clouds. In NeurlPS, 2018.
[33] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualle and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
[34] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[35] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR, 2022.
[36] Alex M Andrew. Multiple view geometry in computer vision. Kybernetes, 2001.
[37] Yitong Xia, Hao Tang, Radu Timofte, and Luc Van Gool. Sinerf:
Sinusoidal neural radiance
fields for joint pose estimation and scene reconstruction. In BMVC, 2022.
[38] Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. In ECCV, 2022.
[39] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. CVPR, 2023.
[40] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. GNeRF: GaN-based Neural Radiance Field without Posed Camera. In ICCV, 2021.
12

---

[41] Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguee Stochastic neural
radiance fields: Quantifying uncertainty in implicit 3d representations. 3DV, 2021.
[42] Jianxiong Shen, Antonio Agudo, Francesc Moreno-Noguer, and Adria Ruiz. Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification. In ECCV, 2022.
[43] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D
2 nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. In NeurlPS, 2022.
[44] Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David Fleet, and Andrea Tagliasac- chi. Robustnerf: Ignoring distractors with robust losses. In CVPR, 2023.
[45] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained real- world arbitrary image collections. In NeurlPS, 2022.
[46] Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. Neroic: neural rendering of objects from online image collections. TOG, 2022.
[47] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurlPS, 2022.
[48] Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. Local-to- global registration for bundle-adjusting neural radiance fields. In CVPR, 2023.
[49] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2 /2.05814, 2021.
[50] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
[52] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017.
[53] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[54] Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani. LASSIE: Learning Articulated Shape from Sparse Image Ensemble via 3d part discovery. In NeurIPS, 2022.
[55] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. In CVPR, 2023.
[56] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
[57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[58] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 2022.
13


Reviews:
[{"review_id": "xOfGWUeKJo", "reviewer": "Reviewer_7yJa", "summary": "The paper proposes a joint camera pose and NeRF optimisation method that can handle transient scenes, including moving objects and various light conditions, by integrating NeRF-W, BARF, NoPe-NeRF, and DINO-based feature-metric loss in a sophisticated way. \n\nThe method is evaluated on four scenes in the Phototourism dataset, showing good performance compared with (modified) BARF baselines.\n\n---\n**After rebuttal**: I have read authors' rebuttal and it addresses my concerns.", "questions": "See the weakness section.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It’s a challenging task to estimate camera poses in scenes with moving objects and different light conditions, especially in a NeRF setup. This method proposes to make the pipeline more robust by considering\n* Moving objects and lighting conditions similar to NeRF-W;\n* Pose optimisation similar to BARF;\n* Monocular depth un-distortion similar to NoPe-NeRF; and \n* DINO-based Feature-metric.\n\nIn short, the proposed method is novel and leads to promising results. It’s also a plus that code is also provided in supplementary.", "weaknesses": "I think a primary concern is how robust the pose estimation is in more scenes. The method is only evaluated in 4 scenes. Is it possible to have an experiment, especially for pose estimation successful rate on more scenes?\n\nI understand that it takes a long time for NeRF to reach the best rendering quality, but we should be able to tell if the pose optimisation is successful far before NeRF converges. For example, we can consider pose estimation successful if rotation error is lower than $\\theta$ degrees in $x$ epochs. In this way, we can see the pose estimation success rate in many more scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the constructive comments, including new experimental settings. We address the reviewer's questions below.**\n\n---\n**Comment 1**: Experiments in more scenes.\n\n**Answer**: \nWe provide more experimental results. As requested, we set an error criterion of 20 degrees, which was used by NeRF-- [3] to determine whether the pose estimation was successful or not . And at 10% of the total iterations, it was measured whether it was lower than the error criterion. The result is provided in Fig. 2 of the PDF. The experiments demonstrate that our method successfully learns pose estimation (< 20 degree) in all five scenes below before 10% of the total iterations. \n\n- **3 more scenes in phototourism dataset.** Since NeRF-W did not release pre-processed images of all scenes, we evaluated the model only in four scenes (Brandenburg Gate, Taj Mahal, Trevi Fountain, and Sacre Coeur) on the phototourism dataset in the main paper. We implemented the preprocessing step as similarly as possible and evaluated three more scenes (Buckingham Palace, Pantheon Exterior, Nara Temple).\n- **2 more scenes in outdoor image dataset.** We additionally presented a nice outdoor image dataset with color variation and transient occluders as in NeRF-OSR [2].\n\n### References\n[1] NIMA: Neural image assessment. IEEE TIP, 2018.\n\n[2] NeRF-OSR: Neural Radiance Fields for Outdoor Scene Relighting. ECCV 2022.\n\n[3] NeRF−−: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064,338 2021.\n"}, {"review_id": "jDZWxEuLZg", "reviewer": "Reviewer_6T1t", "summary": "The paper proposes a novel method for optimizing NeRF without a pose-prior and on in-the-wild image collections containing transient occluders and varying lightings. The main contributions are four fold. Firstly, the authors propose a candidate head for NeRFs that uses image-level representations via a learned embedding for compensating inaccurate poses during the early optimization. Secondly, learning a view-independent feature field based on DINO features as intermediate representation increases the robustness of the joint optimization wrt Varying lightings, weather and time. To reduce the impact of transient occluders the authors suggest using a separate network that predicts occluder in 2D image level based on the feature maps. Additionally, to achieve higher accuracy in the geometry the authors apply monocular depth supervision on regions without occluders. Experiments are conducted on four scenes of the Phototourism dataset with initial camera poses set to the identity.", "questions": "1) The explanation of the loss schedule applied to the candidate head training makes sense however there is no ablation study. What would happen if the weight is not reduced? Why should there be a negative impact if the loss?\n2) Can you explain the z(t) in equation 1?\n3) Would it make sense to introduce a candidate head with feature output in equation 4?\n4) Please provide evidence about the computation time, see Weakness 2.\n", "limitations": "Limitations are discussed in the supplementary.\nI'd suggest that the authors discuss the overall optimization time as a general limitation of the method, if applicable. There are state-of-the-art NeRF architecture, such as Instant-NGP that facilitate optimization in a few minutes instead of hour or days, which might be a good follow up.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The paper is very nicely presented and easy to read.\n2) The paper tackles the challenging and relevant problem of in-the-wild NeRF reconstruction without given posen. The contribution is clear and solves different subproblems that emerge in this domain, e.g. transient occluders.\n3) Most parts are well motivated, the methodology is technical sound and experimentally justified e.g.:\n - Candidate head sounds plausible and improves pose optimization and image quality significantly, see Figure 4 and Table 3.\n - Feature field optimization seems to help for in-the-wild images, see Table 3.\n - Depth supervision improves performance, see Table 3 and Figure 5.\n4) The experimental section contains a comparison to BARF, adapted variants with additional supervision cues and numbers for NeRF-in-the-wild. It appears to be a plausible choice and supports the contribution of the paper.\n5) The authors provide code in the supplementary materials. \n", "weaknesses": "1) The candidate head is introduced to output color and density, equation 4, however it is later used to predict features, see Figure 2. This appears to be misleading for the readers in the beginning.\n2) One of the main limitations of NeRF based methods is the optimization time and an analysis is missing on that. It would be good to discuss the optimization time for the method and the baselines. To get a sense if the time and computational effort is comparable to COLMAP + NeRF-W and others.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1**: Mismatch between Equation 4 and Figure 2. \\\n**Answer**:  In lines 158-159, we mentioned the loss $\\mathcal{L}_{\\text{rgb}}$ is replaced with the loss $\\mathcal{L}_{\\text{feat}}$, but readers can be confused because feature-surrogate bundle adjustment (Sec 3.2) was described after candidate head (Sec 3.1). We will revise the text (e.g. reorder the sections) so that it can be delivered more clearly.\n\n---\n**Comment 2-1**: Analysis of computation time between baseline and UP-NeRF.\\\n**Answer**: Here, we share a comparison of training time including preprocessing time. Our method requires a relatively shorter training time. Also, NeRF-W needs to obtain camera poses with COLMAP, and it takes over 25 hours as a preprocessing step. However, 10 minutes is enough for UP-NeRF (ours) to prepare DINO features and DPT mono-depth predictions for training. \n\n|  | NeRF-W | | UP-NeRF | |\n| --- | --- | --- | :---: | --- |\n|  | Preprocessing (\bCOLMAP) | Training | Preprocessing (DINO&DPT) | Training |\n| Brandenburg Gate | 29h 29m | 50h 30m | 10m | 46h 30m|\n| Sacre Coeur | 25h 27m | 50h 30m | 10m | 46h 30m|\n\n**Comment 2-2**: Long optimization time as a general limitation of NeRF. Incorporate the proposed method into faster models such as InstantNGP as a future direction.\\\n**Answer**: That is a great suggestion. It’ll be good future work. UP-NeRF only requires about 30 minutes of preprocessing time, so using faster models such as Instant-NGP will greatly reduce the training time. According to our quick research, Instant-NGP has not been proven effective on in-the-wild datasets yet. We believe Tackling the challenge and applying the proposed method will lead to interesting technical contributions.\n\n---\n**Comment 3**: Ablation study of loss scheduling \\\n**Answer**: In our final loss Eq. (16), when the schedule is crucial for the initial pose optimization because of color inconsistency. We provide the ablation study of loss scheduling at Brandenburg Gate, and the results table is below. Without loss scheduling, a significant degradation was observed.\n\n|Method |w/o scheduling|Ours|\n| --- | --- | --- | \n|Rotation (°) ↓|172.7|**0.797**|\n|Translation ↓|3.796|**0.148**|\n|PSNR ↑|11.33|**23.60**|\n|SSIM ↑|0.607|**0.801**|\n|LPIPS ↓|0.615|**0.180**|\n\nIf you want more explanation of candidate head, please refer to the Author Rebuttal.\n\n---\n**Comment 4**: Explanation of $\\mathbf{z}(t)$ in equation 1. \\\n**Answer**: We apologize we didn’t explain $\\mathbf{z}(t)$ notation properly. $\\mathbf{z}(t)$ is the intermediate feature of MLP, which is just an output of MLP $\\theta_1$. It will pass through both MLPs $\\theta_2$, $\\theta_1$.\n\n---\n**Comment 5**: Would it make sense to introduce a candidate head with feature output in equation 4?\\\n**Answer**: The MLP parameterized by $\\theta_1$ a base network shared by two heads: shared head and candidate head. The feature $\\mathbf{z}(t)$ is output of MLP $\\theta_1$ and it is transformed the positional information. The feature (or transformed position embedding) was fed into MLP theta 3 as input. "}, {"review_id": "JT1fHGHemX", "reviewer": "Reviewer_qVYM", "summary": "The paper produces a novel approach for optimisation of pose in NeRF scenarios. The core novelty is the addition of a candidate head that improves network stability when the images poses are not yet converged, along with some other tweaks like a transiency inference head or a feature field. \n\nEdit: I have read the rebuttal, and given the scores from other reviewers and I would like to keep my score.", "questions": "* Figure 5 seems to show a case where the transiency is wrong … should the horse really be transient?", "limitations": "* To some extent, but the paper could benefit from a more clear failure case section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The idea is novel, interesting and timely.\n* The results show good improvements over the current s-o-t-a.\n* The experiments section is thorough enough to demonstrate the qualities of the proposed approach. \n\nOverall, while my negatives might sound long and would seen to outweigh the positives, I think it brings a potentially valuable contribution to the community, so I very much support the acceptance of the paper. My main worry is about the somewhat confusing and short explanation of the core contribution of the “candidate” heads, which should be expanded and improved.", "weaknesses": "* The paper is somewhat confusing to read at times. For example “unconstrained images” are introduced and used without introducing the term. The meaning does become later in the paper, but it would be good to introduce things earlier on. Similarly, I do not see much point for Figure 1, as features have been used for this type of matching for years.\n* Even though the literature review section of the paper is quite good, I think the paper does anchor things a bit too much on BARF, and ignores other works such as NoPe-NERF / GARF / etc in the comparisons. I do understand that some of these works were arxiv at the time of the submission (but are publications now), so this should not penalise the paper too much. That being said, it’d be nice if some of these comparisons could be added.\n* I also found some of the notation difficult to follow initially as I found not find any outline of it’s meaning. An example is the (c) in line 126 which is not explained. \n* Looking at the core contribution of the paper (i) the intuition (i.e. the “roughly speaking … “ part at line 125+) could be expanded and (ii) the size of the embedding should have been ablated in the results section.\n* The results section could have been expanded e.g. with (i) the ablation noted above, (ii) extra ablations where, e.g., the feature matching part is turned off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the detailed review and comments. We appreciate your support for our work. The questions will be addressed below.**\n\n---\n**Comment 1-1**: The definition of “unconstrained images” should be introduced earlier on. \\\n**Answer**: We appreciate for the good point. We will add the definition of “unconstrained images” in line 23 where the concept is firstly introduced. \\\n \\\n**Comment 1-2**: Similarly, I do not see much point for Figure 1, as features have been used for this type of matching for years. \\\n**Answer**: There have been many works that use deep features to find matching points between images. However, there have been no attempts to use feature fields to optimize NeRF and pose jointly. This approach, optimizing the feature field instead of the radiance field, succeeded to tackle unconstrained images, unlike prior works.\n\n---\n**Comment 2**: The review section has too much focus only on BARF. \\\n**Answer**: In lines 242-246 of the main paper, although NoPe-NeRF is the most relevant baseline which succeeded in training unposed-NeRF in the outdoor scene, it requires training data to be successive frames in a video sequence, but phototourism dataset is not. So, we cannot get a performance of NoPe-NeRF. However, we agree with you that we need more comparisons, so we share the quantitative results of GARF below. It fails to estimate the poses like BARF because it uses photometric loss for unconstrained image optimization.\n\n|  Scenes | ​　　PSNR ↑  | ​　　SSIM ↑ | ​　　LPIPS ↓ | Rotation (°) ↓ | Translation ↓ |\n| --- | --- | --- | --- | --- | --- |\n|   |GARF　　Ours|GARF　　Ours|GARF　　Ours|GARF　　Ours|GARF　　Ours\n| Brandenburg Gate | 07.75　　**25.79** | 0.211　　**0.881** | 1.074　　**0.122** | 133.5　　**0.426** | 4.659　　**0.058** |\n| Trevi Fountain | 10.82　　**22.44** | 0.218　　**0.693** | 1.112　　**0.233** | 154.4　　**1.498** | 7.232　　**0.127** |\n| Taj Mahal | 11.71　　**25.05** | 0.268　　**0.839** | 1.033　　**0.203** | 91.13　　**0.619** | 5.442　　**0.183** |\n| Sacre Coeur | 10.53　　**21.10** | 0.204　　**0.791** | 1.084　　**0.160** | 82.54　　**1.516** | 11.64　　**0.224** |\n| Mean | 10.20　　**23.60** | 0.225　　**0.801** | 1.076　　**0.180** | 115.4　　**0.797** | 7.243　　**0.148** |\n---\n**Comment 3**: Difficulty of notation in the paper. \\\n**Answer**: We will find a way to simplify the notations and reflect them in the final version.\n\n---\n**Comment 4**: Intuition of candidate embedding and ablation study of its size. \\\n**Answer**: We added a more detailed explanation about candidate head to Author Rebuttal. And we share the ablation study on candidate embedding size below. You can see that the performance is not much sensitive to the candidate embedding size, however, there is a big difference when the candidate embedding is not used.\n\n|  Size of $\\ell^{(c)}_i$ | ​　0  | ​　16 | ​　32 | ​　64 \n| --- | --- | --- | --- | --- |\n| Rotation (°) ↓ | 4.358 | 1.225 | 1.057 | **1.047**\n| Translation ↓ | 0.843 | 0.164 | **0.147** | 0.152\n| PSNR ↑ | 21.45 | 23.26 | **23.47** | 23.40\n| SSIM ↑  | 0.745 | **0.800** | 0.799 | **0.800**\n| LPIPS ↓ | 0.294 | **0.181** | 0.183 | 0.183\n\n---\n**Comment 5**: More ablation studies \\\n**Answer**: We agree with you and we have several additional experiments for ablation including the feature matching part. You can see the results at Brandenburg Gate below. As we mentioned in lines 149-152 of the main paper, it is difficult to estimate pose by using photometric loss because of the diverse appearances of images. As expected, UP-NeRF without feature-surrogate fails to estimate the poses.\n\n|  Method | w/o feature-surrogate  | w/o scheduling | Ours \n| --- | --- | --- | --- | \n| Rotation (°) ↓ | ​　　　　　175.3 | ​　　　172.7 | **0.797** \n| Translation ↓ | ​　　　　　3.846 | ​　　　3.796 | **0.148** \n| PSNR ↑ | ​　　　　　12.29 | ​　　　11.33 | **23.60**\n| SSIM ↑  | ​　　　　　0.645 | ​　　　0.607 | **0.801** \n| LPIPS ↓ | ​　　　　　0.666 | ​　　　0.615 | **0.180**\n\n\n---\n**Comment 6**: Figure 5 seems to show a case where the transiency is wrong … should the horse really be transient? \\\n**Answer**: Static objects (non-transient objects) mean objects that are always there, but “the horse statue” in Figure 5 doesn't appear in every image. It is a temporary installation."}, {"review_id": "Babi9K2bsj", "reviewer": "Reviewer_Kmtq", "summary": "This paper tackles NeRF training from in-the-wild Internet photos without pre-computed camera poses. The main idea is to leverage (self-supervised) image features and a carefully designed optimization strategy, together with ideas from existing work, including modeling transient regions and using mono-depth supervision. These components work in combination to avoid local minima and enable joint optimization of both pose and NeRF from scratch.\n\nExperiments show that this joint optimization pipeline successfully recovers camera poses in several in-the-wild scenes and leads to good NVS results comparable to those with COLMAP preprocessing, whereas the previous work of BARF and its variants all fail.", "questions": "1. Clarify the motivation for the \"candidate embeddings\";\n2. If possible, present more evaluation results on other datasets **with the same hyperparameters and training schedule**, to validate the robustness;\n3. It might be difficult as the pipeline is indeed complicated, but if possible, it would be helpful to further simplify the notations, in particular, the super- and sub-scriptions.", "limitations": "The authors included a brief paragraph on the limitation of the image features. I would expect some discussion on the robustness of the proposed pipeline.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "### S1 - Technically sound approach to a challenging problem\n- The task of estimating camera poses on raw in-the-wild Internet photos using a gradient-descent-based optimization is challenging, due to noisy correspondences and local minima.\n- The proposed method incorporates many good ideas from existing work and works well on a number challenging scenes.\n-- For instance, the use of self-supervised image features for registering semantic correspondences via rendering;\n-- The mechanism to model transient objects using learned opacity. This paper further adapts the original volumetric opacity in NeRF-W to per-image opacity, which is claimed to be more effective.\n-- The use of depth supervision from pre-trained models.\n\n### S2 - Promising results\n- The paper demonstrates good results on a number of challenging scenes using Internet photos, where existing methods clearly fails.", "weaknesses": "### W1 - Complicated pipeline\n- The resulting pipeline is complicated, involving many components, eg, 6 MLPs in total, and easily becomes confusing. It took me several passes back and forth to understand the exact implementation.\n- It also requires a heavily crafted training schedule, gradually activating and deactivating some of the components.\n- A critical concern on such a pipeline is its robustness across various scenes. Would one need to fight against all the hyperparameters and the training schedule when training on other scenes. I strongly suggest the authors also present results on other standard datasets (W3).\n\n### W2 - Unclear motivation and potential redundancy of some technical designs\n- It is unclear to me why the per-image \"candidate embeddings\" can help with the pose optimization. It seems the motivation is to allow some of the per-image variations (multi-view inconsistencies) to be factored into these per-image embeddings. The paper presents ablation results which shows numerical benefits of this component, but why is it useful for pose estimation?\n- Are they still useful in general in the case of a multi-view static scene without transient entities?\n- Isn't this the job of the separate mechanism for handling transient regions?\n- Also, it is confusing to me why the transient confidence weights $\\mathcal{W}_i^{\\text{depth}}$ are calculated from the candidate densities $\\sigma^{(c)}$, rather than from the transient opacities $\\alpha^{(\\tau)}$. Are there some redundancy between the two?\n\n### W3 - Only on one dataset\n- The paper only presents results on one dataset, consisting of Internet photo collections of 4 scenes, which the method is specifically tailored to.\n- However, the paper claims to solve general \"unposed NeRF\". I strongly recommend the authors to also test it on standard multi-view datasets, eg DTU, CO3D etc, to assess the robustness of this complicated pipeline.\n\n### (minor) W4 - Do intrinsics matter?\n- How are the camera intrinsics obtained? From COLMAP? If so, this would undermine the value of avoiding cumbersome COLMAP preprocessing.\n- Or are they estimated or assumed to be some value for all images?\n\n### Other comments\n- There are a few existing work that leverages DINO for pose estimation of objects, which the authors should consider referencing, eg: Zero-Shot Category-Level Object Pose Estimation [1], LASSIE [2], MagicPony [3].\n- Line 125: when the \"candidate embeddings\" are first introduced, it was not immediately obvious to me they are per-image embedding vectors. Also, are they jointly optimized? The term \"candidate head\" is also quite obscure to me.\n- Fig 2: the blue arrow in the middle connecting $\\hat{\\mathbf{F}}_i^{(c)}$ to $\\theta_4$ is slightly inaccurate, as the $\\hat{\\mathbf{F}}_i^{(c)}$ is the result of ray integration, whereas the input to $\\theta_4$ should be the raw per-point feature $\\hat{\\mathbf{f}}_i$, if I understand correctly.\n- What is the computational overheads compared to vanilla NeRF?\n\n### References:\n- [1] Zero-Shot Category-Level Object Pose Estimation. ECCV 2022.\n- [2] LASSIE: Learning Articulated Shape from Sparse Image Ensemble via 3D Part Discovery. NeurIPS 2022.\n- [3] MagicPony: Learning Articulated 3D Animals in the Wild. CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the detailed comments and suggestions. We address all the questions below. We hope that our answers resolve the reviewer's concerns and lead to support.**\n\n---\n**Answer of \"W1 - Complicated pipeline with hyperparameters\"**\n\nAlthough our method has several hyperparameters, the method is relatively robust to the choice of hyperparameters. We here analyze the effect of the candidate embedding size $\\ell^{(c)}_i$. Our experiments on the phototourism dataset show that when the candidate embedding size is reasonably big $(\\ge 16)$, it stably achieves good performance. All other hyperparameters are set in the same way as previous works.\n\n|Size of $\\ell^{(c)}_i$ |0|16|32|64\n| --- | --- | --- | --- | --- |\n|Rotation (°) ↓|4.358|1.225|1.057|**1.047**\n|Translation ↓|0.843|0.164|**0.147**|0.152\n|PSNR ↑|21.45|23.26|**23.47**|23.40\n|SSIM ↑|0.745|**0.800**|0.799|**0.800**\n|LPIPS ↓|0.294|**0.181**|0.183|0.183\n---\n**Answer of \"W2-1 Motivation and more explanations of candidate head.\"**\n+ **Candidate head (motivation and intuition)**: \\\nThank you for asking for more explanations of our contributions. We provided more explanations of the candidate head in the General Response (Author Rebuttal) above. Please refer to it.\n+ **The usefulness of candidate embedding in the case of a multi-view static scene without transient entities**:\\\nOur candidate head prevents the hard-pose images from falling into local minima. Handling transient regions is only an additional role. To validate the usefulness of candidate embedding, we had additional experiments on the BLEFF dataset presented by the authors of NeRF-- [3], which is a synthetic dataset without transient entities with several perturbations by random rotations and translations. As the table below shows, candidate embedding shows more robust results as the poses of images become more perturbed.\n\n|  Scene | Perturb | ​PSNR ↑  | | SSIM ↑ | | LPIPS ↓  | | Rot (°) ↓ | | Trans↓  | |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ----- | --- |\n|  |  |BARF|BARF+$\\ell^{(c)}_i$|BARF |BARF+$\\ell^{(c)}_i$|BARF   |BARF+$\\ell^{(c)}_i$|BARF  |BARF+$\\ell^{(c)}_i$|BARF  |BARF+$\\ell^{(c)}_i$\n| Classroom | 20 | 25.43|26.13 (**+0.70**) | 0.877|0.893 (**+0.016**) | 0.133|0.105 (**-0.028**) | 0.265|0.293 (+0.028) | 0.009|0.022 (+0.013) |\n| Classroom |30 | 12.36|21.36 (**+9.00**) | 0.480|0.806 (**+0.326**) | 0.749|0.183 (**-0.566**) | 29.17|6.182 (**-22.99**) | 2.176|0.354 (**-1.822**) |\n||||||||||||\n| Bed |20 | 37.42|37.06 (-0.36) | 0.969|0.967 (-0.002) | 0.050|0.048 (**-0.002**) | 0.839|0.296 (**-0.543**) | 0.243|0.230 (**-0.013**) |\n| Bed |30 | 13.61|32.47 (**+18.86**) | 0.449|0.890 (**+0.441**) | 0.897|0.142 (**-0.755**) | 141.8|7.077 (**-134.7**) | 16.57|10.43 (**-6.14**) |\n\n**Answer to \"W2-2 Redundancy of some technical designs.\"**\n+ **Redundancy between the transient opacities and the candidate densities**: \\\nIf we use the transient network in the early stage of training, it will generate static regions as well as transient regions because the camera poses are not learned. So we should use the candidate densities for transient confidence weights.\n\n---\n**Answer of \"W3 - Only on one dataset\"**\n\nAs mentioned, we had additional experiments on BLEFF dataset which is a standard multi-view dataset. We use the same hyperparameters and training schedule. We will add more scene results in the final version.\n\n---\n**Answer of \"W4 - (minor) W4 - Do intrinsics matter?\"**:\n\nAs previous works such as BARF, SPARF, and NoPe-NeRF, we assume that intrinsic parameters are given. Our work focuses only on estimating extrinsic parameters (camera pose). In real-world applications, we may get intrinsic parameters from image metadata. Also, we may leverage previous works [1, 2] to estimate the intrinsic parameters.\n\n---\n**Other comment 1**: Missing references.\n\n**Answer**: Thank you for letting us know missing reference. We will include these in the final version.\n\n---\n**Other comment 2**: Fig 2 is slightly inaccurate.\n\n**Answer**: We appreciate it for pointing out an inaccurate connection in our figure. The update is available in Fig. 1 in PDF.\n\n---\n**Other comment 3**: What is the computational overheads compared to vanilla NeRF?\\\n**Answer**: Vanilla NeRF is not suitable for in the wild dataset, so we conducted comparisons with NeRF-W. You can refer to the table below. NeRF-W needs to obtain camera poses with COLMAP, and it takes over 25 hours. However, 10 minutes is enough for UP-NeRF to prepare for training; the process of obtaining DINO features and DPT mono-depth predictions is included.\n\n|  | NeRF-W | | UP-NeRF | |\n| --- | --- | --- | :---: | --- |\n|  | Preprocessing (\bCOLMAP) | Training | Preprocessing (DINO&DPT) | Training |\n| Brandenburg Gate | 29h 29m | 50h 30m | 10m | 46h 30m|\n| Sacre Coeur | 25h 27m | 50h 30m | 10m | 46h 30m|\n\n---\n**Other comment 4**: Complicated notations.\\\n**Answer**: We agree with your opinion, we will try to find out the way to simplify the notations.\n\n---\n### References:\n[1] DeepCalib: A Deep Learning Approach for Automatic Intrinsic Calibration of Wide Field-of-View Cameras\n\n[2] DeepPTZ: Deep Self-Calibration for PTZ Cameras\n\n[3] NeRF−−: Neural radiance fields without known camera parameters.\n\n"}, {"review_id": "OYXv6Y2SEF", "reviewer": "Reviewer_FYmx", "summary": "This paper solves an interesting problem — joint pose and NeRF optimization on in-the-wild image collections. Unlike prior works such as BARF, this work aims at handling unconstrained images with varying illumination and transient occluders. To tackle this problem, this work incorporates learnable camera parameters, depth prior, and semantic features from DINO as supervision into the NeRF-W framework. Experiments demonstrate that the proposed method outperforms BARF and its variants on unstructured Internet images.    ", "questions": "What is the effect of using semantic features as supervision, i.e. L_feat?  \n", "limitations": "I did not find discussions on the limitations of this paper. It would be great to discuss the limitations and analyze the possible reasons and future works.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.  This work identifies an interesting and important research problem — joint pose and NeRF optimization on in-the-wild images. Prior works on the joint pose and NeRF optimization are limited to controlled settings. \n2. The proposed method is intuitive and combines the NeRF-W framework and several components (e.g., depth prior, semantic features, etc.) from other works (such as NoPe-NeRF).  ", "weaknesses": "1. The main contribution of this paper is the problem setting — pose and NeRF estimation in the wild. The proposed method combines NeRF-W and BARF, which has limited technical novelty. \n2. The presentation may be improved. Figure 1 and Figure 2 have not been referred to and explained in the main text. \n3. Missing references:  Meng et al. GNeRF: GAN-based Neural Radiance Field without Posed Camera. ICCV 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate your constructive feedback on our work and we will reflect them. Below are our responses to the reviewer's questions.**\n\n---\n**Comment 1**: The proposed method has limited technical novelty.\\\n**Answer**: As the reviewer admitted, Unposed-NeRF in the outdoor scene is a challenging problem, and only a handful of recent works (e.g., NoPe-NeRF) partially addressed this (clean images NOT in-the-wild setting). As reviewer qVYM and others mentioned, we believe that our work is novel. Furthermore, as shown by BARF-W in Table 1 and 2 , a simple combination of NeRF-W and BARF fails to optimize unposed-NeRF with unconstrained images. \nTo solve the problem, we proposed 4 novel techniques: candidate head (Sec3.1), feature-surrogate bundle adjustment (Sec3.2), isolated transient network (Sec3.3) and transient-aware depth prior (Sec3.4). \n\n---\n**Comment 2**: Figure 1 and Figure 2 have not been referred to and explained in the main text.\\\n**Answer**: We will add more explanations in the main text with explicit figure numbers. \n\n---\n**Comment 3**: Missing references.\\\n**Answer**: Thank you for letting us know the missing reference. We will include these in the final version.\n\n---\n**Comment 4**: What is the effect of using semantic features as supervision, i.e. L_feat? \\\n**Answer**: As we mentioned in lines 149-152 of the main paper, because of the diverse appearances of images it is difficult to optimize the model with the photometric loss. In contrast, DINO features, or semantic features, are less sensitive to the variation of appearance, as shown in Figure 1. Therefore, by using semantic features as supervision, UP-NeRF succeeded in jointly optimizing NeRF and camera poses, unlike the prior methods that are based on photometric loss (See Table 1, 2 and Figure 3).\n\n---\n**Comment 5**: I did not find discussions on the limitations of this paper.\\\n**Answer**: We discussed the limitations in the supplement, and additionally shared some failure cases in the Fig. 3 in pdf."}]

Author rebuttal:
Multiple reviewers asked about our Candidate Head. We address the common comment below. 

Q. **Additional explanation of Candidate Head**

During the initial phases of joint training for NeRF and camera pose estimation, NeRF struggles to accurately capture intricate scene details. This limitation is particularly pronounced in the case where images are hard to infer their pose based solely on coarse information. We refer to these images as "hard-pose images." These inaccurately aligned images subsequently hinder the NeRF training, introducing erroneous supervision.
To mitigate this problem, we proposed a novel architecture with two distinct NeRF heads: the shared head and the candidate head. The intuition is to consider easy/distinct pieces first to complete a large jigsaw puzzle and handle hard pieces later. The shared head is responsible for generating essential parameters, namely density $\sigma$ and color $\hat{\mathbf{c}}$, which are shared across all images. On the other hand, the candidate head generates unique parameters, including density $\sigma^{(c)}_i$ and color $\hat{\mathbf{c}}^{(c)}_i$, tailored to each individual image. In the initial stages of training, the shared head is predominantly learned by easy-pose images. Meanwhile, the candidate head is primarily trained by hard-pose images. This remedy ensures that the candidate head effectively prevents the shared head from being influenced by incorrect guidance. As the shared head becomes accurate enough to facilitate the pose estimation of hard-pose images, these challenging instances are gradually incorporated into the shared head by our loss scheduling that adjusts the weights between two heads, i.e. $w_s$$_e$ in Equation 7 of the main paper.

Meta-review: All the reviewers recommend the acceptance of the work. Reviewers appreciated the well-written paper with good technical contributions. Reviewers raised several clarification questions and also raised concerns related to results on only one dataset. Authors addressed several of these questions in their responses. The reviewers did raise some valuable concerns and suggestions that should be addressed in the final camera-ready version of the paper, which include adding the relevant rebuttal discussions and revisions in the main paper. The authors are encouraged to make the necessary changes to the best of their ability.
Decision: Accept (poster)