TimeNeRF: Building Generalizable Neural Radiance Fields across Time from Few-Shot Input Views

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:44:41

Abstract: We present TimeNeRF, a generalizable neural rendering approach for rendering novel views at arbitrary viewpoints and at arbitrary times, even with few input views. For real-world applications, it is expensive to collect multiple views and inefficient to re-optimize for unseen scenes. Moreover, as the digital realm, particularly the metaverse, strives for increasingly immersive experiences, the ability to model 3D environments that naturally transition between day and night becomes paramount. While current techniques based on Neural Radiance Fields (NeRF) have shown remarkable proficiency in synthesizing novel views, the exploration of NeRF's potential for temporal 3D scene modeling remains limited, with no dedicated datasets available for this purpose. To this end, our approach harnesses the strengths of multi-view stereo, neural radiance fields, and disentanglement strategies across diverse datasets. This equips our model with the capability for generalizability in a few-shot setting, allows us to construct an implicit content radiance field for scene representation, and further enables the building of neural radiance fields at any arbitrary time. Finally, we synthesize novel views of that time via volume rendering. Experiments show that TimeNeRF can render novel views in a few-shot setting without per-scene optimization. Most notably, it excels in creating realistic novel views that transition smoothly across different times, adeptly capturing intricate natural scene changes from dawn to dusk.

Main paper: 
 related works 2.1 neural radiance field: 
 Neural Radiance Field (NeRF) [35] revolutionized novel view synthesis by modeling 3D scenes as an implicit neural representation. It employs a multi-layer perceptron (MLP) to map 3D positions and camera directions to colors and densities. Subsequent works improve NeRF by combining implicit scene representation with traditional grid representation [7, 14, 29, 36, 50, 56] or reducing samples taken along each ray [24, 28, 38, 42] . Nevertheless, these techniques are limited in generating views for different time points and face challenges with varied illumination. NeRF-W [33] extends the NeRF model by introducing a learned appearance code for each image, allowing it to synthesize novel views at different times by inputting different appearance codes. The subsequent works [3, 23, 48, 52] adopt a similar idea. However, these methods require per-scene optimization and a large number of training views. Moreover, they lack the ability to query the view at a specific time.On the other hand, NeRF to dynamic scene [13, 31, 41, 44, 48 ] aims to model a dynamic scene by learning a 4D implicit scene representation. The inputs of the MLP include not only 3D coordinates and direction but also time. While these approaches target short-range temporal changes, like moving people or objects, we emphasize modeling long-range time shifts, enabling realistic view rendering from day to night. Furthermore, these methods also require many image views and re-optimizing the model for each new scene.Recently, three major approaches have been proposed to enable the synthesis of novel views from limited inputs. First, some methods [17, 20, 40, 49, 58, 61] incorporate semantic and geometry regularizations to constrain the output color and density. Second, another line of works attempts to leverage additional depth information such as sparse 3D points [8, 46] or depth prediction from a pre-trained model [12] . Third, some other approaches [2, 19, 27, 30, 39, 54, 57, 60, 63] attempt to condition the model with features extracted from inputs, allowing it to be generalizable to new scenes. For example, MVSNeRF [2] proposes incorporating a 3D cost volume constructed from the extracted feature of input views [11, 62] with the NeRF model for assistance. This allows the model to develop geometry awareness of scenes and increases model generalization ability. Later works [19, 30, 39, 60] also adopt this idea. GeoNeRF [19] and NeuRay [30] further take the occlusion problem into account by predicting the visibility of each source view. ContraNeRF [60] introduces a geometry-aware contrastive loss to improve generalization in the synthetic-to-real setting. However, the methods above do not support rendering a novel view at arbitrary times.Transferring an image to another time zone, for example, from day to night or summer to winter, usually involves style transfer methods. The typical approach is to disentangle content and style information, where style corresponds to the time-specific characteristics. Subsequently, an image is transferred by replacing its original style with the desired time-related style, which is extracted from a reference image [18, 25, 26, 45] . However, this kind of method is limited to domain transfer, lacks the ability to query a specific time, and cannot achieve continuous translation across different time periods. Earlier works [10, 47, 55] on the continuous image translation task assume linear domain manifolds, which may not be suitable for daytime translation. Daytime translation should be cyclic, allowing for translation from day to dusk, dusk to night, night to dawn, and so forth. CoMoGAN [43] proposes the first continuous image translation framework, which enables cyclic or non-linear translation through the design of the functional instance layer and the guidance of a non-neural physical model. Besides, some research efforts are focusing on timelapse generation [1, 4, 5, 37, 59] , aiming to generate a timelapse video based on a single source image. However, none of these works are equipped for time transitions in a 3D space, which enhances the overall immersive experience in digital realms, especially in the realm of the metaverse. 
proposed method: 
 Our goal is to synthesize novel views at any given time by learning implicit scene representations across time from sparse input views that are not captured at the desired moment. We begin by reviewing NeRF and highlighting the distinctions in our approach (section 3.1). After detailing our training process (section 3.2), we introduce our proposed framework. As depicted in Fig. 2 , our framework comprises five main components: 1) extracting geometry features for each source view (section 3.3), 2) constructing a content radiance field (section 3.4) which stores densities and content features, whose environmental change factors have been excluded, 3) estimating time and extracting time-irrelevant factors (section 3.5), 4) transferring content features at 3D locations to RGB colors of the specific time point and complete a time-dependent radiance field (section 3.6), and 5) rendering a novel view at the desired time through volume rendering (section 3.1). Finally, the proposed loss functions are designed to further improve cyclic changes and enhance the content adaptation based on the input data (section 3.7). Due to space constraints, an in-depth description of the network architecture and extra discussion are relegated to the supplementary material. The codebase is ready to open and will be made publicly available in due course.First of all, we briefly review the idea of NeRF [35] . NeRF optimizes an MLP, whose input consists of 3D spatial location x and viewing direction d, and whose output corresponds to colors c and densities 𝜎, to represent a scene implicitly. In other words, it aims to learn a continuous function: (c, 𝜎) = 𝐹 𝜃 (x, d). To render a pixel in an image, NeRF first samples 𝑀 points on the corresponding 3D ray r and obtains the colors and densities of these sample points. Then, NeRF renders the 2D-pixel color using the volume rendering technique; the formula is described as follows.EQUATION)EQUATIONwhere 𝑀 is the number of sample points along a ray r and (c 𝑖 , 𝜎 𝑖 ) represent the color and density of point x 𝑖 , respectively. 𝛿 𝑖 denotes the distance of the adjacent sample points.In our method, we introduce novel modifications to the original NeRF. First, following IBRNet [54] , 𝛿 𝑖 in eq. ( 2) is removed for better generalizability. Second, to enable generalizability, we replace the position x in the model's input with the feature f x , which is derived from the interpolation of features extracted from input views at the projection pixels corresponding to the 3D position x. Third, we aim to render a novel view at the arbitrary time 𝑡. Therefore, our TimeNeRF is designed to learn a continuous function:(c 𝑡 , 𝜎) = 𝐹 𝜙 (f x , d, 𝑡), where c 𝑡 is the color in direction d at position x and time 𝑡 within a scene.We employ a two-stage training process to effectively disentangle content and environmental change factors and subsequently construct implicit scene representations. In Stage 1, we achieve feature disentanglement by leveraging existing style translation models comprising three modules: content extractor, style extractor, and generator. Specifically, we train DRIT++ [26] on the Ithaca365 dataset [9] , which contains images across varied weather and nighttime conditions. To enhance the content extractor in DRIT++, instead of only using the content extractor's output, we extract/select content features from 3 different convolutional layers corresponding to different semantic levels for styled image generation. This strategy refines early-layer content extraction, as these features are utilized to generate images, thereby boosting the model's overall content extraction ability.After Stage 1 training, DRIT++ offers disentangled content features and enables the creation of stylized images as pseudo ground truth for Stage 2 training. In Stage 2, we train our TimeNeRF (detailed in section 3.3 to section 3.6) using input (source) views from the Ithaca365 dataset and using reference images from the Waymo dataset [51] . By merging source images' content features with reference images' style features for training guidance, we can learn the implicit 3D scene representations across time 𝐹 𝜙 (•). The reason behind using the two datasets is elaborated in section 4.1.Given input views {𝐼 𝑖 } 𝑁 𝑖=1 and their corresponding camera parameters {𝐸 𝑖 , 𝐾 𝑖 } 𝑁 𝑖=1 , we extract content features at three levels {𝑐 𝑓 (𝑙 ) 𝑖 } 𝑁 𝑖=1 (i.e., 𝑙 = 0, 1, 2 and 𝑙 = 0 is the lowest level feature) for the 𝑁 input views using the pre-trained content extractor. The content features exclude environmental change factors, such as temporal and weather factors, because the image style has been removed using the pre-trained model. This allows us to handle input views under varying capture conditions. Next, for each input view 𝑣, we select 𝑆 nearby views and construct cost volumes [11, 19, 30, 62] at three levels {𝐶 (𝑙 ) 𝑣 } 2 𝑙=0 by warping the content features of the 𝑆 nearby views to align with view 𝑣 using the corresponding camera parameters. Finally, the cost volumes {𝐶 After obtaining geometry features, we may use them to predict colors and densities (section 3.1). However, this would lose the ability of modeling style change over time. Instead, we propose constructing a content radiance field, an implicit scene representation without environmental change factors. To achieve this, inspired by GeoNeRF [19] , we aggregate the extracted geometry features to predict each sample point's density and content features. For a sample point x ∈ R 3 along a ray r, we interpolate geometry features {𝑔 𝑖 } 𝑁 𝑖=1 and 2D content features {𝑐 𝑓 𝑖 } 𝑁 𝑖=1 , as opposed to the 2D features used in GeoNeRF, to get its corresponding features {𝑔 x 𝑖 } 𝑁 𝑖=1 and {𝑐 𝑓 x 𝑖 } 𝑁 𝑖=1 at point x. These are then aggregated to estimate the density and content feature for x. The procedure is described below.First, we aggregate features via fully-connected layers and multihead attention layers [53] , which facilitate the exchange of information between different views.EQUATIONwhere 𝐻 consists of fully-connected layers and multi-head attention layers. 𝜎 x is the density of x and { wx 𝑖 } 𝑁 𝑖=1 denotes the enhanced features of x for each input view. To obtain the 3D content feature 𝜉 x for a point x ∈ R 3 , we predict the weights of input views {𝑤 x,𝑣 𝑖 } 𝑁and use them to calculate the weighted sum of 2D content features {𝑐 𝑓 x 𝑖 } 𝑁 𝑖=1 . That isEQUATIONEQUATIONwhere Δ𝑑 x 𝑖 represents the direction difference between the query view 𝑣 and input view 𝑖 by computing the cosine similarity, and the concatenation is denoted by ||. We predict the weights via an MLP that considers both the direction difference Δ𝑑 x 𝑖 and wx 𝑖 in eq. ( 4). 𝜉 stands for {𝜉 (𝑙 ) } 2 𝑙=0 . Each individual 3D content feature 𝜉 (𝑙 ) is computed from the 2D content features {𝑐 𝑓 (𝑙 ) 𝑖 } 𝑁 𝑖=1 at level 𝑙 by eq. ( 5).To infuse environmental change information into the content radiance field, one intuitive approach is to use the style feature extracted from the pre-trained style extractor. The style feature inherently contains both time-relevant and time-irrelevant environmental factors since the translation model is trained to transfer images to diverse weather conditions and varying times. However, to purely manipulate the timing of radiance fields, it's imperative to separate time-related and time-irrelevant information further from the style feature. To this end, we employ two MLPs: 𝑔 𝑡 (•) predict time information 𝑡 and 𝑔 𝑎 (•) extracts time-irrelevant features 𝑎 from an image. During training, we extract 𝑡 from a reference image and 𝑎 from input images.To encapsulate the cyclical progression of a day, we design 𝑔 𝑡 (•) to map the style feature of a reference image onto the interval [0, 2𝜋), symbolizing the entire 24-hour cycle. The mapping is trained in an unsupervised manner, utilizing our pseudo stylized image loss 𝐿 𝑠𝑡 𝑦𝑙𝑒𝑚𝑠𝑒 (eq. ( 8)) to guide the model in extracting time-related information from reference images and mapping it to the time range [0, 2𝜋). By this design, without time labels for training, we can still simulate the variations occurring at different times within a day. After training, the transition of a day is encoded within [0, 2𝜋), enabling us to recreate scenes at a desired time 𝑡 by tuning 𝑡 during testing.The objective of time-dependent radiance field constructor "𝑇 " is to transform the 3D content feature 𝜉 x into a time-variant color c x 𝑡 for a point x. This involves fusing 𝜉 x with time-irrelevant environmental features 𝑎 of input views and time 𝑡. Essentially, we learn a mapping:c x 𝑡 = 𝑇 (𝜉 x , 𝑎, 𝑡).The design of 𝑇 is based on two main ideas: Firstly, we embed 𝑡 into (𝑐𝑜𝑠 (𝑡), 𝑠𝑖𝑛(𝑡)), ensuring uniqueness for each value in [0, 2𝜋). This, in conjunction with 𝐿 Δ𝑡 (eq. ( 10)), facilitates cyclic changes. Secondly, to ensure the model focuses on learning variations based on time, we introduce a two-branch network. The first branch, fusing 𝜉 with 𝑡, acts as the template for the change over time. The second branch, merging 𝜉 with both 𝑡 and 𝑎, is to further tune the color according to time-irrelevant features 𝑎.The network (detailed in Supplementary) is designed in a coarseto-fine manner. We exploit content features at three different levels {𝜉 (𝑙 ) } 2 𝑙=0 . The predicted color c 𝑡 is refined progressively by integrating high-level features down to low-level information. Specifically, we obtain the feature of branch 1, 𝑓 1 , by combining content features The image in the orange box is synthesized for the time of input views 𝑡 0 (eq. ( 11)), whose image style is consistent with input views. from three levels with time 𝑡. Meanwhile, we derive the feature of branch 2, 𝑓 2 , by integrating content features from three levels with both time 𝑡 and time-irrelevant features 𝑎. Finally, we combine 𝑓 1 and 𝑓 2 to produce the color c 𝑡 .MSE loss. The mean square error loss L 𝑚𝑠𝑒 in eq. ( 7) ensures accurate 3D scene construction in our model. Different from the colors c 𝑡 described in section 3.6, the predicted colors c in eq. ( 6) for calculating Ĉ (r) in this loss are derived from the weighted sum of the original input image colors {𝐼 𝑖 } 𝑁 𝑖=1 , aiding in the learning of densities 𝜎 (eq. ( 3)) and weights {𝑤 𝑖 } 𝑁 𝑖=1 (eq. ( 4)).EQUATIONEQUATION)where 𝑅 is the set of rays in each training batch. 𝐼 x 𝑖 is the projection color of point x to image 𝐼 𝑖 . Ĉ (r) is the predicted color computed by eq. ( 1) using color c in eq. ( 6); 𝐶 (r) is the ground truth color of the target view.Pseudo stylized loss. The pseudo stylized loss L 𝑠𝑡 𝑦𝑙𝑒𝑚𝑠𝑒 aims to guide the model to learn temporal transition by minimizing the difference between our predicted time-dependent pixel color and the pseudo ground truth colors.EQUATIONwhere Ĉ𝑡 (r) is the predicted color computed by eq. ( 1) using c 𝑡 illustrated in section 3.6, and 𝐶 𝑝𝑠𝑒𝑢𝑑𝑜 (r) is the pixel color of the pseudo ground truth, which is the style-transferred output via the pre-trained translation model (DRIT++) given the target view and reference image.Loss term of Δ𝑡. Besides the design where we transform time 𝑡 into (cos(𝑡), sin(𝑡)) to ensure cyclic changes, we introduce the Δ𝑡 loss. This loss function in eq. ( 10) leverages a small MLP, denoted as 𝐷, to estimate the time difference based on color variations. The predicted time difference is then compared to the pseudo time difference Δ𝑡:EQUATION)EQUATION)where 𝑡 and 𝑡 ′ are randomly sampled from [0, 2𝜋), and c 𝑡 and c 𝑡 ′ are the predicted colors at times 𝑡 and 𝑡 ′ , respectively. By minimizing this loss, our color predictions are guaranteed to align with timebased changes.Reconstruction loss. This loss is to ensure that rendered views match the original appearance when the input time aligns with the capture time. This further allows the rendered views of different times to adapt to the inputs. Particularly,EQUATIONEQUATIONwhere 𝑡 0 is derived from predicting the time of an input view 𝐼 (eq. ( 11)) and 𝑠 𝐼 is the style feature of 𝐼 .Total loss. The total loss is as follows:EQUATIONwhere we set 𝜆 1 = 0.5, 𝜆 2 = 0.01, and 𝜆 3 = 0.5. 
experiments 4.1 datasets: 
 Collecting a dataset with different views and diverse capture times is challenging and unavailable. To address this, we train our model using both the Ithaca365 dataset [9] and the Waymo dataset [51] . Ithaca365 provides different capture views but limited time variations, while Waymo offers images from different times (day, dusk, dawn, and night) on sunny days. From Ithaca365, which includes conditions like sunny, cloudy, rainy, snowy, and nighttime, we randomly selected 2180 scenes with 3 distinct views. This allows TimeNeRF to be versatile for various conditions during testing. On the other hand, reference images for learning time information are taken from Waymo. In all experiments, TimeNeRF is trained with the aforementioned dataset configurations. For testing, we evaluate the model's generalizability using scenes outside the training set, as well as the T&T dataset [22] and the LLFF dataset [34] .During training, we use 𝑁 = 2 input views due to the Ithaca365 dataset's limitations of having only 4 distinct viewpoints per scene, yet 2 of them are nearly overlapping. However, our model is flexible enough to handle more views during testing. We train TimeNeRF over 15 epochs, sampling 1024 rays as the training batch. The number of sample points 𝑀 is set to 128. The Adam optimizer [21] is applied with a 5 × 10 -4 learning rate and a cosine scheduler without restarting the optimizer [32] . In the testing phase, we neither re-train nor fine-tune our model for new scenes.To show TimeNeRF's capability of rendering novel views over varying times, we use the same input views to render 16 novel views, querying the same target viewpoint but specifying times asEQUATION𝑖=0. As shown in Fig. 3 , TimeNeRF correctly produces distinct appearances for each specific time, capturing transitions that resemble the natural shifts seen throughout the day. Comparison. To the best of our knowledge, no existing method is designed for rendering novel views across time. For comparison, we combine view synthesis algorithms with image translation models, which transform images to reflect different times of the day. We utilize our model to produce novel views. Next, DRIT++ [26] , HiDT [1] , and CoMoGAN [43] are used to transfer images across times. Note that DRIT++ and HiDT require extra reference images for the desired time-relevance features. Thus, we use frames from 24-hour time-lapse videos as reference images to generate images spanning an entire day. Conversely, CoMoGAN, similar to our approach, can directly specify times. The results in Fig. 4 synthesize the 𝑝𝑙𝑎𝑦𝑔𝑟𝑜𝑢𝑛𝑑 scene in the T&T dataset using 3 input views. While these methods can generate variations based on time, they sometimes yield undesirable effects, such as incorrect black patches during nighttime and color bias on the ground. Furthermore, all of these methods have the cross-view appearance inconsistency issue (geometric inconsistency), detailed in section 4. 4 .In comparison to previously discussed methods, our approach achieves more accurate cyclic appearance changes through consistent time translation. To validate this, we conducted a comprehensive user study. In this study, we presented participants with transformation results at eight distinct times of day: pre-dawn, dawn, mid-morning, afternoon, dusk, evening, and late night. These results, generated by our method and other referenced methods, were shown alongside a reference image corresponding to each specific time. Participants were tasked with identifying the method that ensured consistent translation between frames and most accurately matched the time depicted in each reference image. The findings of this study, depicted in Fig. 7 , demonstrate that our method facilitates smoother transitions across different times of the day. Please refer to the supplementary materials for more analysis. Generalizability. Additionally, we evaluate our model on the T&T dataset [22] in Fig. 4 , demonstrating its adaptability to different datasets. More results under varying conditions are shown in the supplementary material.Following [6, 15, 16] , we employ the warped LPIPS metrics [65] and RMSE to measure consistency across different views. The score is computed by 𝐸 (𝑂 𝑣 , 𝑂 𝑣 ′ ) = 𝑓 (𝑂 𝑣 ,𝑊 (𝑂 𝑣 ′ ), 𝑀 𝑣,𝑣 ′ ), where 𝑂 𝑣 and 𝑂 𝑣 ′ represents the generated image Î𝑣 , Î𝑣 ′ , and their camera parameters. We warp the result from view 𝑣 ′ to view 𝑣 using the depth estimation in our model, which is estimated by replacing c 𝑖 in eq. ( 1) to the HiDT* [1] CoMoGAN* [43] Ours 12)) (b) illustrates the outcome without 𝐿 Δ𝑡 (eq. ( 10)). (c) depicts the result when the two-branch network is not utilized (section 3.6). Finally, (d) is the result from our complete framework. 6 ) demonstrate that our method achieves better cross-view consistency. This improvement stems from our innovative approach of directly modeling time-relevant appearance changes within a 3D space, in contrast to the image translation approaches that operate in 2D space.To demonstrate TimeNeRF's capability in the traditional novel view synthesis task, we evaluate it on the Ithaca365 and LLFF datasets [34] , comparing its performance with other few-shot generalizable 3D modeling approaches. For a fair comparison, we train all the models on the Ithaca365 dataset. HiDT* [1] CoMoGAN* [43] Ours Figure 6 : View inconsistency issue. We render 2 target views corresponding to the same time using the same set of 3 input views. The red boxes highlight the inconsistent regions between views. Among these results, our method produces more consistent views. outperform MVSNeRF and matches the capabilities of GeoNeRF in in view synthesis. The designed loss functions. We study the effectiveness of our designed loss functions. The reconstruction loss is designed to ensure that the model renders views with accurate appearances when the input time coincides with the input views' capture time. Moreover, it allows the rendered views to adapt based on these inputs for different times. In Fig. 5 (a), the model without 𝐿 𝑡 0 generates a road that appears blue at 𝑡 = 𝑡 0 , although it should be gray. The delta t loss aims to ensure that the model produces unique outputs for each 𝑡 ∈ [0, 2𝜋). Fig. 5 (b) shows the results obtained when training the model without 𝐿 Δ𝑡 . We observe that the model renders similar outputs at 𝑡 = 𝜋 2 and 𝑡 = 3𝜋 2 . Two-branch network. In Fig. 5 (c), we note that omitting the two-branch network from our time-dependent radiance field constructor (section 3.6) leads to mapping failures within [0, 2𝜋). The model generates nearly black-and-white scenes at 𝑡 = 0 and 𝑡 = 𝜋 2 . This limitation is due to the lack of the first branch, which serves as a template to constrain the output. By incorporating our proposed two-branch network and the designed loss functions (Fig. 5(d )), we achieve consistent and accurate results across all time points. 
conclusion and discussion: 
 We propose TimeNeRF, a novel framework for view synthesis that renders views at any time from limited input views without perscene optimization. We have effectively leveraged existing datasets to address the lack of data availability for the aforementioned task. Our model transitions smoothly across time by creating a content radiance field and transforming it into a time-dependent radiance field. In addition, our designed loss functions ensure cyclic changes and adaptive results based on inputs. Evaluations show TimeNeRF's capability to produce photorealistic views across time.As the initial solution in few-shot, generalizable novel view synthesis across arbitrary viewpoints and times, TimeNeRF has showcased its efficacy on various datasets. Nevertheless, significant opportunities for advancement remain. For a comprehensive 3D scene model, the dynamics of lighting angles from diverse sources-such as the sun and streetlights-and the associated constraints of object shadows relative to these light sources need to be accounted for. Additionally, achieving appearance and view consistency in both static and dynamic objects throughout varying times of the day poses a substantial challenge that could dramatically enhance user experience. Moreover, incorporating diffusion models as a tool for data augmentation presents a promising direction to address the scarcity of available data, which could catalyze further enhancements in model performance. These considerations may underline future research on 3D modeling technologies.  
: 
 their reliance on per-scene optimization and the need for hundreds of different viewpoint images. In practical scenarios, input views are often limited, and re-optimizing the model for new scenes is inefficient. Moreover, to achieve complete immersion in virtual reality or the metaverse, creating environments that can transition seamlessly from day to night is essential. Nevertheless, the exploration of NeRF's potential for temporal 3D scene modeling remains limited, with few dedicated datasets available for this purpose.In this paper, we present TimeNeRF, a framework designed to achieve novel view synthesis in three aspects: 1) a few-shot setting, 2) generalizing to new scenes captured under varying conditions, and 3) handling temporal transitions in 3D scenes. Several studies are devoted to realizing novel view synthesis from sparse viewpoints [20, 40, 49, 58] and also ensuring generalizability to previously unseen scenes [2, 19, 27, 30, 54, 63] . However, when it comes to rendering novel views at different times, the viable option for these methods is capturing an additional set of views at the desired time. On the other hand, NeRF-W [33] and Ha-NeRF [3] explore constructing neural radiance fields from images taken at various times and under different illuminations. Though capable of generating novel views at different times, they still depend on appearance codes from reference images and require many viewpoints and per-scene optimization. Recently, CoMoGAN [43] introduces a continuous image translation model capable of altering an image's appearance to correspond to a different time. Yet, when integrated directly with a novel view synthesis model, it succumbs to the view inconsistency problem, a limitation highlighted in earlier works [6, 15, 16] .Consequently, a unified 3D representation model that generalizes across different 3D scenes over time, especially in a few-shot setting, remains a challenging open question. To address this issue, we introduce TimeNeRF. Distinct from prior approaches, TimeNeRF synthesizes novel views from limited viewpoints without necessitating model retraining for previously unseen scenes. Moreover, by constructing novel time-dependent neural radiance fields, our method can render novel views at specific times without relying on reference images.The key idea of TimeNeRF, as shown in Fig. 1 , is to first construct a content radiance field, then transform it into the neural radiance field of a specific moment by infusing the environmental change information relevant to the desired time into the content radiance field. Specifically, our approach involves a two-stage training process. The first stage focuses on disentangling content and environmental change factors, achieved through an image translation model. The second stage begins by extracting geometry information from a few input views via the appearance-agnostic geometry extractor. We build our model upon previous generalizable NeRF-based methods but introduce a significant adjustment: the cost volume is constructed from content features instead of being extracted from standard convolutional network features, allowing our model to handle various capture conditions. Next, the implicit scene network constructs a content radiance field based on the extracted geometry and features. We then predict the time and extract time-irrelevant features from the style feature obtained from a pre-trained style extractor. Finally, we create time-dependent radiance fields by incorporating the time and the extracted features into the content radiance field for novel view synthesis over time.To train the model, it is ideal to have a dataset with various capture views and different capture times. However, collecting dense inputs of a scene is expensive, especially considering our specific problem setup, which requires generating novel views at particular times. This means that, in addition to the substantial effort required to gather multiple views of the scene, we also need to collect data at different times throughout the day, which in practice becomes very costly. To overcome this, we train our model on both the Ithaca365 dataset [9] , which offers few different capture views but has limited time information, and the Waymo dataset [51] , covering a range of capturing times (i.e., day, dusk, dawn, night). Since these datasets do not contain time labels, we train the model without relying on exact time data. Specifically, we leverage our two-stage training approach with the design of our network to map reference images into the time domain. The design also benefits the testing phase, allowing us to render novel views at any time by specifying the time code directly. Additionally, we introduce several loss functions to ensure that our model exhibits cyclic changes and that the transition results are adapted based on the inputs. We will explain the details in the "Proposed Method" section. In general, our main contributions are:• An extended novel setting for view synthesis over time, which is more practical for real-world applications. • A novel time-dependent and NeRF-based approach that renders novel views at arbitrary times from few inputs and achieves generalizability to new scenes captured under any conditions. • Extensive experiments showing TimeNeRF's capability to transition smoothly across different times of the day. 
 
 Appendix:  


Decision: This paper targets at the problem of learning dynamic NeRF with few-shot views, which is quite interesting. All the reviewers agree to accept the manuscript. I recommend a decision of acceptance. The authors should include the comparisons and some details in the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a method to solve time-related appearance changes using few-views for novel-view synthesis. More specifically, they solve a challenging problem: "How to render novel-views at different time"?  Previous approaches like NeRF-W and Ha-NeRF represent the underlying geometry under different illumination conditions and appearance changes. However, they depend on appearance codes and per-scene optimization. The proposed method can render novel views simultaneously without relying on reference images or appearance codes.
strength_weakness: strength_weakness: - The proposed method requires only a few views to learn geometry and appearance changes. 
- Two-stage training process to disentangle content and environmental factors.  
- The proposed method outperforms baselines, which combine view-synthesis and image translation methods.
limitations: limitations: - **Blurry and inconsistent output in the supplementary videos**: In the videos shared in the supplementary, there are clear artifacts in the rendered novel views. For example, in "T&T_playground.mp4", the structure in the playground is blurry and has visual artifacts. 

- **Missing comparison with Instruct NeRF2NeRF(IN2N)**: The proposed work proposes a method to generate novel-views with time-varying appearance changes. It's also important to compare with works like IN2N, which are capable of editing the appearance of a scene. 

- **How about effects like rain/fog/snow?:** Although this is not a limitation, can the proposed work also handle these changes? This will further show that the proposed method is generalizable to different scenarios.  

- **Missing details like training time**: Details like training time, no. of parameters and inference time are not discussed in the current manuscript. Authors should discuss this. 

- **Missing details for 3D-Unet mentioned in L393-397**: Proposed method does not discuss the 3D-Unet details which is used to obtain geometric features.
paper_summary: paper_summary: This work introduces TimeNeRF, which can render novel views at different times of the day in a feed-forward manner. Specifically, TimeNeRF starts by disentangling the content features from the input images, which are then used to learn a generalizable NeRF model. To inject the time-dependent information into NeRF, TimeNeRF leverages a factor extraction module to pass the time and related time-irrelevant information before predicting the color for each point. Experiments on several existing benchmarks demonstrate the soundness of the introduced TimeNeRF.
strength_weakness: strength_weakness: * The setting of rendering generalizable NeRF at different times is interesting.
* It is technically sound to map the style features onto a 24-hour cycle, removing the requirement of exact time labels and making the framework more applicable to different datasets.
* The paper is well-written, and it is easy to follow.
limitations: limitations: * The visual results are not surprising enough. I would imagine that for those street views (‘itheca_street.mp4’ in the supp.), the streetlights should be on at night, leaving lights and shadows on the street. Or the shadows cast by the sun should be gone at night (left part of the ‘T&T_train.mp4’). However, there is no such effect in TimeNeRF. The current visual results are very similar to applying a transparent black mask on top of rendered images, making it hard to understand whether TimeNeRF has effectively learned the time-relevant information. Is there any scene that presents contents where the streetlights will be turned on at night? 

* It is unclear what is in the $\textrm{content}^x_\textrm{3D}$. Typically, in a 2D image, the content might refer to geometry/structure, and style refers to color information. However, in the NeRF representation, geometry is already captured by the density, so I am curious about what is being captured in the $\textrm{content}^x_\textrm{3D}$ intuitively. Besides, I can see that ray direction is still fed into the Implicit Scene Network, does it mean that the $\textrm{content}^x_\textrm{3D}$ is also view-dependent, similar to the color of the original NeRF? The authors are recommended to volume render the $\textrm{content}^x_\textrm{3D}$ at different viewpoints and further analyse what kinds of physical attributes $\textrm{content}^x_\textrm{3D}$ aims to learn.
paper_summary: paper_summary: TimeNeRF can generate novel views of scenes at any time of day with minimal input views, without requiring per-scene optimization. It combines multi-view stereo, nerf, and disentanglement strategies to create an implicit radiance field that generalizes in a few-shot setting.  TimeNeRF excels in rendering realistic transitions between day and night, capturing detailed scene changes smoothly, compated to GAN based methods.
strength_weakness: strength_weakness: 1. The topic addresses a good setting by focusing on view synthesis that not only generates new perspectives but also smoothly transitions across different times of the day. This adds a temporal dimension to 3D scene modeling, enhancing its applicability in dynamic real-world scenarios and virtual environments like the metaverse.

2. The method achieves generalizability by leveraging multi-view stereo, neural radiance fields, and disentanglement strategies. This allows it to adapt to new scenes captured under various conditions with minimal input views, making it robust and versatile for diverse applications.

3. The model's optimization process does not depend on precise time-stamped data, enabling it to render realistic scene changes without the need for exact temporal alignment. This flexibility simplifies data collection and broadens the model's usability across different temporal contexts and environments.
limitations: limitations: 1. It is a weakness that the method did not include comparisons with other NeRF-based approaches. Instead, it was compared to 2D generation models, which do not incorporate 3D modeling. This limits the evaluation's relevance, as it does not directly benchmark against state-of-the-art techniques specifically designed for 3D view synthesis.

2. Some straightforward baselines could have been considered, such as integrating instruct-nerf2nerf and few-shot generalizable NeRF methods with style prompts (e.g., "make it dark"). This would provide a more comprehensive assessment by including methods that address similar challenges in temporal scene representation and view synthesis.

3. The claim of time modeling is somewhat misleading. The method seems to focus more on style-conditioned modeling, akin to what is seen in WildNeRF, rather than genuine temporal dynamics. This distinction is important, as true time modeling would require capturing and predicting changes over continuous time rather than applying style variations to static scenes.
paper_summary: paper_summary: This paper proposes TimeNeRF for rendering novel views at any viewing angle and at any time.
strength_weakness: strength_weakness: + A new setting.
+ A novel NeRF-based approach that renders novel views at arbitrary times from few inputs and achieves generalizability to new scenes captured under any conditions.
limitations: limitations: + The method section write is very poor, many places lack some explanation, I give some examples here:
   + The DRIT (see line 332 on page 3 of the paper) is cited in many places, but no explanation is provided in the paper.
   + There are no details on how to build the cost.
+ Lack of quantitative ablation results, it is difficult to determine whether the proposed module is effective, especially in Figure 5 (a) and Figure 5 (d).
+ Does this task have corresponding ground truth (as shown in Figure 4)? If it is a style transfer task, changing the rendering color of the scene at a certain moment, I strongly recommend conducting user studies. This is because such tasks are highly subjective, and it is difficult to determine which result is better based solely on visual results.
+ Although the proposed method can effectively transform colors based on time, it does not provide specific time points. What criteria are used to determine these time points? How does this time-based transformation differ from directly using text-based transformation (such as Instruct-NeRF2NeRF)?