Exposure Completing for Temporally Consistent Neural High Dynamic Range Video Rendering

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 10:33:53

Abstract: High dynamic range (HDR) video rendering from low dynamic range (LDR) videos where frames are of alternate exposure encounters significant challenges, due to the exposure change and absence at each time stamp. The exposure change and absence make existing methods generate flickering HDR results. In this paper, we propose a novel paradigm to render HDR frames via completing the absent exposure information, hence the exposure information is complete and consistent. Our approach involves interpolating neighbor LDR frames in the time dimension to reconstruct LDR frames for the absent exposures. Combining the interpolated and given LDR frames, the complete set of exposure information is available at each time stamp. This benefits the fusing process for HDR results, reducing noise and ghosting artifacts therefore improving temporal consistency. Extensive experimental evaluations on standard benchmarks demonstrate that our method achieves state-of-the-art performance, highlighting the importance of absent exposure completing in HDR video rendering. The code will be made publicly available upon the acceptance of this paper.

Main paper: 
 introduction: 
 Compared with the high dynamic range (HDR) of natural scenes, the standard digital camera can only capture low dynamic range (LDR) information. To achieve the goal of "what you see is what you get" in photography, delicate optical systems [22, 33, 38] are designed to simultaneously capture LDR images with different exposures which cover the whole dynamic range to reconstruct HDR results. However, these systems are expensive and un-portable, making HDR contents not easily accessible for general users [18] . To address this limitation, computational methods [17, 18, 29] are designed to render HDR videos from LDR videos whose frames are exposed alternately with different exposures (as shown in 1-st row in Fig. 1 ).In this setting, the incomplete exposure information varies along the time dimension. Existing neural HDR rendering methods [3, 6, 15, 40] align neighbor LDR frames according to the LDR frame of current time stamp (called reference frame in this paper) and fuse the aligned neighbor LDR frames into the reference frame to supplement the missing exposure information. The exposure of reference frame changes at every time stamp, which means that the reference frames of different exposures may have different defects: saturation for highly exposed LDR frames, and noise for lowly exposed ones. Due to that the methods mentioned above heavily depend on reference frames, HDR results from these methods may inherit defects of the reference frames, suffering from the artifacts, i.e., the ghosting (2-nd row in Fig. 1(a) ) and noise (2-nd row in Fig. 1(b )), thereby causing temporal inconsistency.To tackle this problem, we revisit early traditional methods [17, 18] for HDR rendering and are motivated to complete the missing exposure information for each time stamp [17, 18] . This idea follows the philosophy of optical HDR systems [38] that are designed to obtain different exposed LDR frames at exact the same time. In this way, defects of different exposed frames can be mutually compensated. In this paper, we propose the Neural Exposure Completing HDR (NECHDR) framework to reconstruct the LDR frames with missing exposure information. In this way, full exposure information at each time stamp can be covered by the reference and completed LDR frames, therefore the exposure information is complete and consistent along the time dimension, which benefits the temporal consistency of the rendered HDR videos.In the proposed NECHDR framework, pyramid features of input LDR frames are extracted by the feature encoder, then fed into the exposure completing decoder and the HDR rendering decoder. The exposure completing decoder interpolates the features of neighbor LDR frames at every level of the feature pyramid. The interpolated LDR features are combined with the features from input LDR images as the inputs to HDR rendering decoder. The HDR rendering decoder estimates coarse HDR results and the optical flows at every level. The flows can facilitate feature interpolating in exposure completing decoder. At the end of NECHDR, a simple blending network is used to integrate interpolated LDR frames, input frames, and coarse HDR frames, which can achieve high-quality and temporally consistent HDR reconstruction results.Extensive experimental results on multiple public benchmarks demonstrate the superiority of the proposed NECHDR: by completing the missing exposure information (3-rd row in Fig. 1 ), our method mitigates ghosting resulting from large motions for the time stamps with highly exposed reference frames (4-th row in Fig. 1(a) ), and reduces the noise level for the lowly exposed reference frames (4-th row in Fig. 1 (b)), hence achieves better temporal consistency. Our work sheds light again on the exposure completing for HDR video rendering. The contributions can be summarized as follows:• Our work firstly implements the idea of exposure completing for neural HDR rendering.• Our work proposes a novel HDR video rendering framework, a.k.a., the NECHDR, which completes the missing exposure information by interpolating LDR frames.• Our NECHDR achieves new state-of-the-art performance on current benchmarks. 
related work 2.1 hdr image rendering: 
 HDR imaging technology aims to extend the dynamic range of images by merging LDR images with different exposures. Both the HDR image and video tasks encounter the misalignment between LDR images. Pioneer studies for HDR image rendering employ image alignment techniques to address this issue. Early works [9, 12, 19, 24, 35, 47] often align LDR images globally, then detect and discard unaligned regions. Directly discarding these regions leads to significant information loss, posing challenges for HDR image rendering. To address these issues, optical-flow-based [1, 14, 49] and patch-based methods [13, 28, 37] are proposed. However, large motion still makes these methods fail to produce artifactfree HDR results. With the rapid development of deep learning, many studies for HDR image rendering switch their focus to neural networks [2, 5, 16, 25, 27, 34, 39, [42] [43] [44] , achieving promising results. Kalantari et al. [16] propose using neural networks to align input LDR images with predicted optical flow. Wu et al. [39] directly map LDR images to HDR images, thereby avoiding alignment errors. Yan et al. [43, 45] use spatial attention to implicitly align LDR images, achieving further improvement. Liu [27] et al.introduce a ghost-free imaging model based on swin-transformer [26] . Yan et al. [42] utilize patch-based and pixel-based fusion to search for information to complement the reference frame from the other frames. However, these HDR image rendering methods assume a fixed exposure for reference frames, typically medium exposure, therefore is not suitable for HDR video rendering where reference frames have alternating exposures.HDR video can be obtained by delicate optical systems, such as scanline exposure/ISO [4, 11] , internal/external beam splitter [23, 31, 38] , modulo camera [48] and neuromorphic camera [10] . However, these expensive systems are hardly accessible to general users. Therefore, rendering HDR videos from LDR videos is investigated. Kang et al. [18] utilize both global and local alignment for reference frames with neighbor frames. Kalantari et al. [17] propose a patch-based optimization algorithm to reconstruct HDR video by synthesizing missing exposures for each frame. These traditional methods offer Our network mainly consists of three processes: feature encoding for LDRs, exposure completing for LDR frames with missing exposures, and HDR rendering. We extract pyramid features from the input LDR frames using a parameter shared feature encoder. Then, the optical flows predicted by the lower-level HDR rendering decoder is used to warp neighbor frames to the reference frame. Subsequently, the exposure completing decoder performs exposure completing based on the warped neighbor frame features. The input features and the completed feature are fed into the HDR rendering decoder to render the coarse HDR frame. Finally, the original frames, the completed frame, and the coarse HDR frame are fused together through a simple blending network to produce a high-quality HDR result.a fundamental idea for completing missing exposures, but they require iterative optimization and are prone to producing artifacts. Recently, Kalantari et al. [15] introduce an end-to-end neural network consisting of an optical flow prediction model and a fusion network. Chen et al. [3] propose a coarse-to-fine alignment framework, which utilizes optical flow for coarse alignment and then employs deformable convolution [7] for fine alignment. Chung et al. [6] employ spatial attention instead of optical flow to achieve alignment from adjacent frames to the reference frame. Xu et al. [40] propose the HDR-domain loss, which uses optical flow supervised by HDR frames for LDR alignment, yielding favorable results. However, when complex situations such as large motion, saturation, and others occur simultaneously, these methods struggle to achieve precise alignment, resulting in artifacts in the final outputs. Besides, as the exposures of reference frames alternate, the reconstructed HDR frames tend to inherit the defects of reference frames, leading to flicker in the output videos. The fundamental issue lies in the lack of exposure information at every time stamp. Different from previous methods, we propose a renaissance work which focuses on completing missing exposures for the reference frame in neural HDR video rendering. Through compensating for the defects of the alternatively exposed reference frames by utilizing the completed results, our approach avoids artifacts of ghost and noise and obtains a temporally consistent HDR video result. Following previous work [40] , we try to render a high-quality HDR video with LDR video under two settings of different alternate exposures: two alternate exposures (𝑧 = 2) with three frames as input, and three alternate exposures (𝑧 = 3) with five frames. We use the two-exposure setting as an example to demonstrate how to achieve HDR rendering. In Sec. 3.5, we introduce how to extend our approach to the three-exposure setting. The frame 𝑙 𝑡 corresponding to the current time stamp 𝑡, is treated as the reference frame. 𝑙 𝑡 is combined with its neighbors as input {𝑙 𝑡 -1 , 𝑙 𝑡 , 𝑙 𝑡 +1 }. To render HDR frames, we explicitly complete frames 𝑙 𝑡 with absent exposure 𝜖 𝑖 , where 𝑖 ≠ (𝑡 ⊘𝑧)+1. Therefore, with complete exposure information, we can render a high-quality HDR video.Our method consists of two processes: exposure completing for the frame with the absent exposure at the time stamp 𝑡 and coarse-to-fine HDR rendering. These two processes are coupled together through an encoder-decoder architecture, as illustrated in the Fig. 2 . The feature encoder E takes LDR frames {𝑙 𝑡 -1 , 𝑙 𝑡 , 𝑙 𝑡 +1 } as input and generates pyramid features for each frame. As the spatial dimensions decrease and the feature channels increase, the encoder outputs feature sets {Φ 𝑡 -1 , Φ 𝑡 , Φ 𝑡 +1 } for input frames, each of which is a set of 𝑘-level pyramid features Φ = {𝜙 𝑘 |𝑘 = 1, 2, 3, 4}. Considering the distinct characteristics of the two processes, we devise two dedicated decoders. Initially, an exposure completing decoder D 𝐼 is employed to interpolate frame 𝑙 𝑡 and its corresponding featuresΦ 𝑘 𝑡 = { 𝜙 𝑘 𝑡 |𝑘 = 1, 2, 3}. Subsequently, a coarse-to-fine HDR rendering decoder D 𝑅 takes {Φ 𝑡 -1 , Φ 𝑡 , Φ 𝑡 +1 } and Φ 𝑡 as input, and outputs optical flows 𝐹 𝑡 -1→𝑡 = {𝑓 𝑘 𝑡 -1→𝑡 |𝑘 = 0, 1, 2, 3}, 𝐹 𝑡 +1→𝑡 = {𝑓 𝑘 𝑡 +1→𝑡 |𝑘 = 0, 1, 2, 3}, coarse HDR features Ψ 𝑡 = { 𝜓 𝑘 𝑡 |𝑘 = 1,Φ 𝑡 -1 = { 𝜙 𝑘 𝑡 -1 |𝑘 = 1, 2, 3}, Φ 𝑡 +1 = { 𝜙 𝑘 𝑡 +1 |𝑘 = 1, 2, 3} or frames { 𝑙 𝑡 -1 , 𝑙 𝑡 +1 }.The warped neighbor features { Φ 𝑡 -1 , Φ 𝑡 +1 } and frames { 𝑙 𝑡 -1 , 𝑙 𝑡 +1 } are fed into D 𝐼 . Subsequently, the exposure completing features Φ 𝑡 from D 𝐼 are fed into the HDR rendering decoder D 𝑅 , assisting it in better restoring lost details due to saturation and noise when decoding coarse HDR features Ψ 𝑡 and coarse HDR frame ℎ 𝑐 𝑡 . This design coupling two decoders enables mutual enhancement of the two processes, thus obtaining more accurate exposure completing frame 𝑙 𝑡 and better coarse HDR result ℎ 𝑐 𝑡 . To obtain the final HDR rendering result, we map {𝑙 𝑡 -1 , 𝑙 𝑡 , 𝑙 𝑡 +1 } and 𝑙 𝑡 to the linear HDR domain. The function defining the mapping of LDR frames to the linear HDR domain is as follows:EQUATIONwhere 𝛾 is a hyperparameter and 𝑒 𝑡 is the exposure time of 𝑙 𝑡 . Finally, the ultimate high-quality HDR rendering result ℎ 𝑡 is obtained by fusing the input and completed frames with coarse HDR frame in linear HDR domain using a simple blending network. The details of each process will be presented in the following sections. 
exposure completing: 
 Under the setting that input LDR frames are with alternating exposures, completing the absent exposure information is essential for rendering HDR frames. However, existing neural HDR rendering methods overlook this essential problem, making it difficult for these methods to accurately recover detailed information when motion occurs with saturation or noise simultaneously, potentially leading to artifacts. Furthermore, due to these methods heavily depend on reference frames, the alternate appearance of noise and saturation caused by the alternate exposures in reference frames affects the temporal consistency of the video. Therefore, to better address the aforementioned issues, we explicitly tackle the fundamental problem of HDR rendering by completing frames with absent exposure corresponding to reference frames. Given the LDR frame input{𝑙 𝑡 -1 , 𝑙 𝑡 , 𝑙 𝑡 +1 } with alternate exposures {𝑒 𝑡 -1 , 𝑒 𝑡 , 𝑒 𝑡 +1 } , where the reference frame and neighbor frames have two different exposures ( 𝑒 𝑡 = 𝜖 1 and 𝑒 𝑡 -1 = 𝑒 𝑡 +1 = 𝜖 2 ), our task is to complete the frame 𝑙 𝑡 with absent exposure 𝜖 2 corresponding to reference frame through interpolation.Our exposure completing process takes the neighbor features {Φ 𝑡 -1 , Φ 𝑡 +1 } and the optical flows {𝐹 𝑡 -1→𝑡 , 𝐹 𝑡 +1→𝑡 } as input. During this process for the features with absent exposure information, the 𝑘-th level neighbor features {𝜙 𝑘 𝑡 -1 , 𝜙 𝑘 𝑡 +1 } and optical flows {𝑓 𝑘 𝑡 -1→𝑡 , 𝑓 𝑘 𝑡 +1→𝑡 } are processed to obtain completed feature 𝜙 𝑘 𝑡 :EQUATIONEQUATIONEQUATIONwhere W (•, •) denotes using optical flow to warp neighbor features to the reference feature, U 2 represents the bilinear upsampling operation with scale factor 2, D 𝑘 𝐼 (𝑘 = 1, 2) represents the middle levels of the exposure completing decoder D 𝐼 , and [•] indicates concatenation operation. In the final step of the exposure completing process, the completed frame 𝑙 𝑡 is explicitly interpolated with neighbor frames 𝑙 𝑡 -1 , 𝑙 𝑡 +1 and optical flow 𝑓 0 𝑡 -1→𝑡 , 𝑓 0 𝑡 +1→𝑡 as input:EQUATIONEQUATIONwhere D 0 𝐼 represents the highest level of the exposure completing decoder D 𝐼 .As illustrated in the Fig. 1 , with the above design, we have obtained realistic exposure completing result for the challenging areas with motion, saturation and noise, providing accurate and complete exposure information for subsequent HDR rendering.Building upon the results of above process, we designed a coarse-tofine HDR rendering network based on completed features Φ 𝑡 and frame 𝑙 𝑡 . Taking the original features {Φ 𝑡 -1 , Φ 𝑡 , Φ 𝑡 +1 }, completed features Φ 𝑡 and warped neighbor features { Φ 𝑡 -1 , Φ 𝑡 +1 } as input, the HDR rendering decoder D 𝑅 reconstructs HDR features Ψ 𝑡 and optical flows {𝐹 𝑡 -1→𝑡 , 𝐹 𝑡 +1→𝑡 } in a coarse-to-fine manner, and finally obtains the coarse HDR frame ℎ 𝑐 𝑡 :EQUATIONEQUATIONEQUATIONwhere D 𝑘 𝑅 (𝑘 = 2, 3) stand for the 𝑘-th level of the HDR rendering decoder D 𝑅 . The warped neighbor features { 𝜙 𝑘 𝑡 -1 , 𝜙 𝑘 𝑡 +1 } help D 𝑅 to identify regions with poor alignment and subsequently obtain the refined optical flow {𝑓 𝑘 -1 𝑡 -1→𝑡 , 𝑓 𝑘 -1 𝑡 +1→𝑡 }. Based on the completed frame 𝑙 𝑡 and the coarse HDR frame ℎ 𝑐 𝑡 , we combine them with original LDR frames to obtain the final HDR rendering result ℎ 𝑡 . We adopted a blending network with U-net architecture from Xu et al. [40] . Thus, original and completed LDR frames {𝑙 𝑡 -1 , 𝑙 𝑡 , 𝑙 𝑡 +1 , 𝑙 𝑡 }, along with their corresponding frames {𝑥 𝑡 -1 , 𝑥 𝑡 , 𝑥 𝑡 +1 , 𝑥 𝑡 } in linear HDR domain, and coarse HDR frame ℎ 𝑐 𝑡 are fed into the blending network. The blending network calculates fusion weights 𝑊 = {𝑤 𝑖 |𝑖 = 0, 1, 2, 3, 4} for the input five HDR domain frames {𝑥 𝑡 -1 , 𝑥 𝑡 , 𝑥 𝑡 +1 , 𝑥 𝑡 , ℎ 𝑐 𝑡 } and obtains the final HDR rendering result ℎ 𝑡 through a weighted average based on the computed weights 𝑊 :EQUATIONFollowing Xu et al. [40] , we also fuse with the neighbor frames {𝑥 𝑡 -1 , 𝑥 𝑡 +1 } to provide information about static regions. As depicted in Fig. 1 , when encountering complex scenes involving both motion and saturation or noise, the fused exposure completion frames effectively provide the missing exposure information, thereby obtaining ghost-free and low-noise HDR results. In summary, based on completing the frame with missing exposure information, we finally achieve a high-quality HDR rendering process.We calculate the losses for the completed LDR features Φ 𝑡 and frame 𝑙 𝑡 , rendered HDR features Ψ 𝑡 and frame ℎ 𝑡 , and optical flows {𝐹 𝑡 -1→𝑡 , 𝐹 𝑡 +1→𝑡 }:The losses for images. The widely adopted L 1 loss is used to supervise the completed LDR frame 𝑙 𝑡 and rendered HDR frame ℎ 𝑡 , and is defined as follows:EQUATIONEQUATIONEQUATIONwhere 𝑙 𝑡 and ℎ 𝑡 are the corresponding ground truth for the completed LDR frame 𝑙 𝑡 and rendered HDR frame ℎ 𝑡 , respectively. The T (•) is a widely used function to map HDR frame to the tonemapped HDR domain, since HDR images are typically displayed after tone mapping. This simple differentiable 𝜇-law function T (•) is defined as follows:EQUATION)where 𝜇 is a hyperparameter.The losses for features. Feature space geometry loss proposed in [21] is employed to ensure that the intermediate features obtained from frame interpolation and HDR reconstruction can be more effectively refined to conform to geometrically structured features. The parameter shared encoder E is used to obtain corresponding pyramid featuresΦ 𝑡 = {𝜙 𝑘 𝑡 |𝑘 = 1, 2, 3} and Ψ 𝑡 = {𝜓 𝑘 𝑡 |𝑘 = 1, 2, 3} from the ground truth of completed LDR and rendered HDR frames. Then, the loss function for supervising the features of completed LDR and rendered HDR frames can be written as:EQUATIONEQUATIONEQUATIONwhere the L 𝑐𝑒𝑛 is census loss [32] and computed using the soft Hamming distance between census-transformed [46] feature maps, considering 3×3 patches in a channel-by-channel manner.The loss for optical flows. The HDR-domain alignment loss [40] is used hierarchically to supervise the learning process of optical flow, which is defined as follows:EQUATIONEQUATIONEQUATIONwhere W (•, •) denotes using optical flow to warp neighbor frames to the reference frame, U 𝑠 represents the bilinear upsampling operation with scale factor 𝑠. The mask 𝑚 𝑡 indicates well-exposed regions in reference frame. First, 𝑙 𝑡 is converted to YCbCr color space to extract luminance 𝑦. Then, 𝑚 𝑡 is defined as 𝛿 low < 𝑦 < 𝛿 high , where 𝛿 low and 𝛿 high are the low and high luminance thresholds, respectively. This allows the optical flow computation to focus more on regions with poor exposure in the reference frame. Total loss. Our total loss can be summarized as follows:EQUATION)In the alternate exposure setting with three exposures, five LDR frames {𝑙 𝑡 -2 , 𝑙 𝑡 -1 , 𝑙 𝑡 , 𝑙 𝑡 +1 , 𝑙 𝑡 +2 } are used as input to the network for HDR ℎ 𝑡 rendering. The five LDR frames are with corresponding exposure sequences {𝑒 𝑡 -2 , 𝑒 𝑡 -1 , 𝑒 𝑡 , 𝑒 𝑡 +1 , 𝑒 𝑡 +2 }. In this way, the reference frame has only one certain exposure 𝑒 𝑡 = 𝜖 3 . Specifically, the first frame 𝑙 𝑡 -2 and the fourth frame 𝑙 𝑡 +1 have the same exposure 𝑒 𝑡 -2 = 𝑒 𝑡 +1 = 𝜖 1 , while the second frame 𝑙 𝑡 -1 and the fifth frame 𝑙 𝑡 +2 also share the same exposure 𝑒 𝑡 -1 = 𝑒 𝑡 +2 = 𝜖 2 . Therefore, the 𝑡 through the coarse-to-fine HDR rendering process. A total of fifteen images, including the exposure completing frames and multiple original LDR frames, along with their corresponding frames in linear HDR domain, and the coarse HDR result, are fed into the blending network. This blending network calculates seven weights to fuse the seven HDR domain images in a weighted average manner and obtain the final HDR rendering result. More details are provided in supplementary materials. 
experiments 4.1 experimental setup: 
 Datasets. We utilize synthetic training data generated from the Vimeo-90K dataset [41] . To adapt the Vimeo90K dataset for HDR video reconstruction, we follow prior research [18] to convert the original data into LDR sequences with alternate exposures. To create the ground truth of completed LDR frames, we also generated LDR sequences with missing exposures in the same way. Our framework is tested on the Cinematic Video dataset [8] and DeepHDRVideo dataset [3] . The Cinematic Video dataset has two synthetic videos from indoor and outdoor scenes. The DeepHDRVideo dataset [3] contains both real-world dynamic scenes and static scenes with random global motion augmentation. The HDRVideo dataset [17] is employed solely for qualitative evaluation, as it lacks ground truth. Implementation details. We implement our approach using Py-Torch and conduct experiments on an NVIDIA RTX3090 GPU. We employ AdamW optimizer [20] with 𝛽 1 = 0.9 and 𝛽 2 = 0.999. The learning rate is set to 10 -4 . In our experiments, we set 𝛾 in Eq. 1 to 2.2 and 𝜇 in Eq. 14 to 5000. Following HDRFlow [40] , we set 𝛿 low to 0.2 and 𝛿 high to 0.8. The weighting hyperparameters 𝛼 and 𝛽 for the loss function in Eq. 21 are set to 0.01. Evaluation metrics. PSNR 𝑇 , SSIM 𝑇 and HDR-VDP-2 [30] are adopted as the evaluation metrics. PSNR 𝑇 and SSIM 𝑇 are computed on the tone-mapped images. HDR-VDP-2 is computed with the number of pixels per visual degree set to 30, which means the angular resolution of the image.Quantitative comparisons between our and other state-of-theart methods on the Cinematic Video [8] and DeepHDRVideo [3] datasets are shown in Table 1 and Table 2 state-of-the-art methods, our approach consistently achieves superior or comparable performance. Especially, our approach achieves state-of-the-art performance on Cinematic Video [8] dataset, outperforming the second-best method by 1.29dB and 0.59dB in terms of PSNR 𝑇 for the 2-exposure and 3-exposure settings, respectively. Qualitative comparisons are shown in Fig. 3 and Fig. 4 . we compare our NECHDR with the previous methods: Chen [3] , LAN-HDR [6] and HDRFlow [40] . Fig. 3 illustrates the results from scenes with saturation and motion on Cinematic Video dataset [8] and HDRVideo dataset [17] under the 2-exposure setting. Apart from our method, other methods tend to exhibit severe artifacts or detail loss when saturation and motion occur simultaneously. In such challenging scenarios, we achieve accurate and artifact-free exposure completion results for saturated regions by leveraging frame interpolation from neighboring frames. This enables us to fuse high-quality HDR results. And in Fig. 4 , we also show the results encountering noise and motion from the Cinematic Video dataset [8] and DeepHDRVideo dataset [3] . The scene on the left side of Fig. 4 is captured in low-light conditions, resulting in very low signal-to-noise ratio in the low-exposure frames. This can lead to very high noise levels in rendered HDR frames after tone mapping, which makes this scene particularly challenging. Specifically, methods based on optical flow [3, 40] tend to produce noisy artifacts, while attention-based method [6] exhibit more pronounced noise. Our method explicitly reconstructs the absent high-exposure frame with low noise at the current time stamp, achieving the best noise suppression, even with some regions having noise intensity lower than ground truth. The visualization results above explain why we achieve a significant performance improvement compared with other state-of-the-art methods. More qualitative comparisons in the 3-exposure setting are provided in the supplementary materials. Ablation Study. We conduct ablation experiments under the 2exposure setting on Cinematic Video dataset [8] and DeepHDRVideo dataset [3] , and the quantitative and qualitative results are shown in Table 3 and Fig. 5 , respectively. We devised a baseline that employs IFRNet [21] to utilize the neighbor frames to complete the middle time frame that with missing exposure information and achieve HDR results by simply fusing the completed LDR frame with the original LDR frames. The performance of this baseline is shown as "EC Baseline" in the first row in Table 3 . Another baseline directly renders HDR results based on IFRNet, which is the "HR Baseline" in second row in Table 3 . However, relying solely on either exposure completing or HDR rendering results is with limited performance. By adding the exposure completing decoder, we get a network (third row in Table 3 ) that contains both decoupled exposure completing and HDR rendering processes. Finally, through feeding the completed features and frames from exposure completing decoder into the process of HDR rendering, we achieve our NECHDR framework in the forth row in Table 3 . Based on the qualitative and quantitative results, we can see: (a) coupled exposure completing and HDR rendering processes benefit the quality Temporal consistency. We show the visual comparisons of temporal consistency in Fig. 6 . In Fig. 6 , we record a two-pixel-height line traversing all frames of a scene in the Cinematic Video dataset [8] over time and lay them out sequentially to form temporal profiles.Base on the illustration of temporal profiles, we can observe that the horizontal stripes exist in the temporal profiles of other methods. The horizontal stripes comes from the differences between adjacent frames, which represents the temporal inconsistency. In contrast, the horizontal stripes can hardly be observed in our temporal profiles, which means that our proposed method achieves better temporal consistency. Additional visual comparisons regarding temporal consistency can be found in the supplementary materials. 
conclusion: 
 In this paper, we implement the idea of exposure completing for neural HDR video rendering and propose the Neural Exposure Completing HDR (NECHDR) framework. The NECHDR leverages interpolation of neighbor LDR frames to complete missing exposures, providing a complete set of exposure information for each time stamp. This process of exposure completing creates a novel neural HDR video rendering pipeline, which can generate results of less noise and ghosting artifacts, thereby enhancing temporal consistency. Experimental results on multiple public benchmarks demonstrate the superiority of our NECHDR, which may shift the focus of researchers in this area to the exposure completing.  
: 
 is complete and consistent. Our approach involves interpolating neighbor LDR frames in the time dimension to reconstruct LDR frames for the absent exposures. Combining the interpolated and given LDR frames, the complete set of exposure information is available at each time stamp. This benefits the fusing process for HDR results, reducing noise and ghosting artifacts therefore improving temporal consistency. Extensive experimental evaluations on standard benchmarks demonstrate that our method achieves state-of-the-art performance, highlighting the importance of absent exposure completing in HDR video rendering. The code will be made publicly available upon the acceptance of this paper.• Computing methodologies → Artificial intelligence; Computer vision; Image and video acquisition; Computational photography. 
 
 Appendix:  


Decision: The authors argued that existing HDR video rendering method generate flickering results. To address this issue, they proposed to render HDR frames via completing the absent exposure information. It is believed this operation can benefit “the fusing process for HDR results, reducing noise and ghosting artifacts therefore improving temporal consistency”.  Generally, this method proposed is straightforward and the results (including the video presented in supplementary) seem good. This paper is also well organized and easy to read.

After rebuttal, the authors have well addressed most of the reviewers’ concerns. A remaining issue is its insufficient validation of temporal consistency. Since this paper claims temporal consistency as its major contribution while the other methods “generate flickering HDR results”, it is necessary to provide enough experiments to validate this statement. However, the paper fails to present any experiment. In Table 2 of the rebuttal file, the authors provide a rough comparison with three methods: Chen, LAN-HDR and HDRFlow, evaluated by ForVideoVDP.  However, it is still far less than all 8 methods compared in this work. Even in this table, the authors just proved that the proposed method generates better quality than its peers, but it does not support the above claim: the proposed method can guarantee temporal consistency while the others cannot. It is strongly recommended that the authors address this important consideration in their future work.

Reviews: review: paper_summary: paper_summary: While the idea of completing missing exposure frames to improve temporal consistency in neural HDR video rendering is new and interesting, there are several issues that need to be addressed.
strength_weakness: strength_weakness: 1. The proposed idea of completing missing exposure frames is well-motivated and inspired by traditional optical HDR imaging systems.
2. The experiments demonstrate the effectiveness of the proposed method in reducing ghosting and noise artifacts, leading to improved temporal consistency.
limitations: limitations: 1. The details of the technical approach are occasionally challenging to comprehend, particularly due to dense notations and equations that lack sufficient explanations.
2. It is recommended to evaluate the influence of varying exposure patterns (for instance, configurations involving more than three exposures) on the performance of the proposed method.
3. A comprehensive ablation study is necessary to delineate the contributions of different framework components, particularly focusing on the role of various loss functions.
4. Figures 3 and 4 reveal that the proposed method introduces noticeable halo artifacts around high-contrast regions, such as around the arms in Figure 4 (right).
5. Although the paper emphasizes temporal consistency, it lacks quantitative comparative analysis in the temporal domain, which is critical for substantiating the claims made.
6. Include a runtime analysis and discussion on the computational complexity of the proposed method.
paper_summary: paper_summary: 1. The paper proposed a method for HDR video reconstruction based on the exposure Completing. 
 the complete set of exposure information is useful to reduce noise and ghosting artifacts and then provides solid information for the HDR reconstruction. The paper shows the effectiveness of the proposed method, outperforming other promising HDR video models. It shows the ghosting artifacts and the noise can be effectively eliminated with the proposed method.
strength_weakness: strength_weakness: 1. The paper is well organized and easily followed.

2. It evaluates their proposed method in various datasets and show the effectiveness of the proposed method, which are convincing.
limitations: limitations: 1. The processing time, model complexity, and GFLOPs in the evaluation results should be provided. And also the comparisons with the previous methods also should be provided.

2. There is no loss for the coarse HDR frame \hat(h_t^c) when training. If there is a loss for the coarse HDR frame, would the performance be better?

3. The value for the losses for features. It is easy to understand the image loss and the optical flow loss. But what is the value for the features loss? Ablution study could be performed to verify the effectiveness of this loss.

4. In introduction section line 139-142, the authors claim that the previous alignment based methods rely on the reference image while the exposure of reference frame changes at every time stamp, which means that the reference frames of different exposures may have different defects. But the exposure completing and the coarse HDR image generation are also relied on the reference image. So this part could be refined.
paper_summary: paper_summary: This paper presents an HDR video reconstruction method based on alternating LDR exposures as input.
strength_weakness: strength_weakness: 1. The task is of practical importance.

2. The quantitative results look good.
limitations: limitations: 1. The authors emphasize temporal consistency as a key feature of the proposed method. However, the reviewer does not find any specific designs for promoting it in network architecture or loss function.

2. What are the key differences between the architectural designs and other DL-based HDR video reconstruction methods? It seems like the proposed method follows a two-stage approach, which has been extensively explored in the field.

3. The overall loss as the combination of many losses looks ad hoc.

4. The authors are encouraged to use more recent and perceptually relevant loss functions such as HDR-VDP-3, PU-encoded PSNR, and SSIM with and without CRF correction.

5. A psychophysical experiment is suggested to quantify the perceptual gains of the proposed method.

6. For video-based applications, the authors are encouraged t
paper_summary: paper_summary: This paper proposes an HDR video rendering approach named Neural Exposure Completing HDR (NECHDR) that explicitly predicts a missing exposure image before fusion to improve reconstructed HDR image.
strength_weakness: strength_weakness: The key modules include an exposure completing decoder that incorporates neighboring LDR frames to predict absent exposures at each time frame guided by predicted optical flows. This is coupled with an HDR rendering decoder that fuses the completed exposures with the original LDRs to produce the HDR output via a coarse-to-fine process. Experiments show NECHDR outperforming prior work in terms of HDR-VDP-2 metrics while reducing ghosting artifacts.
limitations: limitations: This work introduces a two-stage process for reconstructing HDR content from bracketed LDR images. Optical flow is utilized in reconstructing the coarse HDR image before it undergoes processing by the Blending Network. However, the blending network has to address misaligned LDR images and linear HDR images to mitigate ghosting artifacts in the final output, which may be inefficient.

The proposed training pipeline incorporates several loss functions; however, ablation studies analyzing the effectiveness of these loss functions are absent.

Crucial information such as the number of parameters, inference speed, or FLOPS (or equivalent metrics) of the entire network is necessary for evaluating the effectiveness of the proposed method. This information is currently missing for comparison with existing methods.

PSNR, SSIM and HDR-VDP-2 are image quality metrics.  A more comprehensive evaluation would benefit from assessing video quality metrics, such as FovVideoVDP or ColorVideoVDP.