Video Bokeh Rendering: Make Casual Videography Cinematic

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 16:11:01

Abstract: Bokeh is a wide-aperture optical effect that creates aesthetic blurring in photography. However, achieving this effect typically demands expensive professional equipment and expertise. To make such cinematic techniques more accessible, bokeh rendering aims to generate the desired bokeh effects from all-in-focus inputs captured by smartphones. Previous efforts in bokeh rendering primarily focus on static images. However, when extended to video inputs, these methods exhibit flicker and artifacts due to a lack of temporal consistency modeling. Meanwhile, they cannot utilize information like occluded objects from adjacent frames, which are necessary for bokeh rendering. Moreover, the difficulties of capturing all-in-focus and bokeh video pairs result in a shortage of data for training video bokeh models. To tackle these challenges, we propose the Video Bokeh Renderer (VBR), the model designed specifically for video bokeh rendering. VBR leverages implicit feature space alignment and aggregation to model temporal consistency and exploit complementary information from adjacent frames. On the data front, we introduce the first Synthetic Video Bokeh (SVB) dataset, synthesizing authentic bokeh effects using ray-tracing techniques. Furthermore, to improve the robustness of the model to inaccurate disparity maps, we employ a set of augmentation strategies to simulate corrupted disparity inputs during training. Experimental results on both synthetic and real-world data demonstrate the effectiveness of our method. Code and dataset will be released.

Main paper: 
 introduction: 
 The bokeh effect is a popular photographic technique that creates aesthetic blur in out-of-focus areas, commonly used in movie shooting to highlight emphasized subjects. While DSLR cameras naturally produce a shallow depth of field effect, they are expensive, lack portability, and require expertise, making them less accessible for casual users. Smartphones, on the other hand, are widely used in our daily lives. However, limited by their small aperture sizes, they often produce videos with deep depth of field, lacking the aesthetically pleasing bokeh effects. Our work aims to bridge the gap between smartphones and cameras, by rendering refocusable videos from all-in-focus videos, making casual videography cinematic.Compared to video bokeh rendering, single-image-based bokeh rendering has been extensively studied, beginning with the classical rendering approaches [1, 19, 24, 27, 33, 34] that determine the blur radius from disparity map and focal point. However, disparity maps often contain errors, especially in regions with discontinuous depth, resulting in artifacts in the rendering results. To address this issue, neural rendering methods [16, 17, 28, 31] have been proposed to generate authentic bokeh rendering effects by learning from data.Nevertheless, these methods are designed for single-image inputs, and applying them directly to each frame of the video often results in the following issues: (i) Flicker between frames arises due to the absence of spatial-temporal constraints; (ii) The inability to handle the disocclusion phenomenon (Fig. 2 ). The disocclusion phenomenon occurs when occluded objects in an all-in-focus image reappear during bokeh rendering with a large aperture. In a video sequence, the potential exists to recover information about occluded objects near the edge, which is necessary for bokeh rendering. Moreover, disparity maps for video sequences often contain flaws and temporal flickers, even when generated by state-of-theart depth estimation methods. However, current single-image-based methods lack robustness to disparity maps, making them susceptible to producing artifacts. While a previous work, RVR [34] , has proposed a video bokeh rendering system, they utilize optical flow to smooth the inputs of the disparity map. However, they still achieve bokeh rendering frame-by-frame using single-image-based models, thereby leaving inter-frame interactions and correlations untouched, as shown in Fig. 1 .To address the aforementioned issues, from a model design perspective, we present an end-to-end model for video bokeh rendering, termed Video Bokeh Renderer (VBR). The primary distinction between VBR and previous single-image-based bokeh rendering models lies in the capability to enforce temporal consistency and recover occluded information through a temporal fusion mechanism. To handle high resolution and large blur sizes, VBR adopts a coarseto-fine framework comprising two sub-modules: a coarse bokeh generator and an iterative bokeh refiner. The coarse bokeh generator initially renders bokeh effects at a low resolution, and then the iterative bokeh refiner improves the result gradually. To harness information from adjacent frames for achieving aesthetically pleasing and temporally consistent video bokeh rendering results, we introduce the Temporal Fusion Block (TFB) into our model. Due to camera and object motions, identical spatial positions may not correspond to the same object across frames. Thus, in TFB, we first use deformable convolutions to align multiple features from adjacent frames and then utilize stacked convolution layers to merge the aligned features. Moreover, since disparity maps inevitably contain errors and flaws, we employ augmentation strategies during training to mimic inaccurate disparity maps. This enhances the robustness of our model to inaccuracies in disparity maps.From the data perspective, due to the lack of paired all-in-focus and bokeh video data for training, we elaborate a video bokeh data synthesizing pipeline to generate the first video bokeh rendering dataset called Synthetic Video Bokeh (SVB). The training set of SVB comprises 3, 000 videos, each containing 16 frames, resulting in a total of 144, 000 frames. When generating video bokeh data, we mimic different shooting techniques, such as adjusting the focal plane, focusing on a moving target, or modifying the size of the aperture. Additionally, we synthesize a test set for validation purposes, comprising 300 videos and a total of 14, 400 frames.We evaluate our methods on both real-world data and the synthetic test set in the SVB dataset. The results demonstrate the advantages of the proposed video bokeh renderer. Our method achieves state-of-the-art results in rendering quality and temporal consistency on the synthetic test set. Moreover, our method shows robustness to inaccurate disparity maps and more stable performance compared with other methods. To further assess the quality of bokeh rendering from a subjective aspect, we also conduct a user study involving 55 participants on 20 videos collected from the internet. The comparison with other models reveals a preference among users for the rendering effects produced by our model.In summary, our main contributions are as follows:• We introduce VBR, the video bokeh rendering model that first leverages information from multiple frames to generate refocusable videos from all-in-focus videos. • We propose the first synthetic video bokeh rendering dataset (SVB) using our ray-tracing-based video bokeh data synthesis method. • Experimental results demonstrate that our method produces rendering results with better temporal consistency, enhanced edge rendering effects, and increased robustness to lowquality disparity maps. 
related work 2.1 bokeh rendering: 
 Image-space-based methods [1, 16, 24, 28] operate on single RGB-D input in a post-processing manner and are typically divided into classical rendering methods [19, 27, 34] and neural rendering methods [16, 28, 31] . Classical rendering methods. Classical rendering methods [11, 19, 27, 34] determine the blur radius at different spatial positions based on the depth map and focal position. SteReFo [1] proposes a layer-based rendering method, dividing the input image into layers and applying a fixed convolution kernel to blur each layer. Although classical methods are efficient and can generate authentic bokeh effects in regions with smooth depth variations, they often produce artifacts in areas with discontinuous depth, such as edges.Neural rendering methods. To address the artifacts at edges, neural rendering methods [16, 17, 28, 31] have been proposed to generate bokeh effects by learning from data. BokehMe [16] proposes a hybrid network to combine the classical renderer and neural renderer, leveraging the advantages of both techniques to address artifacts commonly encountered at edges in classical methods. To tackle the disocclusion phenomenon (Fig. 2 ), MPIB [17] employs an inpainting network to recover information of occluded objects in the all-in-focus image. However, relying on an inpainting network to fill in occluded information can introduce instability, potentially causing the model to collapse when the inpainting module fails.While RVR [34] is the first work to introduce the notion of video bokeh rendering, it uses optical flow to smooth the disparity maps of input and renders the video frame by frame using a layer-based classical method, overlooking the relationship between adjacent frames during rendering. In contrast, our approach addresses these challenges by leveraging information from neighboring frames to generate temporally consistent bokeh rendering results and mitigate artifacts at edges.Different from single-image bokeh rendering, achieving a satisfactory video bokeh rendering requires not only high rendering performance per frame but also temporal consistency. Moreover, bokeh rendering with a large aperture needs occluded information, further emphasizing the necessity of temporal fusion. Actually, leveraging and fusing information from adjacent frames has been a long-standing video understanding and processing [22, 25, 26] . 3D convolution [22] is used to learn spatial-temporal features. However, since adjacent frames are usually misaligned due to camera and object motions, directly fusing unaligned features may result in artifacts. Transformer-based temporal fusion [3, 23, 25] can bypass the feature alignment due to the global receptive field, but they often entail high computational costs. Another stream of work involves leveraging deformable convolutions to align frames before fusing them. Deformable convolutions [6] are widely employed for implicit feature alignment, as the learned offsets can overcome the fixed sampling position limitations in traditional convolutions and offer more flexible modeling capabilities. TDAN [26] introduces deformable convolutions into the network to align the reference frame and target frames at the feature level. Inspired by TDAN [26] , subsequent works [4, 5, 29] start to align and fuse information from adjacent frames. Although they have shown promising ability in feature alignment and fusion, the heavy complexity and workload brought by the addition priors and complex structures remain undesirable. Additionally, inaccurate optical flow estimation can further compromise feature alignment. In this work, we also modulate cross-frame relationships with deformable convolution, but opt to discard the cascade structure and additional priors for a simpler and lighter-weight implementation. 
disocclusion phenomenon: 
 The disocclusion phenomenon refers that the occluded objects in an all-in-focus image can "re-appear" under a large aperture, i.e., the occluded object can also contribute to the imaging process, indicating that the occluded information is necessary for rendering bokeh effects. As depicted in Fig. 2 , the first row illustrates the imaging process of an all-in-focus image with a pinhole camera. As all the light must travel through the pinhole in a straight line, the blue object is blocked by the green foreground object. However, things are different when it comes to a large aperture, as shown in the second row. The rays from the occluded blue object can reach the green dot on the image plane because light can travel along other routes within the large aperture lens. Hence, perfectly achieving bokeh rendering from a single image is an ill-posed problem, as the occluded information is necessary but absent, resulting in artifacts at edges. Fortunately, with a video sequence, there is potential to restore the occluded information from other frames. Therefore, our model utilizes the temporal fusion block to integrate information from multiple frames to deal with the disocclusion phenomenon, mitigating the artifacts at edges.The defocus map provides the blur radius of each pixel, where any neighboring pixels within this radius contribute to the rendering result of the center pixel. The blur radius is determined by the depth and the focus point [27, 33] . Given the disparity of a pixel, its blur radius can be calculated byEQUATIONwhere 𝑑 denotes the disparity of the pixel, 𝑑 𝑓 represents the disparity of the focal position, 𝐾 indicates the degree of blur, and 𝑟 denotes the blur radius of the pixel. Each element in the defocus map corresponds to the size of the blur radius. The defocus maps 𝑆 can be calculated by the disparity maps 𝐷 and control parameters 𝐾 and 𝑑 𝑓 :EQUATION 
method 4.1 overview: 
 As illustrated in Fig. 3 , the core of the video bokeh renderer is a cascaded model comprising: (i) a coarse bokeh generator that produces low-resolution coarse bokeh rendering results, (ii) an iterative VBR adopts a cascaded architecture to generate high-resolution results. First, a coarse bokeh generator renders bokeh effects in low resolution. Subsequently, the iterative bokeh refiner takes the all-in-focus frames, defocus maps, and the coarsely rendered results as inputs and gradually refines the coarse results into high-resolution outputs. To utilize information from multiple frames, we employ a temporal fusion block at the bottleneck of the sub-modules to fuse features from adjacent frames.refiner that progressively upscales and refines the coarse bokeh rendering results to high resolution. To leverage information from adjacent frames, we design a temporal fusion block positioned at the bottleneck of both the coarse bokeh generator and the iterative bokeh refiner. Our framework generates a bokeh video 𝐵 from an all-in-focus video 𝐼 , a disparity video 𝐷, and controlling parameters. The controlling parameters comprise blur parameters 𝐾 and refocus disparities 𝑑 𝑓 , which respectively determine the overall amount of blur and the focal plane of each frame. Instead of directly inputting the disparity map and controlling parameters into the network, we utilize defocus maps to integrate the controlling parameters and disparity information, serving as inputs to the network.Coarse Bokeh Generator. The coarse bokeh generator initially downsamples both the all-in-focus frames and defocus maps, simultaneously decreasing the numerical range of the defocus maps. This process yields rendering results with equivalent blur levels but at a lower resolution. The downsampling rate, denoted as 𝛼 (0) , depends on maximum absolute value within the defocus map 𝑆 and the maximum blur capability 𝑅 𝑚 of the module:𝛼 (0) = 𝑚𝑖𝑛(1, 𝑅 𝑚 𝑚𝑎𝑥 (|𝑆 |)) .(EQUATIONTo preserve low-level details effectively, we integrate pixel unshuffle and pixel shuffle [20] within the sub-modules for downsampling and upsampling. Following the encoding of the input sequence into high-level features, the coarse bokeh generator utilizes a proposed temporal fusion block to integrate temporal information extracted from adjacent frames. Subsequently, the fused spatialtemporal features are passed into the decoder to obtain the coarsely rendered bokeh effects. We denote the output of the coarse bokeh generator as 𝐵 (0) . Iterative Bokeh Refiner. To achieve aesthetic rendering results in high resolution, we design the iterative bokeh refiner to gradually enhance the rendered output. At the 𝑡-th iteration, the iterative bokeh refiner improves and upscales coarse bokeh result 𝐵 (𝑡 -1) to double the resolution. We also use the video sequence 𝐼 and the disparity maps 𝑆 as additional inputs to provide information. Before each iteration, we will resize the video sequence 𝐼 and the disparity maps 𝑆 to twice resolution of the coarse bokeh result 𝐵 (𝑡 -1) . The scale factor can be determined as follows:EQUATIONIn the first iteration, the coarse bokeh rendering results 𝐵 (0) generated by the coarse bokeh generator are inputted into the iterative bokeh refiner. In each subsequent iteration, the bokeh rendering results 𝐵 (𝑡 -1) from the iterative bokeh refiner become the inputs for the next iteration. The iteration ceases when the 𝛼 (𝑡 ) in Eq. 4 equals 1, resulting in the final rendering result with the same resolution as the input.To enforce temporal consistency and mitigate disocclusion, we design a temporal fusion block to leverage information from adjacent frames. Due to the movement of the objects and the camera, the same spatial position in different frames may not correspond to the same object. Therefore, in the temporal fusion block, we employ deformable convolution [6] to align features from adjacent frames before applying stacked convolution layers to fuse them. The detailed architecture is illustrated in Fig. 4 . The temporal fusion block utilizes features extracted by the encoder as inputs. Given the features 𝑓 𝑖 -1 , 𝑓 𝑖 , 𝑓 𝑖+1 , we concatenate the target frame and reference frames and pass them forward through convolutional layers to obtain offsets, which are then used to align the features:EQUATIONwhere [•] denotes the concatenation operation, 𝑔 denotes the convolutional layers responsible for predicting the offsets, and △𝑂 𝑡 +𝑖 represents the predicted offsets used for feature alignment. Then, the adjacent frames are aligned by deformable convolutions with the predicted offset:EQUATIONwhere DCN denotes the deformable convolutional layers utilized to align the features. The temporal fusion block serves the purpose of aligning and merging information from adjacent frames, thereby reducing interframe flicker to maintain temporal consistency. Additionally, it enables the utilization of information from multiple frames to recover details in occluded regions, mitigating artifacts at edges caused by the disocclusion phenomenon.We adopt the widely-used l1 loss and gradient loss as the spatial loss, where 𝑛 represents the position of the frame:EQUATIONAs for the temporal loss, we adopt the basic relation loss [7] to supervise temporal consistency:EQUATIONwhere 𝐵 * , 𝐵 indicate the ground-truth bokeh video and predicted video, respectively. The temporal constraint ensures temporal consistency by enforcing that the change values between consecutive frames in the rendering results match those in the ground truth. The total loss is defined by Eq. 9, where 𝑇 represents the size of the sliding window,EQUATION)where 𝜆 is the weight factor and is empirically set to 5. 
dataset: 
 Due to the challenges in capturing paired video data using DSLR cameras, currently, there is no dataset for video bokeh rendering. Existing datasets [8, 17, 34] for bokeh rendering only comprise image pairs and lack temporal information. To acquire paired bokeh videos for model training, we employ a synthetic method to generate pairs of all-in-focus and bokeh videos. In each video, we randomly select a background and some foreground objects, such as people or animals. Then, each foreground object will move along a random trajectory in the 3D world. Since the 3D coordinates of all the foreground objects and the background are known, following [17] , we use the ray-tracing-based method to accurately produce bokeh effects in each frame, ensuring precise relation between adjacent frames. SVB Dataset Construction. We utilize a landscape dataset [21] as the background and choose foreground objects from alpha matting datasets [9, 12-15, 18, 32] . To ensure data quality, we manually filter poorly annotated foreground objects and retain 1, 044 objects. The training set of SVB dataset consists of 3, 000 videos with accurate depth and bokeh rendering results, each containing 16 frames. In each video, 4 foreground objects are randomly chosen. To evaluate different models, we also synthesize a test set comprising 300 videos.To assess the ability of models to render complex scenes, we increase the number of foreground objects to 9, adding complexity to the test set. All the videos in the SVB dataset have a resolution of 256 × 256 pixels. In addition to the movement of the objects, we also mimic the process of changing camera parameters found in real-life scenarios, such as altering the focal plane and adjusting aperture sizes. Specifically, we achieve three types of bokeh techniques: (i) maintaining the focus target while varying the degree of blur, (ii) keeping the degree of blur constant while adjusting the focal plane, and (iii) maintaining the degree of blur while varying the focal plane from the farthest to the nearest or vice versa. These three control parameters encompass most bokeh effects observed in our daily lives. Please refer to the supplementary materials for detailed information and examples of the dataset. 
experiments 6.1 implementation details: 
 Data Augmentation. To enhance the robustness of our model, we introduce a set of augmentation strategies to simulate corrupted disparity maps during training. Specifically, we randomly apply Gaussian blur, dilation, erosion, and elastic transformation [2] to the disparity maps.Training. We train the coarse bokeh generator and the iterative bokeh refiner sequentially. Each training sample is a video sequence with 4 consecutive frames, and the training batch size is set to 8. All the modules are trained for 10 epochs with Adam optimizer [10] . The learning rate is set to 10 -4 . We conduct all the experiments with Pytorch framework on two NVIDIA A6000 GPUs. Evaluation Metrics. We evaluate both the video rendering quality and temporal consistency of different methods. Following [1, 16] , we use PSNR and SSIM as the video rendering quality metrics. Moreover, we use PSNR 𝑜𝑏 and SSIM 𝑜𝑏 to evaluate the rendering quality at edges following [17] . For temporal consistency, we apply the basic relation loss defined in Eq. 8 on the whole video as:1EQUATIONwhere 𝑁 denotes the length of the video.To verify the performance of our proposed method, we compare the effectiveness and the robustness of our model, VBR, with two classical bokeh rendering methods: RVR [34] and SteReFo [1] , and four neural rendering methods: DeepFocus [31] , DeepLens [28] , MPIB [17] and BokehMe [16] on the proposed synthetic test set. We found that DeepFocus [31] sometimes collapses under parameter settings with large blur size. Therefore, we resize the inputs of DeepFocus [31] to a range that the network can handle, and then upscale it back to the original resolution for evaluation. This modified method is marked with a superscript †. Quantitative experiment. As shown in Table . 1, our model achieves the highest PSNR and SSIM scores, indicating better rendering quality. Moreover, our method shows better rendering quality at edges compared with BokehMe, according to PSNR 𝑜𝑏 and SSIM 𝑜𝑏 . These results further demonstrate that extracting occluded information Table 2 : Levels of disparity corruption. The scale shift is designed to mimic the disparity's flicker on the overall scale. We randomly choose a scale factor within [0.9, 1.1] and multiply the disparity with this scale factor.Level Gaussian blur Erosion/Dilation Elastic Transformation [2] Scale ShiftI ✓ II ✓ ✓ III ✓ ✓ ✓ IV ✓ ✓ ✓ ✓from adjacent frames can help mitigate artifacts at edges, e.g., addressing the disocclusion phenomenon proposed in Sec. 3.1. Our method also achieves the best temporal consistency, with a 33% relative improvement compared to BokehMe. This result further reveals the benefits of temporal fusion in reducing flicker between frames to generate temporal consistent results. Robustness. To validate the model's robustness to inaccurate disparity maps, we test various methods with impaired disparity maps as inputs. Further details regarding the disparity corruption can be found in Table . 2. As illustrated in Fig. 5 , VBR outperforms other methods in all metrics under different levels of corrupted disparity settings, indicating its superior robustness to disparity maps. Additionally, in Fig. 6 , even in zones with inaccurate disparity, our method can still generate aesthetic bokeh effects.Due to the subjective nature of perception towards video bokeh rendering results, we conduct a user study on real-world videos to better evaluate different methods from the subjective perspective.We collect 20 all-in-focus videos from the internet with a resolution of 1080 × 1920, featuring people, natural landscapes, and other diverse themes. Since there is no ground-truth of the disparity, we obtain the disparity maps by a depth prediction model [30] .Quantitative results. We use different methods to render the videos under the same controlling parameters. During the test, All-in-Focus RVR [34] SteReFo [1] DeepLens [28] DeepFocus † [31] Disparity MPIB [17] BokehMe [16] Ours GT Ours vs. RVR [34] 87.63% / 12.37% Ours vs. SteReFo [1] 83.16% / 16.84% Ours vs. DeepLens [28] 71.71% / 28.29% Ours vs. MPIB [17] 82.32% / 17.68% Ours vs. BokehMe [16] 64.04% / 35.96% each participant is shown 2 videos at a time, comprising two bokehrendered videos produced by our approach and a method randomly chosen from RVR [34] , SteReFo [1] , DeepLens [28] , MPIB [17] , BokehMe [16] , in random order. Participants are asked to choose the method with more consistent and aesthetic bokeh effects or choose none if it was difficult to judge. Additionally, each participant needs to complete at least 10 tests before submitting results. The user study involves 55 participants, and the results in Table . 3 indicate that our method is consistently preferred by most users.Qualitative results. To intuitively show why our method is more frequently preferred by the users, we showcase some examples in Fig. 7 . In the first row, the focal plane is on the woman. Other methods mistakenly blur the edges of the hand and lipstick as they are considered as the background in the disparity map(The disparity maps are shown in supplementary materials). Instead, the generated video of VBR perfectly preserves the details on the edges of the hand and the lipstick, demonstrating its robustness against inaccurate disparity maps. In the second row, the focal plane is on the boy in the mirror. All the other methods produce abnormal artifacts at edges, especially on the boy's hair. Our model generates correct blur results while keeping the edge of the focused object clear. The single-image-based methods can not produce satisfying results as they either overlook the disocclusion phenomenon or utilize an inpainting network (MPIB [17] ) to address it in an unstable manner. The results further verify that leveraging information from adjacent frames benefits our model in mitigating the disocclusion phenomenon. See supplementary materials for more visualization and video results.In this section, we explore the effectiveness of each component in the VBR framework, from modules tailored to temporal consistency to augmentation strategies for robustness. Effects of Temporal Designs. Since the main difference between the image-based and the video-based rendering methods lies in their capability to leverage the information from neighboring frames, we explore whether our design can truly benefit video-based rendering.we implement a plain model with only spatial loss (row 1). Under this setting, our model degrades to a single-image-based model, neglecting information from adjacent frames. As illustrated in Table . 4, through comparing row 1 with row 3 and row 2 with row 5, utilizing the temporal fusion block enables the model to achieve superior render quality and temporal consistency. This result demonstrates the effectiveness of the temporal SteReFo [1] DeepLens [28] MPIB [17] BokehMe [16] Ours fusion block in mitigating flicker between frames and addressing disocclusion issues. Furthermore, we conduct an ablation study on the deformable convolution (row 4 vs. row 5) within the temporal fusion block, replacing it with standard convolution to assess the necessity of alignment. The results show that unaligned features pose challenges for integrating information from adjacent frames, highlighting the importance of feature alignment.To evaluate the effectiveness of temporal loss, from Table . 4, one can see that incorporating temporal consistency loss enhances performance across all metrics (row 3 vs. row 5) and can reach a synergy with the temporal fusion block. To intuitively explore the significance of each component, we provide visual examples in Fig. 8 . As shown in the figure, the full model yields superior rendering outcomes, especially in edge regions (zoomed-in areas), highlighting the necessity of using the temporal fusion block to leverage temporal information. Furthermore, by constraining temporal consistency with the temporal loss, the output exhibits more evenly distributed blur circles. Effects of disparity augmentation. As depicted in Fig. 5 , disparity augmentation strategy plays a crucial role in enhancing the model's robustness to inaccurate disparity maps. Please refer to the supplementary materials for a more in-depth ablation study on disparity augmentation. 
conclusion: 
 In this work, we propose a novel framework named Video Bokeh Renderer (VBR) to generate aesthetic bokeh effects from all-in-focus videos. The framework consists of two sub-modules: a coarse bokeh generator and an iterative bokeh refiner. To mitigate rendering flicker between frames and address the disocclusion phenomenon, we introduce a temporal fusion block into our sub-modules to align and fuse features from adjacent frames. Experimental results show that our method produces rendering results with better temporal consistency, enhanced edge rendering quality, and increased robustness to low-quality disparity maps. This work also contributes a video bokeh rendering dataset, SVB, to alleviate data shortages. To the best of our knowledge, VBR is the first framework to consider temporal information in video bokeh rendering. We hope our work can serve as a solid baseline and inspire further research. Limitations and future work. Although the synthetic bokeh rendering dataset captures authentic bokeh effects, which can be utilized for training video bokeh models, it still exhibits a gap compared to real-world datasets. In our future work, we intend to explore additional methods to generate more realistic datasets. 
 
 Appendix: PRELIMINARYIn this section, we illustrate some preliminary concepts related to video bokeh rendering. First, we analyze the disocclusion phenomenon employing a camera model. Then, we introduce the defocus map, which is used as input in our model to control the amount of blur at different spatial positions. 


Decision: The paper is well written and clearly presents the proposed dataset and rendering approaches.
Out of the 4 reviewers, 1 proposes a definite accept, 2 a weak accept, 1 a borderline accept.
Overall, all are very positive.
Only reviewer 4 is more critical, with criticism mainly addressing the aspects of (1) the topic being niche and (2) the resolution addressed in the paper not being practically realistic.
The authors provide a comprehensive rebuttal. Some of the additional analyses, such as the one regarding resolution-stability, should be included in the camera-ready version of the paper. 
Apparently not yet including additional information in the final paper version is what slightly reduces my "confidence" in my "accept" vote. In case this information from the rebuttal shall be included, I see very good reasons to accept the paper. 
The concern that the paper is "niche" is not shared by me. In my view, as Bokeh and DoF rendering are highly relevant aspects in image and video capture and appeal perception. Also, a conference like ACM MM can benefit from some diversity of topics, which this quite high-quality paper can contribute to.

The positive points given by the reviewers can be summarized as follows:

(feW3)

- topic of interest to researchers in related areas.
- paper has lot of potential contributions: dataset, content analysis,  explorative user study. 
- Moreover,  authors intend to conduct a decent experiment, except for rationale for selecting the baseline algorithms.

(nBQj)

- "Video Bokeh Renderer" (VBR) is new approach that extends bokeh rendering from static images to dynamic video content
- addresses  challenge of maintaining temporal consistency across frames
- technically sound
- clear explanation of how the model utilizes both the spatial and temporal information to generate high-quality bokeh effects in videos
- paper is well-structured and clearly written, making complex concepts accessible and understandable.

(Jct2)

- The experiments are sufficient and convincing.
- Writing, figures and tables are satisfying.
- The first video bokeh rendering dataset is constructed, and the proposed solution has excellent performance.

(36VE)

- Proposed approach is sound and carries sufficient novelty, especially the temporal fusion component is interesting.
- Experiments and evaluation section is very comprehensive, consisting both of quantitative and qualitative (human preference) comparison studies.
- Evaluation shows that the proposed approach gives better performance than the state of the art, both quantitatively and qualitatively.

For the negative points, most have been addressed in the rebuttal.
As stated above, it is somewhat unclear to which extent the explanations in the rebuttal will be implemented in the camera-ready version of the paper.
At this stage, this is the only concern I have with the paper.

Hence, in case of final acceptance of the paper, authors are strongly encouraged to make sure that all points mentioned in the rebuttal and further reviewer comments are considered for the camera-ready version of the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel framework named Video Bokeh Renderer (VBR) to generate aesthetic bokeh effects from all-in-focus videos. Experimental results show that VBR produces rendering results with better temporal consistency, enhanced edge rendering quality, and increased robustness to low-quality disparity maps. Besides, this paper also contributes a video bokeh rendering dataset, SVB, to alleviate data shortages.
strength_weakness: strength_weakness: It is a topic of interest to researchers in related areas. The paper has a lot of potential contributions: the dataset, the content analysis, and the explorative user study. Moreover, the authors intend to conduct a decent experiment, except for the rationale for selecting the baseline algorithms.
limitations: limitations: Overall, the context part is easy to read; however, several improvements can be made to increase the quality and contribution of the paper. My detailed comments are as follows. 
1.	Page 1, Line 115, this paper aims to bridge the gap between smartphones and cameras, making casual videography cinematic. However, the method proposed will introduce additional overhead of smartphones. Conducting performance testing on power consumption will make this work more convincing.
2.	Page 5, Line 540-545, this paper contributes a dataset consisting of 3000 videos. Please briefly describe the categorizations of these videos and include some graphs for demonstration.
3.	This paper selects MPIB and BokehMe as the latest baselines which are proposed in 2022. Is there any related method in 2023 and 2024?
4.	Page 6, Section 6.2, when conducting quantitative analysis of the results, providing the percentage of improvement helps to enhance the understanding of the effectiveness of this method and make it more convincing.
5.	Page 6, Section 6.2, this paper compares the results on the SVB dataset of the proposed method with baselines. However, the analysis of the results is not comprehensive. For example, can the better robustness of this model be explained from the perspective of its model design?

Minor comments
1.	Page 8, Table 4 and Figure 8, please maintain consistent capitalization of the usages of “W/o”. Please explain the meanings of w/o briefly.
2.	The capitalization of subsection titles is inconsistent. For example, "2.1 Bokeh Rendering" is different from "3.1 Disocclusion phenomenon".
paper_summary: paper_summary: This paper introduces the "Video Bokeh Renderer" (VBR), a model that transforms all-in-focus videos into ones with shallow depth of field to simulate cinematic background blur. Unlike traditional methods requiring professional equipment, VBR can achieve similar effects using common smartphones. The model addresses issues like flickering and edge artifacts in video by enhancing temporal consistency and utilizing inter-frame information. Additionally, the paper presents a synthetic video bokeh dataset (SVB) to overcome training data shortages.
strength_weakness: strength_weakness: The "Video Bokeh Renderer" (VBR) is a new approach that extends bokeh rendering from static images to dynamic video content, addressing the challenge of maintaining temporal consistency across frames; 
The methodology described is technically sound, with a clear explanation of how the model utilizes both the spatial and temporal information to generate high-quality bokeh effects in videos; 
The paper is well-structured and clearly written, making complex concepts accessible and understandable.
limitations: limitations: The reliance on the synthetic dataset (SVB) for training and validation might limit the model’s performance in real-world scenarios where lighting, motion, and other environmental factors are more complex and less predictable than in synthetic datasets; 
The paper could explore further how well the model generalizes across different types of videos involving various motion dynamics, backgrounds, and lighting conditions. There may be a need for more extensive testing on diverse datasets to validate the robustness of the model under varied conditions; 
The methodology and demographic details of the user study might require more depth to ensure that the preferences and perceptions are representative of the target audience.
paper_summary: paper_summary: This paper focuses on video bokeh rendering and constructs a novel datasets. The theoretical modeling and network are detailed, and sufficient experiments prove the superior performance of the VBR.
strength_weakness: strength_weakness: 1. The experiments are sufficient and convincing.
2. Writing, figures and tables are satisfying.
3. The first video bokeh rendering dataset is constructed, and the proposed solution has excellent performance.
limitations: limitations: 1. How does the method generate more obvious bokeh effects outside the training set. Whether the actual maximum blur effect is limited by the training set?
2. Reporting computational costs can further refine the manuscript.
3. More works in relation to bokeh rendering should be discussed. [1] Zheng B, Chen Q, Yuan S, et al. Constrained predictive filters for single image bokeh rendering[J]. IEEE Transactions on Computational Imaging, 2022, 8: 346-357. [2] Mandl D, Mori S, Mohr P, et al. Neural Bokeh: Learning Lens Blur for Computational Videography and Out-of-Focus Mixed Reality[C]//2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 2024: 870-880.
4. This work is beneficial to the Bokeh community, looking forward to codes and datasets.
paper_summary: paper_summary: The authors propose a novel method to generate bokeh effects from all-in-focus videos (+ disparity maps) , consisting of a coarse bokeh generator and an iterative refiner module. A novel temporal fusion block is added for better temporal consistency (mitigating flicker effects) and handling of disocclusions.
strength_weakness: strength_weakness: - Proposed approach is sound and carries sufficient novelty, especially the temporal fusion component is interesting.
- Experiments and evaluation section is very comprehensive, consisting both of quantitative and qualitative (human preference) comparison studies.
- Evaluation shows that the proposed approach gives better performance than the state of the art, both quantitatively and qualitatively.
limitations: limitations: - The topic of bokeh effect rendering is quite a niche topic in the multimedia field.
- The synthetically generated video dataset (SVB dataset) has a very small resolution of only 256x256 pixels.
   This is not representative of modern video content, which is at least Full HD.
   Therefore, it is not clear whether the quantitative results from table 1 (utilizing the SVB dataset) apply also for modern video content.