Using Depth Mapping to realize Bokeh effect with a single camera Android device

In this paper we seek to produce a bokeh effect with a single image taken from an Android device by post processing. Depth mapping is the core of Bokeh effect production. A depth map is an estimate of depth at each pixel in the photo which can be used to identify portions of the image that are far away and belong to the background and therefore apply a digital blur to the background. We present algorithms to determine the defocus map from a single input image. We obtain a sparse defocus map by calculating the ratio of gradients from original image and reblured image. Then, full defocus map is obtained by propagating values from edges to entire image by using nearest neighbor method and matting Laplacian. Based on the defocus map, foreground and background segmentation is carried out. We show that by enhancing the blurriness of the background while preserving the foreground, bokeh effect is achieved. Finally, we propose an algorithm to add cartoon-like effect on the input image. Keyword: bokeh effect; matting Laplacian; defocus map; cartoon; Android device I.INTRODUCTION Artistic enhancement of photographs is a very popular field of research with several applications that are common place in the world of smartphone photography. Recently researchers have sought to produce an effect known as Bokeh in smartphone photos using digital image processing techniques. Bokeh is the shallow-depth of field effect which blurs the background of portrait photos (typically) to bring emphasis to the subject in the foreground. Figure 1 shows the same image with and without Bokeh effect. Figure 1. image without and with Bokeh effect Bokeh effect is usually achieved in high end SLR cameras using portrait lenses that are relatively large in size and have a shallow depth of field. It is extremely difficult to achieve the same effect (physically) in smart phones which have miniaturized camera lenses and sensors. However, the latest iPhone 7 has a portrait mode which can produce Bokeh effect thanks to the dual cameras configuration. To compete with iPhone 7, Google recently also announced that the latest Google Pixel Phone can take photos with Bokeh effect, which would be achieved by taking 2 photos at different depths to camera and combining then via software. There is a gap that neither of two biggest players can achieve Bokeh effect only using a single image from a single smartphone camera. In this project we seek to fill this gap. We are planning to produce a bokeh effect with photos taken using an Android device by post processing the photos within the Android device. The photos can also come from Photo Stream. The new images with Bokeh can be saved into Photo Stream. At the core of Bokeh effect production in smartphone photography is depth mapping. A depth map (an estimate of depth at each pixel in the photo) is used to identify portions of the image that are far away and belong to the background and therefore apply a digital blur to the background. As mentioned early, Bokeh effect is typically present in portrait photos. For non-portrait photos, we are seeking to apply cartoon effect on the foreground as a way of artistic enhancement. What’s more, other ways of artistic enhancement like changing the background of images can also be performed based on the depth map. Figure 2: examples of artistic enhancement by taking advantage of depth map II.LITERATURE REVIEW In recent years, there are a variety of methods were proposed to recover the defocus map from a single image. Elder and Zucker [1] used the first and second order derivatives of the input image to determine the size of the Gaussian blur kernel. Bae and Durand [2] extended their work by adding a bilateral filter to remove outliers and estimate the blur amount of the whole image. Using defocus map, they increased the blurriness of the blurry regions and kept the in focus region sharp. Levin et al. [3] converted a statistical model to recover defocus map with a modified camera. Tai and Brown [4] used local contrast prior to measurements into a defocus map and then apply MRF propagation to refine the estimated defocus map. The advantage of their method is that it does not rely on frequency decomposition of the input image. It only compares the local gradients with local contrast. Zhuo and Sim [5] generated a full defocus map with good accuracy by using matting interpolation. They also discussed a method to remove the blur texture ambiguity. In our paper, first, we use a Gaussian blur kernel to reblur the image and calculate the ratio between the gradients. Secondly, we adopt nearest neighbor and matting Laplacian to determine the full defocus map. The defocus map is segmented into foreground and background. Then, bokeh effect is generated by applying blur kernel to further blur the background while keep the foreground sharpen. Finally, we proposed an algorithm based on bilateral filtering and edge detection to add cartoon-like effect on the foreground of the input image. III.ALGORITHM 3.1 Depth Estimation Defocus Blur concept When a picture is taken with a camera, portions of the image look sharp and portions of the image look blurry depending of the depth of the scene. Portions of the scene that lie on the focal plane will look sharp while portions of the image that are away from the focal plan will appear blurry. The blurriness is associated with defocus when the scene is imaged on camera sensor. Light rays from points in the scene that are in-focus will reach points on the sensor whereas light rays from points that are out-of-focus will be spread across multiple points on the sensor within an area known as circle of confusion [5]. The size (diameter) of the circle of confusion is associated with the amount of defocus for a given point in the scene and is given by: