Patent Publication Number: US-11398019-B2

Title: Light field system occlusion removal

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation application of International Patent Application No. PCT/CN2019/091346, filed on Jun. 14, 2019, which is based on and claims the benefit of priority to International Application under Patent Cooperation Treaty, No. PCT/CN2018/083350, filed on Apr. 17, 2018. The above-referenced applications are incorporated herein by their entirety. 
    
    
     TECHNICAL FIELD 
     The present invention relates to image processing, and more particularly, to light field image processing for removing occlusion. 
     BACKGROUND 
     Over the last decades, intelligent video surveillance systems have becoming increasingly important due to the huge number of cameras used for security and surveillance in public areas. Directly monitoring the huge amount of information collected by human is impractical. Hence, reliable automatic processing of the collected video information is in high demand. Facial recognition is one of the advanced technologies that can be used to process such information, as it could be used to spot individual faces within a crowd and prevent crime in real-time. However, crowd occlusion is one of the issues that affects the accuracy of facial recognition. 
     For instance, in security surveillance, the capability of seeing a subject of interest through occluders can be crucial. However, refocusing technology alone often fails to adequately remove or blur an occluding object for sufficient identification and/or recognition of the occluded object at the full potential. In certain circumstances, this may determine whether a suspected terrorist or a dangerous criminal can be identified or not. 
     Therefore, there are needs in the field of light field image rendering for improved methods and systems to create more effective see-through effects. 
     SUMMARY 
     To address issues in the prior art, embodiments of the present invention provide a method of processing light field images for occlusion removal in images and videos captured by light field camera systems. 
     In accordance with an embodiment of the present invention, a method of image processing is provided, the method comprising: capturing a plurality of camera views using a plurality of cameras; generating a depth map for each camera view; calculating a target view on a focal plane corresponding to a virtual camera; projecting a pixel on the target view to the plurality of camera views to obtain a plurality of projected pixels; setting a weighting of a projected pixel to 0 if the depth value of the projected pixel on the first camera view is less than a virtual distance between the pixel on the target view and the corresponding camera, and to 1 if otherwise; and blending the projected pixels in accordance with the weightings to generate a refocused target view. 
     Preferably, the pixel on the target view comprises a color value or a grayscale value, and blending the projected pixel comprises calculating the color value or a grayscale value of the pixel on the target view by weighted averaging a color value or a grayscale value of the projected pixels on the plurality of camera views. 
     Preferably, the method further comprises capturing a plurality of depth maps using a plurality of depth sensors. 
     Preferably, generating a depth map for a camera view comprising warping a depth image captured by a depth sensor in accordance with intrinsic matrices and extrinsic matrices of the depth sensor and camera. 
     In accordance with an embodiment of the present invention, warping the depth image generated by a depth sensor according to a warping algorithm comprises forwarding warping the depth image generated by the depth sensor to obtain an intermediate depth image; filling holes in the intermediate depth image; and backward waring the intermediate depth image to obtain the depth map for the camera view. 
     In accordance with an embodiment of the present invention, the plurality of cameras are configured to capture a plurality of videos, and a refocused target video corresponding to the virtual camera is generated in real time. 
     Preferably, the plurality of video comprise 30 frames per second. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       To better illustrate the technical features of the embodiments of the present invention, various embodiments of the present invention will be briefly described in conjunction with the accompanying drawings. 
         FIG. 1  is an exemplary schematic diagram of a light field camera system with depth sensors in accordance with an embodiment of the present invention. 
         FIG. 2  is an exemplary schematic diagram for calibrating the camera system in accordance with another embodiment of the present invention. 
         FIG. 3  is an exemplary schematic diagram showing the geometry indictors for the refocusing algorithm in accordance with yet another embodiment of the present invention. 
         FIG. 4  an exemplary schematic diagram showing the occlusion removal algorithm in accordance with yet another embodiment of the present invention and two prior art techniques. 
     
    
    
     DETAIL DESCRIPTION OF THE EMBODIMENTS 
     In accordance with embodiments of the present invention, a light field camera array system with wide baseline is introduced to remove occlusion in real-time.  FIG. 1  is an exemplary schematic diagram of a light field camera system with depth sensors in accordance with an embodiment of the present invention. As shown in  FIG. 1 , the system contains several color image sensors and depth sensors. All of these sensors have a synchronous mechanism. The depth information can be generated by various depth sensors, such as Kinect and Realsense sensors. Alternatively, the 3D information can also be produced by a light field color image system by conducting stereo matching. Depth information is used for occlusion detection and occluded region in-painting. A 3D point cloud on the focal plane is generated. 
     Compared to traditional pin-hole camera system, photographs from multiple angles of light filed provide extra information to in-painting the occluded regions. The light field camera system allows us to see “around” the occluder by filling in unseen areas of the photograph. With the real-time light field render, this system can provide reliable occlusion-free footage for face recognition analysis. The occlusion removal mechanism is consisted in the following steps. 
     1. Camera System Calibration 
     First of all, we need to calibrate our camera system. We use traditional calibration methods to calibrate cameras and obtain their intrinsic and extrinsic matrices.  FIG. 2  is an exemplary schematic diagram for calibrating the camera system in accordance with another embodiment of the present invention. 
     As shown in  FIG. 2 , in step  201 , a checkerboard pattern is randomly moved in front of the camera system. In step  202 , the system captures sample images. In step  203 , the intrinsic and distortion parameters of each camera are estimated. In step  204 , the relative pose between each camera and the checkerboard is estimated. In step  205 , the absolute pose of each camera is initialized. Finally, in step  206 , all parameters are globally optimized to minimize re-projection error. 
     2. Depth Maps and Color Images Capture 
     When the system is calibrated, we can start processing pipeline for each frame we captured. All color images and depth maps are captured synchronously, which ensures that all image from different sensors are captured at the same time. 
     3. Depth Projections 
     Next, we need to prepare depth maps for each viewpoint, which generate depth maps for all color images. 
     Depth images for each viewpoint can be obtained by warping the depth images from depth sensors according to the corresponding transformation between RGB cameras and depth sensors. The algorithm consists of three major components: a forward warping, a backward warping, and a hole filling process. 
     We do depth projection for each depth sensor. Here we demonstrate the warping algorithm for a single depth sensor. Let d(x, y) be the depth image obtained from the depth sensor, d i (x i , y i ) be the expected depth image of i-th viewpoint, K k  and K i  be the intrinsic matrices and C k  and C i  be the extrinsic matrices of depth sensor and i-th RGB camera respectively. We can formulate this relation as: 
     
       
         
           
             
               
                 
                   
                     ( 
                     
                       
                         
                           
                             
                               ω 
                               i 
                             
                             ⁢ 
                             
                               x 
                               i 
                             
                           
                         
                       
                       
                         
                           
                             
                               ω 
                               i 
                             
                             ⁢ 
                             
                               y 
                               i 
                             
                           
                         
                       
                       
                         
                           
                             
                               ω 
                               i 
                             
                             ⁢ 
                             
                               d 
                               i 
                             
                           
                         
                       
                       
                         
                           
                             ω 
                             i 
                           
                         
                       
                     
                     ) 
                   
                   = 
                   
                     
                       
                         K 
                         i 
                       
                       ⁢ 
                       
                         
                           
                             C 
                             i 
                           
                           ⁡ 
                           
                             ( 
                             
                               
                                 K 
                                 k 
                               
                               ⁢ 
                               
                                 C 
                                 k 
                               
                             
                             ) 
                           
                         
                         
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             
                               
                                 ω 
                                 ⁢ 
                                 x 
                               
                             
                           
                           
                             
                               
                                 ω 
                                 ⁢ 
                                 y 
                               
                             
                           
                           
                             
                               
                                 ω 
                                 ⁢ 
                                 d 
                               
                             
                           
                           
                             
                               ω 
                             
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         T 
                         i 
                       
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               x 
                             
                           
                           
                             
                               y 
                             
                           
                           
                             
                               d 
                             
                           
                           
                             
                               1 
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     By removing the third column and row of T i , we obtain the forward warping equation given as: 
     
       
         
           
             
               
                 
                   
                     ( 
                     
                       
                         
                           
                             
                               ω 
                               i 
                             
                             ⁢ 
                             
                               x 
                               i 
                             
                           
                         
                       
                       
                         
                           
                             
                               ω 
                               i 
                             
                             ⁢ 
                             
                               y 
                               i 
                             
                           
                         
                       
                       
                         
                           
                             ω 
                             i 
                           
                         
                       
                     
                     ) 
                   
                   = 
                   
                     
                       
                         H 
                         i 
                       
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               x 
                             
                           
                           
                             
                               y 
                             
                           
                           
                             
                               1 
                             
                           
                         
                         ) 
                       
                     
                     + 
                     
                       d 
                       ⁢ 
                       
                         a 
                         ⇀ 
                       
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
     We can rewrite this equation as: 
     
       
         
           
             
               
                 
                   
                     ( 
                     
                       
                         
                           
                             
                               ω 
                               i 
                             
                             ⁢ 
                             
                               x 
                               i 
                             
                           
                         
                       
                       
                         
                           
                             
                               ω 
                               i 
                             
                             ⁢ 
                             
                               y 
                               i 
                             
                           
                         
                       
                       
                         
                           
                             ω 
                             i 
                           
                         
                       
                     
                     ) 
                   
                   = 
                   
                     
                       H 
                       i 
                     
                     ⁡ 
                     
                       ( 
                       
                         
                           
                             
                               
                                 ω 
                                 i 
                                 * 
                               
                               ⁢ 
                               
                                 x 
                                 i 
                                 * 
                               
                             
                           
                         
                         
                           
                             
                               
                                 ω 
                                 i 
                                 * 
                               
                               ⁢ 
                               
                                 y 
                                 i 
                                 * 
                               
                             
                           
                         
                         
                           
                             
                               ω 
                               i 
                               * 
                             
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
             
               
                 
                   
                     with 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             
                               
                                 ω 
                                 i 
                                 * 
                               
                               ⁢ 
                               
                                 x 
                                 i 
                                 * 
                               
                             
                           
                         
                         
                           
                             
                               
                                 ω 
                                 i 
                                 * 
                               
                               ⁢ 
                               
                                 y 
                                 i 
                                 * 
                               
                             
                           
                         
                         
                           
                             
                               ω 
                               i 
                               * 
                             
                           
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ( 
                       
                         
                           
                             x 
                           
                         
                         
                           
                             y 
                           
                         
                         
                           
                             1 
                           
                         
                       
                       ) 
                     
                     + 
                     
                       
                         d 
                         · 
                         
                           H 
                           i 
                           
                             - 
                             1 
                           
                         
                       
                       ⁢ 
                       
                         
                           a 
                           ⇀ 
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
     Given the above formulations, we propose the following warping algorithm: 
     a. Forward warp the depth image d to obtain an intermediate depth image according to the equation (4); 
     b. Fill holes in the intermediate depth image; and 
     c. Backward warp the intermediate depth image to obtain d i  according to equation (3) (the depth image of the desired viewpoint) 
       FIG. 3  is an exemplary schematic diagram showing the geometry indictors for the refocusing algorithm in accordance with yet another embodiment of the present invention. 
     4. Occlusion Removal 
       FIG. 4  an exemplary schematic diagram showing the occlusion removal algorithm in accordance with yet another embodiment of the present invention and two prior art techniques. For each pixel in virtual camera, the process shown in  FIG. 4  is performed to remove occlusion. 
     4.1. Calculate 3D Point on the Focal Plane 
     Like traditional light field refocusing algorithm, we also have a focal plane which represents the estimated geometry of scene. Focal plane is located in virtual camera&#39;s coordinate system, and has a distance away from virtual camera. 
     4.1.1. Get 3D Coordinate 
     For each pixel of target view, we assume a planar proxy and apply ray tracing technique to calculate its 3D coordinate. The focal plane is perpendicular to the view direction of the target camera and the distance between them is z. Given a pixel p v =[u,v] T  in the target view, we calculate its 3D coordinate P v =[x v ,y v ,z v ] T  as 
     
       
         
           
             
               
                 
                   
                     ( 
                     
                       
                         
                           
                             x 
                             v 
                           
                         
                       
                       
                         
                           
                             y 
                             v 
                           
                         
                       
                       
                         
                           
                             z 
                             v 
                           
                         
                       
                     
                     ) 
                   
                   = 
                   
                     z 
                     ⁢ 
                     
                       
                         K 
                         v 
                         
                           - 
                           1 
                         
                       
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               u 
                             
                           
                           
                             
                               v 
                             
                           
                           
                             
                               1 
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     where K v  is the intrinsic of the target camera. 
     4.2. Project 3D Point to Viewpoint 
     4.2.1 Project P onto C i  and Obtain p i    
     We then project the 3D point P v  to all the captured views, from which we generate all the candidates for later pixel blending. Let P i =[x i , y i , z i ] T  and p i =[u i , v i ] T  be the corresponding 3D coordinate and 2D pixel coordinate in the i-th viewpoint, 
     
       
         
           
             
               
                 
                   
                     ( 
                     
                       
                         
                           
                             x 
                             i 
                           
                         
                       
                       
                         
                           
                             y 
                             i 
                           
                         
                       
                       
                         
                           
                             z 
                             i 
                           
                         
                       
                     
                     ) 
                   
                   = 
                   
                     
                       C 
                       i 
                     
                     ⁢ 
                     
                       
                         C 
                         v 
                         
                           - 
                           1 
                         
                       
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               
                                 x 
                                 v 
                               
                             
                           
                           
                             
                               
                                 y 
                                 v 
                               
                             
                           
                           
                             
                               
                                 z 
                                 v 
                               
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
             
               
                 
                   
                     p 
                     i 
                   
                   = 
                   
                     
                       ( 
                       
                         
                           
                             
                               u 
                               i 
                             
                           
                         
                         
                           
                             
                               v 
                               i 
                             
                           
                         
                       
                       ) 
                     
                     = 
                     
                       π 
                       ⁡ 
                       
                         ( 
                         
                           
                             K 
                             i 
                           
                           ⁢ 
                           
                             P 
                             i 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     where π(⋅) is the normalization operator to map the 3D homogeneous coordinate to the 2D pixel coordinate. 
     4.2.2. Calculate Virtual Distance d i ′ 
     By taking the z coordinate from P i , we obtain the z-depth d i ′ between the 3D point and the i-th camera.
 
 d   i   ′=z   i   (8)
 
     4.2.3. Obtain the Depth Value d i    
     At the same time, we retrieve the warped depth value d i (u i ,v i ) and the color c i (u i ,v i ), which can be referred to section 3. 
     4.3. Occlusion Detection 
     We compare these two depth values. If d i (u i ,v i )&lt;d i ′, we become confident that the expected object in the target view is occluded in the i-th viewpoint, therefore we abandon the pixel color from i-th viewpoint by setting the weighting m i  to zero. Otherwise we accept the pixel color by setting the weighting m i  to one. 
     4.4. Occluded Region in-Painting 
     The final pixel blending calculates the pixel color for p v   
     
       
         
           
             
               c 
               ⁡ 
               
                 ( 
                 
                   u 
                   , 
                   v 
                 
                 ) 
               
             
             = 
             
               
                 
                   Σ 
                   i 
                 
                 ⁢ 
                 
                   m 
                   i 
                 
                 ⁢ 
                 
                   
                     c 
                     i 
                   
                   ⁡ 
                   
                     ( 
                     
                       
                         u 
                         i 
                       
                       , 
                       
                         v 
                         i 
                       
                     
                     ) 
                   
                 
               
               
                 
                   Σ 
                   i 
                 
                 ⁢ 
                 
                   m 
                   i 
                 
               
             
           
         
       
     
     with this pixel selecting scheme, the refocused target image is guaranteed to avoid any obstructions in front of the focal plane. 
     After one frame is processed, we repeat these procedures above and obtain frames (video) with occlusions removed.